Paper Title: Contrastive Learning with Dynamic Localized Repulsion for Brain Age Prediction on 3D Stiffness Maps.




Flat-band Fulde–Ferrell–Larkin–Ovchinnikov State from Quantum Geometry Discrepancy
    K. T. Law
    August 5, 2024
==================================================================================



§ ABSTRACT 
In the field of neuroimaging, accurate brain age prediction is pivotal for uncovering the complexities of brain aging and pinpointing early indicators of neurodegenerative conditions. Recent advancements in self-supervised learning, particularly in contrastive learning, have demonstrated greater robustness when dealing with complex datasets. However, current approaches often fall short in generalizing across non-uniformly distributed data, prevalent in medical imaging scenarios. To bridge this gap, we introduce a novel contrastive loss that  adapts dynamically during the training process, focusing on the localized neighborhoods of samples. 

Moreover, we expand beyond traditional structural features by incorporating brain stiffness—a mechanical property previously underexplored yet promising due to its sensitivity to age-related changes.
This work presents the first application of self-supervised learning to brain mechanical properties, using compiled stiffness maps from various clinical studies to predict brain age. 


Our approach, featuring dynamic localized loss, consistently outperforms existing state-of-the-art methods, demonstrating superior performance and laying the way for new directions in brain aging research.




§ INTRODUCTION


Aging causes significant changes in the structure and function of the brain. Magnetic Resonance Elastography (MRE) has recently emerged as a non-invasive technique to measure mechanical brain properties <cit.>, such as stiffness μ, that is, the resistance of a viscoelastic material to an applied harmonic force. Studies have shown a promising age sensitivity of whole-brain stiffness measurements <cit.>, surpassing established neuroimaging age biomarkers, such as volume measurements <cit.>. With advancing MRE protocols, improvements in resolution have enabled the study of stiffness changes in more localized brain regions, revealing regional age-related variability <cit.>. Furthermore, clinical studies in patients with neurodegenerative diseases have revealed stiffness alterations exceeding those observed in healthy aging <cit.>. However, current methods are limited to region-wide averages and thus fail to exploit the rich information available in stiffness maps, which can be accessed through nonlinear relationships at a voxel level.


Brain age prediction leverages neuroimaging data through machine learning by casting it as a regression problem, wherein models are trained on healthy samples to establish a baseline trajectory for aging. Regression is a statistical method used to predict a continuous outcome—such as age—based on input data, in this case, brain imaging data. This approach is particularly promising for identifying deviations from normal aging processes that might indicate neurological conditions.

This approach has traditionally been  applied to structural brain features using T1-weighted MRI <cit.>. More recently, the scope has expanded to include
features obtained from resting-state fMRI and diffusion imaging 
are also being examined 
<cit.>.

Additionally, its potential in detecting and the prognosis of neurodegenerative disorders such as Alzheimer's Disease (AD) is gaining significant attention <cit.>.
In parallel, modeling approaches have evolved from traditional regression methods to state-of-the-art self-supervised learning <cit.>. 
Inspired by advancements in computer vision, self-supervised learning techniques, particularly contrastive learning methods, have been effectively adapted for predicting brain age from structural MRI scans <cit.>. Contrastive learning, a technique that learns by comparing pairs of examples, enhances model performance by focusing on the fine distinctions between similar cases.


Despite their potential, 

current methods often struggle with generalization, particularly across datasets characterized by non-uniform distributions.
To address this limitation, 

we introduce a novel contrastive loss that specifically focuses on localized sample neighborhoods. This method is distinctively designed to adapt dynamically across different stages of training, thereby enhancing model performance where traditional approaches falter.
Furthermore, considering the demonstrated age sensitivity of mechanical properties compared to structural brain properties, this study is pioneering. It represents the first application of contrastive learning to brain stiffness maps, highlighting a new direction in neuroimaging research. Our contributions are summarized next.


[fill=wine]2pt  We introduce a dynamic localized repulsion approach, specifically tailored for the framework of contrastive regression learning. Our new method is designed to address the major challenge of limited generalizability in medical image analysis, particularly within datasets characterized by non-uniform distributions. By concentrating on these challenging distributions, our approach not only enhances the robustness of the models but also performance. 





[fill=wine]2pt We apply our method to brain stiffness maps, introducing a novel, age-sensitive modality for brain age prediction that offers  insights into neurological aging processes. 

[fill=wine]2pt


To validate our novel contrastive regression loss, we have aggregated the largest multi-study dataset of brain stiffness from healthy controls, enabling comprehensive and robust conditions. We conducted experiments and demonstrated higher performance than existing techniques.





§ RELATED WORK


[fill=deblue]2pt Regression Task Regression  is a statistical technique that establishes a relationship between a set of independent variables (X) and a dependent variable (Y), through a function f: X → Y. Regression specifically addresses continuous variables, with Y taking values in ℝ. Known for its robust effectiveness, regression has made significant impacts across various domains including <cit.>. Recent decades have seen a surge in research aimed at advancing deep regression techniques, significantly enhancing their performance and applicability – for example, the works of that <cit.>.  Another interesting family of techniques, which is the focus of this work, falls within the contrastive learning family. This perspective is particularly interesting due to its ability to enhance learning by emphasizing differences between samples, thereby improving model robustness and generalization <cit.>.


[fill=deblue]2pt Contrastive Learning. To construct semantically rich and structured representations, contrastive learning has become a widely adopted method for self-supervised representation learning. This technique involves differentiating between similar (positive) and dissimilar (negative) pairs of data samples x_i and x_k, with the goal of adjusting the distances between representations in the embedding space—bringing similar items closer and pushing dissimilar ones apart <cit.>.

Contrastive learning initially gained popularity through its applications in computer vision, where it demonstrated significant improvements in tasks such as image classification and object detection. Early methods like SimCLR <cit.> introduced a simple yet effective framework that maximized agreement between differently augmented views of the same data sample. This approach utilized a contrastive loss function to bring representations of augmented pairs (positive pairs) closer while pushing apart representations of different samples (negative pairs).

Building on these foundations, a notable advancement was the introduction of supervised contrastive learning <cit.>, which extended the contrastive loss to leverage label information, treating all samples with the same label as positives. This approach enhanced the performance of models by incorporating supervised information into the self-supervised learning framework.

As we shift focus from classification to regression problems, the distinction between positive and negative pairs transitions to a continuous spectrum. This shift necessitates the model's ability to discern varying degrees of similarity, represented as s_i,k = sim(f(x_i), f(x_k)), beyond mere categorical differentiation. Particularly, we are interested in brain age prediction, which requires precise modeling of age-related changes in brain structure and function.
In response to the challenge of integrating continuous labels such as age, recent advancements propose strategies such as the Y-Aware loss <cit.>, which softens the boundary between positive and negative samples. Similarly, the work of that <cit.> proposed the Threshold and Exponential losses, which adjust the strength of alignment and repulsion based on the similarity between continuous labels. Another work introduced the Rank-N-Contrast loss <cit.>, which employs a comparative ranking strategy among samples. This method ranks samples based on their similarity to a given anchor, creating a ranking-based continuous spectrum of positive and negative pairs.



§ PROPOSED TECHNIQUE

This section details the proposed dynamic localized repulsion strategy, a novel approach within contrastive learning tailored for regression tasks.



 §.§ Problem Setup

The primary challenge in brain age modeling lies in accurately mapping high-dimensional brain imaging data to a continuous age variable. Traditional contrastive learning methods are limited in their capacity to handle the subtle variations in brain stiffness associated with aging due to their global approach. Our proposed technique introduces a dynamic, localized strategy, focusing on progressively capture

age-related features effectively.

Formally, in brain age modeling, our objective is to train a neural network capable of accurately mapping brain images x ∈ X to their corresponding target ages y ∈ℝ. The model comprises two key components: a feature encoder f(·): X →ℝ^d, which transforms brain images into a representative feature space, and an age predictor g(·): ℝ^d →ℝ, tasked with estimating the age from these features.



 §.§ Dynamic Localized Repulsion

The dynamic localized repulsion technique is a cornerstone of our methodology, designed to optimize the contrastive learning framework specifically for the regression tasks inherent in brain age modeling. This section outlines the detailed mechanics of this technique, including model components, operational processes, and algorithmic strategies.



To enhance the precision of contrastive regression learning, we introduce a dynamic localized repulsion approach that progressively explores varied scales within the embedding space, as depicted in Figure <ref>. This methodology systematically adjusts the selection of repulsion candidates, taking into account both their proximity and the evolutionary stage of training. The selection process for repulsed samples is defined by:




    NN(x_i;epoch) = { x_k | x_k  is among the  NN_nb (epoch)  nearest neighbors of  x_i  based on  d(x_k, x_i) }


This expression delineates the set of samples subject to repulsion, identified by their distance d(·). Our approach progressively narrows the scope of nearest neighbors involved in the repulsion process, thereby focusing the learning on increasingly localized vicinities within the embedding space. This adaptive mechanism is governed by two critical hyperparameters: the final count of nearest neighbors,NN_nb, final, representing the ultimate scope of repulsion at the end of training, and the decrement frequency, NN_step size, which specifies the interval of epochs for adjustments in the neighbor count, as detailed in Algorithm <ref>.





In datasets that show non-uniform distributions, especially those with multi-modal characteristics, it is common to find some target areas oversampled and others undersampled. This scenario is typical in neuroimaging datasets (see Fig. <ref>). Our dynamic localized strategy is designed to address this issue. It starts by segregating distinct groups and then explores the differences within those groups more deeply. This process is illustrated in Fig. <ref>. Our approach aims to reveal more coherent representations throughout the dataset.





Following the methodology proposed by <cit.>, we utilize kernel functions to determine the degrees of positiveness,  w_i,k = K(y_i - y_k) where 0 ≤ w_i,k≤ 1, between pairs of samples. This metric is defined by the function K(y_i - y_k) and reflects the age similarity between two samples. A higher value of w_i,k suggests that the samples are close in age, prompting the algorithm to minimize the distance between their representations in the embedding space. Conversely, a lower value indicates a significant age difference, thus triggering the repulsion mechanism to increase the distance between their embeddings.




Each sample x_i in a batch is compared against every other sample x_k, with the embeddings being adjusted based on their relative age similarities. The selection of nearest neighbors for this comparison, NN_nb(epoch), dynamically changes as the training progresses. Initially, a larger set of neighbors is considered to establish broad relationships. As training advances, this number is progressively reduced to focus on more immediate and relevant interactions, thus refining the learning towards localized features. The per-sample dynamic localized repulsion loss reads:


    l^i_DynLocRep = -∑_kw_i,k/∑_t w_i,tlog( exp(s_i,k)/∑_x_t ∈ NN(x_i;epoch)exp(s_i,t(1-w_i,t))).



This expression calculates the contribution of each sample pair to the overall loss. It normalizes these contributions by the sum of positiveness weights for all comparisons within the batch, adjusting the impact of each pair based on their age-related similarity. The softmax function is then applied to these normalized and adjusted similarity scores s_i,k, which are recalculated for each dynamically defined nearest neighbor set.

What is the Intuition Behind Our Dynamic Localized Repulsion? We address the challenge of non-uniform data distributions in neuroimaging datasets. Traditional learning models often fail to adequately distinguish between different age groups when these groups are unevenly represented in the training data. Our approach refines this by adjusting embeddings dynamically. This is done via (<ref>) that represents the per-sample loss function in a contrastive learning framework, which aims to dynamically adjust the embeddings based on age-related similarities. The essence of this equation lies in its ability to modulate the degree of repulsion or attraction between samples within the same batch based on their age proximity, which is quantified by w_i,k the positiveness weight.  This formulation allows for adaptive learning where the focus is progressively shifted toward more challenging or informative pairs, potentially those that are not well-aligned in age, thus encouraging the model to learn finer distinctions as training progresses.

The total loss ℒ_DynLocRep is then calculated over all samples as anchors:


    ℒ_DynLocRep   = ∑_i l^i_DynLocRep
       = -∑_i∑_k ≠ iw_i,k/∑_t w_i,tlog( exp(s_i,k)/∑_x_t ∈ NN(x_i;epoch)exp(s_i,t(1-w_i,t))).


This aggregated loss function is key for structuring the embedding space optimally, ensuring that samples with similar ages are located closer together while those with significant age differences are distanced. Such a configuration not only enhances the accuracy of age prediction but also improves the model’s ability on complex neuroimaging datasets.





§ EXPERIMENTAL RESULTS

This section details the complete set of experiments conducted to evaluate the proposed technique.



 §.§ Data Description

We have assembled a dataset of 311 three-dimensional (3D) brain stiffness maps obtained from healthy control subjects (HC). These data were sourced from multiple clinical studies, all of which utilized highly similar Magnetic Resonance Elastography (MRE) protocols. This ensures consistency across the collected data. For detailed information on the individual studies and the data collection methods, please refer to Table <ref>. All datasets were collected in accordance with ethical standards, under protocols approved by the respective local institutional review boards.









The age distribution of all samples from each study is illustrated in Fig. <ref>. Following this, stiffness maps, depicted in Fig. <ref>, were processed to enhance quality and uniformity. Initially, each map underwent skull stripping using Freesurfer <cit.>, a step to isolate the brain tissue from non-relevant anatomical structures. Subsequently, we applied a bias field correction to remove intensity gradients that could affect subsequent analyses.

To address the issue of data heterogeneity across different studies, we performed affine registration of the images to the MNI152 template. This registration was conducted at an isotropic resolution of 2 mm^3 using ANTs <cit.>, ensuring consistent orientation and scale among all datasets. Finally, we normalized the quantitative stiffness images, setting their mean to zero and standard deviation to one across the dataset. 




 §.§ Evaluation Protocol


Our evaluation strategy involved a 3D ResNet-18 model (33.5M parameters), which was pre-trained on the openBHB dataset <cit.>, containing over 5000 T_1 3D MRI brain images from multiple scanning sites, using the best reported method from the OpenBHB challenge <cit.>. To enhance generalization to our stiffness dataset, we utilized quasi-raw images, ensuring uniform image pre-processing, and downsampled the structural MRI images to 2 mm^3 isotropic resolution. The pre-trained ResNet-18 underwent full fine-tuning (i.e. updating all weights) on our brain stiffness dataset, following an 80:20 train-test split, over 50 epochs and a batch size of 32 using the Adam optimizer. This optimization included an initial learning rate of 1× 10^-4, decreased by 0.9 every 10 epochs, and a weight decay of 5×10^-5. Hyperparameters NN_nb, final and NN_step size were optimized via random search across 30 iterations. Our implementation is based on Barbano et al. <cit.>. Our models were trained using an NVIDIA A100-SXM-80GB GPU. Following the training of the representations, we employed a Ridge Regression estimator <cit.> on top of the frozen encoder  to predict age. As an evaluation metric, we calculated the mean absolute error (MAE) on the test set, averaging the results across five random seeds.






 §.§ Results and Discussion


We begin by evaluating the representations of stiffness maps learned using our dynamic localized repulsion loss as in (<ref>) with Manhattan distance, NN_nb, final=14 and NN_step size=1, selected via random search, against those using current state-of-the-art contrastive regression losses.

Table <ref> illustrates the effectiveness of our proposed Dynamical Localized Repulsion approach in the context of brain age prediction from stiffness maps, as evidenced by the Mean Absolute Error (MAE) metric. When comparing our method to existing state-of-the-art contrastive regression losses, our approach demonstrates superior accuracy in predicting brain age. In a closer look, we can observe that Rank-N-Contrast <cit.> shows the highest MAE, suggesting it may be less adept at capturing the nuanced patterns within the data necessary for precise age prediction.
Y-Aware and Exponential <cit.> losses show improvements over Rank-N-Contrast. These methods appear to better align with the underlying age-related changes in brain stiffness but still fall short compared to our approach. Threshold <cit.> loss offers a competitive performance, yet it does not achieve the same level of accuracy as our method. This could indicate that while it handles some aspects of the data variability effectively, it might not fully capture the localized age-related changes as our method does. Our proposed loss achieves the lowest MAE, 
underscoring its ability to more accurately model the age-related changes in brain stiffness. This suggests that our method's focus on localized sample neighborhoods and its dynamic adaptation during the training process significantly contribute to its improved performance.






To investigate the role of the distance norm for nearest neighbor selection of our dynamic localized repulsion approach, we conduct an ablation study, examining the impact of various distance norms on the model's performance, as detailed in Fig. <ref>. The method shows robustness regarding the choice of distance norm. Our findings reveal that the Manhattan norm achieves the lowest MAE of 3.724±0.220 years, outperforming the Cosine (MAE = 3.748±0.142 years), Euclidean (MAE = 3.806±0.154 years), and Chebyshev norm (MAE = 3.842±0.196 years).


Following the ablation study, we turn our attention to the evolution of the representations as the model learns across epochs. The UMAP embedding visualizations presented in Fig. <ref> showcase the progressive refinement of the feature space through epochs 1, 6, 21, 30, 40 and 50. Initially, the representations are scattered and lack clear structure, indicating that the model is yet to learn distinct age-related patterns. As training progresses, we observe the emergence of more defined clusters, reflecting the model's increasing adeptness at capturing age-related variations in brain stiffness. By epoch 50, the embeddings show distinct, well-separated groupings, suggesting that our method has achieved a better understanding of the underlying age-related features. These visualizations not only confirm the effectiveness of our dynamic localized repulsion approach but also offer intuitive insights into how contrastive learning can be harnessed for regression tasks in medical imaging.














§ CONCLUSION



We introduced a novel contrastive regression loss that adeptly prioritizes local regions within embedding spaces and dynamically adjusts these regions throughout the training process. By applying this method to brain stiffness maps obtained from Magnetic Resonance Elastography (MRE), we achieved superior predictive performance in brain age estimation compared to established contrastive learning methods. This advancement not only demonstrates our model's efficacy but also underscores the potential of using localized dynamic adjustments in the analysis of complex neuroimaging data.

Significantly, our research marks the first application of self-supervised learning techniques to explore the mechanical properties of the brain, an area previously uncharted in the literature. The implications of this are profound, opening up new avenues for understanding the structural changes associated with aging and potentially other neurological conditions.

Future work includes to extend our framework to include cohorts with neurological diseases, such as Alzheimer's and Parkinson's. This expansion is expected to provide deeper insights into the progression and early diagnosis of these conditions, leveraging the  detection capabilities of our model. Additionally, we aim to explore the integration of multimodal imaging data to enhance the robustness and accuracy of our predictions, potentially leading to breakthroughs in personalized medicine and neuroimaging analytics.


tmlr



§ APPENDIX




 §.§ Brain Coverage of Stiffness Maps






 Paper Title: Data Contamination Report from the 2024 CONDA Shared Task.


Unsteady slip pulses under spatially-varying prestress
    Eran Bouchbinder
    Received 2 August 2023 / Accepted 7 May 2024
======================================================



§ ABSTRACT 

    The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform.1 The platform continues to be online, open to contributions from the community.




§ INTRODUCTION




Data contamination, where evaluation data is inadvertently included in pre-training corpora of large-scale models, and language models (LMs) in particular, has become a concern in recent times <cit.>. The growing scale of both models and data, coupled with massive web crawling, has led to the inclusion of segments from evaluation benchmarks in the pre-training data of LMs <cit.>. The scale of internet data makes it difficult to prevent this contamination from happening, or even detect when it has happened  <cit.>. 

Crucially, when evaluation data becomes part of pre-training data, it introduces biases and can artificially inflate the performance of LMs on specific tasks or benchmarks <cit.>. This poses a challenge for fair and unbiased evaluation of models, as their performance may not accurately reflect their generalization capabilities <cit.>. And similarly to pre-training contamination, the contamination can also occur during the fine-tuning stage even after a model has been deployed as an API <cit.>.

Although a growing number of papers and state-of-the-art models mention issues of data contamination <cit.>, there is little in the way of organized and compiled knowledge about real, documented cases of contamination in practice <cit.>. Addressing data contamination is a shared responsibility among researchers, developers, and the broader community.

This report compiles the evidence reported in the Data Contamination Database[<https://huggingface.co/spaces/CONDA-Workshop/Data-Contamination-Database>] as part of the Data Contamination Workshop.[<https://conda-workshop.github.io/>] As the Shared Task of the workshop, researchers were invited to discover cases of contamination in available corpora and models, and submit evidence of their discovery. The submissions to the database were collected and compiled on June 23rd, 2024, to be included in this report, but the database continues to run and grow. Overall we collected 566 submissions from 23 contributors, where each submission included a detailed contamination report, indicating the estimated percentage of contaminated data. 

We continue to operate the database, and expect to update it with newer datasets and models as they come out, as well as new report about existing contaminated (or uncontaminated) evaluations.

This report first presents the methodology for collecting evidence, as well as existing papers that report data contamination (Section <ref>). We also report the evidence collected in the Data Contamination Database (Section <ref>), followed by an overview of the trends and statistics in the database, that inform a high-level perspective on the state of data contamination in NLP today (Section <ref>). 



§ METHODOLOGY AND PREVIOUS WORK





Collecting all the contamination evidence —or lack of it— was done openly, through pull requests, and subject to discussions before the admission. Contributors were asked to fill in the information about several aspects, such as the
contaminated resource (a training corpus or model), the evaluation dataset which was found in the contaminated source, a breakdown of the percentage of contamination found in each split of the dataset (train, development, and test), an optional reference to a paper that describes the methodology behind the submission, as well as whether the contamination detection method was data-based or model-based. The contributions provided the HuggingFace Hub id of models, corpus, and datasets when possible. In addition, contributors must provide the evidence or a reference to the scientific paper that reported the evidence originally. Figure <ref> shows the taxonomy of the papers that reported contamination evidence in the shared task.[Note that there are many other works on data contamination detection. In this report we focus on works that were used to detect contamination for this report. We leave a more detailed coverage survey for future work.] We split the these methods into two: data-based and model-based approaches.





  
Data-based approaches. are methods that inspect the pre-training corpora to find contamination evidence. Data-based approaches typically involve string or sub-string matching techniques such as 13-gram overlap <cit.>, 50-character overlap <cit.> or even full-string overlap <cit.>. In Figure <ref> we differentiate between Proprietary and Open data. Papers that fall in the category of Proprietary data are usually LLMs technical reports that run post-hoc data contamination evaluations to identify and remove evaluation instances that appear in the pre-training corpora <cit.>. Papers that fall in the open data category usually involve corpus analysis tools <cit.> or LLMs with publicly available pre-training data <cit.>.




  
Model-based approaches. are those methods that try to estimate the contamination of a model by prompting or analyzing the output, without accessing the pre-training data. These methods are formulated as Membership Inference Attacks (MIA) and range from asking LLMs to generate verbatim of the actual evaluation data <cit.> to analyzing the actual output probabilities given by the model <cit.>. We differentiate between methods applicable to closed and open models. Methods applicable to closed models are usually applicable to open models, but not the other way around due to the limitations established by the API or interface providers.

The collected evidences come from different approaches and sources, making them hardly comparable. For transparency, we included in the database information about the source of the evidence and the link to the discussion. We encourage the users to assess how the evidence was collected for their datasets of interest.





§ COMPILATION OF EVIDENCE
 



The report includes 42 contaminated sources (training corpora or models), 91 datasets, and 566 contamination entries, including 432 contamination events (20 train-set, 95 dev-set, 317 test-set) and 144 non-contamination events, where a contamination event is taken as any report above 0% of contamination. The database contains, for each split (train, dev, and test) of each evaluation dataset, what percentage was found to be contaminated by a subset of the contamination sources (corpora or models). We analyze separately the contaminated corpora and models. 


























  
Contaminated corpora. Figure <ref> shows the number of reported test sets for each corpus often used to pre-train language models. The reported corpora are mainly based on CommonCrawl snapshots, GitHub, or a mix of sources. For CommonCrawl-based corpora, there are 35 events reported for C4 <cit.>, 32 for RedPajama v2 <cit.>, 29 for OSCAR <cit.> and 6 for CommonCrawl <cit.> itself. Regarding the GitHub data, there are 2 events reported for the TheStack <cit.> project. The corpora with various sources, the Pile <cit.> and ProofPile <cit.>, have 30 and 2 reported contamination events respectively. There is also 1 report for xP3 <cit.>, which is a collection of prompts for different NLP datasets.[The report indicates the use of validation data from a specific dataset as training.]

Table <ref> shows for each corpus often used to pre-train language models, the contamination events involving development or test splits. Please refer to the online database for full details of each report.



 












  
Contaminated models. Figure <ref> details the number of contamination events involving test sets that were reported, organised according to each pre-trained model. Most reported evidence is for closed models, for instance: 24 for GPT-3 <cit.>, 17 for GLaM <cit.>, 16 for GPT-4 <cit.>, 13 for GPT-3.5 <cit.>, 8 for PaLM <cit.>, 3 for PaLM-2 <cit.>, 2 for GPT-3.5 Turbo <cit.> and 1 for Calude 3 Opus. In the case of open models: there are 14 reported events for models fine-tuned with FLAN data <cit.>, 5 for Mistral <cit.>, 3 for Llama 2 <cit.>, 2 for Qwen <cit.>, Llema <cit.> and Aquila 2; and a single one for mT0 and Bloom-Z <cit.>.


Table <ref> shows for each pre-trained language model, the contamination events involving development or test splits. Please refer to the online database for full details of each report.























































































































































§ ANALYSIS OF THE REPORTED DATA
 

In this section, we analyze the reported entries to understand the report's data better. 









  
Reported tasks. Figure <ref> shows the percentage of data contamination per task. We use the  assigned to each dataset in the Hugging Face hub. Text-scoring, QA, and multiple-choice-qa are among the most contaminated task types. Figure  <ref> shows the number of downloads for every dataset in the report. We measure the total number of downloads from the Hugging Face hub.[<https://huggingface.co/docs/datasets>] Since one model may be reported as contaminated with a dataset while another model may not, we have entries of both being compromised and non-compromised for some datasets. Relating both tables, we can see that the tasks reported as the most contaminated include very popular datasets such as MMLU (multiple-choice-qa), GLUE (text-scoring), and ai2_arc (multiple-choice-qa), which are standard benchmarks for measuring the performance of LLMs. These benchmarks, as well as other very popular benchmarks reported in instances of data contamination, such as hellaswag or gsm8k are implemented in community leaderboards such as the Open LLM Leaderboard.[<https://hf.co/spaces/open-llm-leaderboard/>]











  
Year of publication of the reported data. Figure <ref> shows the percentage of total test sets included in contamination events per year. We present data for test sets in both contamination events (>0% contamination) and non-contamination events (0% contamination). Most of the reported datasets correspond to the 2018 to 2021 period. 


We further explore the relationship between the year of publication of the datasets and instances of contamination by examining the reported data contamination for the three models with the most instances of data contamination: GPT-4, GPT-3, and GPT-3.5. As expected based on the models' release dates, Figure <ref> shows that more recently released models are contaminated with more recently released datasets. For instance, GPT-3, launched in 2020, is predominantly contaminated with datasets from 2016, while GPT-4, released in 2023, is mainly contaminated with datasets from 2018 to 2022. 














§ CONCLUSIONS


Data contamination has become a significant concern in recent times. Consequently, a growing number of papers and state-of-the-art models mention issues of data contamination. In the CONDA 2024 Shared Task on Evidence of Data Contamination, we have collected and compiled a comprehensive database of available evidence on data contamination in currently available datasets and models. This report includes 566 contamination entries over 91 contaminated sources from a total of 23 contributors. With this shared task, we provide a structured, centralized platform for contamination evidence collection to help the community understand the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. Given the large exploration space, this report does not cover all cases, but a small sample that were reported during our shared task period, in the midst of 2024. We welcome further submissions to the database, and plan to keep this database up-to-date as it provides a valuable source of information for the research community.


















































































































































































































































































































































































































































































































































































































§ ACKNOWLEDGMENTS


We are grateful to Hugging Face and Clémentine Fourrier for their support in establishing the website for the Data Contamination Database hosted on Hugging Face Spaces.



We acknowledge the support of project Disargue (TED2021-130810B-C21, funded by MCIN/AEI /10.13039/501100011033 and by European Union NextGenerationEU/ PRTR) and the Basque Government (Research group funding IT-1805-22). 






























 Paper Title: Transient anisotropic kernel for probabilistic learning on manifolds.










1]Christian Soize cor1
christian.soize@univ-eiffel.fr
2]Roger Ghanem
ghanem@usc.edu
[cor1]Corresponding author: C. Soize, christian.soize@univ-eiffel.fr
[1]Université Gustave Eiffel, MSME UMR 8208, 5 bd Descartes, 77454 Marne-la-Vallée, France
[2]University of Southern California, 210 KAP Hall, Los Angeles, CA 90089, United States




§ ABSTRACT 
PLoM (Probabilistic Learning on Manifolds) is a method introduced in 2016 for handling small training datasets by projecting an Itô equation from a stochastic dissipative Hamiltonian dynamical system, acting as the MCMC generator, for which the KDE-estimated probability measure with the training dataset is the invariant measure. PLoM performs a projection on a reduced-order vector basis related to the training dataset, using the diffusion maps (DMAPS) basis constructed with a time-independent isotropic kernel. In this paper, we propose a new ISDE projection vector basis built from a transient anisotropic kernel, providing an alternative to the DMAPS basis to improve statistical surrogates for stochastic manifolds with heterogeneous data.  The construction ensures that for times near the initial time, the DMAPS basis coincides with the transient basis. For larger times, the differences between the two bases are characterized by the angle of their spanned vector subspaces. The optimal instant yielding the optimal transient basis is determined using an estimation of mutual information from Information Theory, which is normalized by the entropy estimation to account for the effects of the number of realizations used in the estimations.
Consequently, this new vector basis better represents statistical dependencies in the learned probability measure for any dimension. Three applications with varying levels of statistical complexity and data heterogeneity validate the proposed theory, showing that the transient anisotropic kernel improves the learned probability measure.



Transient kernelprobabilistic learning PLoM diffusion maps Fokker-Planck operator spectrum






§ INTRODUCTION






 §.§ Objectives of the paper



PLoM (Probabilistic Learning on Manifolds), introduced in 2016 <cit.>, is a method and algorithm specifically developed for cases where the training dataset consists of a small number of data points. This method is based on projecting an Itô equation associated with a stochastic dissipative Hamiltonian dynamical system, which acts as the MCMC generator from the probability measure estimated using the KDE method applied to the points of the training dataset. The projection basis is the diffusion maps (DMAPS) basis associated with a time-independent isotropic kernel, introduced in <cit.>.

Since 2016, all extensions and applications of PLoM (see Section <ref>) have been carried out using the isotropic kernel. Through these applications, we have seen that the isotropic kernel allows for obtaining quality results, even for heterogeneous data and systems of great statistical complexity in small and large dimensions. However, improving the construction of statistical surrogates for stochastic manifolds involving conditional statistics and very heterogeneous data using PLoM based on a transient anisotropic kernel (time-dependent) was an analysis project. In this paper, we address this problem. We propose a new construction of the ISDE projection vector basis, built from a transient anisotropic kernel, which improves the representation of the statistical dependencies of the learned joint probability measure in any dimension.



 §.§ Framework of the considered problem



Machine learning tools and artificial intelligence <cit.>, such as probabilistic and statistical learning <cit.>, are used in UQ for problems that would require computer resources not available with the most usual approaches. Thus, methods have emerged in the field of engineering sciences, such as learning on manifolds <cit.> and physics-informed probabilistic learning <cit.>.

Probabilistic learning is a very active domain of research for constructing surrogate models (see for instance, <cit.>). Probabilistic Learning on Manifolds (PLoM) is a tool in computational statistics, introduced in 2016 <cit.>, which can be viewed as a tool for scientific machine learning. The PLoM approach has specifically been developed for small dataset cases <cit.>. The method avoids the scattering of learned realizations associated with the probability distribution to preserve its concentration in the neighborhood of the random manifold defined by the parameterized computational model.

his method allows for solving unsupervised and supervised problems under uncertainty when the training datasets are small. This situation is encountered in many problems in physics and engineering science with expensive function evaluations. The exploration of the admissible solution space in these situations is thus hampered by available computational resources.

Several extensions have been proposed to account for implicit constraints induced by physics, computational models, and measurements <cit.>, to reduce the stochastic dimension using a statistical partition approach <cit.>, and to update the prior probability distribution with a target dataset, whose points are, for instance, experimental realizations of the system observations <cit.>. Consequently, PLoM, constrained by a stochastic computational model and statistical moments or samples/realizations, allows for performing probabilistic learning inference and constructing predictive statistical surrogate models for large parameterized stochastic computational models.

This last capability of PLoM can also be viewed as an alternative method to Bayesian inference for high dimensions
<cit.> and is a complementary approach to existing methods in machine learning for sampling distributions on manifolds under constraints. Although a Bayesian inference methodology has also been developed using probabilistic learning on manifolds for high dimensions <cit.>.

PLoM has successfully been adapted to tackle these challenges for several related problems, including nonconvex optimization under uncertainty <cit.>, fracture paths in random composites <cit.>, concurrent multiscale simulations in random media <cit.>, stochastic homogenization in random elastic media <cit.>, ultrasonic transmission techniques in cortical bone microstructures <cit.>, updating digital twins under uncertainties <cit.>, updating under-observed dynamical systems <cit.>, calculation of the Sobol indices <cit.>, dynamic monitoring <cit.>, surrogate modeling of structural seismic response <cit.>, probabilistic-learning-based stochastic surrogate models from small incomplete datasets <cit.>, and polynomial-chaos-based conditional statistics for probabilistic learning of atomic collisions <cit.>, as well as for aeroacoustic liner impedance metamodels from simulation and experimental data <cit.>.



 §.§ Methodology proposed and organization of the paper



Starting with a training dataset of n_d realizations of ν random variables, we consider the probability flow from each of these n_d realizations towards the ν-dimensional sampling probability distribution of the training dataset. We construct the associated coupled Fokker-Planck (FKP) equations with each of the realizations as initial condition. These describe the evolution of the transition probability measures over the graph described by the training dataset, that transport each independent realization, viewed as a concentrated measure at the initial time, into the joint probability measure, consisting of the common stationary probability measure  (also called the steady-state solution) of the FKP equations. This evolution describes a trajectory along which transition probabilities are consistent with both the training dataset and its postulated joint probability density function (that we approximate using a Gaussian Kernel Density Estimate (KDE)). Each of these probabilities provides a distinct geometric characterization of the training dataset, with its own plausible model of statistical dependence, resulting in a transient anisotropic kernel from which we construct a time evolving PLoM. We then set criteria for selecting among these PLoM models, which is tantamount to identifying the most appropriate
statistical dependence structure for the learned dataset, along the flow characterized by the FKP equations. The following
description of the paper’s organization provides a coherent summary of the proposed methodology.

In Section <ref>, we define the training dataset constituted of given realizations of a non-Gaussian normalized vector-valued random variable  (centered and with an identity covariance matrix) and define the associated probability measure P_, whose density is estimated using the Gaussian Kernel Density Estimation (GKDE) method.

Section <ref> deals with a short summary of formal results, introducing an Itô stochastic differential equation (ISDE) and the derived Fokker-Planck (FPK) equation for which P_ is the invariant measure, which is the steady-state solution. We then introduce a formal formulation of the eigenvalue problem of the FPK operator and the nonstationary solution of the Fokker-Planck equation with a deterministic initial condition.

Section <ref> is devoted to the time-dependent kernel k_t derived from the time-dependent solution of the Fokker-Planck equation with a deterministic initial condition. We then define the time-dependent operator K_t associated with the transient kernel k_t, which is a Hilbert-Schmidt operator. We propose a construction of its finite approximation, represented by a matrix [K(t)], using a sampling of P_ of the bilinear form associated with K_t, and introduce the corresponding finite approximation of the eigenvalue problem.

In Section <ref>, we present the direct construction of  matrix [K(t)]. We introduce the transition probability density function as the solution of the ISDE with a deterministic initial condition and study its existence, uniqueness, and properties. We then rewrite the ISDE in matrix form and propose a time-discrete approximation of this matrix-valued ISDE based on an Euler scheme. We introduce convergence criteria to verify convergence. Finally, we construct an explicit representation of the matrix [K(t)] based on the nonstationary solution of the matrix-valued ISDE with a deterministic initial condition.

A numerical illustration of the proposed formulation is given in Section <ref>, for which an explicit solution is known.

Section <ref> deals with the construction and study of the vector basis for PLoM derived from the transient anisotropic kernel, connected to the DMAPS basis constructed with the isotropic kernel. We construct a time-dependent matrix [K̃()] of the transient anisotropic kernel derived from the matrix [K̂()], whose fundamental property is its convergence to the matrix [K_] of the DMAPS isotropic kernel as time approaches zero. The time-dependent reduced-order transient basis is then the eigenvectors of [K̃()] associated with its dominant positive eigenvalues.

In order to qualify and quantify the gain of the constructed reduced-order transient basis, ROTB() at an instant sampled , with respect to the reduced-order DMAPS basis, RODB, we introduce the angle between the subspaces spanned by ROTB() and RODB. We propose a methodology for identifying the optimal instant sampled, which maximizes a selection criterion. This criterion is based on the estimation of mutual information from Information Theory, normalized using entropy estimation to account for the effects of the number of realizations used in the statistical estimators. Such a criterion allows for selecting the best joint learned probability measure with respect to statistical dependencies.

In Section <ref>, we present three applications, each with a specific level of complexity and data heterogeneity in the training dataset. The first application is created such that the probability measure of  in ^9, defined by the points of the training dataset, is concentrated in a multiconnected domain of ^9. The constituent connected parts are manifolds of dimensions much lower than 9, each with different dimensions. These parts may or may not be connected to each other. The training dataset of the second application consists of realizations of the random vector  with values in ^8, generated using a polynomial chaos expansion of degree 6 of a real-valued random variable, whose random germ has a dimension of 2, with each of the two random germs being a uniform random variable with different support. There are therefore 28 terms in this expansion, and the 8 components of  are defined as the random terms of given rank, defining a relatively complex random manifold in ^8. The third application results from a statistical treatment of an experimental database containing photon measurements in the ATLAS detector at CERN. The PCA step of PLoM has been performed, and 45 components have been extracted to obtain the training dataset for the ^45-valued random variable . This application is in higher dimension than the first two but has less statistical complexity.

The paper is completed by two appendices. <ref> presents an overview of the probabilistic learning on manifolds (PLoM) algorithm and its parameterization, using either the DMAPS basis (RODB) or the transient basis (ROTB()) as the projection basis. <ref> provides the formulas for estimating the Kullback-Leibler divergence, mutual information, and entropy from a set of realizations.

It should be noted that there are some very brief repetitions, which have been deliberately made to facilitate reading.



 §.§ Convention for the variables, vectors, and matrices



x,η: lower-case Latin or Greek letters are deterministic real variables.

,: boldface lower-case Latin or Greek letters are deterministic vectors.

X: upper-case Latin letters are real-valued random variables.

: boldface upper-case Latin letters are vector-valued random variables.

[x]: lower-case Latin letters between brackets are deterministic matrices.

[]: boldface upper-case letters between brackets are matrix-valued random variables.



 §.§ Algebraic notations



, ^*: set of natural numbers including 0, excluding 0.

, ^+, ^+*: set of real numbers, subset [0,+∞[, subset ]0, +∞[.

^ν: Euclidean vector space of dimension ν.

_n,m, _n : set of the (n× m), (n× n),  real matrices.

_n^+, _n^+0: set of  the positive-definite, positive,  (n× n) real matrices.

[I_n]: identity matrix in _n.

‖  [x] ‖_F: Frobenius norm of matrix  [x].

∇, : gradient and divergence operators in ^ν.

δ_0: Dirac measure on ^ν at point _ν.




 §.§ Convention used for random variables



In this paper, for any finite integer m≥ 1, the Euclidean space ^m is equipped with the σ-algebra _^m. If  is a ^m-valued random variable defined on the probability space (Θ,,),  is a  mapping θ↦(θ) from Θ into ^m, measurable from (Θ,) into (^m,_^m), and (θ) is a realization (sample) of  for θ∈Θ. The probability distribution of  is the probability measure P_(d) on the measurable set (^m,_^m) (we will simply say on ^m). The Lebesgue measure on ^m is noted d and
when P_(d) is written as p_()  d, p_ is the probability density function (pdf) on ^m of P_(d) with respect to d. Finally, E denotes the mathematical expectation operator that is such that
E{} = ∫_^m  P_(d).




§ DEFINING THE PROBABILITY MEASURE P_ OF RANDOM VECTOR 




Let _() = {^j, j = 1,…, n_d} be the set of n_d > 1 independent realizations ^j∈^ν, with ν≥ 1, of a second-order ^ν-valued random variable defined on a probability space (Θ,,). Let _d∈^ν and [C_d]∈_n_d be the associated empirical estimates of the mean value and the covariance matrix constructed with the points of _(),


    _d = 1/n_d∑_j=1^n_d^j    ,   
    [C_d] =  1/n_d -1∑_j=1^n_d (^j - _d )⊗ (^j - _d )  .


It is assumed that _() is such that


    _d =  _ν   ,   
    [C_d] =  [I_ν]  .


Let ↦ p_() be the probability density function on ^ν, with respect do the Lebesgue measure d, defined by


    p_() = 1/n_d∑_j=1^n_d1/(√(2 π) ŝ)^νexp ( -1/2 ŝ^2‖ - ŝ/s^j‖^2 )    ,   ∀∈^ν  ,


where ŝ and s are defined by


    s =  ( 4/n_d(2+ν) )^1/(ν+4)   ,   ŝ  = s/√(s^2 + (n_d-1)/n_d)  .


Eqs. (<ref>) and (<ref>) correspond to the Gaussian kernel-density estimation (KDE) constructed using the n_d independent realizations of _() involving the modification <cit.> of the usual formulation <cit.>, in which s is the Silverman bandwidth.

Let  be the second-order ^ν-valued random variable, defined on a probability space (Θ,,), whose probability measure P_(d) = p_()  d on ^ν is defined by the probability density function p_ given by Eq. (<ref>). It can be seen that, for any fixed n_d > 1, we have


    E{} = ∫_^ν  P_(d) = 1/2ŝ^2 _d = _ν  ,



    E{⊗} = ∫_^ν⊗  P_(d)
    = ŝ^2  [I_ν] + ŝ^2/s^2 (n_d-1)/n_d [C_d] = [I_ν]   .


Eqs. (<ref>) and (<ref>) show that  is a normalized ^ν-valued random variable. The probability density function p_ defined by Eq. (<ref>) is rewritten, for all  in ^ν, as


    p_() = c_ν ξ()     ,   ξ() = e^-Φ()  ,


in which  c_ν =  (√(2 π) ŝ)^-ν and where Φ() = -log(ξ()) is such that


    Φ() = - log ( 1/n_d∑_j=1^n_dexp ( -1/2 ŝ^2‖ - ŝ/s^j‖^2 )  )    ,   ∀∈^ν  .






§ SHORT SUMMARY OF FORMAL RESULTS




This section is limited to a summary of essential results, which will be used in Section <ref> and which are formally presented.




 §.§ Itô stochastic differential equation related to P_



We introduce an  Itô stochastic differential equation (ISDE) on ^ν, with initial condition, for which P_(d) is the invariant measure. A classical candidate to such an ISDE is written as



    d(t) =     ((t))  dt + d(t)    ,    t > 0   , 
    (0) =     ∈^ν  ,   a.s.   ,


where the drift vector is the function ↦() from ^ν into ^ν defined by


    () = -1/2 ∇Φ()    ,   ∀∈^ν  .


In Eq. (<ref>), (t) = (W_1(t),… ,W_ν(t)) is the normalized Wiener stochastic process <cit.> on ^+, with values in ^ν, which is a stochastic process with independent increments, such that (0) = _ν  a.s, and for  0≤τ < t < +∞, the increment Δ_τ  t = (t) - (τ) is a Gaussian ^ν-valued second-order random variable, centered and with a covariance matrix that is written as


    [C_Δ_τ t] = E{Δ_τ t⊗Δ_τ t} = (t-τ)  [I_ν]   .


It should be noted that Eqs. (<ref>) and (<ref>) is equivalent to


    (t) =  + ∫_0^t ((τ))  dτ + ∫_0^t d(τ)    ,    t ≥  0   .


In Section <ref>, we will see that {(t), t∈^+} is a homogeneous diffusion stochastic process, which is asymptotically stationary for t→ +∞. Assuming that the transition probability measure of (t) given (0) = admits a density with respect to d, such that, for all t > 0, for all  and  in ^ν, and for any Borelian  in ^ν, we have


    { (t) ∈ | (0) =  }    =  ∫_ρ(,t  | ,0)   d  ,  
    lim_t → 0_+ρ(,t  | ,0)   d     = δ_0( - )    ,  
    ∫_^νρ(,t  | ,0)   d    = 1   .






 §.§ FKP equation associated with the ISDE



For all  in ^ν, the transition probability density function (,t)↦ρ(,t  | ,0) from
^ν×^+ into ^+ verifies the following Fokker-Planck (FKP) equation (see for instance <cit.>),


    ∂ρ/∂ t + L_(ρ) = 0    ,    t > 0   ,


with the initial condition for t=0 defined by Eq. (<ref>). The Fokker-Planck operator L_ can be written, after a small algebraic manipulation and for any sufficiently differentiable function ↦() from ^ν into , as


    {L_()}() = -1/2{ p_() ∇ ( ()/p_() ) }  .


The detailed balance (the probability current vanishes) is satisfied and the steady state solution of Eq. (<ref>) is the pdf p_ defined by Eq. (<ref>) <cit.>. We then have, for = p_,


    L_(p_) = 0   .






 §.§ Return to the invariant measure P_



The invariant measure (see Proposition <ref>) P_(d) = p_()  d is such that, for all  in ^ν and for all t≥ 0,


    p_() = ∫_^νρ(,t  | ,0)   p_()   d  .


The ISDE defined by Eqs. (<ref>) and (<ref>) admits an asymptotic (t→ +∞) stationary solution whose marginal probability density function of order one is p_. Consequently, for all  and  in ^ν, we have


    lim_t → +∞ρ(,t  | ,0)  = p_()   .






 §.§ Formal formulation of the eigenvalue problem of the FKP operator



The eigenvalue problem, posed in an adapted functional space, is written as


    L_() =  λ   ,


for which the current must vanish at infinity, yielding the condition,


    lim_‖‖→ +∞  p_()  ‖ ∇  ( p_()^-1()   )  ‖  = 0  .


Continuing the development within  a formal framework, such as that used in <cit.>, we introduce the change of function,


    () = p_()^1/2  q()    ,   ∈^ν   ,    q: ^ν→  .


Let L̂_ be the linear operator defined, for q belonging to an admissible set of functions,


    {L̂_(q)}() =  p_()^-1/2 L_(p_()^1/2  q(y))    ,   ∈^ν  .


Therefore, the eigenvalue problem defined by Eqs. (<ref>) and (<ref>) can  be rewritten in q as


    L̂_(q) =  λ  q   ,


with the condition at infinity,


    lim_‖‖→ +∞ p_()^1/2 ‖ ∇  ( p_()^-1/2 q()   )  ‖  = 0  .






 

Using Eq. (<ref>), which shows that p_()^-1∇ p_() = - ∇Φ, and using
 Eqs. (<ref>) and (<ref>), it can be seen that


    {L̂_(q)}() =  ()  q() - 1/2∇^2 q()    ,   ∈^ν  ,


in which ∇^2 is the Laplacian operator in ^ν and where ↦() is the function from ^ν into , which is defined, for all  in ^ν, as


    () =  1/8‖ ∇Φ()  ‖^2  -  1/4∇^2 Φ()   .










 §.§ Properties of operator L̂_



Let δ q be a function from ^ν into , belonging to the admissible set that allows the evaluation of the bracket


    ⟨L̂_ (q)   , δ q⟩  = ∫_^ν{L̂_(q)}()   δ q()   d  .


Removing  and using Eq. (<ref>) with Eq. (<ref>) yields


    ⟨L̂_ (q)   , δ q⟩  = -1/2∫_^ν
    (p_^-1/2 δ q)  { p_ ∇ ( p_^-1/2  q)}  d  .


Using the condition at infinity, defined by Eq. (<ref>), Eq. (<ref>) can be rewritten as,


    ⟨L̂_ (q)   , δ q⟩  = 1/2∫_^ν
    p_ ⟨∇(p_^-1/2  q)   , ∇( p_^-1/2 δ q) ⟩_^ν  d  .



(a) Eq. (<ref>) shows that L̂_ is a symmetric and positive operator.

(b) Eqs. (<ref>) and (<ref>) show that


    L̂_ (q_0) = 0   for   q_0 = p_^1/2  .


In Proposition <ref>, it will be proven that the ISDE defined by Eq. (<ref>), with the initial condition defined by  (<ref>),  has a unique solution and a unique invariant measure p_()  d. Consequently the dimension of the null space of operator L_ is 1. Since p_()  d is a bounded positive measure (probability measure), the right-hand side of Eq. (<ref>) shows that the null space  of L̂_, which is also of dimension  1, is constituted of the function q_0 = p_^1/2. For q=δ q ≠ q_0, and ‖ q_0‖≠ 0, we have ⟨L̂_ (q)   ,  q⟩  > 0. Therefore, L̂_ is a positive operator (in the quotient space by the null space).



[On the spectrum of operator L̂_] 
It is assumed that p_ defined by Eq. (<ref>), which is constructed with the n_d points {^j, j=1,…, n_d} of the training dataset, is such that the spectrum of L̂_ is countable. Due to (a) and (b), we then deduce that the eigenvalues of L̂_ (defined by Eqs. (<ref>) and (<ref>)) are positive except one that is zero. We will also assume that the multiplicity of each eigenvalue is finite.






 §.§ Eigenvalue problem for operator L̂_



Under Hypothesis <ref>, the eigenvalue problem L̂_(q_α) = λ_α q_α for operator
L̂_, with the condition defined by Eq. (<ref>), is such that


    0 =λ_0 < λ_1 ≤λ_2 ≤…  ,


the multiplicity of each eigenvalue being finite.

We will admit that the family {q_α,α∈} of the eigenfunctions is a Hilbert basis of L^2(^ν). We then have


    ⟨ q_α , q_β⟩_L^2 = ∫_^ν q_α()   q_β()  d = δ_αβ  .


The eigenfunction q_0 associated with λ_0=0, is such that (see Eq. (<ref>)),


    q_0 = p_^1/2   ,  ‖  q_0‖_L^2 = 1   ,


and we have


    ∑_α∈ q_α()   q_α()  d = δ_0(-)   .


From Eqs. (<ref>) and (<ref>), it can be deduced that


    ∀α≥ 1    ,   ∫_^ν p_()^1/2  q_α()   d = 0   .






 §.§ Nonstationary solution of the Fokker-Planck equation with initial condition



The transition probability density function ρ(,t | ,0) introduced in Section <ref> and satisfying Eq. (<ref>) with the initial condition defined by Eq. (<ref>), can  be written, using the Hilbert basis {q_α,α∈} defined in Section <ref>, as


    ρ(,t | ,0) = p_()^1/2  p_()^-1/2 ∑_α∈ e^-λ_α t q_α()   q_α()    ,    t > 0  .


This representation of ρ(,t | ,0), defined by Eq. (<ref>), actually satisfied all the required properties:

Eqs. (<ref>), (<ref>), and  (<ref>) yield ∫_^νρ(,t | ,0)   d = 1.

Eqs. (<ref>) and  (<ref>) yield lim_t→ +∞ρ(,t | ,0)   d = p_().

Eq. (<ref>) yields lim_t→ 0_+ρ(,t | ,0)   d = δ_0(-).

Eqs. (<ref>) and  (<ref>) yield ∫_^νρ(,t | ,0)  p_()   d = p_() that is Eq. (<ref>).




§ TIME-DEPENDENT KERNEL, ITS ASSOCIATED OPERATOR, AND FINITE APPROXIMATION




In this section,  we define the kernel k_t(,) and give its basic properties directly deduced from the properties of ρ(,t | ,0) and p_(), without using the spectral representation defined by Eq. (<ref>). From the spectral representation of k_t(,), we deduce its spectral representation using the spectral representation of  ρ(,t | ,0), defined by Eq. (<ref>). Finally, we define the linear operator K_t associated with kernel k_t and we give the spectral representation of operator K_t.




 §.§ Definition of the kernel k_t and its basic probabilistic properties



The kernel k_t(,) associated with the transition probability density function ρ(,t | ,0) is defined as follows.


 
For every fixed t > 0, the kernel function (,)↦ k_t(,), from ^ν×^ν into ^+, is defined by


    k_t(,) = ρ(,t | ,0)/p_()  .





The following Lemma gives basic properties of kernel k_t.


 

For every fixed t > 0, and for all  and  in ^ν, we have the following properties:


    (a)      ∫_^ν k_t(,)  p_()  d = 1    ,   ∫_^ν k_t(,)  p_()  d = 1                           , 
    
    (b)      ∫_^ν∫_^ν   k_t(,)  p_()  p_() d  d =1   ,  
    
    (c)      lim_t→ 0_+   k_t(,)  p_()  d = δ_0(-)    ,   lim_t→ +∞ k_t(,) =1                                     ,  
    
    (d)       (,)↦ρ(,t | ,0) ∈ C^0(^ν×^ν) ⇒
                          (,)↦  k_t(,) ∈ C^0(^ν×^ν)                   ,  
    
    (e)       k_t(,) = k_t(,)                                                                     .





 (Lemma <ref>). Definition <ref> is used.

(a) The first equation in Eq. (<ref>) is due to Eq. (<ref>) and the second one is due to Eq. (<ref>).

(b)  Eq. (<ref>)  is directly deduced from Eq. (<ref>).

(c) The first equation in Eq. (<ref>) is due to Eq. (<ref>) and the second one is due to Eq. (<ref>).

(d) For all  in ^ν, p_(y) > 0 and p_∈ C^0(^ν). The hypothesis
ρ(· ,t | ·,0)∈ C^0(^ν×^ν) yields Eq. (<ref>).

(e) let p_(t),(,t;,0) be the joint pdf of (t) with  in which (t) is the solution of Eq. (<ref>) for fixed t > 0, with the random initial condition (0) =. We have the classical property related to the definition of the invariant measure,
p_(t)(,t) = ∫_^ν p_(t),(,t;,0)  d =  ∫_^νρ(,t | ,0)  p_()  d = p_(). For all  and  in ^ν, and for t > 0, we have,
ρ(,t | ,0)  p_() = p_(t),(,t;,0) =p_,(t)(,0;,t) =
ρ(,0 | , t)  p_(t)(,t) = ρ(,0 | , t)  p_().
Consequently,  k_t(,)= ρ(,t | ,0) p_() / (p_() p_()) =
ρ(,0 | , t)  p_() / (p_() p_()) = k_t(,).







 §.§ Hypothesis and properties of kernel k_t from its representation



In Section <ref>, we introduced an hypothesis of existence of a discrete (countable) spectrum {λ_α,α∈} of the FKP operator L̂_. In this section, we study the spectral representation of kernel k_t, for every fixed t > 0, deduced from the time-dependent spectral representation of ρ(,t | ,0), defined by Eq. (<ref>).


 

Let = L^2(^ν;p_) be the Hilbert space of the square-integrable real-valued functions on ^ν, with respect to the probability measure p_()  d on ^ν, equipped with the inner product and the associated norm,


    ⟨ u  , ⟩_ = ∫_^ν u() ()   p_()  d   ,   ‖ u‖_ = ⟨ u  , u ⟩_^1/2  .







 

Let {q_α,α∈} be the Hilbert basis of L^2(^ν) introduced in Section <ref>.
For all α∈, we defined the real-valued function ψ_α on ^ν such that


    ψ_α() = q_α()  p_()^-1/2   ,   ∀∈^ν  .


Then, {ψ_α,α∈} is a Hilbert basis of  and we have,


    (a)      ψ_α∈   ,   ‖ψ_α‖_ = ‖ q_α‖_L^2 = 1
                  ,   ∀α∈  , 
    
    (b)      ⟨ψ_α  , ψ_β⟩_ = ∫_^νψ_α() ψ_β()  
                  p_()  d = δ_αβ   ,   ∀ (α,β)∈×  , 
    
    (c)      ψ_0() = 1   ,  ∀∈^ν   ,   ‖ψ_0‖_ = 1                , 
    
    (d)      ∫_^νψ_α()  p_()   d = 0    ,   ∀α∈^*    , 
    
    (e)      ∑_α∈ψ_α() ψ_β()   p_()   d = δ_0(-)
                     ,   ∀ (,) ∈^ν×^ν  .





 (Lemma <ref>).

(a) Since p_^-1/2∈ C^0(^ν), Eqs. (<ref>) yields Eq. (<ref>); combined with
Eq. (<ref>), this yields ‖ψ_α‖_ = ∫_^ν q_α()^2  d = ‖ q_α‖_L^2 = 1.

(b) Using Eq. (<ref>) yields ⟨ϕ_α  , ψ_β⟩_ = ∫_^ν q_α()  q_β()   d = ⟨ q_α  , q_β⟩_L^2= δ_αβ. Thus {ψ_α,α∈} is an orthonormal family in . For all u in L^2(^ν), the linear mapping u↦ = u p_^-1/2 is a continuous injection from L^2(^ν) into   with ‖‖_ = ‖ u ‖_L^2. Therefore, {ψ_α,α∈} is a Hilbert basis of .

(c) The two equations in Eq. (<ref>) are directly deduced from  Eqs. (<ref>) and (<ref>).

(d) Since  ⟨ψ_0  , ψ_α⟩_ = 0 for all α∈^*, we obtain Eq. (<ref>).

(e) Since  {ψ_α,α∈} is a Hilbert basis of , Eq. (<ref>) holds.






 

Let {ψ_α,α∈} be the Hilbert basis of  defined in Lemma <ref>.

(a) For every fixed t > 0, the symmetric kernel k_t can be written, for all  and  in ^ν, as


    k_t(,) =∑_α∈ b_α(t) ψ_α() ψ_α()   ,


in which the family of positive real numbers {b_α(t)= exp(-λ_α  t)  , α∈}, is such that


    1 = b_0(t) > b_1(t) ≥ b_2(t) ≥…  .


(b) If for every fixed t > 0, kernel k_t satisfies


    ∫_^ν∫_^ν k_t(,)^2   p_()  p_()  d  d = c_t^2 < +∞  ,


where c_t > 1  is a positive constant depending on t, then,


    ∑_α∈ b_α(t)^2 = c_t^2 < +∞  .





 (Proposition <ref>).

(a) Substituting  Eq. (<ref>) with q_α() = ψ_α()  p_()^1/2 (see Eq. (<ref>)) into
Eq. (<ref>) yields Eq. (<ref>). From Eq. (<ref>) and since b_α(t)= exp(-λ_α  t), we obtain
Eq. (<ref>).

(b) Assuming Eq. (<ref>), substituting Eq. (<ref>) into Eq. (<ref>), and using Eq. (<ref>) yields
Eq. (<ref>). Since b_0(t) = 1, it can be deduced that c_t^2 > 1 and thus, c_t > 1.






 §.§ Hilbert-Schmidt operator K_t associated with kernel k_t



We now introduce the linear operator in , defined by kernel k_t, and we study its properties and spectrum.


 

For every fixed t > 0, we defined the linear operator K_t from  into  such that, for all u and  in ,


    ⟨ K_t  u  , ⟩_ = ∫_^ν∫_^νk_t(,)  u() ()   p_()  p_()  d  d  ,


where the symmetric kernel k_t verifies the condition defined by Eq. (<ref>).





 

For every fixed t > 0, let K_t be the continuous linear operator defined by Eq. (<ref>), in which kernel k_t is symmetric on ^ν×^ν, and verifies Eq. (<ref>).

(a) For all u and  in , operator K_t is such that


    ⟨ K_t  u  , ⟩_ = ∑_α∈ b_α(t)  ⟨ u  ,ψ_α⟩_ ⟨  ,ψ_α⟩_  ,


and is a positive symmetric operator in . For all u in ,


    K_t  u = ∑_α∈ b_α(t)  ⟨ u  ,ψ_α⟩_ ψ_α  .


(b) For all α in , ψ_α∈ is the eigenfunction independent of t, associated with the positive eigenvalue b_α(t), satisfying Eq. (<ref>), of operator K_t,


    K_t ψ_α = b_α(t)  ψ_α   ,    b_α(t) = exp(-λ_α t)    ,   α∈  ,


which shows that, for all α and β in ,


    ⟨ K_t ψ_α  , ψ_β⟩_ = b_α(t)  δ_αβ  .


(c) For all u in , we have


    ‖ K_t  u ‖_^2 = ∑_α∈ b_α(t)^2  ⟨ u  ,ψ_α⟩_^2   ,


and for Hilbert basis {ψ_α  , α∈} of ,


    ∑_α∈‖ K_t ψ_α‖_^2 =  c_t^2 < + ∞  ,


where c_t^2, defined by Eq. (<ref>), is such that c_t^2 = ∑_α∈ b_α(t)^2, and therefore, is a Hilbert-Schmidt operator in .




 (Proposition <ref>).

Under the condition defined by Eq. (<ref>), it is well known that operator K_t is continuous from  into .

(a) Substituting Eq. (<ref>) into Eq. (<ref>) and using Eq. (<ref>) yield  Eq. (<ref>). This equation shows that K_t is a symmetric and positive operator because ⟨ K_t  u  , u ⟩_ > 0 for all u in  with ‖ u‖_≠ 0. Note that  Eq. (<ref>) is directly deduced from Eq. (<ref>).

(b)  From Eqs. (<ref>) and (<ref>), we have ψ_α∈. Taking u = ψ_β in Eq. (<ref>) and using Eq. (<ref>) yield K_t ψ_β = ∑_α∈ b_α(t)  ⟨ψ_β  ,ψ_α⟩_ ψ_α = b_β(t) ψ_β, Eq. (<ref>) is obtained from Eq. (<ref>) by taking u=ψ_α and = ψ_β.
 The relationship between b_α(t) and λ_α comes from Proposition <ref>.

(c) Using Eqs. (<ref>) and (<ref>) yields Eq. (<ref>). Taking u=ψ_α in Eq. (<ref>) yields
‖ K_t ψ_α‖_^2 = b_α(t)^2. From Eq. (<ref>) and
∑_α∈‖ K_t ψ_α‖_^2 = ∑_α∈ b_α(t)^2, we obtain Eq. (<ref>). It can be deduced (see for instance <cit.>) that K_t is a Hilbert-Schmidt operator in .





 §.§ Finite approximation of operator K_t and of its eigenvalue problem



The Hilbert-Schmidt operator K_t defined by Eq. (<ref>), operates in infinite dimension. The Hilbert basis
{ψ_α  , α∈} (which relates to the Hilbert basis { q_α  , α∈}, see Lemma <ref> and  Eq. (<ref>)) is not explicitly known, thereby preventing the use of the representation defined by Eq. (<ref>).

We must thus construct a finite approximation of K_t.

Since  ^ν is an unbounded set and ν can be very large, classical discretization such as finite-difference or finite-element methods (see <cit.> for Fokker-Planck equation and <cit.> for fractional Fokker-Planck equation) or such methods based on  shape-morphing modes for solving the Fokker-Planck equation as proposed in <cit.>, are not directly adapted for solving the eigenvalue problem of operator L̂_. Another classical method consists in introducing a finite family of functions in , generating a finite dimension subspace of , and in performing the projection of K_t on this finite subspace. Such an approach is not really adapted to operator L̂_ for which a large number of eigenvalues and associated eigenfunctions have to be computed.

It should be noted that a related problem, but distinct from the one addressed, is that of the numerical method for Schrödinger operator and the associated equation (see for instance <cit.> for solving the Schrödinger Equation, <cit.> for the solution of the Schrödinger equation by spectral methods,  <cit.> for numerically solving the time-dependent Schrödinger equation,  and <cit.> for the numerical solution of the Schrödinger equation using finite-difference method).

Nevertheless, such approaches are not well adapted to the objective of the actual developments, which has been detailed in Section <ref>. We then propose to use a statistical sampling of ^ν equipped with the probability measure p_()  d, which will be well adapted to our objective of performing a construction connected to the DMAPS approach.


 

Let us assume that, for every fixed t > 0, we have (,)↦ρ(,t | ,0) ∈ C^0(^ν×^ν). From Eq. (<ref>), it can be deduced that (,)↦  k_t(, ) ∈ C^0(^ν×^ν).
For every fixed t > 0, for all u and  in ∩ C^0(^ν), the restriction to (∩ C^0(^ν))× (∩ C^0(^ν)) of the bilinear form (u,)↦⟨ K_t u  , ⟩_,
defined on × by Eq. (<ref>) with the continuous symmetric function k_t verifying Eq. (<ref>), can be written as


    ⟨ K_t u  , ⟩_  = E{k_t(,)   u()  ()}  ,


in which  is an independent copy of . The joint probability measure P_,(d,d) of  with  is p_()× p_()  d  d.
The real-valued random variables u(), (), and the positive-valued random variable k_t(,), defined on (Θ,,), are second-order random variables,


    E{u()^2 } < +∞   ,    E{()^2 } < +∞   ,    E{k_t(,)^2} < +∞  .


Let Z_t,u, be the real-valued random variable, defined on (Θ,,), such that


    Z_t,u, = k_t(,)   u()  ()  .


Then, Z_t,u, is such that,


    E{Z_t,u,} < +∞  .





 (Proposition <ref>).

(a) Since u and  are continuous functions in , u() and () are real-valued random variables defined on (Θ,,) and are second-order because,


    E{ u()^2} =  ∫_^ν u()^2 p_()  d = ‖ u ‖_^2  < +∞   ,   
    E{()^2}          =  ∫_^ν()^2 p_()  d = ‖‖_^2  < +∞  .


Since k_t∈ C^0(^ν×^ν) (see Eq. (<ref>)) and due to Eq. (<ref>),
k_t(,) is a second-order positive-valued random variable defined on (Θ,,),


    E{k_t(,)^2} =  ∫_^ν∫_^ν k_t(,)^2 p_() p_()    d  d =c_t^2 < +∞  .


(b) For all u in  and for t > 0, from Eq. (<ref>), it can be seen that
(K_t u)() = ∫_^ν k_t(,)  p_()^1/2  u()  p_()^1/2  d.
Using the Cauchy-Schwarz inequality  and Eq. (<ref>) yield


    ‖ K_t u ‖_^2  ≤ ‖ u ‖_^2 ∫_^ν∫_^ν k_t(,)^2 p_() p_()    d  d
    = c_t^2 ‖ u ‖_^2   ,


(which, in passing, shows the continuity of the operator K_t in   as stated at the beginning of the proof of Proposition <ref>).
Consequently, we have ⟨ K_t u  , ⟩_^2 ≤‖ K_t u ‖_^2  ‖‖_^2, which shows that


    ⟨ K_t u  , ⟩_^2  ≤ c_t^2  ‖ u ‖_^2 ‖‖_^2  < +∞  ,


and therefore,  Eq. (<ref>) holds.






 

Let {^i,i=1,…, n_d} and {^j,j=1,…, n_d} be n_d independent copies  of  and , respectively. For every fixed t > 0, and for all u and  in ∩ C^0(^ν), let Z_t,u,^ij be the real valued random variable on (Θ,,), such that, for all i and j in {1,… , n_d},


    Z_t,u,^ij = k_t(^i , ^j)   u(^j) (^i)   .


Let Z_t,u,^ (n_d) be the real-valued random variable on (Θ,,), defined by


    Z_t,u,^ (n_d) = 1/n_d^2∑_i=1^n_d∑_j=1^n_d Z_t,u,^ij  .


Then, Z_t,u,^ (n_d) is an estimator of


    z_  t,u, = E{Z_t,u,} = ⟨ K_t u  , ⟩_  ,


where Z_t,u, is given by Eq. (<ref>).

Let {^j , j=1,… , n_d} be  the n_d independent realizations of  introduced in Section <ref>. Since ^i and ^j are independent copies of  (because  is an independent copy of ), an estimation
of z_  t,u, is a realization z_  t,u,^(n_d) of the estimator  Z_t,u,^ (n_d), which is written as


    z_  t,u,^(n_d) = 1/n_d^2∑_i=1^n_d∑_j=1^n_d k_t(^i,^j)  u(^j) (^i)  .






 

Under the hypotheses and notations of Definition <ref>, the sequence of real-valued random variables
{Z_t,u,^ (n_d)}_n_d on (Θ,,) is convergent in probability to
z_  t,u,,


    ∀ϵ > 0    ,   lim_n_d→ +∞{ | Z_t,u,^ (n_d)
                                   - z_  t,u, |  ≥ ϵ} = 0   .


We have also the almost sure convergence, thanks to the strong law of large numbers,


    {lim_n_d→ +∞Z_t,u,^ (n_d)
                                   = z_  t,u,} = 1   .





 (Lemma <ref>).

The proof uses the usual results from mathematical statistics (see for instance <cit.>).






 

(a) As is well known, the speed of convergence is proportional to 1/√(n_d^2) = 1/n_d and is independent of dimension ν. The quantification of the approximation error could traditionally be estimated using the central limit theorem, which involves the variance of the estimator (see, for instance, <cit.>). In the numerical illustration provided in Section <ref>, we will show the numerical calculation of  the first eigenvalues of the Fokker-Planck operator for which a reference is known.

(b) Lemma <ref> with Definition <ref> and Proposition <ref> allow a finite approximation of ⟨ K_t u  , ⟩_ to be constructed and consequently, to deduce the corresponding finite approximation of the eigenvalue problem defined by Eq. (<ref>).





 

For every fixed t > 0, for all u and  in ∩ C^0(^ν), and for n_d sufficiently large, we have (in the sense of the convergence described in Lemma <ref>),


    ⟨ K_t u  , ⟩_≃⟨   [K(t)]   , ⟩_^n_d  ,


in which ⟨· , ·⟩_^n_d is the usual Euclidean inner product in ^n_d and where [K(t)]∈_n_d^+0 is such that, for all i and j in { 1,…, n_d},


    [K(t)]_ij = 1/n_d  k_t(^i,^j)   ,


and where the vectors  and  in ^ν are such that


    = (u(^1)/√(n_d)  ,  …  ,  u(^n_d)/√(n_d) )   ,   = ((^1)/√(n_d)  ,  …  ,  (^n_d)/√(n_d) )   .


The corresponding finite approximation of the eigenvalue problem defined by Eq. (<ref>) is written as


    [K(t)]  _α(t) = b̂_α(t) _α(t)    ,   _α(t) ∈^n_d  ,


in which


    b̂_0(t) ≥ b̂_1(t)  ≥ … ≥ b̂_n_d-1(t)  ≥ 0  ,


and where the normalization of the eigenvectors is chosen so that for α and β in {0,1,…, n_d-1 },


    ⟨_α(t) , _β(t)  ⟩_^n_d = δ_αβ  .





 (Proposition <ref>).

From Eqs. (<ref>), (<ref>), and (<ref>), it can be deduced that, for n_d sufficiently large,
⟨ K_t u  , ⟩_≃ n_d^-2∑_i=1^n_d∑_j=1^n_d k_t(^i,^j)  u(^j) (^i).
Since k_t(^i,^j) = k_t(^j,^i) by symmetry of k_t (see Eq. (<ref>)), the right-hand side member can be written as ⟨   [K(t)]   , ⟩_^n_d  where [K(t)], , and  are defined by Eqs. (<ref>) and (<ref>). From Proposition <ref>, K_t is a positive operator in , and as Hilbert-Schmidt operator, b_α(t) → 0_+ as α→ +∞. For the finite approximation, matrix [K(t)] is then symmetric  and positive. Using Eq. (<ref>), the finite approximation  of Eq. (<ref>) is then written as Eq. (<ref>). Since
[K(t)] is real, symmetric, and positive, we have Eqs. (<ref>) and we choose the normalization of {_α(t)}_α so that  (<ref>) holds.






 


(a) It should be noted that we have chosen the construction of {_α(t)}_α as an orthonormal basis in ^n_d. Consequently, _α(t) is not related  to ψ_α by a simple sampling, similar to the one described  in  Eq. (<ref>).

(b) In addition, _α(t) depends, a priori, on t, while ψ_α is independent of t. Only for
n_d→ +∞, _α(t) goes to a vector independent of t. Similarly, although b_0(t)= exp(-λ_0 t) = 1 because λ_0 = 0 (see Eq. (<ref>)), we do not have,  a priori, b̂_0(t) = 1, but this equality holds for n_d→ +∞. Nevertheless, for n_d finite, we will exploit this existing dependence on t for the construction of the reduced transient basis at time t, which will be connected to the reduced DMAPS basis for t→ 0.

(c) Note that as n_d →  +∞, b̂_α(t) tends to b_α(t). Since b_α(t) = exp(-λ_α t), that is to say λ_α = -1/t log  b_α(t), we choose to define _α(t) by a similar formula,  such that for t > 0 and α for which b̂_α(t) > 0,


    _α(t)  = -1/t log b̂_α(t)   .


(d) For every fixed t >0, to solve the eigenvalue problem defined by Eq. (<ref>), we have to construct matrix [K(t)] with an adapted methodology. This will be the object of Section <ref>.





§ CONSTRUCTION OF THE MATRIX OF THE FINITE APPROXIMATION




In this section, we present the methodology to construct matrix [K(t)] as defined in Proposition <ref>. This construction requires the numerical evaluation of kernel k_t, because the entry [K(t)]_ij of [K(t)] is
given by k_t(^i,^j)/n_d (see Eq. (<ref>)). According to Definition <ref> (see Eq. (<ref>)), for i and j in {1,… , n_d},


    [K(t)]_ij = 1/n_d ρ(^i,t | ^j,0)/p_(^i)  .


In Eq. (<ref>), p_ is explicitly defined by Eq. (<ref>), and ρ(,t | ,0) is the transient probability density function of the stochastic process {(t), t≥ 0}, starting from (0) = in ^ν. This function is the solution of the ISDE defined by Eq. (<ref>) for t > 0 and with the initial condition (0) = (see Eq. (<ref>)).

We will begin by using a a classical mathematical result concerning the solution of the ISDE, which must be validated for the specific case where the invariant measure is  defined in Section <ref>. Additionally, we will obtain a proof of the properties introduced in Section <ref>.

Then, we will present the numerical method for constructing [K(t)] by numerically solving the ISDE and using nonparametric statistics to estimate ρ(^i,t | ^j,0). Subsequently, we will derive an explicit algebraic formula for [K(t)]_ij.




 §.§ Existence and uniqueness of the solution of ISDE and properties of the transition probability



Let ={(t), t≥ 0} be the ^ν-valued stochastic process satisfying (see Eqs. (<ref>) and (<ref>)),


    d(t) =     ((t))  dt + d(t)    ,    t > 0   , 
    (0) =     ∈^ν  ,   a.s.   ,


with ∈^ν, where the drift ↦() : ^ν↦^ν is defined by Eq. (<ref>) with Eq. (<ref>), and where {(t),t≥ 0} is the normalized Wiener process defined on (Θ,,).



 

Eqs. (<ref>) and (<ref>) define a unique homogeneous diffusion ^ν-valued stochastic process  defined on (Θ,,), whose transition probability measure is homogeneous (that is to say, it depends only on t - 0 = t),


    P_(t)|(0)(,t | ,0) = {(t)∈ | (0)=}   ,    t > 0  ,


where  is any Borel set in ^ν. For all t > 0 and for all  in ^ν, P_(t)|(0)(d , t | ,0) admits a density function ↦ρ(,t | ,0) : ^ν↦ ]0  , +∞[ with respect to the Lebesgue measure d on ^ν, such that


    P_(t) | (0)(d,t | ,0) = ρ(,t | ,0)  d   ,    t > 0   ,
       lim_t→ 0_+ρ(,t | ,0)  d = δ_0(-)   .


Stochastic process  has almost-surely continuous trajectories and for all t > 0, (t) is a second-order random variable,


    ∀ t ≥ 0    ,    E{‖(t)‖^2} < +∞  .


For t→ +∞,  is asymptotic to a stationary stochastic process whose first-order marginal probability measure is the invariant measure p_()  d,


    lim_t→ +∞ρ(,t | ,0)  d = p_()  d  .


The function (,)↦ρ(,t | ,0) is continuous from ^ν×^ν into ^+*,


    ρ(·,t | · ,0) ∈ C^0(^ν×^ν, ^+*)   .





 (Proposition <ref>).
The proof is presented in five steps.

(a) It is easy to prove that drift function b is continuous on ^ν.


(b) Since the diffusion matrix is the identity matrix and b belongs to C^0(^ν,^ν), we can establish the existence of a unique diffusion stochastic process (see <cit.> Ch. VIII, Sec. 2;  <cit.> Ch. IV, Secs. 2, 3, and 5; or <cit.> Ch. V), provided that for all  and ' in ^ν, we have


    ‖()-(')‖ ≤  c ‖ - '‖   ,   ‖()‖ ≤ 
    C  (1 +‖‖)  .


Using Eqs. (<ref>), (<ref>), and (<ref>), () can be rewritten, for all  in ^ν, as


    () =1/2 ξ()^-1∇ξ()    ,  ξ() = 1/n_d∑_j=1^n_dexp{ -1/2ŝ^2‖ŝ/s^j - ‖^2 }  >   0   .


Calculating ∇ξ(), it can be seen that


    2 ‖()‖ ≤ 1/ξ()  n_d  ŝ^2∑_j=1^n_dŝ/s ‖^j‖ exp{ -1/2ŝ^2‖ŝ/s^j - ‖^2 } + 1/ŝ^2 ‖‖  .


Since n_d is finite and since E{‖‖^2} = 1/n_d ∑_j=1^n_d‖^j‖^2 = ν is also finite, we have
sup_j=1,…,n_d‖^j‖ = c_η < +∞ and therefore,


    2 ‖()‖ ≤ 1/2 ( 1/ŝ^2ŝ/s  c_η + 1/ŝ^2 ‖‖ ) ≤  c  (1 + ‖‖)   ,    0 < c < +∞  .


The second inequality in Eq. (<ref>) is then proven. It can now be verified that

    ‖()-(')‖  ≤ 1/2ŝ^2 ( ‖() - (')‖  +  ‖-'‖  )   ,


in which ()= ()/β() where () and β() are written as


    () = ŝ/s∑_j=1^n_d^j exp{ -1/2ŝ^2‖ŝ/s^j - ‖^2 }   ,   β() = ∑_j=1^n_dexp{ -1/2ŝ^2‖ŝ/s^j - ‖^2 }  .


We must have ‖() - (')‖ ≤ c ‖-'‖, that is true if
‖   [∇()] ‖_F  ≤ c̃ < +∞, in which the (ν×ν)-real matrix [∇()] is written as
[∇()] =β()^-1 [∇()] - β()^-2 ()⊗∇β(), and where ‖ · ‖_F is the Frobenius norm. Introducing, temporary, the notation e_j = exp{ -1/2ŝ^2‖ŝ/s^j - ‖^2 }, we have


    [∇()] = 1/ŝ^2 ŝ^2/s^2∑_j=1^n_d∑_j^'=1^n_d e_j e_j^' (^j⊗^j - ^j⊗^j^')/∑_j=1^n_d∑_j^'=1^n_d e_j e_j^'  .


It can then be deduced that,


    ‖  [∇()]  ‖_F  ≤ 1/ŝ^2 ŝ^2/s^2sup_j,j^' ‖^j⊗^j - ^j⊗^j^'‖_F   .


As previously, since sup_j  ‖^j‖ = c_η < +∞, we have ‖  [∇()]  ‖_F  ≤ c̃_F with c̃_F  a finite positive constant independent of , that yields the proof for the Lipchitz continuity of .



(c) We will admit that, for all t >0 and for all  in ^ν, the transition probability measure has a density ↦ρ(,t | ,0) on ^ν with respect to d, and that (,)↦ρ(,t | ,0) is continuous on ^ν×^ν (for all t > 0) (see for instance <cit.> Ch III, Secs. 6 to 9, or <cit.> Ch. 10).


(d)  As a diffusion stochastic process, the trajectories are almost surely continuous functions. Since  is continuous
and ‖()‖ ≤  c  (1+‖‖) implies ‖()‖^2 ≤  2c^2  (1+‖‖^2), and since E{‖(0)‖^2} = ‖‖^2 < +∞ for all fixed  in ^ν, we have Eq. (<ref>) (see <cit.> Ch. IV, Sec. 2).


(e) The existence of the asymptotic stationary solution with Eq. (<ref>) can be found in <cit.> Ch. III, or <cit.> Ch. VI, Secs. 5 and 6.






 §.§ Rewriting the Itô equation in a matrix form



Let {^j(t),t≥ 0} be the solution of Eqs. (<ref>) and (<ref>) with the initial condition ^j(0) =^j∈^ν, in which ^j is defined in Section <ref>. We then have


    ρ(,t | ^j,0)  d = P_^j(t) | ^j(0)(d,t | ^j,0)  .


Let { [ (t) ] , t ≥ 0} be the _ν,n_d-valued stochastic process and [η_d] be the matrix in _ν,n_d such that


    [ (t) ] = [^1(t) …^n_d(t) ]    ,    [η_d] = [^1 …^n_d]   .


Therefore, { [ (t) ] , t ≥ 0} is solution of the matrix-valued ISDE,


    d[(t)] =     1/2 [L([(t)])]  dt + d[(t)]    ,    t > 0   , 
    
    [(0)] =      [η_d]   ,   a.s.   ,


where [y]↦ [L([y])] is the function from _ν,n_d into _ν,n_d, defined, for k∈{1,… , ν} and
j∈{1,… , n_d}, by


    [L([y])]_kj = 1/ξ(^j) ∂ξ(^j) /∂ y^j_k  .


In Eq. (<ref>), ^j is the j-th column of [y] = [^1 …^n_d], where ^j = (y^j_1,… , y^j_ν). Additionally, ξ() is defined by Eq. (<ref>), and { [ (t) ] , t ≥ 0} is the normalized _ν,n_d-valued Wiener stochastic process.




 §.§ Time-discrete approximation of the ISDE and convergence analysis



To estimate [K()]_ij, defined by Eq. (<ref>), using nonparametric statistics, we need to generate realizations of
Eqs. (<ref>) and (<ref>). Consequently, a first stage involves  introducing a time-discrete approximation of
Eq. (<ref>) and analyzing the convergence.


(i) Time sampling Δ t and δ t.

We first define a time sampling , n≥ 1 to be used for estimating [K()]. The time step Δ t
will be defined in Section <ref>. However, Δ t may not be sufficiently small to achieve  a satisfactory
rate of convergence. We then introduce δ t ≤Δ t such that Δ t = n_s ×δ t, with n_s ≥ 1. To discretize Eq. (<ref>), we use the time sampling {μ δ t, μ≥ 1}.


(ii) ISDE discretization.

Assuming that δ t is sufficiently small relative to 1, employing the Euler scheme (as seen, for example, in <cit.>) to discretize the solution { [ (t) ] , t ≥ 0} of Eqs. (<ref>) and (<ref>) yields


    [_μ] =       [_μ -1] + δ t/2  [L([_μ -1])] + √(δ t)  [_μ]
                       ,   μ≥ 1   , 
    
    [_0] =      [η_d]  ,   a.s.          ,


where [_μ] is the approximation of [(t)] at t = μ δ t, and where
{[_μ]_kj  ; k=1,…,ν  ; j = 1,… , n_d  ; μ≥ 1} is an infinite family of independent normalized Gaussian real-valued random variables.



(iii) Convergence of the time-discrete approximation.

For each j in 1,…,n_d, Proposition <ref> can be applied to the stochastic process {^j(t), t≥ 0}, where ^j(t) represents the j-th column of [(t)].
Let { [^δ(t)],t≥ 0} be the time-discrete approximation of { [(t)],t≥ 0}. Take any fixed positive real number T,  let μ_T = int (T/δ t) denotes the nearest integer to T/δ t. It can be observed that as δ t approaches 0, μ_T tends to +∞. The following classical result holds (see, for instance, <cit.>, Page 323).


 

Under Proposition <ref>, the time-discrete approximation { [^δ(t)],t≥ 0} of { [(t)],t≥ 0} converges strongly  at time T if


    lim_δ t → 0 E{ ‖   [_μ_T] - [(T)] ‖^2_F } = 0   ,


where [_μ_T] is determined by Eqs. (<ref>) and (<ref>) up to μ=μ_T.



(iv) Criterion to determine if δ t is small enough.

We can employ a criterion based on weak convergence, which is related to the covariance matrix [C_^j(t)] ∈_ν^+ of the ^ν-valued random variable ^j(t), for t fixed in the interval ]0, T].

Let σ_^2(t) be defined by


    σ_^2(t) = E{ ‖ [(t)] - E{[(t)]} ‖_F^2 } = ∑_j=1^n_d∑_k=1^ν
                   E{ (Y_k^j(t) - E{Y_k^j(t) } )^2}  .


It can easily be seen that


    σ_^2(t) = ∑_j=1^n_d [C_^j(t)]   .


Let σ__μ_T^2  be the corresponding quantity for random matrix [_μ_T],


    σ__μ_T^2 = E{ ‖ [_μ_T] - E{[_μ_T]} ‖_F^2 } = ∑_j=1^n_d [C_^j_μ_T]  .


The criterion can then be based on the following properties,


    lim_δ t → 0  | σ__μ_T - σ_(T) |   =   0   .


We will detailed this criterion in paragraph (vi).



(v) Generation of independent realizations of the time-discrete approximation.

Let n_s≥ 1 and N > 1 be two fixed integers. Let Δ t be fixed and let δ t = Δ t / n_s. Note that μ_T introduced in Section <ref>-(iv) is such that μ_T = n_s× N. We define T, , and _μ by,


    T = N ×Δ t = n_s× N ×δ t    ,    ={1,… , N}   ,   _μ = { 1,… , n_s × N}  .


For all μ in _μ, let { [γ_μ^ℓ]∈_ν,n_d  , ℓ=1,…, n_} be n_ independent
realizations of the random matrix [_μ]. As discussed in  Section <ref>-(ii), it can be seen that the family
{ [γ_μ^ℓ]  , ℓ=1,…, n_  ; μ∈_μ} consists of n_× n_s× N independent realizations. For each ℓ in {1,… , n_}, the realization { [ỹ_μ^ℓ]  , μ∈_μ} of the time-discrete approximation
{ [_μ]  , μ∈_μ} is computed using the following recurrence (refer to Eqs. (<ref>) and (<ref>)),


    [ỹ_μ^ℓ] =      [ỹ_μ-1^ℓ] + δ t/2  [L([ỹ_μ-1^ℓ])] + √(δ t)  [γ_μ^ℓ]
                       ,   μ∈_μ  , 
    
    [ỹ_0^ℓ] =      [η_d]                      ,


In the following, for performing the statistical estimations, we will use the subsequence
{ [y_n^ℓ]  , n∈} of { [ỹ_μ^ℓ]  , μ∈_μ}, such that


    ∀ n ∈   ,    [y_n^ℓ] = [ỹ_μ^ℓ]    ,   μ = n_s× n  .



(vi) Practical criteria for controlling the convergence parameters.

Let Δ t be fixed, as well as n_s, meaning δ t is fixed, to ensure satisfaction of the  convergence criteria introduced in Section <ref>-(iv). An additional practical criterion can be applied to verify the adequacy of all the convergence parameters, as provided in Lemma <ref>.


 

For k∈{1,… , ν}, j∈{1,…, n_d}, and n∈, let [y_ n] and [σ_n] be the matrices in _ν,n_d defined by,


    [y_ n]_kj =  1/n_∑_ℓ=1^n_ [y_n^ℓ]_kj   ,   
    ([σ_n]_kj)^2 =  1/n_∑_ℓ=1^n_ ([y_n^ℓ]_kj - [y_ n]_kj) ^2  .


Let n↦y(n) and n↦σ(n) be the positive-valued functions defined, for n∈, by


    y(n)  = 1/√(ν× n_d)  ‖ [y_ n]  ‖_F
        ,   σ(n) = 1/√(ν× n_d)  ‖ [σ_n]  ‖_F   .


Then, for δ t → 0 and n_→ +∞, we have


    lim_N→ +∞ y(n)  = 0    ,   lim_N→ +∞ σ(n) = 1   .





 (Lemma <ref>).

It can be seen that [y_ n]_kj and ([σ_n]_kj)^2 are the empirical estimates of the mean value and the variance of the time-discrete approximation [_μ]_kj for μ = n_s× n of Y_k^j(). From Proposition <ref>, we know that, for t→ +∞, {(t)  , t≥ 0}, and consequently, {^j(t)  , t≥ 0}, is asymptotically stationary and that Eq. (<ref>) holds. Since E{} = _ν and E{⊗} = [I_ν] (see Eqs. (<ref>) and (<ref>)), if
δ t → 0 and n_→ +∞, we have for N→ +∞,
[y_ n]_kj→ E{H_k} = 0 and  ([σ_n]_kj)^2 → [C_d]_kk = 1.
Since ‖ [y_ n]   ‖_F^2 = ∑_k=1^ν∑_j=1^n_d ( [y_ n]_kj)^2 and
‖ [σ_n]   ‖_F^2 = ∑_k=1^ν∑_j=1^n_d ( [σ_n]_kj)^2, using the same normalization 1/√(ν× n_d) for y(n) and σ(n), we obtain Eq. (<ref>).







 §.§ Estimation of the matrix of the finite approximation



We now can estimate [K()]_ij defined by Eq. (<ref>) using the realizations { [y_n^ℓ],ℓ=1,… , n_} for n∈, defined by Eq. (<ref>).


 
Under Propositions <ref> and <ref>, results and notations of Section <ref>-(v), for
n ∈, an estimate of matrix [K()],  whose entries are defined by Eq. (<ref>), is written as


    [K()] = [B]^-1  [()]  ,


in which [B] is the diagonal matrix in _n_d for which its entries are


    [B]_ij = δ_ij ∑_j'=1^n_dexp{ -1/2 ŝ^ 2‖^i - ŝ/s^j'‖^2 }  .


The entries of matrix [()] in  _n_d are written as,


    [()]_ij =
     1/n_∑_ℓ=1^n_ ( ŝ^ ν/ s^ν_∏_k=1^ν [σ_n]_kj )
      exp{ -1/2∑_k=1^ν ( η_k^i - [y_n^ℓ]_kj/s_ [σ_n]_kj )^2  }  ,


in which s and ŝ are defined by Eq. (<ref>), where s_ is written as


    s_ = {4/n_(2+ν)}^1/(ν +4)  ,


where [y_n^ℓ]∈_ν,n_d is defined by Eq. (<ref>), and where [σ_n]∈_ν,n_d is defined by Eq. (<ref>).




 (Proposition <ref>).

Eq. (<ref>) is deduced from Eq. (<ref>) with t= and with p_(^i) given by Eq. (<ref>) in which =^i, and with the estimate of ρ(^i, | ^j,0) given by the Gaussian kernel-density estimation method (see <cit.>) with the Silverman bandwidth s_ defined by Eq. (<ref>), which is written as


    ρ(^i, | ^j,0) =
     1/n_∑_ℓ=1^n_1/(√(2π) s_)^ν∏_k=1^ν [σ_n]_kjexp{ -1/2∑_k=1^ν ( η_k^i - [y_n^ℓ]_kj/s_ [σ_n]_kj )^2  }  .






  

(a) In Section <ref>, we will provide an illustration by numerically solving the eigenvalue problem defined by Eq. (<ref>) for t= n Δ t, using Eq. (<ref>). Specifically, we consider the Gaussian case with dimension ν=1. In this scenario,  represents a normalized Gaussian real-valued random variable. For this case, a reference solution is available.

(b) Nevertheless, it would be challenging to employ such a formulation with reasonable convergence in very high dimensions, where ν equals several tens or even hundreds. This would necessitate a large value of n_d.

(c) In fact, Proposition <ref> will be employed in Section <ref> to derive an expression that connects to the kernel [_]_ij = exp{ -1/4ε_‖^i-^j ‖^2 }, which is used to calculate the DMAPS basis and proves efficient for large values of ν.






§ NUMERICAL ILLUSTRATION OF THE PROPOSED FORMULATION




As explained in Remark <ref>-(a), this section presents a numerical illustration of the formulation introduced in Section <ref> to solve the approximated eigenvalue problem defined by Eq.(<ref>), which is derived from the eigenvalue problem in Eq.(<ref>). To validate the formulation, we select a reference case where the eigenvalue problem defined by Eq. (<ref>) can be exactly solved. This is feasible when the dimension ν of  is 1 and  is a normalized Gaussian real-valued random variable.




 §.§ Reference case definition and explicit solution



The quantities related to the reference case will be indexed by letter r. The probability density function of H_r on 
is p_H_r(y) = (2π)^-1/2exp(-y^2/2). The potential function Φ_r (see Eq. (<ref>)) is Φ_r(y) = y^2/2 and the drift b_r defined by Eq. (<ref>) is b_r(y)  = -y/2. The ISDE defined by Eqs. (<ref>) and (<ref>) are rewritten as



    dY_r(t)      =  -1/2 Y_r(t)  dt + dW_r(t)    ,    t > 0  , 
    
      Y_r(0)      =  x ∈  .


Consequently, {Y_r(t) , t≥ 0} is a second-order Gaussian stochastic process, which is explicitely defined by


    Y_r(t) = x  e^-t/2 + ∫_0^ t e^-(t-τ)/2  dW_r(τ)   .


A simple calculation shows that the mean value m_r(t) = E{Y_r(t)} and the standard deviation
σ_r(t) = (E{(Y_r(t) -m_r(t))^2})^1/2 of the random variable Y_r(t) for fixed t > 0, are written as


    m_r(t) = x  e^-t/2   ,   σ_r(t) = √(1-e^-t)  .


For all t > 0, as  { Y_r(t) |  Y_r(0) = x} is a Gaussian random variable, the transition probability density function is


    ρ_r(y,t  |  x,0) = 1/√(2π) σ_r(t)exp{ -1/2σ_r(t)^2(y-m_r(t))^2}  .


Note that Eqs. (<ref>) and (<ref>) show that, we effectively have
lim_t→ 0_+ρ_r(y,t  |  x,0)  dy = δ_0(y-x) (see Eq. (<ref>)) and
lim_t→ +∞ρ_r(y,t  |  x,0) = p_H_r(y) (see Eq. (<ref>)).
It can be deduced that the kernel k_r,t(y,x) on ×, defined by Eq. (<ref>) is written, for t > 0, as


    k_r,t(y,x) = 1/√(1 -e^-t)exp{- 1/2e^-t/(1-e^-t) (y^2 + x^2 -2e^t/2 y  x)^2}  .


Let {h_α(y)  , α∈} be the Hermite polynomials and {h_α(y)/√(α!)  , α∈} the Hilbert basis in L^2(;p_H_r),


    ∫_h_α(y)/√(α!) h_β(y)/√(β!)  p_H_r(y)   dy  = δ_αβ   ,    h_0(y)=1    ,     h_1(y) = y    ,     h_α+1(y) = y  h_α(y) - d  /dy h_α(y)  .


Let {_α(y)  , α∈} be the polynomials defined by


    h_α(y) = 1/√(2^α) _α(y/√(2))    ,   _α(y) = √(2^α)  h_α(√(2) y)    .


We have the formula <cit.>,


    1/√(1 - 4a^2)exp{4 a/1-4 a^2 (x̂ ŷ - ax̂^2 - a ŷ^2)} =
    ∑_α∈a^α/α ! _α(x̂)  _α(ŷ)  .


Taking x= √(2) x̂, y= √(2) ŷ, and a =1/2e^-t/2 < 1/2 for t > 0, Eq. (<ref>) can be rewritten as


    k_r,t(y,x) = ∑_α∈ e^-λ_r,α tψ_r,α(y) ψ_r,α(x)   ,




    λ_r,α = α/2   ,   ψ_r,α(y)= 1/√(2^α α !) _α(y/√(2))
    = 1/√(α !)  h_α(y)  ,


and thus, for α and β in ,


    ∫_ψ_r,α(y) ψ_r, β(y)  p_H_r(y)  dy = δ_α, β  .


Comparing Eqs. (<ref>) and (<ref>)  with Eqs. (<ref>) and (<ref>) shows that, for this reference case, the eigenvalues of the Fokker-Planck operator L_ are


    λ_r,α = α/2    ,   α∈  .






 §.§ Estimating the eigenvalues with the proposed numerical formulation



(i) For the convergence analysis, we consider 10 values of n_d constituting the set _d = {100, 300, 400, 800, 1000, 1200,  1500, 1800, 2000, 2200}. For each n_d∈_d, the matrix [η_d] =[η^1 …η^n_d]∈_1,n_d is generated with an adapted generator (instruction randn(1,n_d) for Matlab Gaussian generator).

(ii) For each value of n_d in _d, the generation of n_ = n_d independent realizations { [ỹ_μ^ℓ] , μ∈_μ} with _μ = {1,… , n_s× N} is performed using Eqs. (<ref>) and (<ref>) with n_s=1, δ t = Δ t = 0.061796, N=150. For n_d=1200 and n_ = 1200, Figs. <ref> and <ref> display the graphs of functions n↦y(n) and n↦σ(n) computed  using Eq. (<ref>) with Eq. (<ref>). It can be seen that the criterion defined by Eq. (<ref>) is satisfied.




(iii) For every n_d∈_d, matrix [K(2 Δ t)] ∈_n_d^+0 is calculated using Eq. (<ref>) with
ŝ = 0.2486, ŝ / s = 0.9690, and s_ =0.2565. The first 6 largest eigenvalues, b̂_α(2 Δ t) of
[K(2 Δ t)] and  _α(2 Δ t), simply denoted by _α, are obtained from
Eq. (<ref>).

Fig. <ref> shows the graph of function n_d↦_λ(n_d) that quantifies the relative error between α↦λ_r,α and α↦_α, written as
_λ(n_d) = ∑_α=0^5(λ_r,α- _α)^2 / ∑_α=0^5 λ_r,α^2.
It can be seen that the error decreases as n_d increases. For n_d = n_ = 1200, Fig. <ref> compares the reference
eigenvalues λ_r,α =α/2 (see Eq. (<ref>)) with the computed eigenvalues _α  for α=0,1, …,5. The comparison is good enough.







§ VECTOR BASIS FOR PLOM DERIVED FROM THE TRANSIENT ANISOTROPIC KERNEL, CONNECTED TO THE DMAPS BASIS




We revisit the objective presented in Section <ref>.
The diffusion-maps (DMAPS) basis, used by PLoM <cit.>, is associated with the isotropic kernel detailed in <cit.>. This section introduces the construction of a transient vector basis, based on the transient anisotropic
kernel described in Section <ref> (see Proposition <ref>). This approach incorporates the requirement that as Δ t → 0, the transient kernel at the first time Δ t coincides with the DMAPS isotropic kernel. Consequently, the two vector bases will be linked asymptotically as t→ 0. To develop such a "connected" transient anisotropic kernel to the DMAPS isotropic kernel, we undertake a reparameterization of Eq. (<ref>), which defines the matrix [K()]∈_n_d^+0 for n∈. This reformulation enables the construction of the vector basis for high dimensions (large values of ν) and relatively small n_d, as implemented in PLoM, designed for probabilistic learning with small training datasets.




 §.§ Transient anisotropic kernel connected to the DMAPS isotropic kernel



The following definition provides the construction of a transient anisotropic kernel that is linked to the DMAPS isotropic kernel. This construction draws inspiration from the expression of [K(Δ t)] defined by Eq. (<ref>).



 
For each n in , for Δ t defined in Section <ref>-(i), for n_ and [y_n^ℓ] defined in
Section <ref>-(v), for [σ_n] defined by Eq. (<ref>), and for  a given real ε_ > 0, we define the
matrix K̃()]∈_n_d^+0, such that


    [K̃()] = [B]^-1 [()]  ,


such that, for all i and j in {1,…,n_d}, the matrix [()]∈_n_d has entries,


    [()]_ij =
     1/n_∑_ℓ=1^n_ ( Π_k=1^ν{[σ_n]_kj/Δ t} )^-1exp{ -1/4ε_∑_k=1^ν ( η_k^i - [y_n^ℓ]_kj/[σ_n]_kj/√(Δ t) )^2  }  ,


and [B]∈_n_d is a diagonal matrix whose entries are,


    [B]_ij = δ_ij∑_j'=1^n_dexp{ -1/4ε_‖^i -^j'‖^2 }  .







 

An optimal value, ε_, for the smoothing parameter ε_, is proposed for the PLoM algorithm in <cit.> and is detailed in <ref>-(i). This optimal value enables the analysis of high-dimensional problems (large value of ν).
It should be noted that Proposition <ref> will explain the definition of [K̃()] as defined by Eqs. (<ref>) to (<ref>), showing the connection with DMAPS.





 
We use the notation introduced in Definition <ref>. Let [_]∈_n_d^+0 be the matrix of the DMAPS isotropic kernel, defined for all  i, j ∈{1,… , n_d}, by


    [_]_ij = exp{-1/4 ε_‖^i-^j‖^2}  .


Let [K_]∈_n_d be the matrix defined by


    [K_] = [B]^-1 [_]  ,


where [B] is the diagonal matrix defined by Eq. (<ref>). Let Δ t be defined by


    Δ t = ŝ^2 / κ   ,   κ≥ 1   ,


where ŝ is defined by Eq. (<ref>). Then, as κ→ +∞, and consequently Δ t → 0, we have


    [K̃(Δ t)] → [K_]   ,


which means that, for n=1, the matrix [K̃(nΔ t)] of the transient anisotropic kernel, defined by Eq. (<ref>), converges to the matrix [K_] of the DMAPS isotropic kernel as Δ t → 0.





 (Proposition <ref>).
For n=1, for κ→ +∞, that is to say, for Δ t → 0, Eqs. (<ref>) and (<ref>) show that, for all ℓ∈{1,…,n_}, k∈{1,…,ν}, and j∈{1,… , n_d}, we have [y_1^ℓ]_kj→η_k^j and [σ_n]_kj→√(Δ t), and consequently, [(Δ t)] → [_]. From Eqs. (<ref>) and (<ref>), it can be deduced Eq. (<ref>).





 §.§ Construction of the reduced-order transient basis and its counterpart for the DMAPS basis



In this section, we: (i) review the construction of the reduced-order diffusion-maps (DMAPS) basis (RODB) represented by a matrix
[g_]∈_n_d,m_ with m_ < n_d, and (ii) construct, for each n∈, the reduced-order transient basis (ROTB()) represented by a matrix [g(nΔ t)]∈_n_d,m_.




(i) Reminder of the construction of the RODB.

This construction, due to <cit.>, is the one used by the PLoM algorithm <cit.>.
The matrix [K_], defined by Eq. (<ref>), has positive entries and represents the transition matrix of a Markov chain.
The eigenvalues b_,1,…,b_,n_d and the associated eigenvectors _^1,…,_^n_d are such that


    [K_] _^ β = b_, β _^ β   ,   
    1=b_,1 > b_,2≥…≥ b_,n_d  .


Using Eq. (<ref>), this eigenvalue problem  is rewritten as the symmetric eigenvalue problem,


    ^S_ ^ β = b_, β ^ β   ,   ⟨^ β,^ β'⟩ =δ_ββ'   ,   _^ β = [B]^-1/2^ β  ,


in which ^S_ = [B]^-1/2[_] [B]^-1/2 is a symmetric matrix.
The diffusion-maps basis {_^1,…, _^n_d} forms a vector basis of ^n_d.
As explained in <cit.>, PLoM uses the RODB of order m, which  is defined by
{_^1 …_^ m}.

This basis depends on  ε_ and m.

The optimal value m_ of m is defined (see <cit.>) by


    m_ = ν +1   .


The optimal value ε_ of ε_ is estimated to obtain


    1 = b_,1 > b_,2≃…≃ b_,m_≫ b_,m_ + 1≥…≥ b_,n_d > 0  ,


in which the jump amplitude


    J_ = b_,m_ + 1  /b_,m_  ,


must be equal to J_=0.1 (following <cit.>), but which can also be chosen in the interval [0.1  , 0.5] when ν is large.

Therefore, the RODB is defined for m = m_ and is represented by the matrix,


    [g_] = [_^1 …_^m_] ∈_n_d,m_  .




(ii) Construction of the ROTB().

For each n in , the ROTB() represented by [g(nΔ t)] is constructed as the eigenvectors of matrix
[K̃(nΔ t)] defined by Eq. (<ref>). Taking into account Eq. (<ref>), we want that for n=1, [g(Δ t)]∈_n_d,m_ goes to [g_]∈_n_d,m_ as Δ t → 0. For n > 1, matrix [K̃(nΔ t)] is a priori symmetric, but matrix [(nΔ t)] is not symmetric. For applying a similar approach to the one defined by Eq. (<ref>), we then symmetrize the matrix [()] = [B]^-1/2 [(nΔ t)]   [B]^-1/2, introducing
[^S()] = ([()] + [()]^T)/2.

The transient basis {^1(), … , ^n_d()} associated with the eigenvalues
b̃_1() ≥b̃_2() ≥…≥b̃_n_d()
are then computed by solving the symmetric eigenvalue problem


    [^S()] ^ β() =  b̃_β()  ^ β()    ,   ⟨^ β()   , ^ β'()⟩ = δ_ββ'  .




    ^ β() = [B]^-1/2^ β()   .


The ROTB() is then defined by the matrix



    [g()] = [^1() …^m_()]∈_n_d,m_  ,


in which m_ is defined as explained in Section <ref>-(i).




 §.§ Criteria for comparing the reduced-order transient basis with the reduced-order DMAPS basis



The PLoM algorithm is based on the use of the RODB (see <ref>). With such an RODB based on the isotropic kernel, PLoM has proven generally efficient, even in extremely difficult cases, as demonstrated in numerous publications since 2016. This efficacy will also be evident through the applications presented in Section <ref>. However, for cases involving very heterogeneous data in the training dataset, the learned statistical dependence between the components of the random vector  (through the learned probability measure for ) can a priori be improved using ROTB() for given n. To assess a possible improvement with this reduced-order transient basis compared to the reduced-order DMAPS basis, quantitative criteria are necessary. In this section, we introduce criteria for comparing the two vector bases, and then, in Section <ref>, we will present a methodology for identifying the instance, , that maximizes the selection criteria of the ROTB().


(i) The first criterion will be the angle between the two vector subspaces generated by the two vector bases, ROTB() and RODB. If this angle is close to zero then the two bases coincide. This must be the case when we choose  the instant n=1 for the ROTB()  (See Proposition <ref>).

(ii) The second criterion is the concentration of the learned probability measure in relation to the concentration of the probability measure of the training dataset. It has been proven that the PLoM algorithm, which uses the RODB, was designed to preserve the concentration of the learned probability measure from a small training dataset. For this, in <cit.>, we introduced the indicator d^2, linked to the mean-square convergence, which we will recall and use. We also introduce a second concentration criterion, KL, based on the Kullback-Leibler divergence <cit.> between the learned probability measure and the probability measure of the training dataset.

(iii) The third criterion is mutual information <cit.>, which is defined as the relative entropy introduced by Kullback and Leibler <cit.>. This will be used to quantify the level of statistical dependencies among the components of the centered random vector , whose covariance matrix is the identity matrix. Such mutual information will be estimated for the learned probability measure generated by PLoM, comparing the RODB and the ROTB() as a function of n.

(iv) The last criterion is the Entropy from Information Theory <cit.>, introduced by Shannon <cit.>. The estimation of this entropy is a function of the number of realizations  used for estimating the probability measure. This property will be used in Section <ref> to normalize the estimation of the mutual information.




  §.§.§ Angle between the subspaces spanned by ROTB() and RODB



Let n be fixed in . It is assumed that rank([g_]) = rank([g()])= m_ in which the two matrices are defined by Eqs. (<ref>) and (<ref>). In addition, it is assumed that the null space of the matrix [g()]^T  [g_] is {0}.
Let V() = span{^1(), … , ^m_()} be the m_-dimension subspace of ^n_d (spanned by the ROTB()).
Let V_ = span{^1_, … , _^m_} be the m_-dimension subspace of ^n_d (spanned by the RODB).

For β∈{1,… m_}, let ĝ_^ β = g_^ β / ‖ g_^ β‖ be the normalized vector g_^ β and let ĝ^ β() = g^ β() / ‖ g^ β()‖ be the normalized vector g^ β().
Let [ĝ_] = [ĝ^1_…ĝ_^m_ ] and
[ĝ()] = [ĝ^1()…ĝ()^m_ ] be matrices in _n_d, m_.
The angle γ () between the subspaces V() and V_ is defined by


    γ () = arccos ( σ_min([ĝ()]^T  [ĝ_] ) )   ,


in which σ_min denotes the smallest singular value. If the angle is close to 0, the two subspaces are nearly linearly dependent.



  §.§.§ Indicators related to the learned probability measure



In this section, we detail the indicators used for comparison: concentration of the learned probability measure, mutual information, and entropy. These will be employed to compare the probability measures associated with the training dataset and the learned dataset generated by the PLoM algorithm, as summarized in <ref>. Specifically, we use the reduced-order DMAPS basis (RODB) and, alternatively, the reduced-order transient basis (ROTB()). To facilitate the comparisons presented in the numerical illustrations, we introduce the necessary notations for clarifying the diverse quantities and their numerical calculations.

- Training dataset. In Section <ref>, the independent realizations of the ^ν-valued random variable  are ^1,… ,^n_d, and the matrix [η_d] = [^1…^n_d]∈_ν,n_d represents one realization of the random matrix [], used by PLoM, and defined in <ref>. The probability density function of  is p_ defined by Eq. (<ref>).

- Learned dataset. In <ref>, the learned realizations generated by PLoM are those of the _ν,n_d-valued random variable [_] and are written as [η_^ℓ], ℓ=1,…, n_ (see <ref>). The corresponding realizations _^ℓ', ℓ=1,…, n_, with n_ = n_d × n_, of the ^ν-valued random variable _ are obtained by reshaping. The learned probability density function of _ is denoted by p__. When PLoM is used with RODB, [_], [η_^ℓ], _^ℓ', and p__ will be rewritten as [_], [η_^ℓ], _^ℓ', and p_, respectively. When PLoM is used with ROTB(), [_], [η_^ℓ], _^ℓ', and p__ will be rewritten as [_()], [η_^ℓ()], _^ℓ'(), and
p_( .   ; ), respectively.



(i) Concentration of the learned probability measure for RODB and ROTB().

Based on the mean-square norm, for PLoM with RODB and ROTB(), the learned probability measure concentration is written (see Eq. (<ref>)) as


    d_^  2(m_) = E{ ‖  [_] - [η_d] ‖^2 }   / ‖   [η_d] ‖^2  ,



    d_^  2(m_;) = E{ ‖  [_()] - [η_d] ‖^2 }  /  ‖  [η_d] ‖^2  .


Using the realizations, these quantities are estimated by


    d̂_^  2(m_) = (1/n_)∑_ℓ=1^n_{ ‖   [_^ℓ]  - [η_d] ‖^2 }   / ‖   [η_d] ‖^2   .



    d̂_^  2(m_; ) = (1/n_)∑_ℓ=1^n_{ ‖   [_^ℓ()]  - [η_d] ‖^2 }  / ‖  [η_d] ‖^2   .


The concentration of the learned probability measure can also be estimated using the Kullback-Leiber divergence and its estimation from a set of realizations as presented in <ref>. For the PLoM formulated with RODB and ROTB(), we then have,


    D(p_ ‖  p_ )  = ∫_^ν p_() log ( p_()/p_())   d  ,



    D(p_( .  ;) ‖  p_ )  = ∫_^ν p_(  ; ) log ( p_( ; )/p_())   d  .




(ii) Mutual-information-based components statistical dependencies of the learned probability measure for  RODB and ROTB().

The mutual information and its estimation from a set of realizations are presented in <ref>. For the training dataset and for PLoM formulated with RODB and ROTB(), we have , for ^ν-valued random variables , _, and _(),


    I() = D(p_ ‖ ⊗_k=1^ν p_H_k) =
                  ∫_^ν p_() log ( p_()/Π_k=1^ν p_H_k(η_k))   d  ,




    I(_) = D(p_ ‖ ⊗_k=1^ν p_,k) =
                 ∫_^ν p_() log ( p_()/Π_k=1^ν p_,k(η_k))   d  ,




    I(_;) = D(p_(·  ; )  ‖ ⊗_k=1^ν p_,k(·   ; )) =
       ∫_^ν p_(  ;) log ( p_(  ;)/Π_k=1^ν p_,k(η_k  ;))   d  ,


in which p_,k is the pdf of component H_,k of _ and where p_,k(·   ; ) is the pdf of component
H_,k() of _().



(iii) Entropy for RODB and ROTB().
The entropy and its estimation from a set of realizations are presented in <ref>. For the training dataset and for PLoM formulated with RODB and ROTB(), we have for ^ν-valued random variables , _, and _(),


    S_ = -∫_^ν p_() log p_()   d   ,   
    S_ = -∫_^ν  p_() log p_()   d  ,



    S_() = -∫_^ν  p_(  ; ) log p_(  ; )   d  .





 §.§ Identification methodology for the instant maximizing the selection criterion of the ROTB()



In this section, all quantities denoted with a hat (such as Î()) represent the estimated values of the corresponding quantities without a hat (such as I()), using the realizations as explained in <ref>.
Let n_d be the number of points in the training dataset and n_ = n_d× n_≫ n_d be the number of learned realizations with PLoM (see <ref>), either with the RODB or with the ROTB().



(i) Defining the subset _N of  containing the admissible values of n.

Let N be the largest value of n for which the ROTB(), represented by matrix [g()], is computed. This value N being fixed, the set = {1,…, N} defined by Eq. (<ref>) is fixed.

The subset _N of the admissible values of n is then defined by


    _N = { n∈  ,  1/ν d̂_^ 2(m_;) ≤τ_c ≪ 1}  ,


where d̂_^  2(m_;) is defined by Eq. (<ref>) and where 0 < τ_c ≪ 1 is fixed sufficiently small with respect to 1 in order to preserve the concentration of the learned probability measure (see Section <ref>-(i) and <ref>). It should be noted that the maximum value of d_^  2(n_d) is 2 (corresponding to a measure concentration completely lost, which can be obtained  with classical MCMC algorithms). The PLoM algorithm allows for preserving the concentration of the learned probability measure <cit.> yielding values of the order 0.01 to 0.1 through all the performed applications. For defining the concentration criterion in Eq. (<ref>), we have normalized with respect to dimension ν.


(ii) Characterizing a better ROTB()  compared to RODB.

From Proposition <ref>, in particular from Eq. (<ref>), it can be deduced that, for n=1, we have Î(_  ; 1 ×Δ t) ≃Î(_) for Δ t sufficiently small (which is the considered case).

Let us assume that Î() < Î(_). For n fixed in _N, we will say that the ROTB(), represented by matrix [g()], is better than the RODB, represented by matrix [g_], if


    Î() ≤Î(_  ; ) <  Î(_)   .



(iii) Determining the optimal value n_ of n in _N.

The underlying idea is to select a reduced-order transient basis that gives a learned probability measure whose mutual information is as close as possible to the mutual information of the probability measure p_(), d.

Based on paragraph (ii) above, the optimal transient basis [g(n_)] among the set of possible
transient bases { [g()] , n∈} is obtained for n=n_ such that


    n_ = min_n ∈_NÎ(_  ; )   .



(iv) Defining the normalized estimate of the mutual information.

As explained in <ref>, if n_samp is the number of realizations (either n_d or n_), the entropy estimation asymptotically decreases as -log(n_samp) when n_samp increases.
For comparing the estimate of the mutual information for , which uses n_d realizations, with the one of _ or _(), which uses n_≫ n_d learned realizations, we must normalize the estimated mutual information with respect to the number of realizations. Taking into account the relationship between the mutual information and the entropy estimates (see <ref>), we chose to normalize the estimated mutual information by dividing by the function χ + log(n_samp) (with n_samp=n_d or n_samp=n_), where χ is a real number that must be identified and which must be such that χ + log(n_d) > 0. Since n_ > n_d this condition implies that  χ + log(n_) > 0 is automatically verified.

For the ^ν-valued random variables , _, and _(), the normalized estimated mutual information are then defined by


    Î_() = Î()/χ + log(n_d)   ,   Î_(_) = Î(_)/χ + log(n_)   ,   Î_(_  ; ) = Î(_  ; )/χ + log(n_)  .


in which  Î(), Î(_), and Î(_  ; ) are the estimates (computed with Eq. (<ref>)) of the mutual information I(), I(_), and I(_  ; ) defined by Eqs. (<ref>) to (<ref>).

Based on paragraph (iii) above, constant χ is defined as follows. Let χ_ be the solution  in χ of the equation,


    Î()/χ + log(n_d) = Î(_  ; n_ Δ t)/χ + log(n_)  .


If χ_ + log(n_d) > 0, then χ_ is the desired value of χ.
It should be noted that the optimal value n_ is calculated with the estimation of the non-normalized mutual information (see Eq. (<ref>). Normalization is only introduced for the purpose of comparing the value of this criterion for  and _.




(v) Identification of the instant maximizing the selection criterion of the ROTB().

Let n_ be defined by Eq. (<ref>), and let χ_ be identified by solving Eq. (<ref>). Then, we have


    Î_() = Î_(_  ; n_Δ t) ≤Î_(_  ; )
       ,   ∀ n∈_N  .


We can then conclude that the ROTB(n_Δ t), represented by matrix [g(n_Δ t)], is better than the RODB, represented by matrix [g_]. The proof is straightforward.




§ NUMERICAL APPLICATIONS






 §.§ Preamble




We will present three applications, each with its own specificities. However, as the generation of the training dataset related to the random vector  is relatively complex to describe, we will start with the training dataset related to the normalized random vector  with values in ^ν. The number of realizations in the training dataset is n_d, and the training dataset is represented by the matrix [η_d] ∈_ν,n_d. This means that the PCA step of PLoM, which transforms  into  (see <ref>), is not detailed here. Readers interested in the training dataset for  can request, from the "corresponding author" of the article, the transfer of the matrix [η_d] for each application presented.

We will still briefly give the main specificities of these 3 applications.


Application 1. This application (Appli 1) was created so that the probability measure of  in ^9 (ν=9), which is defined by the points of the training dataset, is concentrated in a multiconnected domain of ^9, with the constituent connected parts being manifolds of dimensions much lower than 9, each having different dimensions. These parts may or may not be connected to each other.

Application 2. For the second application (Appli 2), the n_d realizations of the random vector  with values in ^8 are generated using a polynomial chaos expansion of degree 6 of a real random variable, whose random germ is of dimension 2, with each of the two random germs being a uniform random variable of different support. There are therefore 28 terms in this expansion, and the 8 components of  are defined as the random terms of rank 2, 3, 6, 8, 12, 13, 17, and 19. We thus define a relatively complex random manifold in ^8.

Application 3. The third application (Appli 3) results from a statistical treatment of an experimental database containing photon measurements in the ATLAS detector at CERN. This dataset was obtained by loading the file 'pid22_E262144_eta_20_25_voxalisation.csv' from the free access CERN Open Data Portal. The PCA step of PLoM has been performed, and an extraction of 45 components has been done to obtain the training dataset for the ^45-valued random variable . This application is in higher dimension than the first two, but the statistical complexity is less.



 §.§ Additional convergence analysis conducted for the three applications



For each of the three applications, we will provide detailed analyses of the calculation of the optimal value of the instant n_Δ t, which allows for the selection of the best reduced-order transient basis (ROTB). We will also present the convergence results of the PLoM algorithms under the normalization constraints (see <ref>). However, we cannot show all the convergence results for the other parameters. Below are the different analyses that were carried out with respect to the parameters that control the construction of the reduced-order transient basis:


(i) Convergence with respect to the value of κ. This point is important because Δ t must be sufficiently small to apply Proposition <ref>. For the three applications, we found that κ = 30 is an appropriate value. 


(ii) Convergence with respect to the size of δ t, that is to say, the value of n_s. For the three applications, we found that once Δ t is fixed by the value of κ, the criterion associated with Eq. (<ref>) was satisfied for n_s=1, as well as the evolution of the angle γ(n_Δ t) at the optimal instant was very insensitive to the values of n_s greater than
1. This is due to the fact that Δ t is already small enough to achieve good accuracy with the Euler scheme used to integrate the ISDE. We found that n_s=1, thus δ t = Δ t is a good value.


(iii) Convergence with respect to n_. This analysis was performed by examining the convergence of the angle γ() for n ∈. We observed that a large value of n_ was necessary to achieve convergence. This point is particularly important and must be carefully checked for the applications.




 §.§ Parameters defining the training dataset and controlling  the construction of the RODB



For each application, Table <ref> provides the values of the parameters that define the training dataset and the probability measure P_(d) = p_()  d on ^ν, as well as the parameters defined in Section <ref>-(i), which control the construction of the reduced-order DMAPS basis (RODB).







 §.§ Parameters controlling the construction of the ROTB()



For each application, Table <ref> provides the values of the parameters that control the construction of the reduced-order transient basis ROTB(). In particular, the optimal time n_Δ t is estimated using the indicators defined in Section <ref>. The evolution of these indicators as a function of n will be presented in detail in Sections <ref> to <ref> for the three applications.






 §.§ Parameters of PLoM with RODB and ROTB



For each of the three applications, Table <ref> provides the values of the parameters defined in <ref> and <ref> used by the PLoM algorithm with the reduced-order DMAPS basis (RODM) or the reduced-order transient basis ROTB(). Taking into account the convergence analysis carried out on n_ and the expertise on the statistical convergence of the quantities considered, we choose n_ = n_, a sufficiently large value (see <ref>).
For the PLoM algorithm, the constraints related to normalization, defined by Eq. (<ref>), are always applied in the computation (see <ref>)













 §.§ Results for Application 1




(i) Figure <ref> displays the graphs of the probability density function (pdf)  of components 1, 2, 4, and 5 for  estimated with the n_d realizations of the training dataset, and the pdf estimated with n_ learned realizations, for _ using MCMC without PLoM (a,d,g,j), for _ using  PLoM with RODB (b, e, h, k), and for _ using PLoM with ROTB(n_Δ t) (c, f, i, l).

(ii)  Figure <ref> shows the joint probability density function of components 4 and 5 of  estimated with the n_d realizations of the training dataset (a) and estimated with n_ learned realizations for _ using MCMC without PLoM (b), for _ using PLoM with RODB (c), and for _ using PLoM with ROTB(n_Δ t) (d).

(iii)  In Fig. <ref>, the clouds of n_ points corresponding to n_ learned realizations can be seen for components 1, 2, 3 (a,b,c) and components 3, 4, 5 (d,e,f). These are shown for _ using MCMC without PLoM (a,d), for _ using PLoM with RODB (b,e), and for _ using PLoM with ROTB(n_Δ t) (c,f).

(iv) Figure <ref> plots the functions that characterize the reduced-order transient basis ROTB() as a function of time :




  * The eigenvalues of matrix [K_] and those of the of symmetrized matrix [K̃()] are shown in
Fig. <ref>.


  * The probability-measure concentration using the d^ 2/ν-criterion is shown in Fig. <ref>.
For the learning without PLoM, the d^ 2-concentration is 0.6465, which shows that the concentration is lost, and for the PLoM with the RODM, the concentration is 0.0116, which shows that the concentration is preserved.


  * The other criterion of the probability-measure concentration is given by Kullback measure, shown in Fig. <ref>.
For the learning without PLoM, Kullbach is 0.3486, and for the PLoM with the RODM, Kullback is 3.9435. Comparing Figs. <ref> and <ref> shows that the two criteria are consistent and give the same analysis of the concentration.


  * The angle between the subspaces spanned by RODB and ROTB() is displayed in Fig. <ref>. It can be seen that, for the optimal time 9 Δ t, the angle is 53.03^∘, which is a significant angle showing that the two bases are different while the d^ 2-concentration remains small at 0.0174.


  * The entropy estimation of pdf p_(·  ; ) is given in Fig. <ref>.


  * The normalized mutual information (MI) of the pdfs p_ and p_(·  ; ) is shown in Fig. <ref>.
This figure shows that the optimal value of n is n_ = 9.

For the non-normalized estimation of the mutual information, we have Î() = 4.4668, Î(_  ; n_Δ t) = 6.9996, and Î(_) = 7.0945. For the normalized one, we have
Î_() = Î_(_  ; n_Δ t) = 0.3666 and Î_(_) = 0.3716.



(v) Finally, examination of these figures shows that traditional learning without PLoM gives poor results compared to PLoM, which allows the concentration to be preserved and properly learns the geometry of the probability measure support. We also see that PLoM with the optimal ROTB provides an improvement in learning compared to PLoM with RODM and, therefore, should improve the estimates of conditional statistics thanks to better learning of the joint probability measure.

























 §.§ Results for Application 2




(i) Figure <ref> displays the graphs of the probability density function (pdf)  of components 3, 5, 6, and 8 for  estimated with the n_d realizations of the training dataset, and the pdf estimated with n_ learned realizations, for _ using MCMC without PLoM (a,d,g,j), for _ using  PLoM with RODB (b, e, h, k), and for _ using PLoM with ROTB(n_Δ t) (c, f, i, l).

(ii)  Figure <ref> shows the joint probability density function of components 6 and 8 of  estimated with the n_d realizations of the training dataset (a) and estimated with n_ learned realizations for _ using MCMC without PLoM (b), for _ using PLoM with RODB (c), and for _ using PLoM with ROTB(n_Δ t) (d).

(iii)  In Fig. <ref>, the clouds of n_ points corresponding to n_ learned realizations can be seen for components 2, 5, 6 (a,b,c) and components 3, 6, 8 (d,e,f). These are shown for _ using MCMC without PLoM (a,d), for _ using PLoM with RODB (b,e), and for _ using PLoM with ROTB(n_Δ t) (c,f).

(iv) Figure <ref> plots the functions that characterize the reduced-order transient basis ROTB() as a function of time :




  * The eigenvalues of matrix [K_] and those of the of symmetrized matrix [K̃()] are shown in
Fig. <ref>.


  * The probability-measure concentration using the d^ 2/ν-criterion is shown in Fig. <ref>.
For the learning without PLoM, the d^ 2-concentration is 0.951, which shows that the concentration is lost, and for the PLoM with the RODM, the concentration is 0.0091, which shows that the concentration is preserved.


  * The other criterion of the probability-measure concentration is given by Kullback measure, shown in Fig. <ref>.
For the learning without PLoM, Kullbach is 0.4594, and for the PLoM with the RODM, Kullback is 3.4473. Comparing Figs. <ref> and <ref> shows, similarly to Application 1 that the two criteria are consistent and give the same analysis of the concentration.


  * The angle between the subspaces spanned by RODB and ROTB() is displayed in Fig. <ref>. It can be seen that, for the optimal time 5 Δ t, the angle is 17.1^∘, which is significant, although less than the optimal angle of Application 1. This shows that the two bases are different while the d^ 2-concentration remains small at 0.0153.


  * The entropy of pdf p_(·  ; ) is given in Fig. <ref>.


  * The normalized mutual information (MI) of the pdfs p_ and p_(·  ; ) is shown in Fig. <ref>. This figure shows that the optimal value of n is n_ = 5. Unlike Application 1, the normalized mutual information presents a local minimum, which is also a global minimum over the admissible set _N.
    For the non-normalized estimation of the mutual information, we have Î() = 3.4994, Î(_  ; n_Δ t) = 5.8763, and Î(_) = 5.9382. For the normalized one, we have
    Î_() = Î_(_  ; n_Δ t) = 0.3441, and Î_(_) = 0.3477.



(v) As for Application 1, examination of these figures shows that traditional learning without PLoM gives poor results compared to PLoM, which allows the concentration to be preserved and properly learns the geometry of the probability measure support. We also see that PLoM with the optimal ROTB provides an improvement in learning compared to PLoM with RODM. However, this improvement is less than in the case of Application 1 for which the data are much more heterogeneous (in correlation with the geometric complexity of the probability-measure support). Nevertheless, PLoM with the optimal ROTB is an improvement over PLoM with RODB and consequently, should improve the estimates of conditional statistics thanks to better learning of the joint probability measure.


























 §.§ Results for Application 3




(i) Figure <ref> displays the graphs of the probability density function (pdf)  of components 1, 6, 25, and 40 for  estimated with the n_d realizations of the training dataset, and the pdf estimated with n_ learned realizations, for _ using MCMC without PLoM (a,d,g,j), for _ using  PLoM with RODB (b, e, h, k), and for _ using PLoM with ROTB(n_Δ t) (c, f, i, l).

(ii)  Figure <ref> shows the joint probability density function of components 25 and 40 of  estimated with the n_d realizations of the training dataset (a) and estimated with n_ learned realizations for _ using MCMC without PLoM (b), for _ using PLoM with RODB (c), and for _ using PLoM with ROTB(n_Δ t) (d).

(iii)  In Fig. <ref>, the clouds of n_ points corresponding to n_ learned realizations can be seen for components 1, 6, 12 (a,b,c) and components 12, 25, 40 (d,e,f). These are shown for _ using MCMC without PLoM (a,d), for _ using PLoM with RODB (b,e), and for _ using PLoM with ROTB(n_Δ t) (c,f).

(iv) Figure <ref> plots the functions that characterize the reduced-order transient basis ROTB() as a function of time :




  * The eigenvalues of matrix [K_] and those of the of symmetrized matrix [K̃()] are shown in
Fig. <ref>.


  * The probability-measure concentration using the d^ 2/ν-criterion is shown in Fig. <ref>.
For the learning without PLoM, the d^ 2-concentration is 0.574, which shows that the concentration is lost, and for the PLoM with the RODM, the concentration is 0.067, which shows that the concentration is preserved.


  * The other criterion of the probability-measure concentration is given by Kullback measure, shown in Fig. <ref>.
For the learning without PLoM, Kullbach is 14.35, and for the PLoM with the RODM, Kullback is 5.72. Comparing Figs. <ref> and <ref> shows, similarly to Applications 1 and  2 that the two criteria are consistent and give the same analysis of the concentration.


  * The angle between the subspaces spanned by RODB and ROTB() is displayed in Fig. <ref>. It can be seen that, for the optimal time 9 Δ t, the angle is 9.7^∘, which is significant, although less than the optimal angle of Applications 1 and  2. This shows that the two bases are different while the d^  2-concentration remains small at 0.078.


  * The entropy of pdf p_(·  ; ) is given in Fig. <ref>.


  * The normalized mutual information (MI) of the pdfs p_ and p_(·  ; ) is shown in Fig. <ref>. This figure shows that the optimal value of n is n_ = 9. The behavior of the normalized mutual information is similar to that of Application 1 and does not present a local minimum as in Application 2.
    For the non-normalized estimation of the mutual information, we have Î() = 24.167, Î(_  ; n_Δ t) = 25.115, and Î(_) = 25.673. For the normalized one, we have
    Î_() = Î_(_  ; n_Δ t) = 0.1418 and Î_(_) = 0.1450.



(v) As for Applications 1 and 2, examination of these figures shows that traditional learning without PLoM gives poor results compared to PLoM, which allows the concentration to be preserved and properly learns the geometry of the probability measure support. We also see that PLoM with the optimal ROTB provides an improvement in learning compared to PLoM with RODM. However, this improvement is less than in the case of Applications 1 and 2. For this application, relative  to a relatively high dimension of  ν = 45, the data are more homogeneous than for the other applications (in correlation with the geometric complexity of the probability-measure support). Nevertheless, PLoM with the optimal
ROTB is an improvement over PLoM with RODB and, consequently, should improve the estimates of conditional statistics thanks to better learning of the joint probability measure.





















§ CONCLUSION




In this paper, we have presented the theoretical elements of constructing a time-dependent anisotropic kernel, which allows us to create a data projection basis for PLoM. This basis serves as an alternative to the DMAPS basis built using a time-independent isotropic kernel used by PLoM. We have demonstrated that an optimal time can be determined to obtain an optimal transient basis, best respecting the statistical dependence between the components for the learned joint probability measure.

The proposed theory has been  developed to improve PLoM in cases of highly heterogeneous data. The improvement of the learned joint probability measure is quantified by estimating an objective criterion from information theory, namely the mutual information, which we have normalized relative to the number of realizations using entropy.

This theory is consistent in the sense that, for a time close to the initial time, the DMAPS basis constructed with the time-independent isotropic kernel coincides with the transient basis constructed with the time-dependent anisotropic kernel. Thus, we can characterize the difference between the two bases by the angle of the vector subspaces they generate.

The theory is illustrated through three applications with decreasing levels of data heterogeneity. The three applications confirm that PLoM with the DMAPS basis (time-independent isotropic kernel) always results in learning that preserves the concentration of the measure, unlike the classic MCMC approach. The applications show that it is possible to improve the learned joint probability measure with the transient anisotropic kernel, which a priori allows for better estimates of conditional statistics.



§ ACKNOWLEDGMENTS


The authors acknowledge partial funding from DOE SciDAC FASTMath  Institute, and an ONR MURI on Modeling Turbulence and Chemistry in High Speed Reactive Flows.








§ OVERVIEW OF THE PROBABILISTIC LEARNING ON MANIFOLDS (PLOM) ALGORITHM AND ITS PARAMETERIZATION



The PLoM approach <cit.>, which has specifically been developed for small
data (as opposed to big data) starts from a training dataset _d made up of a relatively small number
n_d of points. It is assumed that _d is generated with an underlying stochastic manifold related
to a ^n_x-valued random variable = (,), defined on a probability space (Θ,,),
in which  is the quantity of interest that is a ^n_q-random variable, where  is the
control parameter that is a ^n_w-random variable, and where n_x = n_q + n_w. Another
^n_u- valued random variable  defined on (Θ,,) is also be considered,
which is an uncontrolled parameter and/or a noise.  Random variable  is assumed to be
written as = (,) in which the measurable mapping  is not explicitly known.
The joint probability distribution P_,(d,d) of   and  is assumed to be given.
The non-Gaussian probability measure P_() = P_,(d,d) of =(,)
is concentrated in a region of ^n_x for which the only available information is the cloud of
the points of training dataset _d. The PLoM method makes it possible to generate the learned dataset
_ for  whose n_≫ n_d points (learned realizations) are generated by the
non-Gaussian probability measure that is estimated using the training dataset. The concentration of
the probability measure is preserved thanks to the use of a diffusion-maps basis that allows to
enrich the available information from the training dataset.

The training dataset _d is made up of the n_d independent realizations
_d^j= (_d^j,_d^j) in ^n_x = ^n_q×^n_w for j∈{1,… , n_d}
of random variable =(,). The PLoM method allows for generating the learned dataset
_ made up of n_≫ n_d  learned realizations {_^ℓ,ℓ = 1, … ,n_}
of random vector . As soon as the learned dataset has been constructed, the learned realizations
for  and  can be extracted as (_^ℓ,_^ℓ) = _^ℓ for ℓ = 1, … ,n_.

Using the learned dataset _, PLoM allows for carrying out any conditional statistics such as ↦ E{()|=} from _w in ^n_ξ, in which  is a given measurable mapping from ^n_q into ^n_ξ, that is to say to construct statistical surrogate models (metamodels) in a probabilistic framework.



 §.§ Reduced representation



The n_d independent realizations {_d^j , j=1,… , n_d} are represented by the matrix [x_d]= [_d^1 …_d^n_d] in _n_x,n_d. Let []= [^1,… ,^n_d] be the random
matrix with values in _n_x,n_d, whose columns are n_d independent copies of random vector . Using the PCA of , random matrix [] is written as,


    [] = [x] + [φ]  [ζ]^1/2  []  ,


in which [] = [^1,… , ^n_d] is a _ν,n_d-valued random matrix, where ν≤ n_x, and where [ζ] is the (ν×ν) diagonal matrix of the ν positive eigenvalues of the empirical estimate of the covariance matrix of . The (n_x×ν) matrix [φ] is made up of the associated eigenvectors such [φ]^T [φ]= [I_ν]. The matrix [x] in _n_x,n_d has identical columns, each one being equal to the empirical estimate ∈^n_x of the mean value of random vector .
The columns of [] are n_d independent copies of a random vector  with values in ^ν.
The realization [η_d] = [^1…^n_d] ∈_ν,n_d of []  is computed by [η_d] =  [ζ]^-1/2 [φ]^T  ([x_d] - [x]).

The value ν is classically calculated in order that the L^2- error function ν↦_(ν) defined by
 

    _(ν) = 1 - ∑_α=1^νζ_α/E{‖‖^2}  ,


be smaller than ε_. If ν < n_x, then there is a statistical reduction.




 §.§ Construction of a reduced-order diffusion-maps basis (RODB) and reduced-order transient basis (ROTB())



 In this section, we begin with the construction of the RODB that is the basis initially used in the PLoM algorithm (see <cit.>).
Concerning the construction of the ROTB(), we refer the read to <ref>-(ii).



(i) Construction of RODB.

This construction corresponds to the one initailly proposed in the PLoM algorithm.
For preserving the concentration of the learned realizations in the region in which the points of the training dataset are concentrated, the PLoM relies on the diffusion-maps method <cit.>. This is an algebraic basis of vector space ^n_d, which  is constructed using the diffusion maps.
Let [_]  and  [B] be the matrices such that, for all i and j in {1,… , n_d},
[_]_ij = exp{-(4 ε_)^-1‖^i-^j‖^2} and
[B]_ij = δ_ij exp{-(4 ε_)^-1‖^i-^j‖^2}, in which ε_ >0 is a smoothing parameter. The eigenvalues b_,1,…,b_,n_d and the associated eigenvectors _^1,…,_^n_d of the right-eigenvalue problem [B]^-1[_] _^β = b_,β _^β are such that 1=b_,1 > b_,2≥…≥ b_,n_d and are computed by solving the eigenvalue problem [B]^-1/2[_] [B]^-1/2 ^β = b_,β ^β with the normalization ⟨^β,^β'⟩ =δ_ββ', and _^β = [B]^-1/2^β. The eigenvector _^1 associated with b_,1=1 is a constant vector.

The diffusion-maps basis {_^1,…,_^α,…, _^n_d} is a vector basis of ^n_d.
For a given integer m < n_d, the reduced-order diffusion-maps basis of order m is defined as the family
{_^1,…,_^m}.

This basis depends on two parameters, ε_ and m, which have to be identified.

As explained in <cit.>, the optimal value m_ of m_ is chosen as m_ = ν +1, and the optimal value ε_ of ε_ is such that


    1 = b_,1 > b_,2≃…≃ b_,m_≫ b_,m_ + 1≥…≥ b_,n_d > 0  ,


with the jump amplitude J_ = b_,m_ + 1  /b_,m_, which is J_=0.1 (following <cit.>), but which can also be chosen in the interval [0.1  , 0.5].
Consequently, the RODB is defined for m=m_ and is represented by the matrix


    [g_] = [_^1 …_^m_]∈_n_d,m_  .




(ii) Construction of ROTB().
Because PLoM will also use the reduced-order transient basis to quantify its efficiency relative to the reduced-order DMAPS basis, we introduce this basis  in this Appendix.
For n fixed in {1,… , N}, the reduced-order transient basis, ROTB(), is represented by the matrix


    [g()] = [^1() …^m_()]∈_n_d,m_  ,


in which m_ is the optimal value identified in <ref>-(i), and where [g()] is constructed in Section <ref>-(ii) (see Eq. (<ref>).



(iii) Reduced-order basis for PLoM.
In this Appendix, the PLoM reduced-order basis will be represented by the matrix [g_]∈_n_d,m_. Depending on the context of its use, this matrix will either be [g_], representing the reduced-order DMAPS basis (RODB) as used in the initial construction of the PLoM, or [g()] for a fixed n, representing the reduced-order transient basis (ROTB()) as proposed in this paper. The latter is introduced with the goal of comparing the efficiency of the two reduced-order vector bases.



 §.§ Reduced-order representation of the random matrices



The reduced-order basis represented by matrix [g_]∈_n_d,m_  spans a subspace of ^n_d that characterizes, for the optimal values m_ and ε_, the local geometry structure of  dataset {^j, j=1,… , n_d}. So the PLoM method introduces the _ν,n_d-valued random matrix [_] = []  [g_]^T with m_ < n_d, corresponding to a data-reduction representation of random matrix [], in which  [] is a _ν,m_-valued random matrix. The MCMC generator of random matrix [] belongs to the class of Hamiltonian Monte Carlo methods, is explicitly described in <cit.>, and is mathematically detailed in Theorem 6.3 of <cit.>. This  generator allows for computing n_ realizations {[_^ℓ],ℓ=1,… , n_} of  [] and therefore, for deducing the n_ realizations {[_^ℓ],ℓ=1,… ,n_} of [_]. The reshaping of matrix [_^ℓ] ∈_ν,n_d allows for obtaining n_ = n_× n_d learned realizations
{_^ℓ',ℓ' =1,… ,n_} of _. These learned realizations allow for estimating converged statistics on _ and then on _ = x + [φ]  [ζ]^1/2 _, such as pdf, moments, or conditional expectation of the type E{()  |  = } for  given in ^n_w and for any given vector-valued function  defined on ^n_q.




 §.§ Criterion for quantifying the concentration of the probability measure of random matrix [_]



The concentration of the probability measure of random matrix [_] is defined (see <cit.>) by


    d^ 2(m_) = E{‖ [_] - [η_d]‖^2} /‖ [η_d]‖^2  .


Let ={m_,m_+1,… , n_d} in which m_ is the optimal value of m.
Theorem 7.8 of <cit.> shows that
min_m∈ d^ 2(m)  ≤ 1 + m_/(n_d-1) < d^2(n_d), which means that the PLoM method, for m=m_ and [g_] is a better method than the usual one corresponding to d^2(n_d)= 1+n_d/(n_d-1)≃ 2.
Using the n_ realizations {[_^ℓ],ℓ=1,… ,n_} of [_], we have the estimate,


    d̂^ 2(m_) = (1/n_)∑_ℓ=1^n_{‖ [_^ℓ]  - [η_d]‖^2} /‖ [η_d]‖^2   .





 §.§ Generation of learned realizations {^ℓ'_, ℓ'=1,…, n_} of random vector _



Let  { ([ (t)], [(t)]), t∈^+ } be  the unique asymptotic (for t→ +∞) stationary diffusion stochastic process with values in _ν,m_×_ν,m_, of the following reduced-order ISDE (stochastic nonlinear second-order dissipative Hamiltonian dynamic system), for t >0,


    d[(t)]      =  [(t)]   dt   , 
    
      d[(t)]     =  [([(t)])]  dt -1/2 f_0 [(t)]  dt + √(f_0)  [d^(t)]   ,


with [(0)] = [η_d]  [a] and [(0)] = [ ]   [a], in which


    [a]  = [g_]  ([g_]^T  [g_])^-1∈_n_d,m_  .


(1) [([(t)])]= [L ( [(t)]   [g_]^T ) ]   [a] is a random matrix  with values in _ν,m_.
For all [u] = [^1 …^n_d] in _ν,n_d with ^j=(u^j_1,… ,u^j_ν) in ^ν, the
matrix [L([u])] in _ν,n_d is defined, for all k = 1,… ,ν and for all j=1,… , n_d, by


    [L([u])]_kj    = 1/p(^j) {∇_^j  p(^j) }_k   , 
                                                p(^j)         = 1/n_d∑_j'=1^n_dexp{ -1/2 ŝ^  2‖ŝ/s^j'-^j‖^2 }  , 
    ∇_^j  p(^j)     = 1/ŝ^ 2  n_d∑_j'=1^n_d (ŝ/s^j' -^j)
                   exp{ -1/2 ŝ^ 2‖ŝ/s^j' -^j‖^2 }  ,


in which ŝ  is the modified Silverman bandwidth  s, which has been introduced in <cit.>,


    ŝ =   s/√(s^2 +n_d -1/n_d)   ,   
    s = {4/n_d(2+ν)}^1/(ν +4)  .



(2) [^(t)] = [^(t)]   [a] where {[^(t)], t∈^+} is the _ν,n_d-valued normalized Wiener process.

(3)  [ ] is the _ν,n_d-valued normalized Gaussian random matrix that is independent of process [^].

(4)  The free parameter f_0, such that 0 < f_0 < 4/ŝ, allows the dissipation term of the nonlinear second-order dynamic system (dissipative Hamiltonian system)  to be controlled in order to kill the transient part induced by the initial conditions. A common value is f_0=4 (note that ŝ< 1).

(5)  We then have [] = lim_t→ +∞[(t)] in probability distribution. The Störmer-Verlet scheme is used <cit.> for solving the reduced-order ISDE, which allows for generating the learned realizations, [z_^1], … , [z_^n_], and then generating the learned realizations [η_^1],… , [η_^n_] such that [η_^ℓ] = [z_^ℓ]  [g_]^T. The implementation of the Störmer-Verlet scheme is detailed, for instance, in the Appendix of <cit.> for parallel computation, introducing the following parameters: the integration time step Δ t_, the initial time t_i =0, and the final integration time t_f = M_0×Δ t_, at which the stationary solution is reached.

(6) The learned realizations {_^ℓ',ℓ' =1,…,n_} of random vector  are then calculated (see Eq. (<ref>)) by _^ℓ' = + [φ]  [μ]^1/2 _^ℓ'.




 §.§ Constraints on the second-order moments of the components of _



In general, the mean value of _  estimated using the n_ learned realizations
{_^ℓ',ℓ'=1,…,n_}, is sufficiently close to zero.
Likewise, the estimate of the covariance matrix of _, which must be the identity matrix, is
sufficiently close to a diagonal matrix. However, sometimes  the
diagonal entries of the estimated covariance matrix can be lower than 1.
Normalization can be recovered by imposing constraints


    E{( H_,k )^2 } = 1    ,    k=1,…, ν  ,


in the algorithm presented in <ref>.
For that, we use the method and the iterative algorithm presented in <cit.> (that is based on Sections 5.5 and 5.6 of <cit.>). The constraints are imposed by using the Kullback-Leibler minimum cross-entropy principle. The resulting optimization problem is formulated using a Lagrange multiplier = (λ_1,…,λ_ν) associated with the constraints. The optimal solution of the Lagrange multiplier is computed using an efficient iterative algorithm. At each iteration, the MCMC generator detailed in <ref> is used.

The  constraints are rewritten as


    E{(_)} =   ,


in which the function = (h_1,… ,h_ν) and the vector = (b_1,… , b_ν) are such that
h_k(_) = (H_,k)^2 and b_k =1 for k in {1,… , ν}.
To take into account the constraints in the algorithm presented in <ref>, Eq. (<ref>) is replaced by the following one,


    [L_([u])]_kj  = 1/p(^j) {∇_^j  p(^j) }_k  -2 λ_k u_k^j  .


It should be noted that Eqs. (<ref>) to (<ref>) can be straightforwardly extended to the case in which the constraint defined by
Eq. (<ref>) is replaced by the full second-order moment constraints  E{_} = _ν and E{_⊗_} = [I_ν], that is to say,


    E{ H_,k}  = 0   ,   E{( H_,k )^2}  = 1    ,   k=1,…, ν   ,   
     E{ H_,k H_,k'}  = 0     ,    1 ≤ k < k' ≤ν  .


The iteration algorithm for computing ^i+1 as a function of ^i is the following,


    ^i+1    = ^i -α_i [Γ”(^i)]^-1 '(^i)    ,    i ≥ 0   ,
    ^0             = _ν  ,


in which '(^i) =  - E{(_^i)} and
[Γ”(^i)] = [{(_^i)}] (the covariance matrix), and where α_i is a relaxation function (less than 1) that is introduced for controlling the convergence as a function of iteration number i.
For given i_2 ≥ 2, for given β_1 and β_2 such that 0 < β_1 < β_2 ≤ 1, α_i can be defined by:

- for  i ≤ i_2,  α_i =  β_1 +(β_2-β_1)(i-1)/(i_2-1);

- for i > i_2, α_i = β_2.

The convergence of the iteration algorithm is controlled by the error function i↦(i)  defined by


    (i) =  ‖ - E{(_^i)}‖ / ‖‖  .


At each iteration i, E{(_^i)} and [{(_^i)}] are estimated by using the n_ learned realizations of _^i obtained by reshaping the learned realizations. If i_last is the last iteration corresponding to convergence, we have _ =  _^i with i = i_last.








§ ESTIMATION OF THE KULLBACK-LEIBLER DIVERGENCE, THE MUTUAL INFORMATION, AND THE ENTROPY FROM A SET OF REALIZATIONS



The definition Kullback-Leibler divergence, the mutual information, and the entropy can be found in
<cit.>. The estimation of these quantities from a set of independent realizations is carried out using the Gaussian kernel density estimation (GKDE) method <cit.>.

For ν > 1, let =(X_1,… ,X_ν) and =(Y_1,… ,Y_ν) be ^ν-valued random variables defined on the probability space (Θ,,), whose probability measures are P_(d) = p_()  d and P_(d) = p_()  d,
in which  the probability density functions p_ and p_ are assumed to be strictly positive. Let {^ℓ,ℓ=1,… , N_x} be N_x independent realizations of  and let {^j,j=1,… , N_y} be N_y independent realizations of .
For k=1,… , ν, let σ_X_k and σ_Y_k be the standard deviation of X_k and Y_k that are estimated (empirical estimator) with the independent realizations.
Finally, we introduce the Silverman bandwidth for the Gaussian KDE estimation of p_ and p_,


    s_x = {4/N_x(2+ν)}^1/(ν +4)   ,    s_y = {4/N_y(2+ν)}^1/(ν +4)  .





 §.§ Estimation of the Kullback-Leibler divergence from a set of realizations



The Kullback-Leibler divergence (or the relative entropy) between p_ and p_ is defined by


    D(p_‖ p_) = ∫_^ν p_() log ( p_()/p_())   d =
    E{log( p_()/p_() ) }  .


The GKDE, D̂(p_‖ p_), of D(p_‖ p_) yields the formula,


    D̂(p_ ‖   p_) =    νlog (s_y/s_x ) + log (N_y/N_x )
                + log (σ_Y_1×…×σ_Y_ν/σ_X_1×…×σ_X_ν ) 
        + 1/N_x∑_ℓ'=1^N_xlog{∑_ℓ=1^N_xexp ( -1/2s_x^2∑_k=1^ν ( x_k^ℓ' - x_k^ℓ/σ_X_k )^2 )/∑_j=1^N_yexp ( -1/2s_y^2∑_k=1^ν ( x_k^ℓ' - y_k^j/σ_Y_k )^2 )}  .





 §.§ Estimation of the mutual information from a set of realizations



The mutual information I() of  allows to quantify the level of statistical dependencies of the components X_1,…, X_ν of =(X_1,… ,X_ν). Let p_X_k be the pdf of real-valued random variable X_k,


    p_X_k(x_k) = ∫_^ν-1 p_ ( x_1,… ,x_k-1,x_k,x_k+1,…,x_ν )  
    dx_1 … dx_k-1  dx_k+1… dx_ν  .


The mutual information I() is defined by


    I() = D(p_ ‖ ⊗_k=1^ν p_X_k) = E{log( p_()/p_X_1(X_1)×…× p_X_ν(X_ν) ) }  .


Eq. (<ref>) shows that, if the components X_1,… ,X_ν are statistically independent, then I()=0.
The GKDE, Î() of I() yields the formula,


    Î()  = 1/N_x∑_ℓ'=1^N_xlog{1/N_x∑_ℓ=1^N_xexp ( -1/2s_x^2∑_k=1^ν ( x_k^ℓ' - x_k^ℓ/σ_X_k )^2 )/∏_k=1^ν ( 1/N_x∑_ℓ”=1^N_xexp ( -1/2s_x^2 ( x_k^ℓ' - x_k^ℓ”/σ_X_k)^2  )  )}  .





 §.§ Estimation of the entropy from a set of realizations



The entropy related to p_ is defined by


    S_ = -∫_^ν p_() log p_()   d =
    - E{log p_()  }  .


The GKDE, Ŝ_, of S_ yields the formula,


    Ŝ_ =    νlog (s_x√(2π)  ) +
                 log (σ_X_1×…×σ_X_ν ) 
        - 1/N_x∑_ℓ'=1^N_xlog{1/N_x∑_ℓ=1^N_xexp ( -1/2s_x^2∑_k=1^ν ( x_k^ℓ' - x_k^ℓ/σ_X_k )^2 )
                                          }  .


Since J_S(N_x) = νlog (s_x√(2π)  ) is asymptotically for N_x→ +∞ in -log(N_x), the entropy decreases when N_x increases. 







§ CONFLICT OF INTEREST


The author declares that he has no conflict of interest.



10
url<#>1urlprefixURL href#1#2#2 #1#1

Soize2016
C. Soize, R. Ghanem, Data-driven probability concentration and sampling on
  manifold, Journal of Computational Physics 321 (2016) 242–258.
https://doi.org/10.1016/j.jcp.2016.05.044
  doi:10.1016/j.jcp.2016.05.044.

Coifman2005
R. Coifman, S. Lafon, A. Lee, M. Maggioni, B. Nadler, F. Warner, S. Zucker,
  Geometric diffusions as a tool for harmonic analysis and structure definition
  of data: Diffusion maps, PNAS 102 (21) (2005) 7426–7431.
https://doi.org/10.1073/pnas.0500334102
  doi:10.1073/pnas.0500334102.

Coifman2006
R. Coifman, S. Lafon, Diffusion maps, Applied and Computational Harmonic
  Analysis 21 (1) (2006) 5–30.
https://doi.org/10.1016/j.acha.2006.04.006
  doi:10.1016/j.acha.2006.04.006.

Korb2010
K. B. Korb, A. E. Nicholson, Bayesian artificial intelligence, CRC press,
  Boca Raton, 2010.

Murphy2012
K. P. Murphy, Machine Learning: A Probabilistic Perspective, MIT press, 2012.

Ghahramani2015
Z. Ghahramani, Probabilistic machine learning and artificial intelligence,
  Nature 521 (7553) (2015) 452–459.
https://doi.org/10.1038/nature14541
  doi:10.1038/nature14541.

Russell2016
S. Russel, P. Norvig, Artifical Intelligence, A Modern Approach, Third Edition,
  Pearson, Harlow, 2016.

Vapnik2000
V. Vapnik, The Nature of Statistical Learning Theory, Springer, New York, 2000.
https://doi.org/10.1007/978-1-4757-3264-1
  doi:10.1007/978-1-4757-3264-1.

Hastie2009
T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning,
  Second Edition, Springer, 2009.
https://doi.org/10.1007/b94608 doi:10.1007/b94608.

James2013
G. James, D. Witten, T. Hastie, R. Tibshirani, An Introduction to Statistical
  Learning, Vol. 112, Springer, 2013.

Taylor2015
J. Taylor, R. J. Tibshirani, Statistical learning and selective inference,
  Proceedings of the National Academy of Sciences 112 (25) (2015) 7629–7634.
https://doi.org/10.1073/pnas.1507583112
  doi:10.1073/pnas.1507583112.

Swischuk2019
R. Swischuk, L. Mainini, B. Peherstorfer, K. Willcox, Projection-based model
  reduction: Formulations for physics-based machine learning, Computers &
  Fluids 179 (2019) 704–717.
https://doi.org/10.1016/j.compfluid.2018.07.021
  doi:10.1016/j.compfluid.2018.07.021.

Oztireli2010
A. C. Öztireli, M. Alexa, M. Gross, Spectral sampling of manifolds, ACM
  Transactions on Graphics (TOG) 29 (6) (2010) 1–8.
https://doi.org/10.1145/1882261.1866190
  doi:10.1145/1882261.1866190.

Perrin2017
G. Perrin, C. Soize, S. Marque-Pucheu, J. Garnier, Nested polynomial trends for
  the improvement of Gaussian process-based predictors, Journal of
  Computational Physics 346 (2017) 389–402.
https://doi.org/10.1016/j.jcp.2017.05.051
  doi:10.1016/j.jcp.2017.05.051.

Kevrekidis2020
Y. Kevrekidis, Manifold learning for parameter reduction, Bulletin of the
  American Physical Society 65 (2020).
https://doi.org/10.1016/j.jcp.2019.04.015
  doi:10.1016/j.jcp.2019.04.015.

Kontolati2022
K. Kontolati, D. Loukrezis, K. R. dos Santos, D. G. Giovanis, M. D. Shields,
  Manifold learning-based polynomial chaos expansions for high-dimensional
  surrogate models, International Journal for Uncertainty Quantification 12 (4)
  (2022).
https://doi.org/10.1615/Int.J.UncertaintyQuantification.2022039936
  doi:10.1615/Int.J.UncertaintyQuantification.2022039936.

Pan2020
S. Pan, K. Duraisamy, Physics-informed probabilistic learning of linear
  embeddings of nonlinear dynamics with guaranteed stability, SIAM Journal on
  Applied Dynamical Systems 19 (1) (2020) 480–509.
https://doi.org/10.1137/19M1267246
  doi:10.1137/19M1267246.

Soize2020a
C. Soize, R. Ghanem, Physics-constrained non-Gaussian probabilistic learning
  on manifolds, International Journal for Numerical Methods in Engineering
  121 (1) (2020) 110–145.
https://doi.org/10.1002/nme.6202
  doi:10.1002/nme.6202.

Soize2021a
C. Soize, R. Ghanem, Probabilistic learning on manifolds constrained by
  nonlinear partial differential equations for small datasets, Computer Methods
  in Applied Mechanics and Engineering 380 (2021) 113777.
https://doi.org/10.1016/j.cma.2021.113777
  doi:10.1016/j.cma.2021.113777.

Talwalkar2008
A. Talwalkar, S. Kumar, H. Rowley, Large-scale manifold learning, in: 2008 IEEE
  Conference on Computer Vision and Pattern Recognition, IEEE, 2008, pp. 1–8.
https://doi.org/10.1109/CVPR.2008.4587670
  doi:10.1109/CVPR.2008.4587670.

Marzouk2016
Y. Marzouk, T. Moselhy, M. Parno, A. Spantini, Sampling via measure transport:
  An introduction, Handbook of uncertainty quantification (2016) 1–41https://doi.org/10.1007/978-3-319-11259-6_23-1
  doi:10.1007/978-3-319-11259-6_23-1.

Parno2018
M. D. Parno, Y. M. Marzouk, Transport map accelerated markov chain Monte
  Carlo, SIAM/ASA Journal on Uncertainty Quantification 6 (2) (2018)
  645–682.
https://doi.org/10.1137/17M1134640
  doi:10.1137/17M1134640.

Perrin2018a
G. Perrin, C. Soize, N. Ouhbi, Data-driven kernel representations for sampling
  with an unknown block dependence structure under correlation constraints,
  Computational Statistics & Data Analysis 119 (2018) 139–154.
https://doi.org/10.1016/j.csda.2017.10.005
  doi:10.1016/j.csda.2017.10.005.

Soize2017a
C. Soize, R. Ghanem, Polynomial chaos representation of databases on manifolds,
  Journal of Computational Physics 335 (2017) 201–221.
https://doi.org/10.1016/j.jcp.2017.01.031
  doi:10.1016/j.jcp.2017.01.031.

Soize2019b
C. Soize, R. Ghanem, C. Safta, X. Huan, Z. P. Vane, J. C. Oefelein, G. Lacaze,
  H. N. Najm, Q. Tang, X. Chen, Entropy-based closure for probabilistic
  learning on manifolds, Journal of Computational Physics 388 (2019) 528–533.
https://doi.org/10.1016/j.jcp.2018.12.029
  doi:10.1016/j.jcp.2018.12.029.

Soize2020b
C. Soize, R. Ghanem, C. Desceliers, Sampling of Bayesian posteriors with a
  non-Gaussian probabilistic learning on manifolds from a small dataset,
  Statistics and Computing 30 (5) (2020) 1433–1457.
https://doi.org/10.1007/s11222-020-09954-6
  doi:10.1007/s11222-020-09954-6.

Soize2020c
C. Soize, R. Ghanem, Probabilistic learning on manifolds, Foundations of Data
  Science 2 (3) (2020) 279–307.
https://doi.org/10.3934/fods.2020013
  doi:10.3934/fods.2020013.

Soize2022b
C. Soize, Probabilistic learning inference of boundary value problem with
  uncertainties based on Kullback-Leibler divergence under implicit
  constraints, Computer Methods in Applied Mechanics and Engineering 395 (2022)
  115078.
https://doi.org/10.1016/j.cma.2022.115078
  doi:10.1016/j.cma.2022.115078.

Soize2022a
C. Soize, R. Ghanem, Probabilistic learning on manifolds (PLoM) with
  partition, International Journal for Numerical Methods in Engineering 123 (1)
  (2022) 268–290.
https://doi.org/10.1002/nme.6856
  doi:10.1002/nme.6856.

Soize2022c
C. Soize, Probabilistic learning constrained by realizations using a weak
  formulation of fourier transform of probability measures, Computational
  Statistics 38 (4) (2023) 1879–1925.
https://doi.org/10.1007/s00180-022-01300-w
  doi:10.1007/s00180-022-01300-w.

Kennedy2001
M. C. Kennedy, A. O'Hagan, Bayesian calibration of computer models, Journal
  of the Royal Statistical Society: Series B (Statistical Methodology) 63 (3)
  (2001) 425–464.
https://doi.org/10.1111/1467-9868.00294
  doi:10.1111/1467-9868.00294.

Marzouk2007
Y. M. Marzouk, H. N. Najm, L. A. Rahn, Stochastic spectral methods for
  efficient Bayesian solution of inverse problems, Journal of Computational
  Physics 224 (2) (2007) 560–586.
https://doi.org/10.1016/j.jcp.2006.10.010
  doi:10.1016/j.jcp.2006.10.010.

Gentle2009
J. E. Gentle, Computational statistics, Springer, New York, 2009.
https://doi.org/10.1007/978-0-387-98144-4
  doi:10.1007/978-0-387-98144-4.

Stuart2010
A. M. Stuart, Inverse problems: a Bayesian perspective, Acta Numerica 19
  (2010) 451–559.
https://doi.org/10.1017/S0962492910000061
  doi:10.1017/S0962492910000061.

Owhadi2015
H. Owhadi, C. Scovel, T. Sullivan, On the brittleness of Bayesian inference,
  SIAM Review 57 (4) (2015) 566–582.
https://doi.org/10.1137/130938633
  doi:10.1137/130938633.

Matthies2016
H. G. Matthies, E. Zander, B. V. Rosić, A. Litvinenko, O. Pajonk, Inverse
  problems in a Bayesian setting, in: Computational Methods for Solids and
  Fluids, Vol. 41, Springer, 2016, pp. 245–286.
https://doi.org/10.1007/978-3-319-27996-1_10
  doi:10.1007/978-3-319-27996-1_10.

Dashti2017
M. Dashti, A. M. Stuart, The Bayesian approach to inverse problems, in:
  R. Ghanem, D. Higdon, O. Houman (Eds.), Handbook of Uncertainty
  Quantification, Springer, Cham, Switzerland, 2017, Ch. 10, pp. 311–428.
https://doi.org/10.1007/978-3-319-12385-1_7
  doi:10.1007/978-3-319-12385-1_7.

Ghanem2017
R. Ghanem, D. Higdon, H. Owhadi, Handbook of Uncertainty Quantification, Vol. 1
  to 3, Springer, Cham, Switzerland, 2017.
https://doi.org/10.1007/978-3-319-12385-1
  doi:10.1007/978-3-319-12385-1.

Spantini2017
A. Spantini, T. Cui, K. Willcox, L. Tenorio, Y. Marzouk, Goal-oriented optimal
  approximations of Bayesian linear inverse problems, SIAM Journal on
  Scientific Computing 39 (5) (2017) S167–S196.
https://doi.org/10.1137/16M1082123
  doi:10.1137/16M1082123.

Perrin2020
G. Perrin, C. Soize, Adaptive method for indirect identification of the
  statistical properties of random fields in a Bayesian framework,
  Computational Statistics 35 (1) (2020) 111–133.
https://doi.org/10.1007/s00180-019-00936-5
  doi:10.1007/s00180-019-00936-5.

Ghanem2018a
R. Ghanem, C. Soize, Probabilistic nonconvex constrained optimization with
  fixed number of function evaluations, International Journal for Numerical
  Methods in Engineering 113 (4) (2018) 719–741.
https://doi.org/10.1002/nme.5632
  doi:10.1002/nme.5632.

Soize2018a
C. Soize, Design optimization under uncertainties of a mesoscale implant in
  biological tissues using a probabilistic learning algorithm, Computational
  Mechanics 62 (3) (2018) 477–497.
https://doi.org/10.1007/s00466-017-1509-x
  doi:10.1007/s00466-017-1509-x.

Ghanem2018b
R. Ghanem, C. Soize, C. Thimmisetty, Optimal well-placement using probabilistic
  learning, Data-Enabled Discovery and Applications 2 (1) (2018) 1–16.
https://doi.org/10.1007/s41688-017-0014-x
  doi:10.1007/s41688-017-0014-x.

Farhat2019
C. Farhat, R. Tezaur, T. Chapman, P. Avery, C. Soize, Feasible probabilistic
  learning method for model-form uncertainty quantification in vibration
  analysis, AIAA Journal 57 (11) (2019) 4978–4991.
https://doi.org/10.2514/1.J057797
  doi:10.2514/1.J057797.

Ghanem2019
R. Ghanem, C. Soize, C. Safta, X. Huan, G. Lacaze, J. C. Oefelein, H. N. Najm,
  Design optimization of a scramjet under uncertainty using probabilistic
  learning on manifolds, Journal of Computational Physics 399 (2019) 108930.
https://doi.org/10.1016/j.jcp.2019.108930
  doi:10.1016/j.jcp.2019.108930.

Almeida2022
J. O. Almeida, F. A. Rochinha, A probabilistic learning approach applied to the
  optimization of wake steering in wind farms, Journal of Computing and
  Information Science in Engineering 23 (1) (2022) 011003.
https://doi.org/10.1115/1.4054501
  doi:10.1115/1.4054501.

Capiez2022
E. Capiez-Lernout, C. Soize, Nonlinear stochastic dynamics of detuned bladed
  disks with uncertain mistuning and detuning optimization using a
  probabilistic machine learning tool, International Journal of Non-Linear
  Mechanics 143 (2022) 104023.
https://doi.org/10.1016/j.ijnonlinmec.2022.104023
  doi:10.1016/j.ijnonlinmec.2022.104023.

Almeida2023
J. O. Almeida, F. A. Rochinha, A probabilistic learning approach applied to the
  optimization of wake steering in wind farms, Journal of Computing and
  Information Science in Engineering 23 (1) (2023) 011003.
https://doi.org/10.1115/1.4054501
  doi:10.1115/1.4054501.

Guilleminot2020
J. Guilleminot, J. E. Dolbow, Data-driven enhancement of fracture paths in
  random composites, Mechanics Research Communications 103 (2020) 103443.
https://doi.org/10.1016/j.mechrescom.2019.103443
  doi:10.1016/j.mechrescom.2019.103443.

Chen2024
P. Chen, J. Guilleminot, C. Soize, Concurrent multiscale simulations of
  nonlinear random materials using probabilistic learning, Computer Methods in
  Applied Mechanics and Engineering 422 (2024) 116837.
https://doi.org/10.1016/j.cma.2024.116837
  doi:10.1016/j.cma.2024.116837.

Soize2023c
C. Soize, An overview on uncertainty quantification and probabilistic learning
  on manifolds in multiscale mechanics of materials, Mathematics and Mechanics
  of Complex Systems 11 (1) (2023) 87–174.
https://doi.org/10.2140/memocs.2023.11.87
  doi:10.2140/memocs.2023.11.87.

Ghanem2022
R. Ghanem, C. Soize, L. Mehrez, V. Aitharaju, Probabilistic learning and
  updating of a digital twin for composite material systems, International
  Journal for Numerical Methods in Engineering 123 (13) (2022) 3004–3020.
https://doi.org/10.1002/nme.6430
  doi:10.1002/nme.6430.

Ezvan2023
O. Ezvan, C. Soize, C. Desceliers, R. Ghanem, Updating an uncertain and
  expensive computational model in structural dynamics based on one single
  target frf using a probabilistic learning tool, Computational Mechanics 71
  (2023) 1161–1177.
https://doi.org/10.1007/s00466-023-02301-2
  doi:10.1007/s00466-023-02301-2.

Capiez2024
E. Capiez-Lernout, C. Ezvan, Olivier Soize, Updating nonlinear stochastic
  dynamics of an uncertain nozzle model using probabilistic learning with
  partial observability and incomplete dataset, ASME Journal of Computing and
  Information Science in Engineering 24 (6) (2024) 061006,1–17.
https://doi.org/10.1115/1.4065312
  doi:10.1115/1.4065312.

Arnst2021
M. Arnst, C. Soize, K. Bulthies, Computation of sobol indices in global
  sensitivity analysis from small data sets by probabilistic learning on
  manifolds, International Journal for Uncertainty Quantification 11 (2) (2021)
  1–23.
https://doi.org/10.1615/Int.J.UncertaintyQuantification.2020032674
  doi:10.1615/Int.J.UncertaintyQuantification.2020032674.

Soize2021e
C. Soize, A. Orcesi, Machine learning for detecting structural changes from
  dynamic monitoring using the probabilistic learning on manifolds, Structure
  and Infrastructure Engineering Journal 17 (10) (2021) 1418–1430.
https://doi.org/10.1080/15732479.2020.1811991
  doi:10.1080/15732479.2020.1811991.

Zhong2023
K. Zhong, J. G. Navarro, S. Govindjee, G. G. Deierlein, Surrogate modeling of
  structural seismic response using Probabilistic Learning on Manifolds,
  Earthquake Engineering and Structural Dynamics 52 (8) (2023) 2407–2428.
https://doi.org/10.1002/eqe.3839
  doi:10.1002/eqe.3839.

Soize2023a
C. Soize, R. Ghanem, Probabilistic-learning-based stochastic surrogate model
  from small incomplete datasets, Computer Methods in Applied Mechanics and
  Engineering (2023).

Soize2024
C. Soize, R. Ghanem, Probabilistic-learning-based stochastic surrogate model
  from small incomplete datasets for nonlinear dynamical systems, Computer
  Methods in Applied Mechanics and Engineering 418 (2023) 116498.
https://doi.org/10.1016/j.cma.2023.116498
  doi:10.1016/j.cma.2023.116498.

Soize2023b
C. Soize, Q.-D. To, Polynomial-chaos-based conditional statistics for
  probabilistic learning with heterogeneous data applied to atomic collisions
  of helium on graphite substrate, Journal of Computational Physics (2023).

Sinha2023
A. Sinha, C. Soize, C. Desceliers, G. Cunha, Aeroacoustic liner impedance
  metamodel from simulation and experimental data using probabilistic learning,
  AIAA Journal 61 (11) (2023) 4926–4934.
https://doi.org/10.2514/1.J062991
  doi:10.2514/1.J062991.

Soize2015
C. Soize, Polynomial chaos expansion of a multimodal random vector, SIAM-ASA
  Journal on Uncertainty Quantification 3 (1) (2015) 34–60.
https://doi.org/10.1137/140968495
  doi:10.1137/140968495.

Bowman1997
A. Bowman, A. Azzalini, Applied Smoothing Techniques for Data Analysis: The
  Kernel Approach With S-Plus Illustrations, Vol. 18, Oxford University Press,
  Oxford: Clarendon Press, New York, 1997.
https://doi.org/10.1007/s001800000033
  doi:10.1007/s001800000033.

Givens2013
G. Givens, J. Hoeting, Computational Statistics, 2nd Edition, John Wiley and
  Sons, Hoboken, New Jersey, 2013.

Doob1953
J. L. Doob, Stochastic processes, John Wiley & Sons, New York, 1953.

Guikhman1980
I. I. Guikhman, A. Skorokhod, Introduction à la Théorie des Processus
  Aléatoires, Edition Mir, 1980.

Friedman2006
A. Friedman, Stochastic Differential Equations and Applications, Dover
  Publications, Inc., Mineola, New York, 2006.

Soize1994
C. Soize, The Fokker-Planck Equation for Stochastic Dynamical Systems and
  its Explicit Steady State Solutions, Vol. Series on Advances in Mathematics
  for Applied Sciences: Vol 17, World Scientific, Singapore, 1994.
https://doi.org/10.1142/2347 doi:10.1142/2347.

Gardiner1985
C. W. Gardiner, Handbook of Stochastic Methods, Second Edition, Springer
  Verlag, Berlin, Heidelberg, 1985.

Risken1989
H. Risken, The Fokker-Planck Equation, Second Edition, Springer Verlag,
  Berlin, Heidelberg, 1989.

Guelfand1967
I. M. Gelfand, N. I. Vilenkin, Les Distributions. Tome 4. Application de
  l'Analyse Harmonique, Dunod, 1967.
https://doi.org/10.1016/0375-9474(67)90547-7
  doi:10.1016/0375-9474(67)90547-7.

Spencer1993
B. Spencer, L. Bergman, On the numerical solution of the Fokker-Planck
  equation for nonlinear stochastic systems, Nonlinear Dynamics 4 (1993)
  357–372.
https://doi.org/10.1007/BF00120671
  doi:10.1007/BF00120671.

Masud2005
A. Masud, L. A. Bergman, Application of multi-scale finite element methods to
  the solution of the Fokker-Planck equation, Computer methods in applied
  mechanics and engineering 194 (12-16) (2005) 1513–1526.
https://doi.org/10.1016/j.cma.2004.06.041
  doi:10.1016/j.cma.2004.06.041.

Kumar2006
P. Kumar, S. Narayanan, Solution of Fokker-Planck equation by finite
  element and finite difference methods for nonlinear systems, Sadhana 31
  (2006) 445–461.
https://doi.org/10.1007/BF02716786
  doi:10.1007/BF02716786.

Pichler2013
L. Pichler, A. Masud, L. A. Bergman, Numerical solution of the
  Fokker-Planck equation by finite difference and finite element methods -
  a comparative study, Computational Methods in Stochastic Dynamics: Volume 2
  (2013) 69–85https://doi.org/10.1007/978-94-007-5134-7_5
  doi:10.1007/978-94-007-5134-7_5.

Deng2009
W. Deng, Finite element method for the space and time fractional
  Fokker-Planck equation, SIAM journal on numerical analysis 47 (1) (2009)
  204–226.
https://doi.org/10.1137/080714130
  doi:10.1137/080714130.

Anderson2024
W. Anderson, M. Farazmand, Fisher information and shape-morphing modes for
  solving the Fokker-Planck equation in higher dimensions, Applied
  Mathematics and Computation 467 (2024) 128489.
https://doi.org/10.1016/j.amc.2023.128489
  doi:10.1016/j.amc.2023.128489.

Popelier2011
P. L. Popelier, Solving the Schrödinger Equation: Has Everything Been
  Tried?, Imperial College Press, 2011.
https://doi.org/10.1142/p780 doi:10.1142/p780.

Feit1982
M. Feit, J. Fleck Jr, A. Steiger, Solution of the Schrödinger equation by
  a spectral method, Journal of Computational Physics 47 (3) (1982) 412–433.
https://doi.org/10.1016/0021-9991(82)90091-2
  doi:10.1016/0021-9991(82)90091-2.

Iitaka1994
T. Iitaka, Solving the time-dependent Schrödinger equation numerically,
  Physical Review E 49 (5) (1994) 4684.
https://doi.org/10.1103/PhysRevE.49.4684
  doi:10.1103/PhysRevE.49.4684.

Simos1997
T. Simos, P. Williams, A finite-difference method for the numerical solution of
  the Schrödinger equation, Journal of Computational and Applied
  Mathematics 79 (2) (1997) 189–205.
https://doi.org/10.1016/S0377-0427(96)00156-2
  doi:10.1016/S0377-0427(96)00156-2.

Serfling1980
R. J. Serfling, Approximation theorems of mathematical statistics, Vol. 162,
  John Wiley & Sons, 1980.

Soize2017b
C. Soize, Uncertainty Quantification. An Accelerated Course with Advanced
  Applications in Computational Engineering, Springer, New York, 2017.
https://doi.org/10.1007/978-3-319-54339-0
  doi:10.1007/978-3-319-54339-0.

Ikeda1981
N. Ikeda, S. Watanabe, Stochastic Differential Equations and Diffusion
  Processes, North-Holland, Amsterdam, 1981.

Khasminskii1980
R. Has'minski, Stochastic Stability of Differential Equations, Sijthoff &
  Noordhoff, Alphen aan den Rijn, The Netherlands, 1980, first English edition
  Khasminskii,2021, Spinger.

Stroock1979
D. W. Stroock, S. S. Varadhan, Multidimensional Diffusion Processes, Vol. 233,
  Springer-Verlag, Berlin,Heidelberg, 1997.

Kloeden1992
P. Kloeden, E. Platen, Numerical Solution of Stochastic Differentials
  Equations, Springer-Verlag, Heidelberg, 1992.

Duong2008
T. Duong, A. Cowling, I. Koch, M. Wand, Feature significance for multivariate
  kernel density estimation, Computational Statistics & Data Analysis 52 (9)
  (2008) 4225–4242.
https://doi.org/10.1016/j.csda.2008.02.035
  doi:10.1016/j.csda.2008.02.035.

Hansen1975
E. R. Hansen, A Table of Series and Products, Prentice-Hall, New York, 1975.

Bhattacharyya1943
A. Bhattacharyya, On the measures of divergence between two statistical
  populations defined by their probability distributions, Bulletin of the
  Calcultta Mathematical Society 35 (1943) 99–109.

Kullback1951
S. Kullback, R. A. Leibler, On information and sufficiency, The Annals of
  Mathematical Statistics 22 (1) (1951) 79–86.
https://doi.org/10.1214/aoms/1177729694
  doi:10.1214/aoms/1177729694.

Kolmogorov1956
A. Kolmogorov, On the shannon theory of information transmission in the case of
  continuous signals, IRE Transactions on Information Theory 2 (4) (1956)
  102–108.

Cover2006
T. M. Cover, J. A. Thomas, Elements of Information Theory, Second Edition, John
  Wiley & Sons, Hoboken, 2006.

Kapur1992
J. N. Kapur, H. K. Kesavan, Entropy Optimization Principles with Applications,
  Academic Press, San Diego, 1992.

Gray2011
R. M. Gray, Entropy and Information Theory, 2nd Edition, Springer, New York,
  2011.
https://doi.org/10.1007/978-1-4419-7970-4
  doi:10.1007/978-1-4419-7970-4.

Shannon1948
C. E. Shannon, A mathematical theory of communication, Bell system technical
  journal 27 (3) (1948) 379–423 & 623–659.
https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
  doi:10.1002/j.1538-7305.1948.tb01338.x.

Lafon2006
S. Lafon, A. B. Lee, Diffusion maps and coarse-graining: A unified framework
  for dimensionality reduction, graph partitioning, and data set
  parameterization, IEEE transactions on pattern analysis and machine
  intelligence 28 (9) (2006) 1393–1403.
https://doi.org/10.1109/TPAMI.2006.184
  doi:10.1109/TPAMI.2006.184.




 Paper Title: AI-Assisted Generation of Difficult Math Questions.

































 
    
       
            
       
       
       
      
         






breakable



































































breakablebox[1][]
    breakable,
    colback=mybackcolor2, 
    colframe=myframecolor2, 
    fonttitle=,
    title=#1,


genbox[1][]
    breakable,
    colback=mybackcolor, 
    colframe=myframecolor, 
    fonttitle=,
    title=#1,


example
  Example:  
  




AI-Assisted Generation of Difficult Math Questions
    
  Vedant Shah*^1,2   Dingli Yu^3   Kaifeng Lyu^3   Simon Park^3

  Nan Rosemary Ke^1   Michael Mozer^4   Yoshua Bengio^1,2

  Sanjeev Arora^3   Anirudh Goyal*^1 

  ^1Mila – Quebec AI Institute   ^2Université de Montréal   ^3Princeton University 
   ^4University of Colorado, Boulder

    August 5, 2024
=================================================================================================================================================================================================================================================================================================



























§ ABSTRACT 
Current LLM training positions mathematical reasoning as a core capability. With publicly available sources fully tapped, there is an unmet demand for diverse and challenging mathematics questions. Relying solely on human experts is both time-consuming and costly, while LLM-generated questions often lack the requisite diversity and difficulty.  We present a design framework that combines the strengths of LLMs with a human-in-the-loop approach to generate a diverse array of challenging math questions. Initially, leveraging LLM metacognition skills <cit.>, a strong LLM is used to extract core “skills” from existing math datasets.  These skills serve as the basis for generating novel and difficult questions by prompting the LLM with random pairs of core skills that must be utilized in the question. The use of two very different skills within each question makes finding such questions an “out of distribution” task for both LLMs and humans. Our pipeline employs LLMs to iteratively generate and refine questions and solutions through multi-turn prompting. Human annotators then verify and further refine the questions, with their efficiency enhanced via further LLM interactions. Applying this pipeline on skills extracted from MATH dataset <cit.> resulted in MATH^2 - a dataset of higher quality  math questions,

as evidenced by:
(a)  Lower performance of all models on MATH^2 than on MATH
(b) Higher performance on MATH when using MATH^2 questions as in-context examples. 
Although focused on mathematics, our methodology seems applicable to other domains requiring structured reasoning, and potentially as a component of scalable oversight.

Also of interest is a striking relationship observed between models' performance on the new dataset: the success rate on MATH^2 is the square on MATH. This suggests that successfully solving the question in MATH^2 requires a nontrivial combination of two distinct math skills.





§ INTRODUCTION



Significant improvement in the capabilities of LLMs <cit.> to understand and generate complex mathematical content has been achieved by leveraging all the public data and a fair bit of private data. Sources of high-quality, varied, and difficult mathematical questions are drying up. Even finding new questions for evaluation is getting difficult since newly-released human exams are somewhat similar to past exams,  which are potentially present in the LLMs' training datasets. Hence, there is a pressing need for innovative methods to create new, diverse, and challenging 
questions.


Expert mathematicians and educators possess the deep understanding required to create questions that not only test a wide range of skills but also push the boundaries of what the learners, and by extension, the models, can handle. However, relying solely on human experts is not scalable.

Generating synthetic questions using LLMs is feasible at scale <cit.>, but often fall short in terms of the necessary difficulty. <cit.> employs a similar approach as ours where they extract topics and corresponding keypoints from a set of seed problems using GPT-4, and then combine the topic to generate new questions, again using GPT-4). However, the generated data is meant to be used for the finetuning of models as compared to serving as an evaluation set in our case. As a result, the questions generated in <cit.> are not sufficiently difficult. Similarly, limited work exists on ensuring the necessary diversity in the generated synthetic data. <cit.> proposes prompting frontier models to generate questions where each question is generated in the context of a persona as a way of ensuring diversity. They use 1M different personas to generate questions, which are then used for finetuning models, leading to significant improvements.  This dichotomy between the quality of human-generated questions and the scalability of LLM-generated questions presents a significant challenge <cit.>.













 §.§ Evaluation Saturation Phenomenon




LLM  evaluations getting saturated is a well-known issue. 
Some of the saturation is driven by across-the-board improvements arising from better training and more extensive/better datasets. 

But a lot has to do with evaluation-specific enhancements that optimize model performance on standard evaluations through techniques like supervised fine-tuning (SFT) on synthetic question-answer pairs. These synthetic pairs can be generated by leading proprietary models when provided with a few examples from the dataset or by filtering the model’s own responses <cit.>. Such methods can dramatically boost performance; for example, just 1 million synthetic examples can elevate Llama2 7B’s performance on the MATH dataset to levels comparable to GPT-4 <cit.>.

The distinction between general and evaluation-specific improvements is crucial. The latter may lead to overfitting to particular evaluations rather than a genuine acquisition of mathematical skills. This issue was highlighted when a new version of the GSM8K dataset revealed performance drops in many models, indicating overfitting to the previous dataset version <cit.>. Similarly, leading LLMs performed significantly worse on newer versions of the Chinese GaoKao exam compared to older exams, raising fundamental questions about the depth of their mathematical understanding. 







 §.§ Proposed Framework: AI-assisted Generation of Difficult Math Questions



 
Traditional methods of question generation, which rely heavily on human expertise, are increasingly unsustainable due to their resource-intensive nature, especially for academic research. Purely synthetic questions generated by large language models (LLMs) often fail to meet the desired standards of complexity and novelty. To address this challenge, we propose a design framework that leverages the strengths of both LLMs and human expertise in a synergistic manner.


At first glance, it may seem counterintuitive to use an AI model to generate and correct novel questions that it is unable to solve itself. 
However, recent research <cit.> demonstrated that top LLMs possess a robust understanding of mathematical skills, including the capability to identify the skills required to solve given questions <cit.>. This naturally raises the question: can LLMs operate in the reverse direction, i.e., generate math problems when given a list of skills that have to be tested? Our initial attempts yielded mixed results. While leading models could produce creative math questions when provided with a list of skills, the majority of these questions exhibited one or more of the following shortcomings: too similar to existing questions in datasets;  have errors or nonsensical elements; are too tedious or mechanical to be engaging for human annotators. (See Section <ref>.) 
Moreover, they often conflate “difficulty” with tedious calculations, which actually would play to the strength of machines to leverage external tools such as calculators or Python interpreters.




Nevertheless, there were promising instances where LLMs generated interesting and correct questions that they were unable to solve,  due to incomplete or incorrect reasoning. This observation led us to the concept of AI-assisted creation of evaluation datasets. Our process may also be of interest for human pedagogy since it begins with the extraction of core "skills" from existing math datasets, which serve as the foundational elements of mathematical questions. 
The current paper focuses on the MATH dataset <cit.>, a mainstay of  LLM evaluation in recent years.

Using state-of-the-art LLMs, we generate new, targeted questions that require random pairs of these skills.  The involvement of LLMs in creating questions based on these skills introduces a level of creativity and novelty that human efforts alone may not achieve. Finally, the proposed human-in-the-loop approach ensures that the generated questions meet high standards of quality and challenge, benefiting from human oversight and refinement.



Starting with a list of mathematical skills extracted from the MATH dataset using recently discovered methods <cit.>, we focused on creating questions that involve one skill from pre-algebra and algebra portions of the MATH dataset and one other skill randomly sampled from different sections of MATH. 
 Our generation pipeline uses carefully crafted prompts and multi-turn interactions with leading models to significantly improve the generation of high-quality questions and candidate answers.



In our AI-assisted process, human experts played a crucial role. Using the (question, answer) pairs generated by LLMs and leveraging API access to leading models, experts identified promising questions—-often those incorrectly answered by the LLMs but containing many correct ideas. Experts then refined these questions to enhance their engagement value and provided gold-standard answers. The AI-assisted process not only boosted human productivity but also resulted in high-quality, novel questions distinct from those in existing datasets. 

Importance of using random pairs of skills: We think that  this is key driver of improved diversity and difficulty among generated questions. Recall that MATH dataset is neatly partitioned into sub-areas such as “Geometry” and “Number theory.”  Requiring generated questions to combine skills from two subareas (e.g., a question linking area-and-perimeter calculations with prime number knowledge) necessitates “out of distribution” thinking; some examples appear in Section <ref>. Such questions challenged all LLMs  as well as our human raters. Our new dataset is much harder than MATH for all models. For instance, if a model has success rate 0.5  on MATH, then  success rate on our new dataset is closer to 0.5^2 =0.25.  This trend is more general, and Section <ref>  suggests that  this is a indication that the average question of MATH^2 indeed requires nontrivial use of two distinct underlying skills, which is the reason for naming this new dataset as MATH^2.
We believe that our methodology could also introduce fresh perspectives into math instruction for  AI and human learners. 










Connection to Scalable Oversight: This notion <cit.> looks ahead to how humans might supervise and check AI systems that potentially outperform humans in many relevant skills. While typically discussed in the context of alignment and safety, the concept is pertinent here. How can human experts reliably evaluate LLMs' understanding of high-school or freshman-level math when these models have already been trained on all available exams and textbooks? Could human-AI collaboration lead to more novel evaluations?


Paper organization: Sections <ref> describes our design methodology and generation pipeline for MATH^2. 


Section <ref> discusses the performance of many open-source and proprietary models  on MATH<cit.> as well as on the new MATH^2 dataset of 180 questions (see Table <ref> ).  Section <ref> discusses the interesting relationship between MATH and MATH^2 scores.

Section <ref> shows that MATH^2 questions are more useful than MATH questions when used as in-context exemplars for various LLMs. Section <ref> sheds some light on interesting behaviors and failure modes of leading LLMs that we observed during the question generation process. 















§ PIPELINE FOR AI-ASSISTED QUESTION GENERATION



We present a structured approach to generating challenging mathematics questions by combining the capabilities of large language models (LLMs) and human expertise. Given below is a high-level overview of the process before delving into the details of each step.

We begin our pipeline with skill extraction - identifying and cataloging distinct mathematical skills from a dataset, as described in <cit.>. This step creates a repository of skills linked to specific questions. The motivation behind this is to systematically generate and analyze questions that require specific skills, ensuring a comprehensive evaluation framework.

Next, we focus on generating questions that combine pairs of distinct skills to increase their difficulty. By using advanced models like GPT-4 and Claude, and incorporating in-context examples of multi-way interactions between AI and humans, we enhance the models' performance in generating complex questions. This step aims to produce challenging questions that robustly assess problem-solving abilities.

The final step involves screening and validation to filter out invalid or flawed questions. This rigorous process includes evaluating and solving the questions to identify hidden flaws, such as computational intractability or logical inconsistencies. Advanced techniques like in-context exemplars and self-consistency further ensure the accuracy and quality of the solutions. This step is crucial for maintaining the integrity and reliability of the generated questions and their solutions. Overall, each step in the pipeline is designed to systematically enhance the quality and difficulty of questions, providing a robust and comprehensive evaluation of mathematical skills.

Motivated by these challenges, we employ a five-step approach to generate difficult math questions using advanced models. For each round of generation, we randomly sample a pair of skills and three sample question-solution pairs corresponding to each skill from the skill repository. These reference examples are sourced from the MATH dataset.

Step 1: Skill Pair Validation.
We begin by asking the LLM (GPT-4 or Claude) to validate the skill pair by assessing the qualitative similarity of the two skills. Reference examples are provided in-context to enrich the model's understanding of the skills. If the model deems the skills too similar, they are flagged and excluded from question generation, as similar skills might lead to simpler questions.

Step 2: Question Generation.
Next, we prompt the LLM to generate a question and a brief solution requiring the application of both skills in the sampled pair. We specify two conditions to ensure high-quality questions: the question should either require an exact answer or specify that an approximate answer is acceptable, and it should ask for only a single final result. In-context, we provide two multi-turn conversations between a human and an AI assistant. These conversations demonstrate the human providing feedback on the AI-generated questions, which the AI then refines. This helps the model anticipate and avoid practical issues, such as insufficient involvement of skills or logical inconsistencies. Appendix <ref> provides examples of the responses of different models in the question generation step. 

Step 3: Solution Attempt.
The model then attempts a solution to the generated question, adopting an adversarial approach to identify flaws such as insufficient information, ambiguity, self-contradiction, or excessive computation. If any issues are found, the model stops solving and clearly states the problems. Otherwise, it completes the solution. During this step, the model does not receive the skill names or reference examples to ensure unbiased problem-solving.

Step 4: Question Validation.

We give LLM the generated question and its solution for validation against a fixed rubric consisting of seven criteria:


    
  * Single Answer Requirement: The question should ask for only one final answer.
    
  * Exact Answer Requirement: There should be only one exact answer, unless approximations are explicitly stated.
    
  * Dual Answer Requirement: The question must necessarily and sufficiently involve the application of both skills, with difficulty comparable to or greater than the reference examples.
    
  * Clarity and Completeness: The question should be clear and contain all necessary information.
    
  * Computational Tractability: The question should not require overly complex computations.
    
  * Realism and Logic: The scenario should be realistic and logically consistent.
    
  * Syntax and Grammar: The question should be grammatically correct and clearly written.


The model uses reference examples and validation exemplars - model generated examples of validating questions, to facilitate this step. We employ majority voting (maj @ 4) to enhance robustness.

Step 5: Final Solution and Re-validation.
For questions classified as valid, we ask the LLM to re-solve the question to obtain a final solution. Reference examples are provided in-context to improve the model's understanding. We use majority voting (maj @ 4) to ensure consistency. If all the answers obtained in this step are unique, indicating potential ambiguity, the question is discarded.

The questions obtained from the above pipeline are further screened by humans. This structured approach not only generates challenging and novel math questions but also ensures their quality through rigorous validation, effectively combining the strengths of AI and human oversight. For detailed examples of prompts used at each step, refer to Appendix <ref>.


    
  * Skill Pair Validation. Firstly, we ask GPT-4 / Claude Opus to validate the skill pair by determining the qualitative similarity of the two skills. We supply the reference examples to the model in-context, to enrich its understanding of the two skills. If the model deems a pair of skills to be similar, they are flagged and not used for question generation. The rationale behind eliminating pairs with similar skills is that such pairs might lead to simple questions. 
    
  * Generation. Next, we prompt GPT-4 / Claude to generate a question and a brief solution such that the question requires the application of both skills in the sampled pair, to be solved. In order to ensure high-quality generations, we explicitly give the model a list of two conditions that the generated question should satisfy: Firstly, the answer to the question should either be exact, or it should be specified in the question that the answer is expected to be only approximately correct. Second, the question should ask for only a single final result. It should not ask for multiple things. We also provided in-context, two multi-turn conversations between a human and an AI assistant of the following form.  The conversation shows the human prompting the AI assistant to come up with a challenging math question requiring the application of two mathematical skills (which may not be similar to the current skills at hand). The AI assistant comes up with a question and the human gives feedback, pointing out the problems with the generated question. Then the AI assistant incorporates the feedback and modifies the question. This goes on till the human is satisfied with the generated question. Using these conversations in context helps the models to understand some of the practical issues that arise in the process and possibly avoid them, e.g.,  insufficient involvement of the skills in the generated question, logical inconsistencies in the questions, etc. Similar to the previous step, we also provide the reference in context to help the model understand the skills and their applications better.
    
  * Solution Attempt. In the next step, we ask the model to attempt a solution to the generated question, but to approach this adversarially: look for flaws such as insufficient information in the question,  ambiguity, self-contradiction, need for excessive computation, final answer being too complicated for autograding,  etc.  If the model encounters any of the above problems in any step of the solution, it is told to not solve the question any further and clearly state the issues it found. Else, it should solve the question to completion. Note that we do not supply the model with the skill names or the reference examples in context in this step.
    
  * Question Validation. In this step, we give GPT-4 / Claude the question generated in the first step and the solution obtained in the second step and ask it to validate the question by evaluating it according to a fixed rubric. If the question does not meet any of the criteria in the rubric, the question is classified as invalid and discarded. The model is given the reference examples in-context to facilitate this validation. The rubric is a set of 7 criteria, each described briefly below:
    
        
  * Single Answer Requirement - The question should ask for only final answer
        
  * Exact Answer Requirement - There should be only one possible exact answer to the question, without the need to make any assumptions or approximations whatsoever, unless explicitly states in the question.
        
  * Dual Answer Requirement - The question must necessarily and sufficiently involve the application of both the skills. To ensure sufficiently rigorous application of both skills, we ask the model to ensure that the difficulty of the application of each skill should be similar to or greater than that in the reference examples.
        
  * Clarity and Completeness - The question should be unambiguous and contain all the information necessary to complete the solution.
        
  * Computational Tractability - Solving the question should not involve very complicated computations. The computations involved should be manageable by a human withing a reasonable amount of time.
        
  * Realism and Logic - The question should involve scenarios that are realistic, plausible, and logically consistent with common sense.
        
  * Syntax and Grammar - The question must be grammatically correct and clearly written to prevent misinterpretation.
    

    A question needs to satisfy all the above criteria to be valid. Further, we provide the model with two types of exemplars to facilitate this step. Firstly, we give the model the reference examples for each skill. Secondly, we also give the model a set of 6 question validation outputs which are manually annotated with human feedback on how the model performed while validating a given question. These validation outputs are not human crafted but instead were extracted from the output of older iterations of the validation prompt. We use majority voting (maj @ 4) <cit.> to increase the robustness of the validation step.
    
  * Final Solution and Re-validation. If the question is classified as valid in Step 4, we ask GPT-4 / Claude to (re-)solve the question to get a final solution. The reference examples for both the skills are provided in-context to improve the model's understanding of the skills. We use majority voting (maj @ 4) to improve the model's performance. Additionally, we investigate the final answer obtained by the model in all 4 trials. If all the answers obtained are unique (i.e. if no single answer has a majority), we discard the question since it hints at the possibility of some ambiguity in the question which was not detected in the validation step.






§ EXPERIMENTS AND FINDINGS



Through our experiments, we demonstrate the difficulty and quality of the MATH^2 while also analyzing the behavior of different models on this task of compositional generalization. Firstly, we evaluate a wide range of models spanning a large range of parameter counts on MATH^2 and compare against their performance on MATH <cit.> which is the base dataset used for extracting skills, showing that the MATH^2 is necessarily harder than MATH. Next, we further demonstrate the difficulty and quality of questions in MATH^2 by showing that they are better in-context exemplars as compared to standardly used exemplars. We describe the experimental setup below.





 §.§ Experimental Setup
 

We follow the pipeline proposed in <cit.> to extract skills from the MATH dataset <cit.>. The MATH dataset encompasses seven high-level topics, allowing us to identify and extract finer-grained skills within each topic and label each question accordingly. At the end of the skill-extraction process, we identify a set of 114 skills. We then remove a few simple skills, such as  and , before using the remaining set to generate questions using the proposed approach. We generate and verify 180 difficult questions to create the MATH^2 dataset. Figure <ref> shows the distribution of skills in MATH^2.


r9.76cm


Comparison of Zero-Shot CoT Performance (Accuracy) on the Generated Dataset vs. MATH Test Set:

GPT-4 Omni demonstrates the least drop in percentage terms (13.42%) whereas MAmmoTH-7B shows the highest relative drop (92.91%). 







Model      MATH^2 (Y)     MATH (X)     % Drop   



GPT-4 Omni          66.85%              77.21%        13.42%  

Claude 3.5 Sonnet     47.78%     73.54%     35.45% 

GPT-4 Turbo             57.22%            73.27%          21.90%    

Gemini-1.5-Pro     43.34%     67.70%     35.98% 

Claude 3 Opus                39.66%            61.20 %       35.20%  

Llama-3-70B-Instruct         22.77%           47.89%       52.45% 

MetaMath-70B       6.11%           26.27%      76.74% 

MAmmoTH-70B         5.00%            19.31%      74.11% 

Mixtral-8×7B-Instruct            12.78%           31.52%      59.45%  

MetaMath-13B       2.79%              21.32%      86.91% 

MAmmoTH-13B         2.23%             10.99%      79.71% 

Deepseek-math-7b-instruct     17.88%     45.05%     60.31% 

Llama-3-8B-Instruct        9.45%          28.62%       66.98% 

Gemma-1.1-7B-Instruct       7.78%          23.36%      66.69% 

MetaMath-7B         2.23%            18.69%        88.07% 

MAmmoTH-7B            0.56%             7.90%      92.91%  

Phi-3-mini-128k-instruct     22.78%     45.14%     49.53% 

Gemma-1.1-2B-Instruct           2.78%             7.52%      63.03% 






 


Table <ref> presents details of the changes made to the questions during the human verification process. Out of 180 question-solution pairs included in MATH^2, 79 underwent some form of modification by the human annotators before being included in the dataset. Out of the 41 questions modified, 3 were minor modifications to improve the clarity of the question. Another 24 modifications were minor modifications (up to 3 words), which nevertheless affected the meaning of the question and changed the final answer. But 14 modifications were significant;  either making the given questions harder, or correcting them, or making them more interesting (i.e., less tedious) for humans.

As for the solutions,  62 out of the 180 solutions originally generated by the model were modified to correct them or improve their clarity. 

In total,  56% of the question-answer pairs in MATH^2 appear exactly as phrased by their LLM creator. 

We evaluate the generated set of questions on a variety of language models, both small and large. Specifically, we assess the MetaMath <cit.>, MAmmoTH <cit.>, Gemmma <cit.>, and Llama-3 series, Phi-3, deepseek-math as well as one Mixture-of-Experts model Mixtral-8×7B-Instruct. Additionally, we include evaluations of larger proprietary models such as GPT-4o, GPT-4 Turbo[Points to  at the time of writing] <cit.>, Gemini-1.5-Pro, Claude 3.5 Sonnet [Points to  at the time of writing] and Claude 3 Opus[Points to  at the time of writing]. We compare the performances of these models on our generated questions against their performance on the MATH dataset <cit.>.

For generating responses, we use the MAmmoTH <cit.> evaluation suite. The responses are graded using a GPT-4 grader, where GPT-4 Omni checks the correctness of a solution response against the ground truth solution. During response generation, we set the temperature to 0 and top_p to 1 for all models. All necessary compute details are discussed in Appendix <ref>



 §.§ Performance across the two datasets: A surprising pattern 
 




Table <ref> shows that all tested models   have significantly lower performance on MATH^2 than on the original MATH dataset. Denoting Y as the performance on MATH^2 and X as the performance on MATH, the percentage drop 100(X- Y)/X
for frontier models — GPT-4 Omni, GPT-4 Turbo, Gemini-1.5-Pro, Claude-3.5-Sonnet and Claude 3 Opus — ranges from 13.42% to 35.45%. 
MAmmoTH-7B, a specialist math model, shows the largest drop at 92.91%. 




The fact that performance drops for all models should not be too surprising, since as noted, the 
MATH^2 questions, by combining skills from different subareas of MATH, could be seen as  “out of distribution (OOD).” This makes it tempting to interpret the percentage drop as a measure of a model's (lack of) “OOD-resilience.” For instance,
very large percentage drops seen with open-source models MetaMath and MAmmoTH feel understandable since their training used synthetic data generated using seed questions from MATH and GSM-8k. Lack of diversity in such synthetic data is known to cause overfitting to the  dataset being imitated.  Similarly, GPT-4o and Claude Sonnet 3.5 are suspected to also have been extensively trained with synthetic data.  Although their MATH performance is similar, Sonnet 3.5 has worse MATH^2 performance,  which might suggest lower quality/diversity in its synthetic data.

However,  in our opinion, the overall pattern among proprietary models of similar size does fit with the OOD story.


A much simpler explanation pops out when we plot Y vs X^2 (Figure <ref> and Figure <ref>): we find a linear relationship Y ≈ X^2!

This implies that the relative drop in performance of the models is well-predictable from just their performance on MATH, and does not require taking their training details into account! 

Why should the two scores be expected to have this relationship? Here is a natural (albeit heuristic) explanation.  Suppose there are N skills and s_i denotes the success rate of the model at correctly applying the ith skill. Then,  its X value should reflect the average[With perhaps a small correction factor if the skills are not evenly distributed among the questions] of the s_i's. Furthermore, on a random question using the ith and jth skill, the probability that the model correctly answers it should be  s_i s_j, since it has to successfully apply both skills. 
If the questions are created using  pairs of skills chosen randomly and independently, then the Y value will be the average value of  s_is_j's, which by independence will be roughly X^2. 

This reasoning in fact  suggests that our pipeline has created  questions
that genuinely required applying two very distinct skills (as opposed to, say,  requiring primarily skill i, and mildly using  skill j). The discovered relationship suggests further that if we could create questions where each combines k skills, we might see the relationship Y ≈ X^k, which would tend to further magnify performance differences between models. 







 







 §.§ Generated Questions are Effective In-Context Exemplars for MATH.
 


A possible test for the quality of a Q&A pair on similar topics as MATH dataset is whether performance on MATH improves when using these as in-context exemplars. 

We test as follows. Recall that MATH has 7 sections. Exemplars for a section are chosen from the section area.  However, by design, our new questions cross section boundaries.  We implemented a new procedure to retrieve in-context exemplars from MATH^2 based on the skill requirements of the current question.

Since MATH^2 is limited in size, it does not cover all the skills extracted during the skill extraction process, containing 97 out of 114 skills. Figure <ref> shows the distribution of different skills in the dataset. We filtered the MATH test set to remove examples requiring skills not present in the generated dataset, resulting in the removal of 809 test examples. During evaluation on the filtered MATH test set, for each question Q labeled with skill a (a ∈𝒮, where 𝒮 is the set of extracted skills), we retrieved in-context exemplars from the MATH^2, ensuring each exemplar involved skill a. We used four such exemplars per question (i.e., 4-shot CoT <cit.>). To handle skills represented by fewer than four examples in MATH^2, we run two experiments: (A) Proposed 4-shot CoT: If a given skill is represented by n examples in the MATH^2, where n < 4, we use n in-context examples instead of 4 exemplars. (B) Proposed + Skill Based 4-shot CoT: If a given skill is represented by n examples in MATH^2, where n < 4, we supplement 4-n exemplars for that skill from MATH training set. The relevant in-context exemplars in MATH training set are determined by following the methodology proposed in <cit.>. We compared the performance of models using these targeted prompting strategies against two baselines: (C) MAmmoTH 4-shot CoT: The 4 in-context exemplars are taken from the MAmmoTH evaluation suite <cit.>. (D) Skill Based 4-shot CoT: We use skill-based prompting as proposed in <cit.>, where the in-context exemplars are selected from the MATH training set, in accordance to the skill required by the question at hand, as determined by GPT-4. 

Table <ref> presents the results of this comparison. The two prompting strategies using questions from MATH^2 as in-context exemplars, clearly outperform the two baselines. We conclude that the MATH^2 questions, due to their difficulty and skill relevance, serve as effective in-context exemplars. Performance gains would likely be more significant with larger datasets generated using our approach, reducing the need to supplement with external exemplars.

















§ OBSERVATIONS FROM THE QUESTION GENERATION PROCESS




The question generation pipeline described in Section <ref> was developed through an iterative process of refining prompts and design choices, and evaluating their impact on the quality of the final questions and solutions. Notably, the inclusion of the attempted solution and question validation steps significantly enhanced the pipeline's effectiveness. Despite the sophistication of the pipeline and prompts, we still observe instances where models fail to follow the given instructions. This section highlights prominent failure modes at various stages of the pipeline, which human raters need to be aware of. Additionally, we explore some intriguing behaviors of the models where they successfully create interesting and creative questions. Section <ref> details the role of human raters in improving these questions.





 §.§ Creative questions: Examples of Synergy from Human-AI interaction 
 




The models frequently produced interesting and creative questions, although they often failed to generate correct solutions. In these cases, the incorrect solutions usually contained enough correct ideas for a human to quickly complete them.

Human annotators were tasked with verifying the validity of the questions and the correctness of the solutions. They were instructed to look out for any failure modes discussed in Section <ref>. Their responsibilities included ensuring that the created questions actually employed the intended math skills, and improving the questions in terms of readability, quality, and difficulty when possible. They were encouraged to suggest changes that would make the problems harder for automated tools to solve while allowing easier or more elegant solutions for humans. The following examples illustrate this process:





Our human team had not encountered such questions before. It requires recognizing that k^3 - k = k(k-1)(k+1) is always divisible by 2 and 3. Thus, k must be such that k(k-1)(k+1)/6 is divisible by 15 (both 3 and 5). Additionally, the sum of the digits of k must be a prime number, and ensuring such conditions is challenging even for powerful LLMs.





The original question combined elements in a novel way. The human rater modified the question to change the sequence size from 20 to 6 beads, maintaining the essential difficulty while making it more elegant for humans. All tested models failed on the modified question.







This was one of many questions the models created about exponential growth and geometric series, possibly similar to standard math test questions. The human slightly altered it to simplify calculations by hand and substituted a different condition that the models found challenging, while humans could easily estimate an approximate answer and then verify.







An LLM can solve the original question through simple computation. The modified question, however, requires understanding an underlying pattern.







Models tend to adopt a brute-force approach to the original question by calculating 15^4 + 16^4. After rephrasing, the number 23^17 + 17^17 is too large for direct computation, requiring understanding of arithmetic modulo a prime.

These examples highlight the essential role of human oversight in refining and improving the questions generated by LLMs, ensuring they are challenging, creative, and suitable for advanced mathematical problem-solving.








 §.§ Failure Modes





Despite the sophistication of our pipeline, models frequently exhibit several failure modes: (a) Insufficient Involvement of Skills: Models often generate questions that either miss one of the skills completely or require a very shallow application of one or both skills. For example, a geometry question may fail to involve ratio and proportion adequately, (b) Insufficient Information: Questions may lack essential details needed for solving, making them incomplete or ambiguous. For instance, a trigonometry question might omit necessary angles or distances, (c) Unsolvable or Computationally Intractable Questions: Some questions generated are either unsolvable or require excessive brute-force calculations, which are impractical for evaluating reasoning abilities, (d) Nonsensical Questions: Models sometimes produce questions that are logically inconsistent, confusing, or ambiguous, such as a probability problem with unclear parameters or an impossible geometry scenario, (e) Deceitful Solutions: Occasionally, models fabricate solutions to nonsensical or unsolvable questions, presenting incorrect logic as plausible reasoning and (f) Finding a Needle in the Haystack: Long and complex validation prompts sometimes cause models to confuse or overlook the specified skills, leading to incorrect evaluations. For a more detailed discussion and examples of questions in the various categories listed above, refer to Appendix <ref>.
 















§ CONCLUSIONS






We introduced a framework that leverages the complementary strengths of humans and AI to generate new, challenging mathematics questions. Building on recent insights into LLM metaknowledge, we use LLMs to extract and name key skills necessary for solving math problems. Using these insights, we developed a pipeline that employs named skills from the well-known MATH dataset, and leverages multi-turn interactions with advanced LLMs to generate questions that combine pairs of skills. These questions were subsequently reviewed and refined by human raters. The proposed pipeline produced questions with greater novelty and difficulty compared to those in the original MATH dataset.

This framework also resulted in a new math evaluation MATH^2, that assesses the same skills as the MATH dataset but is significantly more challenging for leading models because each question involves two skills from different parts of MATH.   GPT-4-T and GPT-4-O exhibited the smallest performance drops, which aligns with the subjective evaluations of our human raters. Additionally, we demonstrated that providing the newly generated questions as in-context examples improved GPT-4-O's performance on the MATH dataset more effectively than examples sourced directly from the MATH dataset. This finding further validates the quality of the questions produced by the proposed pipeline.

Open-source models performed disappointingly on these new questions, but this weak performance also suggests that their performance should be improveable using  novel questions of only moderate difficulty —- which our proposed pipeline can generate at scale. 

We plan to release detailed information about our pipeline to encourage further research and development in the field of open-source math models. 

Limitations and Future Work.  Our pipeline incurs moderately high costs due to  extensive API-based use of frontier models as well as significant human verification. To improve efficiency, future work should focus on optimizing prompting strategies to produce higher-quality questions initially, thereby reducing the need for extensive filtering. Additionally, reducing human verification through the development of automated validation tools is crucial. Enhancing our pipeline by integrating a training-based feedback loop, where the model is trained on  the questions that pass human verification, could further streamline the process by progressively improving question quality. 

These measures will reduce dependency on expensive proprietary models, lower overall operational costs, and maintain or even enhance the quality of the generated math evaluation benchmarks.

Looking ahead, an even more exciting prospect is the potential application of the proposed framework to efficiently produce high-quality data in domains beyond mathematics. 




§ ACKNOWLEDGEMENTS

This research used compute resources provided by Mila (mila.quebec) and  GPT4 access as well as compute resources provided by Princeton Language and Intelligence (PLI).  The Princeton participants were funded by NSF, DARPA, and PLI.  VS would like to thank Aniket Didolkar for helpful discussions throughout the project and for proof reading the paper. AG would like to thank Melvin Johnson, James McClelland and Yoram Bachrach for helpful discussions and useful feedback. AG would also like to thank Daan Wierstra, Melvin Johnson, Siamak Shakeri, Murray Shanahan, John Quan, Theophane Weber, Olivier Tieleman, David Silver, Charles Blundell, Behnam Neyshabur, Ethan Dyer and Nicolas Heess for support and guidance.  










abbrvnat












§ APPENDIX


Here we further analyze the quirks of the question generation pipeline and provide additional experimental details and results. In  Appendix <ref>, we discuss several failure modes of the models that we notice during the question generation process as well as interesting behaviors exhibited by the models throughout the pipeline and interesting creative questions that the models came up with. Appendix <ref> discusses the different considerations that human annotators were instructed to take into account while annotating and verifying the questions generated by the proposed pipeline. Appendix <ref> provides details about the compute used for running our experiments as well as some further analysis of the model evaluations. In Appendix <ref> we provide examples of questions generated by different models in the Question Generation step (Section <ref>). Appendix <ref> gives a detailed description of the prompts used for each step in the question generation pipeline as well as evaluation of the models. It also provides a link to the skill exemplar repository and in-context exemplars used in the question generation process.



 §.§ Failure Modes and Interesting Behaviors





  
Insufficient involvement of skills. Despite clearly specifying that solving the question should necessarily require a rigorous application of both skills, the models often generate questions that either miss one of the skills completely or require a very shallow application of one (while the other one is sufficiently involved) or both skills. This is the most prominent failure mode of the models in the context of question generation. This leads to potentially easy questions, defeating the purpose of skill composition. Consider the question given below which was generated by Claude Opus when asked to combine the skills  and . 



    


Upon careful examination of the question, we note that although the question tests , the involvement of  is practically non-existent. Further, the question validation step in some cases also fails to identify these flaws. Supplying multi-turn human-AI interactions where the user prompts a chatbot to generate a question combining two skills, in-context during the generation step helps the models to avoid such questions to a certain extent. Further, to make the question validation step more robust to such questions, we prompt the model to ensure that the complexity of each skill application in the question being validated in similar to or more than the complexity of these skills in the reference examples present in the skill descriptions. The combination of these two techniques helps us nearly eliminate questions where the absent one of the skills is absent completely and reduce questions involving shallow application of skills to a significant extent.



  
Insufficient information in the questions. Another common failure mode of the pipeline is the generated questions missing information or details essential for solving the question. For example in the question given below which is supposed to combine the skills  and , lacks sufficient detail about the inclinations and elevations of the paths relative to the streetlight's position which is necessary to answer the question.


    


To screen such questions, we include and explicit clause in the question validation prompt as described in Section <ref>. Moreover, we also notice that the inclusion of the solution attempt step improves the chances of detecting such errors since the missing information may not always be apparent from just the question itself. In such cases, attempting a solution (with a defeatist approach) can help detect such flaws.





  
Unsolvable or Computationally Intractable Questions. There are instances when the model generates questions which are unsolvable. For example the question given below has no solution which satisfies all three constraints (i.e., the area of the rectangle being 360 and the sides belonging to the two arithmetic progressions defined in the question.) 


    


In other instances, the model generates questions that are computationally intractable or require manually and tediously iterating through a long sequence of values. For example, solving the question given below requires manually calculating the first 100 terms of the sequence to find the sum 


    


While technically not wrong, such questions are not ideal for evaluating the reasoning abilities of the models since they mostly involve brute force calculations. Further, in cases where the sequence of calculations is very long, the LLM's performance may be bottlenecked by other limitations such as the context length of the model. 

Thus, we strive to filter such questions out. We add an explicit condition to check for computational tractability and solvability of the generated questions in the verification prompt. This check is assisted by the solution attempt produced by the model which will potentially point out any such problems. 



  
Nonsensical Questions. In several cases, the model comes up with questions which are nonsensical - confusing, incomprehensible, logically inconsistent or ambiguous. Consider the question given below.

Given below is an example of a question which is logically inconsistent. More concretely, a square plot of land whose side length is equal to the radius cannot fit inside the quarter-circle.


    


We add checks for such cases in the question validation prompt. Further, at the end of the final solution step (maj @ 4), we further check for cases where the final answer produced in all the 4 self-consistency trials are unique. If all answers are unique, we discard the question. The rationale behind this being that it is highly likely that the model produces a different answer every time due to some inherent ambiguity in the question which was not detected in the solution attempt and the question validation checks. 





  
Deceitful Solutions. Although rare, we encounter cases where the model makes up solutions even though the question is nonsensical or cannot be solved with the amount of information provided in the question. This happens very commonly in the solutions which are generated in the question generation prompt. Thus, we do not use these solutions and include the final solution step where the model is asked to solve the question again. Although most of such solutions and thus questions are screened out in the question validation step and consistency check at the end of the final solution check, in rare cases we see this behavior in the solution produced after the final solution step as well. Given below is one such example. 


    


While solving this question, the model arrives at the conditions sin(θ) = 2 or sin(θ) = -2. Clearly, these conditions have no solutions since -1 ≤ sin(θ) ≤ 1, ∀θ∈ [0, 2π]. However, the model goes on to argue that the the closest values to 2 and -2 in the range of sin(θ) are 1 and -1, and thus, goes on to solve the question for sin(θ) = 1 or sin(θ) = -1.



  
Finding a needle in the haystack. In an attempt to make the question validation step robust to as many failure modes as possible, we come up with a long and sophisticated prompt (see Appendix <ref> for an example of this prompt). To elaborate, the validation prompt comprises of 1.) skill descriptions consisting of 3 exemplars for each of the 2 skills, 2.) 6 very long validation exemplars 3.) rest of the prompt consisting of the 7 conditions described in Section <ref>. In prior iterations of the validation prompt, the names of the two skills which are supposed to be combined in the question, were mentioned only once throughout the prompt. In such a case, we observed that more often than not, model confused the pair of skills at hand, with skills mentioned in the validation exemplars. We also observed cases where the model was unable to locate the original skill names, came up with its own skill names depending on the skill exemplars provided in the skill descriptions. More specifically, while checking the question for Dual Skill Requirement in Section <ref>, the models would check the question for the presence of skills distinct from the pair of skills at hand. We attribute this phenomenon to the the lost in the middle or finding a needle in the haystack challenges faced by LLMs. We provide an example of the phenomenon below. Notice how in Point 3 of the reasoning trace the model checks the question for the presence of two completely unrelated skills. We were able to nearly eliminate this problem by replacing referential terms (such as above given skills, these skills, etc.) to the two skills by the skill names.


    

    

     

    

    

    

    

    
        
        

    

    

    

    

    

    

    

    


Note that none of the above failure modes are completely eliminated in the pipeline described in Section <ref>. Thus, human verification is required.

Despite struggling with the failure modes described above, there also exist cases where the models exhibit positively surprising and creative behaviors. We talk about some of them below.



  
Thinking out of the box. Although rare, we observe instances where the models get creative while validating the question. Consider the question below


    


When validating this question using prior iterations of the question validation prompt, which did not consist of the computational tractability check, the model output while validating the question consists of the following excerpt.


    ...This might introduce a significant challenge not solely due to the methodology's complexity but also due to the potential computational requirement, which may not be feasible in a standard test environment without tools. Furthermore, while the connection to practical geometry (reflective and rotational symmetry) and combinatorics (color patterning and adjacency constraints) is strong, the depth of understanding required to manually adjust for these symmetry considerations in a test question might be too intense or require more guided learning than a single evaluation question could provide....


i.e, the model takes into consideration the fact that the question involves a lot of brute force computation, despite there being no explicit check for computation complexity in the prompt, and classifies the question as invalid. We attribute such out of the box thinking behavior to the role-playing nature of our prompts. Our prompts consist of a math teacher evaluating the the fitness of the given question for being used for testing students' reasoning and analytical skills in a math exam. This leaves room open for the model to detect potential problems not explicitly accounted for in the prompts which might make the question unfit for being used for evaluation.




















































    











    
















 §.§ Considerations for human-annotaters



Human annotators were tasked with double checking the validity of the question and the correctness of the solution. They were asked to look out for any of the failure modes discussed in Section <ref>.
They were asked to check that the created question actually used the math skills it was supposed to exhibit and to improve the question with respect to readability, quality and difficulty. They were encouraged to suggest changes that  make the problem harder to solve using automated tools while retaining easiness for the humans. We illustrate with an examples. 

GPT-4 created the following question given the skill-tags 
 and   :


    


An LLM can solve this by simple computation. The human modified the question so that solving the problem requires understanding the underlying pattern. 


    


For the modified question, one leading model mentioned calculation difficulties for the inability to give any answer, and another resorted to an incorrect numerical approximation that led to an incorrect answer. 

Human annotators were also asked to go through the solutions carefully and correct or improve the solution for good questions if necessary. They were also asked to look out for questions that contain lot of enumeration, i.e. questions which are tedious and require significant amount of brute force computation. For such questions, the annotators were encouraged to reword them such that enumeration is not a feasible strategy below. For example, given below is an example of an enumerative question which was modified to avoid enumeration.


    

    


 Models tend to adopt brute force approach on the original question calculating 15^4 + 16^4. After rephrasing the models cannot use brute force on 23^17 + 17^17, instead being forced to check the divisors more analytically, in particular understanding of arithmetic modulo a prime.









 §.§ Further Experimental Details and Results


For open source LLMs, we use 2 80GB A100 GPUs and 72GB of RAM to run inference facilitated by vLLM <cit.>. For evaluating GPT-4 Omni and GPT-4 Turbo, we use 25 parallel workers to make the API calls, and for Claude-3 Opus we use 2 workers.

Analysis of performance degradation on smaller models. Figure <ref> 
shows that smaller models generally experience larger relative performance declines compared to larger models in the same family.

It is known that larger models are better at composing skills or concepts (a phenomenon known as "compositional generalization") than smaller models. This finding aligns with conclusions drawn in <cit.>.

Skill Representation in the Generated Dataset. Figure <ref> shows the distribution of different skills in the generated set of questions. Figure <ref> shows the comparison in relative performance drops between models of different sizes belonging to the same family.



 §.§ Skill composition using different models


In this section, we provide examples of the responses of different models when asked to generate a question combining two different skills. We prompt GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Llama-3-70B-Instruct and Llama-3-8B-Instruct to generate a question combining the skills  and . We use the same prompt used in the Question Generation step (Step 2) of the proposed pipeline described in Section <ref> (see Appendix <ref>). Given below are the responses of each model to the prompt.

[GPT-4 Omni]
Sure! Here is a challenging math question that tests both area calculation skills and algebraic manipulation skills, formatted as requested.


# QUESTION

A rectangle ABCD has vertices A(0, 0), B(a, 0), C(a, b), and D(0, b). Inside this rectangle, there is a right triangle PQR where P is at vertex A, Q is on BC, and R is on CD. The area of triangle PQR is given by 1/6 of the area of rectangle ABCD. Given that the coordinates of Q and R are (a, kb) and (ka, b) respectively, where k is a constant, find the value of k.


# SOLUTION

First, let's find the area of the rectangle ABCD:


    Area of rectangle  ABCD = a · b



The area of the triangle PQR is given as 1/6 of the area of the rectangle ABCD:


    Area of triangle  PQR = 1/6· (a · b) = ab/6



Next, we need to find the area of triangle PQR using the coordinates of P, Q, and R.

The vertices of triangle PQR are P(0, 0), Q(a, kb), and R(ka, b).


The formula for the area of a triangle with vertices at (x_1, y_1), (x_2, y_2), and (x_3, y_3) is:


    Area = 1/2| x_1(y_2 - y_3) + x_2(y_3 - y_1) + x_3(y_1 - y_2) |



Substitute the coordinates of P, Q, and R into the formula:


    Area of  PQR = 1/2| 0(kb - b) + a(b - 0) + ka(0 - kb) |



    Area of  PQR = 1/2| 0 + ab - ka^2b |



    Area of  PQR = 1/2| ab(1 - k) |



    Area of  PQR = ab/2 |1 - k|



We know that the area of triangle PQR is ab/6:


    ab/2 |1 - k| = ab/6



To solve for k, we can divide both sides by ab/2:


    |1 - k| = 1/3



This gives us two possible equations:


    1 - k = 1/3  or   1 - k = -1/3



Solving these equations for k:


    1 - k = 1/3  ⇒   k = 1 - 1/3 = 2/3



    1 - k = -1/3  ⇒   k = 1 + 1/3 = 4/3



Therefore, the possible values for k are 2/3 and 4/3.


# DETAILS

The question involves area calculation by requiring students to find the area of a triangle within a rectangle, and algebraic manipulation by solving an equation involving a variable. The problem complexity is similar to the examples provided, ensuring that students are sufficiently challenged. The final answer consists of the value(s) of k, making it straightforward to evaluate the students' answers.


[Claude 3 Opus]
Here is my attempt at generating 1 (non multiple choice) difficult question which tests both area_calculation_skills and algebraic_manipulation_skills:


# QUESTION

Consider an equilateral triangle ABC with side length 2a. Let points D, E, F be the midpoints of sides BC, CA, AB respectively.
 
Now construct squares ACDE, ABFG, BCHI externally on sides AC, AB, BC respectively. 

Express the area of the shaded region (consisting of the equilateral triangle DEF and squares ACDE, ABFG, BCHI) in the form ka^2, where k is a constant. Find the value of k.


# SOLUTION

Let the side length of the equilateral triangle ABC be s = 2a.

The area of equilateral triangle ABC is √(3)/4s^2 = √(3)/4(2a)^2 = √(3)a^2.


In an equilateral triangle, the distance from a vertex to the midpoint of the opposite side is √(3)/2 times the side length. So DE = EF = FD = √(3)/2s = √(3)a.


The area of equilateral triangle DEF is √(3)/4(√(3)a)^2 = 3√(3)/4a^2.


The side length of each of the squares ACDE, ABFG, BCHI is equal to the side length of the equilateral triangle ABC, which is s = 2a.
 
So the area of each square is s^2 = (2a)^2 = 4a^2.


The total area of the shaded region is:

Area of DEF + 3 * Area of each square

= 3√(3)/4a^2 + 3(4a^2) = 3√(3)/4a^2 + 12a^2 = (12 + 3√(3)/4)a^2


Therefore, the area of the shaded region is ka^2, where k = 12 + 3√(3)/4≈ 15.1981.


# DETAILS

This question combines area calculation of equilateral triangles and squares with algebraic manipulation of side lengths and areas. Key steps involve:


1) Calculating areas of equilateral triangles ABC and DEF in terms of side length s and a respectively. This tests area_calculation_skills.


2) Algebraically manipulating the expressions for side lengths DE, EF, FD and square side lengths in terms of s and a. This requires algebraic_manipulation_skills.


3) Adding up the component areas algebraically and simplifying the expression in the form ka^2 to solve for constant k. 


The question difficulty is calibrated to be similar to the provided examples. The solution is a single numerical constant k which avoids ambiguity in student answers. The equilateral triangle and square constructions provide an interesting geometric visualization.


Let me know if you would like me to modify anything in the question or solution to better suit your needs! I'm happy to refine it further.




[Gemini 1.5 Pro]
# QUESTION

A circle is inscribed in equilateral triangle ABC, and a smaller circle is externally tangent to this circle and to sides AB and AC.  If the side length of triangle ABC is 4, what is the area of the smaller circle?


[asy] size(150); defaultpen(linewidth(0.8)); pair A=(0,2*sqrt(3)), B=(-2,0), C=(2,0), O=(0,sqrt(3)/3), P=(0,sqrt(3)); draw(A–B–C–A); draw(circle(O,sqrt(3))); draw(circle(P,sqrt(3)-1)); [/asy]


# SOLUTION

Let O be the center of the larger circle, and let P be the center of the smaller circle.  Let r be the radius of the smaller circle.


[asy] size(150); defaultpen(linewidth(0.8)); pair A=(0,2*sqrt(3)), B=(-2,0), C=(2,0), O=(0,sqrt(3)/3), P=(0,sqrt(3)); draw(A–B–C–A); draw(circle(O,sqrt(3))); draw(circle(P,sqrt(3)-1)); draw(O–P); draw(O–C); draw(P–C); label("A",A,N); label("B",B,SW); label("C",C,SE); label("O",O,S); label("P",P,NE); label("2",(1,0),S); label("r",(P+C)/2,E); label("1",(C+O)/2,SE); [/asy]


Since O is the centroid of equilateral triangle ABC, OC = 2/3· 2 √(3) = 4 √(3)/3.  Also, PC = r + 1, so by the Pythagorean Theorem on right triangle POC,


    r^2 + 2r + 1 = PC^2 = OC^2 - OP^2 = 16/3 - (r + √(3))^2.
This simplifies to 2r^2 + (2 + 2 √(3)) r - 13/3 = 0.  By the quadratic formula,


    r = -(2 + 2 √(3)) ±√((2 + 2 √(3))^2 - 4 · 2 · -13/3)/4 = -1 - √(3)±√(40)/4.
Since r is positive,


    r = -1 - √(3) + 2 √(10)/4.
Then the area of the smaller circle is


    π r^2 = π( -1 - √(3) + 2 √(10)/4)^2 = 27 π/8 - 13 √(3)π/8 + 5 √(30)π/4.



# DETAILS

This question tests both area calculation skills (area of a circle) and algebraic manipulation skills (solving a quadratic equation with radicals).  The problem is challenging because it involves nested radicals, but the steps are all fairly standard. 




[Llama-3-70B-Instruct]
Here is a difficult question that tests both area calculation skills and algebraic manipulation skills:


# QUESTION

A square with side length 4 is inscribed in a circle. A semicircle is drawn inside the square, with its diameter being one of the sides of the square. Find the area of the shaded region.


[asy]

size(100);

draw(unitcircle);

draw((-2,0)–(2,0)–(2,2)–(-2,2)–cycle);

filldraw((-2,0)–(2,0)–(2,2)–(-2,2)–cycle,lightgray);

[/asy]


# SOLUTION

The area of the shaded region is equal to the area of the square minus the area of the semicircle. The area of the square is 4^2 = 16. The area of the semicircle can be found using the formula 1/2π r^2, where r is the radius of the semicircle. Since the diameter of the semicircle is 4, the radius is 2, so the area of the semicircle is 1/2π(2)^2 = 2π. Therefore, the area of the shaded region is 16 - 2π.


# DETAILS

This question requires students to calculate the area of a square and a semicircle, and then subtract one from the other to find the area of the shaded region. It also requires algebraic manipulation skills to simplify the expression for the area of the semicircle. The answer is not a simple numerical value, but rather an expression involving π, which adds an extra layer of complexity to the question.



[Llama-3-8B-Instruct]
Here is a question that tests both area calculation skills and algebraic manipulation skills:


# QUESTION

Find the area of the region bounded by the curve y=x^2-4x+3 and the x-axis, to the nearest hundredth.


# SOLUTION

The area of the region can be found by integrating the function y=x^2-4x+3 with respect to x from 0 to 2, which is ∫_0^2 (x^2-4x+3) dx = [x^3/3 - 2x^2 + 3x]_0^2 = (8/3) - 8 + 6 - [0] = 2. The area can also be found by using the formula for the area of a region bounded by a curve and the x-axis, which is ∫_0^2 y dx = ∫_0^2 (x^2-4x+3) dx = 2. Therefore, the area is approximately 2 square units, to the nearest hundredth.


# DETAILS

This question tests both area calculation skills, as it requires the student to find the area of a region bounded by a curve and the x-axis, and algebraic manipulation skills, as it requires the student to integrate a function with respect to x.


Note: I've also reviewed the question and solution to ensure that they are accurate and clear for students.





 §.§ Prompt Examples


We detail the prompts used in the various stages of question generation and evaluation in the following sections. We present the skeleton of the prompts with placeholders for skill names, in-context exemplars, etc. in favor of keeping the prompt brief. The in-context exemplars and the skill exemplar repository are made available https://anonymous.4open.science/r/ai_assisted_questions-131A/here.



  §.§.§ Skill Pair Validation



[Prompt for validating a pair of skills]
I have two basic mathematical skills described below, each followed by some sample question - solution pairs such that solving the questions involves the use of the particular skill. 


SKILL 1: <skill_1>


Question 1: <skill_1_question_1> 

Answer 1: <skill_1_solution_1>


Question 2: <skill_1_question_2>

Answer 2: <skill_1_solution_2>


Question 3: <skill_1_question_3>

Answer 3: <skill_1_answer_3>


SKILL 2: <skill_2>


Question 1: <skill_2_question_1>
 
Answer 1: <skill_2_solution_1>


Question 2: <skill_2_question_2> 

Answer 2: <skill_2_solution_2>


Question 3: <skill_2_question_3>

Answer 3: <skill_2_solution_3>


I am going to use these two skills for framing a new question such that the question requires an expertise in both the skills in order to be solved, i.e. the question will compose these two skills. However, I do not want the two skills to be very similar, i.e., they should not mean the same thing. Go through the descriptions of the skills carefully. Based on your understanding of the skills, can you please tell me whether the two skills are essentially entirely the same or not? Think step by step and give a detailed explanation of your answer. The answer should begin with a prefix '# EXPLANATION
'. Note that your understanding of the skills should not be restricted to the sample questions provided previously. They are just example questions. Use your own prior knowledge as well. End your response with a 'Yes' or 'No' answer to whether the skills are similar or not. This final answer should be on a new line and preceded by the prefix '# FINAL ANSWER
'. Thank you very much!









  §.§.§ Question Generation



[Prompt for question generation]
I am a math teacher trying to create challenging math questions for smart students. I was wondering if you could give me 1 (non multiple choice) question which tests both the following skills:
 (<skill_1>, <skill_2>)
Please also provide a brief solution. Then please look over the question and the solution, and fix any issues so that my students do not get frustrated. This being a math exam, the answers should either be exact, or if not possible, then the question should clearly say the answer is only expected to be approximately correct. Further, for ease of evaluating the students' answers, the question should ask for a single final result.This process is difficult so I am attaching two sample conversations where (Agent) is an AI agent and (Query) is teacher feedback. The conversations revolve around framing such mathematical reasoning questions and using them for evaluating students. These should give you some idea of the expectations and the potential difficulties involved in this task. I am also giving three example question - answer pairs for both <skill_1> and <skill_2> skills, such that the example questions test the corresponding skill. Please ensure that the complexity / difficulty of application of <skill_1> and <skill_2> skills in the generated question is similar to the complexity / difficulty of the skills in the example questions. Please format your output as
 

'# QUESTION

<question>


# SOLUTION

<solution>


# DETAILS

<all other text>'


SKILL 1: <skill_1>


Question 1: <skill_1_question_1>
 
Answer 1: <skill_1_solution_1>


Question 2: <skill_1_question_2>

Answer 2: <skill_1_solution_2>


Question 3: <skill_1_question_3>

Answer 3: <skill_1_solution_1>


SKILL 2: <skill_2>


Question 1: <skill_2_question_1>
 
Answer 1: <skill_2_solution_1>


Question 2: <skill_2_question_2>
 
Answer 2: <skill_2_solution_1>


Question 3: <skill_2_question_3>

Answer 3: <skill_2_solution_3>


# CONVERSATION 1

<agent_convo_1>


# CONVERSATION 2

<agent_convo_2>





    




  §.§.§ Attempted Solution



Prompt for solution attempt. Note that we instruct the model to take a defeatist approach towards solving the question

[Prompt for solution attempt]
You are a professional math teacher and you are given a question which is supposed to test the analytical and mathematical reasoning abilities of your students. You are supposed to provide a solution to the given question. However, the question may be flawed. For example, it might have problems like question being unsolvable using the information provided, question being self-contradictory, the final answer being computationally intractable, the question being ambiguous and confusing, question having multiple possible interpretations, etc., which you may encounter while solving the problem. This question being used for evaluating students in math, the question should ideally have a single, exact answer, with no room for any deviations due to factors such as approximations, rounding errors, etc., unless explicitly specified in the question. Problems such as the ones described above, would prevent the students from solving the question properly, and thus, any question with either of these problems is unfit for testing the students. If you encounter any such problems, stop the solution right there and explain the problems. For example, if you encounter the need to make any approximations or rounding which is not specified in the question, stop solving the question along with the reason. You do not need to solve the question further once you encounter any such problem. If you do not encounter any such problem, solve the question to achieve the single exact answer which the question asks for.


# QUESTION

<question>








  §.§.§ Question Validation


Note that how in the first paragraph, the names of the two skills are mentioned even time instead of using referential phrases. This is done to address the lost in the middle problem

[Prompt for validating the questions]
You are a professional math teacher. You want to evaluate the analytical and mathematical reasoning abilities of your students in a math exam. The students are supposed to sit in an examination hall and solve the questions within a given time limit, without access to any computational devices. The evaluation is designed to test the students' expertise in using two given mathematical skills simultaneously, namely <skill_1> and <skill_2>. This is achieved by asking them to solve a question that necessitates expertise in both <skill_1> and <skill_2> skills, to be solved completely. Since evaluating the students is a critical task allowing very little margin for any error in the process, it is very important to ensure that the questions used for evaluating are high quality and fit for being used to evaluate the students. You need to carefully review the question and a given attempt at solving it, and ensure that the question is of high quality and fit to assess students. In order to do this, you should check the quality of the question with respect to several criteria, such as:


- Single Answer Requirement: The question should ask for one and only one final result. It should not request multiple distinct answers or pieces of information.

- Exact Answer Requirement: It should be possible to achieve one,exact answer to the question, without the need of making any approximations or assumptions whatsoever, unless explicitly specified in the question. There should be no margin for the students to arrive at any other possible answer due to things like rounding errors, etc.

- Dual Skill Requirement: The question must require rigorous expertise in both a: a) '<skill_1>' and b) '<skill_2>', for resolution. Application of both <skill_1> and <skill_2> and their subskills should be, necessary and contribute directly to obtaining the final answer; <skill_1> and <skill_2> skill should be applicable separately and critically during the problem-solving process. You are also given three example question - answer pairs for both <skill_1> and <skill_2> skills in order to help you better understand the meaning of each skill. Please carefully review the question and its attempted solution, paying close attention to how well it aligns with the examples provided for each skill. Consider the depth and breadth of knowledge demonstrated in the examples. The complexity / difficulty of application of both <skill_1> and <skill_2> in the question should be similar or greater than the complexity / difficulty of <skill_1> and <skill_2> in the example question-answers given for that respective skill. 

- Clarity and Completeness: The question should be unambiguous and contain all the information necessary to complete the solution. Any required assumptions not common knowledge should be explicitly stated. Check for any ambiguity that might confuse students. Carefully go through the solution to check if it makes any assumption or approximation in order to solve the question.

- Computational Tractability: Since the students are supposed to solve the questions within a given time limit and without access to any computational devices such calculators, computer, mobile phones, etc., you must ensure that the question is computationally tractable and all the computations involved can be done by hand in a limited amount of time.

- Relevancy of Information: The question should not have any extra details that do not contribute to the solving of the problem.

- Realism and Logic: The question should involve realistic scenarios or hypotheses with logically consistent data. The specified operations and the contextual setup should reflect plausible mathematical situations. (e.g., positive amounts for transactions, integers for counts).

- Syntax and Grammar: The question must be grammatically correct and clearly written to prevent misinterpretation.

- etc. (any other problems which you think make the question not fit for being used for evaluating the students)

Your task is to give a 'Yes' or 'No' assessment, indicating whether the question is high quality and suitable for evaluating the students on simultaneous application of the skills <skill_1> and <skill_2>. Provide thorough reasoning for your assessment based on the conditions mentioned above and any other relevant analytical points concerning mathematical reasoning and problem-solving. Your response should be structured as follows:


# REASONING

<Your detailed analysis justifying your decision>


# FINAL ANSWER

<'Yes' or 'No'. No other text should be present in this section>


Ensure to review the combination of skills intended for assessment, and check the logical flow and mathematical correctness from the question's setup to the solution's conclusion. Look out for any problems in the question which are pointed out in the attempted solution. Account for all the potential pitfalls such as logical inconsistencies, unnecessary complexity, or insufficient detail that may obstruct the clarity or solvability of the question. Given below are the two skills and some example question-answer pairs for the two skills. This process is difficult so I am attaching a few sample conversations where (agent) is an AI agent who is trying to verify the questions and (query) is teacher feedback. This should give you some idea of potential difficulties in this task. This is followed by the question which you need to check (preceded by '# QUESTION TO BE CHECKED') and its attempted solution (preceded by '# SOLUTION ATTEMPT'). 


SKILL 1: <skill_1>


Question 1: <skill_1_question_1> 

Answer 1: <skill_1_solution_1>


Question 2: <skill_1_question_2>

Answer 2: <skill_1_solution_2>


Question 3: <skill_1_question_3>

Answer 3: <skill_1_solution_3>


SKILL 2: <skill_2>


Question 1: <skill_2_question_1> 

Answer 1: <skill_2_solution_1>


Question 2: <skill_2_question_2> 

Answer 2: <skill_2_solution_2>


Question 3: <skill_2_question_3>

Answer 3: <skill_2_solution_3>


# CONVERSATION 1

<validation_exemplar_1>


# CONVERSATION 2

<validation_exemplar_2>


......


# CONVERSATION 6

<validation_exemplar_6>


# QUESTION TO BE CHECKED

<question>


# SOLUTION ATTEMPT

<solution>


Thank you very much!









  §.§.§ Final Solution


For the final solution, we make use in-context exemplars from MATH <cit.> as opposed to the attempted solution step.

[Prompt for the final solution]
I have two basic mathematical skills described below, each followed by some sample question - solution pairs such that solving the questions involves the use of the particular skill in order to be solved.


SKILL 1: <skill_1>


Question 1: <skill_1_question_1>
 
Answer 1: <skill_1_solution_1>


Question 2: <skill_1_question_2>

Answer 2: <skill_1_solution_2>


Question 3: <skill_1_question_3>

Answer 3: <skill_1_solution_3>


SKILL 2: <skill_2>


Question 1: <skill_2_question_1>
 
Answer 1: <skill_2_solution_1>


Question 2: <skill_2_question_2>
 
Answer 2: <skill_2_solution_2>


Question 3: <skill_2_question_3>

Answer 3: <skill_2_solution_3>


Go through the descriptions of the skills carefully. Now, here is a new question such that the question requires an expertise all both the skills in order to be solved. That is, the question composes these two skills


QUESTION: <question>


Based on your understanding of the skills, can you please solve the question accurately? Think step by step and explain the solution. Finally, end your response by stating the final numerical answer obtained using the solution. Note that your understanding of the skills should not be restricted to the sample questions provided in their description. They are just example questions. Use your own prior knowledge as well. The explanation of your solution and the final numerical answer should each be on a new line, and should be preceded by the prefixes '# SOLUTION
' and '# ANSWER
' respectively. Thus, your response should be in the format:


'# SOLUTION

<solution>


# ANSWER

<final_answer; no other text should be present in this section>'.


Thank you very much!









  §.§.§ Evaluation



[Prompt given to the GPT-4 for evaluating the model's solution]
You are a professional math teacher and are tasked with evaluating your students on a math exam. You are will be given a question, the correct solution to the question and the student's solution. You need to tell me whether the student solved the question correctly, thus matching the answer obtained by the correct solution. Think step-by-step and give a detailed explanation of your answer. At the end, give a 'Yes' or 'No' answer to whether the student's solution is correct. Your output should be in the following format:


# STEP BY STEP EXPLANATION

<detailed explanation of your thought process>


# CORRECTNESS

<'Yes' if the student's solution is correct. 'No' otherwise. This section should not contain any other text>


Here are the question, correct solution to the question and the student's solution:

QUESTION: <question>


CORRECT SOLUTION: <correct_solution>


STUDENT'S SOLUTION: <student's_solution>





    







 Paper Title: Synth-Empathy: Towards High-Quality Synthetic Empathy Data.






^Peking University     ^University of Chinese Academy of Sciences     ^Shanghai AI Laboratory ^ Institute of Information Engineering, Chinese Academy of Sciences


 ^†hao.liang@stu.pku.edu.cn, ^†sunlinzhuang21@mails.ucas.ac.cn,  wentao.zhang@pku.edu.cn



§ ABSTRACT 
In recent years, with the rapid advancements in large language models (LLMs), achieving excellent empathetic response capabilities has become a crucial prerequisite. Consequently, managing and understanding empathetic datasets have gained increasing significance. However, empathetic data are typically human-labeled, leading to insufficient datasets and wasted human labor. In this work, we present Synth-Empathy, an LLM-based data generation and quality and diversity selection pipeline that automatically generates high-quality empathetic data while discarding low-quality data. With the data generated from a low empathetic model, we are able to further improve empathetic response performance and achieve state-of-the-art (SoTA) results across multiple benchmarks. Moreover, our model achieves SoTA performance on various human evaluation benchmarks, demonstrating its effectiveness and robustness in real-world applications. Furthermore, we show the trade-off between data quantity and quality, providing insights into empathetic data generation and selection. The codebase and data are made available at <https://github.com/Aurora-slz/Synth-Empathy>.


Synth-Empathy: Towards High-Quality Synthetic Empathy Data
    Hao Liang^†, Linzhuang Sun^†, Jingxuan Wei^, Xijie Huang^, Linkun Sun^, Bihui Yu^, Conghui He^, Wentao Zhang^*
    
==================================================================================================================

[† The first two authors have equal contributions. 

* Corresponding Author

]
footnote-1





























§ INTRODUCTION


In recent years, with the rapid advancements in large language models (LLMs) <cit.>, data management has become a crucial aspect of these technologies <cit.>. At the same time, <cit.> also demonstrates that data processing, selection, and management can significantly influence the performance of LLMs.

Empathy, the ability to understand and share the feelings of another, is a critical component of human social interaction and communication. It allows individuals to connect with others on an emotional level, fostering relationships and promoting prosocial behavior  <cit.>. In the future of Human-central Artificial General Intelligence (AGI), excellent empathetic response capability is a crucial prerequisite, which focuses on equipping LLMs to understand and respond appropriately to human feelings  <cit.>.





In previous research, empathy modeling has predominantly concentrated on human-labeled data  <cit.> and integrating information through modifications to the model architecture  <cit.>. While the importance of model architectures is well acknowledged, the quality and quantity of data are also critical  <cit.>. Additionally, <cit.> indicates that new data management methods are needed for LLMs due to their massive data requirements. Previous empathetic studies underscore the significance of data-centric approaches. They frequently overlook effective data generation approaches, leading to the following two key challenges:






C1. High Cost of Human-Labor. 
Previous methods rely on human labor for the creation of empathetic datasets <cit.>. However, this process can be extremely costly and requires substantial human effort.

C2. Poor Effectiveness.
Previous research <cit.> has only a limited amount of data. Despite efforts by  <cit.> to improve model performance, the empathetic effectiveness remains constrained. More empathetic data is needed to train high-performance empathetic models.




To address these issues, as shown in Figure <ref>, we propose a three-step empathetic data generation and curation pipeline and subsequently train an empathetic LLM. First, we utilize prompts to generate empathetic responses. Next, we apply domain knowledge for quality selection to curate the empathetic data. Finally, we conduct diversity selection for further data curation. With the curated synthetic dataset, we fine-tune an LLM and achieve state-of-the-art (SoTA) empathetic response performance.


The core contributions of this paper are summarized as follows:



  * New Perspective.
Limited data and low effectiveness are significant impediments to the practical adoption of empathy models. To the best of our knowledge, this study represents the first approach to generate data from scratch to address these challenges.

  * New Method.
We propose a new data generation and curation pipeline for empathy, introducing the first generated high-quality empathy dataset. Utilizing our meticulously curated synthetic dataset, we pioneer the integration of synthetic data to empathetic model training. Our method, which utilized curated synthetic data, enables robust effective, and user-friendly empathetic response.

  * SoTA Performance. 
    (1) Effectiveness in Empathetic Response. 
    By utilizing the synthetic empathetic response data to fine-tune LLMs, our method can achieve SoTA performance in multiple benchmarks, as shown in Figure <ref>. At the same time, we achieve SoTA performance on human evaluation benchmarks. This means our model have high application potential. 

    
    (2) No Human Labor. 
    By utilizing curated prompts and LLMs, we can obtain high-quality empathetic responses without the need for human labor. With a meticulously designed data curation pipeline, as illustrated in Figure <ref>, we achieve high-quality data, as demonstrated in Figure <ref> and Figure <ref>. This approach significantly reduces the requirement for human labor.
    
    



§ RELATED WORK




 §.§ Empathetic Response Generation

 
 Imbuing models with empathy to enhance emotional experience is a crucial aspect of AGI.  <cit.> constructed the ED dataset, the most authoritative dataset for measuring the empathic abilities of models, which has since become a benchmark for empathy research. Building on this dataset, numerous researchers have focused on improving models' empathy comprehension by integrating emotional labels and external knowledge.

 <cit.> introduces commonsense knowledge inference into this task by enriching historical conversation data with the pre-trained COEMT model <cit.>. Diverged from this approach,  <cit.> incorporate external knowledge through Graph Neural Networks (GNN) for context encoding. Additionally, <cit.> posit that using detailed sentiment labels can improve the accuracy of capturing user sentiment and puts forward a sentiment loss mechanism with multiple levels of granularity to enhance model training.  Moreover,  <cit.> strives to provide empathetic responses by analyzing context at the word level.  <cit.> employed a emotion detection algorithm based on psychological principles to pinpoint important statements in conversations.  <cit.> take a different approach by not only recognizing the emotions of others but also assessing saved model emotional state.  Furthermore,  <cit.> divides the empathetic response task into two steps: verifying the semantic content and infusing emotional expression.  

In the LLM era, many researchs explore ways to boost the empathic potential of models through meticulous design of prompts <cit.>. Differently,  <cit.> delves into the general sensibility and rationality, exploring their respective contributions to empathy. However, the importance of fine-grained sensibility and rationality cognition for empathy still lacks comprehensive investigation. 



 §.§ Data Quality and Data Selection

The advent of large language models has brought about a substantial increase in the volume of training data. <cit.> In this scenario, the quality and quantity of data become paramount. LLMs, trained on vast amounts of data, can capture subtle nuances and complex patterns in language, excelling in various natural language processing tasks. However, the increase in data volume also brings new challenges, particularly in data management, cleaning, and annotation. <cit.> In this section, we mainly discuss the effectiveness of data quality and data selection.



  
Data Quality: High-quality data can significantly enhance the performance of models <cit.>. As the volume of data increases, ensuring high data quality becomes more challenging because it requires more resources for data cleaning, selection and annotation. <cit.> Poor quality data can lead to models learning incorrect patterns and making inaccurate predictions. 



  
Data Selection: 
LLMs-based methods were commonly used in data selection. <cit.> For instance, <cit.> leverages DeBERTa <cit.> for scoring, retaining high-quality data, and combining it with the k-center greedy algorithm to select diverse data. <cit.> score the accuracy of data using ChatGPT to pick out high-quality data. <cit.> use GPT-4 to rewrite data to increase their complexity and then streamline it by reducing its variety and improving its quality. <cit.> train two models using ChatGPT's labeled data to score the quality and complexity of the data. <cit.> rely on ChatGPT to tag each instance, defining its complexity and diversity based on these tags. <cit.> first cluster the data, and then use GPT-4 to select high-quality data for each cluster.

Given the critical role of data quality and selection in enhancing model performance, our paper focuses on leveraging advanced data diversity and quality selection techniques to optimize empathetic data quality. By employing methods that integrate data features and similarity scores, we aim to efficiently identify and utilize high-quality data for empathetic response. 



 §.§ Data Generation

Data has always been the key driver behind the success of large language models (LLMs). Recent advancements of LLMs are largely due to the availability of large-scale, diverse, and high-quality datasets for training these models  <cit.> . However, the scarcity of data and the high costs present substantial challenges in obtaining such datasets  <cit.>. Recent advancements in generating synthetic data and improving the performance of LLMs have shown promising results across various domains. Synthetic data holds great potential in building large-scale, high-quality datasets. Researchers have explored multiple approaches, from leveraging differential privacy to creating instruction-tuning frameworks, to enhance the quality, diversity, and utility of synthetic data  <cit.>. A key component in generating high-quality synthetic datasets is precise alignment. Fan et al.  <cit.> introduce REALIGN, a method that enhances the quality of instruction data by reformatting responses to better align with pre-established criteria and evidence, thereby improving LLMs' alignment with human values while minimizing human annotation and model hallucinations. Li et al.  <cit.> build a high-quality instruction-following language model by automatically labeling human-written text with corresponding instructions and demonstrating highly effective self-alignment. 

In the field of empathetic response, the construction of generative datasets remains relatively underexplored. Empathetic models primarily rely on the utilization and curation of existing datasets. Approaches such as those by  <cit.> have attempted to improve empathetic response performance through data selection methods. However, these methods still face challenges due to the lack of high-quality empathetic response data.



§ METHOD

In this section, we first introduce the empathetic data generation method in subsection <ref>. Then we introduce the newly designed empathetic quality selection module in subsection <ref>. Additionally, we introduce the diversity selection in subsection <ref>. We demonstrated the high-quality of our dataset in subsection <ref>. With the curated dataset, we are able to fine-tune empathetic models.



 §.§ Empathetic Data Generation






In this subsection, we introduce the data generation method. Our data generation is based on the authoritative EmpatheticDialogues (ED) dataset <cit.>, which consists of 25,000 daily conversations encompassing 32 uniformly distributed emotional labels. By utilizing the prompt in Figure <ref>(1), we generate empathetic responses based on the high-quality ED dataset.

The formula for generating data using the prompt is as follows:


    Synthetic Dataset = LLM(Prompt(ED Data)


Following the formula above, we generate more than 250k empathetic data points for further selection.


 §.§ Empathetic Data Quality Selection

In this subsection, we introduce the data quality selection process for empathetic data. Similar to subsection <ref>, we utilize the ED dataset as the baseline and select data with an empathetic style similar to the ED dataset. The selection process is shown in Figure <ref>(2) and involves the following steps:



  
Empathetic Discriminator
We fine-tune a LLM on the ED dataset to serve as the discriminator, denoted as M_D. This discriminator is trained to produce responses that mimic the high-quality empathetic style of the ED dataset, denoted as D_ED.

    M_D = SFT(LLM; D_ED)




  
Similarity Based Quality Selection
In generating empathetic response data, we first use the discriminator M_D to answer the question. Then, we use the SoTA embedding model gte-qwen2-7b-instruct to embed the model's response and the generated response. We calculate the similarity between the discriminator's response and the generated response. We only select the responses with similarity higher than the threshold T. The optimized procedure is as follows:

    
  * Use the discriminator M_D to answer the question, obtaining the response r_D:
    
    r_D = M_D(question)


    
  * Use gte-qwen2-7b-instruct to embed the discriminator's response r_D and the generated response r_G, obtaining the embedding vectors E_D and E_G:
    
    E_D = gte-qwen2-7b-instruct(r_D)

    
    E_G = gte-qwen2-7b-instruct(r_G)


    
  * Calculate the similarity S between the discriminator's response r_D and the generated response r_G:
    
    S = E_D · E_G/E_DE_G


    
  * Select the generated response r_G if the similarity S is higher than the Domain Knowledge (DK) threshold T:
    
    r_G  is selected if  S > T



With the similarity selection based on the empathetic discriminator, we obtain a high-quality synthetic dataset.



 §.§ Empathetic Data Diversity Selection

In this subsection, we introduce the diversity selection method. The aim of diversity selection is to maximize the minimum distance between selected data points, ensuring a diverse subset. The algorithm can be formally described as follows:

Given a dataset X = {x_1, x_2, …, x_n}, our goal is to select a subset S ⊆ X of size k that maximizes the minimum distance between any point in X and the closest point in S. The objective function is:


    max_S ⊆ X, |S| = kmin_x ∈ Xmin_s ∈ S d(x, s)


where d(x, s) denotes the distance between points x and s.

As shown in Figure <ref>(3), we select the KCenterGreedy <cit.> algorithm for its robustness and quick computation. Using the KCenterGreedy algorithm, we obtain a high-diversity synthetic dataset.



 §.§ High Quality Generated Empathetic Data

In this section, we introduce an analysis of the quality of our generated dataset. We use a three-dimensional evaluation to assess the quality of our curated data. Using the prompts in Figure <ref>, we evaluate the dataset's coherence, naturalness, and empathy, scoring each from 1 to 3 points. 

As shown in Figure <ref>, our method significantly improves the dataset's quality. The average score increased from 2.65 to 2.88. Additionally, more than 95% of the data achieved full marks, demonstrating the high quality of our curated synthetic dataset. 

With high-quality generated empathetic response pairs, we can train a more effective empathetic model. Additionally, the empathetic response generation pipeline alleviates concerns about the quantity of empathetic data.









§ EXPERIMENTS

In this section, we first introduce the experimental setups, including data selection and the training process. We then aim to answer the following questions to verify the effectiveness and robustness of our proposed Synth-Empathy model: 

Q1: Can we fine-tune an empathetic model to achieve state-of-the-art (SoTA) performance with the curated synthetic data? 

Q2: Can our model achieve favorable human evaluation results, given the importance of human evaluation for empathy? 

Q3: Can we determine the trade-off between data quantity and quality to guide future empathetic data selection? 

Q4: Are the quality selection and diversity selection modules necessary to enhance model performance?




















 §.§ Experimental Setup



  §.§.§ Datasets


We evaluate our methods on the widely used EMPATHETICDIALOGUES dataset, which includes 25k multi-turn empathetic conversations.   Each conversation involves a speaker and a listener, with an average of 4.31 turns per dialog.   This dataset contains 32 evenly distributed emotion labels, with each conversation being associated with a specific label. And in order to guarantee the equity of the comparison baselines, we follow the same dataset division as the previous research method by splitting the dataset into training, validation, and test sets in an 8:1:1 ratio. 




  §.§.§ Baselines

The baseline models are listed as follows:

1. SEEK: Employ multi-granularity sentiment labels at the word, sentence, and dialogue levels to enhance emotional understanding.
2. CASE: Enhancing the cognitive capabilities of models with external knowledge sources, COMET and ConceptNet, to improve the quality of empathetic responses.
3. ESCM: Leveraging dynamic emotion-semantic vectors along with dependency trees to direct the model in generating empathetic responses.
4. Qwen1.5-7B-F.T.: Conducting SFT of Qwen1.5-7B-Chat model using ED dataset.
5. LLaMA3-8B-F.T.: Conducting SFT of LLaMA3-8B-Instruct model using ED dataset.
6. LLaMA3-8B: Generating empathetic response through designed prompt based on LLaMA3-8B-Instruct[<https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct>] model.
7. Mixtral-8x7B: Generating empathetic response through designed prompt based on Mixtral-8x7B-Instruct[<https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1>] model.
8. Qwen1.5-72B-Chat: Generating empathetic response through designed prompt based on Qwen1.5-72B-Chat[<https://huggingface.co/Qwen/Qwen1.5-72B-Chat>] model.




  §.§.§ Evaluation


To offer a thorough evaluation of our model's capabilities, we assess its performance through both automatic metrics and human evaluations.

Automatic Evaluation Metrics: For automatic evaluation, we employ corpus-level BLEU (B-1 to B-4), sentence-level ROUGE (R-1, R-2), and Distinct (Dist-1, Dist-2) metrics. BLEU and ROUGE scores measure the similarity between the generated text and the reference text, with higher scores reflecting greater similarity. Meanwhile, Distinct-N assesses content diversity, where higher values indicate a broader range of diverse representations. CIDEr aligns more closely with human judgment of sentence similarity by employing TF-IDF to assign varying weights to different n-grams. Specifically, it reduces the weight of frequently occurring phrases while increasing the weight of less common ones, thereby reflecting the distinctiveness and significance of the n-grams in the evaluation process. Besides, we employed xxx to assess semantic similarity from the perspective of vector embedding.

Human Evaluation Metrics: For human evaluation, we choose the following metrics: Coherence, Empathy, Informativeness, and Continuity. Coherence: Assesses how well the model's generated text aligns with the intended response. Empathy: Measures the model's capacity to grasp the speaker's situation and convey appropriate concern. Informativeness: Evaluates the extent of information provided in the responses produced by the model. Continuity: Reflects the model's effectiveness in maintaining the flow of the conversation. 


Then, the A/B test is conducted to evaluate the effectiveness of our model in comparison to several baseline models. Specifically, 200 examples are randomly selected from the test dataset. Each example is paired with two responses: one generated by our model and the other by a baseline model. Three evaluators assess each pair of responses, determining a winner, loser, or tie based on above four criteria.



  §.§.§ Settings

All experiments are carried out on the machine equipped with 8 NVIDIA A100 GPUs, a 120-core CPU, and 960GB of memory. The inference process of LLMs is implemented on the Vllm framework with version 0.4.1.



 §.§ Main Experiments

To address Q1, in this section, we evaluated various models on their performance in generating empathetic responses using several metrics, as summarized in Table <ref>. 

Our model was fine-tuned on the dataset combining the ED training set and our generated dataset. Our generated dataset comprises data with S greater than 60 and diversity selection.

Our model consistently outperformed other models across most metrics. Specifically, it achieved the highest BLEU scores, with 22.05 for BLEU-1, 10.53 for BLEU-2, 5.99 for BLEU-3, and 3.77 for BLEU-4, demonstrating its superior ability to generate text closely aligned with reference translations. In terms of Distinct metrics, which gauge the diversity of generated responses, our model scored 3.23 for Distinct-1 and 18.84 for Distinct-2, indicating a strong balance between diversity and coherence.

Our model also excelled in the Rouge metrics, attaining scores of 18.72 for Rouge-1, 5.27 for Rouge-2, and 16.88 for Rouge-L. These scores reflect its effectiveness in capturing the essential content of the reference texts and producing more contextually relevant responses. In the CIDEr metric, which assesses content relevance and richness, our model achieved a score of 26.96, the highest among the compared models, highlighting its effectiveness in generating captioning that aligns closely with human judgments of content relevance and informativeness.

Furthermore, in embedding-based evaluations using BERT, our model demonstrated superior performance with a BERT-P score of 20.5, a BERT-R score of 18.2, and a BERT-F1 score of 19.4. These results indicate that our model generates text with more meaningful and contextually accurate semantic representations compared to other models. 

Overall, these results confirm that our model provides superior performance across a range of metrics, demonstrating its effectiveness in generating high-quality and diverse outputs. The promising results highlight the effectiveness of the high-quality synthetic dataset, thereby introducing a new paradigm for using synthetic data in empathetic responses.




 §.§ Human Evaluation




To address Q2, we conducted a series of A/B tests. The comparative analysis includes our model versus SEEK, CASE, ESCM, Qwen1.5-7B-F.T., and LLaMA3-8B-F.T. The results of these tests are summarized in Figure <ref>.

The results of our A/B tests demonstrate that our model significantly outperforms SEEK, CASE, and ESCM, with win rates of 66%, 63.3%, and 57.9%, respectively. When compared with the more advanced models, Qwen1.5-7B-F.T. and LLaMA3-8B-F.T., our model still demonstrates competitive performance. Against Qwen1.5-7B-F.T., our model secures a win rate of 31% and maintains a reasonable loss rate of 19.8%, with ties at 49.2%. Similarly, our model achieves a win rate of 26% against LLaMA3-8B-F.T., with a loss rate of 15.1% and the highest tie rate of 58.9%. 


Overall, our model's strong performance in human evaluations underscores its practicality and user-friendliness. These results indicate that our model is not only effective but also well-received by human evaluators, highlighting its potential for real-world applications.





 §.§ Data Quality and Data Quantity


To address Q3, we analyzed the trade-off between the DK threshold, the similarity threshold in section <ref>, and the CIDEr score to demonstrate the robustness of our approach. Table <ref> and Figure <ref> present detailed experimental results, comparing various DK thresholds across the Qwen1.5-7B and LLaMA3-8B models using two different data generation prompts.

As the DK threshold increases from 50 to 70, the CIDEr score of Qwen1.5-7B with Prompt 1 initially rises from 20.45 to a peak of 25.07 at a DK threshold of 60, before slightly decreasing to 22.45 at a DK threshold of 70. For the LLaMA3-8B model, the CIDEr score shows a similar pattern, peaking at 24.34 with a DK threshold of 60, before decreasing at higher DK ratios. In terms of Prompt 2, the Qwen1.5-7B model demonstrates an increase in the CIDEr score from 20.36 at a DK threshold of 50 to a peak of 26.96 at a DK threshold of 60, followed by a slight decline. This observation corroborates a clear trade-off trend, emphasizing that an optimal DK threshold enhances the generation of empathetic responses.

The consistency of the trade-off pattern across different models and prompts underscores the robustness of our method. Regardless of the specific model or prompt used, the CIDEr score consistently peaks around the DK threshold of 60, demonstrating that our method effectively balances data retention and response quality. Additionally, by analyzing the trade-off between data quantity and quality, we provide guidance for optimal empathetic selection methods in the future. Utilizing the best threshold can achieve superior empathetic performance and select the appropriate dataset, thereby avoiding the waste of computational resources.





 §.§ Ablation Study

To address Q4, we provide the following ablation study. We remove the data quality selection module and the data diversity selection module to
demonstrate the effectiveness of our Synth-Empathy pipeline. 



  
Excluding Data Quality Selection Module.
When excluding the data quality selection module, the performance of the model decreases across all benchmarks, as shown in Table <ref>. For both the Qwen1.5-7B and LLaMA-8B-Instruct models, this exclusion results in decreased performance, demonstrating the necessity of the quality selection module.



  
Excluding Data Diversity Selection Module.
Similarly, the model's performance declines when the data diversity selection module is excluded, as shown in Table <ref>. For both data selection ratios of 55 and 60, and with both the Qwen1.5-7B and LLaMA-8B-Instruct models, this exclusion results in decreased performance, demonstrating the necessity of the diversity selection module.

Overall, the ablation study highlights the critical role of data generation and data selection modules in SynthVLM. These experiments provide valuable insights into the contributions of each module, guiding future improvements and optimizations of the SynthVLM model.
































§ CONCLUSION

Empathy is a critical component of human social interaction and communication. However, high-quality empathetic data remains scarce, necessitating efficient and effective data generation algorithms. This paper presents Synth-Empathy, a new paradigm for generating high-quality empathetic data, pioneering a method to obtain empathetic responses without human labor. By leveraging quality and diversity selection, we achieved high-quality synthetic empathetic data. Remarkably, utilizing the curated empathetic data, our model outperforms all previous models. Additionally, our model achieved SoTA performance in human evaluation, demonstrating its contextual appropriateness and user-friendliness.





ACM-Reference-Format





 Paper Title: Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models.



[
Measuring Progress in Dictionary Learning for 
 Language Model Interpretability with Board Game Models














equal*


Adam Karvonenequal,ind
Benjamin Wrightequal,mit
Can Ragerind
Rico Angellumass
Jannik Brinkmannmann
Logan Smithind
Claudio Mayrink Verdunhar
David Baunor
Samuel Marksnor


Adam Karvonenadam.karvonen@gmail.com

indIndependent
mitMIT
umassUniversity of Massachusetts, Amherst
mannUniversity of Mannheim
harHarvard University
norNortheastern University







Machine Learning, ICML

0.3in
]










 

§ ABSTRACT 
What latent features are encoded in language model (LM) representations? Recent work on training sparse autoencoders (SAEs) to disentangle interpretable features in LM representations has shown significant promise. However, evaluating the quality of these SAEs is difficult because we lack a ground-truth collection of interpretable features that we expect good SAEs to recover. We thus propose to measure progress in interpretable dictionary learning by working in the setting of LMs trained on chess and Othello transcripts. These settings carry natural collections of interpretable features—for example, “there is a knight on F3”—which we leverage into supervised metrics for SAE quality. To guide progress in interpretable dictionary learning, we introduce a new SAE training technique, p-annealing, which improves performance on prior unsupervised metrics as well as our new metrics.




§ INTRODUCTION



Mechanistic interpretability aims to reverse engineer neural networks into human-understandable components. What, however, should these components be?
Recent work has applied Sparse Autoencoders (SAEs) <cit.>, a scalable unsupervised learning method inspired by sparse dictionary learning to find a disentangled representation of language model (LM) internals. However, measuring progress in training SAEs is challenging because we do not know what a gold-standard dictionary would look like, as it is difficult to anticipate which ground-truth features underlie model cognition. Prior work has either attempted to measure SAE quality in toy synthetic settings <cit.> or relied on various proxies such as sparsity, fidelity of the reconstruction, and LM-assisted autointerpretability <cit.>.

In this work, we explore a setting that lies between toy synthetic data (where all ground-truth features are known; cf. <cit.>) and natural language: LMs trained on board game transcripts. This setting allows us to formally specify natural categories of interpretable features, e.g., “there is a knight on ” or “the bishop on  is pinned.” We leverage this to introduce two novel metrics for how much of a model's knowledge an SAEs has captured:

    
  * Board reconstruction. Can we reconstruct the state of the game board by interpreting each feature as a classifier for some board configuration?
    
  * Coverage. Out of a catalog of researcher-specified candidate features, how many of these candidate features actually appear in the SAE?


These metrics carry the limitation that they are sensitive to researcher preconceptions. Nevertheless, we show that they provide a useful new signal of SAE quality.

Additionally, we introduce p-annealing, a novel technique for training SAEs. When training an SAE with p-annealing, we use an L_p-norm-based sparsity penalty with p ranging from p=1 at the beginning of training (corresponding to a convex minimization problem) to some p < 1 (a non-convex objective) by the end of training. We demonstrate that p-annealing improves over prior methods, giving performance on par with the more compute-intensive Gated SAEs from <cit.>, as measured both by prior metrics and our new metrics.

Overall, our main contributions are as followsCode, models, and data are available at <https://github.com/adamkarvonen/SAE_BoardGameEval>:

    
  * We train and open-source over 500 SAEs trained on chess and Othello models each.
    
  * We introduce two new metrics for measuring the quality of SAEs.
    
  * We introduce p-annealing, a novel technique for training SAEs that improves on prior techniques.




§ BACKGROUND





 §.§ Language models for Othello and chess


In this work, we make use of LMs trained to autoregressively predict transcripts of chess and Othello games. We emphasize that these transcripts only give lists of moves in a standard notation and do not directly expose the board state. Based on behavioral evidence (the high accuracy of the LMs for predicting legal moves) and prior studies of LM representations <cit.> we infer that the LMs internally model the board state, making them a good testbed setting for studying LM representations.





  
Othello. 
Othello is a two-player strategy board game played on an 8x8 grid, with players using black and white discs. 
Players take turns placing discs on the board, capturing their opponent's discs by bracketing them between their own, causing the captured discs to turn their color. 
The goal is to have more discs turned to display your color at the end of the game. The game ends if every square on the board is covered or either player cannot make a move. 

In our experiments, we use an 8-layer GPT model with 8 attention heads and a n = 512 dimensional hidden space, as provided by <cit.>.
This model had no prior knowledge of the game or its rules and was trained from scratch on 20 million game transcripts, where each token in the corpus represents a tile on which players place their discs. 
The game transcripts were synthetically generated by uniformly sampling from the Othello game tree. 
Thus, the data distribution captures valid move sequences rather than strategic depth.
For this model, <cit.> demonstrated the emergence of a world model—an internal representation of the correct board state allowing it to predict the next move—that can be extracted from the model activations using a non-linear probe.
<cit.> extended this finding, showing that a similar internal representation could be extracted using linear probes, supporting the linear representation hypothesis <cit.>.



  
Chess. 
Othello makes a natural testbed for studying emergent internal representations since the game tree is far too large to memorize. 
However, the rules and state are not particularly complex. 
Therefore, we also consider a language model trained on chess game transcripts with identical architecture, provided by <cit.>.
The model again had no prior knowledge of chess and was trained from scratch on 16 million human games from the Lichess chess game database <cit.>. 
The input to the model is a string in the Portable Game Notation (PGN) format (e.g., “”).
The model predicts a legal move in 99.8 % of cases and, similar to Othello, it has an internal representation of the board state that can be extracted from the internal activations using a linear probe <cit.>.










 §.§ Sparse autoencoders

Given a dataset 𝒟 of vectors 𝐱∈ℝ^d, a sparse autoencoder (SAE) is trained to produce an approximation

    𝐱≈∑_i^d_SAE f_i(𝐱)𝐝_i + 𝐛

as a sparse linear combination of features. Here, the feature vectors 𝐝_i∈ℝ^d are unit vectors, the feature activations f_i(𝐱)≥ 0 are a sparse set of coefficients, and 𝐛∈ℝ^d is a bias term. Concretely, an SAE is a neural network with an encoder-decoder architecture, where the encoder maps 𝐱 to the vector 𝐟 = [     f_1(𝐱)          … f_d_SAE(𝐱) ] of feature activations, and the decoder maps 𝐟 to an approximate reconstruction of 𝐱. 

In this paper, we train SAEs on datasets consisting of activations extracted from the residual stream after the sixth layer for both the chess and Othello models. At these layers, linear probes trained with logistic regression were accurate for classifying a variety of properties of the game board <cit.>. For training SAEs, we employ a variety of SAE architectures and training algorithms, as detailed in Section <ref>.





§ MEASURING AUTOENCODER QUALITY FOR CHESS AND OTHELLO MODELS



Many of the features learned by our SAEs reflect uninteresting, surface-level properties of the input, such as the presence of certain tokens. However, upon inspection, we additionally find many SAE features which seem to reflect a latent model of the board state, e.g., features that reflect the presence of certain pieces on particular squares, squares that are legal to play on, and strategy-relevant properties like the presence of a pin in chess (Figure <ref>). 

Fortunately, in the setting of board games, we can formally specify certain classes of these interesting features, allowing us to more rigorously detect them and use them to understand our SAEs. In Section <ref>, we specify certain classes of interesting game board properties. Then, in Section <ref>, we leverage these classes into two metrics of SAE quality.




 §.§ Board state properties in chess and Othello models


We formalize a board state property (BSP) to be a function g : {game board}→{0,1}. In this work, we will consider the following interpretable classes of BSPs:


    
  * 𝒢_board state contains BSPs which classify the presence of a piece at a specific board square, where the board consists of 8× 8 squares in both games. For chess, we consider the full board for the twelve distinct piece types (white king, white queen, ..., black king), giving a total of 8× 8× 12 BSPs. For Othello, we consider the full board for the two distinct piece types (black and white), yielding 8×8× 2 BSPs.
    
  * 𝒢_strategy consists of BSPs relevant for predicting legal moves and playing strategically in chess, such as a pin detector. They were selected by the authors based on domain knowledge and prior interpretability work on the chess model AlphaZero <cit.>. We provide a full list of strategy BSPs in Table <ref> in the Appendix. Because our Othello model was trained to play random legal moves, we do not consider strategy BSPs for Othello.












 §.§ Measuring SAE quality with board state properties


In this section, we introduce two metrics of SAE quality: coverage and board reconstruction.



  
Coverage. Given a collection 𝒢 of BSPs, our coverage metric quantifies the extent to which an SAE has identified features that coincide with the BSPs in 𝒢. In more detail, suppose that f_i is an SAE feature and t∈[0,1] is a threshold, we define the function

    ϕ_f_i, t(𝐱) = 𝕀[f_i(𝐱) > t· f_i^max]

where f_i^max is (an empirical estimate of) max_𝐱∼𝒟f_i(𝐱), the maximum value that f_i takes over the dataset 𝒟 of activations extracted from our model, and 𝕀 is the indicator function. We interpret ϕ_f_i,t as a binary classifier; intuitively, it corresponds to binarizing the activations of f_i into “on” vs. “off” at some fraction t of the maximum value of f_i on 𝒟. Given some BSP g∈𝒢, let F_1(ϕ_f_i,t; g)∈ [0,1] denote the F1-score for ϕ_f_i, t classifying g. Then we define the coverage of an SAE with features {f_i} relative to a set of BSPs 𝒢 to be

    Cov({f_i}, 𝒢) := 1/|𝒢|∑_g∈𝒢max_tmax_f_iF_1(ϕ_f_i,t; g).


In other words, we take, for each g∈𝒢, the F_1-score of the feature that best serves as a classifier for g, and then take the mean of these maximal F_1-scores. An SAE receives a coverage score of 1 if, for each BSP g∈𝒢, it has some feature that is a perfect classifier for g. Since Cov depends on the choice of threshold t, we sweep over t∈{0, 0.1, 0.2, …, 0.9} and take the best coverage score; typically this best t is in {0, 0.1, 0.2}.



  
Board reconstruction. Again, let 𝒢 be a set of BSPs. Intuitively, the idea of our board reconstruction metric is that, for a sufficiently good SAE, there should be a simple, human-interpretable way to recover the state of the board from the profile of feature activations {f_i(𝐱)} on an activation 𝐱∈ℝ^d. Here, the activation 𝐱 was extracted after the post-MLP residual connection in layer 6.

We will base our board reconstruction metric around the following human-interpretable way of recovering a board state from a feature activation profile; we emphasize that different ways of recovering boards from feature activations may lead to qualitatively different results. This recovery rule is based on the assumption that interpretable SAE features tend to be high precision for some subset of BSPs, in line with <cit.>. For example, features that classify common configurations of pieces are high precision (but not necessarily high recall) for multiple BSPs.
 Thus, using a batch of “train” data 𝒟_train we identify, for each SAE feature f_i, all of the BSPs g∈𝒢 for which ϕ_f_i,t is a high precision (of at least 0.95) classifier. Then for each g∈𝒢 our prediction rule is



    𝒫_g({f_i(𝐱)}) = 
        1,    if ϕ_f_i,t(𝐱) = 1  for any  f_i  which
         is high precision for  g  on 𝒟_train
    
        0,    otherwise.







Let F_1(𝒫({f_i(𝐱)}); 𝐛) denote the F_1-score for a given board state 𝐛, where 𝒫({f_i(𝐱)}) = {𝒫_g({f_i(𝐱)})}_g ∈𝒢 represents the full predicted board (containing predictions for all 64 squares) obtained from the SAE activations. Then, the average F_1-score over all board states in the test dataset 𝒟_test can be calculated as:

    Rec({x_i}, 𝒟_test) = 1/|𝒟_test|∑_𝐱∈𝒟_testmax_t F_1(𝒫({f_i(𝐱)}); 𝐛),





§ TRAINING METHODOLOGIES FOR SAES


In our experiments, we investigate four methods for training SAEs, as explained in this section. These are given by two autoencoder architectures and two training methodologies—one with p-annealing and one without p-annealing—for each architecture. Our SAEs are available at <https://huggingface.co/adamkarvonen/chess_saes/tree/main> (chess) and <https://huggingface.co/adamkarvonen/othello_saes/tree/main> (Othello).






 §.§ Standard SAEs


Let n be the dimension of the model's residual stream activations that are input to the autoencoder, m the autoencoder hidden dimension, and s the dataset size. Our baseline “standard” SAE architecture, as introduced in <cit.> is defined by encoder weights W_e ∈ℝ^m × n, decoder weights W_d ∈ℝ^n × m with columns constrained to have a L_2-norm of 1, and biases b_e ∈ℝ^m, b_d ∈ℝ^n. Given an input 𝐱∈ℝ^n, the SAE computes

    𝐟(𝐱)    = ReLU(W_e (𝐱 - 𝐛_d) + 𝐛_e) 
    𝐱̂   = W_d  𝐟(𝐱) + 𝐛_d

where 𝐟(𝐱) is the vector of feature activations, and  𝐱̂ is the reconstruction. 

For a standard SAE, our baseline training method is as implemented in the open-source  repository <cit.>, optimizing the loss

    ℒ_standard = 𝔼_𝐱∼𝒟_train[ 𝐱 - 𝐱̂_2 + λ𝐟(𝐱) _1].


for some hyperparameter λ>0 controlling sparsity.











 §.§ Gated SAEs

The L_1 penalty used in the original training method encourages feature activations to be smaller than they would be for optimal reconstruction <cit.>. 
To address this, <cit.> introduced a modification to the original SAE architecture that separates the selection of dictionary elements to use in a reconstruction and estimating the coefficients of these dictionary elements. This results in the following gated architecture: 






    π_gate(𝐱) := W_gate(𝐱 - 𝐛_d) + 𝐛_gate


    𝐟̃(𝐱) := 𝕀[π_gate(𝐱) > 0] ⊙ReLU(W_mag(𝐱 - 𝐛_d) + 𝐛_mag)


    x̂(𝐟̃(𝐱)) = W_d 𝐟̃(𝐱) + 𝐛_d

where 𝕀[ · > 0] is the Heaviside step function and ⊙ denotes elementwise multiplication. Then, the loss function uses x̂_frozen, a frozen copy of the decoder:





    ℒ_gated := 𝔼_𝐱∼𝒟_train[    𝐱 - x̂(𝐟̃(𝐱))_2^2 
        + λReLU(π_gate(𝐱))_1 
        + 𝐱 - x̂_frozen(ReLU(π_gate(𝐱)))_2^2 ]









 §.§ p-Annealing


Fundamentally, an L_1 penalty has been used to induce sparsity in SAE features because it serves as a convex relaxation of the true sparsity measure, the L_0-norm. The L_1-norm is the convex hull of the L_0-norm, making it a tractable alternative for promoting sparsity <cit.>. However, the proxy loss function is not the same as directly optimizing for sparsity, leading to issues such as feature shrinkage <cit.> and potentially less sparse learned features. Unfortunately, the L_0-norm is non-differentiable and directly minimizing it is an NP-hard problem <cit.>, rendering it impractical for training.



In this work, we propose the use of nonconvex L_p^p-minimization, with p<1, as an alternative to the standard L_1 minimization in sparse autoencoders (SAEs). This approach has been successfully employed in compressive sensing and sparse recovery to achieve even sparser representations <cit.>. To perform this optimization, we introduce a method called p-annealing for training SAEs, based on the compressive sensing technique called p-continuation <cit.>. The key idea is to start with convex L_1-minimization through setting p=1 and progressively decrease the value of p during training, resulting in closer approximations of the true sparsity measure, L_0, as p approaches 0. We define the sparsity penalty for each batch x as a function of the current training step s:




    ℒ_sparse(𝐱, s) = λ_s 𝐟(𝐱) _p_s^p_s = λ_s ∑_if_i(𝐱)^p_s


In other words, the sparsity penalty will be a scaled L_p^p norm of the SAE feature activations, with p decreasing over time. At p=1, the L_p^p norm is equal to the L_1 norm, and as p → 0, the L_p^p norm limits to the L_0-norm, as lim_p→ 0∑_i f_i(𝐱)^p = ∑_i f_i(𝐱)^0.

The purpose of annealing p from 1 → 0 instead of starting from a fixed, low value for p is that the lower the p, the more concave (non-convex) the L_p^p norm is, increasing the likelihood of the training process getting stuck in local optima, which we have observed in initial experiments. Therefore, we aim to first arrive at a region of an optimum using the easier-to-train L_1 penalty and then gradually shift the loss function. This manifests as keeping p=1 for a certain number of steps and then starting decreasing p linearly down to p_end > 0 at the end of training. We set p_end=0.2.







  
Coefficient Annealing.
Changing the value of p changes the scale of the L_p^p norm. Without also adapting the coefficient λ, the strength of the sparsity penalty would vary too wildly across training. Empirically, we found that keeping a constant λ would lead to far too weak of a sparsity penalty for the larger p's at the start of training, making the process worse than simply training with a constant p from the beginning.

Consequently, we aim to adapt the coefficient λ such that the strength of the sparsity penalty is not changed significantly due to p updates. Formally, the update step is:

    λ_s+1λ_s∑_j=s-q+1^s∑_if_i(𝐱_𝐣)^p_s/∑_j=s-q+1^s∑_if_i(𝐱_𝐣)^p_s+1.


We keep a queue of the most recent q batches of feature activations mid-training and use them to calibrate the λ_s updates. Therefore, the strength of the sparsity penalty is kept locally constant.



  
Combining p-annealing with other SAEs.

Since the p-annealing method only modifies the L_1 terms in the loss function without affecting the SAE architecture, it is simple to combine p-annealing with other SAE modifications. This allows us to create the Gated-Annealed SAE method by combining the Gated SAE architecture and p-annealing. Concretely, we modify ℒ_gated (Equation <ref>) by replacing the sparsity term λReLU(π_gate(𝐱))_1 in  with λ_s ReLU(π_gate(𝐱)) _p_s^p_s. Our experiments showed that the optimum values for coefficients λ and λ_s differ.





§ RESULTS



In this section, we explore the performance of SAEs applied to language models trained on Othello and chess. Consistent with <cit.>, we find that interpretable SAE features typically track properties relative to the player whose turn it is (e.g. “my king is pinned” rather than “the white king is pinned”). To side-step subtleties arising from this, we only extract our activations from the token immediately preceding white's move.
Specifically, we consider SAEs trained on the residual stream activations after the sixth layer using the four methods from Section <ref> (see Table <ref> for additional hyperparameters). 
In addition to our metrics introduced above, we also make use of unsupervised metrics previously appearing in the literature <cit.>:

    
  * L_0 measures the average number of active SAE active features (i.e., positive activation) on a given input.
    
  * Loss recovered measures the change in model performance when replacing activations with the corresponding SAE reconstruction during a forward pass.
    This metric is quantified as (H_* - H_0)/(H_orig - H_0), where H_orig is the cross-entropy loss of the board game model for next-token prediction, H_* is the cross-entropy loss after substituting the model activation 𝐱 with its SAE reconstruction during the forward pass, and H_0 is the cross-entropy loss when zero-ablating 𝐱.

Our key takeaways are as follows.





  
SAE features can accurately reconstruct game boards. 
In general, we find that SAE features are effective at capturing board state information in both Othello and chess (see Table <ref>, Figure <ref>d and <ref>d). 
In contrast, SAEs trained on a model with random weights perform very poorly according to our metrics, showing that SAE performance is driven by identifying structure in the models' learned representation of game boards.
Nonetheless, SAEs do not match the performance of linear probes in terms of reconstructing the board state.
This performance gap suggests that SAEs do not capture all of the information encoded in the model's internal representations.



  
Standard SAEs trained with p-annealing perform on par with Gated SAEs.
We find that standard SAEs trained using p-annealing consistently perform better than those trained with a constant L_1 penalty (Equation <ref>), as measured by existing proxy metrics and in terms of improvement in coverage (see Figure <ref>a and <ref>a).
In fact, standard SAEs trained using p-annealing show a coverage score that is comparable to Gated SAEs trained without p-annealing. However, we find cases where our coverage metric disagrees with existing metrics. In Figure <ref>, for example, Gated SAEs perform achieve a higher loss recovered score than Standard SAEs trained using p-annealing. We emphasize that the training and inference of Gated SAEs is more computationally expensive, requiring 50% more compute per forward pass compared to Standard SAEs <cit.>. 













  
Coverage and board reconstruction reveal differences in SAE quality not captured by unsupervised metrics.
Our metrics reveal improvements in SAE performance that traditional proxy metrics fail to capture.
For example, we trained SAEs with hidden dimensions 4096 and 8192 (expansion factors of 8 and 16, respectively).  
We expect the SAEs with 8192 hidden dimensions to perform better since they have greater capacity.
However, we observe that they perform equally well according to prior unsupervised metrics (see Figures <ref> a, c and <ref> a, c). 
In contrast, our metrics reveal that SAEs with larger hidden dimensions are better. For the Standard architecture, this is reflected by the parallel lines (of purple diamonds) in Figures <ref> b, d and <ref> b, d.
Thus, our metrics are able to capture improvements from larger expansion factors.
In addition, we find that the performance of p-annealing closely resembles that of Gated SAEs when evaluated using standard proxy metrics; it demonstrates clear improvements under our proposed metrics.



  
Coverage and board reconstruction are consistent with existing metrics.
Figures <ref>, <ref>, and <ref> demonstrate that both coverage and board reconstruction metrics are optimal in the elbow region of the Pareto frontier. 
This region, where SAEs reconstruct internal activations efficiently with minimal features, also yielded the most coherent interpretations during our manual inspections. This provides precise, empirical validation to the common wisdom that SAEs in this region of the Pareto frontier are the best.










































§ LIMITATIONS


The proposed metrics for board reconstruction and coverage provide a more objective evaluation of SAE quality than previous subjective methods. 
Nevertheless, these metrics exhibit several limitations. 
Primarily, their applicability is confined to the chess and Othello domains, raising concerns about their generalizability to other domains or different models. 
Additionally, the set of BSPs that underpin these metrics is determined by researchers based on their domain knowledge. 
This approach may not encompass all pertinent features or strategic concepts, thus potentially overlooking essential aspects of model evaluation.
Developing comparable objective metrics for other domains, such as natural language processing, remains a significant challenge. 
Moreover, our current focus is on evaluating the quality of SAEs in terms of their ability to capture internal representations of the model. 
However, this does not directly address how these learned features could be utilized for downstream interpretability tasks.















§ RELATED WORK




  
Sparse dictionary learning. Since the nineties, dictionary learning <cit.>, sparse regression <cit.>, and later, sparse autoencoders <cit.> have been extensively studied in the machine learning and signal processing literature. The seminal work of <cit.> introduced the concept of sparse coding in neuroscience (see also <cit.>, building upon the earlier concept of sparse representations <cit.> and matching pursuit <cit.>. Subsequently, a series of works established the theoretical and algorithmic foundations of sparse dictionary learning <cit.>. Notably, <cit.> introduced LISTA, an unrolled version of ISTA <cit.> that learns the dictionary instead of having it fixed.

In parallel, autoencoders were introduced in machine learning to automatically learn data features and perform dimensionality reduction <cit.>. Inspired by sparse dictionary learning, sparse autoencoders <cit.> were proposed as an unsupervised learning model to build deep sparse hierarchical models of data, assuming a certain degree of sparsity in the hidden layer activations. Later, <cit.> generalized sparse autoencoders (SAEs) to convolutional SAEs. Although the theory for SAEs is less developed than that of dictionary learning with a fixed dictionary, some progress has been made in quantifying whether autoencoders can, indeed, do sparse coding, e.g., <cit.>.



  
Feature disentanglement using sparse autoencoders. 

The individual computational units of neural networks are often polysemantic, i.e., they respond to multiple seemingly unrelated inputs <cit.>.
<cit.> investigated this phenomenon and suggested that neural networks represent features in linear superposition, which allows them to represent more features than they have dimensions. 
Thus, in an internal representation of dimension n, a model can encode m ≫ n concepts as linear directions <cit.>, such that only a sparse subset of concepts are active across all inputs – a concept deeply related to the coherence of vectors <cit.> and to frame theory in general <cit.>. 

To identify these concepts, <cit.> used SAEs to perform dictionary learning on a one-layer transformer, identifying a large (overcomplete) basis of features. 
<cit.> applied SAEs to language models and demonstrated that dictionary features can be used to localize and edit model behavior. <cit.> proposed a scalable method to discover sparse feature circuits, as opposed to circuits consisting of polysemantic model components, and demonstrated that a human could change the generalization of a classifier by editing its feature circuit. Recently, <cit.> explored autoencoders for attention layer outputs. These works have benefited from a variety of open-source libraries for training SAEs for LLM interpretability <cit.>.












  
Improving and evaluating sparse autoencoders. 

Despite the success of SAEs at extracting human-interpretable features, they fail to perfectly reconstruct the activations <cit.>.
One challenge in the training of SAEs with an L_1 penalty is shrinkage (or 'feature suppression'); in addition to encouraging sparsity, an L_1 penalty encourages feature activations to be smaller than they would be otherwise. 
<cit.> approached this problem by fine-tuning the sparse autoencoder without a sparsity penalty. 
<cit.> and <cit.> explored alternative sparsity penalties to reduce feature suppression during training.
<cit.> introduced Gated SAEs, an architectural variation for the encoder which both addresses shrinkage and improves on the Pareto frontier of L_0 vs reconstruction error. Recently, <cit.> systematically evaluated the
scaling laws with respect to sparsity, autoencoder size, and language model size.


The goal of dictionary learning in machine learning is to produce human-interpretable features and capture the underlying model's computations <cit.>. 
However, quantitatively measuring interpretability is difficult and often involves manual inspection. 
Therefore, most existing work assesses the quality of SAEs along different proxy metrics: 
(1) The cross-entropy loss recovered, which reflects the degree to which the original loss of the language model can be recovered when replacing activations with the autoencoder predictions.
(2) The L_0-norm of feature activations 𝔼_z ∼𝒟 h(z) _0, measuring the number of activate features given an input <cit.>. <cit.> proposed to compare SAEs against supervised feature dictionaries in a natural language setting.
However, this requires a significant understanding of the model's internal computations and is thus not scalable.




§ CONCLUSION


Most SAE research has relied on proxy metrics such as loss recovered and L_0, or subjective manual evaluation of interpretability by examining top activations. However, proxy measures only serve as an estimate of interpretability, monosemantic nature, and comprehensiveness of the learned features, while manual evaluations depend on the researcher's domain knowledge and tend to be inconsistent.

Our work provides a new, more objective paradigm for evaluating the quality of an SAE methodology; coverage serves as a quantifiable measure of monosemanticity and quality of feature extraction, while board reconstruction serves as a quantifiable measure of the extent to which an SAE is exhaustively representing the information contained within the language model. Therefore, the optimal SAE methodology can be judged by whether it yields both high coverage and high board reconstruction.

Finally, we propose the p-annealing method, a modification to the SAE training paradigm that can be combined with other SAE methodologies and results in an improvement in both coverage and board reconstruction.



§ AUTHOR CONTRIBUTIONS

A.K. built and maintained our infrastructure for working with board-game models.
A.K., S.M., C.R., J.B., and L.S. designed the proposed metrics. 
B.W. performed initial experiments demonstrating the benefits of training SAEs with p < 1. B.W., C.M.V., and S.M. then proposed p-annealing, with B.W. leading the implementation and developing coefficient annealing.
The basic framework for our dictionary learning work was built and maintained by S.M. and C.R. The training algorithms studied were implemented by S.M., C.R., B.W., R.A., and J.B. 
R.A. trained the SAEs used in our experiments. 
A.K., C.R., and J.B. selected and implemented the BSPs.
A.K. and J.B. trained the linear probes. 
Many of the authors (including L.S., J.B., R.A.) did experiments applying traditional dictionary learning methods and exploring both toy problems and natural language settings, which helped build valuable intuition.
The manuscript was primarily drafted by A.K., B.W., C.R., R.A., J.B., C.M.V., and S.M., with extensive feedback and editing from all authors. 
D.B. suggested the original project idea.



§ ACKNOWLEDGMENTS

C.R. is supported by Manifund Regrants and AISST. 
L.R. is supported by the Long Term Future Fund.
S.M. is supported by an Open Philanthropy alignment grant.

The work reported here was performed in part by the University of Massachusetts Amherst Center for Data Science and the Center for Intelligent Information Retrieval, and in part using high performance computing equipment obtained under a grant from the Collaborative R&D Fund managed by the Massachusetts Technology Collaborative.


icml2024
















§ SPARSE AUTOENCODER TRAINING PARAMETERS


We used a single NVIDIA A100 GPU for training SAEs and experiments. It takes much less than 24 hours to train a single SAE on 300 million tokens. Given a trained SAE, our evaluation requires less than 5 minutes of computing time.





§ LIST OF BOARD STATE PROPERTIES


Table <ref> summarizes the high-level board state properties considered in 𝒢_strategy. The selection of concepts was inspired by <cit.>. The column indicated by # denotes the number of individual BSPs per concept. A single BSP per concept indicates we match this condition globally for any corresponding piece.





§ PERFORMANCE OF LINEAR PROBES AND SAES ON BOARD STATE PROPERTIES



In Figure <ref>, we present a mean coverage score over strategy board state properties 𝒢_BSP.
Properties within 𝒢_BSP vary significantly in complexity.
For example, queen detection can be inferred directly from the move history, while fork detection requires an accurate representation of the board state. 

Table <ref> shows that linear probe F1-score is below 0.95 for 6 out of 15 properties in 𝒢_BSP. This suggests that chess-GPT <cit.> does not represent these properties linearly. Additional experiments are required to determine whether the representation is present at all.

For the board state case, reconstruction is significantly higher than coverage. This is because there are many SAE features that are high precision classifiers for a configuration of squares, such as "white pawn on , white knight of ". In cases where coverage is higher than reconstruction (such as for ), it is because there are not many features that are over 95% precision for “there is a check move available” from which we can recover if there is an available check move. Coverage is significantly higher because there is at least one feature that has an F_1-score of 0.54 for , which may not have a precision greater than 95%.

Y>X


Y>X



 










§ MODEL INTERNAL BOARD STATE REPRESENTATION





 §.§ Othello Models

Previous research of Othello-playing language models found that the model learned a nonlinear model of the board state <cit.>. Further investigation found a closely related linear representation of the board when probing for "my color" vs. "opponent's color" rather than white vs. black <cit.>. Based on these findings, when measuring the state of the board in Othello, we represent squares as (, ) rather than (, ).




 §.§ Chess Models

Similar to Othello models, prior studies of chess-playing language models found the same property, where linear probes were only successful on the objective of the (, ) representation and were unsuccessful on the (, ) representation <cit.>. They measured board state at the location of every period in the Portable Game Notation (PGN) string, which indicates that it is white's turn to move and maintain the (, ) objective. Some characters in the PGN string contain little board state information as measured by linear probes, and there is not a clear ground truth board state part way through a move (e.g., the “” in “”). We follow these findings and measure the board state at every period in the PGN string.

When measuring chess piece locations, we do not measure pieces on their initial starting location, as this correlates with position in the PGN string. An SAE trained on residual stream activations after the first layer of the chess model (which contains very little board state information as measured by linear probes) obtains a board reconstruction F_1-score of 0.01 in this setting. If we also measure pieces on their initial starting location, the layer 1 SAE's F_1-score increases to 0.52, as the board can be mostly reconstructed in early game positions purely from the token's location in the PGN string. Masking the initial board state and blank squares decreases the F_1-score of the linear probe from 0.99 to 0.98.



 Paper Title: On the Perturbed States for Transformed Input-robust Reinforcement Learning.


Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
10.1109/ACCESS.2017.DOI






[1]School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea

Author : Preparation of Papers for IEEE TRANSACTIONS and JOURNALS
Author : Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

^∗ Equal contribution, Corresponding author: Chang D. Yoo (e-mail: cd_yoo@kaist.ac.kr).


§ ABSTRACT 
Reinforcement Learning (RL) agents demonstrating proficiency in a training environment exhibit vulnerability to adversarial perturbations in input observations during deployment. This underscores the importance of building a robust agent before its real-world deployment. To alleviate the challenging point, prior works focus on developing robust training-based procedures, encompassing efforts to fortify the deep neural network component's robustness or subject the agent to adversarial training against potent attacks. In this work, we propose a novel method referred to as Transformed Input-robust RL (TIRL), which explores another avenue to mitigate the impact of adversaries by employing input transformation-based defenses. Specifically, we introduce two principles for applying transformation-based defenses in learning robust RL agents: (1) autoencoder-styled denoising to reconstruct the original state and (2) bounded transformations (bit-depth reduction and vector quantization (VQ)) to achieve close transformed inputs. The transformations are applied to the state before feeding it into the policy network. Extensive experiments on multiple environments demonstrate that input transformation-based defenses, , VQ, defend against several adversaries in the state observations. The official code is available at <https://github.com/tunglm2203/tirl>



Transformed Input-robust Reinforcement Learning (TIRL), Bounded Transformation, Adversarial Attack, Autoencoder-styled Denoising, Bit-Depth Reduction, Vector Quantization (VQ).


=-15pt

On the Perturbed States for Transformed Input-robust Reinforcement Learning
    Tung M. Luu^∗, Haeyong Kang^∗, Tri Ton, Thanh Nguyen, and Chang D. Yoo
    August 5, 2024
===========================================================================





§ INTRODUCTION


Modern deep reinforcement learning (RL) agents <cit.> typically rely on deep neural networks (DNNs) as powerful function approximations. However, it has been discovered that even a well-trained RL agent can experience significant failures due to small adversarial perturbations in its input during deployment<cit.>. This vulnerability raises concerns about deploying such agents in safety-critical applications, such as autonomous driving <cit.>. Therefore, the development of techniques to help RL agents withstand adversarial attacks on state observations becomes crucial before their deployment in the real world <cit.>.

Several approaches have been suggested in the literature to safeguard against adversarial attacks on an agent's state observations. One avenue of research concentrates on improving the robustness of the agent's DNNs component, such as the policy network or Q-value network, by promoting properties like invariance and smoothness through regularization techniques <cit.>. These regularizers yield policy outputs that generate similar actions under bounded input perturbations. However, these defenses do not account for the dynamics of the RL environment and can be vulnerable to stronger attacks <cit.>. An alternative approach involves adversarial training of the RL agent, in which an adversary is introduced to perturb the agent's input as it takes actions within the environment. The episodes collected under these adversarial conditions are then utilized for training, leading to a more robust RL agent. Perturbations in this context can be induced from the policy or value function <cit.>, or more recently, from another RL-based adversary <cit.>. RL-based adversaries are typically acquired through online learning. Although training with RL-based adversaries can yield robust agents, it often requires additional samples and computations due to the training of the additional agent involved. Furthermore, online attacks may lead to several unsafe behaviors, posing a potential risk to the control system if adversarial training occurs in a physical environment rather than a simulated one.

The approaches above can be seen as robust training-based defenses, wherein the RL agent undergoes optimization to concurrently fulfill the requirements of an RL task and meet the robust performance criteria. Meanwhile, within the realm of image classification, an alternative strategy for countering adversarial attacks is recognized as input transformation-based defenses <cit.>. These defenses aim to eliminate adversarial perturbations from the input by applying transformations before presenting it to the classifier. The transformational process commonly relies on denoising techniques to purify perturbations <cit.> or employs image preprocessing techniques to mitigate the impact of adversaries <cit.>. Moreover, as this strategy exclusively modifies the input and not the model itself, input transformation-based defenses could benefit RL agents without substantially altering the underlying RL algorithms, serving as a plug-and-play module. Despite their potential to mitigate adversarial attacks, many of these transformations are tailored to image data <cit.> and may not easily extend to vector inputs such as low-dimensional states in continuous control systems. This paper explores a new input transformation-based defensive approach, referred to as Transformed Input-robust RL (TIRL), against adversarial attacks on state observations by applying input transformations suitable for low-dimensional states in continuous control tasks. To validate the effectiveness of the input transformations, various techniques are investigated, including bit-depth reduction, vector quantization, and autoencoder-style denoisers in continuous control tasks. Our findings indicate that these novel defenses exhibit notable resilience against adversarial attacks. Our contributions are summarized as follows:

    
  * We propose a new method, Transformed Input-robust RL (TIRL) with two principles, , (1) Bounded Transformation and (2) Autoencoder-styled Denoising, to use input transformations to counter adversarial attacks on state observations in deep RL.
    
  * Based on the two proposed principles, we explore the best transformation, , vector quantization (VQ), which is beneficial for the RL agent in continuous control tasks for the first time.
    
  * Extensive experiments on continuous control tasks from environments demonstrates that these defenses can be surprisingly effective against existing attacks.



§ RELATED WORK
 

Adversarial perturbed states. 
Since the discovery of adversarial examples in image classification <cit.>, corresponding vulnerabilities in deep reinforcement learning (RL) were first demonstrated in <cit.>, <cit.>. Huang et al. <cit.> evaluated the robustness of Deep Q-Network (DQN) agents by employing a weak Fast Gradient Sign Method (FGSM) attacker <cit.> to initiate attacks at every step. Lin et al. <cit.> focused on attacking specific steps within trajectories, using a planner to devise perturbations that steer the agent toward a target state. Pattanaik et al. <cit.> introduced a more potent attack strategy that leverages both the policy and the Q function, departing from perturbations based solely on the policy. Zhang et al. <cit.> later introduced the state-adversarial Markov decision process to formally define adversarial attacks on state observations, showing that optimal attacks can be learned as an RL problem. Building upon this foundation, Zhang et al. <cit.> and Sun et al. <cit.> introduced RL-based adversaries for black-box and white-box attacks, respectively. Although these attackers appear powerful, their effectiveness often relies on specific assumptions, such as requiring knowledge of the victim's policy <cit.>. Consequently, the introduced defenses <cit.> might be vulnerable to a more sophisticated, yet unknown, attacker. In contrast, we focus on developing defense strategies that are agnostic to the attacker's characteristics.

Adversarial training. 
The robustness of RL agents can be improved by training with adversarial samples <cit.>. In the Atari domain, Kos and Song <cit.> and Behzadan et al. <cit.> used weak attacks to generate perturbations for training DQN agents, achieving limited improvement. Alternatively, regularization-based methods have emerged to enhance the robustness of deep neural network components in RL algorithms. Zhang et al. <cit.> introduced a hinge loss regularizer to control the smoothness of the Q function under bounded perturbations, while Russo and Proutiere <cit.> proposed Lipschitz regularization. In continuous control tasks, Huang et al. <cit.> generated attacks using both the policy and Q function, then used the resulting trajectories for training. However, recent work <cit.> shows this approach is unreliable for enhancing robustness against new attacks. Smoothness regularization has also been proposed to strengthen policy model robustness in both online <cit.> and offline RL <cit.>. Based on the theory of optimal attacks from <cit.>, Zhang et al. <cit.> introduced a training paradigm that alternates between learning the RL agent and a black-box RL-based adversary to develop robustness. Similarly, Sun et al. <cit.> applied this approach with a white-box RL-based adversary, yielding a more robust RL agent. Training robust RL agents with additional RL-based adversaries requires extra samples and modifications to the underlying algorithms, making the process more complex and sample-intensive <cit.>. Our approach, however, focuses on modifying the agent's input rather than altering the algorithm. This simplifies training and offers greater flexibility when incorporating other RL algorithms, providing a more practical and effective solution for developing robust RL agents.

Input Transformation-based Defenses. 
Studies have used input transformations to mitigate adversarial attacks in image classification due to their simplicity and flexibility <cit.>. Traditional image processing techniques such as image cropping <cit.> and image rescaling <cit.> have been used to reduce the effectiveness of adversarial attacks. Powerful denoising models have also been employed to purify input perturbations. For example, Meng and Chen <cit.> used an autoencoder-style denoiser to reconstruct denoised images from randomly perturbed ones, while Samangouei et al. used GANs <cit.> for image reconstruction. Nie et al. <cit.> applied diffusion models <cit.> to refine adversarial examples before feeding them to classifiers. However, many of these transformations are specifically designed for image data <cit.> and may not easily adapt to vector inputs, such as low-dimensional states in continuous control systems. To alleviate those issues, this paper investigates a novel input transformation-based defense that mitigates the impact of adversarial attacks on state observations for RL agents.


§ PRELIMINARIES

This section presents the general reinforcement learning framework and an RL algorithm, Soft Actor-Critic (SAC), before introducing our novel input-transformation-based defenses. To evaluate the effectiveness of our method with SAC, we also describe test-time adversarial attacks.



 §.§ Reinforcement Learning

A reinforcement learning (RL) environment is modeled by a Markov decision process (MDP), defined as a tuple of (𝒮, 𝒜, p, r, γ), where 𝒮 is the state space, 𝒜 is the action space, p=Pr(s_t+1|s_t, a_t) is the transition probability distribution which is usually unknown, r: 𝒮×𝒜×𝒮→ℝ is the reward function, and γ∈ [0,1) is a discount factor. An agent executes actions based on a policy π: 𝒮→𝒜. In this work, we use the terms state and state observation to indicate the underlying state of the environment and the state perceived by the agent, respectively. If there are no attacks, both of them are identical. The target of RL agent is to maximize the expected discounted return 𝔼_π,p[∑_t=0^∞γ^t r(s_t,a_t,s_t+1)], which is the expected cumulative sum of rewards when following the policy π in the MDP. The state value function can measure this objective as follows:

    V^π(s):=𝔼_π,p[∑_t=0^∞γ^t r(s_t,a_t,s_t+1) | s_0=s],

or the state-action value function:

    Q^π(s,a) := 𝔼_π, p[∑_t=0^∞γ^tr(s_t,a_t,s_t+1)|s_0=s, a_0=a].





 §.§ Training Soft Actor-Critic
 
Soft Actor-Critic (SAC) <cit.> is an actor-critic off-policy RL algorithm widely used for solving continuous control tasks. This paper adopts SAC as the backbone of the proposed method for robustness evaluation due to its stability and sample efficiency in continuous control benchmarks. However, our method can also be easily extended to other algorithms, such as Proximal Policy Optimization (PPO) <cit.> and Twin Delayed Deep Deterministic Policy Gradient (TD3) <cit.>. 

SAC alternates between the soft policy evaluation and soft policy improvement steps to learn the optimal stochastic policy that maximizes the expected returns and entropy of actions. Denoting π_θ as a stochastic policy parameterized by θ. In soft policy evaluation step, SAC learns two critic networks, denoted as Q_ϕ_1 and Q_ϕ_2, parameterized by ϕ_1 and ϕ_2, respectively, by minimizing the soft Bellman residual:

    J_Q(ϕ_i) = 𝔼_(s, a, s', r)∼ℛ, a'∼π_θ[(Q_ϕ_i(s, a) - y)^2],

where, y = r + γ[min_i=1,2Q_ϕ̂_i(s',a') - αlogπ_θ(a'|s')], ℛ is the replay buffer, α is the temperature parameter. The parameter ϕ̂_i denotes the exponential moving average (EMA) of ϕ_i, which empirically improves training stability in off-policy RL algorithms <cit.>. In soft policy improvement step, the actor policy π_θ is optimized by maximizing the following objective:

    J_π(θ) = 𝔼_s∼ℛ, a∼π_θ[min_i=1,2Q_ϕ_i(s, a) - αlog (π_θ(a|s))],

where, the stochastic policy π_θ(a|s) is a parametric tanh-Gaussian that given the state s, an action a is sampled from tanh(μ(s) + σ(s)ϵ), with ϵ∼𝒩(0, 1), and μ and σ are parametric mean and standard deviation. Finally, the temperature α is learned with following objective:

    J(α) = 𝔼_s ∼ℛ, a ∼π_θ[-αlogπ_θ(a|s)-αℋ̅],

where, ℋ̅∈ℝ is the target entropy hyperparameter that policy tries to match, which is usually set to -|𝒜| in practice with 𝒜 is the action space.




 §.§ Test-time Adversarial Attacks
 



After training, the agent is deployed within its designated environment and operates according to a fixed pretrained policy π_θ, , the parameter θ is frozen. During this testing phase, an adversary can observe interactions between the victim agent and the environment, including states, actions, and rewards, to mislead the agent. However, the adversary does not possess knowledge of the environment's dynamics nor the capability to alter the environment directly. We focus on a typical state adversary that targets the manipulation of the state observation of the RL agent <cit.>. The MDP with a state adversary involved is formally defined by the state-adversarial Markov decision process (SA-MDP) introduced in <cit.>. Let Ψ: 𝒮→𝒮 represent a state adversary. At each time step, the adversary Ψ, equipped with a certain budget ϵ, is allowed to inject perturbations to state s returned by the environment before the agent perceives them, as illustrated in Fig. <ref>. It is important to note that the underlying states of the environment, returned rewards, and executed actions remain unaffected. This setting aligns with many realistic scenarios, such as sensor measurement errors, noise in sensory signals, or man-in-the-middle (MITM) attacks on a deep RL system. For instance, in robotic manipulation, an attacker could inject imperceptible noises into the camera, capturing an object without altering the actual object's location. As in many of the existing works investigating adversarial robustness of deep RL <cit.>, we focus on the ℓ_∞-norm attack, in which the adversary Ψ is constrained to perturb the state s∈𝒮 into another “neighboring” state s̃ within an ϵ-radius ℓ_∞-norm ball around s. Formally, the perturbed state s̃ = Ψ(s) ∈ℬ_ϵ(s), with ℬ_ϵ(s) = {s̃: s - s̃_∞≤ϵ}. The details of state adversaries are described in Section <ref>.


§ TRANSFORMED INPUT-ROBUST RL (TIRL)

In this section, we theoretically analyze the effectiveness of input transformation-based defenses in improving the robustness of RL agents. Subsequently, two principles are proposed to design transformation defenses for RL agents: 1) Bounded Transformation and 2) Autoencoder-styled Denoising. Based on these principles, we explore potential transformations to counter adversarial attacks.



 §.§ Effective Input Transformation Defenses

We first characterize the agent's performance changes before and after attacks by measuring the difference in value functions between the policy in MDP and SA-MDP. Let π be the policy, V^π(s) be its state value function in the regular MDP, and V^π∘Ψ(s) be its state value function in the SA-MDP with the adversary Ψ. Prior work <cit.> provides an upper bound on the difference between
value functions caused by an optimal adversary. Specifically, theorem 5 in <cit.> states that, under the optimal adversary Ψ^* in SA-MDP, the performance gap between V^π(s) and V^π∘Ψ^*(s) can be bounded as follows:

    max_s∈𝒮{V^π(s)-V^π∘Ψ^*(s)}≤κmax_s∈𝒮max_s̃∈ℬ_ϵ(s)D_TV(π(·|s), π(·|s̃))

where, D_TV(π(·|s), π(·|s̃)) represents the total variance distance between π(·|s) and π(·|s̃), κ is a constant independent of π, ℬ_ϵ(s) is defined as in Section <ref>, and π∘Ψ^* denotes the policy under perturbations, , π(a|Ψ^*(s)). The detailed proof for Eq. <ref>, which is based on tools developed in constrained policy optimization <cit.>, can be found in the Appendix of <cit.>.

Based on the upper bound between value functions, <cit.> proposed regularizing D_TV(π(·|s), π(·|s̃)) during policy training. In contrast, we focus on simple yet effective input transformations to reduce the performance gap. Let 𝒯 be a transformation that maps 𝒮→S̅, where S̅ is a set of all transformed states. This transformation can be applied during the training and testing phases or exclusively during the testing phase. Our following proposition establishes the connection between the performance gap and the input transformations.


Consider a K-Lipschitz continuous policy π parameterized by the Gaussian distribution with a constant variance independent of state for a regular MDP.
Let the corresponding value function be V^π(s). Define 𝒯_t and 𝒯_d as the transformations applied during training and testing, respectively. Under the optimal adversary Ψ^* in SA-MDP, for all s∈𝒮 we have:

    max_s∈𝒮{V^π∘𝒯_t(s)-V^π∘𝒯_d ∘Ψ^*(s)}≤ζmax_s∈𝒮max_s̃∈ℬ_ϵ(s)𝒯_t(s)-𝒯_d(s̃)_2,

where, ζ is a constant independent of π. Here, π∘𝒯_t and π∘𝒯_d ∘Ψ^* denote π(·|𝒯_t(s)) and π(·|𝒯_d(Ψ(s))), respectively.
 


Proof:
Based on Pinsker’s inequality, we first upper bound the total variance distance in RHS of Eq. (<ref>) by Kullback-Leibler (KL) divergence:

    D_TV(π(·|s), π(·|s̃)) ≤√(1/2D_KL(π(·|s) ∥π(·|s̃))).

Assuming the state space has dimension d, and given that the policy is Gaussian with constant independence variance, we denote π(·|s) ∼𝒩(μ_s,Σ_s) and π(·|s̃) ∼𝒩(μ_s̃,Σ_s̃), where μ_s, μ_s̃∈ℝ^d are produced by neural networks μ_θ(s), μ_θ(s̃), repectively, and Σ is a diagonal matrix independent of the state, i.e., Σ_s = Σ_s̃ = Σ. Using the K-Lipschitz assumption of the policy network, we can bound the KL divergence as follows:

    D_KL(π(·|s) ∥π(·|s̃))    = 1/2(log|Σ_s̃|/|Σ_s| - d + tr(Σ_s̃^-1Σ_s) 
            + (μ_s̃ - μ_s)^⊤Σ_s̃^-1(μ_s̃ - μ_s)) 
       ≤ Lμ_s̃ - μ_s^2_2 
       = Lμ_θ(s) - μ_θ(s̃)_2^2
       ≤ LKs - s̃_2^2.

The first inequality holds because Σ_s̃, Σ_s are positive, then ensuring there exists a positive constant L∈ℝ^+ that satisfies this inequality. The second inequality holds because the policy network is K-Lipschitz continuous. Let 𝒯_t and 𝒯_d be the transformations applied into state s and s̃, respectively. By combining Equation (<ref>), (<ref>), and (<ref>), we obtain:

    LHS of (<ref>)≤ζmax_s∈𝒮max_s̃∈ℬ_ϵ(s)𝒯_t(s) - 𝒯_d(s̃)_2

where ζ:=1/√(2)κ√(LK). The proof is complete. □ 



 §.§ Input Transformation Principles

Proposition <ref> says that the performance gap can be bounded by the difference between the “origin” state and ”perturbed” state in the transformed space, , 𝒯_t(s)-𝒯_d(s̃)_2. It suggests two distinct principles for reducing the performance gap through transformations. We summarize our two principles as follows: 

    
  * Bounded Transformation: If 𝒯_t and 𝒯_d are used for both training and testing, we can design 𝒯_t and 𝒯_d such that, after transformation, the difference between the state and its perturbed version is small, , max_s̃∈ℬ_ϵ(s)𝒯_t(s) - 𝒯_d(s̃)_2 is minimized.

    
  * Autoencoder-styled denoising: If 𝒯_t is the identity function, , 𝒯_t is not used during training, we need to design 𝒯_d such that it minimizes max_s̃∈ℬ_ϵ(s)s - 𝒯_d(s̃)_2. In other words, 𝒯_d needs to reconstruct origin state s from the perturbed state s̃. This naturally leads to the selection of denoisers for 𝒯_d.
 

In the autoencoder-styled denoising approach, we can theoretically maintain the performance under attacks, V^π∘𝒯_d∘Ψ^*(s), close to the natural performance V^π(s) as long as the denoiser can effectively reconstruct the original state s from the perturbed state s̃. However, this approach might be vulnerable to fully white-box gradient-based attacks, where adversaries have access to the entire policy's architecture, including its parameters and the denoiser. In the bounded transformation approach, the strength of these defenses lies in their non-differential nature, making them more resilient to white-box attacks. However, due to the use of transformations during training, it is not guaranteed that the natural performance under the transformation, V^π∘𝒯_t(s), is identical to V^π(s), as some information in the state may be lost. Nevertheless, as long as 𝒯_t retains the essential information from the original space, we can expect V^π∘𝒯_t(s) to be close to V^π(s), as empirically observed in our experiments. We present input transformation-based defenses following these two principles in the following parts. 





 §.§ Various Input Transformations

We investigate various input transformations to satisfy the two principles. In each principle, we can summarize as follows: (1) Bounded Transformation (bit-depth reduction and vector quantization (VQ) to achieve close transformations of origin and perturbed state and (2) Autoencoder-styled Denoising, i.e., VAED to reconstruct origin state.




  §.§.§ Bounded Transformation: Bit-Depth Reduction

 The bit-depth reduction (BDR) transformation, initially explored as a defense in image classification tasks <cit.>, is adopted here for our control tasks. The BDR performs uniform quantization for each input dimension, assigning each value to the nearest quantized point (Fig. <ref> top). If the adversary's budget ϵ is not too large, the state s and its perturbed state s̃ within the ϵ-radius can be assigned the same value after transformation (Fig. <ref> bottom). The BDR adheres to the bounded transformation principle, keeping 𝒯_t(s) - 𝒯_d(s̃)_2 small overall. The original design of BDR for images does not directly apply to continuous vector states due to varying value ranges and the continuous nature of the data. To address the limited flexibility, we redesigned BDR by partitioning each dimension into equally spaced bins, controlled by the bin width (bW) parameter, and rounding real-valued scalars to the nearest bin value. To balance robustness and natural performance, we adjust the bW parameter: increasing the bin width enhances defense against attacks but may reduce overall performance and vice versa.







  §.§.§ Bounded Transformation: Vector Quantization

We propose a new sparse input transformation using vector quantization (VQ) to discretize the entire state space into discrete vectors more strongly, as shown in Fig. <ref>. To perform VQ for state space, we employ the K-means algorithm. Specifically, given a set of state vectors {s_i}_i=1^N, the K-means algorithm partitions the N points into K clusters. Subsequently, each state is then represented by its nearest centroid in the codebook C=[c_1, c_2, …, c_K]. The training of K-means is achieved by minimizing the following objective:

    min_C, {p_i}∑_i=1^Ns_i - Cp_i_2^2

where p_i∈{0,1}^K is a one-hot vector with the value 1 indicating the index of the centroid nearest to data point s_i and K controls the codebook size. In the backward pass, the VQ module is treated as an identity function, referred to as straight-through gradient estimation <cit.>. The above objective is also convenient for learning the codebook online, which is suitable for concurrent training with RL algorithms since the data collected by the RL agent continuously evolves during training time. Unlike BDR, after transformation, VQ can produce a more “sparse” state space, thus significantly reducing the space of attacks, leading to the agent being more robust against adversarial attacks. 



  §.§.§ Autoencoder-styled Denoising

Training denoisers requires perturbed states, which are additionally collected by interacting with the environment under attack, making the process inefficient. To obtain stable denoising results, we only apply the autoencoder-styled denoiser <cit.> after completing the training of the RL agent. Moreover, sampling data under online adversarial attacks can result in potentially unsafe behaviors, especially in a physical system. For instance, in manipulation tasks using a robotic arm, collecting samples under attacks might lead to the breakage of objects. To mitigate these challenges, we leverage samples from the replay buffer collected during the training of RL agents for denoiser training. To generate perturbed states, we use Action Diff <cit.> and Min Q <cit.> attacks, as they only require a policy or Q-value network and do not necessitate interaction with the environments. This strategy helps reduce costs for acquiring additional data and avoids unsafe behaviors. In this approach, we must select an appropriate ϵ-radius for generating perturbations. If the ϵ is too large, the denoiser might be unable to reconstruct the original states.







The pseudo-code for applying BDR or VQ during agent training is shown in Algorithm <ref>. Note that we only apply these transformations to the input of the policy network. After training is complete, the transformations are applied to mitigate adversarial attacks, as demonstrated in Algorithm <ref>.




§ EXPERIMENTS

In this section, we evaluate the efficacy of our defenses across continuous control tasks. Section <ref> details the setup and adversaries used to evaluate the robustness. The experiment in Section <ref> evaluates performance in a gray-box setting, where adversaries can access the model architecture and parameters but are unaware of the applied transformations. The experiment in Section <ref> focuses on a white-box setting, where the adversaries are aware of the defense strategies.



 §.§ Experimental Setup




Environments and Baseline. We evaluate the effectiveness defenses on five environments, as commonly used in the literature <cit.>, including Walker2d, Hopper, Ant, Inverted Pendulum, and Reacher, as shown in Fig. <ref>. We use SAC as a base RL algorithm with the implementation from <cit.>, the hyperparameters for the five environments similar to those in <cit.>. When applying the input transformations, we maintain the same hyperparameters for the underlying RL algorithm and only conduct a grid search for the transformation's hyperparameter. We run Walker2d and Ant 3 × 10^6 steps, Hopper 10^6 steps, and Inverted Pendulum and Reacher 2× 10^5 steps. 

Adversaries. We attack the trained policies with multiple existing attack methods, as used in <cit.>, including Random, Action Diff, Min Q, Robust Sarsa (RS), and Policy Adversarial Actor Director (PA-AD). Specifically, given an attack budget ϵ, a state s is adversarially perturbed into state s̃ by adversaries as follows:

    
  * Random uniformly samples perturbed states within an ϵ-radius ℓ_∞-norm ball.
    
  * Action Diff <cit.> is an attack method relied on the agent's policy. It directly finds the perturbed states within an ϵ-radius ℓ_∞-norm ball to satisfy: max_s̃∈ℬ_ϵ(s)D_KL(π_θ(·|s)||π_θ(·|s̃)) with D_KL is the Kullback-Leibler divergence.
    
  * Min Q <cit.> relies on the agent's policy and Q function to perform attacks. It generates the perturbed states within an ϵ-radius ℓ_∞-norm ball to satisfy: min_s̃∈ℬ_ϵ(s)Q_ϕ(s, π_θ(s̃))
    
  * Robust Sarsa (RS) <cit.> is performed similar to Min Q attack with a robust action-value function.
    
  * Policy Adversarial Actor Director (PA-AD) <cit.> is currently the most powerful attack on RL agents, as it learns the optimal adversary using RL. Specifically, given a pretrained policy π_θ, PA-AD learns an adversarial policy π_adv, which takes the state s as input to generate the adversarial state s̃. The adversarial policy is trained using Proximal Policy Optimization (PPO) <cit.>.


Generally, the strength of attackers is ranked in ascending order as follows: Random < Action Diff < Min Q < RS < PA-AD. For Action Diff, Min Q, and RS, we use projected gradient descent (PGD) to generate the optimal perturbed state, as in <cit.>. For example, the perturbed state s̃ is obtained by PGD in Action Diff attack as follows:

    s̃^k+1 = s̃^k + η proj [∇_s̃^k D_KL(π_θ(·|s)||π_θ(·|s̃^k))]

where s̃^0 = s, k = 0, …, H - 1, proj[.] is a projection to ℬ_ϵ(s), η is the step size, and H is the number of iterations. Through experiments, we use 10-step PGD for optimization for these attacks (, H=10) with a step size η = 0.1. For PA-AD, we follow the setup in <cit.> to evaluate pre-trained SAC policies. The architecture of the adversarial policy is similar to the SAC policy, consisting of two 256-dimensional hidden layers followed by  activations, except for the last layer, which uses  activation. The ϵ-radius of attackers for each environment is shown in the tables. 





Input Transformations for Robust RL 

    
  * TIRL-BDR: We apply the BDR transformation to the input of the policy network of SAC, while leaving the input of the Q networks unchanged. Note that BDR is applied in both training (in a regular environment) and testing (with adversaries). The hyperparameter bW is searched in {0.1, 0.25, 0.4, 0.5} for Reacher, and {0.05, 0.1, 0.15, 0.2} for the other environments.
    
    
  * TIRL-VQ: Similar to BDR, the VQ transformation is applied only to the input of the policy network and is used during both training and testing. The codebook is learned concurrently with the agent while training by optimizing Eq.( <ref>); during testing, the learned codebook is fixed. The codebook size K is searched in {2× 10^5, 5× 10^5, 10^6} for Ant, Hopper, and Walker2d, and {5× 10^4, 1× 10^5, 2× 10^5, 3× 10^5, 5× 10^5} for the other environments.
    
    
  * TIRL-AED: We use an autoencoder-based denoiser to reconstruct an original state from a perturbed state. To train the denoiser, we collect a set of N states {s_i}_i=1^N from a pretrained policy's replay buffer, then use Min Q attacks to generate perturbed states  {s̃_i}_i=1^N. The autoencoder is trained to reconstruct the original states by optimizing the mean square error: s - D_ω_2(E_ω_1(s̃)_2^2, where E_ω_1 and D_ω_2 are the encoder and decoder, respectively. Note that the denoiser is only used to denoise the perturbed inputs of the policy network during testing. We use the Adam optimizer <cit.> for training the denoiser, with a learning rate of 3× 10^-4. For the Min Q attacker, we search the ϵ-radius in {0.25, 0.3, 0.35, 0.4} for Inverted Pendulum, and {0.05, 0.1, 0.15, 0.2, 0.25} for the other environments.
    
    
  * TIRL-VAED: We follow a similar setting as TIRL-AED, except for the training objective. We utilize the variational autoencoder <cit.> objective: -β D_KL(E_ω_1(z|s̃)  p(z))+𝔼_E_ω_1(z|s̃)[log D_ω_2(s|z) ], where p(z) is the standard normal distribution, and β is set to 1× 10^-4. We also search the ϵ-radius in the same range as TIRL-AED.

We summarize the best hyperparameters for transformations across environments in the Table <ref>.





 §.§ Evaluation with Gray-box Attacks

In this scenario, adversaries can access the policy architecture and its parameters but not the applied transformations. When generating perturbations, the clean state is fed directly into the policy network, bypassing transformations. For example, in the Action Diff attack, the adversarial state s̃ is generated using the formula: s̃^k+1 = s̃^k + η proj [∇_s̃^k D_KL(π_θ(·|s)||π_θ(·|s̃^k)], ignoring any transformations applied during policy training. Min Q and RS are analogous to Action Diff. In RL-based attacks like PA-AD, the transformation is also excluded from the pre-trained policy when learning the adversarial policy.

The performance of both the vanilla SAC agent and the TIRL with various transformations under different attacks is presented in Table <ref>. The results illustrate the effectiveness of the investigated transformations in mitigating the impact of attacks. The TIRL-BDR improves performance by 31% over the vanilla SAC across five environments, achieving the highest robustness in the Walker2d and Inverted Pendulum environments at 45% and 44%, respectively. The TIRL-VQ significantly enhances robustness by 55% over the vanilla SAC, with the most substantial improvements observed in the Ant (117%) and Hopper (56%) environment among transformations. In the Reacher environment, the TIRL-VAED excels with the highest performance, reaching 40%. When comparing the TIRL-AED and TIRL-VAED, both demonstrate comparable performance. Overall, in the gray-box setting, the TIRL-VQ transformation appears to be the most effective among the transformations considered. However, it is essential to note that applying VQ impacts natural returns due to the loss of information in quantized states. This highlights the trade-off between balancing robustness and maintaining natural performance.










 §.§ Evaluation with White-box Attacks

While the previous experiment demonstrates that input transformation defenses are relatively effective in mitigating adversarial attacks, we now conduct experiments using white-box attacks in a more challenging setting. In this scenario, adversaries can access the policy architecture, parameters, and applied transformations. Specifically, in the Action Diff, Min Q, and RS attacks, adversaries are allowed to backpropagate through the transformations while generating perturbations. For example, the Action Diff generates an adversarial state as follows: s̃^k+1 = s̃^k + η proj [∇_s̃^k D_KL(π_θ(·|𝒯(s))||π_θ(·|𝒯(s̃^k))]. Similarly, RL-based adversaries can use the complete pretrained policy, including its transformations, in RL-based attacks to train the adversarial policy.

Table <ref> presents the performance results in this scenario. The results show that TIRL-VQ maintains the highest robustness among all transformations, achieving a 50% improvement over vanilla SAC.  Although TIRL-BDR's improvement decreases, it still partially mitigates the attacks' effects, showing a 20% improvement across five environments. In contrast, the autoencoder-style denoising approach exhibits a performance decline in this setting, resulting in outcomes similar to vanilla SAC. This decline is because the denoisers are composed of deep neural network components, which adversaries can easily exploit to generate more robust perturbations. Additionally, we observe that TIRL-VQ performs best in 4 out of 5 environments in the white-box setting, compared to 2 out of 5 environments in the gray-box setting. This indicates that VQ's performance hardly deteriorates as the adversary's strength increases. 



 §.§ Comparison with prior work


In this experiment, we compare our defenses with prior robust training-based methods, including SAC-SA <cit.> and PA-ATLA <cit.>. We conduct experiments on the Ant, Hopper, and Walker2d environments, which were also used in previous works. The results are presented in Table <ref>. In the Ant environment, we observe that SAC-TIRL-VQ performs comparably to PA-ATLA and better than SAC-SA. In the Walker2d environment, SAC-TIRL-VQ and TIRL-BDR are comparable to SAC-SA and outperform PA-ATLA. Across all three environments, SAC-TIRL-VQ achieves an average return of 4216.1, SAC-SA achieves an average return of 4196.1, and PA-ATLA achieves an average return of 3823.6. Overall, SAC-TIRL-VQ shows comparable improvement to SAC-SA across environments, with an approximate 52% improvement compared to vanilla SAC. It is worth noting that, unlike SAC-SA and PA-ATLA, we do not use adversarial examples during training. Moreover, our method only modifies the model's input, making it possible to combine with robust training-based defenses for further robustness enhancement. We leave this investigation for future work.



 §.§ Ablation experiments

In this section, we conduct an ablation study on hyperparameters for TIRL-BDR and TIRL-VQ transformations. We first train the agents with different hyperparameters. Subsequently, with each pretrained model, we evaluate robustness as a function of adversary strength (, the ϵ-radius) for each of the three attackers: Random, Action Diff, and Min Q. 

Fig. <ref> illustrates the robustness of TIRL-BDR in the Hopper environment with varying values of bin width (bW). As observed, if bW is small (, 0.05), it resembles the robustness of vanilla SAC, particularly noticeable in Action Diff and Min Q attacks. This is because a smaller bW essentially reflects the original state space, thus having less impact on countering perturbations caused by attacks. However, a value that is too large, bW, dramatically drops natural performance due to inaccurate representations.

Fig. <ref> showcases the results for various codebook sizes of the VQ transformation in the Reacher environment. Across all considered values, robustness in policy performance remains consistently strong. However, our primary objective is to identify smaller codebook sizes that yield higher natural rewards. This choice is motivated by the increasing computational cost of larger codebook sizes in the VQ transformation. Notably, in our experiments on the Hopper environment, the wall-clock time for training SAC-VQ with K=5× 10^5 is approximately 2.5 times longer compared to SAC-BDR. This cost differential becomes even more pronounced in higher-dimensional environments. Consequently, when selecting hyperparameters for VQ, we recommend commencing with smaller values and incrementally adjusting them until achieving a satisfactory balance between robustness and computational efficiency. 



 §.§ Computational Cost Comparison



We compare the training and testing times for different transformations on a single machine equipped with a Tesla V100 16GB GPU. Note that TIRL-AED and TIRL-VAED are trained after completing the training of RL agents, which takes approximately 15-20 minutes for training the denoiser. The results are presented in Table <ref>. Although TIRL-VQ is effective in mitigating adversarial attacks, its training time with SAC is longer, primarily due to the K-means algorithm. This poses a limitation when extending our method to high-dimensional states, such as images. Future investigations could explore more advanced variants of VQ to optimize efficiency further and expand the scope of potential applications.



§ CONCLUSIONS

Reinforcement Learning (RL) agents demonstrating proficiency in a training environment exhibit vulnerability to adversarial perturbations in input observations during deployment. This underscores the importance of building a robust agent before its real-world deployment. To alleviate the challenging point, prior works focus on developing robust training-based procedures, encompassing efforts to fortify the deep neural network component's robustness or subject the agent to adversarial training against potent attacks. In this work, we proposed a novel method referred to as Transformed Input-robust RL (TIRL), which explores another avenue to mitigate the impact of adversaries by employing input transformation-based defenses. Specifically, we introduced two principles for applying transformation-based defenses in learning robust RL agents: (1) autoencoder-styled denoising to reconstruct the original state and (2) bounded transformations (bit-depth reduction and vector quantization (VQ)) to achieve close transformed inputs. The transformations are applied to the state before feeding them into the policy network. Extensive experiments on multiple environments demonstrated that input transformation-based defenses, , VQ, defend against several adversaries in the state observations.

In this paper, we focus primarily on low-dimensional environments due to the high computational cost of applying VQ to high-dimensional environments like those in Atari games. Future research can address this challenge by developing more advanced and efficient VQ techniques. Additionally, our proposed TIRL method holds the potential to synergistically complement other robust training-based defenses, significantly enhancing the overall robustness of RL agents. This direction opens exciting opportunities for developing more resilient RL systems capable of operating effectively in complex and adversarial environments.






ieeetr



 Paper Title: Vera Verto: Multimodal Hijacking Attack.









Vera Verto: Multimodal Hijacking Attack
    
Minxing Zhang1  Ahmed Salem2  Michael Backes1  Yang Zhang1




1CISPA Helmholtz Center for Information Security    
2Microsoft

    
====================================================================================================================================




§ ABSTRACT 
The increasing cost of training machine learning (ML) models has led to the inclusion of new parties to the training pipeline, such as users who contribute training data and companies that provide computing resources.
This involvement of such new parties in the ML training process has introduced new attack surfaces for an adversary to exploit.
A recent attack in this domain is the model hijacking attack, whereby an adversary hijacks a victim model to implement their own – possibly malicious – hijacking tasks.
However, the scope of the model hijacking attack is so far limited to the homogeneous-modality tasks.
In this paper, we transform the model hijacking attack into a more general multimodal setting, where the hijacking and original tasks are performed on data of different modalities.
Specifically, we focus on the setting where an adversary implements a natural language processing (NLP) hijacking task into an image classification model.
To mount the attack, we propose a novel encoder-decoder based framework, namely the Blender, which relies on advanced image and language models.
Experimental results show that our modal hijacking attack achieves strong performances in different settings.
For instance, our attack achieves 94%, 94%, and 95% attack success rate when using the Sogou news dataset to hijack STL10, CIFAR-10, and MNIST classifiers.
Our code is available https://github.com/minxingzhang/ModalHijackinghere.[Vera Verto is a magical transformation incantation used in the fictional wizarding world of J.K. Rowling's Harry Potter series, which transforms NLP sentences into CV images in the paper.]







§ INTRODUCTION



As a critical component of various applications, machine learning (ML) models have been increasingly expensive to train.
Hence, the training of ML models has transformed gradually into a joint process, e.g., new parties are included in the training of the model either by providing data or computational resources.
However, the involvement of these new parties has created new attack surfaces against ML models, e.g., poisoning and backdoor attacks <cit.>.

Recently, the model hijacking attack has been proposed by Salem et al. in this domain <cit.>, where the adversary is able to stealthily implement their own hijacking task into a target victim model.
Concretely, assuming the same ability as in poisoning attacks, the adversary first camouflages their own hijacking dataset for stealthiness to have a similar visual appearance as the target model's training dataset.
Then the adversary poisons the training dataset of the target model with their camouflaged hijacking dataset.
This attack could induce two different risks.
The first one is about accountability which is the main threat for hijacking attacks, where the model owner can be framed by the adversary to perform illegal or unethical tasks without knowing.
For instance, a completely different classification of some unethical tasks could be mounted in a benign classifier.
The second one is parasitic computing, where the model owner pays the model maintenance costs, while the adversary uses/offers it for their own application/service for free.
For instance, one NVIDIA V100 GPU chip costs $0.992-$2.48 per hour on the Google Cloud Platform.[<https://cloud.google.com/compute/all-pricing#gpus>]
On the other hand, the model hijacking technique can also be adapted to compress models, i.e., training a single model for multiple tasks.



 §.§ Our Contributions



Motivation
The model hijacking attack mandates that the hijacking and original tasks have the same modality, which limits the applicable domains to computer vision (CV) related tasks.
However, ML has achieved great success in many domains, e.g., the multiple available translators such as DeepL and Google Translate, and the different face detectors on social media platforms.
Hence, relaxing this assumption will significantly increase the risks of the model hijacking attack, as the adversary can now target models with different modalities, i.e., more target models exist for the adversary to perform their attack.



In this paper, therefore, we transform the model hijacking attack into a more general multimodal setting, i.e., implementing a hijacking task in a completely different domain.
More concretely, the adversary can implement an NLP hijacking task by a CV target model, as illustrated in <ref>.
For short, we refer to our attack as the modal hijacking attack.
The goal of our modal hijacking attack can be summarized as two aspects:
(a) The effectiveness corresponds to the successful implementations of hijacking attacks in victim models;
(b) The stealthiness corresponds to victim models not realizing that they are hijacked.

Our modal hijacking attack follows the same assumption as the attacks requiring poisoning <cit.>, i.e., the adversary is only able to poison the training dataset without any access to the target model's architecture or hyperparameters.
Our modal hijacking attack would increase the risks of accountability and parasitic computing.
Concretely, our attack could pose a more severe threat to accountability since multiple modalities are involved instead of a single one.
For instance, by model hijacking attacks, a benign classifier would be forced to claim some unethical statements are valid, where the scope of feasible hijacking tasks is extended.
Besides, the threat of parasitic computing can be more significant when NLP tasks are involved.
This is because the establishment of NLP models normally requires an enormous training budget.[<https://research.aimultiple.com/large-language-model-training/>]
And the risk of parasitic computing will get further increased when our Blender (introduced in the following soon) is reusable.

Modal Hijacking Attack
To perform the modal hijacking attack, the adversary needs to transform the NLP-based hijacking dataset into the victim model's CV-based original dataset.
To this end, we propose the Blender, a novel encoder-decoder-based model which integrates a language model, i.e., BERT <cit.> and multiple CNN models.
The Blender integrates two losses, i.e., visual and semantic losses, 
to fuse both the hijacking and original inputs.
Then the crafted sample has a similar visual appearance to the original inputs but maintains the semantic features of the hijacking one, as shown in <ref>.
A successful modal hijacking attack should enable the target victim model to preserve its utility, i.e., has the same performance on the original CV task (“Stealthiness”), while performing the hijacking NLP task with high accuracy (“Effectiveness”).

Different from the previous work, our modal hijacking attack expands the scope from a single type of data to a multimodal setting.
The challenge that stems from the change is the needed transformation from a discrete domain (NLP) to a continuous one (CV), in which there exists a comprehension gap.
We believe this transformation is not trivial, as demonstrated in <ref>.
Meanwhile, our modal hijacking attack is more general, i.e., the Blender can be applied with different hijacking and original datasets.
Besides, the reusability property provides a cheaper option for the adversary to hijack target models as shown later in <ref>.
To the best of our knowledge, the modal hijacking attack is the first work to combine different data modalities, which increases the capability and flexibility for hijacking attacks.
Moreover, this work can encourage the exploration of the applicability of performing hijacking attacks for different modalities (which can also be with the benign aim of model compression).

Evaluation
To evaluate our modal hijacking attack, we use two NLP datasets <cit.>, namely Yelp Review (Yelp) and Sogou News (Sogou), and three CV datasets, i.e., MNIST <cit.>, CIFAR-10 <cit.>, and STL-10 <cit.>.
We extensively evaluate the different setups for our modal hijacking attack.
Our results show that our modal hijacking attack can achieve strong performances with respect to both the attack success rate and the victim's model utility.
For instance, when victim models trained on MNIST, CIFAR-10, and STL-10 datasets are hijacked by the Yelp (Sogou) datasets, our modal hijacking attack achieves an attack success rate of 65% (94%), 68% (94%), and 65% (95%), respectively.
Meanwhile, the victim models' utility is not jeopardized, i.e., our modal hijacking achieves the utility of 99% (99%), 93% (93%), and 93% (92%), respectively, which is less than 2% drop compared to clean models.
Moreover, we show the generalizability of our modal hijacking attack by evaluating it against different setups, e.g., different models to construct the Blender and target model.
Finally, we involve two possible defenses to evaluate the modal hijacking attack.

In addition, we discuss the advantages of our work from the aspects of practical usage, stealthiness, and computational efficiency in <ref> for a better understanding.




§ BACKGROUND AND RELATED WORKS



Training Time Attacks
The inclusion of new parties in the training pipeline for ML has induced a new attack vector.
Adversaries can utilize this to interfere with the training of the victim model.
These attacks are referred to as training time attacks.
One of the most famous training time attacks is data poisoning <cit.>.
This class of attacks allows the adversary to insert malicious samples into the victim model's training data.
One widespread target for the poisoning attacks is to jeopardize the model's utility, i.e., to make the training of the victim model fail, which is different from our modal hijacking attack.

Another related training time attack is the backdoor attack <cit.>.
In this attack, the adversary takes a further step and tries to associate a malicious behavior – of the target model – with a trigger, e.g., a white square on the corner of the input.
A successful backdoor attack results in a victim model that behaves benignly on clean inputs; while predicting a specific label when queried by backdoored data, i.e., inputs with triggers, but triggers are not as flexible as our modal hijacking attack.

Testing Time Attack
A similar attack is adversarial reprogramming <cit.>.
In this attack, the adversary also tries to perform their own task using a victim model.
However, this is a testing time attack, i.e., the adversary only accesses the model after its training.
Unlike our hijacking attack, this attack requires assumptions on the target model such as white-box access.




§ MODAL HIJACKING ATTACK





 §.§ Threat Model



Adversary's Goals
The main goals of our attack could be summarized as:
(a) The effectiveness guarantees that victim models can successfully conduct our hijacking tasks, which will be evaluated by the metric of attack success rate;
(b) The stealthiness ensures that victim models will not realize the existence of hijacking attacks, which will be measured from multi-faceted aspects, e.g., the metric of utility, the visual appearances of fused images, and the poisoning rate.
The results and analyses in <ref> and <ref> show the superiority of our modal hijacking attack.

In this paper, we generalize hijacking attacks to a multi-modal setting, which represents a more practical scenario in the real world.
It might be hard for the adversary to find a victim model of the exact same task domain.
Thus, the previous model hijacking attack, which only focuses on the same modality, limits the applications of hijacking attacks.
However, the transformation to a multi-modal setting is challenging.
Specifically, NLP tasks are located in a discrete domain while CV tasks are in a continuous one.
In that case, how to understand the discrete information in a continuous form is not trivial.
To address this challenge, we propose the modal hijacking attack which enables the adversary to hijack the victim model of CV tasks by NLP tasks, enhancing the applicable scope and practical usage of hijacking attacks.

Adversary's Knowledge
We follow the same assumption as that widely used in poisoning <cit.> and backdoor attacks <cit.>, i.e., we only assume the ability to poison the training dataset of the target model.
Besides the involvement of users who contribute training data and companies that provide computing resources, another possible approach under this assumption is intrusion techniques <cit.>.
However, when aiming at a malicious goal (e.g. frame-up or parasitic computing), no matter in which cases, fewer poisons indicate a more practical scenario and also more stealthiness, which will be further discussed in <ref>.
On the other hand, when a model owner conducts model compression, the target model's training data is of course accessible.
In short, our modal hijacking attack does not require any extra information about the target model architecture or hyperparameters.

Moreover, we assume the adversary to have a container image dataset to fuse with the hijacking one.
This container dataset does not have to follow the same distribution as the original training dataset of the victim model.
However, the adversary can construct it relying on their preference for the visual appearance of the Blender's outputs, i.e., the fused dataset.
In addition, as victim models are used to perform the hijacking task, our modal hijacking attack assumes that the number of labels of the original dataset is at least equal to the hijacking dataset's one.



 §.§ Dataset Terminologies



The modal hijacking attack uses four different datasets which are defined now for clarity:
First, the Original Dataset () is the victim model's training dataset for training the original task;
Second, the Hijacking Dataset () is the adversary's training dataset for training the hijacking task;
Third, the Container Dataset () is a set of images the adversary constructs/collects to fuse with the hijacking dataset samples;
Finally, the Fused Dataset () is the container dataset after being fused with the hijacking one.



 §.§ Blender Design



In general, the Blender aims to generate a fused dataset, which is used to hijack the victim model.
This is performed by fusing the hijacking NLP sentences with the container CV images.
We first present the design of our Blender, then how to deploy it.

Design
To fuse the text-hijacking dataset with an image container one, we first extract the hijacking dataset's features, following state-of-the-art works by using a language model <cit.>.
Then to construct the fused dataset, we first try the naive approach of building an adapter, which is a CNN, to resize the NLP features to the size of the victim model's input.
However, this approach does not perform well for some datasets as shown later in <ref>.
Moreover, using this naive approach results in random-looking images as illustrated in <ref>, which can be easily detected.
To circumvent the limitations of the naive approach, we propose our encoder-decoder-based model, i.e., Blender.
More concretely, the Blender consists of an NLP feature extractor , i.e., a language model, an adapter , two encoders  and , and a decoder .
Further, to improve the computational efficiency, we relax the requirement from using double encoders to a single one .

Another design decision we make is to use the complete embeddings of the hijacking sentence instead of only the last “[cls]” token.
As shown later in <ref>, using all of the embeddings significantly improves the performance of the modal hijacking attack.

Our final design decision is to fine-tune the NLP feature extractor using the hijacking dataset before deployment.
This will allow the Blender to better understand the hijacking dataset, thus improving its ability to extract semantic information and integrate it into container images.
The performance gains of this step are evaluated in <ref>.
It's worth noting that this step does not require any additional assumptions as the adversary, who owns the hijacking dataset, can control the NLP feature extractor independent of the target victim model.
Additionally, the cost of fine-tuning is not a major concern for the adversary, as it can be mitigated by reusing the Blender as demonstrated in <ref>.

Operation
We first explain how double-encoder Blender operates.
The Blender uses the NLP feature extractor  to extract the features/embeddings of the text hijacking input ∈.
Next, these features are input to the adapter  to preprocess before being input to the first encoder .
In parallel, the container image ∈ is input to the second encoder .
Next, the outputs of both encoders are concatenated together and input to the decoder .
Finally, the decoder constructs the output fused image , which has the visual appearance of the container image , while having the features of the text one .
More formally,

    (((())) || ()) = ,

where || is the concatenation operator.

In the single-encoder setting, the container image  is directly concatenated with the adapter result (·), which is then sent to the encoder .
This process is formulated as

    (((())|| ) ) = ,


Training
To train the Blender, we use two losses, namely the visual and semantic losses.

Visual Loss:
The visual loss () is responsible for forcing the fused image to have a similar look to the container one.
To accomplish this, we utilize the mean squared error (MSE) to measure the pixel-wise difference between the fused and container inputs, i.e., = || - ||_2^2.

Semantic Loss:
The semantic loss  is designed to fuse the NLP features in the container image.
Similar to the visual loss, we use MSE for the semantic loss too.
However, the MSE here is calculated between the features extracted from the text input and the ones extracted from the fused image.
Feature extraction here is performed with different feature extractors according to the input type, i.e., the text/image feature extractor (/) is used to extract the features of /.
Since the MSE expects the same sizes for both inputs, we further process the adapter's output with a linear layer () to adjust its size to match the CV ones.
More formally,

    = ||((())) - ()||_2^2.


The Blender is trained as the following 4 steps:

(1) The adversary first constructs their container dataset .
The only requirement for this dataset is to be an image dataset.
Ideally, this dataset has a similar visual appearance as the original dataset , to make the modal hijacking attack more stealthy.
However, the adversary can construct this dataset as they desire.

(2) Second, every sentence in the hijacking dataset is randomly mapped to an image in the container dataset.
This mapping does not have to be one-to-one, i.e., a single container image can be associated with multiple sentences.

(3) Next, (sentence, image) pairs are processed by our Blender as previously presented; both semantic and visual losses are calculated.

(4) Finally, both losses are added and the Blender is updated accordingly, i.e., = argmin_(+), where  is the parameters of the Blender and the linear layer .

It is important to mention that the CV feature extractor  and the linear layer  are only needed when training the Blender, then they can be discarded.



 §.§ Attack Deployment



Our modal hijacking attack is executed in two phases.

Poisoning the Victim Model
The adversary starts by training the Blender as previously presented in <ref>.
Next, they use the Blender to create the fused dataset, i.e., by fusing the container and hijacking datasets.
They then perform a label mapping between the original and hijacking datasets to decide the labels of the fused dataset, i.e., by unidirectionally one-to-one mapping the hijacking samples' labels to the fused dataset's ones.
Finally, the fused dataset with its labels is used to poison the victim model.

Inference Time
After training the victim model, the adversary executes the modal hijacking attack on a target hijacking input  as presented in <ref>.
As the figure shows, the adversary first samples a container image and then uses the trained Blender to fuse it with the hijacking input and create the fused image.
The adversary then queries the fused image to the victim model and receives the output label.
Finally, they map the received label back to its corresponding one in the hijacking dataset.



 §.§ Advantages



There are 3 main advantages of our multimodal hijacking attack over the previous work <cit.>:



- Hijacking attacks are generalized from homogeneous to heterogeneous modalities, which improves practical usage.



- A single encoder in our Blender can support two completely different modalities in our attack, which largely improves the computational efficiency.



- Our poisons (i.e., fused images) look much more natural, where the hijacking content is not exposed.
This advantage enhances our attack's stealthiness.




§ EVALUATION





 §.§ Dataset Description



CV Datasets
We use three commonly-used benchmark datasets in our evaluation, i.e., MNIST, CIFAR-10, and STL-10.
MNIST is a handwriting digits datasets, which contains 70,000 28×28 gray-scale images;
CIFAR-10 is a real-world objects dataset, which contains 60,000 32×32 color images;
Finally, STL-10 is also a real-world objects dataset, which has some common (e.g., airplane, cat, and dog) classes with CIFAR-10.
We use the labeled subset of STL-10 that consists of 13,000 96×96 color images.

NLP Datasets
We use two well-established NLP datasets, i.e., Yelp Review (Yelp) and Sogou News (Sogou).
Yelp is a dataset of English reviews with labels corresponding to scores (between 1 and 5).
It includes 650,000 training and 50,000 testing samples.
Sogou is a dataset of news articles with labels associated with five categories, i.e., sports, finance, entertainment, automobile, and technology.
It includes 90,000 training and 12,000 testing samples.



 §.§ Model Architectures



Blender
To recap, our Blender is composed of an NLP feature extractor, an adapter, two encoders/a single encoder, and a decoder.
The architectures of these components are presented as below: 

NLP feature extractor.
We use the Bidirectional Encoder Representations from Transformers (BERT) language model <cit.>, and fine-tune it.
Then we discard the last layer, and use the embeddings of each token as our NLP features (as previously described in <ref>).
We also try different language models and present the results in <ref>.

Adapter.
The adapter is composed of an average pooling layer, and 4 convolutional ones.

Encoder(s).
We use the same architecture for both double- and single-encoder settings.
Each encoder consists of four convolutional layers with batch normalization and ReLU activation function.

Decoder.
The decoder consists of four convolutional transpose layers.
The first three use batch normalization and ReLU activation function, while the fourth one only uses a tanh activation function.

CV Feature Extractor
We adopt VGG11 <cit.> as our CV feature extractor and use its output as the features.
We evaluate the attack performances when using other CV feature extractors in <ref>.

Victim/Target Model
We use the ResNet18 <cit.> model for our evaluation and show the generalizability of our modal hijacking attack with different target models in <ref>.



 §.§ Evaluation Metrics



Utility
We use utility to measure the performance of the victim model's original task.
To this end, we use a clean testing dataset to evaluate its performance on the clean and the hijacked victim models, where the clean testing dataset is the testing dataset of the original CV model, i.e., from the same distribution as the original dataset but not used when training.
The closer the performances of these two models on the clean test dataset are, the better the utility is.

Attack Success Rate
We use attack success rate (ASR) to measure the effectiveness of the modal hijacking attack, i.e., the performance of the hijacking task on the victim model.
We first create a hijacking test dataset by fusing a clean testing dataset (for the hijacking task) with the container dataset using the Blender.
Next, we compute the accuracy of the victim model on that hijacking test dataset.
Moreover, we train an NLP classification model for the hijacking task to compare its performance with the victim model on the hijacking task, where the data and labels of the hijacking dataset are not changed.
This model represents the case where the adversary wants to perform the hijacking task without the modal hijacking attack, i.e., the upper bound of the attack.
The closer the performance between our modal hijacking attack and this model is, the better the modal hijacking attack is.



 §.§ Results









To recap, the adversary first needs to construct a container dataset to train the blender, as previously mentioned in <ref>.
We randomly sample 100 images from each target dataset to construct their corresponding container datasets.
These 100 images are then removed from the target datasets, i.e., ∩ = Φ.
Next, we sample 5,000 samples from each hijacking dataset, with 1,000 instances per label.
For each hijacking-original dataset pair, we randomly map the corresponding 5,000 hijacking samples with the 100 container images and train the Blender as mentioned in <ref>.
Afterward, we use the trained Blender to fuse the 5,000 hijacking samples and poison the victim model.
The fused samples are combined with the complete training dataset for each victim model, i.e., 60,000, 50,000, and 5,000 samples for the MNIST, CIFAR-10, and STL-10 datasets, respectively.

To evaluate the performance of our modal hijacking attack in terms of both utility and attack success rate, we train clean models for all original and hijacking tasks. 
For the original tasks, we use the complete original dataset when training the models.
And we use the whole clean testing dataset to evaluate the performances.
As mentioned in <ref>, the closer the performances – with respect to the clean testing dataset – of the hijacked and clean models are, the better the modal hijacking attack is.
For the hijacking tasks, we calculate the upper bound of the performance by using the complete training datasets, not just the 5,000 sentences used to hijack the victim model.
We then sample a testing dataset from the hijacking's task test dataset and use it to evaluate these models.
Finally, we fuse this dataset and use the fused version of the dataset to evaluate the attack success rate of the victim model.
The closer the attack success rate to the upper bound performance is, the better the modal hijacking attack is.

Quantitative Evaluation
We plot the attack success rate (ASR) in <ref>.
As the figure shows, our modal hijacking attack achieves strong performance independent of the hijacking and the original datasets.
For instance, our attack achieves 94%, 94%, and 95%, when using the Sogou dataset to hijack MNIST, CIFAR-10, and STL-10 victim models by our Blender of double encoders, respectively, which is only 1% worse than then upper bound models.
Similarly, for the Yelp dataset, our attack's ASR is only 4%, 1%, and 4% less than the upper bound for the MNIST, CIFAR-10, and STL-10 victim models, respectively.
This clearly demonstrates the effectiveness of our modal hijacking attack.
Especially taking into consideration that we only use 5,000 hijacking samples to hijack the victim models unlike the entire dataset when training the upper bound ones, we next evaluate our modal hijacking attack's utility.
We plot the performances of the victim models and the ones trained with clean datasets in <ref>.
As the figure shows, our modal hijack attack achieves comparable performance on the clean testing dataset with the clean models.
For instance, the victim model achieves the utility of 99% (99%), 93% (93%), and 93% (92%) on the MNIST, CIFAR-10, and STL-10 models when being hijacked by the Yelp (Sogou) dataset in the double-encoder case, respectively.
This shows the negligible drop in model utility for our modal hijacking attack.
In addition, we evaluate the attack performance on the Tiny ImageNet <cit.> as the original dataset.
The results show that our attack is generalizable to higher-quality images, as shown in <ref>.
Furthermore, when we restrict the Blender using a single encoder, our method achieves comparable attack performance.
Besides, in the following exploration of different hyperparameters/decision designs (<ref>), using a single encoder consistently shows similar power as double encoders.
These results provide a potential option to gain strong attack performance with less cost of computation resources.

Qualitative Evaluation
Concretely, <ref> shows randomly sampled fused samples together with their container images, as well as corresponding details.
As the figure shows, the output of our Blender is very similar to the original dataset, with few visible artifacts.
As shown later (<ref>), these artifacts do not make the fused images easier to detect.
Besides, compared to the fused images of <cit.>, where the hijacking digits are visible, it is hard to recognize the hijacking content in our attack.
This advantage further increases the stealthiness of our modal hijacking attack.
Moreover, increasing the training epochs of the Blender can reduce the artifacts as shown in <ref>.
To compare the performances of the modal hijacking attack with the naive approach, i.e., only using the adapter but not the Blender, we plot the outputs of the adapter of the Yelp samples in <ref>.
Comparing both figures, the output of our Blender is clearly more similar to the original dataset, hence, showing the stealthiness of our modal hijacking attack.




 §.§ Hyperparameters/Design Decisions



























Using Different NLP Features
To recap, our attack utilizes the embeddings on all tokens instead of only using the“[cls]” one <cit.>.
We now evaluate the performance of our attack when using only the “[cls]” token's embedding.
To this end, we use the Yelp and CIFAR-10 hijacking and original datasets, respectively.
As depicted in <ref>, with double encoders, using only the “[cls]” token's embedding reduces the attack success rate from 68% to 22%.
However, surprisingly, using a single encoder could even slightly improve the attack success rate.
Besides, using the “[cls]” token's embedding does not change the model's utility (similar to our decision of encoding all tokens as <ref> shows), as both approaches result in fused images – with features – that are distinct from the original images.
Hence, the target hijacked model is able to learn the original task without any loss of utility.

Naive Attack
We now evaluate the naive approach for the modal hijacking attack, i.e., directly using the output of the adapter instead of the Blender.
In this part, we take the Blender with double encoders as an example.
To this end, we evaluate the naive approach for all of the hijacking and original datasets.
The results show that the naive approach can achieve almost the same performance as our attack for some datasets, e.g., MNIST and CIFAR-10.
However, the victim models' utility drops significantly on STL-10.
For instance, the utility of the victim models drops separately to 46% and 33% when using the Yelp and Sogou datasets.
We present the full results in <ref>.
Moreover, we plot the resulting images from the naive approach in <ref>.
As the figures show, the resulting images clearly look random and consequently easy to detect, which violates the goal of stealthiness.
These results show the advantage of using the Blender to perform the attack over the naive approach.

Different Feature Extractors
We now show the generalizability of our modal hijacking attack by using different feature extractors.
More concretely, instead of using BERT and VGG11 to train the Blender, we use BART <cit.> and MobileNetv2 <cit.> as the NLP and CV feature extractors, respectively.
The results show that changing the feature extractors yields similar performance regardless of the encoder setting.
For instance, with double encoders, it achieves 68%, 69%, and 63% ASR with a negligible drop in utility, when using the Yelp dataset to hijack MNIST, CIFAR-10, and STL-10, respectively.
We present the full results in <ref>.
This result shows that our attack can use different feature extractors depending on the adversary's preference.

Different Victim Models
We now show the generalizability of our modal hijacking attack against different target victim models.
To this end, we use the Yelp dataset as the hijacking task for all of the CV tasks while using the AlexNet <cit.> as the victim model's architecture.
As expected, our modal hijacking attack still achieves strong performance against the AlexNet-based models in both cases of using double encoders and a single one.
For example, with double encoders, it achieves 67%, 67%, and 66% ASR with the utility drop within 3% when hijacking MNIST, CIFAR-10, and STL-10 classification models, respectively.
We present the full results in <ref>.
These results show that our modal hijacking attack is indeed independent of the victim's model architecture.
We believe this independence is due to the robust feature extraction of our attack.
More concretely, using state-of-the-art language models to extract features from the hijacking datasets results in robust features that can be picked up by different target model architectures when fused with container images.

Intuitively, we hypothesize that the effectiveness of our hijacking attack relies on the redundant model capacity.
That is, the victim model could simultaneously support its original task (i.e., stealthiness) and the hijacking task (i.e., effectiveness).
To this end, we explore the case of using an MLP model as the victim model, as the MLP model can only achieve the utility of 53%, which is much lower than CNNs.
Such a utility gap indicates that MLP models do not own redundant capacity.
In that case, if the hypothesis is accepted, our hijacking attack against MLP models should achieve poor performance.
However, in <ref>, it is surprising to find that our attack performs strongly in both stealthiness and effectiveness.
These results indicate that low-capacity victim models are also vulnerable, which further increases the risks of our attack.

Number of Training Epochs for Blender
We evaluate the effect of varying the number of the Blender training epochs using the Yelp-CIFAR-10 setting, where we take the Blender with double encoders as an example.
We train Blenders using from 50 to 500 epochs with steps of 50, then evaluate our attack performance on them.
Our results show that the utility and ASR do not change much with the number of epochs (<ref>).
However, the quality of the fused images does get better with a higher number of epochs.
The visual quality saturates at approximately 200 epochs.
Hence, the adversary can use 50 epochs if they care less about the visual appearance of the fused images; otherwise 200 epochs would be a good compromise.
We show a set of randomly sampled fused images for the different epochs in <ref>.

Effect of Fine-tuning BERT
We now evaluate the effect of fine-tuning the NLP feature extractor ().
To this end, we use the Yelp dataset to hijack a CIFAR-10 classification model with a non-fine-tuned BERT.
Our results show that the utility of the victim model does not change; however, the ASR is significantly impacted, i.e., the ASR significantly drops.
This shows the need to fine-tune  on the hijacking dataset.
Besides, even the non-fine-tuned one with more poisons is not comparable to a fine-tuned NLP feature extractor.
Another interesting finding is that our single-encoder Blender is more resilient to the non-fine-tuning case.
We present the full results in <ref>.

The Poisoning Rate Effect
We now, to better understand the adversary's capacity, evaluate the influence of the poisoning rate on our attack.
We use the Yelp-CIFAR-10 setting (with 60,000 clean images) and set the number of poisoning – fused – samples from 2,500 (4.2%) to 10,000 (16%) with steps of 2,500 to hijack different models.
The results show that using 2,500 samples is too few to hijack the model, i.e., the ASR is less than 60%.
However, increasing the points beyond 5,000 (8.3%) does not improve the ASR.
Hence, we use 5,000 hijacked samples for our evaluations in <ref>.

In application, the amount of data to poison the dataset here depends on the adversary.
On the one hand, as our attack implements a new task, it just depends on how accurate the adversary wants their hijacking task to be.
On the other hand, even if the poisons could be injected by involving new parities or intrusion techniques <cit.>, less modification of the dataset is intuitively less possible to be revealed, which indicates a more practical setting and more stealthiness.
It is also important to note that even if the accuracy is not state-of-the-art, the ability of the model to perform an unethical task (even with low performance) can make the model owner accountable for a violation.
Even, in that case, one successful attack case might be enough to support it.
Besides, state-of-the-art models usually have millions of inputs, e.g., ImageNet has more than 1.2 M images.
Hence, the amount of poisoning data can be negligible depending on the application.
Further, we find that the Blender with double encoders are more sensitive to the poisoning rate.
We present the full results in <ref>.

Reusability of the Blender
As the Blender can be expensive to train, we evaluate reusing a trained Blender to hijack different settings. 
We try four different setups:
The first hijacks models using the same container dataset but different victim models;
The second increases the flexibility and uses a different container dataset;
The third and the fourth further increase the efficiency of the attack by using a pre-trained Blender to camouflage different hijacking datasets.
For the first case, we already presented its results when evaluating our attack against different target victim models, i.e., AlexNet and MLP.
For the second case, we use the Blender trained using the Yelp hijacking and CIFAR-10 container datasets to hijack MNIST and STL-10 models.
In other words, we use the CIFAR-10 trained Blender to separately fuse images from the MNIST and STL-10 datasets.
Our results show that the modal hijacking attack still achieves strong performance.
More concretely, it achieves the same utility compared to using their original Blender (<ref>) with a non-significant ASR drop for the MNIST and STL-10 hijacked models, respectively.
We present the full results in <ref>.
For the third case, we use the Blender trained on Yelp (Sogou) but conduct the attack with the hijacking dataset of Sogou (Yelp).
In that setting, our modal hijacking can gain almost the same performance.
For instance, when the adversary trains a double-encoder Blender on the Yelp (Sogou) dataset to hijack a CIFAR-10 model using the Sogou (Yelp) hijacking dataset, our attack achieves an ASR of 93% (66%) and utility of 94% (94%), which is only 0%-2% lower compared to training a specific Blender for each hijacking dataset.
Meanwhile, the single-encoder cases show similar results.
We present the full results in <ref>.
For the final case, we try a different setup where two hijacking datasets (Yelp and Sogou) are used to build a Blender.
Then this Blender is used to attack a third dataset (CoLA <cit.>, which is a binary dataset).
This approach shows that it is indeed better to use two datasets instead of one when building the Blender.
More concretely, using the double-encoder Blender trained jointly on both datasets (Yelp and Sogou) improves the performance to 80% ASR and 93% utility, which is 4% (3%) stronger in ASR than when using a Blender trained on a single dataset, e.g., Yelp (Sogou).
We present the full results in <ref>.
These results demonstrate that a single Blender is reusable to different setups, which significantly reduces the cost and increases the flexibility of our modal hijacking attack.

Container Dataset Creation
Finally, we propose a way of constructing the container dataset.
So far, we use a randomly selected container dataset.
One problem is that the labels might not align with the fused samples.
Hence, a manual inspection can raise some flags.
To this end, we now propose a more stealthy way to construct the container dataset.
We first train a CV classifier on the container dataset.
Next, we sort the images that are misclassified with the most confidence.
Finally, we manually select the container dataset out of these images. <ref> shows a subset of these images.
As the figure shows, it is hard to manually assign a label to these images, which makes the fused samples more stealthy.




 §.§ Defense







We evaluate using two established defenses against data poisoning attacks to mitigate our modal hijacking attack <cit.>.
The first defense, intuitively, clusters a clean dataset and computes the centroid of each label.
Then for any given input, the distance between it and its corresponding centroid – depending on its label – is calculated.
Inputs with distances exceeding the threshold are then discarded.
We evaluate this defense technique on the hijacking tasks of Yelp and Sogou and the original tasks of MNIST, CIFAR-10 and STL-10.
The results consistently show that the attack performance drops to almost random guesses, however, it induces an average drop of the utility of around 15%, where using double encoders is more robust.
Moreover, this type of defense requires access to clean data which is sometimes hard to get for many applications in practice.
The second defense EPIC aims to find effective poisons and drop them in low-density gradient regions during training.
The results show that the defense indeed reduces the attack success rate, however, our hijacking attack is still effective in most cases.
We present the complete results in <ref>.




§ DISCUSSION





 §.§ Challenges in Multiple-Modality Scenario



As aforementioned, the main challenge in our work is that NLP and CV tasks are separately located in discrete and continuous feature spaces.
This gap boosts the design of our Adapter and of course our Blender, which generalizes our straightforward but effective framework of hijacking attacks from the homogeneous to the heterogeneous modalities.
Such an improvement largely benefits the practical usage of hijacking attacks.
In practice, this multimodal setting is prevalent since deep learning is now deployed in various applications, e.g., facial recognition, self-driving, and text-to-image generation.
Intuitively, our technique can be adapted for different scenarios where the hijacking task has feature extractors, e.g., language models for the NLP tasks.
However, different modalities have different difficulties, i.e., how easy/hard the original data can be modified, which indicates a non-trivial challenge of adaptively implementing hijacking attacks.

Utilizing the representative tasks in NLP and CV domains, our work serves as the pioneer in exploring hijacking attacks in the multimodal scenario.
We hope that inspired by our work, more complicated scenarios could be applicable in future works.
For instance, hijacking an NLP model with a CV task is an interesting next step but would be significantly difficult/different, as text inputs are not continuous and hence cannot be changed as the images do.



 §.§ Stealthiness of Fused Images



Compared to the fused images of previous work <cit.> (where MNIST digits are visible in the fused CIFAR-10 images), ours look much more natural as shown in <ref>, indicating more stealthiness.
Nevertheless, few noises could still be observed.
We plan to reduce these artifacts by adding a GAN-like discriminator <cit.> to the training of the Blender in future work.
This discriminator is trained to differentiate between the container and fused samples.
The Blender is penalized for any container sample that the discriminator can identify, hence improving the appearance of the fused samples.



 §.§ Computational Costs of Training Blender



Another issue to consider is the computational cost of training the Blender.
Even though this is not the main scope of our work, minimizing the cost of hijacking attacks could improve the adversary's ability in practice.
To this end, we show in <ref> that our Blender can be trained once and then utilized for multiple hijacking attacks, even for different hijacking tasks, making it more efficient.
Besides, our extensive evaluations show the feasibility of only using a single encoder in our Blender instead of double ones, which largely saves computational costs.




§ CONCLUSION



Model hijacking attack is a new threat that exploits the inclusion of new parties in the ML training pipeline.
In this attack, the adversary can hijack CV-based models to implement their own image classification task.
However, as ML has improved in multiple domains besides CV, the hijacking and original tasks might be from different data modalities.
In the paper, we propose a more general hijacking attack in the multimodal scenario, where the adversary can hijack CV models using NLP tasks, which increases the practical usage of hijacking attacks.
To this end, we propose an autoencoder-based model that mixes language models with CNNs, namely the Blender, to perform our modal hijacking attack.
Using the Blender, the adversary can hijack image classification models using text/NLP hijacking tasks.
We extensively evaluate our attack using five different datasets, including three image classification datasets and two text ones.
Our results show that the modal hijacking attack achieves strong performances (i.e., effectiveness) with a negligible drop in the model's utility (i.e., stealthiness).
Besides, the relaxed encoder requirement and improved visual appearances of poisons indicate the advantages of computational efficiency and stealthiness over the previous model hijacking attack.

We aim by this work to first raise awareness of possible accountability risks in some of the realistic machine learning training pipelines.
Then, to motivate the community to work on different mitigation techniques to address this risk, we already present a couple of possibilities.
Finally, our modal hijacking technique can also be used for compressing target models, hence reducing the training or maintenance costs.



plain






§ VISUAL RESULTS















