Paper Title: Using CSNNs to Perform Event-based Data Processing & Classification on ASL-DVS.









Using CSNNs to Perform Event-based Data Processing & Classification on ASL-DVS
    
    
        Ria Patel,
        Sujit Tripathy, Zachary Sublett, Seoyoung An and Riya Patel
    
        
            Department of Electrical Engineering & Computer Science,
        
    
    
        
            University of Tennessee, Knoxville, TN 37916, USA
        
   

    
===========================================================================================================================================================================================================================================================================================




Using CSNNs to Perform Event-based Data Processing & Classification on ASL-DVS
    
    
        Ria Patel,
        Sujit Tripathy, Zachary Sublett, Seoyoung An and Riya Patel
    
        
            Department of Electrical Engineering & Computer Science,
        
    
    
        
            University of Tennessee, Knoxville, TN 37916, USA
        
   

    
===========================================================================================================================================================================================================================================================================================

§ ABSTRACT 


Recent advancements in bio-inspired visual sensing and neuromorphic computing have led to the development of various highly efficient bio-inspired solutions with real-world applications. One notable application integrates event-based cameras with spiking neural networks (SNNs) to process event-based sequences that are asynchronous and sparse, making them difficult to handle. In this project, we develop a convolutional spiking neural network (CSNN) architecture that leverages convolutional operations and recurrent properties of a spiking neuron to learn the spatial and temporal relations in the ASL-DVS gesture dataset. The ASL-DVS gesture dataset is a neuromorphic dataset containing hand gestures when displaying 24 letters (A to Y, excluding J and Z due to the nature of their symbols) from the American Sign Language (ASL). We performed classification on a pre-processed subset of the full ASL-DVS dataset to identify letter signs and achieved 100% training accuracy. Specifically, this was achieved by training in the Google Cloud compute platform while using a learning rate of 0.0005, batch size of 25 (total of 20 batches), 200 iterations, and 10 epochs. 









§ INTRODUCTION


The application of deep learning has been instrumental in solving numerous computer vision problems, notably image classification <cit.>, object detection and semantic segmentation <cit.>, and gesture recognition <cit.>, etc. For some specific applications such as static image recognition, Deep Neural Networks (DNNs) have been demonstrated to outperform human performance <cit.>. Training state-of-the-art deep learning models is computationally expensive, leading to high energy usage. To reduce the energy cost of future AI systems, researchers are exploring the use of energy-efficient, brain-inspired spiking neural networks (SNNs) in various fields <cit.>. SNNs typically take inputs from neuromorphic sensors that generate spikes with changes in input signals. Next, SNNs interpret such spikes, which contain sparsely encoded spatial and temporal information. SNNs implemented on neuromorphic hardware allow the hardware to utilize sparse activations within the SNNs to gain significant power and latency improvements compared to conventional hardware performing the same tasks <cit.>.

For data acquisition, dynamic vision sensors (DVS) are used to capture individual pixel changes within each frame of a recording, making them capable of capturing motion very similar to humans. Pixels that change are considered events whereas those that don’t change are not considered events by the camera. Next, the event-based data is encoded into spikes such that the SNN can interpret the spatiotemporal features.

Conventional frame-based cameras capture information from all pixels within the frame, making the image densely rich in information and memory demanding. Event-based cameras mitigate this issue by reducing this pixel information to make them sparsely encoded. As a result, these DVS cameras consume low power enabling operation at a finer time resolution of a few milliseconds <cit.>. 

Recently, SNNs have been used for problems that involve data with strong spatiotemporal correlations such as hand gesture recognition, object tracking, action and object recognition etc. <cit.>. In this work, we contribute a CSNN that performs classification on event-based data from ASL-DVS dataset  <cit.>. The rest of the report is organized as follows: Section II includes a
literature review that provides some background on CSNN architecture as well as discusses some prior work that utilized SNNs including CSNNs. Section III presents the proposed CSNN framework for ASL-DVS data processing and classification. Section IV discusses the characteristics of the ASL-DVS dataset and its preprocessing steps in detail. Next, we present the experimental set-up and results before concluding the report.






§ LITERATURE REVIEW




In this section, we first present the SNN and CSNN model architecture. Next, we discuss some prior works that utilized SNNs including CSNNs for event-based data processing.





 §.§ SNNs

An SNN shares similarities with an artificial neural network (ANN). Both models calculate neuron outputs using the weighted sums of inputs. However, SNNs process single-bit spikes as input, which are sparse in both space and time. Also, SNNs employ different types of activations compared to ANNs. The weights contribute to the membrane potential, and when this potential reaches a certain threshold, the neuron generates a spike. Spikes introduce a temporal component to the data processed by an SNN. In the ASL-DVS dataset, this temporal aspect is measured in μs to capture subtle movements occurring within the images  <cit.>.



There are several different types of spiking neuron models, but one of the most commonly used is called the Leaky Integrate and Fire (LIF) neuron. This neuron is closely modeled after biological neurons, making it suitable for neural networks. It integrates the weights of its input over time until it reaches or exceeds a certain threshold and then, the neuron will fire a spike. The membrane potential will decay towards a lower value at each step controlled by the decay rate β, enforcing the "leaky" aspect of the neuron. Figure <ref> demonstrates this feature, which is taken from  <cit.>.




The LIF membrane is modeled with:

    τdU(t)/dt = -U(t) + RI_int


This equation models how the membrane potential changes over time. Where I_in represents the input-weight matrix, R represents the membrane resistance, and the membrane potential at a given time t is represented by U(t) = β U(t - 1) + (1 - β) I_in(t), where β is the decay rate of the membrane potential. 




 §.§ CSNNs


A convolutional spiking neural network typically comprises spiking convolution operations and spiking encoding. Assume an input sequence of S(n), for n = 0, 1, 2,..., S(n) is a tensor at every time step that has shape u,v, where u and v are the width and height of each frame. Output from event cameras can be processed using a variable batch window to achieve the desired temporal resolution. For example, a 3 s 240 × 180 resolution events data stream with 100 ms temporal resolution will generate S(n), n = 0, 1, 2, ..., 30. For each segment, the tensor shape is (240,180). Next, the input tensor is convolved with a 2-D convolutional kernel to generate a spiking feature map  <cit.>. The convolution operation can be described using the following equations  <cit.>:


    y[i,j] = ∑_j = -∞^∞∑_i = -∞^∞ = x(i + m, j + n).K(m,n)



Where, y[i,j] represents the spiking feature map, x is the input tensor, K is the kernel, (i,j) are coordinates on the feature map and (m,n) are the coordinates in the kernel. Neurons in the feature map receive spikes, which are aggregated to generate membrane potential. 

The convolutional layers extract features. The LIF neurons at the output of each layer then gather all the input spikes and output a spike when the membrane potential crosses the set threshold. Lastly, similar to the traditional convolutional neural networks (CNNs), the last layer of CSNNs is a fully connected (FC) layer with LIF neurons and has an output shape equal to the number of classes for classification.



 §.§ Prior Technical Work

In this subsection, we review prior works discussing applications of SNNs and CSNNs for event-based data processing.

Amir et al. implemented a gesture recognition system that utilized a TrueNorth neurosynaptic processor alongside a DVS for real-time hand gesture recognition with minimal power consumption. They achieved an out-of-sample accuracy of approximately 96.5% on a diverse dataset, demonstrating the robustness of the approach for various hand gestures under different lighting conditions <cit.>.

Weijie et. al used CSNN for EMG (Electromyogram)-based hand gesture recognition. They utilized the Strathclyde and CapgMyo open-source datasets for model development. The average recognition accuracy was observed to be around 98.76% based on the Strathclyde dataset and 98.21% based on the CapgMyo open source dataset <cit.>.

Xing et al. introduced a spiking convolutional recurrent neural network (SCRNN) model for event-based hand gesture recognition, using the DVS gesture dataset. They achieved an accuracy of 96.59% for 10-class gesture recognition and 90.28% for 11-class gesture recognition <cit.>.

Samadzade et al. investigated the spatiotemporal feature extraction capabilities of CSNNs for event-based data processing. They observed superior performance of CSNNs methods on N-MNIST (97.6%), DVS-CIFAR10 (69.2%), and DVS-Gesture (96.7%) datasets. Additionally, the model achieved results comparable to ANN methods on UCF-101 (42.1%) and HMDB-51 (21.5%) datasets <cit.>.

The availability of comprehensive neuromorphic datasets plays a crucial role in the increased implementation of SNN-based architecture. Yin et. al <cit.> and Orchard et. al <cit.> created neuromorphic ASL-DVS and MNIST datasets. Eshraghian et al. compiled a comprehensive list of neuromorphic vision and audio benchmark datasets, widely accessed for SNN-based model development <cit.>. Moreover, efforts have been made to convert traditional computer vision datasets to neuromorphic vision datasets. A method for converting static image datasets into spiking neuromorphic datasets using an actuated pan-tilt camera platform was presented. By simulating motion updates through sensor movement, this approach enabled the utilization of popular image datasets like MNIST and Caltech101 for spike-based recognition algorithms <cit.>.

Lastly, to bridge the gap between deep learning and SNNs, Eshraghian et al. presented principles from deep learning, gradient descent, backpropagation, and neuroscience to develop SNNs. By elucidating the relationship between encoding data as spikes and the learning process, this work helps to advance the understanding and application of SNNs in NN research and development <cit.>.





§ TECHNICAL APPROACH


 
In this section, we present the overall deep learning pipeline for ASL-DVS data processing and classification. Then, we briefly discuss data selection and processing, the architecture of the CSNN model trained for ASL-DVS classification, and the CSNN learning method.



 §.§ Deep Learning Pipeline

Figure <ref> shows the overall deep-learning pipeline. We start with acquiring neuromorphic ASL-DVS datasets. Subsequently, we pre-process the data to bring it to a format so that it can be imported into the CSNN model. Next, we have an iterative process of model training and tuning to improve the CSNN model performance. Then, we validate our model that performs ASL-DVS gesture classification.



 §.§ Data Selection and Pre-Processing

For this project, we have used the ASL-DVS dataset to perform event-based data processing and classification. ASL-DVS dataset was acquired in AEDAT format and then, IniVation DV Processing software was used to pre-process it before the data could be imported into a CSNN model for training and validation. In the next section, we discuss the characteristics of the ASL-DVS dataset and its preprocessing steps in detail.




 §.§ Network Architecture

The CSNN model used to process the neuromorphic ASL-DVS dataset is built upon a similar CSNN implemented for N-MNIST data processing and classification  <cit.>. However, we updated the network
architecture to accommodate the ASL-DVS dataset. The model architecture comprises of three convolutional layers and a FC layer with 24 outputs for the 24 classes of the ASL-DVS dataset. Figure <ref> is a diagram showing the model's architecture. Each convolutional layer includes 5x5 filters with layers 1, 2, and 3 containing 12, 32, and 45 filters respectively. Each convolutional layer outputs to a 2x2 max pool layer and then to a LIF (leaky integrate and fire) neuron. The model then feeds into a FC layer with 24 output LIF neurons, that will determine the class or letter in the ASL-DVS dataset that the input belongs to. Figure <ref> shows the details involved in calculating the size of CSNN layers.



 §.§ CSNN Learning Method

For model training, the Adam optimizer is utilized. The loss function employed is the Mean Square Error (MSE) of the spike count, calculating the MSE between the correct and incorrect spike count estimates. These estimates are typically expressed as proportions of spikes over the total number of time steps. The target proportions for the loss function can vary, but a common choice is an 80% correct rate and a 20% incorrect rate <cit.>, which is used in this model.

To perform backpropagation using spikes, a specific method known as BackPropagation Through Time (BPTT) is utilized. This algorithm computes gradients by determining the loss of all descendants in the computational graph and aggregating them. Consequently, the derivative of the loss with respect to the weights ∂ℒ/∂ W is computed for each time step. The equation below illustrates how backpropagation computes this derivative  <cit.>:


    ∂ℒ/∂ W = ∑_t∂ℒ(t)/∂ W = ∑_t∑_t≤ s∂ℒ(t)/∂ W(s)∂𝒲(s)/∂ W


This allows for the model to learn the parameters, but there is still the problem of calculating the gradients when spikes themselves are non-differentiable (also, called "dead neuron problem"). The adopted solution to this is known as the surrogate gradient method. Essentially, this swaps the non-differentiable function for one that is differentiable during the calculation of gradients.














§ DATASET & IMPLEMENTATION




 §.§ Dataset Characteristics




The ASL-DVS dataset was created using the IniVation DAVIS240C NVS event-based camera under constant lighting conditions, which captures at a 240 × 180 resolution and has a 12 µs latency <cit.>. The events are captured and stored in the AEDAT 2.0 format, but a conversion process is used to achieve AEDAT 4.0, the newest AEDAT format that allows for the usage of the IniVation DV Processing software framework. Using this software, the AEDAT files were converted to CSV files to better view the event-based information and perform data exploration. 



This dataset contains recording data from five subjects who signed each letter of the alphabet, excluding J and Z. The ASL for J and Z are excluded from the dataset due to the nature of how those letters are signed (they include hand motion along with gesture). In this project, recording data from the first four subjects is used to train the CSNN. 

Figure <ref> illustrates the components of the ASL-DVS dataset. The dataset comprises 100,800 samples, also referred to as "frames," each lasting approximately 100 microseconds (μs). The number of samples varies for each letter in each subject, and each sample contains a varying number of events detected by the DVS camera. The terminology used in the data pre-processing section is explained in Figure <ref>.

Each event has four attributes: timestamp, x, y, and polarity. The timestamp denotes the exact μs that this event took place in the specified pixel via x and y coordinates. Lastly, polarity denotes whether there is an increase or decrease in the pixel brightness or intensity, eventually to be encoded as a spike.




 §.§ Dataset Exploration


To visualize the ASL-DVS dataset, we used the DV-Processing software to load each subject and each letter in the AEDAT4.0 format. The Python module library contains classes that load in each recording as a batch of events, which renders an image as shown in Figure <ref>. This image contains 1,515 events, creating the silhouette of Subject 1's hand signing the letter 'f' as they move it in an oscillating left-to-right motion.
The DVS camera picks up all movements and stores them as the black pixels shown in Figure <ref>, however, in the AEDAT files, the polarity field denotes them as either 0 or 1. This signifies that at the specific pixel, either there was a decrease or increase in pixel brightness. For the sake of visualization and ease of use for the SNN model, we encode all events that occur as 1 (black pixels) while the rest are encoded as 0 (white space).



Figure <ref> illustrates the overall sample distribution for each class across all subjects in the original ASL-DVS dataset. It is important to note that there are fewer samples for the letters Y and Z. This discrepancy is due to Subject 5 excluding recordings for the letters J and Y, while Subjects 1-4 exclude recordings for the letters J and Z.



Figure <ref> presents the sample distribution for each subject concerning vowel letters. Vowels were chosen for visualization as they are the most frequently used letters in English. Notably, Subject 5 contributes significantly to the sample distribution, indicating that Subject 5's dataset contains the longest recordings.



Figure <ref> provides a visualization of the number of events captured from Subject 1's recording of the letter F. The recording is divided into 3-second batches, each containing a varying number of events within the time window. This figure illustrates the subject's hand movement patterns at the beginning, middle, and end of the recording, demonstrating how the oscillation path of the hand changes throughout the recording.





 §.§ Dataset Pre-processing


We have extracted a subset from the ASL-DVS dataset by randomly choosing batches of events from each subject's recordings for a letter. Each recording varies in duration (e.g. 3 minutes, 5 minutes). We divide each recording into 3-second event batches, each containing a different number of events in each sample. Figure <ref> provides an example of an event batch, although it is not from the preprocessed dataset.

Following the event batch creation, we randomly select 3 event batches from each recording to include in our subset. For each subject and class, we save the randomly selected event batches into Numpy binary files. Subsequently, we combine all event batches from each subject and organize them into 24 classes.

To create the train and validate datasets, we randomly shuffle the event batches for each class. We then allocate 70% of the event batches for training and reserve the remaining 30% for validation. Our training set contains 495 event batches and our validation set contains 214 event batches.





§ EXPERIMENTS & RESULTS





 §.§ Experimental Setup

As shown in <ref>, the input to the model is a tensor of dimension: [time steps, batch size, channels, height, width]. An input event batch of 3 sec duration has 30 time steps with bin window duration of 100 milliseconds. The batch size is set at 25 implementing mini-batch gradient descent (GD). Also, the frame height and width are 180 and 240 respectively with 2 number of channels. The training dataset that consists of 495 event batches is passed onto the CSNN model and the model is trained based on the learning method discussed in section III. For this work, the 'Adam' optimizer is utilized with a learning rate = 0.0005, Betas = (0.9, 0.999). The loss function used is 'MSE of the spike count' with the 80% correct rate and a 20% incorrect rate. With 495 training data points (event batches) and a mini-batch batch size of 25, we have 20 iterations in each epoch. For training, we have experimented with varying numbers of epochs. Similarly, for validation, we have a batch size of 214 with all 214 event batches being processed simultaneously to generate validation results.




 §.§ Hardware


Given the size of the dataset, we turned to Google Cloud Platform (GCP) for training our model using various hyperparameter combinations. When attempting to run the model locally on the dataset, we encountered issues with insufficient memory.

In terms of hardware, we employed the N2 machine series, featuring 8 Intel Cascade Lake and Ice Lake virtual CPUs (referred to as threads by GCP) and 32GB RAM. This series utilizes x86 architecture and the SCSI and NVMe disk interface type. We selected the N2 machine series due to its suitability for general-purpose workloads and batch processing, aligning perfectly with our requirements.

On GCP, we leveraged the Vertex AI Machine Learning platform to conduct our model training within a Jupyter Notebook. Before training, we set up a Bucket, a storage solution in GCP, to house our dataset. Subsequently, within the Vertex AI-hosted Jupyter Notebook, we trained the model using the dataset stored in the Bucket, which already contained the separated training and testing data.



 §.§ Results

This subsection shows the results from our model for the best-performing hyper-parameters, in the form of loss and accuracy plots as well as a table.

Figures <ref> and <ref> show the model's training accuracy and loss plots for training cases with 1 epoch and 10 epochs each. This was the best training performance the model had achieved. It starts to stabilize around 90%-100% for the last several GD batches. Also, the loss is gradually decreasing across all batches. 

Figure <ref> shows the validation loss and accuracy across models trained for 1, 3, 5, 9, and 10 epochs. The models that had been trained for longer epochs generally got a higher accuracy and a lower loss indicating that longer training will result in the model learning better and in turn performing better. 

Table <ref> shows an overview of results from the different epochs. As supported by the graphs, the training loss gradually decreases while the training accuracy increases with the number of epochs. It even hits 100% accuracy for a single training batch (GD batch) as early as 5 epochs. Similarly, the validation loss also decreases while accuracy increases for each of the epochs. However, we also noted the mismatch between training and validation accuracy. This might indicate that the model was overfitting to the training dataset although it could otherwise be attributed to noise in event batches (some event batches captured background objects behind some subjects showing ASL gestures). This leads to the proposed future work, which may aim at exploring different regularization techniques to improve validation accuracy.


 

 





§ CONCLUSION



For this project, we aimed to develop a CSNN model capable of processing and classifying the ASL-DVS dataset. To begin, we obtained the ASL-DVS dataset and pre-processed it to prepare it for training the CSNN model. Using GCP, we successfully developed and implemented the CSNN, achieving a high training accuracy of 100% and a validation accuracy of 81%. Our findings indicate that as the number of training epochs increased, the losses decreased and the accuracies improved. However, we observed a disparity between training and validation accuracies, suggesting the need for further research to explore different regularization techniques aimed at enhancing validation accuracies.







§ ACKNOWLEDGMENT

We would like to express our gratitude to Dr. Charlie Rizzo from the TENNLab for his invaluable assistance in clarifying DVS datasets. Despite recently defending his Ph.D. thesis, he generously devoted his time to aid us in understanding the intricacies of neuromorphic dataset processing.






§ APPENDICES






 §.§ Coding

Our code repository <cit.> is organized into three main directories. The  directory contains visualization GIFs showcasing the event-based data. The  directory houses our model code and dataloader scripts. The  directory contains code for data preprocessing, analysis, and visualization.

For our implementation, we utilize the PyTorch framework along with the snnTorch framework, which serves as a PyTorch extension for spiking neural networks. Additionally, we incorporate Tonic, a library that streamlines the loading of neuromorphic datasets into the PyTorch framework. Tonic enables the application of transforms specifically designed for spatiotemporal datasets, while also being compatible with native PyTorch transforms.




 §.§ Code Design




  §.§.§ Data Pre-processing

Our data pre-processing code utilizes the DV-Processing library mostly. In the `utils/` directory of our repository, we created a `helper_funcs.py` that contains utility functions to perform batching of the recording data. These functions are used in the Jupyter notebooks `data_explore.ipynb` and `data_preprocess.ipynb`, which are to be run in the respective order. The `generate_letter_datasets.py` script, which must be run before the `data_preprocess.ipynb` code, utilizes the other two scripts in the directory `export_aedat4_to_csv.py` and `process_aedat_files.py` to create the training and validation datasets. 




  §.§.§ Model

The model design code was influenced by a snnTorch tutorial demonstrating the use of Convolutional Spiking Neural Networks (CSNNs) on the Neuromorphic MNIST (N-MNIST) dataset <cit.>. This tutorial provided insights into importing the DVS dataset into Tonic. 

To adapt the CSNN architecture for the ASL-DVS dataset, which has larger and non-square input dimensions compared to the 34 × 34 dimensions of the N-MNIST dataset, we made significant changes to the network structure. These changes included modifying the layers, adjusting the number of neurons in each layer, and refining the convolutional filters.







 Paper Title: Aligning Multiple Knowledge Graphs in a Single Pass.











Journal of  Class Files, August 2024
Yang et al.: Interpretable and Efficient Heterogeneous Graph Convolutional Network








§ ABSTRACT 
Entity alignment (EA) is to identify equivalent entities across different knowledge graphs (KGs), which can help fuse these KGs into a more comprehensive one. Previous EA methods mainly focus on aligning a pair of KGs, and to the best of our knowledge, no existing EA method considers aligning multiple (more than two) KGs. To fill this research gap, in this work, we study a novel problem of aligning multiple KGs and propose an effective framework named MultiEA to solve the problem. First, we embed the entities of all the candidate KGs into a common feature space by a shared KG encoder. Then, we explore three alignment strategies to minimize the distances among pre-aligned entities. In particular, we propose an innovative inference enhancement technique to improve the alignment performance by incorporating high-order similarities. Finally, to verify the effectiveness of MultiEA, we construct two new real-world benchmark datasets and conduct extensive experiments on them. The results show that our MultiEA can effectively and efficiently align multiple KGs in a single pass.



Knowledge Graphs, Entity Alignment, Graph Neural Networks




Aligning Multiple Knowledge Graphs in 
 a Single Pass
    Yaming Yang,
        Zhe Wang,
        Ziyu Guan^** Corresponding author,
        Wei Zhao,
        Weigang Lu,
        Xinyan Huang

Y. Yang, Z. Wang, Z. Guan, W. Zhao, and W. Lu are with the State Key Laboratory of Integrated Services Networks, School of Computer Science and Technology, Xidian University, Xi'an, China 710071.
E-mail: {yym@, zwang@stu., zyguan@, ywzhao@mail., wglu@stu.,}xidian.edu.cn
X. Huang is the Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, the School of Artificial Intelligence, Xidian University, Xi'an, China 710071.
E-mail: xinyanh@stu.xidian.edu.cn



    August 5, 2024
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================











§ INTRODUCTION

knowledge graphs (KGs) are a special kind of graph that can store a wealth of structural facts (i.e. knowledge) about the real world. Each fact is usually structured as a triple (h,r,t), representing that head (subject) entity h and tail (object) entity t hold relation r between them. In recent years, KGs have successfully supported many web applications such as search engines <cit.>, question-answer systems <cit.>, recommender systems <cit.>, etc.

In practice, different KGs are constructed based on diverse data sources and different extraction approaches, and a single KG can usually cover only a specific aspect of structural facts. For example, an English KG usually contains more facts about the English-speaking society, while a Chinese KG contains more facts about the Chinese-speaking society. Considering that more KGs together can provide more comprehensive structural facts from various aspects, researchers have proposed many methods (Cf. surveys <cit.>) to fuse a pair of KGs into a unified one. The general process is to first identify equivalent entities between the candidate KGs, and then let them serve as the “bridge entities” to link the candidate KGs into a unified one.

Technically, existing mainstream methods typically solve this problem by embedding entities into a common latent representation space and minimizing the distance between pre-aligned entity pairs. This process is also referred to as entity alignment (EA). Depending on how to obtain the entity embeddings, they can be classified into two main categories: (1) Trans-based EA methods <cit.> learn entity embeddings by letting each triple satisfy some specific geometric properties in the embedding space. Most of these methods adopt TransE <cit.> as the translation module to preserve the property: h + r ≈ t. (2) GNN-based EA methods <cit.> adopt graph neural network (GNN) models such as graph convolutional network (GCN) <cit.> to learn entity embeddings by iteratively aggregating the embeddings of neighbor entities.



However, all the existing EA methods are specifically designed for aligning only a pair of candidate KGs, which can cover limited knowledge for downstream applications. To the best of our knowledge, none of the existing EA methods consider aligning multiple (more than two) KGs. For example, Figure <ref> shows a case where four candidate KGs need to be aligned, which triggers a more challenging research problem. 

There are three obstacles that prevent traditional pair-wise KG alignment methods from being applied to the multiple KG alignment problem studied in this work.
First, existing EA datasets are particularly constructed for pair-wise KG alignment. At present, there is no benchmark dataset that can support the study of aligning multiple KGs.
Second, if we trivially adapt existing methods to this task, they need to be separately executed multiple times since they can only utilize the local (pair-wise) alignment information in a single pass, which is inefficient.
Last and most importantly, they cannot capture some useful global (beyond pair-wise) alignment information, affecting the final alignment performance. For the example in Figure <ref>, when they align KG-en with KG-fr, and align KG-en with KG-zh independently, the entities of the three KGs will be projected into four separate embedding spaces. Consequently, it is easy to yield inconsistent results like e_1^(en)≡ e_1^(fr), e_1^(en)≡ e_1^(zh), and e_1^(fr)≡ e_2^(zh) (wrong), violating the transitivity of the alignment relationship.

To fill the research gap in this field, in this work, we innovatively study the problem of aligning multiple KGs in a single pass. First of all, to facilitate the study of this problem, we construct two new benchmark datasets that contain multiple KGs to be aligned. Then, we formulate a novel research problem of aligning multiple KGs concurrently. Finally, we propose an effective framework named MultiEA to solve the problem. As illustrated in Figure <ref>, our MultiEA can concurrently align the four candidate KGs in a single pass, which is more practical. In addition, it embeds the entities of all the candidate KGs into a unified space and thus can capture useful global alignment information, which helps achieve better alignment performance. Recall the example above, if MultiEA pulls e_1^(en) and e_1^(fr) closer and pulls e_1^(en) and e_1^(zh) closer. Then, e_1^(fr) and e_1^(zh) will definitely be pulled closer as well, leading to the consistent result: e_1^(en)≡ e_1^(fr), e_1^(en)≡ e_1^(zh), and e_1^(fr)≡ e_1^(zh) (right). 

The main contributions of this work are summarized as follows.


  * To the best of our knowledge, in the EA research community, we are the first to study the problem of concurrently aligning multiple (more than two) KGs. We are also the first to construct the benchmark datasets and define the evaluation metric for this problem.

  * We propose an effective framework named MultiEA to solve the problem. Three alignment strategies are explored to capture the global alignment information. An innovative inference enhancement technique is proposed to significantly boost the alignment performance.

  * We conduct extensive experiments on the two constructed benchmark datasets. The results demonstrate that MultiEA can effectively and efficiently align multiple KGs in a single pass. We will release our source codes and datasets to facilitate further research on this problem.






§ RELATED WORK

In this section, we comprehensively review existing methods that are related to ours.


 Trans-based EA Methods. Early EA methods exploit KG embedding methods, e.g., TransE <cit.>, to project a pair of candidate KGs into a common low-dimensional Euclidean space by letting the embeddings of head entity h, relation r, and tail entity h satisfy the triangular relationship: h + r ≈ t. Given a set of pre-aligned entity pairs (i.e., seed alignment labels), they minimize the distance between these equivalent entity pairs in the embedding space. Thus, more potential equivalent entity pairs can be discovered after model optimization. Representative Trans-based EA methods include JEwP <cit.>, JAPE <cit.>, MTransE <cit.>, IPTransE <cit.>, BootEA <cit.>, TransEdge <cit.>, MultiKE <cit.>, etc. 


 GNN-based EA Methods. Motivated by the extraordinary success of graph neural networks (GNNs) in extracting the structural features of graphs, GCN-Align <cit.> first introduces GCN <cit.> into EA, which has shown superior performance than Trans-based EA Methods, e.g., JEwP <cit.>, MTransE <cit.> and JAPE <cit.>. Further, AVR-GCN <cit.> deftly combines the graph convolutional operator in GNNs and the translation operator in Trans-based methods to extract better entity embeddings. Afterward, a series of following GNN-based EA methods, e.g.,  <cit.> are proposed. Recently, RREA <cit.> insightfully summarizes KG embedding-based EA methods and GNN-based EA methods into a unified framework consisting of two modules: shape-builder and alignment. The former constraints KGs into a specific distribution in the embedding space, and the latter minimizes the distance between the embeddings of pre-aligned entity pairs.


 EA Enhancement. In recent years, several works further enhanced the EA performance by exploiting attribute information and entity name information. HMAN <cit.>, BERT-INT <cit.>, and ACK-MMEA <cit.> can leverage the rich attributes of entities. Besides, PSR <cit.>, GMNN <cit.>, NMN <cit.>, HGCN-JE <cit.>, RDGCN <cit.>, LightEA <cit.> and SelfKG <cit.> can use the word embeddings of the entity names to effectively initialize entity embeddings, greatly improving the performance of EA.


 Alignment Methods in Other Fields.
In the social network analysis field, there are several methods that have made some efforts to align multiple social networks. COSNET <cit.> leverages multiple networks to enhance the alignment performance of two networks. It requires exponential time complexity to build a matching graph, which is impractical for many scenarios. ULink <cit.> learns a latent user space based on user attributes. It cannot exploit structural information and thus it is not a strict social network alignment approach. MC^2 <cit.> combines both structural information and attribute information to infer a common base by matrix factorization. It requires all social networks to have attributes, and its time computational complexity is square to the number of uses. MASTER <cit.> embeds multiple social networks in a common latent space through collaborative matrix factorization, which also requires square computational complexity. In the data integration field, a recent work <cit.> is proposed to align multiple tables based on Sentence-BERT <cit.>, table-wise hierarchical merging, and density-based pruning. These methods are specially designed methods for other fields and they generally require sophisticated optimization.

In summary, different from existing methods, in this work, we study the problem of aligning multiple (more than two) KGs based on the structural information. Accordingly, we develop a GNN-based neural network framework named MultiEA to solve the problem in an end-to-end optimization manner.





§ PRELIMINARIES


In this section, we first give the formal definition of knowledge graphs. Then, we formulate the novel problem of aligning multiple KGs.


Knowledge Graph (KG).
A knowledge graph is defined as 𝒢 = (ℰ, ℛ, 𝒯), where ℰ is the set of entities (nodes) and ℛ is the set of relations (edges). 𝒯⊆ℰ×ℛ×ℰ is the set of triples, and each triple t ∈𝒯 is represented as <e_i, r_k, e_j>, which means that head entity e_i ∈ℰ and tail entity e_j ∈ℰ hold relation r_k ∈ℛ between them. We use 𝐡_i∈ℝ^d to denote the representation of entity e_i in one model layer, where d is the dimensionality of embeddings. We use 𝐠_k∈ℝ^d to denote the embedding of relation r_k, which is a randomly initialized learnable parameter vector.


Multiple KG Alignment.
The input is a set of KGs {𝒢^(1), 𝒢^(2),  𝒢^(m), ..., 𝒢^(M)}, where M is the number of the input KGs, and M>2. The m-th KG is denoted by 𝒢^(m) = (ℰ^(m), ℛ^(m), 𝒯^(m)).
The problem is to identify equivalent entities that refer to the same real-world thing across all the M KGs, based on a set of seed alignment labels (i.e., pre-aligned entities), denoted as 𝒮 = { l_1, l_2, ..., l_n, ..., l_N }, where N is the number of labels. The n-th label is denoted as l_n = (e_n^(1), e_n^(2), ..., e_n^(m), ..., e_n^(M)), where entity e_n^(m)∈ℰ^(m), and all the M entities associated with l_n are equivalent.

As shown in Figure <ref>, KG-en, KG-zh, KG-fr, and KG-ja are four candidate KGs that need to be aligned, where the four entities marked with red dashed lines refer to the same object, i.e., “Sweden”. We treat the four equivalent entities as one of the seed alignment labels, to help our EA algorithm find more potential equivalent entities. Finally, we can link (merge) the four KGs based on these detected equivalent entities across the four KGs.






§ FRAMEWORK

In this section, we describe our proposed MultiEA framework in detail. Firstly, we design a GNN-like KG encoder to embed the entities of all the candidate KGs into a common vector space. Then, we propose three alignment strategies to measure the distances among pre-aligned entities, which guide the training of EA models. Finally, we develop an innovative inference enhancement technique to boost the alignment performance by incorporating high-order similarities.





















 §.§ KG Encoding


Given a set of KGs: {𝒢^(1), 𝒢^(2), ..., 𝒢^(m), ..., 𝒢^(M)}, we use a shared KG-oriented GNN encoder to compute their entity embeddings. In the following, we take the m-th KG, i.e., 𝒢^(m) = (ℰ^(m), ℛ^(m), 𝒯^(m)) as an example to elaborate on the details of our encoder. The superscript m is omitted for the sake of notation simplicity.

Firstly, we augment the original KG data to add a virtual self-relation r_self into the relation set ℛ, formally described as follows:

    ℛ = ℛ∪{ r_self}.

Then, for each entity, we add a virtual triple to describe that each entity e_i has the self-relation r_self between itself, which is added to the triple set as follows:

    𝒯 = 𝒯∪{ <e_i, r_self, e_i> | e_i ∈ℰ}.

To facilitate the aggregation of the GNN encoder, for entity e_i, we define the set of its neighbor relation-entity tuples as follows:

    𝒩_i = { <r_k, e_j> | <e_i, r_k, e_j> ∈𝒯, or <e_j, r_k, e_i> ∈𝒯}.

Although the relations in KGs are usually directed, following most previous works, we treat them as bi-directed (or undirected). This is reasonable because the inverse of each relation is usually intuitive as well. For instance, for the triple <Obama, wife, Michelle>, we can also rewrite it as: <Michelle, husband, Obama>. Note that due to the addition of self-connection, entity e_i will have a special neighbor relation-entity tuple: <r_self, e_i> ∈𝒩_i.

In a layer of our KG encoder, the output representation of entity e_i is computed as follows:

    𝐡_i' = σ( ∑_<r_k, e_j> ∈𝒩_iα_i,k,j·𝐖_k·𝐡_j),

where 𝐡_j is the neighbor entity e_j's feature vector output by the previous layer, 𝐖_k is a relation-specific projection matrix, α_i,k,j is the attention coefficient for aggregation, and σ is the non-linear activation function.

There are often thousands of relations in common KGs, and thus the relation-specific projection matrices 𝐖_k, ∀ r_k ∈ℛ may introduce too many trainable parameters, risking overfitting. Therefore, following previous studies <cit.>, we leverage the relation embedding vector 𝐠_k to define its related projection matrix, as follows:

    𝐖_k= 𝐈 - 2 ·𝐠_k·𝐠_k^T.

In this way, we no longer need to introduce additional parameters. In particular, the normalization constraint 𝐠_k_2 = 1 is imposed, so that the orthogonality of 𝐖_k is naturally guaranteed since we can easily derive: 
    𝐖_k^T·𝐖_k = (𝐈 - 2 ·𝐠_k·𝐠_k^T)^T (𝐈 - 2 ·𝐠_k·𝐠_k^T) = 𝐈.

The orthogonality of 𝐖_k is proven to be very beneficial for EA task <cit.>.

The attention coefficient is computed as follows. Given a neighbor tuple <r_k, e_j> ∈𝒩_i of entity e_i, we first compute the proximity among the head entity e_i, the relation r_k, and the tail entity e_i:

    β_i,k,j = σ( 𝐚_h^T ·𝐡_i + 𝐚_r^T ·𝐠_k + 𝐚_t^T ·𝐖_k·𝐡_j),

where 𝐚_h, 𝐚_r, and 𝐚_t are learnable parameter vectors for head entities, relations, and tail entities, respectively. Then, we compute the attention coefficient α_i,k,j by normalizing β_i,k,j across all the elements of 𝒩_i:

    α_i,k,j = exp( β_i,k,j)/∑_<r_x, e_y> ∈𝒩_iexp( β_i,x,y).

Note that by adding the self-relation r_self, each entity e_i itself acts as one of its own neighbors (Cf. Eqs. (<ref>, <ref>)). In addition, we use the attention mechanism to aggregate all the neighbors (Cf. Eqs. (<ref>, <ref>)). Thus, as proven by <cit.>, our encoder is able to automatically learn the importance of arbitrary hops of neighborhood information. This is more flexible than <cit.>, which directly concatenates all hops of neighborhood information.

Recall that the encoder is shared among all the KGs, and thus we can obtain the embeddings of the entities from every KG.





 §.§ Training for Alignment


In order to pull equivalent entities together in a unified embedding space, we adopt the following margin ranking loss as the training loss of our MultiEA model:

    L=∑_l ∈𝒮∑_l' ∈𝒮'max(d(l) - d(l') + λ, 0),

where 𝒮 is the set of positive examples, i.e., ground-truth alignment labels, 𝒮' is the set of negative examples, i.e., randomly generated false alignment labels, d(·) is a distance function, and  λ > 0 is a margin hyperparameter for separating positive and negative examples.

The negative examples in 𝒮' are generated as follows. For each positive example l = (e^(1), e^(2), ..., e^(m), ..., e^(M)) ∈𝒮, we fix one entity and replace the other entities by randomly sampling from their corresponding KGs, formally described as follows:

    η×{[ (e^(1), ẽ^(2), ..., ẽ^(m), ..., ẽ^(M)),; (ẽ^(1), e^(2), ..., ẽ^(m), ..., ẽ^(M)),;                                    ...,; (ẽ^(1), ẽ^(2), ..., e^(m), ..., ẽ^(M)),;                                    ...,;  (ẽ^(1), ẽ^(2), ..., ẽ^(m), ..., e^(M));                                         ]},

where ẽ^(m) is randomly sampled entity from ℰ^(m) to replace the original entity e^(m). Here, η is an integer, indicating that for each positive example, we generate η groups of negative examples in this way. Thus, each positive example corresponds to η× M negative examples.

In this work, we propose three strategies for minimizing the distance among equivalent entities. The ideas are intuitively shown in Figure <ref>. For a positive example l = (e^(1), e^(2), ..., e^(m), ..., e^(M)) ∈𝒮, we firstly obtain its associated entity embeddings from the KG encoder and denote them as: (𝐡^(1), 𝐡^(2), ..., 𝐡^(m), ..., 𝐡^(M)). Then, the distance d(l) is minimized by three strategies as follows.



 (1) Moving toward Mean.
This strategy lets all the equivalent entities approach their mean, as illustrated in Figure <ref>. We first compute the mean as follows:

    𝐮 = 1/M·∑_m=1^M𝐡^(m).

Then, we sum the Euclidean distances between each entity embedding and the mean:

    d(l) = ∑_m=1^M𝐡^(m) - 𝐮_2.



 (2) Moving toward Anchor.
In practice, the candidate KGs are usually unbalanced due to various realistic factors. For instance, on our constructed multi-lingual KG dataset DBP-4, the English KG contains many more triples than the other three KGs (Cf. Table <ref>). Therefore, as shown in Figure <ref>, we designate entities from KG DBP-4: en as anchor entities and let the equivalent entities from the other KGs approach these anchor entities. This idea is formally described as follows:

    d(l) = ∑_m=1^M𝐡^(m) - 𝐡^(a)_2,

where 𝐡^(a) is embedding of the anchor entity[Note that when 𝐡^(m) is designated as anchor entity, then 𝐡^(m) - 𝐡^(a)_2 = 0.].


 (3) Moving toward Each Other.
This strategy lets equivalent entities approach each other. As illustrated in Figure <ref>, we minimize the distances between all possible pairs of equivalent entities, formally described as follows:

    d(l) = ∑_m_1 ≠ m_2𝐡^(m_1) - 𝐡^(m_2)_2,

where m_1 and m_2 are in the set { 1, 2, ..., m, ..., M }.








 §.§ Training Time Complexity


Let |ℰ|, |ℛ|, and |𝒯| denote the total numbers of the entities, the relations, and the triples of all the M candidate KGs, respectively. Let d denote the dimensionalities of entity embeddings and relation embeddings. Let |𝒮| denote the number of alignment labels. Recall that each label corresponds to η× M negative examples.

In the KG encoding phase (Cf. Section <ref>), the time complexity involves three main computational operations. First, the aggregation operation described by Eq. (<ref>) has time complexity 𝒪( |ℰ| · d^2 ). Second, the computation of relation-specific projection matrices described by Eq. (<ref>) has time complexity 𝒪( |ℛ| · d^2 ). Finally, as described by Eqs. (<ref>, <ref>), the computation of attention coefficients has time complexity 𝒪( |𝒯| · (d + d + d^2) ). Considering that in practice, the embedding dimensionality d is a relatively small value, the time complexity of KG encoding is 𝒪( |ℰ| + |ℛ| + |𝒯| ). 

In the alignment phase (Cf. Section <ref>), as described by Eq. (<ref>) and Eq. (<ref>), both the mean strategy and the anchor strategy require time complexity 𝒪(M). As described by Eq. (<ref>), the each other strategy requires time complexity 𝒪(M^2). As described by Eq. (<ref>), taking the number of labels into consideration. The mean strategy and the anchor strategy have time complexity of 𝒪(|𝒮| ·η· M^2), and each other strategy has time complexity 𝒪(|𝒮| ·η· M^3).

In practice, |𝒮|, η, and M are much smaller than |ℰ|, |ℛ|, and |𝒯|. Therefore, the overall training time complexity of MultiEA is equal to 𝒪( |ℰ| + |ℛ| + |𝒯| ).




 §.§ Inference Enhancement


After training the model, we can obtain the embeddings of the entities from all the KGs. Based on these embeddings, we can compute the similarity between any two entities that are from different KGs. For example, given entity e_i^(m_1)∈ℰ^(m_1), and entity e_j^(m_2)∈ℰ^(m_2), we compute the similarity between them as follows[Since entity embeddings are normalized, i.e., 𝐡_i_2 = 1, ∀ i, the computed similarities lie in the range [0,1]. Thus, we can compose high-order similarities by matrix product operation later.]:

    𝐒_i,j^m_1 - m_2 = 1 - 𝐡_i^(m_1) - 𝐡_j^(m_2)_2/2.


Thus, we can obtain a matrix 𝐒^m_1 - m_2 to describe the entity similarities between 𝒢^(m_1) and 𝒢^(m_2). However, Eq. (<ref>) only captures the first-order similarity, which may not be sufficient since some higher-order similarity information is ignored. Therefore, in this work, we propose to further enhance the similarity matrix by incorporating higher-order similarities, which can be composed by matrix product operation. For example, if there are three KGs, 𝒢^(m_1), 𝒢^(m_2), and 𝒢^(m_3), we can enhance the similarity matrix 𝐒^m_1 - m_2 as follows:

    𝐒^m_1 - m_2 = γ_1 ·𝐒^m_1 - m_2 + γ_2 ·𝐒^m_1 - m_3·𝐒^m_3 - m_2,

where γ_1 and γ_2 are hyperparameters to balance the two terms. They are real numbers in the range [0,1] and satisfy γ_1 + γ_2 = 1. Similarly, if there are four KGs, we can enhance 𝐒^m_1 - m_2 as follows:

    𝐒^m_1 - m_2   = γ_1 ·𝐒^m_1 - m_2 + γ_2 ·𝐒^m_1 - m_3·𝐒^m_3 - m_2
       + γ_3 ·𝐒^m_1 - m_4·𝐒^m_4 - m_2,

where γ_1, γ_2, and γ_3 are hyperparameters in the range [0,1] and satisfy γ_1 + γ_2 + γ_3 = 1.

Here, we only incorporate the two-order similarities by the matrix product operation. In practice, higher-order similarities can be similarly incorporated through more matrix product operations. Finally, we can identify equivalent entities according to the enhanced similarity matrix. In this way, the model can utilize more comprehensive information, helping improve the inference performance.





§ EXPERIMENT


In this section, we construct the benchmark dataset and define the evaluation metric for the new task. Then, we conduct extensive experiments to verify the effectiveness of our MultiEA.



 §.§ Benchmark Datasets


As far as we know, there is no existing EA benchmark dataset that can support the task of aligning multiple KGs. To this end, we construct two novel benchmark datasets based on previously widely used real-world datasets <cit.>. One dataset DBP-4 contains four candidate KGs, and another dataset DWY-3 contains three candidate KGs to be aligned. The key statistics of the constructed datasets are listed in Table <ref>.




•DBP-4.
This dataset contains four KGs from different language sources, including English (en) KG, Chinese (zh) KG, Japanese (ja) KG, and French (fr) KG. It is constructed based on the DBP15K dataset, which was originally released by <cit.>. The original alignment labels describe the correspondence from English KG to the other three KGs, which can be described as: en-fr, en-zh, and en-ja. We treat the English KG as an intermediary and complement the alignment information among all four KGs, which can be described as: en-fr-zh-ja. Specifically, we filter entities from different KGs that correspond to the same English entity. Intuitively, if we have: e^(en) = e^(fr), e^(en) = e^(zh), and e^(en) = e^(ja). Then, we can infer: e^(en) = e^(fr) = e^(zh)= e^(ja), which means that all these four entities from different KGs are equivalent.

•DWY-3.
This dataset contains three KGs, i.e., DBpedia (dbp), Wikidata (wiki), and YAGO (yago). It is constructed based on the DWY100K dataset, which was originally released by <cit.>. Its original alignment labels describe the correspondence from DBpedia to the other two KGs, which can be described as: dbp-wiki, and dbp-yago. Similar to the construction of DBP-4, we treat the DBpedia KG as an intermediary to complement the alignment information among all three KGs, which can be described as: dbp-wiki-yago.

It is worth noting that the condition under which we construct labels is very strict. As a result, the number of labels is very small compared to the number of entities and triples. To address the issue, we further induce a smaller knowledge graph. Specifically, inspired by the previous work <cit.>, for each entity involved in labels, we select its popular neighbor entities whose degrees are larger than a threshold (we set 15 in this work). This target entity and its high-degree neighbor entries are added to a new entity set, and the involved relations are added to a new relation set. Then, we induce a new set of triples based on the selected entities and relations.







 §.§ Evaluation Metric

Traditional pair-wise KG alignment methods generally use Hits@K to evaluate the alignment performance. Specifically, given two KGs: 𝒢^(l) and 𝒢^(r), we first consider aligning 𝒢^(r) to 𝒢^(l), and define a temporary metric l_Hits@K to measure the proportion of the entities in 𝒢^(l) whose counterpart in 𝒢^(r) rank in top-K:

    l_Hits@K = ∑_n=1^N𝕀(e_n^(r)ranks in top-K)/N,

where 𝕀 is the indicator function that outputs 1 if the input fact is true and 0 otherwise. Then, we consider aligning 𝒢^(l) to 𝒢^(r), and similarly define a temporary metric r_Hits@K in the opposite direction. The final metric Hits@K is computed as the average of the two temporary metrics:

    Hits@K = 1/2· (l_Hits@K + r_Hits@K).


Unfortunately, Hits@K does not work for our task of multiple KG alignment. Hence, in the spirit of Hits@K, we define a novel evaluation metric. Given M KGs: {𝒢^(1), 𝒢^(2), ..., 𝒢^(m), ..., 𝒢^(M)} where M>2, we treat each KG 𝒢^(m) as the target KG, and consider aligning the other M-1 KGs to it. First, we count the number of entities from 𝒢^(m) whose all M-1 counterparts from the other M-1 KGs rank in top-K, described as follows:

    C = ∑_n=1^N𝕀(e_n^(1)...,e_n^(m-1),e_n^(m+1)...,e_n^(M)rank in top-K),

where 𝕀(·) is an indicator function that outputs 1 if all the input entities rank in top-K, and 0 otherwise. Then, a temporary metric m_Hits@K is defined to compute the proportion of:

    m_Hits@K = C/N.

The final metric is computed as the average of all the M temporary metrics: 

    M-Hits@K = 1/M·∑_m=1^Mm_Hits@K.

Obviously, our newly defined metric M-Hits@K is more challenging than the traditional metric Hits@K since the condition for counting a correct entity is much more strict. This can better reflect the model performance of aligning multiple KGs.




 §.§ Baselines and Variants

Regarding baselines, we select six representative supervised EA baselines that only leverage the structural information of candidate KGs, including two Trans-based EA methods (1), (2), and four GNN-based EA methods (3), (4), (5), (6). They are listed as follows:


  * MTransE <cit.> is a Trans-based EA method that uses TransE <cit.> to encode entities of each KG in a separated embedding space;

  * IPTransE <cit.> further uses PTransE <cit.> to encode KGs and propose an iterative alignment strategy;

  * GCN-Align <cit.> is the first GNN-based EA method that uses GCN <cit.> to encode two KGs;

  * NAEA <cit.> further uses GAT <cit.> to encode the structural information of KGs. Therefore, it learns entity embeddings by aggregating neighbor entities with different importance;

  * KECG <cit.> also uses GAT <cit.> to encode KGs, and additionally restricts the projection matrix to be a diagonal matrix, reducing the number of parameters and computations;

  * PSR <cit.> encodes KGs based on orthogonal projection matrix and attention aggregation like ours. Differently, (1) it dose not add self-connections, and (2) it directly concatenates all hops of neighborhood information.


Recall that in Section <ref>, we introduce three different alignment strategies. Here, we use +mean, +anchor, and +each to denote the three strategies, respectively. In Section <ref>, we propose the inference enhancement module. Here, we use +infer or -infer to indicate whether the module is equipped. By considering all the permutations, we set the following six variants for our MultiEA:



  * MultiEA+mean-infer;

  * MultiEA+anchor-infer;

  * MultiEA+each-infer;

  * MultiEA+mean+infer;

  * MultiEA+anchor+infer;

  * MultiEA+each+infer.










For anchor-based variants, we treat DBP-4: en and DWY-3: dbp as the anchor KGs on the two datasets, respectively.












 §.§ Implementation Details


For the constructed benchmark datasets of DBP-4 and DWY-3, we follow the convention to randomly select 30% elements of the seed alignment set 𝒮 to form the training set, and the rest are treated as the test set. We implement our MultiEA based on a PyTorch-based open-source EA toolkit named EAkit <cit.>. For baselines, we use their implementations provided in the EAkit package. All the experiments are conducted on our constructed benchmark datasets, and all the methods use the same dataset partitioning.

For our MultiEA, we use the same hyperparameter settings for both datasets. This is a challenging configuration since it can better reflect the sensitivity of hyperparameters w.r.t. datasets. All the model parameters are randomly initialized by the Xavier uniform distribution <cit.>. We adopt the Adam optimizer, and the learning rate is set to 0.01. The embedding dimensionalities of both entities and relations are set to 256. The non-linear activation function σ is set to the ELU function <cit.>. The margin hyperparameter λ is set to 1. The number of negative example groups, i.e., the hyperparameter η is set to 10. We adopt early stopping for model training, and the number of patience steps is set to 10.

All the experiments are conducted on an NVIDIA TITAN RTX GPU with 24GB GPU memory and 128GB main memory.










 §.§ Multiple KG Alignment


We first quantitatively evaluate the effectiveness of our MultiEA in aligning multiple (more than two) KGs. Specifically, we apply all six baseline methods as well as all six variants of MultiEA to the two constructed benchmark datasets, and use the metric M-Hits@K newly defined in Eq. (<ref>) to compare their accuracy. The results are reported in Table <ref>.


 Comparison with Baselines. It is worth noting that we cannot directly apply baseline EA methods to the multiple KG alignment task since they are specifically designed for the traditional pair-wise KG alignment task. To make them compatible, we split the concerned task into multiple pair-wise sub-tasks and separately align each pair of KGs. To reflect this, baselines are marked by ^* in the table. We can see that our MultiEA (especially the three variants with the inference enhancement module at the bottom of the table) can significantly outperform all the baselines on the two datasets. This may be due to that our MultiEA can effectively capture the global alignment information by projecting the entities of all the candidate KGs in a unified embedding space. Besides, GNN-based EA baselines show better performance than Trans-based EA baselines. This is consistent with the findings of many previous studies <cit.>.


 Abalation Studies. As we can see, our MultiEA+each+infer achieves the best overall results in most cases, showing its superior effectiveness in aligning multiple KGs. Overall, MultiEA+anchor+infer and MultiEA+mean+infer achieve the second-best and the third-best results, respectively. The situation is the same for the other three variants without the inference enhancement module. Therefore, we recommend the “each other" strategy as the default strategy for most cases. If users prefer higher efficiency and there is an obvious anchor KG, we recommend the “anchor" strategy, and otherwise, we recommend the “mean" strategy. From another dimension, we can observe that all three variants with the inference enhancement module significantly outperform the other three variants without this module. This indicates that the inference enhancement module is significantly beneficial for this task.





 §.§ Case Study


To intuitively show the effectiveness of our MultiEA, in Figure <ref> and Figure <ref>, we list ten groups of equivalent entities discovered by MultiEA (the best-performing variant MultiEA+each+infer) on the two datasets. As we can see, MultiEA can effectively discover equivalent entities across all the candidate KGs.

Further, we visualize the embeddings of these entities. Specifically, we utilize the t-SNE algorithm <cit.> to project them into the 2-dimensional Euclidean space. Figure <ref> and Figure <ref> show the results on DBP-4 and DWY-3, respectively, where different colors and different shapes mark the entities from different KGs, and the equivalent entities are connected by black lines. As we can see, on both datasets, the ten groups of equivalent entities are obviously gathered together, indicating the effectiveness of the alignment mechanism of MultiEA. On the other hand, a proper margin is maintained between different groups, avoiding the risk of model overfitting. The result on DWY-3 shows a relatively better distribution. This is consistent with the higher accuracy on DWY-3 as reported in Table <ref>.




 §.§ Efficiency Study


In this experiment, we demonstrate the efficiency advantage of our MultiEA in aligning multiple KGs. As shown in Table <ref> and Table <ref>, we first report the experimental results of our re-implemented GNN-based EA baselines as described in Section <ref>. That is, they align multiple candidate KGs by splitting this task into multiple separate pair-wise sub-tasks, as marked by ^* in the tables. Then, we apply the “each other” alignment strategy and the inference enhancement module of our MultiEA+each+infer to each baseline X, as denoted as MultiEA (X) in the tables. 

In addition to the M-Hits@K scores, we report the number of model parameters, the memory, and the time required by each method during its training. As we can see, our MultiEA can help each baseline achieve significantly better performance. Moreover, MultiEA can help each baseline significantly reduce the requirements of parameters, space, and time resources. This demonstrates the superior efficiency of our MultiEA.





 §.§ Pair-wise KG Alignment


We also apply our MultiEA (the variant MultiEA+each+infer) to the traditional pair-wise KG alignment task. Firstly, based on the two constructed benchmark datasets, we induce several pair-wise KG alignment sub-datasets. For DBP-4, we induce three sub-datasets, i.e., DBP-4: en-fr, DBP-4: en-ja, and DBP-4: en-zh. For DWY-3, we induce two sub-datasets, i.e., DWY-3: dbp-wiki and DWY-3: dbp-yago. For baselines, we reproduce their results on our induced datasets. Then, we re-implement the encoder of MultiEA as that used in a baseline X, denoted as MultiEA (X). This can eliminate the encoder's influence on the alignment results, and thus investigate whether our proposed “each other" strategy and inference enhancement module helps improve the performance of baselines on this task. By convention, we use the widely used metric Hits@K described by Eq. (<ref>) to evaluate the alignment accuracy.

Table <ref> and Table <ref> shows the final results. In addition to the original experimental results, we also compute the relative gain to show the improvement more intuitively. As we can see, our MultiEA can always help improve the baselines' performance on the traditional pair-wise KG alignment task. Besides, we have some other findings as follows. PSR, KECG, and NAEA perform better than GCN-Align, indicating the effectiveness of the attention mechanism used in their KG encoders. Moreover, MultiEA (PSR), MultiEA (KECG), and MultiEA (NAEA) outperform MultiEA (GCN-Align). This demonstrates that our MultiEA framework can effectively exploit the structural information captured by the graph attention mechanisms. Two Trans-based methods MTransE and IPTransE show worse performance than GNN-based methods, which is consistent with the conclusions of previous works <cit.>. 

The experimental results reported here are slightly different from the results reported in previous studies. The main reason lies in that our constructed datasets are different from existing datasets. For example, the label ratio on our DBP-4 is 2539 × 4/8901+3545+4326+3893≈ 0.4915, while the label ratio of DBP15K used in previous studies is (15000 × 2/19388+19572+15000 × 2/19780+19814+15000 × 2/19993+19661) ×1/3≈ 0.7614, which is much larger than ours.









 §.§ Hyperparameter Study




In this subsection, we study the impact of MultiEA's four key hyperparameters on multiple KG alignment accuracy.

First, we study the balance hyperparameters introduced in the inference enhancement module, as described by Eqs. (<ref>, <ref>). In practice, we only search the weight for the first term, and the weights for the other terms are set to the same value, thus compressing the hyperparameter search space. For one example of DBP-4 dataset, we can rewrite Eq. (<ref>) as: 

    𝐒^en - zh   = γ·𝐒^en - zh + 1 - γ/2·𝐒^en - fr·𝐒^fr - zh
       + 1 - γ/2·𝐒^en - ja·𝐒^ja - zh.

For another example of DWY-3 dataset, we can rewrite Eq. (<ref>) as: 

    𝐒^dbp - wiki = γ·𝐒^dbp - wiki + (1 - γ) ·𝐒^dbp - yago·𝐒^yago - wiki.
 
The sensitivity of γ on the two datasets is shown in Figure <ref> and Figure <ref>, respectively. We can see that on both datasets, our MultiEA achieves the best performance when γ = 0.2. It means that the importance of the first-order similarity is 0.2 and the importance of the other higher-order similarities is 0.8. This indicates that incorporating high-order similarities is very beneficial for the multiple KG alignment task, demonstrating the effectiveness of our proposed inference enhancement module. This finding is also consistent with the observation in Section <ref>.

Second, we study the sensitivity of the margin hyperparameter λ introduced in the loss function as described by Eq. (<ref>). The results are shown in Figure <ref> and Figure <ref>. As we can see, the performance is very poor when λ = 0, because the model cannot separate the positive examples and the negative examples, causing the underfitting issue. There is a clear inflection point when λ = 1, and thus in the other experiments, we set it to 1 by default. After that, the performance gradually declines, probably because the model suffers the overfitting issue when the margin is too large.

Third, we investigate the sensitivity of η, which has been introduced to describe the groups of negative examples. As shown in Figure <ref> and Figure <ref>, our model is not sensitive to this hyperparameter. Therefore, in practice, we set η to 10 to save computational resources.

Finally, we show the model performance w.r.t. the training ratio in Figure <ref> and Figure <ref>. As expected, the training ratio shows a clear positive effect on improving the model performance, which is consistent with the findings of many previous studies, e.g., <cit.>.





§ CONCLUSION

In this work, we note that existing methods mainly focus on aligning only a pair of candidate KGs, ignoring the multiplicity of the candidate KGs to be aligned. To fill the research gap in this field, we formulate a novel problem of aligning multiple (more than two) KGs and propose the MultiEA framework to effectively solve this problem. To enhance the inference performance, we innovatively propose to incorporate higher-order similarities. Last but not least, we construct two new benchmark datasets and define the evaluation metric for the problem. The experimental results show that our MultiEA framework can effectively and efficiently align multiple KGs in a single pass. We will make our source codes and the constructed benchmark datasets publicly available to facilitate further research on this problem, which we believe will inspire more interesting works in the future.





§ ACKNOWLEDGMENTS

This work was supported in part by the National Natural Science Foundation of China under Grants 62133012, 61936006, 62303366, and 62073255, in part by the Innovation Capability Support Program of Shaanxi under Grant 2021TD-05, and in part by the Fundamental Research Funds for the Central Universities under Grants QTZX23039, XJSJ23030.



IEEEtran


[
    < g r a p h i c s >
]Yaming Yang received the B.S. and Ph.D. degrees in Computer Science and Technology from Xidian University, China, in 2015 and 2022, respectively. He is currently a lecturer with the School of Computer Science and Technology at Xidian University. His research interests include data mining and machine learning on graph data.


[
    < g r a p h i c s >
]Zhe Wang received the B.S. degree in Management from Hefei University of Technology, China, in 2020. He is currently working towards a Ph.D. degree with the School of Computer Science and Technology, Xidian University, China. His research interests include data mining and machine learning on knowledge graph data.


[
    < g r a p h i c s >
]Ziyu Guan received the B.S. and Ph.D. degrees in Computer Science from Zhejiang University, Hangzhou China, in 2004 and 2010, respectively. He had worked as a research scientist in the University of California at Santa Barbara from 2010 to 2012, and as a professor in the School of Information and Technology of Northwest University, China from 2012 to 2018. He is currently a professor with the School of Computer Science and Technology, Xidian University. His research interests include attributed graph mining and search, machine learning, expertise modeling and retrieval, and recommender systems.


[
    < g r a p h i c s >
]Wei Zhao received the B.S., M.S. and Ph.D. degrees from Xidian University, Xi’an, China, in 2002, 2005 and 2015, respectively. He is currently a professor in the School of Computer Science and Technology at Xidian University. His research direction is pattern recognition and intelligent systems, with specific interests in attributed graph mining and search, machine learning, signal processing and precision guiding technology.




[
    < g r a p h i c s >
]Weigang Lu received the B.S. degree in Internet of Things from Anhui Polytechnic University, China, in 2019. He is currently working towards a Ph.D. degree with the School of Computer Science and Technology, Xidian University, China. His research interests include data mining and machine learning on graph data.




[
    < g r a p h i c s >
]Xinyan Huang received the B.S. degree from the School of Electronic Science and Technology, Shaanxi University of Science and Technology, Xi'an, China, in 2015, and the M.S. degree from the School of Physics and Information Technology, Shaanxi Normal University, Xi' an, in 2017. She is currently working towards a Ph.D. degree with the School of Artificial Intelligence, Xidian University, China. Her research interests include machine learning, computer vision, and pattern recognition.




 Paper Title: Distributionally Robust Optimization as a Scalable Framework to Characterize Extreme Value Distributions.


Energy-lowering symmetry breaking creates a flat-band insulator in paramagnetic Nb_3Cl_8
    Alex Zunger
    August 5, 2024
========================================================================================



§ ABSTRACT 
The goal of this paper is to develop distributionally robust optimization (DRO) estimators, specifically for multidimensional Extreme Value Theory (EVT) statistics. EVT supports using semi-parametric models called max-stable distributions built from spatial Poisson point processes. While powerful, these models are only asymptotically valid for large samples. However, since extreme data is by definition scarce, the potential for model misspecification error is inherent to these applications, thus DRO estimators are natural. In order to mitigate over-conservative estimates while enhancing out-of-sample performance, we study DRO estimators informed by semi-parametric max-stable constraints in the space of point processes. We study both tractable convex formulations for some problems of interest (e.g. CVaR) and more general neural network based estimators. Both approaches are validated using synthetically generated data, recovering prescribed characteristics, and verifying the efficacy of the proposed techniques. Additionally, the proposed method is applied to a real data set of financial returns for comparison to a previous analysis. We established the proposed model as a novel formulation in the multivariate EVT domain, and innovative with respect to performance when compared to relevant alternate proposals. 




§ INTRODUCTION



Modeling rare and extreme events is an important task in many disciplines such as finance, climate science, and medicine <cit.>. Estimating distributions of rare events from data is difficult due to the lack of observations within this region, making it challenging to understand the risks deep in the tail of a distribution. Extreme Value Theory (EVT) studies the class of distributions arising as the possible distributional limits that can be used to estimate multivariate distributions in distant (relative to the origin) regions (i.e., tails) which by their nature witness very few observations (or none at all). These distributional limits are derived as the possible asymptotic statistical laws of shifted and re-scaled data as the sample size increases. It turns out that such possible distributional limits form a semi-parametric class called max-stable distributions, which are constructed in terms of a spatial Poisson point process. 

Naturally, because of the lack of data in extremal regions and because of the asymptotic nature of max-stable models, their use in inferential tasks involving tails is exposed to high variance due to model misspecification. Moreover, when using max-stable models, the assumptions that the data converges to the distributions specified by EVT must be made. This leads to an important question: How can we robustify against scenarios deep in the tails while observing potentially sub-asymptotic data where the assumptions of EVT may be violated? To answer this question, we propose a solution based on distributionally robust optimization (DRO). DRO involves a zero-sum game in which the statisticians play against an adversary that perturbs (in a non-parametric way) the nominal / baseline distribution assumed by the statistician. Building on classical DRO, we carefully design constraints to retain the extrapolation properties of EVT for the robustified distribution.

Since we are interested in robustifying the tails, we will consider max-stable baseline distributions given by extreme value distributions (EVDs), as presented by <cit.>. Max-stability roughly states that the distribution of the maximum of independently and identically distributed (i.i.d.) samples belongs to  the same distribution up to a change in the location and scale parameters. This means that the “shape” of the distribution is preserved under the max operation, and it is this property that allows for extrapolating to regions outside the observation domain. In our robustification framework, we therefore wish to preserve this max-stability property. We do this by searching over the space of distributions that are also max-stable, in effect constraining our search to only distributions that extrapolate according to EVT, which form a semi-parametric class. We do this by carefully designing the uncertainty set such that the necessary max-stable properties are preserved. 

To illustrate our desired result, we refer to Figure <ref>. This figure visualizes how a completely unconstrained adversary (solid line) may be too conservative and may not consider the extrapolating properties of the distribution. A well constrained adversary to a max-stable distribution (dashed line) provides an appropriate balance of coverage while maintaining underlying structural properties. This figure demonstrates a case when even if we consider two adversarial formulations that achieve a similar minimum error value, the properly constrained adversary (dashed line) is preferable because the size of uncertainty is very hard to calibrate. Therefore, a curve that is "flatter" around the minimum as a function of the uncertainty size parameter is preferable.




Our work focuses on the case of multivariate extreme value distributions (MEVs) which characterize the joint risk between different variables. Unlike univariate EVDs which have a fully parametric form, MEVs are semi-parametric, where the dependence structure is an infinite dimensional object, and much harder to estimate. We focus on robustifying the dependence structure while preserving the MEV character. This representation leads to additional difficulties which we overcome through the proposed DRO framework.

In view of the fact that the lack of data will naturally induce model error, we introduce an approach to quantify model misspecification based on optimal transport DRO. We select the Wasserstein metric for optimal transport because this approach, together with moment constraints, encompasses most DRO formulations as demonstrated by <cit.>.

The cost structure in the optimal transport discrepancy allows one to balance various objectives when modeling data, specifically the trade-off between tractability and control of pessimism in extremal behavior. A too-powerful adversary makes the size of uncertainty hard to calibrate since a slight increase may result in adversarial policies that perturb a distribution in ways that are too pessimistic and may not be consistent with the constraints imposed by EVT.

We mitigate these concerns by studying DRO formulations which constrain the adversarial perturbations to explore non-parametric models that induce robust estimators while preserving MEV characteristics. One of the approaches we employ is based on optimal transport for point processes, while the other is based on a neural network architecture. We illustrate the performance of the neural network architecture in the context of a Conditional Value at Risk (CVaR) metric applied to a multi-variable extreme value distributed data set. The network is evaluated across several synthetic data sets, specifically constructed to challenge the assumptions associated with EVT. The neural network architecture uses the tractability of the optimal transport distance metric to parameterize model uncertainty. Furthermore, our analysis is extended to a real data set of financial returns as a baseline comparison, similar to the data proposed by <cit.>.This experiment demonstrates precisely the anticipated behavior shown in Figure <ref>, where our methodology leverages a properly constrained adversarial estimator.

Related Work
A number of related research directions exist that consider both estimating EVDs from data as well as robustifying them using DRO. Since our focus is on multivariate EVDs, we will review a few of the estimators from the literature. For estimating multivariate EVDs, copula based approaches have been developed in a variety of instances including work by  <cit.>. Samplers for MEVs have also been considered as demonstrated by  <cit.>. <cit.> provides a flexible framework that uses neural networks to estimate and sample from multivariate EVDs irrespective of dimension, which we use in the computational component of this work. However, all of these methods only consider the case where the model is well-specified and do not consider uncertainty associated with the model class. We build upon these works with the addition of the DRO perspective. 

With regards to the DRO literature, <cit.> described the general framework for Wasserstein DRO that we will use throughout this work. Additional discussion is available in  <cit.> and  <cit.>. Other DRO frameworks for EVT have been considered, but only in the univariate case, by  <cit.>. <cit.> considers DRO under the Kullback-Liebler divergence, which is appropriate for the univariate analysis but may not be appropriate for the multivariate case where supports are likely to be disjoint.

 Finally, <cit.> considers DRO estimators specifically for MEV distributions leveraging extremal coefficient constraints. 
 In this work, upper and lower bounds are established on a Value at Risk (VaR) loss and applied to a real data set of financial returns. 
 These bounds are established over the infinite domain of spectral measures, using a finite set of constraints formulated as a linear semi-infinite program. 
 This is a limited subset of application problems when compared to our investigation. 
 While the general goal of <cit.> is similar to our framework, our method extends to more general risk measures and considers a flexible uncertainty set specified by the Wasserstein distance. 
 Furthermore, we achieve a single robust loss (upper bound), as opposed to less precise (upper and lower) boundaries. 
 In Section <ref> we generate a similar data set to <cit.> and employ our proposed method for comparison. 
 In summary, the main contributions of this paper are: 


  * We introduce a framework to produce DRO estimators for MEV distributions based on optimal transport.


  * We provide tractable DRO formulations for various estimators of interest when adversaries live in the (infinite dimensional) space of point processes.


  * We test the performance of our estimators with the goal of showing that our MEV-constrained adversaries improve performance in the sense of Figure <ref>, and compare to a previous work for baseline analysis.



§ BACKGROUND AND PROBLEM FORMULATION
 
In this section we focus our discussion on concepts critical to our proposed results and we introduce the framework for distributionally robust estimators in MEV distributions.
We first provide a brief overview of EVT and describe how it is used to extrapolate beyond the observed data. 
We then introduce the multivariate counterpart, which we use throughout this work, to describe EVT in multi-dimensional settings.
Finally, we discuss the distributionally robust optimization framework we use based on the Wasserstein distance. 



 §.§ Extreme Value Theory Background


We begin by reviewing concepts surrounding MEV distributions. Consider a sequence of n i.i.d. random vectors {X^(1), , X^(n)}, with X^(i)∈ℝ^d and i = 1, , n and denote the maxima over each dimension as M_k,n:=max_i =1^n X_k^(i), where k ∈{ 1, …, d }. Similarly to univariate EVT analysis, we consider MEV distributions G, where P((M_1,n- b_1,n)/a_1, n≤ z_1, , (M_d,n- b_d,n)/a_d, n≤ z_d) → G(z_1, , z_d), for some normalizing constants a_k, n > 0 and b_k, n, as the number of observations n increases to infinity. 





 §.§ Spectral Representation of Component-wise Maxima and Asymptotic Characterization


We will now introduce specific properties of MEV distributions that we will exploit in our framework. Consider i.i.d. random vectors {X^(1), , X^(n)}, with X^(i)∈ℝ^d and i = 1, , n, with unit Frechét margins such that F_X_k(x) = exp(- 1 /x), with x > 0 for all k = 1, …, d. Following <cit.>, let a sequence N_1, , N_n be a point process, where N_n(·)=∑_i=1^n1_X^(i)/n(·) with N_n(·)d→ N(·) as n →∞ with d denoting convergence in distribution and N is a Poisson point process. We will apply this result further in Section <ref> to define the proposed robustificaiton of the Poisson point process.



  
Decomposing Max-Stable Random Variables
Max-stable distributions can be decomposed according to the radial and spectral decomposition <cit.>. Specifically, if we let Y^(n)∈ℝ^d ∼ H be a sample from the spectral distribution and A^(n) to be the n^th arrival of a unit rate Poisson point process, then a max-stable random variable is represented as


    M = max_n≥ 1Y^(n)/A^(n).


Under the condition that 𝔼[Y_k] =1, the variable M is distributed with unit Frechét margins. This decomposition provides a semi-parametric class of distributions whose structure we will use throughout the rest of the text. With the spectral decomposition of the MEV we are now able to analyze an MEV distribution. Additionally, we introduce Lemma <ref> which allows for the explicit transformation of the MEV cumulative distribution function (CDF). 


    For X^(k) with standard Frechét marginal distributions, we have:
    
    P   (M_1,n/n≤ x_1,⋯, M_d,n/n≤ x_d ) 
       →exp(-V(x_1,…,x_d)),

    where
    
    V(x_1,⋯,x_d)=d∫_Δ_d-1max_k = 1^dw_k/x_kH(dw),

    Δ_d-1 represents the unit d-dimensional simplex and H(·) satisfies: ∫ w_k H(dw)=d^-1 for all k ∈{1, …, d}.


When dealing with MEVs, the important component in estimation is the spectral measure H. When placed in its standardized form, H is a measure with support over the simplex that describes the dependence between the covariates, specifically which components become extreme simultaneously. When a face of the simplex has H non-zero, it implies that covariates associated with the vertices of the face experience extreme events simultaneously. For more details on estimating the spectral measure, we refer the reader to <cit.>. In Section <ref>, we describe how H plays the role of an intensity function when taken in the perspective of the point process.



 §.§ Wasserstein DRO Formulations for EVDs

Referencing the framework to quantify model uncertainty via DRO described by <cit.>, we define the probability space (Ω, ℱ, P), where a candidate robust distribution P is feasible such that d(P, P_0) ≤δ. P_0 is a baseline distribution and d(·) is a distance measure, constrained by the parameter δ. Two methods are commonly employed to quantify model uncertainty when constructing distributional ambiguity sets. The first considers the corruption of the likelihood baseline model to be misspecified, which is addressed via ϕ-divergence ambiguity sets. The second method perturbs the actual data, which leads to the use of Wasserstein distances to quantify model misspecification.  

A recent investigation by <cit.> has demonstrated that both considerations of likelihood and perturbations of data may be unified under the Wasserstein distance, where d(·) = W_c(·). Consider a loss function ℓ : ℝ^d →ℝ_+ and the Wasserstein distance transport cost c: ℝ^d ×ℝ^d →ℝ_+. We define the primal optimization problem as follows:

    max_P : W_c(P, P_0) ≤δ𝔼_X ∼ P[ℓ(X)]

In the context of our problem, P_0 could be the estimated distribution from available samples, but one may not have enough coverage since rare events may not have been observed within the data collection period. Following <cit.>, the dual form of (<ref>) is given by the following problem

    min_λ≥ 0 [ λδ + 𝔼_X ∼ P_0[ max_Zℓ(Z) - λ c(X, Z)  ]  ].

As we will discuss, the dual form is particularly amenable for computational purposes, especially when one can exploit specific properties of c, ℓ, and P. Our investigation will describe two perspectives of (<ref>) for EVDs. The first we present in Section <ref>, and is based on interpreting an MEV through a point process and the second is based on the interpretation using a copula. Each perspective has different properties that are useful for computational purposes, which we will discuss in Section <ref>. 


§ ROBUSTIFICATION OF THE POISSON POINT PROCESS


In this section, we provide a more general formulation for DRO problems using a characterization with a point process. 
For the probability space (Ω, ℱ, P), we denote a counting measure N(·) on a Polish space (S,d). 
We define N(·)=∑_i=1^∞1_x_i(·) and N'(·)=∑_i=1^∞1_y_i(·) as two random counting measures. 
Additionally, we define scaling functional κ: ℤ→ℝ_≥0, and a lower semi-continuous cost function c:S× S→ℝ_≥0. 
Let σ(·) denote the permutation function defined on the support of N(·), and we then define the distance as:

    c(N(·), N'(·))=   ∞1_N(S)≠N'(S) or N(S)∨ N'(S)=∞
       + κ(N(S))inf_σ(·)∑_i=1^N(S)c(x_i, y_σ(i)).


This distance is also similar to the ones presented by <cit.>. Additionally, we define the 1-Wasserstein distance between two random counting measure by:

    W_c(μ,ν)=inf_γ∈Γ(μ,ν)𝔼_(N, N')∼γc(N, N'),

Here μ is the measure corresponding to N(·) while ν is the measure corresponding to N'(·). Analogously to standard DRO problems, we can write down a new formulation for this DRO problem with point process as follows:


Let N(·) denote a Poisson point process, f(·) denote a Borel function, and W_c(P, P_0) denote the Wasserstein distance between measures induced by N' and N under cost metric c. For a distance δ≥ 0, the following problems are equivalent:

    sup_ W_c(P, P_0) ≤δ𝔼_N' ∼ P[ℓ(N'(f))] = 
    inf_λ≥ 0   {λδ  + 𝔼_N∼ P_0 [sup_N' [ ℓ(N'(f) ) - λc(N, N') ] ] },

where N(f):=∫ f(x)N(dx).

    This follows from the application of <cit.>. 


As mentioned in the background material in (<ref>), MEV observations are given by the product of  radial (A^(n)) and spectral (Y^(n)) components. In the perspective of the point process, the associated measures for each of these components forms the intensity of the point process. Specifically, we let the atoms of the point process be given by (A^(n), Y^(n)) where A^(n) can be thought of as a time coordinate (since A^(n) is the n^th arrival time) and Y^(n) as a space (or the mark) component. Define N(da, dv) = ∑_n=1^∞1_((A^(n), Y^(n)) (da, dv). This provides a useful characterization in terms of the dual problem insofar as we can optimize over the boundaries of the integrals to form our adversary point process. As we will illustrate, this is equivalent to finding a new point process with modified intensity. 
 
With regards to the cost in the Wasserstein distance, we place an infinite cost on changes in the arrival times of the events. This has the interpretation that our uncertainty is not on the arrival time of the tail events but rather on the dependence. We formalize this through the following cost function between two points:

    c((s, u), (t,v)) = ∞1_s ≠ t + |u - v| 1_s = t

Our Wasserstein cost is then between the spectral component while fixing the radial component. We formalize this case in the following corollary.

    
    For the given setting in Theorem <ref>, if the point process satisfies N(x)=lim_n∑_i=1^n1_X^(i)/n(x), where X^(i) are i.i.d. standard Frechét margins. Then, we can rewrite the primal problem as:
    
    sup_W_c(P, P_0)≤δ𝔼_N∼ P[ℓ(N(f∘ T^-1)],

    where N(da, dv)=∑_n1_(A^(n), Y^(n))(da, dv) and T(X)=(1/X_1, X/X_1). 
    Under P_0, A^(n) is the n-th arrival of the Poisson point process with intensity 1 and Y^(n) follows the spectral measure H(·). 
    Here we introduce the composition of the Borel function f on the transformation T.

 



§ OPTIMIZATIONS FOR SPECIFIC LOSS FUNCTIONS


Having introduced the different perspectives of the optimization, we provide specific loss functions where each perspective is particularly conducive towards specific robustification cases.


 §.§ Robustifying the CDF


Consider the cumulative density function (CDF) of M∈ℝ^d satisfying M_i=max_n=1^∞Y_i^(n)/A^(n), where Y^(n)∈ℝ^d are i.i.d. random vectors and A^(n) is the n-th arrival time of a unit Poisson point process. 
The CDF of M satisfies:

    P(M_1 ≤ x_1^-1,    …,  M_d ≤ x_d^-1 ) = 
       exp ( - d∫_Δ^d-1max_i=1^d x_i w_i H(dw)).

Here, Equation <ref> is derived from Lemma <ref> by using the max stable decomposition of the random variable, as demonstrated in Equation <ref>. 
We will now exploit specific geometric properties of the point process to derive a dual form. 
Given the definition of M_i, we may make a transformation of the CDF algebraically as follows:

    P     (M_1 ≤ x_1^-1, …,  M_d ≤ x_d^-1 )= 
        P(max_k=1^d Y_k^(n)x_k≤ A^(n),∀ n ) :=P(V_x^(n)≤ A^(n),∀ n).

In this case, we can reduce our analysis to this two dimensional space and consider the robustification of the point process on this space, rather than the full d dimensional space of the Y^(n)'s. 
A schematic of this representation before robustification is given in Figure <ref>. Using this form leads to a dual form given by the following theorem:


Consider the setting in Theorem <ref> and the loss function ℓ(X) = 1_X≥ 1, the point process N(·):=∑_n=1^∞1_(A^(n),V^(n)_x)(·), the function f(a,v)=1_𝒞(a,v) and 𝒞={(a,v)∈ℝ^2 | a≥0,a≤ v}. Then, the dual objective is:

    inf_W_c(P, P_0)≤δP'(M_1 ≤ x_1^-1, …,  M_d ≤ x_d^-1 )
    
    =   1-inf_λ≥ 0λδ + 𝔼_P_0 [1 - λmin_n≥ 1( A^(n)- V_x^(n))^+  ].



See Appendix <ref>.


This formulation is advantageous in many ways. First, we reduced the original optimization problem over measures to one over a scalar variable λ. Second, this formulation includes the relevant constraints such that the perturbed distribution maintains the max-stability property of EVT. Third, the optimal solution is given by a shift in the rate function, as illustrated in the following corollary: 


The minimizer of the problem in Theorem <ref> has the following form:

    inf_P:W_c(P,P_0)≤δP(X_1 ≤ x_1^-1, …, X_d ≤ x_d^-1) 
       = exp ( - 𝔼_N∼ P_0 [N( 𝒞_1/λ^⋆)]  )

where 𝒞_1/λ^⋆ is the region 
{(a, v) : V_x^(n) > A^(n) - 1/λ^⋆}
and λ^⋆ is the minimizer of (<ref>).
 
 
     See Appendix <ref>.
 

The ideas behind Corollary <ref> are illustrated in Figure <ref> where the region over which the expectation is taken is shifted by 1/λ^⋆.

 

 §.§ Probability an Event in a Rare Set Occurs

In this part, we describe a generalization of the CDF case where we are interested in robustifying the probability that an event in the set A ⊂ℝ^d_+ occurs, i.e. P(N(A) ≥ 1). Our primal problem in this case is:
 
    sup_W_c(P, P_0) ≤δ P(N(A) ≥ 1) = sup_W_c(P, P_0) ≤δ𝔼_X ∼ P[1_X ∈ A]

 and we let the distance be any ℓ_p-norm.
 We can write the dual form and obtain the following simplification:
 
    Consider the setting in Theorem <ref> and the loss function ℓ(X)=1_X≥1, the point process N(·)=∑_n=1^∞1_X^(n)(·), and the function f(x)=1_A(x). Then the dual objective is:
    
    sup_W_c(P, P_0) ≤δ P(N(A) ≥ 1) 
    
        =   inf_λ≥ 0{λδ +  𝔼_P_0 ( 1- λinf_ninf_y ∈ A y - X^(n))^+ } .

    
 
  
 See Appendix <ref>.
 

 Relatedly, we can also compute the expected number of events to occur within a set using the following theorem:
 
     Consider the setting in Theorem <ref> and the loss function ℓ(X)=X, the point process N(·)=∑_n=1^∞1_X^(n)(·), and the function f(x)=1_A(x). The the dual objective is:
     
    sup_W_c(P, P_0) ≤δ𝔼_N ∼ P[N(A)]
    
             =   min_λ≥ 0 [ λδ + 𝔼_P_0∑_n=1^∞ ( 1 - λinf_y ∈ A y - X^(n))^+  ].

 
 

 §.§ Conditional Value at Risk

Conditional value at risk (CVaR) is a commonly used risk measure in finance. Often there is uncertainty around the calculation, so we derive an efficient form for the calculation. In the DRO case, we can write the CVaR problem with the given level α as:
 
    sup_P: W_c(P,P_0)≤δinf_z𝔼_P((∫⋁_k=1^d x_k N(dx)-z)^+/1-α+z).

 By exchanging the sup-inf to inf-sup, we can write the dual formulation as the following theorem:

 
      Consider the setting in Theorem <ref> and the loss function ℓ(X)=(X-z)^+/1-α+z, the point process N(·)=∑_n=1^∞1_X^(n)(·), and the function f(x)=x_∞. The the dual objective is:
    
    inf_zsup_P: W_c(P,P_0)≤δ𝔼_P((∫⋁_k = 1^d x_i N(dx)-z)^+/1-α+z)
    
             =   δ/1-α + 𝔼_P_0 [ ∑_n=1^∞X^(n)_∞ | ∑_n=1^∞X^(n)_∞ > q_α. ],

    where P_0(∑_n=1^∞X^(n)_∞≤ q_α)=α and q_1<∞.
 
 
 See Appendix <ref>.
 
Having introduced these simplifications, we see that the optimization problems for commonly used losses under the point process interpretation of EVT can be made feasible.
 



§ EXPERIMENTS AND RESULTS


We now consider empirical validation of the proposed optimization schemes using two experiments. The first uses a synthetic data set constructed to pathologically challenge our model and represent real world risk scenarios. 
The second validation is a baseline comparison using a real data set of financial returns similar to the DRO MEV experiment proposed by <cit.>. 
For the synthetic and real data experiments the observations are extreme; however, the standard EVD model is shown to not provide appropriate coverage for computing the worst-case risks. 
For both data sets, we provide empirical results validating that the proposed method satisfies the properties that we desired and outlined in the introduction, i.e. those of sufficient coverage while not being too conservative.




 §.§ Computational Implementations

For losses that have a reduced dual problem such as those in Section <ref>, we use the empirical data to represent the expectations and optimize over λ.
However, for general losses we use the equivalence stated in Corollary  <ref> to estimate the adversarial distribution.
Our procedure follows the method in <cit.> where we first estimate the spectral measure from the observations using a generative neural network and then optimize over the space of networks within some ball of the initial distribution.
The case for arbitrary ℓ is illustrated in Algorithm <ref>.







 §.§ Experiments with Synthetic EVT Data


We evaluate our proposed methodology on two synthetic two-dimensional EVD data sets.
 
We hypothesize that a properly formulated DRO approach, where max-stability is enforced via constraints, demonstrates a greater invariance to increased model misspecification relative to a non-constrained models. Our methodology is designed to account for the complexities of real data and produce more accurate and reliable predictions. 
Details concerning computational parameters, runtime, and code base reference is available in Appendix <ref>.



  §.§.§ Synthetic Datasets


For our synthetic data experiments, we consider mixture distributions where one mixture component with a more concentrated risk appears less frequently than the other mixture component. This is illustrated in Figures <ref> and  <ref>, where the ten thousand observations of the two datasets are visualized. In Figure <ref> the component has a smaller tail index and more realizations with smaller magnitude whereas in Figure <ref> the component has a larger tail index, but is more rare.  
For more information see Appendix <ref>.

A standard EVD approach would be sufficient when modeling a non-mixture Symmetric Logistic (SL) or Asymmetric Logistic (ASL) set of data. However, this mixture inserts a pathological modeling issue, which emulates the practical concerns often observed in real data where extrapolating to out-of-sample extreme values is difficult given the scarcity of data in these scenarios. 
These generated mixture distributions exhibit different tail behavior, which make it difficult for standard approaches to capture the dependencies accurately, and thus we turn to the proposed DRO estimator and evaluate our ability to recover the true risk.








  §.§.§ Synthetic Experiment Results



We now illustrate the results of the computational method for the loss corresponding to CVaR of the ℓ^1-norm of the observations.
Specifically, we aim to robustify the calculation of 1/α𝔼[X_1 1_X_1 ≤ x_α] where x_α = inf{ x : P(X_1 ≤ x) ≥α}.
As α→ 0, this corresponds to the CVaR for values deep in the tail.
We approximate this region as a MEV under both the SL and ASL distributions described previously. 
Our goal in this experiment is to understand the behavior of the error, defined as the difference between the true risk and the risk computed using the different methods, as a function of the budget δ.
We compare 3 different robustifications: (1) a totally unconstrained robustification where P can be in any class of distributions; (2) an EVT constrained case where P must also be an MEV and shares the same A^(n)'s; (3) an EVT constrained case where the margins are assumed to be well calibrated and only the dependence is modified (such that w ∼ P,   s.t.  𝔼[w_i] = 1/d).
In terms of standard EVT methods, (3) corresponds to the case where the adversarial distribution has a different dependence structure but maintains the same margins.
Specifically, this would suggest estimating the margins is easier and therefore should be conserved, but the dependence structure should be perturbed.
Figures <ref> and <ref> illustrates the error of using a DRO approach with max-stability constraints compared to the unconstrained approach, producing a distribution which is not of the MEV model class as functions of δ. 
This allows for the understanding of performance based on the behavior of the loss function as δ changes. 



  
Synthetic Experiment Results Analysis
In Figures <ref> and <ref>, we visualize that the models which are constrained to be max-stable, specifically P_⋆ and P_⋆ (𝔼[w_i] ≈1/d), demonstrate a lower and robust (positive) error when compared to the unconstrained model P_⋆ (unc). Additionally, we observe a slight improvement in the error of the constrained P_⋆ model for an increase of model uncertainty (δ) and subsequent monotonically increasing error as  uncertainty increases. This behavior confirms our conceptual illustration of the DRO max-stable constrained approach in Figure <ref>.




  §.§.§ Convergence of CDF as Distance is Increased

We now consider an experiment that corresponds to Theorem <ref> described in Section <ref>.
We illustrate the behavior in Figure <ref> where we see an improvement in the error for a wide range of values of δ. 





 §.§ Experiment on High Dimensional Financial Data


We consider a dataset on financial returns of equities in the S&P 500 index similar to the observations used in the validation experiments proposed by <cit.>. 
The dataset is composed of all current members of the S&P 500 who have been members since at least January 1983 to provide forty years of equity price data. 
We then average returns across companies, where each company is uniquely assigned to one of eleven industrial sectors (industrial, health care, consumer staples, etc.), leading to an eleven dimensional dataset. 
Maximum daily returns are taken both annually and weekly across the forty year data set, where each of these annual and weekly maximum data sets now constitute extreme observations.

This dataset is in the dimension of the number of industries considered, in this case d = 11. 
A visualization of this extreme data for highest weekly and annual returns is provided in Appendix <ref>. 
Obtaining a ground truth risk for real data is nearly impossible since we may never observe realizations deep in the tails, so validating against a ground truth risk is difficult.
To circumvent this, we consider training on less extreme data given by maxima over shorter time scales and use maxima over longer time scales as the ground truth risk.
Specifically, we fit the models using weekly maxima and test on yearly maxima.
The reason for this is that large events are more likely to occur over the longer time scale (the year) than over the day, leading to a more representative set of measurements deeper in the tails. 

We use the exact same methodology outlined in Section <ref> and illustrate the results of this experiment in  Figure <ref>.



  
Real Data Experiment Results Analysis
We observe that when δ = true risk, a minimal error is achieved between the true and calculated risk.
Additionally, this experiment demonstrates that for a real extreme data set the models constrained by EVT achieve a lower error level when compared to the unconstrained model, as anticipated.
We note that the comparable formulation in <cit.> does not provide a modifiable parameter such as our formulation does with δ, to adjust model uncertainty. 
This demonstrates the additional value of our proposed formulation over a comparable method. 



§ DISCUSSION

In this paper, we introduced a framework for robustifying MEV distributions according to the point process viewpoint. 
We provided a novel framework and discussed simplifications that are possible under certain cost functions. 
We numerically demonstrated that, as we proposed, adding additional constraints from EVT improves the robustification without being too conservative. 
The framework is evaluated on a synthetic data set which is constructed to challenge the model and a high-dimensional real world data set derived from a similar investigation, demonstrating expected and improved behavior. 
The proposed methodology has a variety of applications in many fields, particularly in risk sensitive settings where model misspecification may be present. 

There are many avenues for extending the proposed methodology.
One important extension involves the case where one must specify a policy to mitigate a worst-case risk.
This applies in, for example, portfolio optimization problems where the optimal portfolio minimizes the conditional value at risk over a distribution of portfolio losses.
Additionally, further exploration of the proposed formulation leveraging max-stable processes extended to a more general max-infinitely division process may be insightful. 



  
Limitations
There exist a number of limitations associated with the proposed framework. 
With regards to the computational method, there may be numerical instabilities associated with the optimization particularly in cases of δ being small. 
This is because the λ parameter must grow large, which leads to unstable optimization schemes.



 
Material in this paper is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-20-1-0397.










Distributionally Robust Optimization as a Scalable 

           Framework to Characterize Extreme Value Distributions 
 (Supplementary Material)
    Alex Zunger
    August 5, 2024
================================================================================================================================================





§ PROOFS






 §.§ Proof of Equivalence in Representations


Let us first recall the two problems that we are solving. 
On one hand, we take an explicit point process view point in:

    max_ W_c(P, P_0) ≤δ𝔼_N ∼ P[ℓ(N(f))],
    Point Process

where N(dx)=∑_nδ_X^(n)(dx) is a non-homogeneous Poisson process. By the properties of EVT, the intensity satisfies

    𝔼[N(f)] = ∫ f(T^-1(r,w)) r^-2dr H(dw).

By change of variables, we could decompose X^(n)=Y^(n)/A^(n), where A^(n) is the n-th arrival of Poisson process with intensity 1 and Y^(n)∼ H(·). Thus, we could rewrite the problem with:

    max_W_c(P, P_0)≤δ𝔼_N'∼ P[ℓ(N'(f∘ T^-1)],

where N'(dt, dy)=∑_n1_(A^(n), Y^(n))(dt, dy) and T(X)=(1/X_1, X/X_1).



 §.§ CDF proof




We now detail the proofs for specific loss functions, beginning with the cumulative distribution function (CDF), followed by the conditional value at risk (CVaR), probability, and expected number of events in rare sets. 


 
 Define the upper triangular region as 𝒰 = { (a, v) :  a ≥ 0, v ≥ a }.
 To compute the CDF, we want to condition on the Poisson process having no points within 𝒰, i.e. N(𝒰) = 0. 
 We can compute the rate of the point process according to 𝔼[N(𝒰)]. 
 Since this is a Poisson point process, the probability that no points occur within 𝒰 is given by
 exp(- 𝔼[V_x]).
 Having written the CDF in terms of the rate of the Poisson point process, we now follow through to describe the simplified form written in (<ref>).
 First, we write the dual problem when directly applying <cit.> as
 
    min_λ≥ 0{λδ + 𝔼_N ∼ P_0 [max_N'1_N'(𝒰) ≥ 1 - λc(N, N')  ] }.

Conditioning directly on the first inequality, we can rewrite this as
 
    min_λ≥ 0{λδ + 𝔼_N ∼ P_0 [1- λmin_N'(𝒰) ≥ 1c(N, N')  ]^+ }.

 Here we also set c((s,u), (v,t))=|u-v|1_s=t+∞1_s≠t. Moreover, we notice the equivalent representation between X^(n) and (A^(n), V_n(x)), which means we can write N(dx) = ∑_n1_X^(n)(dx)=∑_n1_(A^(n), V_n(x))(dt, dv). Thus, if N(𝒰)≥ 1, it means ∃ n s.t. V_n(x)-A_n≥0, we can simply set N'(·)=N(·) to obtain the minimum of min_N'(𝒰) ≥ 1 c(N, N'). On the other hand, if N(𝒰)=0, it means ∀ n, V_n(x)-A_n<0. Then we denote δ=min_n(A^(n)- V_n(x)) and n^*∈min_n(A^(n)- V_n(x)) and set N'(dt,dv)=1_(A^(n^*), V_n^*(x)+δ)(dt,dv)+∑_n≠n^*1_(A^(n), V_n(x))(dt,dv), which guarantees N'(𝒰)≥ 1, and the minimum of min_N'(𝒰) ≥ 1 c(N, N') is exactly δ.  To conclude, we have the following equivalence holding:
 
    min_N'(𝒰) ≥ 1 c(N, N') = min_n≥ 1 [ A^(n) - V^(n)_x]^+.

Substituting, we obtain the desired result of (<ref>):
 
    min_λ≥ 0λδ + 𝔼_N∼ P_0 [1 - λmin_n≥ 1( A^(n)- V_x^(n))^+  ]^+






  §.§.§ Minimizer of CDF Proof


 
 Following where we ended in the proof of Theorem <ref>, 
Denote f(A,V_x):=min_n≥ 1( A^(n)- V_x^(n))^+. By first order condition of the Equation (<ref>), we have:
 
    δ=𝔼_N∼ P_01(f(A,V_x)≤1/λ^*)f(A,V_x).

 Substituting it to objective, we have:
 
    min_λ≥ 0λδ + 𝔼_N∼ P_0 [1 - λmin_n≥ 1( A^(n)- V_x^(n))^+  ]^+
       =P_0(f(A,V_x)≤1/λ^*).

 Thus, we have:
 
    min_P:W_c(P,P_0)≤δP(X_1 ≤ x_1^-1, …, X_d ≤ x_d^-1)
       =1-P_0(f(A,V_x)≤1/λ^*)
       =P_0(f(A,V_x)>1/λ^*)
       =P_0(N(𝒞_1/λ^*)=0)
       =exp(-𝔼_P_0N(𝒞_1/λ^*)  )

 



 §.§ Proof of Rare Region Probability

 

 
     We write the primal problem as 
     
    sup_D(P, P_0) ≤δ P(N(A) ≥ 1)

     and writing it in the dual form we get
     
    inf_λ≥ 0{λδ + 𝔼_P_0 [sup_Z^(n)( sup_n=1^∞1_Z^(n)∈ A - λ∑_n=1^∞ Z^(n) - X^(n) )  ]}.

     Replacing the inner maximization, we can write
     
    sup_Z^(n)( sup_n1_Z^(n)∈ A - λ∑_n=1^∞ Z^(n) - X^(n) ) = ( 1- λinf_ninf_z ∈ Az - X^(n))^+,

     which is due to:
     
    sup_Z^(n)( sup_n1_Z^(n)∈ A - λ∑_n=1^∞ Z^(n) - X^(n) ) = ( 1- λinf_∃ n, Z^(n)∈ A∑_n=1^∞Z^(n) - X^(n))^+.

     And solving inf_∃ n, Z^(n)∈ A∑_n=1^∞Z^(n) - X^(n), we have it equals inf_ninf_z ∈ Az - X^(n) as the region only requires one sample z^(n)∈ A and the others can be equal to corresponding X^(n).
 

 

 §.§ Proof of Number of Rare Region Occurrences


    We begin with the dual definition
    
    inf_λ≥ 0{λδ + 𝔼_N ∼ P_0 [sup_ Z^(1)⋯ Z^(N) ( ∑_n=1^∞1_Z^(n)∈ A  - λ∑_n=1^∞ Z^(n) - X^(n))  ]}.

    Rewriting the part within the sup, we obtain the desired result, since
    
    sup_Z^(1)⋯ Z^(N) ( ∑_n=1^∞1_Z^(n)∈ A  - λ∑_n=1^∞ Z^(n) - X^(n)) = ∑_n=1^∞( 1- λinf_z ∈ Az - X^(n))^ +.

    



 §.§ CVaR proof

 

 

At first, we consider the finite point case, where N(·)=∑_n=1^N δ_X^(n)(·). The CVaR problem is given as
 
    min_z max_D(P, P_0) ≤δ𝔼_P (  ( ∫⋁_i=1^d x_i N(dt, dx) - z )^+ /1- α+ z )_primal problem.

 Taking the primal problem component, we can write the dual problem as:
      
    min_λ≥ 0{λδ + 𝔼_P_0 [ max_Z^(n)( ∑_n=1^∞ Z^(n)_∞ - z )^+ - λ∑_n=1^∞ Z^(n) - X^(n)_∞ ] }
       = min_λ≥ 1{λδ + 𝔼_P_0 ( ∑_n=1^N  X^(n)_∞ -z)^+  }   (Lemma <ref>)
       = δ + 𝔼_P_0 ( ∑_n=1^N  X^(n)_∞ -z)^+ .

     Adding the minimization over z, we obtain
     
    min_z [ δ + 𝔼_P_0 ( ∑_n=1^N  X^(n)_∞ -z)^+/1-α + z   ] .

Taking the minimum, we get

    δ/1-α + 𝔼_P_0 ( ∑_n=1^N  X^(n)_∞|∑_n=1^N X^(n)_∞≥ q_α )

where P_0 ( ∑_n=1^N  X^(n)_∞≤ q_α) = α. Finally, as q_1<∞, the summation converges a.s., we can safely let N→∞ and obtain the final result.
 

    


    
        For x∈ℝ^d_+ and λ≥0, we have:
        
    max_y_1⋯ y_N∈ℝ^d_+(∑_n=1^N y_n_∞-z)^+-λ∑_n=1^Ny_n-x_n_∞=
                ∞,    when λ∈[0,1),
    (∑_n=1^N x_n_∞-z)^+,   when λ∈[1,∞).

    
    
        For λ∈[0,1), we claim the objective value would diverge to infinity. In fact, we let y_n=x_n for n≥2. The objective becomes:
        
    max_y_1∈ℝ^d_+(y_1_∞+∑_n=2^Nx_n_∞-z)^+-λy_1_∞.

        Then, we can pick y_1 such that y_1_∞≥ z-∑_n=2^Nx_n_∞ and letting it goes infinity, we find the objective will also go infinity.

        For λ∈[1,∞), we have:
        
    (∑_n=1^N y_n_∞-z)^+-λ∑_n=1^Ny_n-x_n_∞   ≤(∑_n=1^N y_n-x_n_∞+x_n_∞-z)^+-λ∑_n=1^Ny_n-x_n_∞
       =(1-λ)∑_n=1^Ny_n-x_n_∞+(∑_n=1^N x_n_∞-z)^+.

        Thus, maximizing over y, we have the upper bound for the objective is (∑_n=1^N x_n_∞-z)^+. And this value is attained when y_n=x_n for n=1,⋯,N.
    
 

 



§ DATA DESCRIPTION


Here we provide more details on the data distributions used.
The symmetric logistic (SL) and asymmetric logistic (ASL) distributions are commonly used parametric models that have the property of max-stability.
We describe some of their properties and how they are applied in the mixture model.
We use the sampling algorithms described in <cit.> to generate all the datasets.

As mentioned, for computing the cumulative distribution, the central object of interest is the spectral measure. 
In each of these datasets, the spectral measure is given by a mixture of two spectral measures.
Our baseline model P_0 is fit to the mixture without consideration that the model contains two components and results in model misspecification.



 §.§ Symmetric Logistic Mixture

For these experiments, we consider a mixture model with two components: one that is almost completely dependent and another that is almost completely independent. 
The component with dependence is sampled with lower probability than the one with independence. 
Therefore, a naive estimator would largely concentrate on the component with independence than the one with dependence. 
The dependence function for this distribution is given by

    A_SL(w) =  (∑_i=1^d w_i^1/α )^α,   α∈ (0, 1].

Notice that the influence α is equal for all components of the vector.
We specifically sample the distribution with large dependence (α=0.1) with probability 0.1 whereas we sample the distribution with small dependence (α = 0.9) with probability 0.9.



 §.§ Asymmetric Logistic Mixture

While the SL distribution is symmetric and all components have the same level of dependence, the ASL can impose specific dependencies between any subset of variables.
Denoting 𝒫_d as the power set of {1, …, d }, the dependence function for this family of distributions is given by

    A_ASL(w) = ∑_b ∈𝒫_d (∑_i ∈ b ( λ_i, b w_i)^1/α_b )^α_b,    w ∈Δ_d-1, α∈ (0, 1), λ_b ∈Δ_|b|-1.


We then consider a mixture of two different components, one again with a heavier tail and another with a lighter tail.
This is given by setting λ to be the same random point on the simplex for both mixture components but then considering α = 0.1 and α=0.9 for the components with high dependence and low dependence, respectively. 



 §.§ Financial Data and Visualization


Below we've provided a visualization of the maximum daily returns taken over weekly blocks, and maximum daily returns taken over annual blocks. The returns are provided for the eleven industries, where the values are averages taken across selected companies uniquely belonging to each industry. To compute the average of each industry, we first selected the 93 members of the S&P 500 that have been listed on the index since January 1983, in order to provide forty years of financial data. Each of these company's daily returns were averaged across their assigned industry to create the average daily returns for each industry over forty years.





§ ALGORITHM DETAILS

Here we provide additional details that were left out of the main manuscript. 
Specifically, we go over the Maximize and Minimize steps in greater detail.
The basic idea is we have an iterative training algorithm similar to the min-max problems in generative adversarial networks.
λ is initialized according to 1/δ + ε for some small ε. 
P is initialized according to the same parameters as P_0.
For each of the K iterations, we

Maximization step:
We fix λ to be the value from the last iteration.

    
  * Sample N points from the base distribution X^(n)∼ P_0,    n=1,…,N to obtain the X points from the base distribution.
    
  * Sample N points from the current estimate of the adversarial distribution P to get X̃^(n)∼ P,    n = 1 … N.
    
  * Compute ℒ for each X^(n), X̃^(n) which results in a N × N matrix of values, i.e. 
    
    𝐋 :=
        [ ℒ_λ(X̃^(1), X^(1))                  ⋯ ℒ_λ(X̃^(1), X^(n));                  ⋮                  ⋱                  ⋮; ℒ_λ(X̃^(n), X^(1))                  ⋯ ℒ_λ(X̃^(n), X^(n));                    ]

    
  * Take the max over the X̃^(n) dimension, i.e. max_i𝐋_i, ·
    
  * Average over the remaining Y^(n) dimension, i.e. R = 1/N∑_j=1^N max_i=1^N 𝐋_i, j.
    
  * Compute the gradient of R with respect to the parameters of P, update the parameters of P via gradient descent.

Adding different constraints to the problem results in different forms of the Ỹ, which we describe in the next section.


Minimization step:
Now we fix the parameters of P and only minimize over λ.

    
  * Compute R̂ = λδ + 1/N∑_j=1^N max_i=1^N 𝐋.
    
  * Compute gradient of R̂ with respect to λ, update according to gradient descent.

We next describe a few algorithmic considerations of the maximization step.



 §.§ Unconstrained Case

When sampling from an MEV, one samples according to A^(n), Y^(n),    n=1,…, N, N ≫ 0.
Then samples are generated according to max_n=1^N Y^(n)/A^(n) (e.g. see <cit.> for additional considerations).
In the unconstrained problem, we let the X^(i) in (<ref>) be the MEV, i.e. X^(i) = max_n=1^N Y^(i,n)/A^(i,n). 

Then, P does not use any of the factorization of the Y^(n)/A^(n) and is free to be whatever distribution maximizes the quantity.
The problem is unconstrained because it does not consider any of the constraints given by EVT, specifically that the factorization in terms of the radial A and spectral component Y must hold.


 §.§ Constrained Case

On the other hand, we consider the explicit factorization in the constrained case.
Here we share the A^(n) across both distributions of P, P_0. 
Then, X remains as described in the unconstrained case but X̃ is given by X̃^(i) = max_n=1^N Ỹ^(i,n)/A^(i,n) where the Ỹ^(i,n) are sampled from the estimated P distribution.
Note that the A^(n) are the same for the calculation of X and X̃.
The constraints then inform the adversarial distribution that it must be max-stable and that the radial component is consistent between both.



 §.§ Computational Details and Code Reference


The code base for the DRO MEV model, written in python, is available at the following github link:  <https://github.com/patrick-kuiper/mev_dro>. Below, in Table <ref>, we provide the run time for each of the synthetic data experiments: Unconstrained, Constrained Extreme Value Distributed (EVD), and Constrained Extreme Value Distributed: with Unit Margin (EVD-SM). We chose to highlight the synthetic data sets as they represent the most challenging experiments computationally. Each of these experiments were run individually on NVIDIA GeForce RTX 3090 GPUs with CUDA Version 12.2. Each experiment was run for 2001 epochs.  






 Paper Title: GNUMAP: A Parameter-Free Approach to Unsupervised Dimensionality Reduction via Graph Neural Networks.


A Group Theory Proof of Pascal's Theorem
    Kaylee Wiese
    
========================================






§ ABSTRACT 

With the proliferation of Graph Neural Network (GNN) methods stemming from contrastive learning, 
unsupervised node representation learning for graph data is rapidly gaining traction across various fields, from biology to molecular dynamics, where it is often used as a dimensionality reduction tool. However, 

there remains a significant gap in understanding the quality of the low-dimensional node representations these methods produce, particularly beyond well-curated academic datasets. To address this gap, we propose here the first comprehensive benchmarking of various unsupervised node embedding techniques tailored for dimensionality reduction, encompassing a range of manifold learning tasks, along with various performance metrics. We emphasize the sensitivity of current methods to hyperparameter choices — highlighting a fundamental issue as to their applicability in real-world settings where there is no established methodology for rigorous hyperparameter selection. Addressing this issue, we introduce GNUMAP, a robust and parameter-free method for unsupervised node representation learning that merges the traditional UMAP approach with the expressivity of the GNN framework. We show that GNUMAP consistently outperforms existing state-of-the-art GNN embedding methods in a variety of contexts, including synthetic geometric datasets, citation networks, and real-world biomedical data — making it a simple but reliable dimensionality reduction tool. 






A Group Theory Proof of Pascal's Theorem
    Kaylee Wiese
    
========================================





§ INTRODUCTION

Consider the following biological problem: given a set of gene expressions at various locations within a tissue sample (for instance, a slice of mouse brain tissue), how can we aggregate information to discover spatial domains with coherent gene expression patterns? Recent methods, including <cit.> from which this example is adapted, have turned towards using a graph-based approach — and subsequently, Graph Neural Networks (GNNs) — to resolve this conundrum. In this setting, the data is first represented as a graph 𝒢 on n nodes corresponding here to the various spatial locations within the sample, each endowed with a feature vector X_v ∈^d  (the gene expression data). Under this new formalism, Graph Neural Networks (GNNs) <cit.> come as a natural tool for visualization and the subsequent discovery of new patterns in the data. Through the use of recursive neighborhood “convolutions”, GNNs allow indeed the creation of rich node representations that capture topological information, feature data and essential neighborhood properties, and integrate this information in a Euclidean vector representation that is amenable to any downstream machine learning task.


Unsupervised Learning on Graph Data
While the early rise in Graph Neural Network (GNN) advancements predominantly emphasized supervised learning, there has been growing interest in developing unsupervised GNN methods for learning node representations. This shift is driven by the scarcity of labeled data in numerous real-world situations, such as in the example described above, along with an increasing demand for techniques that facilitate exploratory data analysis and dimensionality reduction for graph data. However, there seems to have been a concurrent development of unsupervised learning approaches in the application community <cit.> and in the method community <cit.>. This parallel development may underscore a disconnect between the approaches devised by the methods community and the practical reality of real-world data. Specifically, the deployment of state-of-the-art approaches evaluated on academic benchmarks appears to be impeded by two significant challenges: (a) Ease of deployment, and (b) Trustworthiness of the learned representations.

(a) Ease of deployment. From a methodological point of view, a growing number of recent approaches are leaning toward the adoption of self-supervised contrastive learning frameworks to effectively represent nodes <cit.>. In this setting, the trick usually consists of perturbing the input data (e.g., by masking features with probability p_f, dropping edges with probability p_e) to create two modified versions of the original data. Each of these perturbed versions of the data is then processed by a Graph Neural Network, which is trained to identify pairs of node embeddings from the two perturbed graphs that correspond to the same node in the original dataset. Examples of such training objectives include the loss proposed by <cit.> for their method GRACE, ℒ = 1/2N( ∑_i=1^N [ ℓ( u_i,v_i)+ℓ(v_i,u_i)] ), 
where
ℓ(u_i,v_i)=log(e^s(u_i,v_i)/τ/∑_k=1^N[ 1_{k≠ i} e^s(u_i,u_k)/τ+e^s(u_i,v_k)/τ]), s:ℝ^p×ℝ^p→ℝ is the cosine similarity function,  and u_i,v_i∈ℝ^p are the node representation for node i stemming from the two perturbed versions of the graph. Finally, τ>0 is a temperature parameter that must be tuned. Alternatives include the loss used in CCA-SSG <cit.>, ℒ=∑_i=1^nu_i - v_i^2+λ(  U^T U- I _F^2+ V^T V-I _F^2 )
where U,V∈ℝ^N× p are the node representation matrix of two views, λ>0 is a hyperparameter, and ·_F denotes the Frobenius norm. 
While these self-supervised learning losses have achieved state-of-the-art performance in a number of academic benchmarks, few of these methods have yet been deployed in applied settings.

One hypothesis explaining this gap is the heavy reliance of these "state-of-the-art" approaches on the correct choice of hyperparameters (for instance, λ, τ, p_f or p_e in the approaches described above). We exemplify this phenomenon through an example case in Figure <ref>, where we propose to deploy CCA-SSG <cit.> using a simple 2-layer GCN <cit.> to learn a 2D visualization of the nodes. We then assess the quality of the learned embeddings by learning a support vector classifier to classify the nodes, and evaluate the performance of the classification on held-out data. We note a substantial variation in embedding quality as the edge drop rate p_e, the feature mask rate p_m and the regularization parameter λ vary. This highlights the importance of selecting the “correct” set of hyperparameters: a wrong choice of hyperparameters could cause the method to significantly underperform or to learn uninformative embeddings (see Figure <ref> left).

However, in the unsupervised context, there is no established cross-validation technique for GNNs or other principled technique for parameter selection at large. This significantly complicates the practical application of these methods in settings where there are no labels to evaluate the method on for hyperparameter selection.

(b) Trustworthiness. 
Moreover, despite the increasing adoption of graph neural networks (GNNs), there is a notable gap in research focusing on the evaluation of the quality of GNN node representations: How effectively do these embeddings capture the structural details within the data? Are they capable of accurately encoding topological information? Hypothetically, a good embedding should preserve both local and global structure in the graph while conveying a maximum amount of information. 

Beyond the graph setting, the need to benchmark new unsupervised approaches is underscored by the multiplication of recent publications in applied domains providing tips for performing a dimensionality reduction <cit.> or comparing existing tools <cit.>: with the rapidly increasing number of new methods, it becomes difficult to know which one to adopt — particularly when these methods can be quite sensitive.
Many applications — particularly in biology — have a long track record of using graph-based techniques such as UMAP <cit.> and t-SNE <cit.> for dimensionality reduction. These methods have been well established, understood, and have been vetted by the myriad of applications and benchmarking tasks, including manifold learning and classification tasks, to which they have been deployed. On the other hand, while GNNs offer the potential for richer representation learning by capturing both feature and node information, these methods have not yet attained a similar level of reliability.
This gap underscores (a) the need for a comparison of GNN-based approaches to current methods, allowing practitioners to place GNNs in the landscape of unsupervised learning methods. While GNNs offer more flexibility as they allow combining both graph data and node covariates, this comparison is nevertheless indispensable to start evaluating them as dimensionality reduction techniques; and (b) the need for more in-depth analysis and validation of GNNs in a variety of settings that go beyond node classification benchmarks, and incorporate a wider variety of tasks – including learning manifolds. 





Contributions To fill this gap, we propose here a systematic comparison of learned node representations, focusing more specifically on the context of dimensionality reduction and data visualization. To bridge classical dimensionality reduction methods that do not incorporate node features with state-of-the-art GNN-based approaches, we first introduce an unsupervised learning technique inspired by UMAP <cit.>, named GNUMAP, which integrates the UMAP framework for learning low-dimensional embeddings while being less parameter-intensive compared to contrastive learning methods. This approach facilitates a comparative analysis of self-supervised and reconstruction-based unsupervised embedding methods. We then propose to evaluate existing methods on two sets of tasks: (a) a suite of manifold learning tasks, thereby allowing us to compare GNN methods with more established benchmarks; and (b) classification tasks; Our goal is to provide practitioners with a clearer understanding of effective approaches in the unsupervised application of GNNs, thereby promoting their broader use across various applications.






§ GNUMAP: BRIDGING CLASSICAL DIMENSIONALITY REDUCTION AND GRAPH NEURAL NETWORKS


In this section, we introduce Graph-Neural UMAP (GNUMAP), a method that allows us to bridge the Graph Neural Network (GNN) framework with established dimensionality reduction techniques proven effective on real-world data. We begin by briefly reviewing UMAP, before extending it to the analysis of graph data. 

UMAP UMAP <cit.>, is a standard technique for dimensionality reduction of Euclidean data denoted as {x_i}_i=1^n ∈^p_h, and relies partly on the graph formalism. The first step of the method consists indeed in constructing a nearest-neighbor graph based on the data points {x_i}_i=1^n. The connection probability between nodes in this graph is given by the formula 

    p_ij = p_i|j + p_j|i - p_j|i× p_i|j,
 where 


    p_j|i = exp(-d(x_i, x_j)-ρ_i/σ_i).

In this context, x_i ∈ℝ^p_h denotes the original high-dimensional coordinates of input data point i, ρ_i represents the distance to the nearest neighbor of point i, and σ_i represents the local density factor around point i.
UMAP finds a low-dimensional representation y_i ∈ℝ^p_l, p_l << p_h for each point x_i that minimizes the cross-entropy between connection probabilities in the high and low-dimensional space. In the low-dimensional space, UMAP uses the family of curves f(x) = 1/1 + α x^2β to compute the connection probability analogous to p_ij, which is expressed as 

    q_ij = 1/1 + α× d(y_i, y_j)^2β.


By default, UMAP assumes the spread of the low-dimensional embeddings is 1 and the desired minimum distance between embeddings is 0.1. In such case, UMAP notes that f(x) = 1/1 + α x^2β where α=1.57 and β=0.89 best models the low-dimensional connection probability. While α=1.57 and β=0.89 are the default parameters, the authors of UMAP allow customizing of α and β based on the desired embedding spread and minimum distance.

The algorithm then computes the cross-entropy loss between the pairwise connection probabilities in both high (p_ij) and low (q_ij) dimensions. This cross-entropy loss is formulated as:

    ℒ = -∑_i∑_j[p_ijlog(q_ij) + (1 - p_ij) log(1 - q_ij) ]


This approach enables UMAP to find a low-dimensional representation while preserving the intrinsic topological structure of the data.

GNUMAP. By contrast, in our proposed adaptation of UMAP to the graph setting, the input of the algorithm is already a graph with either binary or weighted edges. We will denote this given input graph as 𝒢 = (𝒱, ℰ, X), where 𝒱∈ℝ^n is a set of n nodes, ℰ is a set of edges along with their connection probability, and X ∈ℝ^n × p_h be the p_h-dimensional node feature matrix. This inherent graph structure of the input allows us to bypass the initial step of traditional UMAP, which converts data into a graph with connection probability defined as in Equation <ref>. Similar to UMAP, our approach focuses on achieving dimensionality reduction by identifying a low-dimensional representation that preserves the topology of the original graph. 

(a) High Dimensional Node Connectivity. Let 𝐀∈ℝ^n × n be the input graph's (possibly weighted) adjacency matrix. We replace the connection probability  p_ij in Equation <ref> by A_ij, the observed  (and potentially weighted) connection between node i and j. This alleviates in particular any ambiguity on the choice of the parameters ρ_i and σ_i in the original method. Note here that compared to the original UMAP algorithm, if the adjacency matrix is binary, the p_ijs take values in {0,1}.

(b) Low Dimensional Node Connectivity. Let y_i denote the GNN embedding of datapoint i (y_i = GNN(x_i, 𝒩(i)), where 𝒩(i) denote the neighborhood of point i. We further feed the GNN outputs into a differentiable batch normalization layer that scales and decorrelates the input features to reduce internal covariate shift. The effect of the DBN layer is demonstrated in Table <ref> and Figure <ref> in the Appendix. 

Then, 
    q_ij = 1/1 + α× d(y_i, y_j)^2β
 denotes the weight of the low-dimensional node connections between y_i and y_j. This way of defining the low-dimensional connection strength q_ij draws significant inspiration from UMAP. As our DBN objective ensures the embedding distances are normalized, the default UMAP minimum distance of 0.1 allows interpretability in our setting, and therefore we incorporated UMAP's default constants α=1.57 and β=0.89. These values allow indeed smaller tails (so more spread-out embeddings) than a baseline choice of α=β=1 (see Figure <ref> in the Appendix). For a demonstration of GNUMAP performance with varying α and β, see Table <ref> in the Appendix.


(c) Loss Calculation. GNUMAP employs a cross-entropy loss between high and low dimensional node connectivity, so the loss becomes

    ℒ = -∑_i∑_j[p_ijlog(q_ij) + (1 - p_ij) log(1 - q_ij) ]


For scalability purposes, GNUMAP downsamples the loss corresponding to absent edges, and samples as many negatives as positives to compute the loss. The procedure is summarized in Algorithm <ref>.





Remark 1 GNUMAP is an auto-encoder. Using the reconstruction objective in Equation <ref>, it is is obvious that GNUMAP is simply an auto-encoder <cit.>. Auto-encoders are a class of reconstruction-based techniques that aim to find a low-dimensional representation of the data such that the distance or similarity between learned node embeddings is predictive of the existence of an edge between nodes in the original graph. The best known version of this type of approach, GAE, along with its variational version, VGAE, were first suggested in <cit.>. However, our proposed adaptation of UMAP differs slightly from these methods in several aspects: (i) the probability q_ij is defined differently.
Indeed, in GAE, q_ij is a function of the inner product between node representations: 
    q^GAE_ij = (1  + e^-X_i^TX_j)^-1 = ( 1+ e^d(X_i,X_j)^2 -  X_i^2 -   X_j^2/2)^-1.
 

Edge probabilities therefore depend both of the distance d(X_i,X_j) and on the norms X_i_2 and X_j_2 of the embeddings themselves. Consequently, embeddings close to the origin will typically exhibit a higher number of connections than embeddings farther from the origin. This allows the model to  potentially build in degree heterogeneity, and places all high degree nodes towards the origin.   However, in our proposed construction, this probability is solely a function of the distance. This can be beneficial, particurlarly when we expect the graph to have several cores, whose center (high degree nodes) should be placed far from one another. We will come back to this point in our synthetic experiments in section 3. (ii) GNUMAP uses a whitening step that allows to regularize the objective function. GNUMAP's differentiable batch normalization design closely follows that of CLGR <cit.> since the effectiveness of batch whitening was already demonstrated in the previous work. 

Remark 2 Contrary to self-supervised techniques for node embeddings, this proposed method does not rely on the choice of specific hyperparameters to perform well.

Remark 3 The driving hypothesis that GNUMAP leverages is that connected nodes should be close. GNUMAP is therefore amenable to homophilic networks (where adjacent nodes are presumed to be similar), rather than heterophilic networks — a limitation of our framework compared to self-supervised learning techniques.

Remark 4 Another method related to this direct extension of UMAP is SpaGCN <cit.>. Developed for transcriptomics applications, SpaGCN is a GCN algorithm that incorporates histological data to identify spatial domains and the associated gene expressions. From the method perspective, while SpaGCN is also an adaptation of UMAP to the graph setting, it differs from our approach in two significant ways: (i) SpaGCN directly targets creating a pre-specified number of clusters in the embedding space. The associated clusters are selected by minimizing the  KL divergence between their distance in low-dimensional space, and their distance in high-dimensional space. The definitions of q and p are also quite different: SpaGCN's q is assigned to a lower value when the embeddings are further from the cluster centers, and p is updated as the twice-normalized square of q at every three epochs. 
GNUMAP, by contrast, is amenable to learning a wider variety of manifolds, as it does not explicitly target the clustering of embeddings, but simply seeks to reconstruct a low-dimensional representation of the nodes that aligns with the original graph.



Overall, GNUMAP is a simple auto-encoder structure that leverages the success of UMAP in the Euclidean setting to perform dimensionality reduction. While this method is not conceptually novel (it is just an autoencoder), as we will show in our experiments section, the choice of the loss, as well as the addition of a whitening step are key to its success. This simple approach can perform well on a variety of tasks, and, as previously argued, is more interpretable and closer to many practioners' requirements in dimensionality reduction.




§ EVALUATING GNNS' MANIFOLD LEARNING ABILITIES


Having established a natural extension of UMAP to the GNN setting, we propose to benchmark these approaches on a set of toy examples evaluating their ability to correctly learn an underlying manifold structure. While this is a standard test for any dimensionality reduction technique, GNNs have not yet been evaluated in this specific context. 



We first evaluate GNUMAP performance on four synthetic geometric graph datasets considered as benchmarks in the realm of unsupervised dimensionality reduction methods: the Blobs, Swissroll, Moons, and Circles datasets (see Figure <ref>). The datasets were generated with ground-truth low-dimensional coordinates and cluster labels, which allows us to calculate our proposed metrics and also to inspect if our metrics are coherent with the embedding visualization. The four datasets were generated with 500 nodes, each connected to its 20 nearest neighbors. For simplicity, the experiments in this work have assigned a value of 1 to positive edges and 0 to negative edges. However, the GNUMAP algorithm is applicable to any input graph preprocessed such that weaker edges are mapped closer to 0 and stronger edges closer to 1. The features were instantiated as the embeddings of a 10-component Laplacian eigenmap decomposition of the induced 20-nearest graph. The latter is indeed an established procedure to instantiate features on a graph <cit.>. We compare GNUMAP with state-of-the-art GNN embedding methods tailored for dimensionality reduction: DGI <cit.>, BGRL <cit.>, CCA-SSG <cit.>, GAE <cit.>, VGAE <cit.> and SpaGCN <cit.>. We also compare with well-established dimensionality reduction methods PCA, t-SNE <cit.>, Isomap <cit.>, Laplacian Eigenmap <cit.>, UMAP <cit.>, and DenseMAP <cit.>. Note all methods mentioned above are Euclidean methods, which enable a consistent assessment with respect to our proposed metrics. To evaluate embedding quality, we introduce metrics including but not limited to: agreement in local geometry, Spearman correlation, and classification accuracy. We refer the reader to Appendix <ref> for a more detailed description of these metrics.

We implement a standard 2-layer GCN architecture for all GNN-based methods. Note that SpaGCN, CCA-SSG, GRACE, and BGRL are hyperparameter-heavy methods. SpaGCN requires the louvain resolution and number of neighbours for cluster initialization, as well as the penalty coefficient for embedding distance from cluster centers. Similarly, GRACE, BGRL and CCA-SSG require the specification of the regularization λ, an edge drop rate, as well as a feature mask rate. Autoencoder methods such as GNUMAP and GAE, on the other hand, are completely parameter free.

Since it is unclear which set of hyperparameters would be optimal for each dataset when deploying contrastive-learning techniques, we ran a full search for all possible hyperparameter combinations.  Note that this provides the most favorable comparison of these methods to GNUMAP, but deviates from the original unsupervised learning setting, since we are choosing the hyperparameters of the methods based on the labels — therefore making the methods more supervised. In real-world experiments, however, we would not benefit from such supervision. Despite this selection procedure, as we will show, GNUMAP still manages to outperform most methods.

We visualize some of the embedding from the best hyperparameter combination in Figure <ref>. In addition to GNUMAP, we visualized GAE for comparisons with explicitly learned distance between low-dimensional embeddings, CCA-SSG and GRACE for comparisons with contrastive learning methods. We also show results from related method SpaGCN. For a complete illustration including DGI, BGRL, VGAE, as well as PCA, t-SNE, Isomap, Laplacian Eigenmap, UMAP, and DenseMAP, see Figure <ref> in the appendix. Note the dimensionality reduction methods were not fitted on spectral features like the GNN-based models.





Furthermore, we compute proposed metrics over 100 experiments and summarize the results in Table <ref>.





 

Discussion Overall, GNUMAP consistently outperforms other self-supervised GNN-based methods in these small examples. GNUMAP successfully embeds blobs, swissroll, circles, and moons in 2-dimensions in a way that differentiates ground truth labels more clearly than state-of-the-art GNN-based methods. In Table <ref>, our method outperforms state-of-the-art GNN-based methods in all five of our proposed metrics and the GNUMAP embedding visualization in Figure <ref> is coherent with the metrics. Note that GNUMAP successfully unrolls the Swissroll to closely resemble the original data, which is a standard synthetic dataset where linear methods fail to correctly embed. 








§ EVALUATING UNSUPERVISED GNNS' EMBEDDING PERFORMANCE ON REAL-WORLD DATA


To assess GNUMAP's performance across real-world data, we conducted a comparative analysis using well-established graph benchmark datasets: Cora, Citeseer, and Pubmed. They are standard citation network benchmark datasets <cit.>. In these networks, nodes represent scientific publications, and edges denote citation links between publications. Node features are the bag-of-words representation of papers, and node label is the academic topic of a paper.

We also incorporated the Mouse Spleen cell data <cit.> to assess GNUMAP performance in biological applications. The dataset was originally generated by “co-detection by indexing"(CODEX) techniques, which is to take the image of a tissue section, and at each of the tissue locations, the relevant biomarkers information is measured and recorded <cit.>. For our experiments, we created a 5-nearest-neighboring-cell graph from the CODEX data. While the 3-topic cluster labels in the Mouse Spleen dataset are generated by the spatial LDA model, previous work <cit.> has established that such topic annotations are biologically consistent with immunologist-labeled topics. Therefore, we will regard the 3-topic cluster labels as ground truth labels in the Mouse Spleen dataset for the subsequent analysis.

For all experiments with real-world datasets, GNUMAP was compared against CCA-SSG, SPAGCN, and GAE.

All datasets come with or were assigned class labels, enabling a comparison between embedding visualization and our calculated metrics. For scalable metric evaluation, we randomly sampled 1000 datapoints and calculated the following metrics: classification accuracy, Calinski Harabasz score, and Davies Bouldin score. These metrics convey the general degree of embedding informativity and the quality of clustering, which is appropriate for evaluating Cora, Citeseer, Pubmed, and Mouse Spleen data as one expects some clustering among the same classes. The results are presented in Figure <ref> and Table <ref>.









Discussion
On homophilic citation networks, GNUMAP is consistently the best or second-best for two of the three metrics: classification accuracy and Davies Bouldin score. For Cora, GNUMAP outperforms all other models in classification accuracy and Davies Bouldin score, and GNUMAP produces the second best Clinski Harabasz score after CCA-SSG. On Pubmed, GAE was the best-performing model on all three metrics, and GNUMAP was a close second. The autoencoder framework appears especially effective in this data set. GAE was the best-performing model on all three metrics for Citeseer, and GNUMAP was a close second. Autoencoder framework appears especially effective on this dataset. On the Mouse Spleen dataset, CCA-SSG and GAE perform best or second-best over all three evaluation metrics. GNUMAP is the third-best method across all metrics, and SpaGCN is the worst. However, since SpaGCN was designed specifically for biological application, such poor performance was unexpected, and this suggests the 5-nearest-neighboring-cells graph preprocessing method was unable to reflect the mouse spleen's biological properties. This allows for future exploration for a separate preprocessing pipeline for converting biological data to graph. We further emphasize again hat for these examples, we chose the best performing hyperparameters for SPAGCN and CCA-SSG, thereby conferring these methods a significant advantage over the autoencoder setting. 



§ CONCLUSION

With the exponential growth in data complexity, statisticians and computer scientists have recognized graphs as a modeling framework for high-dimensional data types. Therefore, modeling with graphs, predominantly through GNNs, is gaining traction in various areas, especially the biological <cit.> domain. Through this work, we contribute to the ongoing research on unsupervised graph learning by proposing GNUMAP: a GNN inspired by theoretical research on the dimensionality reduction technique UMAP. Through our unique metrics that evaluate informativeness (classification accuracy), preservation of original graph information (agreement in local geometry, spearman correlation, frechet distance), and cluster quality (Davies-Bouldin score, Calinski-Harabasz score, silhouette score), we establish GNUMAP as a robust and expressive GNN embedding method that performs outperforms many existing GNN embedding methods, as well as dimensionality reduction methods depending on the task. Clear clustering of GNUMAP embeddings across synthetic geometric datasets as well as Cora, Citeseer, Pubmed, and Mouse Spleen further highlights the adaptive power of GNUMAP. GNUMAP performance can be characterized as “data-agnostic"–unlike state-of-the-art GNNs, there is minimal hyperparameter tuning required to enhance GNUMAP performance, allowing an easy application to any dataset.

Limitations and Future Work
The reconstruction-based approach that we propose here in this paper, GNUMAP, is by design extremely simple, and its objective is interpretable. While it performs well on the examples that we provide here, this method is only valid for homophilic datasets – where edges encode similarities between nodes. However, this method also lends itself to natural extensions, such as accounting for cluster densities using the density coefficient of DenseMAP <cit.>, to better capture the original data characteristics in low-dimensional embeddings.


unsrt




Appendix




§ METRICS
 

Current approaches to evaluating embeddings generally fall into one of two types: (a) visual methods, where individuals visually inspect the learned embeddings to assess if they align with their expectations, and (b) classification-based methods. In the classification-based approach, it is presumed that there exists a set of labels, not used during training, which effectively represent the structure of the underlying data. In our work, we extend these latter metrics for application to manifold scenarios (which involve continuous structures rather than discrete ones) and quantify the performance of the learned representations using the following metrics.

    
  General Metrics To assess the embedding quality, we propose measuring the agreement between the learned embedding space and the original data. This agreement can in particular be measured using:
    
        
  * Agreement in Local Geometry
        We compute the percentage of overlap between the original graph's k-nearest neighbors and knn graphs on the embedding space. A higher percentage denotes better local geometry preservation. 
        
    1/n∑_i=1^n# of overlapping neighborhood on  x_i/node degree(x_i)

        
  * Spearman Correlation
        Inspired by a methodology introduced in <cit.>, we evaluate the Spearman correlation between the pairwise distance in high-dimensional embedding space and embedding space from the model. Inspired by Isomap<cit.>, the pairwise distance in the original space is evaluated by the shortest path distance, assuming that the geodesic distance on the manifold can be reasonably approximated by the shortest path distance. The pairwise distance in the low-dimensional space is evaluated by standard Euclidean distance. This metric evaluates the relationship between learned embedding space and the original graph using a monotonic function. The metric spans [-1,1] where -1 denotes a relationship modeled by a perfectly decreasing monotonic function, and vice versa. A higher Spearman correlation implies a similarity between the learned embedding and the original data.
        
  * Classification Accuracy
        While our embeddings are unsupervised, we assume quality embeddings would form well-separated and helpful features for the support vector machine(SVM) multi-class classification problem. Consequently, we divide our manifold into different clusters (in intrinsic space) and assess the accuracy of the embedding space in recovering the different clusters through SVM with radial basis function(RBF) kernel. We record the average accuracy in predicting cluster labels from embedding space coordinates in a 10-fold. Higher accuracy thus implies the informativity of the learned embeddings.
        
  * Frechet Distance
        Finally, we implemented the Frechet inception distance <cit.>, a metric originally proposed to measure the similarity between images, to evaluate the 2d Frechet distance between the learned embedding coordinates and the original data. Smaller Frechet distance denotes better similarity between embedding and the original data coordinates.
    
    
  Metrics for Clustered Data For data that are expected to form clusters (e.g. synthetic blobs, citation networks, Mouse Spleen gene expression prediction), we propose metrics evaluating the quality of cluster separation in the embedding space.
    
        
  * Davies Bouldin Score
        This metric is the average ratio of inter-cluster distance to intra-cluster distance to the most similar cluster<cit.>. Davies Bouldin score is lower when clusters are concentrated and far apart. A lower score denotes better cluster separation, and the minimum score is zero.
        
  * Calinski Harabasz Score
        This metric is the ratio of the sum of intra-cluster dispersion and of inter-cluster dispersion<cit.>. Calinski Harabasz score is higher when the intra-cluster variability is low and inter-cluster variability is high. A higher score indicates better score cluster separation.
        
  * Silhouette Score
        This metric is the mean silhouette coefficient<cit.> of all data points. Let a be the mean intra-cluster distance and b be the mean nearest-cluster distance. Then, the silhouette coefficient for each data point is computed by b - a/max(a, b). Silhouette score is concerned with how similar a data point is to its assigned cluster compared to other clusters. The best value, 1, indicates well-separated and correctly assigned clusters, while the worst value, -1, often indicates that a sample is more similar to a different cluster. A silhouette score of 0 indicates overlapping clusters.
    





§ EFFECT OF Α AND Β IN LOW DIMENSIONAL REPRESENTATION


 We will investigate the effect of α and β in eq-<ref>. Recall the formula defining the edge connection probability in the low-dimensional space.


    q_ij = 1/1 + α× d(y_i, y_j)^2β

 










UMAP uses the family of curves f(x) = 1/1 + α x^2β to compute the low-dimensional connection probability. If x, the distance between embeddings, is close to zero, connection probability becomes closer to 1. Else, the probability becomes closer to zero. UMAP's low-dimensional connection probability calculation is dependent on α and β values. Figure <ref> illustrates low-dimensional connection probability with respect to distance between embeddings for different  α and β combinations.




§ EFFECTS OF DBN

Decorrelation Batch Normalization (DBN)

DBN layer performs batch whitening, which removes linear correlation between input channels and, therefore, stabilizes GNN model convergence. DBN layer is effective in unrolling the higher-dimensional graph object into 2-dimensions, which can be confirmed both by visual inspection in Figure <ref> and by metric evaluation in Table <ref>.









§ ADDITIONAL FIGURES








 Paper Title: Image-Based Deep Reinforcement Learning with Intrinsically Motivated Stimuli: On the Execution of Complex Robotic Tasks.


A Cooperation Control Framework Based on Admittance Control and Time-varying Passive Velocity Field Control for Human–Robot Co-carrying Tasks

    Dang Van Trong, Sumitaka Honji, and Takahiro Wada


        
The authors are with the Graduate School of Science and Technology, Nara Institute of Science and Technology, Ikoma, Nara 630-0192, Japan (e-mail: van_trong.dang.ve6@naist.ac.jp, honji.sumitaka@naist.ac.jp, and t.wada@is.naist.jp).



    August 5, 2024
===========================================================================================================================================================================================================================================================================================================





§ ABSTRACT 
    
    Reinforcement Learning (RL) has been widely used to solve tasks where the environment consistently provides a dense reward value. However, in real-world scenarios, rewards can often be poorly defined or sparse. Auxiliary signals are indispensable for discovering efficient exploration strategies and aiding the learning process. In this work, inspired by intrinsic motivation theory, we postulate that the intrinsic stimuli of novelty and surprise can assist in improving exploration in complex, sparsely rewarded environments. We introduce a novel sample-efficient method able to learn directly from pixels, an image-based extension of TD3  with an autoencoder called NaSA-TD3. The experiments demonstrate that NaSA-TD3 is easy to train and an efficient method for tackling complex continuous-control robotic tasks, both in simulated environments and real-world settings. NaSA-TD3 outperforms existing state-of-the-art RL image-based methods in terms of final performance without requiring pre-trained models or human demonstrations.






§ INTRODUCTION
 

    Motivation could be seen as an internal force or energy to encourage someone to execute or avoid a task.
    It is related to several internal or external factors, such as personality, own beliefs, culture, the surrounding environment, or earlier experiences.
    Motivation permits agents, organisms such as an animal or artificial organisms such as a robot, to play, explore, and discover new behaviours without external reward signals <cit.>. 
    Psychologists distinguish two groups of motivation, intrinsic and extrinsic <cit.>. When you do something for its satisfaction, i.e. without any immediate obvious intention, it is considered intrinsic motivation <cit.>. In contrast,  extrinsic motivation is when you do something to obtain positive feedback associated with that behaviour and the environment.
    Inside the intrinsic motivation group, novelty and surprise play an essential role in the agent's learning process. They stimulate interest and exploration of new stimuli <cit.>. As mentioned in <cit.>, only novel, unexpected and challenging circumstances make an organism intrinsically motivated. Specialized literature  <cit.> claims that exploration of new environments enriches plasticity in the brain to stimulate learning. This phenomenon is easy to recognize in young human infants, who show significant preferences for situations or stimuli never experienced before, even when these stimuli do not provide an associated direct reward <cit.>. 
    
    The underlying idea is simple: a novel situation or a surprise emotion encourages the agent to explore and learn better solutions faster than an agent using only external reward signals <cit.>. This is particularly important in the real world, especially in robotics, where most basic tasks do not have an immediate external reward or are absent. Furthermore, not only are intrinsic and extrinsic reward signals required to learn to solve tasks, but rich and abundant information is also needed, information that can describe and provide key features of the environment and the task itself.  This is particularly important in RL applications, where the state space representation plays a critical role during the learning process. 
    
    Learning from images has proven to be a feasible choice in RL <cit.>. Images encapsulate valuable information that may not always be readily available in other representations, such as proprioceptive states of the dynamics. Moreover, learning directly from pixels offers scalability for real-world applications. Utilizing cameras to gather environmental information proves to be more accurate, cost-effective, and straightforward compared to other sensor types, especially in complex setups featuring high dynamics, such as robots with multiple degrees of freedom (DoF).
    
    Nonetheless, learning directly from high-dimensional images presents a considerable challenge. Most of the state-of-the-art RL algorithms struggle when the input state is pure pixels <cit.>, as well as the computational cost increase <cit.>. Thus, having an accurate but simplified compact representation of the image that the agent can use to learn a control policy might help reduce the complexity and cost. However, creating a compact and valuable representation of an image with all relevant information about the environment and the task from which to learn is a challenge <cit.>. 
    
    Mapping images into a useful encoding usually demands a large training dataset, which is not a particular feature in RL. Ideally, mapping the images into a compressed representation and learning a control policy should co-occur and depend on each other. This interdependence resembles a chicken-and-egg dilemma: an optimal policy necessitates an effective representation, while an effective representation relies on meaningful gradient information derived from the policy. Therefore, achieving synergy between representation learning and policy optimization is paramount.
    
    
    Thus, in this study, aiming to address the challenges of sparse reward environments and image-based learning representations, we introduce a novel RL algorithm called Novelty and Surprise Autoencoder-TD3, NaSA-TD3. We extend and adapt the TD3 framework proposed by <cit.> from a conventional deterministic actor-critic method to a comprehensive image-based actor-critic approach. Drawing inspiration from intrinsic motivation theory, we translate human motivational behaviours into a computational framework by representing novel and surprising stimuli. We leverage these stimuli to uncover more effective exploration strategies to enhance the learning process. Additionally, to support and facilitate the endeavour of learning directly from images, we employ an autoencoder network in conjunction with an actor-critic methodology. This combination enables learning control policies tailored for executing complex robotic tasks, such as dexterous manipulations. The paper's contributions can be summarized as follows:
    
    
        
  * A computational representation of human intrinsic stimuli novelty and surprise.  
        
  * Evaluation of the influence of intrinsic stimuli novelty and surprise, introduced as a reward bonus, on RL for complex robotics tasks.
        
  * An image-based extension of TD3 <cit.> with an autoencoder for learning control policies for continuous action problems.
    
    


§ DEFINITIONS
 
    
    Novelty and surprise are closely related to each other. Their definitions are often incorrectly interchanged or interpreted in the same way; however, there are precise attributes that make them distinguishable <cit.>. 
    
    It is important to mention that there are several other detailed concepts relevant to motivation and intrinsic stimulus from a psychological perspective, along with extensive neuroscience literature; however, they are beyond this article's scope, and we do not address them because it would put us in a category too far from our primary goal. Nevertheless, for more in-depth definitions, we encourage the reader to review <cit.> and <cit.>. This work uses the following definitions.
    


 §.§  What is Novelty?


    The Oxford  English Dictionary defines novelty as “the quality of being new, different, and interesting" <cit.>. Psychologists, psychiatrists, and scientists differ in describing and measuring novelty depending if the stimuli are physical, visual, or behavioural <cit.>. In all cases, however, the common word is different <cit.>. We define novelty as something that we have not seen before; in other words, the process of recognizing new stimuli that are distinguishable from anything known before. 
    Detecting novelty involves digging into the contents of the agent's memory to associate or recognize if an experience has previously been seen or attended. Consequently, whether an observation is novel depends on whether a representation of that observation can be found, partial or identical, in the memory <cit.>.
    For instance, in dexterous manipulation using a robotic gripper, a novel stimulus might manifest as a joint configuration or object state that the agent has not previously encountered.
    


 §.§ What is Surprise?


    A surprise could be defined as a discrepancy, presented as a feeling or emotion excited, between an expected observation and an actual observation <cit.>. As mentioned in <cit.>, “Surprise requires an internal world model that formulates an expectation about the future". Measuring surprise involves a mechanism of prediction and comparison between expectation and actual observation.  Besides, here we do not examine the current observation with past observations; i.e. surprise does not directly correlate with the memory, even though a predicted world model is built using previous experiences <cit.>. 
    In dexterous manipulation with a robot gripper, a surprise stimulus can be conceptualized as the disparity between the expected position of the gripper and the target and the actual positions after taking an action. With surprise, time is important since prediction involves a specific time based on the current observation <cit.>, while novelty can not be associated with predictions since we can not predict situations or events never been experienced <cit.>. 
    


§ RELATED WORK


    The idea of using intrinsic motivation has been used in RL to encourage the agent's exploration. 
    In low-domain environments, a common method for computing exploration bonus signals is having a tabular record of how often a state has been visited, i.e. a count-based strategy. Previous proposals using count-based approaches, such as <cit.>, have shown satisfactory results and demonstrated tackling the exploration problem. However, maintaining state-action visitation count is impractical, inefficient and expensive in complex tasks with continuous state spaces,  making count-based approaches unscalable <cit.>. For example, it would be impractical in a robotics application to try to save all possible states when we deal with an infinite state space. 
    
    To avoid a tabular record of every visited state, <cit.> proposes EX2, a pseudo-count exploration method based on a discriminative exemplar model where the authors use a K-exemplar conditioned network. This approach, however, acts as a binary classifier that may tend to overfit due to the nature of the classifier itself. In <cit.>, the authors present a method using self-organizing maps, a CNN autoencoder and a predictive ensemble model to estimate an intrinsic reward. This method was tested in a simple environment using a single actuator for a learning-to-reach task. While work by <cit.> presents a concept that uses an agent's ability to reconstruct observations by giving their “contexts" (a downsample or noise version of the input image) to generate an intrinsic bonus that is decomposed into fast and slow rewards signals. Due to the need to balance the importance of the intrinsic signals, this method can only be applied to on-policy RL methods. No experiments in real-world or continuous action spaces are presented. Learning without extrinsic rewards has also been examined in <cit.>, where the novelty of a particular state is calculated as the distance of the current behaviour features to the k-nearest neighbours in a memory of behaviour features. However, the agent can end in “procrastination" on the main task instead. 
    
    
    On the other hand, <cit.> proposes a curiosity-based method to help the agent explore the environment more by generating an intrinsic signal based on an inverse dynamics model. In this proposal, the authors transform the observation input into a feature space with the relevant information and only consider the predictions that can affect the agent and ignore the rest. In <cit.>, a goal-based curiosity method is presented, where the authors decompose a task into several easier sub-tasks (called goals) given an observation. The agent samples K goals and predicts the probability that each goal can be achieved from the current observation using a predictor network. The number of samples needed to train the policy is too high to scale this method to real-world setups.

    
    
    Most closely related to our method is the work presented in <cit.>, which introduces a curiosity-based approach coupled with a comprehensive analysis of state representation learning (SRL) to enhance the sample efficiency of image-based RL. In this study, two methods are proposed: one utilizing a regularized autoencoder (RAE), and the other employing a contrastive learning approach. However, this work differs from ours in that it does not integrate intrinsic signals as part of the reward; instead, a separate curious policy based on the SRL error is trained to foster interest in problematic states. This approach is applicable only to off-policy methods and has yet to be validated in real-world environments.
    
    
    Despite the promising results demonstrated in these studies, intrinsic motivation remains underutilized as a potential solution in RL. While related works have illustrated that employing intrinsic stimuli as bonus rewards or additional policy promotes agent exploration, only a subset of these approaches are viable for real-world deployment due to limitations in sample efficiency or computational cost. Moreover, many related works have been evaluated in simplistic real-world environments or simulations, often relying on intricate calculations, predefined initial conditions, or pre-trained models <cit.>. Addressing the majority of these challenges, we present a scalable and robust method capable of tackling control tasks in continuous action spaces. Our algorithm leverages human-inspired stimuli of novelty and surprise to enhance RL learning directly from images. In comparison to traditional heuristic approaches, our method has the ability to solve complex tasks across both simulated and real-world environments.
    
    


§ NASA-TD3
 
    
    The goal of an agent in RL is to find the best action or decision for each state to maximize future rewards R; nevertheless, in some complex tasks, the reward R  may not be present or is sparse, complicating the search for an optimal policy. However, by employing intrinsic motivation stimuli, i.e. augmenting the reward function to deliver a bonus for visiting interesting, novel or surprising states, we can encourage exploration and improve the policy in an environment where the reward is sparse. Thus, intrinsic stimuli encourage the agent to explore more than an agent using extrinsic reward signals only. 

       
    
    The total R signal given a particular state s and an action a can be presented as:
    
    
    R_total(s, a)    = R_ext + R_int
    
            R_int(s, a)      = α * R_novel + β * R_surprise

    
    where R_ext is the extrinsic reward from the environment, R_int is the intrinsic reward formed by R_novel and R_surprise, which are functions designed to capture the novelty and surprise of a given state-action pair and α and β are the weights to balance the degree each term. In our experiments, we fix α =1 and β=1.



 §.§ Novelty and Surprise Detection

    
    Studies <cit.> have demonstrated that even after viewing a thousand pictures, each for a moment and single exposure, we humans can remember with high precision details we have seen in the picture. Recognizing whether we have seen an image is at some degree equivalent to detecting visual novelty, i.e. if we cannot remember seeing something, it can be considered a novel stimulus. As presented in <cit.>, a familiarity memory system involves the medial temporal lobe, perirhinal cortex, and inputs from the visual system to learn and recognize objects without paying attention to details. Consequently, we do not need to look for identical aspects or details in each image. Inspired by this idea, we present a novelty detection method using an unsupervised representation learning with images and an Autoencoder (AE). We determine if an observation looks “familiar" to what we have seen previously, mimicking the human brain. 

                

    At each step, we pass the current agent's observation (an RGB image from the camera) into the AE. If the output of the AE, representing a reconstruction of the observation, deviates significantly from the original input, we infer that the observation is novel and has not been encountered previously. Although we do not explicitly compare the input image with every image stored in memory, we postulate that the AE inherently possesses a memory representation of all previously seen states within its internal layers. This implicit memory mechanism reduces computational costs and eliminates the necessity of comparing each incoming state with the entire stored memory. To quantify novelty, we propose calculating the Structural Similarity Index (SSIM) between the input observation and its reconstruction from the AE, as illustrated in Fig <ref>. Essentially, we measure the disparity between the reconstructed and original images, utilizing this difference as the intrinsic reward bonus. Our choice of SSIM stems from its ability to identify similarities within pixels with better consistency compared to other metrics, such as Mean Square Error (MSE) or Peak Signal to Noise Ratio (PSNR), which are commonly used. This selection sets our approach apart from previous works, such as <cit.>. Thus, at each time step for each observation s_t,  the intrinsic reward signal for novelty is calculated by:    
     
    R_novel (s_t) = 1 - SSIM (s_t, Dec(Enc(s_t))

    
    
    Likewise, we aim to encourage agent exploration by providing a bonus signal for surprise events, i.e., to promote curiosity in unexpected situations. We measure surprise as the discrepancy between what is expected by the agent and what is actually obtained. In other words, the difference between the predicted observation and the true observation. As mentioned earlier, surprise requires an internal world model; however, predicting and generating high-dimensional observations, e.g. images, can be difficult and expensive.  Instead, we use a model to predict the next encoded image, i.e. the latent representation. This simplifies the computational cost compared to the full image reconstruction, presenting another distinctive aspect of our approach compared to previous proposals.
    
    From this, z_t+1 denotes the encoding of the image state s_t+1, and z'_t+1 the predicted representation of the next encoding state given the previous encoding image state z_t and action a_π using a predictive model P_ρ. The intrinsic reward signal for surprise is calculated by:  
    
    
    R_surprise(s_t, a_t)    = MSE(z_t+1, z'_t+1)
     
            z'_t+1   = P_ρ(z_t, a_t)

    
    Additionally, inspired by our previous work <cit.>, we use an ensemble of predictive models. This simple but effective combination helps improve the predictions and keeps stability <cit.> and makes the model less sensitive to noise, especially in the real world. Fig <ref> illustrates the configuration of the predictive model.

    
    
    
    
    It is important to underscore the distinctiveness in our approach regarding intrinsic signals and their respective computation methodologies. While previous studies, such as <cit.>, commonly adopt a unified intrinsic signal strategy, our research clearly emphasises delineating between novelty and surprise, employing separate methodologies for their assessment.
    
    
    
    


 §.§ Image-based policy learning


    Inspired by <cit.>, we introduce our Auto-Encoder Actor-Critic Off-Policy method. Our approach builds upon TD3 <cit.> and adheres to its training dynamics. The selection of TD3 is motivated by its simplicity and stability during training, as well as its ability to mitigate overestimation bias through the utilization of two critic networks. However, what sets our proposal apart is the integration of an AE within the main networks. This addition empowers our method to directly process high-dimensional observations, enabling the agent to learn directly from them. The complete architecture, including the number of layers and activation functions, is illustrated in Fig <ref>.

    
    
    The Actor-Critic network learns and evaluates a policy. This network receives as input the low-dimensional vector z from the encoder, which attempts to extract relevant features from the s observations at time t. Here, both the encoder and decoder are trained by minimizing the L2 reconstruction loss between the input observation and the reconstructed observation. To determine the optimal latent space size, we conducted experiments training the agent under identical conditions while varying only the latent space size. The results of this analysis are presented in Fig. <ref>. Based on these findings, a latent size of z=200 was selected for the remainder of the experiments. 
    
    In terms of policy learning methodology, we adhere closely to the established process of the base TD3 approach. However, instead of utilizing the true observation state s, we employ the latent representation vector z. Both the AE and the policy π are trained simultaneously. Notably, the encoder and decoder networks employed for novelty detection are identical to those utilized for policy learning. A graphic representation illustrating how and where we update the gradients can be seen in the broken lines in Fig <ref>. The actor's gradients are not used to update the encoder networks, while the critic's gradients update the critic networks and the encoder at each training step. 
    
    One of the primary distinctions between our proposal and previous works, such as <cit.> or <cit.>, lies in the comprehensive update of the entire encoder at each step, encompassing both the convolutional and linear layers. We observed that this simple modification enhances the stability of the networks. We attribute this improvement to the fact that the linear layers play a crucial role in constructing the image representation. Furthermore, we maintain consistency by utilizing the same encoder for both the target and main networks (actor and critic) throughout the training process. Consequently, there is no requirement to update the encoder weights of the target networks at a faster rate than the rest of the network's parameters, as observed in <cit.>.  




 §.§ Simulated Environments


    

    Repeating the nominal action is a popular technique used in RL. Several proposals <cit.> claim that action repetition positively affects the learning process. However, suppose the number of repetitions is not set adequately. In that case, it can cause instability in the control dynamics and lead to sub-optimal behaviour <cit.>, making choosing the correct number of repetitions a tuning hyperparameter that depends on each environment <cit.>. Likewise, repetition might not be suitable for real-world applications where servo-motors or linear actuators have fixed movements where action repetitions won't have any effect. Since our goal is an easily scalable algorithm capable of working in a wide range of applications, including real robots, we are not using this technique in our method; instead, we use novelty and surprise signals to obtain the same performance as the algorithms that use action repetition. More details of the training process as well as a list of hyperparameters, full source code and videos of each task can be found on the paper’s website at: <https://sites.google.com/aucklanduni.ac.nz/nasa-td3-pytorch/>
    
    


§ EXPERIMENTAL RESULTS
 
     We assess our algorithm's performance across diverse, complex continuous control robotics tasks in both virtual and real-world scenarios. Additionally, we conduct comparative analyses against SAC-AE <cit.>, utilizing the official source-code implementation provided by the author [<https://github.com/denisyarats/pytorch_sac_ae>]. We choose challenging continuous control tasks of the DeepMind Control Suite <cit.>. 
    
        
    
    This test-bed platform provides complex environments for continuous control tasks with normalized actions and image state observations. We choose six tasks with different complexity and rewards; see Fig <ref>. 
    A third-person camera image of size (84×84) pixels with three channels is used for all tasks. A stack of three consecutive full-colour images is used as the state observation. We run each environment for 1×10^6 training steps with the same hyperparameters for five independently seeded runs. All parameters are trained with the Adam optimizer, and we perform G=5 (based on a trial error experimentation) gradient updates per environment step. We regularly evaluate the agent's performance in each environment during the training; after every 1 × 10^4 training steps, we compute the average reward over 10 episodes. During the evaluation, no gradient updates are performed.

    
    


 §.§ Dexterous Robotics Manipulation in the Real World

    To test the applicability and efficiency of our proposed method to solve real-world tasks, we describe an experiment involving a 4-DoF robot gripper <cit.>; see Fig <ref>. The gripper has two identical fingers equipped with Dynamixel XL-320 servomotors and a standard webcam (C270 Logitech with 720p) on top of the structure. The task requires turning a valve tap to an arbitrary target orientation. Despite the apparent simplicity, the dexterous manipulation task poses significant challenges due to its requirement for visual perception and physical coordination of 4-DoF. Specifically, the robot must learn to synchronize the movements of its two fingers to prevent cancellation of rotational movements, while also mastering a complex finger gait <cit.>. The resulting solution needs to account for real-world factors such as noise, control inaccuracies, and environmental conditions like lighting dynamics and friction. It is worth noting that we are conducting this task without the use of commercially expensive hardware platforms, and we aim to learn directly from raw images without modelling or system identification.
    
    The extrinsic reward for this task is computed as the normalized negative L2-norm distance between the current valve's position and the desired angle. 
    The intrinsic reward is calculated with the novelty and surprise strategy mentioned above. The horizontal episode is limited to 50 steps where we use an action space that outputs a four-element vector ℝ^4 with the desired position of each servomotor. We maintain the same image size as in the previous section to ensure consistency across experiments. Thus, the state space consists of the RGB images from the camera (840×640) downscaled to an image of size (84×84) pixels with three channels. 

    
    
    The robotic task has two scenarios at the end of the episode. In the first case, the valve orientation is left in place between episodes with no resetting or reinitialisation. In the second case, the valve orientation is reinitialised to a random angle (using a servomotor attached to the valve) once the episode ends. Previous related tasks rotate the valve to the same initial orientation at the end of each episode <cit.>. Reinitialising the valve to any random orientation between [0, 2 π) increases the complexity of the task as it requires the agent to learn to use the information from the images to perceive the current valve's orientation. 
    We refrain from comparing this task with SAC-AE <cit.> due to the necessity of modifying its original code. Our primary objective is to showcase that our proposal is readily applicable to real-world applications. We train the model for a total of 5 × 10^4 environment steps, which amounts to approximately 11 hours of real-world training time. Five independently seeded runs are used to confirm the stability of our proposal. All hyperparameters used in this experiment and task-solved videos are listed on our website.



§ RESULTS AND DISCUSSION 

    Fig. <ref> shows the average evaluation reward comparing our NaSA-TD3 proposal, TD3-AE (which is our image-based extension of TD3 without any intrinsic rewards), SAC-AE <cit.> and Pixel-TD3 on the six continuous control tasks under identical conditions[For a fair comparison with our proposal with the exact number of steps for an episode, we present the result of SAE-AE passing action repetition parameter equal to 1]. Utilizing the novelty and surprise signals as intrinsic rewards significantly enhances task performance, with instances of achieving maximum reward values (corresponding to a 1 × 10^3 max return) observed in certain environments. In contrast, some other image-based approaches fail to reach the maximum value or struggle to solve the task entirely, as exemplified by Pixel-TD3. 

        
    
    It is worth noting that AE-TD3’s performance can match or improve SAC-AE in some environments. By further incorporating novelty and surprise stimuli as intrinsic rewards, NaSA-TD3 outperforms all variations. Notably, our method does not require a complex tuning of hyperparameters or complicated training steps as other image-based RL algorithms. Nevertheless, in the spirit of transparency regarding the results, it is worth noting that the novelty and surprise signals do not significantly impact the final results when the task is not highly complex, or the extrinsic reward is not sparse. This can be seen more clearly in environments such as  Walker or Finger Spin, where the agent performed similarly to SAC-AE. Presumably, this is because the intrinsic reward is not required, implying that the problem is easy enough to be solved with an extrinsic reward. In contrast, when the environment is complex or has sparse rewards, as in the Reacher or Cheetah environments, the intrinsic signals help significantly improve the performance.
    
    Fig <ref> shows the average evaluation reward for the real-world manipulation experiments. The results show the effect of novelty and surprise signals in executing the dexterous manipulation task. The superiority of our proposal over extrinsic-reward-only methods is even more evident. Our proposal solves the task and gets the highest reward in both tested scenarios. Pixel-TD3 failed to solve the tasks, while AE-TD3 struggled when the valve was reinitialised to a random orientation. These results underscore the effectiveness of intrinsic stimuli in encouraging the agent to explore a broader range of state-action pairs in complex environments, ultimately leading to faster and improved performance in real-world scenarios.
    
    Our proposal does have some limitations, with one of the most significant being the amount of RAM required. Utilizing pure images as input and storing them in a replay buffer can be computationally demanding, necessitating at least 64 GB of RAM to run a simulation environment for 1 million environment steps. However, this challenge can be mitigated by tuning the buffer size. Our experimentation revealed that other related proposals, such as <cit.> and <cit.>, require similar or even greater amounts of RAM, as they often train models for more extensive training steps. Notably, this issue is not explicitly addressed in their papers. Furthermore, despite our method demonstrating superior and faster convergence, the computational time for each environment step is higher compared to other methods such as SAC-AE <cit.>. This is primarily due to the additional gradient updates for the autoencoder and policy in each step, as well as the computation of  SSIM and MSE for novelty and surprise values, which involve additional calculations and resources.
    


§ CONCLUSION

    
    In this work, we set out to investigate the use of intrinsic rewards as a mechanism for improving the performance of RL in the execution of complex tasks such as dexterous robotic manipulation. We formulated reward metrics inspired by novelty and surprise, using the predictability of patterns across image sequences. A key novelty was the use of Autoencoders, which reduced our system's computational and memory needs to support real-world computation. Our findings show the ability to learn directly from raw images in simulation and real-world robots. This opens the door to scaling this proposal to more complex scenarios and tasks where the only information needed can come from a camera. Likewise, we verify that when the reward is sparse, auxiliary intrinsic signals such as novelty and surprise can help to discover efficient exploration strategies and assist the learning process.
    
    Future work will seek to include other intrinsic signals such as boredom, frustration, and pleasure. Furthermore, we will seek to improve the efficiency in the number of samples necessary, perhaps with the adaption of model-based reinforcement learning algorithms, plus a more comprehensive analysis of other autoencoder architectures and their effect on policy learning.

    































                        








§ ACKNOWLEDGEMENTS

    This research was supported by New Zealand's Science For Technical Innovation (SfTI) on contract UoA3727019. We want to thank the Human Aspects in Software Engineering Lab (HASEL) from the University of Auckland. 


ieeetr





 


 Paper Title: Disentangling Dense Embeddings with Sparse Autoencoders.

























Scaling Backwards:
Minimal Synthetic Pre-training?

    Ryo Nakamura1,*
0009-0009-0858-1229 
Ryu Tadokoro2,*
0009-0001-9473-3832
Ryosuke Yamada1
0000-0002-2154-8230


Yuki M. Asano 30000-0002-8533-4020
Iro Laina40000-0001-8857-7709
Christian Rupprecht40000-0003-3994-8045
Nakamasa Inoue50000-0002-9761-4142
Rio Yokota50000-0001-7573-7873
Hirokatsu Kataoka10000-0001-8844-165X

    August 5, 2024
====================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Sparse autoencoders (SAEs) have shown promise in extracting interpretable features from complex neural networks. We present one of the first applications of SAEs to dense text embeddings from large language models, demonstrating their effectiveness in disentangling semantic concepts. By training SAEs on embeddings of over 420,000 scientific paper abstracts from computer science and astronomy, we show that the resulting sparse representations maintain semantic fidelity while offering interpretability. We analyse these learned features, exploring their behaviour across different model capacities and introducing a novel method for identifying “feature families” that represent related concepts at varying levels of abstraction. To demonstrate the practical utility of our approach, we show how these interpretable features can be used to precisely steer semantic search, allowing for fine-grained control over query semantics. This work bridges the gap between the semantic richness of dense embeddings and the interpretability of sparse representations. We https://huggingface.co/charlieoneill/embedding-saesopen source our embeddings, trained sparse autoencoders, and interpreted features, as well as a https://huggingface.co/spaces/charlieoneill/saerch.aiDweb app for exploring them.




§ INTRODUCTION


The advent of large language models has revolutionised natural language processing, enabling the representation of text in rich semantic spaces <cit.>. These dense neural vector embeddings capture nuanced semantic relationships, significantly enhancing downstream applications such as information retrieval (IR) and semantic search <cit.>. However, the power of these representations comes at a cost: reduced interpretability and limited user control, presenting significant challenges for fine-tuning and explaining search results <cit.>.



To address these limitations, recent research has explored methods to disentangle and interpret the information encoded in dense representations <cit.>. Sparse autoencoders (SAEs) have emerged as a promising solution for extracting interpretable features from high-dimensional representations <cit.>. By learning to reconstruct inputs as linear combinations of features in a higher-dimensional sparse basis, SAEs can disentangle complex representations into individually interpretable components. This approach has shown success in analysing and steering generative models <cit.>, but its application to dense text embeddings remains unexplored.

In this work, we present the first application of SAEs to dense text embeddings derived from large language models. We demonstrate that this approach bridges the gap between the semantic richness of dense embeddings and the interpretability of sparse representations, offering new possibilities for understanding and manipulating textual semantic spaces. In a direct demonstration of their utility for semantic search, we show how SAE features can be used to steer search results, which has been previously been applied to decoder-only transformers and diffusion models for guided generation <cit.>. By causally manipulating features in the SAE's hidden representation of an embedding vector, we can perform precise adjustments of the semantic meaning of the vector upon reconstruction.

Our research makes the following key contributions:



  * We train SAEs with varying sizes on embeddings from a large corpus of scientific papers, demonstrating their effectiveness in learning interpretable features from dense text representations.

  * We conduct a comprehensive analysis of the learned features, examining their interpretability, behaviour across different model capacities, and semantic properties.

  * We introduce the concept of SAE “feature families”, clusters of related features that allow for multi-scale semantic analysis and manipulation, and look at how features “split” across levels of abstraction.

  * We demonstrate the practical utility of our approach by applying these interpretable features to enhance semantic search, allowing for fine-grained control over query semantics. We develop and open-source this as a tool that implements our SAE-enhanced semantic search system, as well as open-sourcing the underlying SAEs.


This paper is organised as follows: Section <ref> provides background on embedding representations and sparse autoencoders. Section <ref> details our methodology for training SAEs on text embeddings and analysing the resulting features, and the interpretability metrics of the learned features, as well as scaling behaviour of features across model capacities. Section <ref> outlines our identification of feature families and feature splitting. Section <ref> demonstrates the application of our approach to semantic search. Finally, Section <ref> discusses the implications of our work and potential future directions.







§ BACKGROUND AND RELATED WORK





Whilst dense embeddings have dramatically improved performance across various NLP tasks, they present significant challenges in terms of interpretability.



 §.§ Embeddings and Representation Learning


The evolution of word representations in NLP has progressed from simple one-hot encodings to sophisticated dense vector embeddings, each stage offering improvements in semantic expressiveness and contextual understanding. Early distributional semantics models like Latent Semantic Analysis <cit.> paved the way for prediction-based word embeddings such as Word2Vec <cit.> and GloVe <cit.>, which capture semantic and syntactic relationships. The advent of contextualised word representations, such as ELMo <cit.> and BERT <cit.>, marked a significant leap by generating dynamic word representations based on surrounding context. Recent developments have extended to sentence and document-level embeddings, with models like Sentence-BERT <cit.> proving particularly effective for tasks such as semantic search and information retrieval <cit.>.

While dense embeddings have dramatically improved the performance of NLP systems, they present challenges in terms of interpretability and fine-grained control. The high-dimensional, continuous nature of these representations makes it difficult to understand or manipulate specific semantic aspects encoded within them <cit.>. This opacity can be particularly problematic in applications where explainability or precise semantic control is critical. Semantic search with dense embeddings has largely replaced traditional keyword search <cit.>, but faces ongoing challenges, including the “curse of dense retrieval” <cit.> — a phenomenon where the performance of dense retrieval methods degrades more rapidly than sparse methods as index size increases — and lack of interpretability, which hinder fine-tuning of search results <cit.>.











 §.§ Sparse autoencoders


In large language models, the superposition hypothesis suggests that dense neural networks are highly underparameterised, and perform computations involving many more concepts than neurons <cit.>. Because these semantic concepts, or features, are quite sparse, models compensate encoding multiple features within the same set of neurons. However, this also leads to complex, overlapping representations that are difficult to interpret on a single-neuron basis. Similarly, in embedding spaces, features are not represented monosemantically in individual dimensions. Instead, each feature is typically distributed across multiple dimensions, and conversely, each dimension may contribute to the representation of multiple features. This distributed representation allows embedding models to efficiently encode a large number of features in a relatively low-dimensional space, but it also makes the embeddings challenging to interpret directly.

To address this challenge, sparse autoencoders (SAEs) have emerged as a promising solution. SAEs learn to reconstruct inputs using a sparse set of features in a higher-dimensional space, potentially disentangling superposed features <cit.>. By encouraging this disentanglement, SAEs aim to reveal more interpretable and semantically meaningful representations, demonstrating efficacy in uncovering interpretable features in large language model activations <cit.>. In a well-trained SAE, individual features in the hidden dimension align with the underlying sparse, semantically meaningful features. 






  §.§.§ Architecture and training


Sparse autoencoders (SAEs) are neural network models designed to learn compact, interpretable representations of high-dimensional data while enforcing sparsity in the hidden layer activations. The architecture of an SAE consists of an encoder network that maps the input to a hidden representation, and a decoder network that reconstructs the input from this representation.

Let 𝐱∈ℝ^d be an input vector, and 𝐡∈ℝ^n be the hidden representation, where typically n ≫ d. The encoder and decoder functions are defined as:

    Encoder:   𝐡   = f_θ(𝐱) = σ(W_e𝐱 + 𝐛_e) 
    Decoder:   𝐱̂   = g_ϕ(𝐡) = W_d𝐡 + 𝐛_d

where W_e ∈ℝ^n × d and W_d ∈ℝ^d × n are the encoding and decoding weight matrices, 𝐛_e ∈ℝ^k and 𝐛_d ∈ℝ^d are bias vectors, and σ(·) is a non-linear activation function (e.g., ReLU or sigmoid). The parameters θ = {W_e, 𝐛_e} and ϕ = {W_d, 𝐛_d} are learned during training.

The training objective of our SAE combines three main components: a reconstruction loss, a sparsity constraint, and an auxiliary loss. The overall loss function is given by:


    ℒ(θ, ϕ) = 1/d𝐱 - 𝐱̂_2^2 + λℒ_sparse(𝐡) + αℒ_aux(𝐱, 𝐱̂)


where λ > 0 and α > 0 are hyperparameters controlling the trade-off between reconstruction fidelity, sparsity, and the auxiliary loss.

For the sparsity constraint, we use a k-sparse constraint: only the k largest activations in 𝐡 are retained, while the rest are set to zero <cit.>. This approach avoids issues such as shrinkage, where L1 regularisation can cause feature activations to be systematically lower than their true values, potentially leading to suboptimal representations shrinkage, <cit.>. 

We also use an auxiliary loss, similar to the “ghost grads” technique <cit.>, to model the reconstruction error using the top k_aux dead latents, where we typically set k_aux = 2k. Latents are flagged as dead during training if they have not activated for a predetermined number of tokens (in our case, one full epoch through the training data). Given the reconstruction error of the main model 𝐞 = 𝐱 - 𝐱̂, we define the auxiliary loss as:


    ℒ_aux(𝐱, 𝐱̂) = 𝐞 - 𝐞̂_2^2


where 𝐞̂ = W_d 𝐳 is the reconstruction using the top k_aux dead latents, and 𝐳 is the sparse representation using only these dead latents. This additional loss term helps to revive dead features and improve the overall representational capacity of the model <cit.>.






  §.§.§ Autointerpretability
 

As SAEs provide a framework for more interpretable representations, researchers have sought ways of automatically interpreting the features that they learn (autointerpretability). <cit.> introduced sparse probing, where sparse linear models are used to detect the presence of concepts and features in internal model activations through supervised training. Many now use LLMs to directly interpret artifacts of unsupervised models i.e. SAEs. <cit.> used GPT-4 to generate and simulate neuron explanations by looking at text on which the neuron activates strongly, while <cit.> and <cit.> applied similar techniques to analyse sparse autoencoder features. <cit.> further introduced a specificity analysis to rate explanations by using another LLM to predict activations based on the LLM-generated interpretation, providing a quantification of how interpretable a given neuron or feature actually is. <cit.> demonstrated that cheaper methods, such as Neuron to Graph <cit.> with n-gram based explanations, allow for a scaleable feature labelling mechanism that does not rely on using more expensive LLMs. 












  §.§.§ Structure in SAE features
 
State-of-the-art automated interpretability techniques, as described in Section <ref>, have resulted in the discovery of a large volume of highly interpretable, monosemantic features in SAEs trained over language models <cit.>. With features being the base unit of interpretability for SAEs, recent work has focused on understanding the geometric structure of features. <cit.> report feature splitting in geometrically close groups of semantically related features, where number of learned features in the cluster increases with model size. They also report the existence of universal features which re-occur between independent SAEs and which have highly similar activation patterns. <cit.> find feature splitting also occurs in SAEs trained over production-scale models, with larger SAEs also exhibiting novel features for concepts that are not represented in smaller SAEs. <cit.> report over-splitting of binary features. <cit.> find clusters of SAE features that represent inherently multi-dimensional, non-linear subspaces.












§ TRAINING SAES AND AUTOMATED LABELLING





 §.§ Training and automated interpretability methods


Training: We train our top-k SAEs on the embeddings of abstracts from papers on arXiv with the  tag (astrophysics, 272,000 papers) and the  tag (computer science, 153,000 papers). The embeddings were generated with OpenAI's  model.[https://openai.com/index/new-embedding-models-and-api-updates/] We train our SAEs on these collections of embeddings separately. We normalised the embeddings to zero mean and unit variance before passing them to the SAE as inputs. Our trained SAEs are available for download https://huggingface.co/charlieoneill/embedding-saeshere.



Hyperparameters: Notable hyperparameters include the number of active latents k, the total number of latents n, the number of auxiliary latents k_aux, the learning rate, and the auxiliary loss coefficient α. We found learning rate and auxiliary loss coefficient to not have a significant effect on final reconstruction loss; we set the former to 1e-4 and the latter to 1/32. We vary k between 16 and 128, and n between two to nine times the embedding dimension d_input. Whilst we train SAEs with many different combinations of these hyperparameters, we largely focus on what we hereon refer to as SAE16 (k=16, n=2d_input=3072), SAE32 (k=32, n=4d_input=6144) and SAE64 (k=64, n=6d_input=9216). We train each model for approximately 13.2 thousand steps.

Automated interpretability:
Following the training of a Sparse Autoencoder (SAE), it becomes necessary to interpret its features, each corresponding to a column in the learned decoder weight matrix. To facilitate feature interpretation and quantify interpretation confidence, we employ two Large Language Model (LLM) instances: the Interpreter and the Predictor. The Interpreter is tasked with generating labels for each feature. It is provided with the abstracts that produce the top 5 activations of the feature across the dataset, along with randomly selected abstracts that do not activate the feature. The Interpreter then generates a label for the feature based on this input (for the complete prompt, refer to Appendix <ref>).
Subsequently, the generated label is passed to the Predictor. The Predictor is presented with three randomly sampled abstracts where the feature was activated and three where it was not. It is then instructed to predict whether a given abstract should activate the feature, expressing its confidence as a score ranging from -1 (absolute certainty of non-activation) to +1 (absolute certainty of activation).[We use 3 activating and 3 non-activating abstracts for the Predictor, rather than 5, due to LLM costs. We used  as the Interpreter and  as the Predictor. Notably, we predict each abstract separately, rather than batching abstracts like <cit.>.] We measure the Pearson correlation between this confidence and the true activation (binary; +1 or -1). We also measure the F1 score, when framing the confidence as a binary classification (active if confidence is above 0, inactive otherwise).









Evaluation metrics: In order to compare SAEs, we evaluate both their ability to reconstruct the embeddings, as well as the interpretability of the learned features. For the former, we examine the normalised mean squared error (MSE), where we divide MSE by the error when predicting the mean activations. We also report the log density of the activation of features across all papers. We do not report dead latents (those not firing on any abstract) as all models contained zero dead latents at the end of training. We also report the mean activation of features, when their activation is non-zero. To measure interpretability, we use Pearson correlation, as outlined above.



 §.§ SAE Performance


We observe precise power-law scalings for sparse autoencoder (SAE) performance as a function of the number of total latents n, active latents k, and compute C used for training. The normalised mean squared error (MSE) scales as L(n) = cn^-α for fixed k, where α ranges from 0.12 to 0.18, increasing with k, while c generally decreases (Figure <ref>, left panel). The  dataset shows slightly higher α values compared to . For compute scaling, we calculate the number of training FLOPs C at each step for each SAE. We find L(C) = aC^b, where a generally increases with k (3.84 for k=16 to 8.03 for k=64) and b ranges from -0.11 to -0.16, becoming more negative as k increases from 16 to 64 (Figure <ref>, right panel). Both relationships show high accuracy with R-squared values above 0.93. Detailed fits are provided in Appendix <ref>.




   








 §.§ Interpretability of SAE features


The most direct way to evaluate the interpretability of features is to look at the distribution of automated interpretability scores, discussed above. Specifically: given a feature label from our interpreter model, how well can a predictor model predict the feature's activation on unseen text? We show in Figure <ref> that the Pearson correlation between predictor model confidence of a feature firing and the ground-truth firing is quite high, with median correlations ranging from 0.65 to 0.71 for  and 0.85 to 0.98 for . We note that Pearson correlation increases as k and n decrease, likely due to models learning coarser-grained features that are easier for the interpreter to identify.

























§ CONSTRUCTING FEATURE FAMILIES THROUGH GRAPH-BASED CLUSTERING


We find that our SAEs trained over arXiv paper embeddings recover a wide range of features. These features cover both scientific concepts, from niche to broad and multi-disciplinary, and also more abstract semantic artifacts, such as humorous writing or critiques of scientific theories. Features and activating examples can be found in Appendix <ref>.

To analyse how features evolve across different SAE capacities and to identify meaningful groupings of related features, we studied two distinct phenomena: feature splitting and feature families. Feature splitting – the tendency of features appearing in larger SAEs to “split” the direction spanned by a feature from a smaller SAE, and activate on granular sub-topics of the smaller SAE's feature – has been observed in previous work on sparse autoencoders (e.g. <cit.>). Examples of feature splitting, as well as features recurring across SAEs, can be found in Figures <ref> and <ref>/<ref>. 



In contrast, feature families exist within a single SAE, and exhibit a clear hierarchical structure with a dense “parent” feature and several sparser “child” features; we suggest that the “parent” feature encompasses a broader, more abstract concept that is shared among the “child” features. An example feature family from  can be seen in Figure <ref>. 



 §.§ Feature splitting


We investigated how features in smaller SAEs relate to features in larger SAEs through a nearest neighbour approach. For each pair of SAEs (i.e. SAE16 and SAE32) with n_1 and n_2 features respectively, we calculated an n_1 × n_2 similarity matrix S where S_ij = 𝐰_i^T 𝐰_j/𝐰_i𝐰_j. Here, 𝐰_i and 𝐰_j are decoder weight vectors for features in the smaller and larger SAE, respectively. For each feature in the larger SAE, we identified the most similar feature in the smaller SAE, allowing us to trace how features potentially “split” or become more refined as model capacity increases. 



Our results are shown in Figure <ref>. We find that increasing both number of active latents k and the latent dimension n reduces the similarity between nearest neighbours in differently sized SAEs. This agrees with intuition: larger models with more capacity (higher k and n) can learn more fine-grained and specialised features, leading to greater differentiation from features in smaller models. 

Qualitatively, matching features from small to large SAEs, we find both recurrent features and novel features. Recurrent features appear with extremely high S_ij and activation similarity across one or more model pairs, and have highly similar auto-generated interpretations, suggesting semantic closeness; these are much more common for lower k (see Figure <ref>). In contrast, novel features have distinct semantic meaning from their nearest-neighbour match, and activate similarly on some but not all documents; novel features thus split the semantic space covered by their nearest-neighbour match from a smaller SAE. However, some novel features share little semantic or activation overlap with their nearest-neighbour feature, as in Fig. <ref>, indicating smaller SAEs may not sufficiently cover the feature space; see <ref> in the Appendix for more details.



 §.§ Feature families




  
Feature family identification
To identify feature families, we developed a graph-based approach using co-activation patterns across the dataset. We consider only highly interpretable features (F1 ≥ 0.8, Pearson ≥ 0.8).

We first compute co-occurrence matrix C and activation similarity matrix D. For all data points k:

    C_ij   = ∑_k A_ik A_jk    D_ij = ∑_k B_ik B_jk

where A_ik = 1 if feature i is active on example k (0 otherwise), and B_ik = 𝐡_𝐤, 𝐢 if feature i is active on example k with hidden vector 𝐡_𝐤 (0 otherwise). We normalise the co-occurrence matrix by feature activation frequencies and apply a threshold to focus on significant relationships:

    C_ij^norm = C_ij/f_i + ϵ

where f_i = ∑_k A_ik is the activation frequency of feature i and ϵ is a small constant for numerical stability. We then apply a threshold τ to obtain C_ij^thresh (hereafter just C). We construct a maximum spanning tree (MST) from C, capturing the strongest relationships between features while avoiding cycles. We convert the MST to a directed graph, with edges pointing from higher-density to lower-density features, representing a hierarchy from more general to more specific concepts. We identify feature families via depth-first-search in this directed graph, starting from root nodes (i.e., no incoming edges) and recursively exploring hierarchical sub-families.

We iterate this process, removing parent features after each iteration to re-form the MST and reveal overlapping, finer-grained feature families. We de-duplicate families with high set overlap (|F_1 ∩ F_2|/|F_1 ∪ F_2| > 0.6). In practice, we choose τ = 0.1 and use n = 3 iterations.




























































  
Feature family interpretability To evaluate the interpretability of feature families, we analysed their collective properties and the effectiveness of high-level descriptions in capturing their behaviour. For each family, we generated a “superfeature” description using GPT-4o, based on the individual feature descriptions within that family. We then uniformly sampled high-activating examples across all activations of child features, and assessed the interpretability of the superfeature using a prediction task, where GPT-4o predicted whether test abstracts would activate the superfeature. We compared these predictions to ground truth activations to compute Pearson correlation and F1 scores. Additionally, we calculated several metrics to characterise the structure and coherence of the feature families. Table <ref> presents the mean values of these metrics across all families for both the  and  datasets.



  
Matrix structure We conjecture that feature families are equivalent to diagonal blocks in some permutation of the co-occurrence matrix C and activation similarity matrix D. If feature families are indeed meaningful clusters in the graph, then in C and D in-block elements should co-activate much more strongly than off-diagonal elements. We also argue that due to the hierarchical nature of feature families, matrix “blocks” are highly sparse, since child features all co-occur with the parent feature but rarely co-occur with one another. Subsets of the co-occurrence matrix, permuted by feature family, are shown in <ref>. 

Motivated by these structures, we compute the parent-child co-occurrence ratio R(p, 𝒞) for every family with parent feature p and children 𝒞, avg(∑_i ∈𝒞 A_ip)/avg(∑_i ∈𝒞∑_j ∈𝒞, j ≠ i A_ij). We also permute the co-occurrence and activation similarity matrices by greedily selecting feature families, and compute the in-block to off-diagonal ratios C_diag / C_off and D_diag / D_off (excluding the i = j diagonal), capturing the clustering strength of the block diagonal. Statistics are listed in Table <ref>.


















§ EVALUATING EFFECTIVENESS OF SEARCH INTERVENTIONS





 §.§ Intervening on embeddings with SAE features


As an implementation detail, we note that intervening on a feature by up- or down-weighting its hidden representation and then decoding is equivalent to directly adding the scaled feature vector to the final embedding. We explore an alternative process in Appendix <ref> where we iteratively optimise the encoded decoded latents to minimise the difference between the desired feature activations and the actual activations.




 §.§ Experiments



We incorporate SAE-based embedding interventions into a literature retrieval system for  and . To assess the effectiveness of SAE feature intervention on semantic search, we evaluate the specificity and interpretability of feature-centric query modifications. We select random samples (N = 50 each) real literature retrieval queries relevant to machine learning and astronomy, which are answerable with information in papers from  and . For each query, we return the top k = 10 most relevant papers using embedding cosine similarity, making up the original retrieval results ℛ. We then select a random feature i in the top-k from the query's hidden representation 𝐡_𝐪, and another orthogonal feature j that has no overlap with the top-k; we limit our selection only to features that are highly interpretable (F1 > 0.9, Pearson > 0.9). Given these features, we create a modified query embedding with 𝐡'_𝐪,𝐢 = λ_- and 𝐡'_𝐪,𝐣 = λ_+, letting λ_- = 0 and sampling λ_+∈ [0, 5]. This effectively “down-weights” and “up-weights” the importance of i and j, respectively, in the modified query, which is used to generate new retrieval results ℛ'.

To the effect of up-weighting and down-weighting query modifications on the end retrieval results, we provide both ℛ and ℛ' to an external LLM instance. The external LLM then compares ℛ and ℛ' and determines which features, out of a multiple-choice subset of 5 options, have been up-weighted or down-weighted; we use this to compute the intervention accuracy, which measures the precision and efficacy of causal query interventions. As a baseline, we compare our SAE-based method against traditional query rewriting, by using another LLM instance to re-write the original query such that it up-weights j and down-weights i entirely using natural language.



Our results are shown in Figure <ref>. We find that SAE feature interventions consistently outperform traditional query rewriting across various levels of query fidelity. This Pareto improvement demonstrates that our method can achieve higher intervention accuracy while maintaining greater similarity to the original query. For instance, at a cosine similarity of 0.75, SAE interventions achieve approximately 20% higher accuracy compared to query rewriting. 












We also experiment with intervening on feature families, sampling highly interpretable families containing features in the query top-k. We uniformly adjust weights for all features in the family, including the parent, using the auto-generated family interpretation as the multiple-choice option. Results show that feature family interventions achieve accuracy comparable to individual features, but only down-weighting interventions outperform query re-writing. This may be because feature families can comprehensively down-weight related concepts, while up-weighting a general concept doesn't necessarily require activating all granular child features. Notably, lower cosine similarity isn't inherently undesirable, as changing the query will naturally reduce similarity. These experiments simply demonstrate that SAE features provide a way to parameterise semantic aspects of a query, allowing controlled modifications of its meaning.





§ DISCUSSION



In this work, we have presented the first application of sparse autoencoders (SAEs) to dense text embeddings derived from large language models. By training SAEs on embeddings of scientific paper abstracts, we have demonstrated their effectiveness in disentangling semantic concepts while maintaining semantic fidelity. We introduced the concept of “feature families” in SAEs, which allow for multi-scale semantic analysis and manipulation. Furthermore, we showcased the practical utility of our approach by applying these interpretable features to enable fine-grained control over query semantics. This aligns with recent work on controlled text generation, where <cit.> proposed using SAE-based interventions on embeddings to precisely alter generated text in specific semantic directions.

Importantly, our approach offers a novel solution to a growing challenge in scientific literature exploration. With the exponential proliferation of research papers, traditional methods of browsing and discovering relevant literature are becoming increasingly ineffective <cit.>. Our SAE-based method provides a new way to navigate and find pertinent papers, especially when researchers are not entirely certain about what they're looking for—a key function that libraries used to serve <cit.>. By allowing users to explore and manipulate interpretable semantic features, our system enables a more intuitive search process. This capability is particularly valuable in interdisciplinary research or when exploring emerging fields, where relevant work may not be easily discoverable through conventional keyword searches or citation networks <cit.>.

By providing a proof-of-concept for extracting interpretable features from dense embeddings, our work opens several promising research directions and applications across various NLP tasks. In text classification, our method could offer fine-grained insights into model decision boundaries with global features. For machine translation, it could enable targeted semantic manipulations, potentially addressing issues like gender bias in translations <cit.>. Scaling this approach to larger and more diverse datasets could reveal insights into how feature families and interpretability evolve with data size and domain breadth. While our current SAEs are trained on narrow scientific domains, extending this to the entirety of arXiv or even internet-scale text corpora could yield general-purpose SAEs with exceptionally rich feature spaces, providing insights into the structure of human knowledge as represented in large-scale text embeddings <cit.>. Beyond these applications, our work contributes to the broader goal of making language models more transparent and controllable, which is crucial for building trust in AI systems as they become more integrated into critical decision-making processes <cit.>.





On an evaluation note, we'd like to see how our reconstructed embeddings fare on a standard semantic embedding benchmark such as MTEB <cit.> in comparison to the original underlying embeddings. We'd also like to be able to conduct an evaluation of SAE features against some proxy of ground-truth features, much like <cit.> propose. For instance, the Unified Astronomy Thesaurus <cit.> could provide a basis for evaluating individual feature overlap with astronomy concepts, and even family features as groupings of these individual concepts.

In regard to scientific discovery, our method offers a powerful tool for analysing the evolution of scientific fields over time. By examining statistics of SAE features—such as clustering patterns, co-occurrences, and temporal trends—researchers could gain novel insights into how scientific domains have changed and interacted. This approach could significantly enhance existing efforts to map the landscape of scientific research, which have relied primarily on citation networks and keyword analysis <cit.>. For instance, in astronomy, <cit.> have used knowledge graphs to track field evolution; our SAE-based method could provide a more nuanced and interpretable view of conceptual shifts within the field. Furthermore, the interpretability offered by our SAEs could facilitate interdisciplinary research by providing a common semantic framework for comparing concepts across different scientific domains <cit.>.







 §.§ Limitations


Our work focused on relatively small datasets from specific scientific domains. Although this specificity allowed us to demonstrate the effectiveness of our approach in targeted areas, future work should investigate how well these methods generalise to larger, more diverse datasets. Additionally, our automated interpretability process, while effective, does not utilise the full spectrum of activations, potentially missing nuanced patterns in feature behaviour. 

The computational requirements for training SAEs on large embedding datasets also present scalability challenges that need to be addressed for wider adoption of this approach. Our SAEs are quite small in comparison to more general language model SAEs. This proved adequate given the narrow domains we trained on, but SAEs for general text embeddings would need to be scaled up by at least 2-3 the total number of latents.

Acknowledgments. Part of this work was done at the 2024 Jelinek Memorial Summer Workshop on Speech and Language Technologies and was supported with discretionary funds from Johns Hopkins University and from the EU Horizons 2020 program’s Marie Sklodowska-Curie Grant No 101007666 (ESPERANTO). Advanced Research Computing at Hopkins provided cloud computing to support the research. We are also grateful for the support from OpenAI through the OpenAI Researcher Access Program. Christine Ye is supported by the Regeneron Science Talent Search. Charles O'Neill is supported by a Tuckwell Scholarship at the Australian National University.  Support for KI was provided by NASA through the NASA Hubble Fellowship grant HST-HF2-51508 awarded by the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Inc., for NASA, under contract NAS5-26555. 
















§ TRAINING DETAILS





 §.§ Training setup

Our sparse autoencoder (SAE) implementation incorporates several recent advancements in the field. Following <cit.>, we initialise the bias b_pre using the geometric median of a data point sample and set encoder directions parallel to decoder directions. Decoder latent directions are normalised to unit length at initialisation and after each training step. For our top-k models, based on <cit.>, we set initial encoder magnitudes to match input vector magnitudes, though our analyses indicate minimal impact from this choice.

We augment the primary loss with an auxiliary component (AuxK), inspired by the “ghost grads” approach of <cit.>. This auxiliary term considers the top-k_aux inactive latents (typically k_aux=2k), where inactivity is determined by a lack of activation over a full training epoch. The total loss is formulated as ℒ + αℒ_aux, with α usually set to 1/32. This mechanism reduces the number of dead latents with minimal computational overhead <cit.>. We found that dead latents only occurred during training the k=16 models, and all dead latents had disappeared by the end of training. We show how dead latents evolved over training the k=16 SAEs for the  abstracts in Figure <ref>.

For optimisation, we employ Adam <cit.> with β_1=0.9 and β_2=0.999, maintaining a constant learning rate. We use gradient clipping. Our training uses batches of 1024 abstracts, with performance metrics showing robustness to batch size variations under appropriate hyperparameter settings.

The primary MSE loss uses a global normalisation factor computed at training initiation, while the AuxK loss employs per-batch normalisation to adapt to evolving error distributions. Following <cit.>, we apply a gradient projection technique to mitigate interactions between the Adam optimiser and decoder normalisation.





 §.§ SAE training metrics


Table <ref> shows the final training metrics for all combinations of SAEs trained. We note clear trends in normalised MSE, log feature density and activation mean as we vary the number of active latents k and the overall number of latents n. 






 §.§ Scaling laws


For the left panel of Figure <ref>, which shows the scaling of normalised MSE with the number of total latents n, we observe the following power-law relationships:

    k = 16    : L(n) = 0.61n^-0.12 (astro.ph);  L(n) = 0.67n^-0.13 (cs.LG)
    
    k = 32    : L(n) = 0.49n^-0.13 (astro.ph);  L(n) = 0.56n^-0.14 (cs.LG)
    
    k = 64    : L(n) = 0.46n^-0.15 (astro.ph);  L(n) = 0.60n^-0.17 (cs.LG)
    
    k = 128    : L(n) = 0.31n^-0.13 (astro.ph);  L(n) = 0.51n^-0.18 (cs.LG)

For the right panel of Figure <ref>, which shows the scaling of normalised MSE with the amount of compute C (in FLOPs), we observe the following power-law relationships:

    k = 16    : L(C) = 3.84C^-0.11
    
    k = 32    : L(C) = 5.25C^-0.13
    
    k = 64    : L(C) = 8.03C^-0.16
    
    k = 128    : L(C) = 2.80C^-0.13

These equations demonstrate the consistent power-law scaling behaviour of sparse autoencoders across different values of k, n, and compute C.




 §.§ Feature density and similarity


We find an intuitive relationship between k and n and the log feature density (essentially, how often a given feature fires). As k increases, we get a sharper peak of log feature density, shifted to the right, suggesting features fire in a tighter range as we increase the instantaneous L0 of the SAE's encoder (Figure <ref>).



To compare features across different SAEs trained on the same input data, we analyse the cosine similarity between the decoder weight vectors corresponding to each feature. Decoder weights, represented by columns in the decoder matrix, directly encode each feature's contribution to input reconstruction. Encoder weights, on the other hand, are optimised to extract feature coefficients while minimising interference between non-orthogonal features. This separation is important in the context of superposition, where we have more features than input dimensions, precluding perfect orthogonality. 







§ 


To demonstrate the practical applications of our sparse autoencoder (SAE) approach to semantic search and feature interpretation, we developed https://huggingface.co/spaces/charlieoneill/saerch.aiSAErch.ai, a web application that allows users to interact with the SAE models trained on arXiv paper embeddings. 



 §.§ Overview


SAErch.ai is built using the Gradio framework and consists of three main tabs: Home, SAErch, and Feature Visualisation. The application allows users to switch between the Computer Science () and Astrophysics () datasets.

The SAErch tab implements the core functionality of our semantic search system, allowing users to:


  * Input a search query

  * View the top 10 search results based on embedding similarity

  * Interact with the SAE features activated by their query


For each query, the system displays sliders corresponding to the top-k SAE features activated by the input. Users can adjust these sliders to modify the query embedding, effectively steering the search results towards or away from specific semantic concepts; see Figure <ref>. This directly demonstrates the fine-grained control over query semantics discussed in Section <ref> of our paper. Users can also search for and add specific features not initially activated by their query (Figure <ref>).












 §.§ Feature Visualisation Tab


The Feature Visualisation tab is divided into two sub-tabs: Individual Features and Feature Families. This section of the application directly relates to our analysis of SAE features and feature families discussed in Sections <ref> and <ref>.



  §.§.§ Individual Features


For any selected feature, this tab displays:


  * Top 5 activating abstracts, demonstrating the semantic content captured by the feature

  * Top and bottom 5 correlated features, illustrating the relationships between different SAE features

  * Top 5 co-occurring features, showing which features tend to activate together

  * A histogram of activation values, providing insight into the feature's behavior across the corpus

  * The most similar features in SAE16 and SAE32


















  §.§.§ Feature Families


The Feature Families tab in our web application offers an in-depth exploration of related features discovered by our sparse autoencoder. We show an example feature family in Figure <ref>.

The table displays the parent feature (superfeature) and its child features, along with key metrics, such as the name of the parent and child features, the frequency of co-occurrence between the child feature and the parent feature, ranging from 0 to 1, and the F1 Score and Pearson correlation.

The interactive directed graph provides a visual representation of the feature family structure. Each node represents a feature. The size of the node corresponds to the feature's density (frequency of activation), while the color intensity indicates the Pearson correlation (interpretability). Arrows between nodes show relationships between features, with the direction typically pointing from more general to more specific concepts. Users can hover over nodes to view detailed information about each feature, including its name and log density.








§ AUTOMATED INTERPRETABILITY DETAILS





 §.§ Examples of features




We show some examples of perfectly interpretable features (Pearson correlation >0.99) in Table <ref>. The strength of the activation of the feature on its top 3 activating abstracts is shown in parentheses next to the abstract title.





 §.§ Automated interpretability prompts


We provide the prompts used for the Interpreter model and the Predictor model in the boxes below. Where  text is used, it represents an input to the model. We found that performance significantly increased when including the instruction to use “Occam's razor”, whereby the simplest feature at the appropriate level of granularity was selected.



[colback=green!5!white, colframe=green!75!black, title=Interpreter Model Prompt, fonttitle=, coltitle=white]

You are a meticulous  researcher conducting an important investigation into a certain neuron in a language model trained on  papers. Your task is to figure out what sort of behaviour this neuron is responsible for – namely, on what general concepts, features, themes, methodologies or topics does this neuron fire? Here's how you'll complete the task:




INPUT DESCRIPTION:  You will be given two inputs: 1) Max Activating Examples and 2) Zero Activating Examples.


  * You will be given several examples of text that activate the neuron, along with a number being how much it was activated. This means there is some feature, theme, methodology, topic or concept in this text that `excites' this neuron.

  * You will also be given several examples of text that don't activate the neuron. This means the feature, topic or concept is not present in these texts.


OUTPUT DESCRIPTION: Given the inputs provided, complete the following tasks.


  * Based on the  provided, write down potential topics, concepts, themes, methodologies and features that they share in common. These will need to be specific - remember, all of the text comes from , so these need to be highly specific  concepts. You may need to look at different levels of granularity (i.e. subsets of a more general topic). List as many as you can think of. Give higher weight to concepts more present/prominent in examples with higher activations.

  * Based on the zero activating examples, rule out any of the topics/concepts/features listed above that are in the zero-activating examples. Systematically go through your list above.

  * Based on the above two steps, perform a thorough analysis of which feature, concept or topic, at what level of granularity, is likely to activate this neuron. Use Occam's razor, as long as it fits the provided evidence. Be highly rational and analytical here.

  * Based on step 4, summarise this concept in 1-8 words, in the form . Do NOT return anything after these 1-8 words.


Here are the max-activating examples:





Here are the zero-activating examples:





Work through the steps thoroughly and analytically to interpret our neuron.





[colback=blue!5!white, colframe=blue!75!black, title=Predictor Model Prompt, fonttitle=, coltitle=white]

You are a  expert that is predicting which abstracts will activate a certain neuron in a language model trained on  papers.  Your task is to predict which of the following abstracts will activate the neuron the most. Here's how you'll complete the task:




INPUT DESCRIPTION: You will be given the description of the type of paper abstracts on which the neuron activates. This description will be short. You will then be given an abstract. Based on the concept of the abstract, you will predict whether the neuron will activate or not.




OUTPUT DESCRIPTION: Given the inputs provided, complete the following tasks.


  * Based on the description of the type of paper abstracts on which the neuron activates, reason step by step about whether the neuron will activate on this abstract or not. Be highly rational and analytical here. The abstract may not be clear cut - it may contain topics/concepts close to the neuron description, but not exact. In this case, reason thoroughly and use your best judgement. However, do not speculate on topics that are not present in the abstract.

  * Based on the above step, predict whether the neuron will activate on this abstract or not. If you predict it will activate, give a confidence score from 0 to 1 (i.e. 1 if you're certain it will activate because it contains topics/concepts that match the description exactly, 0 if you're highly uncertain). If you predict it will not activate, give a confidence score from -1 to 0.

  * Provide the final confidence score in the form  e.g. . Do NOT return anything after this.


Here is the description/interpretation of the type of paper abstracts on which the neuron activates:





Here is the abstract to predict:





Work through the steps thoroughly and analytically to predict whether the neuron will activate on this abstract.





 §.§ Exploring the effectiveness of smaller models


Although we eventually used  as the Predictor model, we initially did some ablations to understand how effective  and  would be as different combinations of the Interpreter and Predictor models. We measured this by randomly sampling 50 features from our SAE64 (trained on  abstracts) and measuring the interpretability scores of different model combinations, in terms of both F1 score (does the model's binary classification of a feature firing on an abstract agree with the ground-truth) and the Pearson correlation (described in the main body). Interestingly, we observe that using  as the Interpreter and  as the Predictor leads to similar scores as using  for both, as shown in Figures <ref> and Figures <ref>. This suggests that the challenging task in the autointerp is not necessarily labelling but rather predicting the activation of a feature on unseen abstracts.



Another observation is that using  as the Predictor only leads to a moderate degradation of F1 score, it leads to a significant degradation of Pearson correlation. This is likely because we only use 6 abstracts for each feature prediction (3 positive, 3 negative) and thus there are only a few discrete F1 scores possible. Additionally, it appeared that  was generally less likely to assign higher confidence scores in either direction, with a much lower variance in assigned confidence than when  was the Predictor. This affects Pearson correlation but not F1.






§ CROSS-DOMAIN FEATURES

The intersection between our  (n = 153,146) and  (n = 271,492) corpora contains n = 330 cross-posted papers. Motivated by these papers, as well as the observation of similar features re-occurring in models of different sizes (see Section <ref>), we search for the max cosine similarity feature between  and  SAEs at a fixed k and n_dir. As expected, we find significant mis-alignment between the vast majority of feature vectors between SAEs trained on different domains, with mis-alignment increasing with k and n_dir (see Figure <ref>; this is unsurprising given how k and n_dirs correlate with feature granularity). 

However, a small subset of features appear in both sets of SAEs, with relatively high max cosine similarity. For example, Table <ref> shows the nearest  neighbours for every feature in the  “Machine Learning” feature family (average cosine similarity = 0.59, average activation similarity = 0.40). To test whether the features represent the same semantic concepts, we substitute the natural language description of the best-match  feature for each listed  feature and test the interpretability of the substituted descriptions; we find Δ_Pearson = -0.07 and Δ_F1 = -0.06. The existence of these features suggests that both sets of SAEs learn a semi-universal set of features that span the domain overlap between  and .

Interestingly, we find a number of near-perfectly aligned pairs (cosine similarity > 0.95) of highly interpretable features with little semantic overlap. A number of these features share similar wording but not meaning, such as “Substructure in dark matter and galaxies" () and “Subgraphs and their representations". Of these 10 feature pairs, the average activation similarity is 0.91. 








§ FEATURE FAMILY DETAILS




 §.§ Feature splitting structures
 
Figure <ref> shows an example of a recurrent feature across SAE sizes that does not exhibit feature splitting. While the feature has extremely high activation and cosine similarity across every model pair, each model only learns 1 feature in this direction. In Figures <ref> and <ref> we show two examples of feature splitting across SAE16 – SAE32 – SAE64 trained on . <ref> appears to show canonical feature splitting as originally described in <cit.>, with an increasing number of features splitting the semantic space at each SAE size. 

There exists a top-level “periodicity”/“periodicity detection” feature universal to all three SAEs, with relatively high similarity to all other features, as well as novel, more granular features appearing in smaller SAEs, i.e. “Quasi-periodic oscillations in blazars”, which only appears in SAE64 and is highly dissimilar from other split features.

In contrast, <ref> demonstrates nearest-neighbour features across models that do not exhibit semantically meaningful feature splitting. While the top-level “Luminous Blue Variables (LBVs)" feature occurs at every model size, SAE64 also exhibits two additional features, “Lemaitre-Tolman-Bondi (LTB) Models" and “Lyman Break Galaxies (LBGs)", that are highly dissimilar to each other, the LBVs feature, and every other feature in the smaller models. We claim these are novel features, occurring for the first time in SAE64, and that SAE16/SAE32 do not learn features for any related higher-level concepts; instead, this grouping could be a spurious token-level correlation (LBV/LTB/LBG as similar acronyms).








  
Feature triplets In Figure <ref>, we search for features that occur in n_dirs = 3072 models and have highly aligned features in larger (n_dirs = 6144, 9216) models; we use this as a rough proxy for the number of re-occurring features. We find that significantly more features re-occur between models for higher k, with over 1100 feature triplets at > 0.95 cosine similarity for k = 16; as k increases, the number of triplets drops sharply.



  
Self-consistency In <ref> we show the set overlap between nearest-neighbour matches between SAE16 and SAE64 found directly, and nearest-neighbour matches between SAE16 and SAE64 found via nearest-neighbour matches to SAE32. If features exhibit perfectly clean splitting geometry, then these two sets of SAE64 features should be consistent. However, we find that the distribution of set overlap is roughly bimodal; other than triplet features with perfect overlap, overlap generally ranges from 0 to 0.6.  The vast majority of intersection = 1 sets are ≤ 3 features in size. This corroborates findings in <ref> which suggests  features across models with different k are not well-aligned.






 §.§ Feature family structure


We compute feature family sizes (including the parent), co-occurrence ratios (R(p, 𝒞), see section <ref>), and activation similarity ratios (computed identically to R(p, 𝒞), just using activation similarities). Statistics for variants of  and  are shown in <ref>. We find a positive correlation (Spearman = 0.22) between R(p, 𝒞) and feature family interpretability.



We reproduce the projection method of <cit.>, running all documents through the SAE and ablating features not in the feature family, to produce Figure <ref>. Visualizing the resulting principal components confirms that the feature families we find do not represent manifolds or irreducible multi-dimensional structures. We can instead think of feature families as linear subspaces in the high-dimensional latent space; in fact, the component vectors can be seen in the lines of points representing documents only activating on one feature in the family.



In <ref> we use n = 3 iterations of feature family construction. We select this hyper-parameter based off Figure <ref>. In the first 2-3 iterations, removing parent nodes and re-constructing features preferentially creates additional smaller families, suggesting iterations are necessary to fully explore the graph. But given the sparse co-occurrences (C_i, j > 0.1) used to build the graph, the number of additional feature families found at each iteration drops off steeply after n = 3.






 §.§ Feature family interpretability

We show example feature families and their interpretability scores in Figure <ref>.












§ EXPLORING LEARNED DECODER WEIGHT MATRICES




Encoder and decoder representations Figure <ref> reveals an intriguing relationship between feature distinctiveness and the similarity of encoder and decoder representations in our sparse autoencoder. In an ideal scenario with orthogonal features, encoder and decoder vectors would be identical, as the optimal detection direction (encoder) would align perfectly with the representation direction (decoder). This is because orthogonal features can be uniquely identified without interference. However, in our high-dimensional space with more features than dimensions, perfect orthogonality is impossible due to superposition. 



The right panel of Figure <ref> shows a negative correlation between a feature's decoder-encoder cosine similarity and its maximum similarity with other features. Features more orthogonal to others (lower maximum similarity) tend to have more similar encoder and decoder representations. This aligns with intuition: for more isolated features, the encoder's detection direction can closely match the decoder's representation direction. Conversely, features with higher similarity to others require the encoder to adopt a more differentiated detection strategy to minimise interference, resulting in lower encoder-decoder similarity. The left panel, showing a mean cosine similarity of 0.57 between corresponding encoder and decoder vectors, further emphasises this departure from orthogonality. This phenomenon points to the importance of untied weights in sparse autoencoders.

Clustering feature vectors
Motivated by structure in the feature activation graph, we explore whether similar structure can be found in the decoder weight matrix W itself. <cit.> find 2 such clusters; we reproduce their method across our embeddings and SAEs, permuting the left singular vectors U of W using a one-dimensional UMAP. We also experiment with permuting U and W using reverse Cuthill-McKee. We do not find any meaningful block diagonal structure or clustering in W.












































§ ITERATIVE ENCODING OPTIMISATION



We noted in Section <ref> that intervening on a feature by up- or down-weighting its hidden representation and then decoding is equivalent to directly adding the scaled feature vector to the final embedding. To demonstrate this equivalence, let's consider an intervention on feature i by an amount δ. The modified hidden representation is 𝐡' = 𝐡 + δ𝐞_i, where 𝐞_i is the i-th standard basis vector. Decoding this modified representation gives 𝐱̂' = W_d𝐡' = W_d𝐡 + δ W_d𝐞_i = 𝐱̂ + δ𝐰_i, where 𝐰_i is the i-th column of W_d. Thus, intervening on the hidden representation and then decoding is equivalent to directly adding the scaled feature vector to the original reconstruction.

We show in Figure <ref> how cosine similarity between the original query embedding and the modified query embedding changes as we change the upweighting and downweighting strength for different features. Cosine similarity drops rapidly as soon as upweight or downweight exceeds 0.1.









There is an implicit challenge in SAE-based embedding interventions: the trade-off between steering strength and precision. When directly manipulating feature activations, we observed that strong interventions often led to unintended semantic shifts, activating correlated features and potentially moving the embedding far from the SAE's learned manifold. Our goal is to achieve precise semantic edits that express the desired feature strongly while minimising interference with unrelated features. To this end, we developed an iterative optimisation approach that leverages the SAE's learned feature space to find an optimal balance between these competing objectives.

Let 𝐱∈ℝ^d be the original embedding, f_θ(·) the SAE encoder, and g_ϕ(·) the SAE decoder. We define a target feature vector 𝐭∈ℝ^k representing the desired feature activations after intervention, where k is the number of active features in our SAE. The iterative latent optimisation aims to find optimised latents 𝐡^* that satisfy:


    𝐡^* = argmin_𝐡'{f_θ(g_ϕ(𝐡')) - 𝐭_2^2}


We solve this optimisation problem using gradient descent, starting from the initial latents 𝐡 = f_θ(𝐱) and iteratively updating 𝐡'. We use the AdamW optimiser with a cosine annealing learning rate schedule.

To evaluate the effectiveness of this approach, we compare it to a direct intervention method where we simply set the target feature to a specific value in the latent space. For each abstract in our dataset, we embed the abstract using an OpenAI embedding model to obtain 𝐱. We then encode the embedding to get initial latents 𝐡 = f_θ(𝐱). We randomly select a target feature i and target value v. We then apply both intervention methods: our iterative optimisation of 𝐡' as described above, with 𝐭_i = v and 𝐭_j = 𝐡_j for j ≠ i, and direct intervention: setting 𝐡'_i = v and 𝐡'_j = 𝐡_j for j ≠ i.



Figure <ref> (left panel) shows the trajectory of normalised MSE during the iterative optimisation process, when setting a random zero feature to active. Similarly, the right panel shows the optimisation when setting a random active feature to zero. Normalised MSE improves in the former case but not the latter.





























 




























































































































    

























    




































































 Paper Title: AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Models.















































 Dataa Robotics
 
 Shenzhen
 
 China
daqinluo@outlook.com



  Meituan Inc.
  
  
  Shenzhen
  China
fcjian@outlook.com



  Dataa Robotics
  
  Shenzhen
  
  China
  
  
yuxuan.nong@dataarobotics.com



Corresponding author.

  Johns Hopkins University
  
  Baltimore
  USA
yshen92@jhu.edu






































§ ABSTRACT 
  Automated Machine Learning (AutoML) offers a promising approach to streamline the training of machine learning models.
  
  However, existing AutoML frameworks are often limited to unimodal scenarios and require extensive manual configuration.
  
  Recent advancements in Large Language Models (LLMs) have showcased their exceptional abilities in reasoning, interaction, and code generation, presenting an opportunity to develop a more automated and user-friendly framework.
  
  To this end, we introduce , an innovative Automated Multimodal Machine Learning framework that leverages LLMs as controllers to automatically construct multimodal training pipelines. 
  
   comprehends data modalities and selects appropriate models based on user requirements, providing automation and interactivity. 
  
  By eliminating the need for manual feature engineering and hyperparameter optimization, our framework simplifies user engagement and enables customization through directives, addressing the limitations of previous rule-based AutoML approaches.
  
  We evaluate the performance of  on six diverse multimodal datasets spanning classification, regression, and retrieval tasks, as well as a comprehensive set of unimodal datasets.
  
  The results demonstrate that  achieves competitive or superior performance compared to traditional rule-based AutoML methods.
  
  Furthermore, a user study highlights the user-friendliness and usability of our framework, compared to the rule-based AutoML methods.
  
  Code is available at:
    https://github.com/tim120526/AutoM3Lhttps://github.com/tim120526/AutoM3L. 










  
































<ccs2012>
   <concept>
       <concept_id>10010147.10010178</concept_id>
       <concept_desc>Computing methodologies Artificial intelligence</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010257</concept_id>
       <concept_desc>Computing methodologies Machine learning</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10003120.10003121</concept_id>
       <concept_desc>Human-centered computing Human computer interaction (HCI)</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Artificial intelligence
[500]Computing methodologies Machine learning
[500]Human-centered computing Human computer interaction (HCI)




















AutoM3L: 

An Automated Multimodal Machine Learning Framework with Large Language Models

    Yiqing Shen
    August 5, 2024
=========================================================================================







§ INTRODUCTION

Multimodal data is crucial in various machine learning (ML) tasks as it provides the ability to capture more comprehensive feature representations.

Real-world data often combines heterogeneous sources, such as integrating table product information with associated images and textual descriptions.

Similarly, in the financial sector, user photos, text, transactions, and other data types are frequently consolidated in tabular formats for analysis and management.

However, the inherent diversity of these modalities introduces complexities, particularly in selecting optimal machine learning or deep learning model architectures and seamlessly synchronizing features across modalities

Consequently, there is often a heavy reliance on manual involvement in the ML pipeline.



 
Automated Machine Learning (AutoML) has emerged as a promising approach to reduce the need for manual intervention in the ML pipeline <cit.>. 

However, a significant gap exists for multimodal data, as the majority of AutoML solutions primarily focus on unimodal data. 

AutoGluon[<https://github.com/autogluon/autogluon>] made an initial attempt at multimodal AutoML but suffers from several limitations.

Firstly, it lacks comprehensive automation of feature engineering, which is crucial for effectively handling multimodal data.

Secondly, it presents a steep learning curve for users to become familiar with its configurations and settings, contradicting the user-friendly automation principles that AutoML aims to embody.

Moreover, AutoGluon's adaptability is constrained by pre-set settings such as the search space, model selection, and hyperparameters, necessitating significant manual intervention.

Lastly, extending AutoGluon's capabilities by integrating new techniques or models often requires complex manual code modifications, hindering its agility and potential for growth.


The scientific community has been captivated by the rapid rise of large language models (LLMs), particularly due to their transformative potential in task automation <cit.>.

LLMs have evolved beyond their initial purpose as text generators and have now become highly autonomous entities capable of self-initiated planning and execution <cit.>.

This evolution presents a compelling opportunity to enhance the performance and adaptability of multimodal AutoML systems.

Leveraging this potential, we introduce , an innovative LLM framework for Automated Multimodal Machine Learning.

Unlike platforms such as AutoGluon, which are constrained by predefined pipelines,  distinguishes itself through its dynamic user interactivity.

Specifically, it seamlessly integrates ML pipelines tailored to user instructions, enabling unparalleled scalability and adaptability throughout the entire process, from data pre-processing to model selection and optimization.

The major contributions are four-fold, summarized as follows.

(1) We introduce AutoM3L, a novel framework that automates the development of machine learning pipelines for multimodal data. 

AutoM3L enables users to derive accurate models for each modality from a diverse pool of models and generates an executable script for cross-modality feature fusion, all with minimal natural language instructions. This approach simplifies the process of building multimodal ML pipelines and makes it more accessible to a wider range of users.

(2) We advance the automation of feature engineering by leveraging a LLM to intelligently filter out attributes that could hinder model performance while simultaneously imputing missing data.

This automated feature engineering process reduces the need for manual intervention and improves the overall quality of the input data.

(3) We automate hyperparameter optimization by combining the LLM's self-generated suggestions with external API calls.

This approach eliminates the need for labor-intensive manual explorations and enables more efficient and effective hyperparameter tuning.

(4) We conduct comprehensive evaluations, comparing AutoM3L with conventional rule-based AutoML on a diverse set of multimodal and unimodal datasets.

Additionally, a user study further highlighted the distinct advantages of our framework in terms of user-friendliness and a significantly reduced learning curve.




§ METHODS




In this paper, we propose an Automated Multi-Modal Machine Learning () framework that utilizes Large Language Models (LLMs) to automate the machine learning pipeline for multimodal scenarios.

This section begins by introducing the organization of the multimodal dataset in Sec.<ref>. In Sec.<ref> to <ref>, we elaborate on the five functional components enhanced by LLMs in : (1) modality inference, (2) automated feature engineering, (3) model selection, (4) pipeline assembly, and (5) hyperparameter optimization, as illustrated in Fig. <ref>.



 §.§ Organization of Multimodal Dataset



Most existing studies utilize the JavaScript Object Notation (JSON) to represent multimodal data. However, JSON cannot capture the interplay between different modalities, making it unsuitable for analysis by language models. To address this limitation, we follow <cit.> and employ the structured tables to represent multimodal data. Structured tables offer a clear representation that captures the interaction between different modalities and effectively aggregates information from various formats into a unified structure. Additionally, these tables encompass a diverse range of data modalities, including images, text, tabular data, and more.






 §.§ Modality Inference Module



 begins with the Modality Inference-LLM (MI-LLM) component, which identifies the associated modality for each column in the structured table. 

To simplify its operation and minimize additional training costs, MI-LLM leverages in-context learning. 

As illustrated in Fig. <ref>(a), the guiding prompt for MI-LLM consists of three essential parts:

(1) An ensemble of curated examples is utilized for in-context learning, assisting MI-LLM in establishing strong correlations between column names and their associated modalities, thereby generating the desired format responses. 

These examples serve as a foundation for MI-LLM to learn from and adapt to the specific dataset at hand, enabling it to accurately infer the modality of each column based on the provided examples.

(2) A subset of the input structured table, consisting of randomly sampled data items paired with their respective column names, is included.

The semantic richness of this subset acts as a guiding force, directing MI-LLM towards accurate identification of modalities. 

By providing a representative sample of the dataset, MI-LLM can better understand the context and characteristics of each column, allowing it to make more informed modality inferences.

(3) User directives go beyond mere instructions, enriching the process with deeper contextual information.

These directives leverage the LLM's exceptional interactivity to enhance the refinement of modality inference.

For instance, a directive such as “This dataset delves into the diverse factors influencing animal adoption rates” provides MI-LLM with valuable contextual information, enabling a more insightful interpretation of column descriptors.

This additional context helps MI-LLM to make more accurate and relevant modality inferences by considering the overall theme and purpose of the dataset.





 §.§ Automated Feature Engineering Module


Feature engineering is a critical pre-processing phase to address common data challenges, such as handling missing values. 

While conventional AutoML solutions rely heavily on rule-based feature engineering, our  framework leverages the exceptional capabilities of LLMs to enhance this process. 

Specifically, we introduce the Automatic Feature Engineering-LLM (AFE-LLM), as illustrated in Fig. <ref>(b).

This module utilizes two distinct prompts, resulting in two core components: AFE-LLMfilter and AFE-LLMimputed.

The AFE-LLMfilter component effectively sifts through the data to eliminate irrelevant or redundant attributes.

On the other hand, the AFE-LLMimputed component is dedicated to data imputation, ensuring the completeness and reliability of essential data.

Importantly, these components work together in synergy. After AFE-LLMfilter refines the features, AFE-LLMimputed then addresses relevant data gaps in the dataset.


To enhance feature filtering, AFE-LLMfilter incorporates the following prompts:

(1) An ensemble of examples for in-context learning, which includes introducing attributes from diverse datasets and intentionally incorporating irrelevant attributes. The objective of AFE-LLMfilter is to effectively distinguish and eliminate irrelevant attributes

(2) Column names in the structured table, containing abundant semantic information about each feature component, thereby enhancing the LLM's capability to distinguish between crucial and dispensable attributes.

(3) Modality inference results derived from MI-LLM, guiding the LLM to remove attributes of limited informational significance. 

For instance, when comparing a binary attribute that indicates whether someone is over 50 with a continuous attribute such as age, it becomes apparent that the binary attribute may be somewhat redundant. 

In this case, the binary attribute can be identified and removed.

(4) User instructions or task descriptions can be embedded when available, aiming to establish a connection between column names and the corresponding task.



On the other hand, the AFE-LLMimputed component is dedicated to data imputation, ensuring the completeness and reliability of essential data. 

Regarding data imputation, AFE-LLMimputed exploits its inferential capabilities to effectively identify and fill missing data. 


The prompt includes the following:

(1) Data points with missing values, enabling AFE-LLMimputed to fill these gaps by discerning patterns and inter-attribute relationships.

(2) A selected subset of data instances from the training set that involves deliberately masking individual attributes and presenting them in Q&A pairs, laying down an inferential groundwork.

(3) Where available, user instructions or task descriptions are incorporated, offering a richer context and further refining the data imputation process.


Importantly, these components work together in synergy. After AFE-LLMfilter refines the features, AFE-LLMimputed then addresses relevant data gaps in the dataset. By combining feature filtering and data imputation, this module ensures that the dataset is optimized for the subsequent steps in the  pipeline.






 §.§ Model Selection Module

Upon successfully performing the modality inference and automated feature engineering modules,  proceeds to determine the optimal model architecture for each data modality. 

The candidate models are cataloged within a model zoo, with each model stored as a model card. 

The model card captures a wide range of details, including the model's name, type, applicable data modalities, empirical performance metrics, hardware requirements, and other relevant information.

To streamline the generation of these cards, we leverage LLM-enhanced tools, such as ChatPaper<cit.>, to eliminate the need for laborious manual writing processes. 

We generate embeddings for these model cards using a text encoder, thereby allowing users to retrieve relevant model cards and seamlessly expand the model zoo by appending new cards, as illustrated in Fig. <ref>(a).


Following the model card generation, we propose the Model Selection-LLM (MS-LLM) to effectively match each modality with the appropriate model.

We view this task as a single-choice dilemma, where the context provides a range of models for selection. 

However, due to limitations on the context length of LLM, it is not feasible to present a complete array of model cards.

Hence, we initially filter the model cards based on their applicable modality type and keep only those that are aligned with the specified data modality. 

Next, a subset of the top 5 models is identified using text-based similarity metrics to compare the user's requirements with the model cards' descriptions.

These high-ranking model cards are then incorporated into the prompt of MS-LLM, along with user instructions and data descriptions.

This combination guides MS-LLM in making its final decision, ultimately identifying the most suitable model for the given modality, as illustrated in Fig. <ref>(b).



The MS-LLM prompt fuses the following components:

(1) A selected subset of five model cards, providing insight into potential model candidates.

(2) An input context that intertwines data descriptions and user instructions. The data descriptions clarify important aspects such as data type, label type, and evaluation standards.

Meanwhile, user instructions can provide clarification on specific model requirements. 

For example, a user instruction such as “deploy the model on the CPU device” would guide MS-LLM to models optimized for lightweight deployments. This enhances the user-friendliness and intelligence of the framework by enabling interactive execution.






 §.§ Pipeline Assembly Module

After retrieving the unimodal models, a crucial step involves fusing them. We employ a late fusion strategy for integration, which can be mathematically expressed as:



    _i    = _i(_i(_i)),
    _cat    = (_1,...,_n), 
    _   = 
    ((_cat)),




    _i = _i(_i(_i)),



    _cat = (_1,...,_n),



    _ = 
    ((_cat)),

where  denotes concatenation, _i represents the input data of modality i (i=1,⋯,n), and _i adapts the output of _i to a consistent dimension. 

The  and  are the target models that need to be built.

Determining the architectures for  and  using rule-based methods that require manual scripting is impractical, as the architectures depend on the number of input modalities.

Instead, we reframe this process as a code generation challenge, where the Pipeline Assembly-LLM (PA-LLM) is responsible for generating the fusion model architecture.

PA-LLM leverages the code generation capabilities of LLMs to produce executable code for both model fusion and data processors, as depicted in Fig. <ref>(a).

This is achieved by providing the module with relevant model configuration files within the prompt. 

The data processors are generated based on the specified data preprocessing parameters in the configuration file.

We prioritize the integration of pre-trained models from various modalities, sourced from well-known libraries such as  and .

By establishing ties with the wider ML community, we have significantly enhanced the versatility and applicability of our model zoo.













 §.§ Automated Hyperparameter Optimization Module


In conventional ML pipelines, hyperparameters such as learning rate, batch size, hidden layer size, and loss weight are commonly adjusted manually, which is labor-intensive and time-consuming. 

Although external tools like  allow users to conduct optimization by specifying hyperparameters and their search intervals, there is still room for further automation.

To bridge this gap, we propose the HyperParameter Optimization-LLM (HPO-LLM), which extends the foundational capabilities of .

The core functionality of HPO-LLM lies in its ability to determine optimal hyperparameters and their corresponding search intervals through careful analysis of a provided training configuration file, as depicted in Fig. <ref>(b).

Leveraging the extensive knowledge base of LLMs in ML training, we first utilize LLM to generate comprehensive descriptions for each hyperparameter specified in the configuration file. 

The descriptions, combined with the original configuration file, constitute the prompt for HPO-LLM, which then provides recommendations on the most suitable hyperparameters for optimization.

The input prompt provided to HPO-LLM encompasses the following components:

(1) The training configuration file, containing a comprehensive set of hyperparameters, assists HPO-LLM in selecting the most suitable hyperparameters for optimization.

(2) LLM-generated text descriptions for each hyperparameter, enabling HPO-LLM to gain a comprehensive understanding of the significance of each hyperparameter.

(3) Optional user directives provide a personalized touch, allowing users to incorporate additional instructions that guide HPO-LLM's decision-making process. 

These directives can include emphasizing specific hyperparameters based on unique requirements, resulting in a tailored optimization approach. 

By integrating the capabilities of  with our HPO-LLM, we have pioneered an approach that enhances hyperparameter optimization by combining automation with advanced decision-making.







§ EXPERIMENTS




 §.§ Experimental Settings



  §.§.§ Datasets
 To assess the effectiveness of the  system, we performed experiments on six multimodal datasets, including some obtained from the Kaggle competition platform. 
These datasets cover various tasks, such as classification, regression, and retrieval. 

Table <ref> describes the details of the datasets.



We utilized three classification datasets as follows:


(1) PetFinder.my-Adoption Prediction (PAP): This dataset aims to predict the adoptability of pets by analyzing image, text, and tabular modalities.

(2) Multi-Modal Sarcasm Detection (MMSD): This dataset is curated to determine whether an utterance contains ironic sentiment, utilizing image and text modalities.

(3) CH-SIMS: This dataset focuses on sentiment recognition and leverages video and text modalities.

Turning our attention to regression, we utilized two datasets:

(1) PetFinder.my-Pawpularity Contest dataset (PPC): This dataset aims to predict the popularity of shelter pets by leveraging image and tabular modalities.

(2) PARA: This dataset provides diverse image and tabular attributes for personalized image aesthetics assessment.

For the retrieval-based tasks, we employed the Shopee-Price Match Guarantee dataset (SPMG), which aims to determine if two products are identical, relying on data from image and text modalities.

Our performance metrics include accuracy for multiclass classification tasks, the area under the ROC curve (AUC) for binary classification tasks and retrieval tasks, and the root mean square error (RMSE) for regression tasks.

We also evaluated  on a large number of unimodal datasets from the AutoML Benchmark<cit.> available from OpenML[<https://www.openml.org/>], which cover regression and binary/multiclass classification tasks.



  §.§.§ Baseline

Given the scarcity of specialized multimodal AutoML frameworks, our evaluations were exclusively performed using the AutoKeras[<https://github.com/keras-team/autokeras>] and AutoGluon framework. 

AutoKeras is dedicated to neural architecture search (NAS) and hyperparameter optimization for a given dataset.

Setting up training pipelines in AutoGluon required meticulous manual configurations. 

This involved specifying which models to train and conducting an extensive pre-exploration to determine the suitable parameters and their respective search ranges for hyperparameter optimization.

It's crucial to highlight that the automation and intelligence levels of AutoGluon remain challenging to quantify, and in this work, we innovatively measure them through the user study from the human perspective. 

See Appendix <ref> for detailed experimental settings.




  §.§.§ IRB Approval for User Study
 

The user study conducted in this research has received full approval from the Institutional Review Board (IRB).

All methodologies, protocols, and procedures pertaining to human participants were carefully reviewed to ensure they align with ethical standards.





 §.§ Quantitative Evaluation



We first carried out quantitative evaluations, drawing direct comparisons with AutoKeras and AutoGluon, focusing on the modality inference, automated feature engineering, and the automated hyperparameter optimization modules.

For modality inference evaluation, apart from the modality inference component, all other aspects of the frameworks are kept consistent.

For feature engineering and hyperparameter optimization, we aligned the modality inference from AutoKeras and AutoGluon with the results of  to analyze their respective impacts on performance.


To enhance the robustness of our results, we performed 10-fold cross-validation experiments on all datasets. The accuracy is reported as the mean value with its corresponding standard deviation.

Afterwards, we evaluate the pipeline assembly module in terms of intelligence and usability through user study in the next section, due to its inherent difficulty in quantitative evaluation.






  §.§.§ Evaluation for Modality Inference


Table <ref> depicts the comparative performance between AutoGluon's modality inference module and our LLM-based modality inference approach across various multimodal datasets. Since AutoKeras utilizes manually predefined data modality for each column, we excluded it from the comparisons.

Within AutoGluon, modality inference operates based on a set of manually defined rules. For instance, an attribute might be classified as a categorical modality if the count of its unique elements is below a certain threshold. 

Upon observing the results, it's evident that  offers accuracy on par with AutoGluon for most datasets. This similarity in performance can be primarily attributed to the congruence in their modality inference outcomes.

However, a notable divergence is observed with the CH-SIMS dataset.

Due to the manually defined rules in AutoGluon being unable to infer video modality and misclassified the "text" attribute as "categorical", the assembly of the training pipeline was hindered, resulting in the failure of the training task.

We manually corrected the misinference of the "text" attribute in AutoGluon, achieving the accuracy of 0.543(0.032). In comparison,  demonstrated a significantly superior accuracy, with a notable 3.2% improvement.

Such a result highlights the robustness of our LLM-based modality inference approach, which effectively infers modality details from column names and associated data through in-context learning with only a few examples, making it significantly more efficient than cumbersome manually designed rules.



  §.§.§ Evaluation for Feature Engineering

Table <ref> illustrates the comparisons of data preprocessing modules using AutoGluon and AutoKeras with our LLM-based automated feature engineering module on multimodal datasets.

Given the completeness of these datasets, we randomly masked portions of the tabular data and manually introduced noisy features from unrelated datasets to assess the effectiveness of automated feature engineering. For datasets without tabular modality, only noise features are introduced. Note that, AutoGluon lacks a dedicated feature engineering module for multimodal data, making this experiment a direct assessment of our automated feature engineering. We observed that automated feature engineering, which implements feature filtering and data imputation, effectively mitigates the impact of noisy data. Across all test datasets, automated feature engineering showed improvements,

while AutoGluon and AutoKeras suffered from performance degradation as they struggled to handle noisy data.



Since retrieval tasks and video modality inputs are not supported, we did not test AutoKeras on relevant datasets.










  §.§.§ Evaluation for Hyperparameter Optimization
 

We also conduct experiments to evaluate the automated hyperparameter optimization module within . 

Contrasting with AutoKeras and AutoGluon, which often require users to manually define the hyperparameter search space,  simplifies this process.




From Table <ref>, it's evident that the integration of hyperparameter optimization during the training phase contributes positively to performance. 

Impressively,  matches AutoGluon's accuracy on most datasets and, due to its effective utilization of video information, it has realized a 2.7% improvement on the CH-SIMS dataset.

However, the standout advantage of  lies in its automation. 


While AutoGluon requires a manual setup, which can often be tedious,  significantly reduces the need for human intervention, providing a more seamless and automated experience.

Another finding is that AutoKeras achieves lower accuracy on all datasets. In our analysis, we attribute it to the network structures obtained within its limited network search space, which lacks pretraining on large-scale datasets.

In contrast, our approach leverages the strength of pretrained models by linking with open-source communities such as HuggingFace and Timm. 

This integration allows us to access more powerful pretrained models, contributing to the improved performance demonstrated in our work.



















  §.§.§ Uni-Modal Scenario Evaluation

Given that most AutoML frameworks currently focus on single-modality AutoML, to demonstrate the scalability of , we also evaluated  on a large-scale tubular modality AutoML Benchmark<cit.> from OpenML. 



We compared it with a plethora of popular single-modal AutoML frameworks<cit.> on large and representative  datasets 

covering binary classification, multi-class classification, and regression tasks.

The metrics include logarithmic loss for multi-class classification tasks, the root mean squared error for regression tasks, and the area under the ROC curve for binary classification tasks. We reported the mean and standard deviation based on 10-fold cross-validation.
In the experiment, we employed the user instruction: “the model with the best performance on tabular data” to drive MS-LLM for model selection, and FT-Transformer was chosen and integrated with other components to form a training pipeline.
The results in Table <ref> and Table <ref> demonstrate 's strong performance even in single-modality settings.



Notably, in most frameworks compared, model ensemble techniques are employed to produce final predictions. However,  solely utilized a single model for evaluation and achieved competitive results, even outperforming others on most experimental tasks. 
















 §.§ User Study




  §.§.§ Hypothesis Formulation and Testing

To assess 's effectiveness, we conducted a user study focused on whether the LLM controller can enhance the degree of automation within the multimodal AutoML framework.

We formulated null hypotheses:


  * H1:  does not reduce time required for learning and using the framework.

  * H2:  does not improve user action accuracy.

  * H3:  does not enhance overall framework usability.

  * H4:  does not decrease user workload.


We performed single-sided t-tests to evaluate statistical significance. Specifically, we compared  and AutoGluon on the following variables: task execution time, the number of attempts, system usability, and perceived workload. 





  §.§.§ User Study Design
 
As depicted in Fig. <ref>, our user study's workflow unfolds in structured phases. 

Note that the user study has been reviewed by IRB and granted full approval.

The study begins with the orientation phase where voluntary participants are acquainted with the objectives, underlying motivations, and procedural details of the user study. 

This phase is followed by a user background survey, which gleans insights into participants' professional roles, their prior exposure to technologies such as LLM and AutoML, and other pertinent details.

The core segment of the study involves hands-on tasks that participants undertake in two distinct conditions: perform multimodal task AutoML with AutoGluon and with .

These tasks center around exploring the automation capabilities of the AutoML frameworks, as well as gauging the user-friendliness of their features. 

Participants, guided by clear instructions, are tasked with constructing multimodal training pipelines employing certain models and defining specific hyperparameter optimization domains.


To ensure a balanced perspective, participants are randomly split into two groups: the first interacts with AutoGluon, while the second delves into .

Upon task completion, the groups swap platforms. 

For a holistic understanding of user interactions, we meticulously track both the time taken by each participant for task execution and the number of attempts before the successful execution.

The study culminates with a feedback session, where participants articulate their impressions regarding the usability and perceived workload of both AutoGluon and  via questionnaire. 

Their feedback to the questionnaire, captured using Google Forms, form a crucial dataset for the subsequent hypothesis testing and analysis.

Our study cohort consisted of 20 diverse participants:
6 software developers, 10 AI researchers, and 4 students, which ensured a rich blend of perspectives of the involved users. 









  §.§.§ Results and Analysis of Hypothesis Testing
 

The data we gathered spanned four variables, visualized in Fig. <ref>.

To validate our hypotheses, we performed paired two-sample t-tests (essentially one-sample, one-sided t-tests on differences) for the aforementioned variables across two experimental conditions: AutoGluon and .

These tests were conducted at a significance level of 5%. 

The outcomes in Table <ref> empower us to reject all the null hypotheses, underscoring the superior efficacy and user-friendliness of .

The success of  can be largely attributed to the interactive capabilities endowed by LLMs, which significantly reduce the learning curve and usage costs.




Since most researchers were familiar with LLMs but had limited AutoML experience, increasing their learning curve on AutoGluon. Whereas the majority of engineers and students were novices in both these spheres, facing steeper challenges in grasping AutoGluon. Interestingly, even researchers acquainted with AutoML felt that  demonstrated superior ease of use comparatively. Collectively across backgrounds,  attained higher user ratings, lower task completion times, and fewer failed attempts, which quantitatively validates its improved user-friendliness.



§ CONCLUSION

In this work, we introduce , an LLM-powered Automated Multimodal Machine Learning framework.

 explores automated pipeline construction, automated feature engineering, and automated hyperparameter optimization. 

This enables the realization of an end-to-end multimodal AutoML framework.

Leveraging the exceptional capabilities of LLMs,  provides adaptable and accessible solutions for multimodal data tasks.
It offers automation, interactivity, and user customization. 
Through extensive experiments and user studies, we demonstrate 's effectiveness and user-friendliness. This highlights its potential to transform multimodal AutoML.
 marks a significant advance, offering enhanced multimodal machine learning across domains. 

Our future direction is to encompass a diverse range of data modalities, spanning graph, audio, and point clouds, among others.















ACM-Reference-Format











§ APPENDIX
 
tocsectionAppendix

In this appendix, we provide additional details of our approach. We present complete prompts for each LLM module in Sec.A,
while Sec.B offers comprehensive details about the datasets.
In Sec.C and Sec.D, we provide more details about user study, including evaluation metrics, participant background, etc. 
More detailed experimental implementation and related work are given in Sec.E and Sec.F. 
Finally, we discuss the impact of the bias problem of LLMs, computation cost, etc. of our framework in Sec.G.




§ PROMPTS






Prompt 1: Full System Prompt For MI-LLM. 


[colback=gray!10!white,colframe=gray!40!black]
You are a helpful assistant that analyzes data modalities in multimodal Auto-Machine learning task.

Your task is to analyze the data type of each column of the pandas.DataFrame tabular data.

Your answer must be in a strict JSON format: {“column name”: “data type”}.

You can analyze the data type based on the corresponding column name,column data and the user instructions, which may include the context of tasks/datasets, etc..

You should not omit any column of data in your answer.



Here are some examples for your reference:

Input: instructions:{case1_desc},Date:{case1_input}

Output: {case1_output}

...









Input: instructions:{data_desc},Date:{data_input}

Output:







Prompt 2: Full System Prompt For AFE-LLMfilter. 


[colback=gray!10!white,colframe=gray!40!black]
You are a helpful assistant that applies feature engineering, especially feature selection.

Given a set of features, your task is to filter out some features that are not relevant to the specific task.

You should filter out the features based on the feature names, feature type and user instrucions, which may contain the context of tasks/datasets, etc..

You cannot forge features that are not in the Input.

In particular, image features should be preserved.



Here are some examples for your reference:

Input: instructions:{case1_task}, features type:
{case1_feature_type}, features:{case1_feature}

Output: {case1_retained_feature}

...




Input: instructions:{data_task}, features type
:{data_feature_type}, features:{data_feature}

Output:








Prompt 3: Full System Prompt For AFE-LLMimputed. 


[colback=gray!10!white,colframe=gray!40!black]
You are a helpful assistant that applies feature engineering, especially data imputation.

Given a feature sequence, your task is to predict missing values in it. Missing values are represented by "???".

You should predict missing values based on other feature values in the sequence and, you can refer to user instructions, which may contrain context of the task/dataset, etc...

Your output format must be a certain element value, don't reply the reasoning process.



Here are some examples for your reference:

Input: instructions:{case1_task}, feature sequence:
{case1_data_sequence}

Output: {case1_miss_value}

...









Input: instructions:{data0_task}, feature sequence:
{data0_sequence}

Output:







Prompt 4: Full System Prompt For HPO-LLM. 


[colback=gray!10!white,colframe=gray!40!black]

You are a helpful assistant that infers the hyperparameters and their search ranges for hyperparameter optimization in machine learning task.

You can use the format:[,,,] to represent a discrete search range.

You can choose up 3 hyperparameters that you think are most suitable for hyperparameter optimization. 

Your answer must be in a strict JSON format: {“hyperparameter_name”:“search_range”}.














Here are some things you need to focus on:

(1).If the values in the search space are of type INT or FLOAT, then the search space needs to have at least 3 values.

(2).The search ranges should refer to the original value of the config. The search ranges should include the original value of the config.

(3).You should not output the hyperparameters don't need to optimize.

(4).You cannot forge parameters that are not in the configuration file.

(5).If the “checkpoint_name” is in config, only the “loss_weight” is taken.




Here are some comments to help you understand the parameters better: {self_desc}

Given the config as follow: {config}

Given the user requirements: {user requirements}

Your answer:







Prompt 4: Full System Prompt For MS-LLM. 


[colback=gray!10!white,colframe=gray!40!black]
I am a deep learning software develop engineer, you're a code compiler, and we're working together on a multimodal Auto-Machine learning task.

Given the dataset description and user request , your task is to help the user to select a suitable model.

You should focus more on the description of the models and find the model that has the most potential to solve requests and tasks.

Your answer must be in a strict JSON format: {“name”: “model name”, “reason”: “your reasons to select the model”}.
Please choose the most suitable model from:
{model_cards}



User: Assume we have a dataset:{data_desc} and user request: {user_request},please select the most suitable model.

Your answer:








Prompt 5: Full System Prompt For PA-LLM(Data Processors Generation). 



You are a helpful assistant that writes data processors code to load different types of data for multimodal Auto-Machine learning task.

Since different types of models need different data preprocessing, your task is to write a function to return the corresponding data processors based on models' config.

Specifically, you do not need to define the data processor for fusion model, and the label data processor is also required to provide label data for each model.

The function return must be in a strict dict format: {“data type”: “data processor”}.

Please specify the library you imported in the code.



Here are some data processors code for you reference:



from multimodal.data import ImageProcessor

class ImageProcessor:

    def __init__(self,model_config):

    ...
    


from multimodal.data import TextProcessor

class TextProcessor:

    def __init__(self,model_config):

    ...
    


from multimodal.data import CategoricalProcessor

class CategoricalProcessor:

    def __init__(self,model_config):

    ...
    










    











...
    


Given some models' config as follow:{configs}

Your answer:








Prompt 6: Full System Prompt For PA-LLM(Pipeline Assembly). 



You are a helpful assistant that writes the Deep learning model code.

You task is to write a fusion model to fuse different base models' features.

Use # before every line except the python code.

Here are some model code for you reference:

from multimodal.models import CategoricalTransformer

class CategoricalTransformer(nn.Module):

    def __init__(self,model_config):

    ...
    
from multimodal.models import NumericalTransformer

class NumericalTransformer(nn.Module):

    def __init__(self,model_config):

    ...
    
from multimodal.models import TimmAutoModelForImagePrediction

class TimmAutoModelForImagePrediction(nn.Module):

    def __init__(self,model_config):

    ...
    
from multimodal.models import HFAutoModelForTextPrediction

class HFAutoModelForTextPrediction(nn.Module):

    def __init__(self,model_config):

    ...

Given some base models' config as follow:{base_configs};

Give the fusion model config as follow:
{fusion_config}

You should then respond to me the code with:

1). Fusion technique should be learnable, MLP is recommended.

2). The fusion model structure should be defined as fusion_model and fusion_head,which output features and logits, respectively.

3). Base models instance should be defined in Fusion model Class.You should not change the value of the output of base model instances.

4). All base models have a uniform variable(self.out_features_dim) to represent the output features dimension.

5). Finding the maximum dimension of all base models' output features, and define learnable linear layers to adapt all base models' output features to the maximum dimension as the input of fusion_model. For example, if three models have feature dimensions are [512, 768, 64], it will linearly map all the features to dimension 768.

6). Output the logits,features,loss weights of fusion model and base models.The return must be in a JSON format: {model_name:{“logits”:...,“features”:...,“weight”:...}}. 


7). All the network layers and variable self.model_name,self.loss_weight should be defined in function __init__, not in function forward.

8). Some variables are not present in each model's config,you cannot use a variable that does not exist in the corresponding model config.



You should only respond in the format as described below :

Class Fusion:

    def __init__(self,...)

    ...

    def forward(self,batch)

    ...
    
    fusion_features = self.fusion_model(...)

    fusion_logits   = self.fusion_head(fusion_features)

    ...



























§ STRUCTURED TABLE DATASETS




For the purpose of reproducibility, we provide the downloading links to the datasets used in this work and, Table <ref> and Table <ref> describe the details of the sample datasets.


  * PetFinder.my-Adoption Prediction dataset (PAP):
 
 <https://www.kaggle.com/competitions/petfinder-adoption-prediction>


  * PetFinder.my-Pawpularity Contest dataset (PPC)

<https://www.kaggle.com/competitions/petfinder-pawpularity-score>


  * Multi-Modal Sarcasm Detection (MMSD):

<https://github.com/headacheboy/data-of-multimodal-sarcasm-detection>


  * Shopee-Price Match Guarantee dataset (SPMG):

<https://www.kaggle.com/competitions/shopee-product-matching>


  * PARA:

<https://cv-datasets.institutecv.com/#/data-sets>


  * CH-SIMS:

<https://github.com/thuiar/MMSA>




































§ QUESTIONNAIRE AND VARIABLES IN USER STUDY




 §.§ User Background Survey Questionnaire



    
  * Age? Single-choice question.
    
        
  ◯ 18
        
  ◯ 18-24
        
  ◯ 25-34
        
  ◯ 35-44
        
  ◯ 44
    

    
  * Gender? Single-choice question.
    
        
  ◯ Male
        
  ◯ Female
    

    
  * What is your highest level of education? Single-choice question.
    
        
  ◯ High School or Below
        
  ◯ Bachelor's Degree
        
  ◯ Master's Degree
        
  ◯ Ph.D.
        
  ◯ Other:                             
    

    
  * What is your occupation? Single-choice question.
    
        
  ◯ Student
        
  ◯ Engineer
        
  ◯ Data Scientist/Analyst
        
  ◯ AI Algorithm Engineer
        
  ◯ Educator
        
  ◯ Doctor/Medical Professional
        
  ◯ Other:                             
    

    
  * Are you familiar with Python? Single-choice question.
    
        
  ◯ Yes
        
  ◯ No
    

    
  * Are you familiar with terminal operation? Select only one bullet point.
    
        
  ◯ Yes
        
  ◯ No
    

    
  * Do you have any experience with machine learning? Select only one bullet point.
    
        
  ◯ Yes, experienced
        
  ◯ Yes, some experience
        
  ◯ No, no experience
    

    
  * Have you used any AutoML tools or platforms before? Select only one bullet point.
    
        
  ◯ Yes, very familiar
        
  ◯ Yes, somewhat familiar
        
  ◯ No, not familiar
    

    
  * Are you familiar with the AutoGluon used in this experiment? Select only one bullet point.
    
        
  ◯ Yes
        
  ◯ No
    

    
  * Are you familiar with the Large language model? Select only one bullet point.
    
        
  ◯ Yes, very familiar
        
  ◯ Yes, somewhat familiar
        
  ◯ No, not familiar
    

    
  * Would you be willing to participate in this experiment? Select only one bullet point.
    
        
  ◯ Yes, I am willing to participate
        
  ◯ No, I am not willing to participate
    

    
  * What are your expectations for automated machine learning methods? Select only one bullet point.
    
    
  *                                     
    




 §.§ Questionnaire After Task Execution




 

    
  * How much time did it take in total to complete all the tasks? (in seconds)         

    
  * How many script execution attempts did you make in total to complete the tasks?        

    
  * I think that I would like to use this system frequently.
    Select only one bullet point.

    
    1     2     3     4     5

    Strongly Disagree  ◯      ◯      ◯       ◯       ◯  Strongly Agree 

    

    
  * I found the system unnecessarily complex.
    Select only one bullet point.

    
    1     2     3     4     5

    Strongly Disagree  ◯      ◯      ◯       ◯       ◯  Strongly Agree 

    

    
  * I thought the system was easy to use.
    Select only one bullet point.

    
    1     2     3     4     5

    Strongly Disagree  ◯      ◯      ◯       ◯       ◯  Strongly Agree 

    

    
  * I think that I would need the support of a technical person to be able to use this system.
    Select only one bullet point.

    
    1     2     3     4     5

    Strongly Disagree  ◯      ◯      ◯       ◯       ◯  Strongly Agree 

    

    
  * I found the various functions in this system were well integrated.
    Select only one bullet point.

    
    1     2     3     4     5

    Strongly Disagree  ◯      ◯      ◯       ◯       ◯  Strongly Agree 

    

    
  * I thought there was too much inconsistency in this system.
    Select only one bullet point.

    
    1     2     3     4     5

    Strongly Disagree  ◯      ◯      ◯       ◯       ◯  Strongly Agree 

    

    
  * I would imagine that most people would learn to use this system very quickly. 
    Select only one bullet point.

    
    1     2     3     4     5

    Strongly Disagree  ◯      ◯      ◯       ◯       ◯  Strongly Agree 

    

    
  * I found the system very cumbersome to use.
    Select only one bullet point.

    
    1     2     3     4     5

    Strongly Disagree  ◯      ◯      ◯       ◯       ◯  Strongly Agree 

    

    
  * I felt very confident using the system.
    Select only one bullet point.

    
    
    1     2     3     4     5

    Strongly Disagree  ◯      ◯      ◯       ◯       ◯  Strongly Agree 

    

    
  * I needed to learn a lot of things before I could get going with this system.
    Select only one bullet point.

    
    1     2     3     4     5

    Strongly Disagree  ◯      ◯      ◯       ◯       ◯  Strongly Agree 

    

    

    
  * Mental Demand:How mentally demanding was the task?
    Please assign a score between 1 and 20, where 1 = very low, and 20 = very high.


            

    
  * Physical Demand:How physically demanding was the task?
    Please assign a score between 1 and 20, where 1 = very low, and 20 = very high.


            

    
  * Temporal Demand:How hurried or rushed was the pace of the task?
    Please assign a score between 1 and 20, where 1 = very low, and 20 = very high.


            

    
  * Performance: How successful were you in accomplishing what you were asked to do?
    Please assign a score between 1 and 20, where 1 = very low, and 20 = very high.


            

    
  * Effort:How hard did you have to work to accomplish your level of performance?
    Please assign a score between 1 and 20, where 1 = very low, and 20 = very high.


            

    
  * Frustration:How insecure, discouraged, irritated, stressed and annoyed wereyou?
    Please assign a score between 1 and 20, where 1 = very low, and 20 = very high.


            

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Mental Demand       ◯ Physical Demand
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Temporal Demand       ◯ Performance
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Effort       ◯ Frustration
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Mental Demand       ◯ Temporal Demand
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Effort      ◯ Physical Demand
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Performance       ◯ Frustration
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Effort      ◯ Mental Demand
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Temporal Demand       ◯ Frustration
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Physical Demand      ◯ Performance
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Mental Demand      ◯ Performance
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Temporal Demand      ◯ Effort
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Frustration       ◯ Physical Demand
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Frustration     ◯ Mental Demand
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Frustration     ◯ Temporal Demand
    

    
  * Main source of workload?
    Select only one bullet point.

    
     ◯ Performance     ◯ Effort
    








§ USER STUDY DETAILS





 §.§ Participant Recruitment


We strategically recruited volunteers to participate in our user study, encompassing potential users of AutoML frameworks, consisting of 20 diverse participants: 6 software developers, 10 AI researchers, and 4 students, ensuring a rich blend of perspectives among the involved users.

Their prior exposure to key techniques relevant to this study is summarized below:


    
  * Large Language Models:
    
        
  * 10 participants were very familiar with LLMs from actively using them in research projects.
        
  * 5 participants had some previous experience with LLMs.
        
  * 5 participants were completely new to LLMs. 
    

    
  * AutoML & HPO Tools:
    
        
  * 3 participants actively used AutoML & HPO libraries like AutoGluon and Optuna in their work.
        
  * 4 participants had tried basic AutoML tutorials before.
        
  * 13 participants had no familiarity with AutoML and HPO tools.
    

    
  * Multimodal Data Experience:
    
        
  * 5 participants worked extensively on multimodal datasets combining image, text, and tabular sources.
        
  * 10 participants only used uni-modal datasets before.
        
  * 5 participants were new to both multi-modal and uni-modal data.
    


We believe that this diverse group of participants provides a comprehensive evaluation of our , considering a range of backgrounds and expertise levels in AutoML methods. 



 §.§ Definition and Calculation of Variables



We denote participant responses to the  question in questionnaire <ref> as . Questions 1-18 are numerical variables, while the remaining are categorical. The dependent variables are as follows:



  * Task Execution Time: Objective, continuous variable measuring the total time taken by participants to successfully complete the task. Derived directly from the response to question 1 in questionnaire <ref>:

    = .




  * Number of Attempts: Objective, continuous variable recording the total script execution attempts by participants to successfully complete the task. Derived directly from the response to question 2 in questionnaire <ref>:

    =  .




  * Usability Score: 
The Usability Score is a subjective, continuous metric gauging the system's perceived usability. 

It is sourced from the System Usability Scale (SUS) survey questionnaire<cit.>, which comprises 10 questions.

Each question offers five response choices from “Strongly Disagree” to “Strongly Agree”, which are numerically scored from 1 to 5. 

Formally, the usability score is based on the responses to questions 3 through 12 in questionnaire <ref>. 

To quantify usability, we apply the standard scoring system of the SUS to convert the scores for each participant on these questions into a new numerical format. 

Subsequently, we calculate the sum of these scores and multiply the result by 2.5. 

This step serves to reposition the original scores, which originally ranged from 0 to 40, into a revised scale spanning from 0 to 100. 

Although interpreted like percentiles, they aren't percentages.  

Higher scores signify better-perceived usability, which is mathematically defined as:

    = 2.5 ×∑_i=1^5(-1) + (5 -).




  * Workload Index: This subjective, continuous variable assesses perceived mental workload and is derived from the NASA Task Load Index (NASA TLX) questionnaire <cit.>. 

Recognized for its comprehensive evaluation of mental workload, the NASA TLX divides workload into six categories: Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration.

Participants rate each category on a scale of 1 to 20 (questions 13 to 18). They also evaluate the significance of 15 pairs of these categories in shaping the overall workload (questions 19 to 33).

The scale score for each dimension is calculated as × 5. The weighted score  is determined based on the frequency of selection for each dimension as more important in questions 19 to 33, divided by 15. 

The overall workload score, ranging from 0 to 100, is then computed by summing the products of the scale score and weighted score for each dimension as follows:

    = ∑_i=13^18· (5 ·.




The most direct independent variables stem from the differences in approaches between participants using  and AutoGluon when performing tasks. Furthermore, various independent variables have the potential to impact user outcomes, including:

    
  * Participant Background: These categorical variables encompass background information about the participants, such as their professional roles, providing deeper insights into potential background knowledge, biases, or preferences that users may bring to task execution.
    
  * Familiarity with Technology: These numerical variables represent each participant's familiarity with terminal operations, the Python programming language, LLM, and AutoML methods. Familiarity levels can potentially impact the ease with which participants complete AutoML tasks, thus influencing the final measurement outcomes.




 §.§ User Study Analysis Process
 



  * Collected Data.
We collected both objective and subjective evaluations from each user regarding the systems, including task execution time, number of attempts, usability scores, and workload indices. Box plots for these four variables are presented individually in Fig <ref>. Each box plot displays the minimum value, first quartile (Q1), median (Q2), third quartile (Q3), and maximum value for these variables. The box represents the interquartile range (IQR) from Q1 to Q3, with a line inside indicating the median. 








  * Normality Testing.
To ensure the validity of our subsequent statistical analyses, we conducted a normality test on our data using Q-Q plots, as depicted in Fig <ref> and Fig <ref>. The proximity of our data points to the theoretical quantile lines, along with the bell-shaped curve observed in the histograms, suggests that task completion time, the number of attempts, usability score and workload reasonably adhere to the assumption of normality.


  * Hypothesis Testing.
We employed hypothesis testing to assess the statistical significance of the observed performance differences between the AutoGluon and  conditions. The differences we are analyzing, denoted as , were calculated by taking the AutoGluon measurements and subtracting the corresponding   measurements. Assuming the null hypothesis, both AutoGluon and  exert an equivalent impact. Consequently, these differences are expected to adhere to a distribution centered around zero, denoted as  = 0. Our dataset for hypothesis testing comprises 20 samples, and we express the null and alternative hypotheses as follows:


    :   = 0      : 0


This applies to the testing of hypotheses H1, H2, and H4. In the case of testing H3, the alternative hypothesis is that 0.
Here, d and  denote the sample mean and sample standard deviation of the observed differences, respectively. With these parameters in mind, the sampling distribution of the test statistic follows a t-distribution with degrees of freedom equal to n-1. Consequently, under the null hypothesis ,


    τ   = d//√(n)∼






§ EXPERIMENT IMPLEMENTATION




 §.§ Implementations for Quantitative Evaluations

In our quantitative assessment, we primarily relied on  OpenAI’s APIs: <cit.> for code generation, and the <cit.> for text completion, and <cit.> for text embedding. 

For all APIs, we set the temperature parameter to 0 to maximize determinism.

The experiments utilized the PyTorch Lightning framework<cit.> for model training, and Ray<cit.> served as our tool for hyperparameter search. 

Furthermore, we set the mode selection of AutoGluon to "best quality", ensuring optimal accuracy.

While we consistently used the same models in our multimodal experiments as those in the AutoGluon assessments for the same modality data, our emphasis on the model selection module was not solely on accuracy. Instead, we were driven by the goal of intelligently choosing models based on data modality and user-specific needs.



In order to obtain more robust experimental results, in all quantitative experiments, we used 10-fold cross-validation and reported the mean and standard deviation. For the retrieval dataset, stratified sampling of IDs is performed for each fold, and matching positive and negative sample pairs are created for verification.



 §.§ Implementations in User Study

For the user study, given the advanced capabilities of GPT-3.5, we chose to employ the  API as the LLM backbone of . Participants in the study were provided execution scripts for both AutoGluon and , allowing them a comparative experience. 























































































































































































































































































































    



    

    

    

    



    

    

    

    

    

    

 





    

    

    

    

    

    





    

    

    

    

    

    



    

    

    

    

    

    

    

    

    




  







  





    

    

    

    

    

    

    

    



    

    

    

    

    

    

    

    



    

    

    

    

    

    

    

    



    

















      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      



      

      

      

      



      

      

      















§ RELATED WORKS



 §.§ AutoML


AutoML has emerged as a transformative paradigm to streamline the design, training, and optimization of ML models by minimizing the need for extensive human intervention. 

Current AutoML solutions predominantly fall into three categories: (i) training pipeline automation, (ii) automated feature engineering, (iii) hyperparameter optimization.

Within the sphere of automated feature engineering, certain methodologies have carved a niche for themselves. 

For instance, DSM<cit.> and OneBM<cit.> have revolutionized feature discovery by seamlessly integrating with databases, curating an exhaustive set of features.

In a complementary vein, AutoLearn<cit.> adopts a regression-centric strategy, enhancing individual records by predicting and appending additional feature values.

Concurrently, training pipeline and hyperparameter optimization automation have also seen significant advancements.

For example, H2O AutoML<cit.> is particularly noteworthy for its proficiency in rapidly navigating an expansive pipeline search space, leveraging its dual-stacked ensemble models.



However, a recurring challenge across these AutoML solutions is their predominant focus on uni-modal data, which limits their applicability to more complex multimodal data. 

At the same time, most of the current multimodal methods<cit.> designed for different application scenarios are not end-to-end and only support limited modality inputs, which limits the usability of the framework.

Recognizing this gap, we introduce a novel LLM framework tailored specifically for multimodal AutoML scenarios.




 §.§ Large Language Models


The domain of Natural Language Processing has undergone a paradigm shift with the introduction of LLMs<cit.>. 

With their staggering parameter counts reaching into the hundreds of billions, LLMs have showcased unparalleled versatility across diverse tasks. 

A testament to their evolving capabilities is Toolformer<cit.>, which equips LLMs to interact with external utilities via API calls, thereby expanding their functional horizons.

AutoGPT further exemplifies this evolution, segmenting broad objectives into tangible sub-goals, subsequently executed through prevalent tool APIs, such as search engines or code executors.

Yet, as we embrace the potential of LLMs to manage AI tasks via API interactions, it's crucial to navigate the inherent intricacies. 

Model APIs, in particular, often require bespoke implementations, frequently involving pre-training phases which highlights the pivotal role of AutoML in refining and optimizing these intricate workflows.

Our proposed AutoML framework aspires to bridge this gap, enabling fluid user-AI engagements through lucid dialogues and proficient code generation.



§ DISCUSION



 §.§ Biases and fragileness in LLMs

Large lauguage models may contain biases that influence system performance and fairness. For instance, LLMs might exhibit gender or racial biases, leading to discriminatory outcomes during training and testing phases. We speculate the bias issues in LLMs may impact the 's Automatic Feature Engineering module. For example, we expect LLM to identify and select skill names (attributes) relevant to job requirements. If LLM encounters bias in the training data, it might result in the following issues:

    
  * Gender Bias: The model might be inclined to select skill names associated with a specific gender, overlooking other equally important skills. For instance, there could be a tendency to select skills related to roles like engineers or programmers, neglecting skills required for roles such as nurses or educators.
    
  * Industry Bias: The model might favor selecting skill names commonly used in that industry, neglecting skills required in other industries. This could lead to an imbalance in attribute selection across diverse industries.


To mitigate this issue, we propose:

    
  * Fine-tuning LLMs: Users should be aware of potential biases in LLMs and take measures to mitigate them. The selection of training data is crucial, ensuring that the dataset is diverse, inclusive, and covers various aspects such as gender, race, and culture.
    
  * Review and Correct Output of  Module: Setting rules or adding post-processing steps to ensure the generated results do not contain adverse biases.
    
  * Improving Prompt Engineering for  Module: For example, in the prompts, include diverse examples covering different genders, races, and industries. More specifically, design prompts relevant to the task's specific context to guide the model in better understanding and selecting attribute names. Additionally, reduce bias impact by introducing positive and negative examples. For instance:
    
        
  * Positive Examples: Include positive examples related to various professions and skills, such as "programming," "project management," "communication skills," etc.
        
  * Negative Examples: Introduce some irrelevant or inappropriate attributes, such as "gender," "appearance," "age," etc. These attributes are ones that I hope the model can ignore.
    
    




 §.§ Computation cost

The entire process of multimodal AutoML consists of two parts: setting up the training pipeline and the actual training of the model.

Our proposed LLM framework only participates in the setup phase. Once training begins, LLM is no longer used.

The majority of computational resources are consumed by the training itself, which remains unchanged regardless of LLM usage. 

Although LLM can introduce additional cost during the setup phase, this cost is minimal and has a negligible impact on the overall computational expense.


Moreover, the current API calls of LLM allow for real-time performance, making the training duration comparable to traditional methods.

For example, using dataset PAP, we conducted experiments on an RTX 3090 with identical settings for AutoM3L and AutoGluon (same model and number of hyperparameter search trials).

Our training took 18.4 hours, while AutoGluon's took 18.3 hours, indicating that LLM only added 0.5% to the total training time.




From an economic perspective, this experiment consumed approximately 200K tokens, costing no more than $0.3 based on the current GPT-3.5 API pricing. 



Thus, incorporating LLMs does not result in a significant economic burden.



 §.§ Different LLMs

We compared the use of GPT-4 and GPT-3.5-turbo in our framework, except for the code generation module. 

In a single trial on the PAP dataset, AutoM3L w/GPT4 achieved an accuracy of 0.441, compared to AutoM3L w/GPT3.5’s 0.439.

AutoM3L w/ GPT4 demonstrated higher accuracy, which we attribute to GPT-4's superior comprehension and reasoning abilities.



 §.§ Robustness of model selection

We employed the model zoo of HuggingFace Transformer, TIMM including different sizes of models for each modality.
As shown in Table <ref>, we generated 10 different commands from a user instruction using GPT-3.5, all of which were accurately mapped to the lightweight models, indicating the robustness of MS-LLM.



 §.§ Interpretability

AutoM3L offers an interactive interface that allows users to actively participate in the pipeline construction process and receive explicit feedback at each stage.
During the model selection and hyperparameter optimization, users receive feedback from the LLM, 
such as why the selected model card matches user requirements, and the rationale behind the chosen hyperparameters, which provides a certain degree of transparency. 





 Paper Title: Analyzing the Effectiveness of Quantum Annealing with Meta-Learning.





[1]Riccardo Pelliniriccardo.pellini@polimi.it

1]Maurizio Ferrari Dacremamaurizio.ferrari@polimi.it


*[1]
Politecnico di Milano,

Milan, 
Italy



The field of Quantum Computing has gathered significant popularity in recent years and a large number of papers have studied its effectiveness in tackling many tasks.
We focus in particular on Quantum Annealing (QA), a meta-heuristic solver for QUBO problems.
It is known that the effectiveness of QA is dependent on the task itself, as is the case for classical solvers, but there is not yet a clear understanding of which are the characteristics of a problem that makes it difficult to solve with QA.
In this work, we propose a new methodology to study the effectiveness of QA based on meta-learning models. 
To do so, we first build a dataset composed of more than five thousand instances of ten different optimization problems. We define a set of more than a hundred features to describe their characteristics, and solve them with both QA and three classical solvers. We publish this dataset online for future research.
Then, we train multiple meta-models to predict whether QA would solve that instance effectively and use them to probe which are the features with the strongest impact on the effectiveness of QA. 
Our results indicate that it is possible to accurately predict the effectiveness of QA, validating our methodology. Furthermore, we observe that the distribution of the problem coefficients representing the bias and coupling terms is very informative to identify the probability of finding good solutions, while the density of these coefficients alone is not enough. The methodology we propose allows to open new research directions to further our understanding of the effectiveness of QA, by probing specific dimensions or by developing new QUBO formulations that are better suited for the particular nature of QA. Furthermore, the proposed methodology is flexible and can be extended or used to study other quantum or classical solvers.





Analyzing the Effectiveness of Quantum Annealing with Meta-Learning
    [
    August 5, 2024
===================================================================





§ INTRODUCTION


In recent years the field of Quantum Computing has gathered significant popularity, thanks to remarkable advancements that led to the development of several quantum computers of different architectures and technologies that can be used to tackle numerous problems. Although quantum computers are still limited both by their relatively small size and by the noise that limits the precision of the computation, the field is rapidly moving forward.
Among the existing Quantum Computing paradigms, Quantum Annealing (QA) is a meta-heuristic that can be used to solve Quadratic Unconstrained Binary Optimization (QUBO) problems, a family of NP-hard optimization problems. 
The key idea of QA is to represent a QUBO problem as an energy minimization problem of a real and configurable quantum device. To do so, the problem variables are mapped onto physical quantum bits, or qubits.
The quantum device is steered towards a state of minimal energy, called ground state, with a controlled evolution. The ground state corresponds to the optimal solution of the original QUBO problem. The devices that implement the QA process are called Quantum Annealers.

The ability of QA to tackle NP-hard optimization problems and its flexibility to heterogeneous domains is what makes it an interesting technology for industries and researchers. Many applications of QA have been proposed in the fields of machine learning <cit.>, chemistry <cit.> and logistics <cit.>, but the results are not always competitive against classical heuristics solvers.

An important issue is that the quality of the solutions found by QA is limited by multiple factors. 
First of all, Quantum Annealers are physical devices which have a limited number of qubits and connections between them. This limits the size of the problems that they can tackle and requires to process the QUBO problem adapting it to the physical structure of the Quantum Annealer. 
A second important aspect is that the quality of the solutions found by QA depends on the behaviour of the underlying physical quantum system, which is very difficult to study. It is known that some problems appear to be more difficult to solve with QA <cit.>, but understanding why is not a trivial task and still an open research question.

Most of the previous studies on QA compare its performance, in terms of required time for computation with respect to other heuristic solvers, rather than on the quality of the solutions it finds, its effectiveness. There are two ways in which one can study the effectiveness of QA, one is by analytically describing the underlying quantum behaviour and the other is to perform empirical experiments.
A theoretical analysis has been performed for very small QUBO instances <cit.>, which however are too simple to assess the effectiveness of QA when compared to other classical solvers. Furthermore, analytically analysing such a quantum system becomes rapidly very expensive and is generally impossible for problems of interesting size. 
On the other hand, the existing empirical studies on the effectiveness of QA have explored much larger problems but focus mainly on specific tasks such as feature selection <cit.>, clustering <cit.> and classification <cit.>, and therefore lack generality.

To the best of our knowledge, there is no published research which has investigated extensively how the characteristics of the problem impact the effectiveness of QA. 
For this reason, in this study we propose a novel empirical methodology for the analysis of the effectiveness of QA, based on the study of the characteristics of QUBO problems with a meta-learning approach. The general idea consists in generating many QUBO instances, defining a set of features which can describe them and train meta-models to predict whether QA would solve that problem or not. 
Our key contributions are as follows:

    
  * The design of an experimental methodology which can be applied to study the effectiveness of QA. This methodology can be used also for other quantum algorithms, such as QAOA <cit.> or VQE <cit.>;
    
  * The selection of ten classes of optimization problems, each one with specific characteristics, from which we generate approximately five thousand QUBO instances; 
    
  * The design and the generation of a meta-learning dataset, which contains for each of the five thousand instances a selection of a hundred features based on probability theory, statistics and graph theory. We show that using them it is possible to effectively predict whether QA would solve a problem instance effectively or not. We share the meta-learning dataset online for further research;
    
  * The analysis of the features of a QUBO problem with the strongest impact on the effectiveness of QA;




§ BACKGROUND
 


 §.§ QUBO and Ising Models

In order to use QA to tackle optimization problems, these should be represented with one of two equivalent formulations called QUBO and Ising, suitable for NP-Complete and some NP-Hard optimization problems <cit.>. While the two are equivalent, the QUBO formulation is closer to traditional Operations Research, the Ising formulation is instead closer to Physics.

The objective function in the QUBO model is given by Equation <ref>, where x ∈{0,1}^n is a column vector representing the assignment of the binary variables x_1, x_2, ..., x_n, n is the number of problem variables, y the cost, and Q ∈ℝ^n × n is a real square matrix, either symmetric or upper triangular.

    min_x y = x^TQx

We will refer to combinatorial optimization problems written in the QUBO formulation as QUBO problems. 
Note that the QUBO formulation does not allow for hard constraints. 
An optimization problem with constraints can be transformed into a QUBO problem by introducing a quadratic penalty term multiplied by a penalty coefficient p. The idea is that the hard constraints are transformed in soft constraints, such that if they are violated a positive penalty p is added to the cost function making the cost of that variable assignment worse. Note that by using soft constraints we do not have the guarantee that the optimal solution will satisfy the constraints, which may happen frequently if the penalty coefficient p has a value that is too low. 
In general, a quadratic binary optimization problem with equality constraints formulated as Ax - d = 0, where d ∈ℝ^m and A ∈ℝ^m × n, can be transformed into the following QUBO problem:

    min_x y    = x^TQx + p · x^TCx 
    
        C    = (Ax - d)^T(Ax - d)

If the quadratic binary optimization problem also has inequality constraints, those need to be transformed first into equality constraints using binary slack variables.
For example, if we have the following constraint:

    x_1 + 2x_2 + 4x_3 ≤ 3

we can transform it into an equality constraint by introducing the binary slack variables x_4 and x_5:

    x_1 + 2x_2 + 4x_3 + x_4 + 2x_5 = 3

There exist no general rule to choose the best number of slack variables, so multiple strategies can be followed.

A second useful formulation is the Ising model, which was developed to describe an energy minimization problem for a system of particles <cit.>.
The objective function of the Ising model is given by Equation <ref>, where s ∈{-1,1}^n is the column vector representing the assignment of the n problem variables s_1, s_2, ..., s_n also called spin variables, J ∈ℝ^n × n is the coupling matrix that describes the quadratic terms of the objective function and has zero diagonal, b ∈ℝ^n is the bias vector, which contains the linear terms of the objective function. The constant term c ∈ℝ is called offset.

    min_s y = s^TJs + b^Ts + c

A QUBO problem can be transformed into an Ising problem through a linear mapping of the variables. In particular, a binary variable x_i is transformed into a spin variable s_i according to the following conversion[This mapping is equivalent to the more commonly used x_i = 1+s_i/2 with the difference that in our case a binary 0 is mapped onto spin 1 and binary 1 is mapped onto spin -1.]:

    x_i = 1-s_i/2




 §.§ Quantum Annealing and Quantum Annealers

Quantum Annealing (QA) is a meta-heuristic solver for QUBO problems.
It is based on the Adiabatic Quantum Computation (AQC) paradigm, with some relaxations <cit.>.
The idea is to represent the optimization problem as an energy minimization one, and then use a configurable device that exhibits the needed quantum behaviour to minimize it. Such a device, the Quantum Annealer, is composed by multiple qubits connected between each other. QA works based on a time evolution of the quantum system. The initial state of the system is a default one, easy to prepare, so that the qubits are in a state of minimal energy, the ground state. Then, the physical system is evolved slowly over a short amount of time by introducing a dependency on the Ising coefficients of the problem one wishes to solve. This means, for example, slowly changing the magnetic fields the qubits are subject to. At the end of the evolution, the physical system will depend only on the problem and, if the evolution was careful enough, it will still be in the ground state. Since the state of minimal energy is also the solution of the optimization problem, measuring the state of the qubits will yield the values that the problem variables should have. 

The evolution of the system in QA occurs in a noisy environment and is subject to quantum fluctuations, quantum tunneling, which helps it explore the solution space. 
The noise of the system and the duration of the evolution influence the results of QA, if the evolution is too fast the system will likely escape its ground state and find a worse solution, while if the evolution is too slow noise may build up and push the system again out of the ground state. 
Due to its stochastic nature, QA acts as a device sampling low-cost solutions in a similar way as other classical solvers do, such as Simulated Annealing.
For this reason, QA is repeated multiple times in order to obtain samples of the final state of the quantum system. 

The physical devices that implement QA are called Quantum Annealers.
Currently, D-Wave Systems Inc. is the company that provides the Quantum Annealers with the largest number of qubits[The documentation of the D-Wave Systems' services: <https://docs.ocean.dwavesys.com/en/stable/index.html>]. 
For example, the D-Wave Advantage has more than 5000 qubits with a topology called Pegasus, where each qubit is connected to other 15 ones.

Solving a QUBO problem with a Quantum Annealer requires the following steps:

    
  * Formulate the problem as a QUBO or an Ising problem: the coefficients that are needed to configure the Quantum Annealers are those of the Ising formulation, as such the problem needs to be in this form. If the problem has a simpler formulation as a QUBO, the transformation is straightforward. Note that some problems can be formulated as QUBO or Ising easily, while others require more expensive processing.
    
    
  * Embed the problem on the topology of the device: since the Quantum Annealer is a physical object, we must fit the problem we want to solve on it, accounting for the limited number of qubits and of the connections between them. 
    This procedure is called minor-embedding <cit.> and maps each problem variable to one or more qubits. If multiple qubits are needed to represent a single problem variable, that is called a qubit chain. If the problem has a large number of quadratic terms, a substantial number of qubits may be needed to create all the physical connections.  
    Figure <ref> shows an example of how a simple problem can be mapped on a Quantum Annealer.
    
    Minor-embedding is an NP-Hard problem but polynomial-time heuristic algorithms are available <cit.>[We use the library  offered by D-Wave: <https://docs.ocean.dwavesys.com/en/stable/docs_minorminer/source/sdk_index.html>].

    
  * Evolution of the system and sampling of the solutions:
    once the minor-embedding is done, the problem is transferred to the Quantum Annealer. First, the device is programmed with the problem coefficients, then we can perform a sequence of multiple evolutions to obtain the desired number of samples n_s. Each sample requires three steps: (i) the evolution is run for the desired duration, called annealing time t_a, (ii) the final state of the system is measured, which requires a read-out time t_r dependent on the number of qubits used, and (iii) the device pauses shortly for cooling.




More formally, the energy of a system can be modeled with an Hamiltonian, H ∈ℝ^2^n × 2^n, and the evolution that occurs in QA is described by the time-dependent Hamiltonian H(t) that models the transition from the initial default Hamiltonian H_i[
The initial Hamiltonian H_i is also called driver Hamiltonian.] and the Hamiltonian describing the problem H_p:

    H(t) = A(t)H_i + B(t)H_p

The coefficient A(t) decreases as the evolution progresses, while B(t) increases introducing the dependency on the characteristics of the problem, but their exact values depend on the hardware. At the beginning of the evolution B(t) is zero, while at the end A(t) is zero. Note that this is just a description of the underlying physical system and there is no need to compute this representation to use QA.

In the ideal Adiabatic Quantum Computing setting, it is possible to compute the exact annealing time needed to ensure the system remains in the ground state and finds the global optimum, this result dates back from a century ago <cit.>. This optimal annealing time is inversely proportional to the smallest difference between the two smallest eigenvalues λ_1(t), λ_2(t) of H(t). Such difference is called minimum gap. Although this result may be useful to understand its behaviour, it is not applicable to QA because it is subject to noise. Furthermore, computing the eigenvalues of H(t) is prohibitive for all but the smallest problems.

To exemplify how this representation works, assume to have an Ising problem of n variables with coupling J and bias b, H_p is a 2^n × 2^n matrix computed as follows:    
    H_p = ∑_i=1^n ∑_j=i^nJ_ijσ_z^(i)σ_z^(j) + ∑_i^n h_i σ_z^(i)

The matrix σ_z^(i) is the Z-Pauli operator σ_z acting on qubit i:

    σ_z = [r]
            1     0 
    
            0     -1


    σ_z^(i) = ⊗_k=1^i-1I  ⊗σ_z ⊗_k=1^n-iI

with ⊗ being the tensor product and I the identity matrix. A useful property of H_p is that it is a diagonal matrix that contains all the cost values for all possible variable assignments of the problem. Since it is diagonal, these values are also its eigenvalues and the corresponding eigenvectors encode the variable assignment that has that cost. The minimum eigenvalue of H_p corresponds to the minimal cost and the corresponding eigenvector to the optimal variable assignment.

As an example, consider the following QUBO problem, which is minimized when x_1 = x_2:

    min_x_1, x_2 y = x_1 + x_2 - 2x_1 x_2

The equivalent Ising formulation is:

    min_s_1, s_2 y = 1/2 - 1/2s_1 s_2

For this small instance we can compute H_p easily.
The matrices σ_z^(1) and σ_z^(2) are:

    σ_z^(1) = σ_z ⊗ I = [  1  0  0  0;  0  1  0  0;  0  0 -1  0;  0  0  0 -1;    ]  σ_z^(2) = I ⊗σ_z = [  1  0  0  0;  0 -1  0  0;  0  0  1  0;  0  0  0 -1;    ]

H_p is then equal to:

    H_p = 1/2[ -1  0  0  0;  0  1  0  0;  0  0  1  0;  0  0  0 -1;    ]

The smallest eigenvalue of H_p is λ_min = -1/2 and has a multiplicity of two corresponding to the first and last eigenvalues. Indeed, both x_0 = 0, x_1=0 and x_0 = 1, x_1=1 are optimal solutions of the problem. If we sum λ_min with the offset of the Ising problem, 1/2, we obtain 0, that is the same value of the QUBO cost function when x_1 = x_2.



 §.§ Studies on the Effectiveness of QA

Most of the previous studies on QA focus on its performance, by measuring the time required to solve a problem and comparing it to that of classical solvers.
To the best of our knowledge there is no consensus on whether QA provides a general and consistent speedup compared to other traditional solvers for QUBO problems <cit.>, while a recent paper claims substantial speedup for a quantum simulation task <cit.>. Although some papers may claim a speedup, this is often based on measurements that only account for part of the process. Indeed, one should consider the time required by all phases: (i) formulating the optimization problem as QUBO or Ising, (ii) embedding the problem on the QA, (iii) sampling the solutions on the device and (iv) postprocessing the results if needed (for example by checking if the constraints are satisfied). Frequently, the efficiency of QA is measured by only accounting for the usage of the quantum device itself (programming time and the repeated annealing and read-out) while ignoring the time needed for minor embedding and for creating the QUBO formulation. This gives an incomplete picture of the technology that does not account for two significant bottlenecks. For example, it may be that in a certain situation QA is faster than other traditional methods in solving a specific QUBO problem, but that may not be the case anymore if one includes the minor embedding phase. Furthermore, if it is very computationally expensive to formulate the problem as QUBO, it may be more efficient to use other traditional methods that do not need a QUBO formulation at all.

When comparing the quality of the solutions found by QA and classical solvers, their effectiveness, the published literature usually focuses on problems related to specific fields or even to very specific instances of those problems. Due to this, there is still a limited understanding of how would QA compare in a more general setting. For example, the effectiveness of QA has been analyzed for feature selection <cit.>, classification <cit.> and  clustering <cit.>, which are typical machine learning tasks. In the field of chemistry, QA has been applied and analyzed to find the equilibria of polymer mixtures <cit.>, to find similarities between molecules <cit.> and to find their ground state <cit.>.
The effectiveness of QA in solving problems related to logistics has been analyzed too, for example in solving the Nurse Scheduling Problem <cit.> and in optimizing the assignments of the gates at the airport <cit.>.

Previous research also studied the effectiveness of QA from a theoretical perspective by representing analytically the evolution of the time-dependent Hamiltonian H(t) (see Equation <ref>) and computing the probability of escaping the ground state <cit.>. This approach is however limited by the fact that the size of the Hamiltonian grows exponentially on the number of QUBO problem variables n, and the analytical analysis of the Hamiltonian becomes rapidly impractical for all but the smallest problems.
An alternative way is to adopt an empirical approach, by using the outcome of multiple experiments to probe the underlying physical system <cit.>. The idea is to allow the evolution to progress up to a certain intermediate stage and then drastically accelerating it (a quench, according to D-Wave terminology), observing how the effectiveness changes based on when was the evolution accelerated. While this approach allows to tackle large problem instances, applying the acceleration at different stages of the evolution requires to repeat the experiment a large number of times and therefore this approach too is very resource intensive.

To overcome the limitations of the methods adopted in the literature, this paper proposes a new empirical approach to study how the quality of the solutions found by QA is impacted by the characteristics of the problem.
To achieve this, we first collect a dataset of problem instances belonging to 10 selected problem classes and solve them using both QA and three classical solvers. Then, for each problem instance, we compute a set of features describing various characteristics, from the distribution of the bias coefficients of its  QUBO formulation to the topology of the graph that describes the instance once it has been embedded on the Quantum Annealer. Using this dataset we train a machine learning classifier to identify whether QA was able to find a good solution for that instance and, finally, use it to assess which are the most important problem features.



§ META-LEARNING DATASET GENERATION

In this section, we present the methodology used to generate our meta-learning dataset, on which we train the meta-models to predict the effectiveness of QA. We publish this dataset online for future research.
First, we describe how we select the ten classes of problems we want to solve with QA and the strategies we use to generate the five thousand instances.
Then, we describe how we evaluate the effectiveness of QA, in terms of closeness to the optimal solution of the problem and by comparing QA with other classical methods (Simulated Annealing, Tabu Search and Steepest Descent).
Third, we describe the representations of the QUBO problem we used to compute the approximately one hundred features used to train the meta-models.
Finally, we describe how we solve the instances with QA and with the classical solvers, with a particular focus on the choice of the optimal hyperparameters of the solvers.



 §.§ Selection of Problems and Instances

We identify a selection of ten different optimization problems that exhibit different characteristics: some have constraints, others do not; some have linear terms, others do not; some have a large number of quadratic terms while others do not, etc. The details on their formulations are reported in Appendix <ref> and the details on the generation of the instances are in Appendix <ref>.

The first group contains five classes of optimization problems defined over a graph: Max-Cut, Minimum Vertex Cover, Maximum Independent Set, Max-Clique and Community Detection.
They were selected for the following reasons. Both the Max-Cut and Community Detection problems have a straightforward QUBO formulation that does not require penalties to represent constraints. The Max-Cut, Maximum Independent Set and Minimum Vertex Cover problems share the same quadratic terms in their QUBO matrix, but not the diagonal (the linear terms or bias). The Max-Clique problem is formulated as a Maximum Independent Set problem but defined on the complement graph. The Community Detection problem has a very dense QUBO matrix as there are quadratic terms between all variables and is a relevant problem in Machine Learning <cit.>.
Since these problems are formulated on a graph, we apply them on four different graph topologies: Erdös-Renyi, Cyclic, Star and 2d-grid. Note that in order to have a diversified set of instances we introduce small random perturbations to each topology, consisting in few edge insertions and deletions. The number of insertions and deletions depends on the number of nodes of the graph. More details are reported in Appendix <ref>.

The second group of five optimization problems contains: Number Partitioning, Quadratic Knapsack, Set Packing, Feature Selection and 4 × 4-Sudoku. These are a more heterogeneous set than the previous graph-based problems ad so require ad-hoc strategies to generate their instances which we detail in Appendix <ref>.
Similarly to the Max-Cut and Community Detection problems, the Number Partitioning problem has a straightforward QUBO formulation with no penalty terms to represent constraints.
Similarly to the Community Detection problem, the Feature Selection problem has a dense QUBO matrix with quadratic terms between all variables.
Finally, the Quadratic Knapsack, Set Packing and 4 × 4-Sudoku problems are all Constraint Satisfaction Problems, each with different types of constraints.
In particular, the Quadratic Knapsack problem has inequality constraints that need to be converted in equality constraints using slack variables.

We generate multiple instances of all the problem classes we selected. 
Concerning the size of the problem instances, measured in the number of problem variables, there are two constraints to take into account. First, the D-Wave Quantum Annealer that we use has more than 5000 qubits but, due to their limited connectivity, it is generally possible to tackle problem instances up to between 100-200 variables depending on the structure of the QUBO problem. This is due to the minor-embedding phase. Second, we want the instances to be representative of problems that are not trivial and with a Hamiltonian that could not be analyzed analytically. 
In order to provide a more complete picture, we are also interested to assess the impact of the distribution of the solution space of the problem. This, formally, corresponds to the set of eigenvalues and eigenvectors of the Hamiltonian of the problem H_p (see Section <ref>). Unfortunately, it is impractical to compute them for instances of more than 32 problem variables, which may be too small and easy to allow a comparison on the effectiveness of different solvers. For this reasons we decided to create two separate sets of instances:

    
  * One set of large instances, with 5114 instances of between 69 to 99 variables, the upper range of what can be tackled with the QA; 
    
  * One set of small instances, with 246 instances of between 27 to 32 variables. With this set of instances we can do a more complete analysis which includes also the distribution of the solution space.


The number of problem instances for each optimization problem is summarized in Table <ref>. 
Notice that the instances of the 4 × 4-Sudoku problem are included only in the small instances set, since the largest possible instance is unique and it has at most 64 variables.
All the generated instances are satisfiable and, when needed, the penalty term coefficient p used in the QUBO formulation is optimized with a Bayesian Search[The range of possible values of p, for a particular instance, depends on the cost function of that instance. More details are given in Appendix <ref>] <cit.>, in order to maximise the number of feasible solutions for Simulated Annealing.





 §.§ Evaluating the Effectiveness of a Solver

In this section, we describe how to evaluate the effectiveness of a solver and, in particular, of QA.
Both QA and the traditional solvers we compare it to are stochastic and are executed multiple times to obtain a set of variable assignments that aim to minimize the cost function, which we call a set of samples. 
A sample is represented by an assignment of the decision variables x and by the related cost value y.

We solve all instances with QA, Simulated Annealing (SA), Tabu Search (TS) and Steepest Descent (SD).
Our definition of how much a solver is effective is based on whether it finds samples that meet some quality constraints. While for the small instance set it is possible to compute the global optimum, for the large ones it is not feasible to do so and therefore we define the effectiveness in relative terms with respect to the other solvers.

In particular, we evaluate the effectiveness of QA on the large instances set by comparing its samples with those of the traditional heuristic solvers. We define the samples associated to the best cost value for a solver S as y^S_min.
An instance I is QA-over-all if the best solution found by QA is at least as good as the best one found by SA, SD, TS combined. More formally, if y^QA_min≤min{y^SA_min, y^TS_min, y^SD_min}.
Comparing QA with a pool of multiple solvers results in a stricter evaluation of its effectiveness, but the condition that QA has to be at least as good as all the other solvers combined may be too strict.
For this reason, we also compare QA with each individual solver.
An instance I is QA-over-S if the best solution found by QA is at least as good as the one found by solver S, hence y^QA_min≤ y^S_min.

For the small instances we can perform a deeper analysis of the effectiveness because we can explore the full solution space and find the global optimum. This is in practice done by computing the Hamiltonian of the problem, H_p, which is a diagonal matrix enumerating the eigenvalues λ, sometimes called energy, of all the variable assignments. The eigenvalue is equivalent to the cost function y but does not include possible constant offsets c from the Ising formulation, therefore y = λ + c. 
The variable assignment x associated to an eigenvalue  λ can be computed starting from the corresponding eigenvector of H_p.
The global optimum of an instance is the assignment x_min corresponding to the minimal eigenvalue of H_p, λ_min. We will refer to the maximum eigenvalue as λ_max.

We define a sample with energy λ as ϵ-Optimal if the following condition holds:
    
    λ≤λ_min + ϵ· (λ_max- λ_min)

The ϵ-Optimality condition describes how close is the eigenvalue of a sample to the solution of the instance.
The coefficient ϵ∈ [0,1] allows to restrict the interval under which λ is considered close enough to the optimal eigenvalue λ_min. Notice that if ϵ=0 only the global optimum of the instance meets the constraint in Equation <ref>. 

We also define a sample x as Hamming-Optimal (h-Optimal) if 
it differs from any solution x_min in at most one decision variable.
This corresponds to check the Hamming distance between a sample and a solution of the instance:
    
    ||x - x_min||_Hamming≤ 1




 §.§ Meta-Learning Features


In this section, we introduce the features we define to describe a problem instance.
We rely on a selection of metrics used in statistics and probability theory, such as the Gini coefficient <cit.>, the Herfindahl-Hirschman index <cit.> and the Shannon entropy <cit.>, as well as metrics used in graph theory, such as the spectral gap, the radius a graph, its diameter and its connectivity. In total we compute 107 features, which we describe in detail in  Appendices <ref> and <ref>.

The features we compute can be grouped in multiple domains of analysis.
Overall we identify seven domains, among which we describe the three most relevant ones:

    
  * Logical Ising Graph (LogIsing): This domain uses the Ising formulation of a QUBO problem. It is represented as a graph having one node per problem variable, associated to the corresponding bias b, and an adjacency matrix that corresponds to the coupling matrix J;
    
  * Embedded Ising Graph (EmbIsing): This domain uses the Ising formulation of a QUBO problem obtained after its minor embedding on the QA. The target architecture is D-Wave Advantage with the Pegasus topology. Therefore, this formulation represents the actual problem solved by the Quantum Annealer, in which multiple qubits may be used to represent one problem variable. This formulation is represented as a graph in the same way as LogIsing; 
    
  * Solution Space (SolSpace): This domain uses the eigenvalues, or energy values, of all possible variable assignments, which can be computed only for the small instances, and aims to describe how they are distributed.

Other domains we identify are: (i) Normalized Multiplicity (NorMul), whose features are related to the multiplicity of the eigenvalues of H_p; (ii) Matrix Structure (MatStruct), which contains features related to the distribution of the values of the matrix Q of a QUBO problem; (iii) 25%-SolSpace and 25%-NorMul, which contain the same features of SolSpace and NorMul, but computed by considering only the 25% lowest eigenvalues of H_p, the energies of the 25% best solutions.

For the LogIsing and EmbIsing domains we compute several features on different mathematical objects, such as the coupling matrix J, the Laplacian matrix of the corresponding graph and the bias vector. We call such objects components. 

We can also identify sets of features that refer to the same mathematical object, but are computed on different domains. For example, both LogIsing and EmbIsing domains include features computed on the bias. We refer to them as component sets and allow us to perform an analysis of the importance of those mathematical objects that is orthogonal to that of the domains. We identified the following component sets: Coupling, Bias,  Laplacian, Structural Adjacency (StructAdj), Structural Laplacian (StructLap), where
StructAdj and StructLap gather features related to the binarized versions of the coupling and Laplacian matrices. 



 §.§ Hyperparameter Optimization of the Solvers

Since the goal of this study is to compare the effectiveness of different solvers, it is essential to ensure that each solver is using the best hyperparameters. Indeed, it is well-known in many fields that comparing methods that are not consistently optimized leads to inconsistent results that cannot be used to draw reliable conclusions <cit.>. The same applies in our case.

We optimize the hyperparameters of each solver (QA, SA, TS, SD) 
on the instance with the largest minor-embedding on D-Wave Quantum Annealer for each optimization problem class. The goal is to identify the hyperparameters that will lead the solver to find the variable assignment with the lowest cost y. Once the optimal hyperparameters have been found, they are used to solve all instances of the corresponding problem class. We optimize separately the hyperparameters used for the large and small instances sets. 
To optimize the hyperparameters of the classical solvers we use the standard QUBO formulation while for QA we use the embedded QUBO formulation: we followed this strategy because the embedded QUBO formulation is required only for QA. In this way, we have a fair comparison between different solvers since, for each one of them, we take into account only the necessary steps to solve a QUBO instance.



  
Optimal Hyperparameters of Quantum Annealing
QA has several hyperparameters that can be optimized[For a detailed explanation of all the hyperparameters of D-Wave Quantum Annealers, refer to the documentation of the devices: <https://docs.dwavesys.com/docs/latest/c_solver_parameters.html>], some of which refer to the evolution process as a whole while others allow to fine-tune it at the level of each individual qubit.
The access to the D-Wave Quantum Annealers is limited and for such a large set of instances we have devised a methodology to optimize the hyperparameters we believe are the most important: 
the annealing time t_a and the number of samples n_s. In order to perform an efficient optimization within the available resources, we define a fixed computational budget T for each instance. 
Using the default annealing time, 20 μ s, and drawing 100 samples requires, in the worst case, at most  37 ms. In our experiments we allocated T=70 ms and T=300 ms per each problem instance. 

The optimization is performed by iterating over 10 values for t_a, approximately equidistant from each other, between 5 μ s and 200 μ s. Given t_a, the number of samples n_s is computed as the maximum value allowed within the computational budget T, according to Equation <ref>. For T =70 ms, n_s is between 145 and 537 while, for T =300 ms, n_s is between 766 and 2826.

    n_s = ⌊T - t_p/t_a + t_r + Δ⌋

The term t_p ≃ 15  ms is the time needed to program the instances on the Quantum Annealer, t_r is the read-out time, needed to read the results of the annealing process, and Δ≃ 20 μ s is the delay applied after each read-out operation. The read-out time t_r is unknown a-priori because it depends on the size of the embedded problem. Based on empirical observation we use t_r=75  μ s for small instances and t_r = 150 μ s for large instances. 
We choose t_a and the related n_s which provide the sample with the lowest energy. If for multiple pairs (t_a, n_s) QA finds samples with the lowest energy, we choose the pair with the smallest t_a.

Since the results we obtained when using both computational budgets are very similar, we report those for T=70 ms. The selected hyperparameters are reported in Table <ref>.
Notice that the annealing time t_a for the large instances is often smaller than for the small instances. This highlight that, for large instances, a larger t_a does not improve the effectiveness of QA, at least in the range of values we considered and for the number of samples it allows to draw. Such result suggests that QA may require an additional optimization, for example, of the annealing schedule, which is not straightforward and it goes beyond the scope of this study.





  
Optimal Hyperparameters of the Classical Solvers

The hyperparameters of Simulated Annealing (SA), Tabu Search (TS) and Steepest Descent (SD) are optimized with the following procedure.
For the optimization of these methods we do not use a fixed computational budget because the technology is fundamentally different and, due to the various stages required by QA, it is not trivial to define such a comparison in a way that is fair.
First, we fix the number of samples to n_s = 200 which is a value comparable to that used for QA. For half of the the large instances QA uses more samples than the classical solvers, while for the remaining half the opposite is true.
We optimize the hyperparameters with a Bayesian Search of 100 iterations <cit.>. The results are available in Appendix <ref>. 
For TS, we optimize the 
number of restarts of the algorithm and the initialization strategy.
For SD, there are no hyperparameters to optimize, except for the number of samples, which we have already set.
For what concerns SA, we optimized the number of [A sweep consists in flipping a randomly chosen decision variable.], the [The schedule determines how the temperature of the system decreases over time.] and the initial state generator. We noticed however that hyperparameters we found for SA produced worse results compared to the default ones in our following analysis, which may be due to the sensitivity of SA to some of them. For this reason, we retain the default hyperparameters of 1000 , a  and a random initial state generator. 



§ META-MODEL TRAINING AND OPTIMIZATION

In this study we aim to identify which are the characteristics of a problem that impact the effectiveness of QA. We do this by first training a classification model on the dataset we have created in order to predict whether QA would solve that instance well or not based on its features. Since the classifier is trained to predict the outcome on another experiment, it is called a meta-model. Once the meta-model is trained, we can use it to probe how important are the various features.

We train the meta-models with Random Forest, AdaBoost, XGBoost and Logistic Regression, using as input data either a specific domain (LogIsing, EmbIsing, SolSpace) or a specific component set (Bias, Coupling, Laplacian), which are described in Section <ref>.
The target labels are described in Section <ref> (Optimal, ϵ-Optimal, h-Optimal) 
and they are binary, according to whether the solver meets that effectiveness condition or not.

The first step is to train the meta-model and optimize its hyperparameters to ensure it is effective in predicting the label. 
In order to measure the effectiveness of the meta-models, we have to account for the significant class imbalance of the labels towards the negative class, instances that are not solved well by QA  (see Section <ref>).
We use Balanced Accuracy (BA) to evaluate the meta-models because it is robust to class imbalance.
Given the true positives as TP, the true negatives as TN, the number of positive labels in the data as P and the number of negative labels as N, the Balanced Accuracy BA is computed as: 

    BA = 1/2(TP/P + TN/N)


The training and optimization of the meta-models is performed with 5-fold Nested Cross-Validation. First, we create a 5-folds testing split with a training fold and a testing one which we will use to train and evaluate the meta-model. In order to find the optimal hyperparameters for the meta model, we split each training fold with a further 5-folds split, the optimization split. This results in 5 optimization splits for each of the 5 training folds of the testing split and is aimed at preventing the overfitting of the meta-models. 
The splits are all stratified with respect to the problem class of the instances, to ensure every split has an equal distribution of the problem classes. All meta-models are trained on the same data splits and we perform different splits for the large and small instances.
The hyperparameters of the meta-models are optimized according to a Bayesian Search <cit.> exploring 50 configurations, we select those that provide the best Balanced Accuracy on the optimization split.

Once the meta-models have been optimized, we use them to assess which problem characteristic, feature, is most important. We use Permutation Feature Importance (PFI), 
which evaluates how the accuracy of a model drops when the values of a certain feature are shuffled. The idea is that the more important a feature is the larger will be the drop when the values of that features are shuffled. For each feature the process is repeated multiple times and the corresponding importance is given by the mean of the drop in accuracy observed.



§ RESULTS AND ANALYSIS

In this section we provide the most relevant insights of our analysis regarding the effectiveness of QA.
We have three goals: (i) determine hitch classes of
problems are more difficult to solve with QA, (ii) understand whether it is possible to predict the effectiveness of QA based on the features we have identified; and (iii) discover the domains, the component sets and the features that impact the effectiveness of QA.
To do so, we describe the results obtained by solving the instances with QA and with the other classical solvers. Then, we describe the results of the validation of the meta-models and of the Permutation Feature Importance performed on their features. We publish online a dataset with all the instances we generated, the features we computed and the samples obtained for each solver.[The instances, the dataset with the features, the results of the solvers and an example script to train meta-models are available at this GitHub repository: <https://github.com/qcpolimi/QA-MetaLearning>.]



 §.§ Effectiveness of QA for Large Instances





In this section we discuss the effectiveness of QA compared to the other classical solvers (SA, TS and SD) on the large problem instances. Table <ref> reports the results on each problem class according to the labels we defined in Section <ref>, whether the best sample found by QA is at least as good as that found by a specific solver (QA-over-SA, QA-over-TS and QA-over-SD) or by all of them combined (QA-over-all).

As a general comment, we can observe that for less than half of the problem classes (4 out of 9) QA solves effectively more instances than at least one classical solver, while for most of the problem classes (7 out of 9) QA is more effective than at least one of the classical solvers for some particular instances.
However, if we combine all classical solvers QA is more effective only in three problem classes but mostly to a limited extent. Only for Max-Cut QA shows a consistently high effectiveness. 
These results confirm that the effectiveness of QA depends on the problem class, as is the case for classical solvers, which is consistent to what observed in previous studies <cit.>.
If we compare QA and classical solvers throughout the problem classes, we can see that QA is very frequently more effective than SA, while it is more effective than TS or SD only on some specific problem classes.
As a result, we conclude that comparing QA only with SA, without considering other solvers, is not the best practice to evaluate the effectiveness of QA.

Regarding the characteristics of the problem classes, a first observation we can make is that QA is more effective on problems that do not require penalties to represent constraints: Max-Cut, Community Detection and Number Partitioning. This suggests that the presence of constraints is a factor that makes a problem more difficult to solve with QA. The reason for this may be due to the type of quadratic terms introduced by the penalties which could open new research directions in whether one could use a different formulation for the same constraint that is more suitable for QA <cit.>.
Furthermore, remember that Max-Cut, Maximum Independent Set and Minimum Vertex Cover share the same Ising coupling matrix J, with the exception of a multiplicative factor, but have a different bias vector b. We can observe how Max-Cut is the only one among them that is solved effectively by QA, suggesting that the bias structure plays an important role as well.

Lastly, a high number of quadratic terms (a dense coupling matrix J) does not always negatively affect QA. In particular, both Community Detection and Number Partitioning have a dense coupling matrix but still 13% of Community Detection instances and the 30% of Number Partitioning instances are solved effectively with QA.



 §.§ Effectiveness of QA for Small Instances



In this section we discuss the effectiveness of both QA and the other classical solvers (SA, TS and SD) on the small problem instances. For these instances we can compute the cost associated to all variable assignments and the global optimum. We do this by computing the Hamiltonian of the problem H_p and use its eigenvalues (its diagonal). 
We also compute the maximum energy values needed to assess the ϵ-Optimality for the samples of QA.



Table <ref> compares the effectiveness of the solvers according to the labels defined in Section <ref>, if the solver finds the global optimum (Optimal), if the energy of the best sample is close to that of the global optimum (for ϵ-Optimal we use ϵ = 10^-5), if the variable assignment of the best sample has an Hamming Distance of at most 1 with any of the global optimum solutions (h-Optimal).

Consistently with what observed for the large instances, QA is less effective than the classical solvers on all the effectiveness conditions. 
If we compare 10^-5-Optimal and h-Optimal, we can see that QA finds more h-Optimal samples than 10^-5-Optimal samples, as opposed to the other solvers. This indicates that QA finds more easily samples which are close to the optimal ones in terms of Hamming distance rather than energy.
Notice that in this experiment SD is the most effective solver, this may be related to the small size of the instances which may make them relatively easy to solve with simple strategies.



As done for the large instances, we compare the effectiveness of the solvers on the problem classes. Table <ref> shows the fraction of problem instances in which the solver finds the global optimum (Optimal). Overall, as opposed to what we observed for the large instances, on the small ones QA is never more effective than the classical solvers. 
QA seems to be more effective for problems that are defined over graphs (Max-Cut, Maximum Clique, Community Detection) compared to the ones that are not. Note however that on two graph problems, Minimum Vertex Cover and Maximum Independent Set, QA performs significantly behind the classical solvers. 
Based on the analysis of the large instances (see Section <ref>) we observed that the bias of the problem seems to play a role in affecting the effectiveness of QA. This observation is confirmed here as well since QA is much more effective in solving Max-Cut problems than in solving Maximum Independent Set and Minimum Vertex Cover ones.
We can also observe that the effectiveness of QA is quite poor for Set Packing and Feature Selection, being significantly behind the classical solvers. 

Interestingly, on Max-Cut and 4 × 4-Sudoku problems almost all the solvers find the global optimum, while for the Quadratic Knapsack hardly any instance can be solved optimally at all, with the most effective solver being TS with 4% of the instances solved optimally.

As a second analysis we study the effectiveness of QA in sampling solutions with a variable assignment that is close to one of the optimal ones in terms of Hamming distance (h-Optimality). The results reported in Table <ref> are consistent with the previous ones in which we assessed the ability of the solvers to find the global optimum, see Table <ref>.
We should note QA exhibits much better effectiveness, when measured in this way, being able to sample solutions close to the optimal ones in the majority of cases. For example, the effectiveness on the Number Partitioning problem goes up from 33% to 96% while for Minimum Vertex Cover goes from 17% to 58%.
The results also confirm that QA is quite effective for problems that do not require penalties to model constraints (Max-Cut, Community Detection and Number Partitioning). On the other hand, QA is still ineffective for the Feature Selection problem. Quadratic Knapsack remains very challenging for all the solvers.






 §.§ Meta-Models and Feature Importance Analysis







The goal of our analysis is to identify the domains and the component sets whose features allow the meta-models to predict well the effectiveness of QA.
We limit our analysis to the meta-models trained to predict whether QA will be at least as effective compared to all the classical solvers combined (target label QA-over-all) for the large instances. We also analyze the meta-models predicting whether QA will find the global optimum on the small instances (QA-Optimal). The full results are available in the online appendix. 

The first important question is whether it is possible to train meta-models able to predict the effectiveness of QA. The results in terms of Balanced Accuracy are shown in Figure <ref> (target QA-over-all) and in Figure <ref> (target QA-Optimal) for the two most effective classifiers.
We can immediately see that for several domains or components of the large instances the Balanced Accuracy is approximately 85%, while for the small instances it is often exceeding 90 or even 95%. This, combined with the fact that we selected a heterogeneous set of problem classes and instances that are solved by QA with different degrees of success, allows us to conclude that it is indeed possible to build accurate meta-models to predict the effectiveness of QA. These meta-models can then be used for many purposes, among which, studying the behaviour of this technology.

We now analyse which are the domains or components that produce the best meta-models. Concerning the domains, the more informative ones are those related to the graph structure of an Ising problem (LogIsing and EmbIsing) which, based on the high accuracy of the meta-model, we conclude are very informative on the effectiveness of QA. 
Among the domains, the distribution of the values in the Q matrix of the QUBO problem (MatStruct) is less informative, this can be explained by the fact that the problem that is actually solved on the quantum device is represented as Ising and not as QUBO.

Secondly, if we consider the domains related to the distribution of the energies of an Ising problem which are only available for the small instances (SolSpace, NorMul, 25%-SolSpace, 25%-NorMul in Figure <ref>) it is possible to build meta-models which, again, predict well the effectiveness of QA with a Balanced Accuracy well above 90%. 
This result shows that the effectiveness of QA also depends on the distribution of the energies of the problem, on how the cost y of the QUBO problem is distributed. Notably, using features based on the solution space allows to achieve comparable Balanced Accuracy with other domains, indicating that both are equally very informative. This is a particularly good result because it is relatively easy to compute the features for the other domains once the problem has been formulated as QUBO.

If we look at the orthogonal grouping of the features, by component sets, we notice that with the Bias, Coupling the Laplacian component sets it is possible to train at least a meta-model with good Balanced Accuracy (higher than 80%). On the other hand, the Structural Adjacency (StructAdj) and Structural Laplacian (StructLap) are the least informative and, in particular for large instances, do not allow to build a meta-model better than random guess. Since both StructAdj and StructLap are computed on the binarized problem structure, they only account for how the problem variables are connected and not the coefficient values, this means that the structure of the problem alone is not informative at all.
The bias and the coupling of an Ising problem, together with the Laplacian matrix related to the graph of the Ising problem, are the most informative on the effectiveness of QA.

In general, we can confirm that the characteristics of the problem are important to determine the effectiveness of QA but one must account for the actual coefficient values of the problem and, preferably, use features derived from the Ising formulation. This could open new research questions on whether one can change the formulation of a problem so that its coefficients have a different distribution that is more adequate for the QA. Furthermore, since the coefficients are a function of the problem class and the data, it may be possible to identify which types of graph topologies may be more or less difficult to tackle based on the distributions of the coefficients that they would produce.

We now move to analyzing which specific features are the most important among the ones we identified.
We limit our analysis on feature importance to the XGBoost and AdaBoost meta-models trained on two domains (LogIsing and EmbIsing) and on two component sets (Bias and Coupling).
In the case of meta-models related to small-instances, we include in the analysis also the domain SolSpace.
The best five features of each of these domains and component sets are listed in Table <ref> (target QA-over-all, large instances) and in Table <ref> (target Optimal, small instances).








  
Domains Feature Importance
We consider in particular the domains LogIsing and EmbIsing.
The majority of the most important features are related to the bias and to the coupling of the problem, which is consistent with our previous analysis. Some features are related to the distribution of the values of the bias and the coupling (Gini index, Shannon entropy, Herfindahl-Hirschman index), while other features are related to precise values of these mathematical objects (minimum value, maximum value).

In particular, notice that  (related to the distribution of the bias),  (related to the values of the bias) and  (related to the eigenvalues of the coupling) are among the best features in the majority of the meta-models. We deduce that the distribution of the values and the values themselves of the bias are important to study the effectiveness of QA, together with the eigenvalues of the coupling. This is again an interesting observation because it would allow us to identify in advance whether a problem could be well-suited for QA.

Notice also that the number of qubits needed to embed the problem on the Quantum Annealer () is important, for one meta-model, to predict the effectiveness of QA for the large instances, but not for the small instances. 
This difference may be linked to the fact that, for the small instances, the number of qubits required after the minor-embedding process is limited and therefore has a lower impact.

We analyze, for these two domains, the least important features too. The majority of them is related to the structural adjacency and to the structural Laplacian matrix. This confirms that the sole structure of a problem is not sufficient to determine the effectiveness of QA. Thus, we must consider also the coefficient values between the variables. 

For what concerns the SolSpace domain (see small instances in Table <ref>), notice that both the meta-models have the same top three features, although in different order: such features are mostly related to the distribution of the eigenvalues of the problem ( and ), which plays therefore a role in determining the effectiveness of QA.



  
Component Sets Feature Importance
If we consider the Bias component set, the majority of the most important features are related to the distribution of the values of the bias, both considered in the LogIsing domain and in the EmbIsing domain.
In particular, observe that the Gini index and the condition number of the Bias, computed in both domains, are among the five most important features, as they were for the meta-models trained on the domains LogIsing and EmbIsing.
This corroborate our statement that the distribution and the values of the bias are important to determine the effectiveness of QA.

If we instead consider the Coupling component set, we observe that most of the important features we identify are related to the values and to the eigenvalues of the coupling. Features computed in both the LogIsing and EmbIsing domains are important and the most important features are related to the values of the eigenvalues of the coupling.
Notice that the spectral gap and the Gini index of the coupling, computed both in the LogIsing and EmbIsing domains, are shared with almost all meta-models as some of the most important features. We conclude that the eigenvalues of the coupling and their distribution are important to analyze the effectiveness of QA. 



  
Feature Correlation with Target Label
In the previous analysis we identified the features that are the most important for the meta-models.
Some of these features, furthermore, are important also if computed in different domains and for different meta-models. These features are, for the LogIsing and EmbIsing domains: , , ; while for the SolSpace domain:  , , and . 
We want now to give an intuition of which values of these features determine a low or an high effectiveness of QA.
For each feature we identified, we compute the Spearman rank with the targets of the meta-models (QA-over-all and QA-Optimal).

For most of these features, the Spearman rank does not highlight a strong correlation with the value of the target (in general, the Spearman rank in absolute value is below 0.40). This suggests that the complexity of the underlying behaviour might require more powerful tools.
Two exceptions are given by  and  in domain SolSpace, which have respectively a Spearman rank of -0.596 and 0.537.
This mean that as the Gini index of the eigenvalues increases, the Ising problem becomes less difficult to solve.
On the other hand, high values of  of the eigenvalues implies that the Ising problem is difficult to solve with QA.



§ CONCLUSIONS


In this paper, we have studied the effectiveness of QA with an empirical approach based on meta-learning models.

First, we select a pool of ten optimization problems which can be formulated as QUBO. Then, we generated more than five thousand instances, based on different problem sizes and structures. In particular, we created two sets, one containing large instances and another with small ones for which we can study also the properties of the whole solution space.

As a second step, we define a set of more than a hundred features to describe each problem instance. 
The features are heterogeneous, based on graph theory or on metrics largely used in statistics, probability theory and economics and account for the structure of the problem, its coefficients and its solution space. 
We gather all the features into a meta-learning datasets, which we share on GitHub for further research.

Third, we compare the effectiveness of QA and three classical solvers: Simulated Annealing (SA), Tabu Search (TS) and Steepest Descent (SD).
We observe that QA is frequently less effective than the classical solvers, for both the large and small instances, except for specific problems. 
In particular, we have observed that QA solves more effectively problems with no constraints in their formulation (Max-Cut, Number Partitioning and Community Detection).

Lastly, we train different classification algorithms to predict whether QA will solve an instance effectively or not and show that it is possible to do so accurately. We then use the meta-models to probe the behaviour of QA.
In particular, by analyzing the feature importance of the meta-models, we can observe how the distribution of the bias and the coupling of a problem play a key-role in determining whether QA will be effective in solving it. 

In conclusion, we successfully applied an empirical analysis of the effectiveness of QA based on meta-learning. 
Possible future directions include the analysis on how different distributions of the coupling and bias values relate to the effectiveness of Quantum Annealing. Such results could be correlated to specific kinds of problems. For example, problems characterized by graphs with a power-law distribution (e.g. problems involving social networks) may be more or less difficult to tackle than those characterized by regular graphs. This can be done, for example, by defining new features which describe how much the distribution of the bias and the coupling differs from another distribution, e. g. from a Gaussian or a uniform distribution.
Thanks to its generality, the methodology can be easily extended to other heuristic solvers of QUBO problems, such as the Variational Quantum Eigensolver (VQE) <cit.> and Quantum Approximate Optimization Algorithm (QAOA) <cit.>, providing a useful tool to further our understanding of how to use these quantum algorithms effectively.



Supplementary information
The meta-learning dataset with all the problem instances, the corresponding graphs and features, as well the samples obtained with each solver can be accessed here: <https://github.com/qcpolimi/QA-MetaLearning>.



§ DECLARATIONS


This version of the article has been accepted for publication, after peer review but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: <https://doi.org/10.1007/s42484-024-00179-8>.

Funding We acknowledge the financial support from ICSC - “National Research Centre in High Performance Computing, Big Data and Quantum Computing”, funded by European Union – NextGenerationEU. We acknowledge the CINECA award under the ISCRA initiative, for the availability of high-performance computing resources and support. We also acknowledge the support and computational resources provided by  E4 Computer Engineering S.p.A.

Competing interests The authors declare no competing interests.

Data availability The instances, the dataset with the features, the results of the solvers and an example script to train meta-models are available at this GitHub repository: <https://github.com/qcpolimi/QA-MetaLearning>.

Author contribution All authors contributed to the study conception and design.
M. Ferrari Dacrema conceived the methodology.
Material preparation, data collection and analysis were performed by R. Pellini.
The first draft of the manuscript was written by R. Pellini and all authors contributed to the final version. All authors read and approved the final manuscript.





§ QUBO FORMULATIONS OF OPTIMIZATION PROBLEMS


In this Appendix, we show the QUBO formulations of the ten optimization problems we selected for our study.
We divide the problems in two different groups: problems defined over a graph, explained in Section <ref>, and problems not defined over a graph, explained in Section <ref>.



 §.§ Graph Problems





  §.§.§ Max-Cut

Given the graph G = (V, E), a cut over graph G induced by the set of vertices S and V - S is the set of edges which connect vertices in S with vertices in V-S.
The Max-Cut problem aims at finding the largest cut which can be induced on graph G <cit.>.
Consider, for each  vertex i of graph G, a binary variable x_i, such that

    x_i = 
            1  if  i ∈ S 
    
            0  otherwise

Then, the Max-Cut problem is formulated as written in Equation <ref>

    min_x y = -∑_(i,j) ∈ E(x_i + x_j - 2x_i x_j)




  §.§.§ Maximum Independent Set


Given the graph G = (V, E), an independent set is a set S of vertices which are not adjacent to each other.
The Maximum Independent Set problem aims at finding the largest independent set in G.
Consider, for each vertex i of graph G, a binary variable x_i, such that

    x_i = 
            1  if  i ∈ S 
    
            0  otherwise

Then, the Maximum Independent Set problem can be formulated as a QUBO problem as it is written in Equation <ref> <cit.>.

    min_x y = -a∑_i ∈ Vx_i + b∑_(i,j) ∈ Ex_i x_j

The penalty term b must outweigh the term a, to penalize the choice of having two connected vertices in S.



  §.§.§ Minimum Vertex Cover

Given the graph G = (V, E), a Vertex Cover C is a set of vertices such that all the edges E are connected at least to a vertex in C.
The Minimum Vertex Cover is the Vertex Cover with the smallest number of vertices <cit.>.
Consider, for each  vertex i of graph G, a binary variable x_i, such that

    x_i = 
            1  if  i ∈ C 
    
            0  otherwise

The Minimum Vertex Cover can be formulated as a QUBO problem as written in Equation <ref>.

    min_x y =  ∑_i ∈ V x_i + p ·∑_(i,j) ∈ E(1 - x_i - x_j + x_i x_j)

As for the Maximum Independent Set problem, also in the Minimum Vertex Cover problem the penalty term p must be chosen large enough, to penalize the selection of a set of vertices which is not a vertex cover of G.



  §.§.§ Max-Clique

Given the graph G = (V, E), a clique C is a sub-graph of G such that C is fully connected.
The Max-Clique problem aims at finding the clique C of G with the highest number of nodes.
The QUBO formulation of the Max-Clique problem has the same QUBO formulation of the Maximum Independent Set, but it leverages the complement graph G of G <cit.>.
Please refer to the Section <ref>.



  §.§.§ Community Detection

Given the graph G = (V, E), the aim of the Community Detection problem is to partition G in two communities C_1, C_2 of similar nodes <cit.>.
Remember that the graph G is represented by the adjacency matrix A, where the  (i,j) entry is A_ij = 1 if nodes i and j are connected by an edge, otherwise it is A_ij.
The degree vector d keeps track of how edges are incident to every node of the graph.
The similarity between nodes is expressed by the modularity matrix B, which is computed according to the following formula:
    
    B = A - dd^T/2|E|

    The binary variable x_i is related to the node i ∈ V of graph G and it is defined as follows:
    
    x_i = 
                1     if  i ∈ C_1 
    
                0     if  i ∈ C_2

    The matrix Q is proportional to B. In particular, it is equal to:
    
    Q = -1/|V|B

    The negative sign is due to the fact that QUBO problems require to be minimization problems in order to be solved with a QA.
    The formulation of the problem is therefore given by:
    
    min_x y = -1/|V| x^T Bx


    


 §.§ No-Graph Problems





  §.§.§ Number Partitioning

Given a set of real numbers Z = {z_1, z_2, ..., z_n }, 
the Number Partition problem aims at finding two partitions Z_1, Z_2 of Z in order to minimize the following expression <cit.>:
    
    (∑_z' ∈ Z_1z' - ∑_z”∈ Z_2 z”)^2


The binary decision variable x_i is defined as follows:

    x_i = 
                1     if  z_i ∈ Z_1 
    
                0     if  z_i ∈ Z_2


The QUBO formulation of the Number Partitioning problem is then derived from the expression <ref>, and it can be written as follows:

    min_x y = ( ∑_i=1^|Z|x_i z_i - ∑_i=1^|Z|(1-x_i) z_i)^2

    


  §.§.§ Set Packing

Given a collection of n sets S_1, S_2, ..., S_n, each one with a given capacity c_1, c_2, ..., c_n, the Set Packing problem aims at finding a selection of sets providing the largest total capacity, while respecting m constraints for the selection of the sets.
The binary decision variable x_i is associated to the set S_i, in particular:

    x_i = 
                1     if  S_i  is selected 
    
                0     otherwise

The Set Packing problem can be formulated as it follows:

    max_x y =    ∑c_i x_i
    s.t.   ∑a_ji x_i ≤ 1,    j = 1,2,...,m,    a_ji∈{0,1}


In order to map the constraints of the Set Packing problem into a quadratic penalty term, we leverage the following conversion rule <cit.>:
    
    ∑_i=1^na_ji x_i≤ 1 → p ·∑_i=1∑_k>ia_ji a_jk x_i x_k

    where coefficients a_ji and a_jk are either 0 or 1.
    The objective function is therefore derived as
    
    min_x y = -∑c_i x_i + p ·∑_i=1^n∑_k>i^nx_i x_k ∑_j=1^m a_ji a_jk

    


  §.§.§ Quadratic Knapsack

Given a set of projects P_1, P_2, ..., P_n such that for each pair (P_i, P_j) it exists a joint revenue r_ij = r_ji, the Quadratic Knapsack problem aims at maximizing the total revenue of activating the projects under a budget constraint. 
The Quadratic Knapsack problem is formulated as it follows:
    
    max_x y =  ∑_i^n ∑_j^nr_ij x_i x_j
    s.t.∑_i^nc_i x_i ≤ b


    To formulate the Quadratic Knapsack problem in the QUBO formulation, we introduce m binary slack variables t_1, t_2, ..., t_m.
    Each slack variable has a budget coefficient c_t_j, which has to be chosen accordingly to the budget b, e. g. by stating that ∑_j^m c_t_j = b.
    With these type of contraints, we rewrite the problem using the formulation given in Equation <ref>, defined in Section <ref>.
    The QUBO formulation of the Quadratic Knapsack problem is therefore:
    
    min_x y = - ∑_i^n ∑_j^nr_ij x_i x_j + p ·(∑_i^nc_i x_i  + ∑_j^mc_t_j t_j  - b )^2

    


  §.§.§ Sudoku

A Sudoku can be considered as a constraint satisfaction problem <cit.>.
In our study, we analyze only small instances of Sudoku problems, so we consider Sudoku with a 4 × 4 grid and some fixed assignments. We call such problems 4 × 4-Sudoku
The binary variable we use is defined as
    
    x_(i,j),k = 
                1   if cell  (i,j)  has value  k 
    
                0  otherwise

There are four kinds of constraints in a Sudoku:

    
   Cell Constraints. 
 A cell can contain only one number;
    
    ∑_k = 1^4x_(i,j),k = 1   ∀ i,j ∈{1,2,3,4}

    
   Row Constraints. 
 Two cells in the same row must have distinct numbers;
    
    ∑_j = 1^4x_(i,j),k = 1   ∀ i,k ∈{1,2,3,4}

    
   Column Constraints. 
 Two cells in the same column must have distinct numbers;
    
    ∑_i = 1^4x_(i,j),k = 1   ∀ j,k ∈{1,2,3,4}

    
   Block Constraints. 
 Two cells in the same block must have distinct numbers;
    
    ∑_(i,j) ∈ Bx_(i,j),k = 1      ∀ k ∈{1,2,3,4}, 
       ∀ blocks  B

    There are totally 64 constraints in a 4 × 4-Sudoku problem.
    If some cells are already assigned, the number of constraints and the number of variables reduces.
    In particular:

    
  * If cell (i,j) is assigned, there exist no variable related to this cell;
    
  * If cell (i,j) is not assigned and the number of values it can have is m, then there exist m variables related to cell (i,j) and each variable refer to a possible value of the cell;


    We can map all the constraints into a single square matrix A, which has at most 64 rows and 64 columns.
    Each column is related to a variable x_(i,j), k, while each row refers to a constraint.
    Assume that x is a vector where each element is a variable x_(i,j), k.
    If 1⃗ is a vector of 64 elements, all equal to 1, the QUBO formulation of the 4 × 4-Sudoku problem is:
    
    min_x y = (Ax - 1⃗)^T(Ax - 1⃗)





  §.§.§ Feature Selection


Given a dataset 𝒟, the Feature Selection problem aims at finding the subset S ⊂ F = {f_1, f_2, ..., f_n}  of the best k features able to represent the dataset. This is done especially in machine learning, to reduce the number of features of a predictive model and therefore its complexity.
Assume to have n features. For each feature f_i, we have the binary decision variable x_i, defined as follows:

    x_i = 
            1     if  f_i  is selected
    
            0     otherwise

We model the Feature Selection problem using Pearson correlation Corr(·, ·) <cit.>. 
For the quadratic terms, we compute the correlation between two different features f_i, f_j of 𝒟.
For the linear terms, we compute the correlation between a feature f_i and the target t of 𝒟.
We can compute directly the QUBO matrix Q, where in particular its (i,j) element of the matrix Q is equal to:
    
    Q_ij = 
            
                Corr(f_i, f_j)     if  i ≠ j 
    
                - Corr(f_i, t)     otherwise

Suppose we want to select k features. To do so, we consider the following QUBO formulation of the Feature Selection problem:
    
    min_x y = x^TQx + (∑_i=1^nx_i - k)^2

Notice that, if less or more than k features are selected, the penalty term is larger than 0 and the objective function value gets worse.



§ INSTANCE GENERATION STRATEGIES


In this section, we explain the strategies we applied to generate the instances of the optimization problems we selected.
The strategies varies whether the problem is defined over a graph or not.
An instance of a problem is characterized by its structure, which is either the topology of a graph or a general title which refers to the strategy used to generate the constraints and the objective function, and the number of variables n.

In general, for all the problems we generate instances with a minimum of n_min variables to a maximum of n_max variables.
For every optimization problem, except for the 4 × 4-Sudoku and for the Feature Selection problem, there exist n_rep instances having a certain structure and with the same number of variables.
Such instances are however different between each other, thanks to graph tweaking (explained in Section <ref>) and to the random generation of the coefficient of the cost function and of the constraints.

For the small instances, we have chosen n_min = 27, n_max = 32 and n_rep = 1.
For the large instances, we have chosen n_min = 69, n_max = 99 and n_rep = 5.



 §.§ Graph Problems


The instance of a problem defined over a graph is totally determined by the topology of the graph.
The strategy we applied to generate instances of graph problems is essentially divided in two distinct phases: the first is the generation of the graph G of n nodes, according to a certain topology; the second is the generation of an instance, for every graph problem, defined over G.

We have selected four different graph topologies to generate the graph G: the Star topology, the Cycle topology, the 2d-grid topology and the Erdös-Rényi topology.
A key step in the generation of G is the random insertion and removal of a limited number of edges, resulting in the tweaking of graph G. In this way, we generate more graphs starting from a given topology and with n nodes, which however get tweaked in different ways. 
We ensure always that the tweaked graph is connected.
In particular, the graph tweaking process happens according to the following rules:

    
  * At most n_ins = ⌊n/6⌋ edges are inserted;
    
  * At most n_rem = ⌊n/8⌋ edges are removed;
    
  * To perform the tweaking, we randomly modify the adjacency matrix A of the original graph.
    We start the tweaking of A from the first element of the first row, A_11, and we proceed left-to-right, till the element in the last column and in the last row, A_nn.
    
  * If A_ij=0 and less than n_ins edges have been inserted, insert the edge (i,j) by setting A_ij = 1 with probability p_ins = 40%;
    
  * On the contrary, if A_ij=1 and less than n_rem edges have been removed, remove the edge (i,j) by setting A_ij = 0 with probability p_rem = 30%;
    
  * If the tweaked graph is not connected, repeat the procedure;


For the Minimum Vertex Cover and the Maximum Independent Set problems we have to set the value of the penalty term coefficient p.
For the Minimum Vertex Cover, we have set p = n, that is to the number of nodes of the graph G.
For the Maximum Independent Set, we have set p = 2n. 
In both cases, we have that an assignment which violates a constraint highly penalize the objective function and that it is not a solution of the instance.



 §.§ No-Graph Problems



For what concerns the problems not defined over a graph, to generate an instance we have to generate the matrices related to the objective function and to the constraints. In particular, we want to analyze satisfiable instances.
We discuss now the strategies we used to generate the instances of each problem we selected.



  §.§.§ Number Partitioning

To generate the instances of the Number Partitioning problem, we have to define the set Z to partition.
For simplicity, we generate only sets of integer numbers.
We selected three probability distribution to generate the set Z, in addition to a fourth strategy where Z contains all the numbers between 1 and the number of variables n.
The three probability distributions are: (i) an uniform distribution between 1 and 99; (ii) a geometric distribution with probability of success p = 0.02; (iii) a Poisson distribution of mean value μ = 50.



  §.§.§ Set Packing

An instance of the Set Packing problem is determined by the capacities of the n sets c_1, c_2, ..., c_n, the coefficients a_j,i of the constraints, and the coefficient of the penalty term p.

For what concerns the capacities c_i, we sample highest possible capacity coefficient, called c_max, from a uniform distribution between 10 and 39.
This number represents the highest possible capacity coefficient, called c_max.
Then, we generate the capacities c_1, c_2, ..., c_n of the sets by sampling them from to a uniform distribution between 1 and c_max.

To generate the coefficients a_j,i, we consider a matrix A where the element in the j-th row and in the i-th columns is equal to a_j,i. Notice that A has always n columns.
We have defined four strategies to generate matrix A:

    
  * A is rectangular, with n-1 rows and the elements a_i,i and a_i,i+1 are equal to 1, for i = 1,2,...,n-1. All the other elements are set to zero. We say that this matrix has a step structure.
    We provide below, as an example, a matrix of this kind with n = 5:
    
    A = [ 1 1 0 0 0; 0 1 1 0 0; 0 0 1 1 0; 0 0 0 1 1;   ]


    
  * A is rectangular, with a number of rows m randomly sampled from a uniform distribution between 2 and ⌊n/2⌋. All the columns of A have exactly one element equal to 1. We say that this matrix has disjoint rows structure.
    Below, we provide an example of a disjoint rows matrix, with n = 5 and m = 3:
    
    A = [ 1 0 0 1 0; 0 0 1 0 1; 0 1 0 0 0;   ]


    
  * A is a square matrix. The elements on the main diagonal are all equal to 1. Furthermore, for each row of A, a random off-diagonal element is set to 1 with probability p=60%. We say that this matrix has an almost diagonal structure;

    
  * A is rectangular, with a number of rows m sampled from a uniform distribution between 10 and ⌊n^2/2⌋.
    The elements of A are randomly generated between 0 and 1, with equal probability. We say that this matrix has a random structure;
    


Also the penalty term p must be chosen, in order to penalize the assignments which violate the constraints.
We use 100 steps Bayesian optimization to choose p, by maximizing the percentage of feasible solutions found by Simulated Annealing in 30 executions.
The range of values of p is determined by the sum c_all = ∑_i^n |c_i|. In particular:

    ⌊c_all/3⌋≤ p ≤ c_all




  §.§.§ Quadratic Knapsack

In the Quadratic Knapsack problem, to generate the instances we have to choose: (i) the values of the joint revenue r_ij; (ii) the values of the coefficients c_1, c_2, ..., c_n of the constraint; (iii) the budget b; (iv) the number m of slack variables t_i to introduce and their constraints coefficients c_t_i; (v) the value of the penalty term coefficient p.

We generate the joint revenue values by generating a matrix.
The general joint revenue value r_ij is the element in the i-th row and in the j-th column of a revenue matrix R. By definition of the Quadratic Knapsack problem, R is symmetric.
We generate R according to the following four strategies:

    
  * R is diagonal. The elements on the diagonal are sampled from a uniform distribution of integer numbers between 15 and 39. We say that the structure of R is diagonal;

    
  * R has the whole diagonal, with some other few random off-diagonal elements.
    The elements on the diagonal of R are sampled from a uniform distribution of integers numbers between 15 and 39. 
    Then, for every column of R, we set with probability 40% a random off-diagonal r_ij with an integer number sampled from a uniform distribution between 1 and 24; when the element is set, we make the matrix symmetric by dividing r_ij by 2 and setting r_ji = r_ij.
    In the case r_ji element is set again, when considering column j, the previous value of r_ji and r_ij is overwritten. We say that R has an almost diagonal structure.
    An example of almost diagonal 4 × 4 matrix is:
    
    [l]
            17.0     13.5     0.0     16.0 
    
            13.5     32.0     0.0     0.0 
    
            0.0     0.0     19.0     0.0 
    
            16.0     0.0    0.0     26.0


    
  * The on-diagonal elements of R are non-zero; also the elements immediately on the right and on the left of on-diagonal elements are non-zero; the on-diagonal elements are integer numbers sampled from a uniform distribution between 15 and 39; the off-diagonal elements immediately to the right of on-diagonal elements are sampled from the same distribution; then, the off-diagonal elements are divided by 2 and the matrix is made symmetric, by setting r_ij = r_ji, for every i, j ∈{1,2,..., n}. We say that R has an enlarged diagonal structure.
    An example of a 4 × 4 enlarged diagonal is:
    
    [l]
                15.0     11.0     0.0     0.0 
    
                11.0     25.0     9.5     0.0 
    
                0.0     9.5     31.0     17.0 
    
                0.0     0.0     17.0     37.0


    
  * R is generated randomly, every element is sampled from the uniform distribution of integer numbers between 15 and 39; then, all the elements below the diagonal are ignored and set to 0; finally, the off-diagonal elements are divided by 2 and the matrix is made symmetric, by setting r_ij = r_ji, for every i, j ∈{1,2,..., n}. We say that R has a random structure.


The coefficients c_1, c_2, ..., c_n which appear in the constraint are integer numbers sampled from a uniform distribution between 1 and 15.
The budget b is set as ϕ * ∑_i=1^n c_i, where ϕ∈ℝ is sampled from a uniform distribution between 0.20 and 0.70.

We have chosen to introduce four binary slack variables t_1, t_2, t_3, t_4 to transform the inequality constraints into equality constraints. The slack variables are respectively associated to the budget coefficients c_t_1, c_t_2, c_t_3, c_t_4. We choose also the values of these coefficients as it follows:
    
    c_t_1 = ⌊b/2⌋   
            c_t_2 = ⌊b/4⌋   c_t_3 = ⌊b/8⌋   c_t_4 = b - ∑_i=1^3 c_t_i


We use 100 steps Bayesian Search to choose p, by maximizing the percentage of feasible solutions found by Simulated Annealing in 30 executions.
The range of the values of p depends on the sum r_all = ∑_i=1^n∑_j=1^n|r_ij|.
In particular:

    1 ≤ p ≤ r_all




  §.§.§ Sudoku

To generate 4 × 4-Sudoku instances, we generated randomly 30 different solved 4 × 4-Sudoku games.
We then iteratively removed the value of a cell, chosen randomly, of the 4 × 4-Sudoku game.
Each time we remove a cell, we increase the number of variables of the related 4 × 4-Sudoku instance.
We ensure that the generated instance has a number of variables between 27 and 32.

Notice that it is not possible to build instances larger than 64 variables of the 4 × 4-Sudoku problem. For this reason, we generate only small instances for this kind of problem.



  §.§.§ Feature Selection

We generated the instances of the Feature Selection problem starting from the following public datasets[Datasets are available on the website OpenML: <https://www.openml.org>]: , , , , , , .
An instance of the Feature Selection problem depends on the dataset 𝒟, on the number of features to select k and on the target variable t.

Assume we generate m instances with a number of variables n, with n bounded between n_min and n_max.
To select the dataset D, we use an iterative procedure.
We start from the dataset  and we check if it has more than n = n_min non-target features: if this condition is true, we select this dataset and we reduce its number of features to n, by deleting randomly chosen features; otherwise, we go to the the next dataset, chosen according to the order we used to list them, and we repeat this check. After the last dataset, , we repeat staring from .
After that m features are selected, we continue the procedure by incrementing n.
The procedure goes on until we generate m instances with n = n_max variables. 

Assume that the dataset 𝒟 is the i-th dataset selected to generate an instance of n variables, in the iterative procedure, the number of features to select is computed as follows:

    k = ⌊n/5⌋ + i


For what concerns the target variable t, every dataset we selected has one or multiple target variables. 
Since this formulation can tackle only one target variable t, in case of multiple target variables we randomly select one of them and delete all the others.

For small instances, we considered n_min = 27, n_max = 32 and m = 4, for a total of 24 instances. The number of features k we select is bounded between 5 and 9.
For large instances, we considered n_min = 69, n_max = 99 and m = 5, for a total of 155 instances. The number of features we select is bounded between 13 and 23.



§ DEFINITION OF THE FEATURES


We introduce in this section the definitions of the features we used in our study.
A subset of these features is based on probability theory and statistics, another subset is based on graph theory, and other features are related to the study of the spectrum of matrices.



  
Gini index
Given a set X of positive numbers, the Gini index Gini(Y) is a real number which measures the degree of inequality between the values y ∈ Y <cit.>. It is comprised between 0 (all the values y ∈ Y are equal) and 1 (only one value of Y is different from 0).
Given an ordered collection of values Y = {y_1, y_2, ⋯, y_m}, such that 0 ≤ y_1 ≤ y_2 ≤⋯≤ y_m, the Gini index is computed as:
    
    g(Y) = 2∑_i=1^miy_i/m∑_i=1^my_i - n+1/n




  
Herdindahl-Hirschman index
Given a set Φ, called industry, composed of m firms F_1, F_2, ..., F_m, each one characterized by a market share S(F_i) ∈ [0,1], such that ∑_i=1^m F_i = 1.
The Herfindahl-Hirschman index (HHI) is a measure used in economics to quantify the competitiveness of an industry with respect to the market share of the firms which compose the industry <cit.>.
The HHI of inustry Φ is defined as:

    HHI(Φ) = ∑_F ∈ΦS(F)^2

Clearly, 0 < HHI(Φ) ≤ 1. In particular, HHI(Φ) = 1 when there exists only one firm inside industry Φ, that is Φ is a monopoly.
On the contrary, if we assume that all the firms have equal market share, the market is competitive and, if m → +∞, we have that HHI(Φ) → 0. 

If two firms F_i, F_j merge into a larger new firm F_z, we have that S(F_z) = S(F_i) + S(F_j). The HHI computed on the new industry, which include F_z, increases.



  
Shannon entropy
Given a random variable Y, distributed according to a certain distribution p_Y(y).
The Shannon entropy of Y measures the level of uncertainty in the distribution p_Y(y).
In the discrete and finite case, assuming that the possible values of Y are y_1, y_2, ..., y_m, Shannon entropy is defined as:

    Sh(Y) = -∑_i=1^m p_Y(y_i)log_2 p_Y(y_i)

In general, Shannon entropy can be greater than 1.
The maximum value of Shannon entropy occurs when Y is distributed according to a uniform distribution, where all the values of Y are equally probable and none of them can be predicted more easily than the others.



  
Condition Number
Given a matrix M, the condition number CN(M) measures how close is matrix M to be singular.
If M is symmetric and real, the CN(M) is computed according to the maximal and minimal eigenvalues of M, respectively λ_max and λ_min:

    CN(M) =  | λ_max/λ_min |




  
Radius of a Graph
Given a graph G(V, E), described by the adjacency matrix A.
The radius of graph G is the largest eigenvalue of A in absolute value.



  
Diameter of a Graph
Given a graph G(V, E), the shortest path between two nodes i, j is the sequence of edges having minimal cost which connects i and j.
The diameter of G is the length of the longest shortest path.
If G is not connected, the diameter is not defined.
In the case of negative weights on the edges that form a cycle, the diameter is not defined too.



  
Spectral Gap of a Graph
Given a graph G(V, E), described by the adjacency matrix A, the spectral gap of G the difference, in modulus, between the the two largest eigenvalues of A.
Another definition we use is related to the Laplacian L of the graph. In this case, the spectral gap is the smallest non-zero eigenvalue of the Laplacian L related to the graph.



  
Connectivity of Graph
Given a graph G(V, E), described by a positive semi-definite Laplacian matrix L, the connectivity is equal to the second smallest eigenvalue of L.



  
Connected Components of a Graph
Given a graph G(V, E), described by a positive semi-definite Laplacian matrix L, the number of connected components of the graph is equal to the multiplicity of the eigenvalue 0 of L.




§ DOMAINS AND COMPONENT SETS




 §.§ Matrices of an Ising Graph


We call Ising graph a graph having the coupling matrix J of an Ising problem as adjacency matrix. Each node of the graph corresponds to a variable of the Ising problem.
The nodes of an Ising graph has a weight too, determined by the bias vector b of the Ising problem.
In the case the Ising problem is mapped onto the topology of a Quantum Annealer, we call it embedded Ising graph; otherwise, we refer to it as logical Ising graph.
We call structural adjacency matrix A of an Ising graph the adjacency matrix related to the unweighted version of the Ising graph.

The degree matrix D_J of the Ising graph is computed starting from the coupling J, while the structural degree matrix D_A of the Ising graph is computed starting from A.

The Laplacian L of an Ising graph is computed according to the following formula:

    L = D - J

The structural Laplacian L_A is computed using A instead of J and D_A instead of D_J.

The normalized adjacency matrix A_N is computed according the following formula:

    A_N = D_A^-1/2 A D_A^-1/2

The normalized Laplacian L_N is instead equal to:

    L_N = D_A^-1/2 L_A D_A^-1/2




 §.§ Domains and Related Features

We list here all the features computed in every domain we consider.
We refer to the definitions of the features given in Appendix <ref>.



  §.§.§ Embedding and Logical Ising Graph (EmbIsing and LogIsing)

In these domain, we gather features related to the embedded and logical Ising graph.
We compute the features starting from the following mathematical objects: bias, coupling, Laplacian, degree, structural adjacency, structural degree, structural Laplacian, normalized adjacency and normalized Laplacian matrices.
In both the domains, we compute the same features, but the considered mathematical objects vary in size and values.
For both the domains, we use the notation  to refer to the feature , in lowercase letters, computed on the mathematical object .



  
Coupling
Given the coupling J, we compute:

    
  * : Gini index of the eigenvalues of J. Since eigenvalues may be negative, we shift all the eigenvalues by the subtracting to all of them the smallest eigenvalue of J;
    
  * : HHI of the eigenvalues of J. The share of the eigenvalues is equal to their multiplicity divided by n;
    
  * : Shannon entropy of the eigenvalues of J. The probability of an eigenvalue is equal to its multiplicity divided by n;
    
  * : condition number of J;
    
  * : radius of the graph described by adjacency matrix J;
    
  * : diameter of the graph described by the adjacency matrix J. It may be not defined;
    
  * : spectral gap of the graph described by the adjacency matrix J;
    
  * : smallest eigenvalue of J;
    
  * : largest eigenvalue of J.




  
Bias
Given the bias b, we compute:

    
  * : Gini index of the values of b. Since some values of b may be negative, the values are all shifted by subtracting the minimal value of b;
    
  * : HHI of the values of b. The share of the values corresponds to their multiplicity divided by n;
    
  * : Shannon entropy of the values of b. The probability of a value corresponds to its multiplicity divided by n;
    
  * : the minimal value of b;
    
  * : the maximal value of b;
    
  * : the fraction between the maximal and the minimal value of b, in absolute value.




  
Degree
Given the degree matrix D of an Ising graph, we compute:

    
  * : Gini index of the eigenvalues of D. Since eigenvalues may be negative, we shift all the eigenvalues by the subtracting to all of them the smallest eigenvalue of D;
    
  * : HHI of the eigenvalues of D. The share of the eigenvalues is equal to their multiplicity divided by n;
    
  * : Shannon entropy of the eigenvalues of D. The probability of an eigenvalue is equal to their multiplicity divided by n;
    
  * : minimal eigenvalue of D;
    
  * : maximal eigenvalue of D;
     
  * : condition number of D.




  
Laplacian
Given the Laplacian matrix L of an Ising graph, we compute:

    
  * : Gini index of the eigenvalues of L. Since eigenvalues may be negative, we shift all the eigenvalues by the subtracting to all of them the smallest eigenvalue of L;
    
  * : HHI of the eigenvalues of L. The share of the eigenvalues is equal to their multiplicity divided by n;
    
  * : Shannon entropy of the eigenvalues of L. The probability of an eigenvalue is equal to its multiplicity divided by n;
    
  * : minimal eigenvalue of L;
    
  * : maximal eigenvalue of L;
    
  * : the second smallest eigenvalue of L. We compute it also if L is not positive semi-definite;
    
  * : the smallest non-zero eigenvalue of L;
    
  * : multiplicity of eigenvalue 0 of L. We compute it also if L is not positive semi-definite.




  
Structural and Normalized Matrices
The features computed for the structural adjacency, degree and Laplacian matrices and for the normalized adjacency and Laplacian matrices are the same computed for their weighted counterparts. The main difference is that the structural and the normalized matrices are positive-semidefinite and there is no need to shift the eigenvalues to compute the Gini index.



  
Graph Structure in EmbIsing Domain
In the EmbIsing domain, we compute also two features related to the size of the Embedded Ising graph.
In particular, we compute:

    
  * : number of nodes of the Embedded Ising graph, which corresponds to the number of qubits used to solve an Ising problem with the Quantum Annealer;
    
  * : number of chains of nodes in the Embedded Ising graph; a chain represents a single node of the Logical Ising graph, a single variable of the original Ising problem.




  §.§.§ Matrix Structure (MatStruct)

In this domain, we compute the features related to the values of the QUBO matrix Q.
In particular, we compute:

    
  * : Gini index of the values of Q. We subtract the minimal value of Q from all the elements to guarantee that the values are positive;
    
  * : HHI of the values of Q. The share of an element of Q is equal to its number of occurrences inside Q, divided by n^2;
    
  * : Shannon entropy of the values of Q. The probability of an element of Q is equal to its number of occurrences inside Q, divided by n^2.




  §.§.§ Solution Space (SolSpace)

In this section, we compute features related to the eigenvalues of the Hamiltonian of the problem H_p, related to a QUBO problem.
A generic eigenvalue of H_p is λ_i.
Assume that λ_min is the minimal eigenvalue of H_p. Remember that, if the QUBO problem has n variable, H_p has 2^n eigenvalues.
In this domain, we compute:

    
  * : Gini index of the eigenvalues of H_p. To have all positive values, eigenvalues are shifted by subtracting λ_min to all of them;
    
    
  * : HHI of the eigenvalues of H_p. We use the value 
    
    S(λ_i) = λ_i-λ_min/∑_j=1^2^n(λ_j - λ_min)

    as the share of an eigenvalue λ_i;

    
  * : HHI of the eigenvalues of H_p, where identical eigenvalues are considered together. The share of the eigenvalue λ_i is equal to the sum of the shares of the eigenvalues equal to λ_i, computed according to the formula <ref>.
    
    
  * : Shannon entropy of the eigenvalues of H_p. We use as probability of eigenvalue λ_i the value described in Equation <ref>;
    
  * : the minimal eigenvalue of H_p;
    
  * : the eigenvalue of the first quartile of H_p;
    
  * : the median eigenvalue of H_p;
    
  * : the eigenvalue of the third quartile of H_p;
    
  * : the maximal eigenvalue of H_p.





  §.§.§ Normalized Multiplicity (NorMul)

In this section, we compute features related to the multiplicity of the eigenvalues of the Hamiltonian of the problem H_p, related to a QUBO problem. Assume that there are m different eigenvalues of H_p.
The multiplicity of the eigenvalue λ_i of H_p is equal to μ_i. We call normalized multiplicity π_i of λ_i the value:

    π_i = μ_i/∑_j=1^m μ_j

In this domain, we compute the following features:

    
  * : Gini Index of the normalized multiplicities of the eigenvalues of H_p;
    
  * : HHI of eigenvalues of H_p, where we use their normalized multiplicities as shares;
    
  * : Shannon entropy of the eigenvalues of H_p, where we use their normalized multiplicities as probabilities;
    
  * : the normalized multiplicity of the smallest eigenvalue of H_p.




  §.§.§ 25%-SolSpace and 25%-NorMul

The features computed in these domains are the same computed in the domain SolSpace and NorMul.
The only difference is that here, if consider the eigenvalues in ascending order, we consider only the eigenvalues of H_p in the first quartile.



 §.§ Component Sets


Component sets gather all the features computed on the same mathematical objects, but in different domains.
In these study, all the component sets contain features computed in both the LogIsing and EmbIsing domains.
For every component set, we use the notation  to refer to the feature , written in lowercase letters, computed in the domain  on the mathematical object related to the component set.
The component sets we consider are:

    
  * Bias: it contains the features related to the bias;
    
  * Coupling: it contains the features related to the coupling;
    
  * Laplacian: it contains the features related to the Laplacian matrix;
    
  * StructAdj: it contains the features related to the structural adjacency matrix;
    
  * StructLap: it contains the features related to the structural Laplacian matrix;




 §.§ 
Complexity in Features Computation

The major complexity in the computation of the features is given by the computation of the Hamiltonian of the problem, H_p.
To do so it is in fact necessary to compute a 2^n real-valued vector, with n number of QUBO variables, according to Equation <ref>. This of course is related only to the small instances.

For what concerns the features related to the large instances, the bottleneck lies in computing all the minor-embeddings of the instances: minor-embedding is, indeed, an NP-Hard problem.

Once H_p and the minor-embedding of the instances have been computed, all the features are easily computed without any particular complexity
.



§ HYPERPARAMETERS OF THE SOLVERS



  
Tabu Search

We optimize the number of restarts of the algorithm.
The default number of restarts is 100000.
Table <ref> contains the optimal hyperparameters found for each problem class.





  
Simulated Annealing
We optimized the number of  and the  type.
The default value of the number of  is 1000, while the defualt  is .
Notice that we solve the instances also with Simulated Annealing with no hyperparameters optimization and we obtained better results.
Table <ref> contains the optimal hyperparameters found for each problem class.





§ ADDITIONAL RESULTS ON THE EFFECTIVENESS OF SOLVERS










 Paper Title: Deep Learning for Options Trading: An End-To-End Approach.









Deep Learning for Options Trading: An End-To-End Approach]Deep Learning for Options Trading:
 An End-To-End Approach










Corresponding Author
weeling@robots.ox.ac.uk
0000-0003-2664-8238


  Oxford-Man Institute of Quantitative Finance
  University of Oxford
  Oxford
  United Kingdom




sjrob@robots.ox.ac.uk



  Oxford-Man Institute of Quantitative Finance
  University of Oxford
  Oxford
  United Kingdom



stefan.zohren@eng.ox.ac.uk



  Oxford-Man Institute of Quantitative Finance
  University of Oxford
  Oxford
  United Kingdom












§ ABSTRACT 
We introduce a novel approach to options trading strategies using a highly scalable and data-driven machine learning algorithm. In contrast to traditional approaches that often require specifications of underlying market dynamics or assumptions on an option pricing model, our models depart fundamentally from the need for these prerequisites, directly learning non-trivial mappings from market data to optimal trading signals. Backtesting on more than a decade of option contracts for equities listed on the S&P 100, we demonstrate that deep learning models trained according to our end-to-end approach exhibit significant improvements in risk-adjusted performance over existing rules-based trading strategies. We find that incorporating turnover regularization into the models leads to further performance enhancements at prohibitively high levels of transaction costs.











<ccs2012>
 <concept>
  <concept_id>00000000.0000000.0000000</concept_id>
  <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
  <concept_significance>500</concept_significance>
 </concept>
 <concept>
  <concept_id>10003752.10010070.10010071.10010261.10010272</concept_id>
  <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
  <concept_significance>300</concept_significance>
 </concept>
 <concept>
  <concept_id>00000000.00000000.00000000</concept_id>
  <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
  <concept_significance>100</concept_significance>
 </concept>
 <concept>
  <concept_id>00000000.00000000.00000000</concept_id>
  <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
  <concept_significance>100</concept_significance>
 </concept>
</ccs2012>


[500]Do Not Use This Code Generate the Correct Terms for Your Paper
[300]Do Not Use This Code Generate the Correct Terms for Your Paper
Do Not Use This Code Generate the Correct Terms for Your Paper
[100]Do Not Use This Code Generate the Correct Terms for Your Paper




<ccs2012>
   <concept>
       <concept_id>10010147.10010257</concept_id>
       <concept_desc>Computing methodologies Machine learning</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178</concept_id>
       <concept_desc>Computing methodologies Artificial intelligence</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10002950.10003648.10003688.10003693</concept_id>
       <concept_desc>Mathematics of computing Time series analysis</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>





















[
    Stefan Zohren
    February 2024
=================







§ INTRODUCTION



Options form a class of derivatives known as contingent claims, granting one of the counterparties the right to transact an underlying asset at a certain time in the future and at a specific price. For instance, the buyer of a call (put) stock option pays a premium to the seller for the right to buy (sell) the underlying stock by a specified expiration date and strike price. Unlike in cash markets that typically exhibit linear payoffs, the ability to generate synthetic and highly non-linear exposures has made options an increasingly important tool among investors as trading instruments alongside other derivatives <cit.>.

The options market has continued to grow significantly over the last decade. According to the Options Clearing Corporation, the average daily combined volume of US equity and non-equity options has steadily risen and roughly tripled over the past decade from about 16.3 million in 2013 to 44.2 million contracts in 2023 <cit.>. Given the increasing popularity of options trading by market participants conducted in modern electronic exchanges, there exists an unprecedented opportunity to engage in large-scale data analysis of options trading from the perspective of an active investor.

Classical option pricing models and their parametric variants, stemming from the seminal works of Black and Scholes <cit.> and Merton <cit.>, regard options as redundant assets. While these approaches offer a tractable no-arbitrage pricing framework, they necessitate thorough specifications of underlying dynamics and require assumptions such as a frictionless market and the ability of market makers to perfectly hedge their exposures. Practically, these models are often vulnerable to model misspecification and are usually too simplistic to sufficiently account for empirically observed variations in option returns <cit.>. Recent works have challenged this framework, demonstrating that option prices are influenced by additional risks beyond the underlying asset's exposure <cit.>, with several studies documenting the existence of mispricing in the options market <cit.>.

Despite the accelerating growth of the options market and the evidence of options mispricing, there is a surprising lack of research on a scalable machine learning-based strategy that is able to directly leverage the abundance of data to conduct options trading on behalf of an active investor. In this work, we address this gap by introducing a novel class of deep learning models capable of managing and trading a portfolio of options in a highly data-driven approach. 

Previous research has conventionally approached the complexities of managing portfolios of options by developing methods for optimal hedging and accurately pricing options. Our model departs fundamentally from the need for these prerequisites, and are directly optimized to learn highly non-trivial mappings from observed data to optimal trading decisions based on risk-adjusted performance. The resulting end-to-end framework does not depend on specific market dynamics, and can be extended broadly across instruments where market data is available.

To illustrate our approach, we construct the framework by training end-to-end neural networks of different complexities and show that our models are able to outperform existing rules-based strategies in managing a portfolio of options, demonstrating strong risk-adjusted returns over a backtest period of over a decade. We subsequently extend our models to incorporate turnover regularization during optimization, leading to further performance enhancements in the presence of high transaction costs.




§ RELATED WORK


To place our work within the context of existing research on options, we briefly review the broad literature in three areas: replication and hedging, pricing and valuation, and predicting option returns. We subsequently demonstrate how our work contributes to existing systematic trading strategies within options markets.



 §.§ Options Literature


Traditionally, options have been extensively studied due to their significance in both replication and hedging strategies. The works of <cit.> and <cit.> apply reinforcement learning techniques towards approximating policies for optimal hedging in the presence of trading costs. While these works are pertinent to a product or liquidity provider needing to accurately hedge its exposures to books of options and other complex derivatives, our analysis fundamentally differs in its focus on options trading from the standpoint of an active investor seeking to profit from an options trading strategy and whose main objective is not predominantly about neutralizing exposures. Furthermore, hedging techniques which are based on reinforcement learning often require generating or simulating possible market paths to arrive at an optimal trading strategy. In contrast, the solution we offer does not require any assumption or simulation of market processes, and scales with available historical data and compute.

There has also been significant emphasis placed on the need to accurately price or value an option in order to facilitate optimal hedging and trading, with notable contributions from the classical Black-Scholes-Merton option pricing model for European-style options <cit.>. Non-parametric models have also been developed, with <cit.> and <cit.> using neural networks to approximate the market's option pricing function. Conversely, our framework focuses on automatically extracting features and making trading decisions directly from available data, effectively circumventing the need for engineering an option pricing or valuation model.

Several studies have also employed machine learning for predicting option returns, often framing the trading strategy as a standard regression problem and subsequently determining the direction of price movements based on forecasted returns. <cit.> use both linear and nonlinear models and ex-ante option-based and stock-based characteristics to predict monthly returns of delta-hedged options, demonstrating high out-of-sample forecast accuracy. However in the case of a trend-following strategy, <cit.> shows that models that accurately predict returns do not necessarily guarantee a positive or superior strategy performance, given that the overall profitability of trading strategies is influenced by other factors including the distribution of returns, position sizes and the presence of risk adjustments like volatility targeting <cit.>. Taking this into consideration, our work effectively integrates both trend prediction and optimal position sizing simultaneously within a single end-to-end function, eliminating the need to forecast option returns in making trading decisions.



 §.§ Systematic Strategies and Risk Premia in Options Markets


Our work contributes to a series of research documenting systematic strategies originating from studies demonstrating additional risk factors <cit.>, and the existence of mispricing <cit.> within the options market. Considering the broad range of possible risk factors and strategies, we concentrate our analysis on systematic trend-based strategies. Given their simplicity of being based primarily on historical price trends and returns, and unlike volatility-based strategies, trend-based strategies often do not require additional assumptions on any underlying option pricing model or specifications of market dynamics.



 §.§ Momentum, Mean-Reversion and Applications of Machine Learning

Trend-based strategies consists of primarily momentum and mean-reversion strategies.
Momentum strategies operate on the principle that asset returns exhibit a tendency to persist in the same direction and aim to trade in the direction of the trend <cit.>. On the other hand, mean-reversion strategies adopt an opposite and contrarian view, taking on positions that bet on the eventual break down and correction of overextended trends <cit.>. Recently, machine learning models have been increasingly utilized in trend-based trading strategies <cit.>.

While momentum strategies have been extensively documented in a range of asset classes from cash equities <cit.> to futures markets <cit.>, these strategies have received relatively less attention in options markets. <cit.> document strong evidence of the presence of momentum in options markets and propose a series of rules-based momentum strategies. However, these rules-based strategies often require explicit specifications for the trading rule, often with insufficient evidence to justify selecting a certain rule over another. In light of these observations, we employ deep neural networks to automatically learn risk-adjusted trading rules in a data-driven approach, and provide a thorough comparison between these trend-based strategies.




§ OVERVIEW OF DATASET


Our dataset consists of option contracts sourced from the OptionMetrics Ivy DB database, comprising end-of-day bid-ask prices, implied volatility and Greeks of individual options. We independently verify the accuracy of the provided implied volatility and Greeks using a binomial tree model of  <cit.> only for the purpose of initial delta hedging (refer to Section <ref>), but otherwise we do not assume any underlying option pricing model throughout this work. We focus our analysis on option contracts of equities listed on the S&P 100 Index, as these companies span a wide range of sectors, representing major large-cap optionable companies in the US market, and are associated with higher option liquidity. We obtain underlying stock prices from the Center for Research in Security Prices (CRSP), which we use for computing option moneyness and accounting for corporate actions such as stock splits. We perform backtesting with the most recent market data (as of this work) from 2010 to 2023, which notably includes the market selloff following the COVID-19 pandemic. 

We impose a series of data filters established in the literature to ensure the consistency of our analysis. We consider only standard monthly option contracts that expire on the third Friday of the month and exclude any special settlements due to corporate actions. We then exclude options that contain price observations that breach American-style option bounds, having a bid price of zero, or where the ask price is smaller or equal to the bid price, and require options to have a positive open interest on the day of portfolio formation. In addition, we disregard options that contain one or more missing observations between the day of portfolio formation and expiration.

On the expiration day of each month and for each stock, we form portfolios of static delta-neutral straddle options by selecting a pair of call and put contracts with identical strike prices that are closest to at-the-money (ATM) and expire in the following month. Since it is usually not possible to select options with moneyness exactly equal to 1.0 (moneyness of call = S/K, put = K/S where K = strike price, S = stock price), we select the pair of options that are closest to ATM within a moneyness range of 0.95 to 1.05. 

Our resulting universe consists of 29984 options that are traded, with a total of 603068 daily returns observations throughout the backtest period. The returns of straddle options have positive means (1.41% monthly), large standard deviations (90.85% monthly), and significant positive skewness as indicated by a low median (-15.61% monthly). We provide an in-depth explanation on portfolio formation and returns computation in Section <ref>. 




§ SYSTEMATIC OPTIONS TRADING STRATEGIES


Let i = 1, 2, ⋯, N_t denote individual underlying stocks. For a given portfolio of (1-month, ATM, static delta-neutral) straddle options of these stocks that is rebalanced daily, the overall returns of a strategy that equally diversifies over N_t straddles at day t can be expressed as follows:

    r_t, t+1^STRATEGY   = 1/N_t∑_i=1^N_t X_t^(i, straddle) ( σ_tgt/σ_t^(i, straddle) )  r_t, t+1^(i, straddle)

whereby

    r_t, t+1^(i, straddle)   = p_t+1^(i, straddle) - p_t^(i, straddle)/p_t^(i, straddle)
    
        p_t^(i, straddle)   = w_norm^(i, call) p_t^(i, call) + w_norm^(i, put) p_t^(i, put)
    
        w_norm^(i, call)   = w^(i, call)/w^(i, call) + w^(i, put)
    
        w_norm^(i, put)   = 1 - w_norm^(i, call)
    
        w^(i, call)   = - Δ_0^(i, put)
     
        w^(i, put)   = Δ_0^(i, call)

On the day of portfolio formation (t=0), we construct static delta-neutral straddle options by holding the call and put options with weights - Δ_0^(i, put) and Δ_0^(i, call) respectively, with Δ_0 representing initial deltas. We subsequently normalize both weights to sum to one, resulting in a respective weightage of the call and put options that is generally close to 50-50. In evaluating the price of the straddle, we take p_t^(i, call) and p_t^(i, put) to be the bid-ask midpoints of the call and put options.

Consistent with other works <cit.>, we focus on static delta-neutral straddles as these instruments are on average invariant to movements in the underlying. <cit.> demonstrate that performing one-time delta hedging at initiation neutralizes the overall directional risks associated with an option by about 70%. As such, we opt for delta hedging at initiation and following similar works, we disregard the early exercise premium embedded in the American-style options and therefore ignore the possibility of early assignment of short options. 

Here, X_t^(i, straddle)∈ [-1, 1] denotes the trading signal or position for the straddle option of stock i at day t, and r_t, t+1^(i, straddle) is the realized returns of the straddle from t to t+1. We find evidence that straddle options in the cross-section of individual constituents of the S&P 100 exhibit different levels of volatility using a Levene's test at a significance level of 1%. Given these differences, we include volatility targeting at the level of individual straddle options to scale the realized returns r_t, t+1^(i, straddle) by their volatility in order to target equal assignments of risk. We set the annualized volatility target σ_tgt to be 15% and estimate the ex-ante volatility σ_t^(i, straddle) with a 20-day exponentially weighted moving standard deviation of daily straddle returns.

All of the following trend-based benchmarks which we incorporate in our work adhere to this general framework and are concerned with constructing an accurate trading signal X_t^(i, straddle):



  
Long Only (Short Only) This strategy takes a maximum long (short) position X_t^(i, straddle) = 1 or -1 for all straddle options in the portfolio. Since the performance of a short only strategy is the exact opposite of a long only strategy, we focus only on the long only strategy. 



  
TSMOM (TSMR) Following the time-series momentum strategy (TSMOM) of <cit.>, we adopt the strategy with a monthly horizon. The position taken for a straddle option is based on the sign of the option's returns over the past 20 days: X_t^(i, straddle) = sgn(r_t-20, t^(i, straddle)). In the time-series mean reversion strategy (TSMR), we modify TSMOM to take on a negative loading on past returns, where X_t^(i, straddle) = -sgn(r_t-20, t^(i, straddle)). This strategy takes a contrarian approach by taking on a long (short) position for straddles with negative (positive) returns over the past 20 days.


  
MACD (MACDMR) We use volatility normalised moving average convergence divergence (MACD) indicators based on <cit.> in place of the sign of returns for estimating the trading signal: 

    MACD(i, t, S, L) = m(i, t, S) - m(i, t, L) 
       MACD_norm(i, t, S, L) = MACD(i, t, S, L)/std(p_t-5 : t)
       Y^(i, straddle)_t = MACD_norm(i, t, S, L)/std(MACD_norm(i, t-20 : t, S, L))
       X_t^(i, straddle) = 1/3∑_k=1^3 ϕ(Y_t^(i, straddle) (S_k, L_k))

where MACD(i, t, S, L) denotes the MACD value of the straddle option of stock i at day t with a short time scale S and long time scale L. m(i, t, j) is defined as the exponentially weighted moving average of the straddle's prices at day t, with a time scale j corresponding to a half-life of HL = log(0.5) / log(1 - 1/j). We combine MACD signals in an equally weighted sum over multiple short and long time scales S_k∈{2, 4, 8}, L_k∈{8, 16, 32} to construct a position X_t^(i, straddle) where ϕ(y) = yexp(-y^2/4)/0.89. Likewise, we consider both the momentum X_t^(i, straddle) (MACD) and mean reversion -X_t^(i, straddle) (MACDMR) strategies. We refer the reader to <cit.> for further details on the strategy implementation.



  
TSHestonMOM (TSHestonMR) Following <cit.>, we consider a strategy that constructs the trading signal for the straddle option of stock i on the day of portfolio formation (t=0), holding the position unchanged to expiry: 

    Y_0:t^(i, straddle)   = r_0-nM, 0^(i, straddle)
    
        X_0:t^(i, straddle)   = sgn(Y_0:t^(i, straddle))

where r_0-nM, 0^(i, straddle) is the average returns for the series of (1-month, ATM, static delta-neutral) straddle options for stock i over a lookback period of n months from the day of portfolio formation. 
A key distinction between TSMOM and TSHestonMOM is that the momentum signals identified in the latter strategy are more accurately referred to as cross-serial correlations, in that the options used to compute the n-period average returns r_0-nM, 0^(i, straddle) are different from those used to construct the trading signal for the options to be traded following t=0. In other words for the TSHestonMOM, all types of momentum signals involve past returns of a set of options predicting the future returns on an entirely new set. Similar to <cit.>, we consider multiple strategies corresponding to average returns of monthly straddle returns over various lookback periods ranging from n = 1, 3, 6, 12 corresponding to monthly, quarterly, semiannual and annual returns. We again consider both the momentum X_0:t^(i, straddle) (TSHestonMOM) and mean reversion -X_0:t^(i, straddle) (TSHestonMR) strategies. 




  
CSHestonMOM (CSHestonMR) Based on the long-short approach of <cit.> and <cit.>, we implement a cross-sectional momentum strategy that scores and ranks a stock based on its average straddle returns computed as per TSHestonMOM. On the day of portfolio formation, the strategy utilizes a high-minus-low decile portfolio, taking a maximum long and short position for the top and bottom 10% of ranked stocks respectively and holding the positions to expiry. We first calculate raw momentum scores Y_0:t^(i, straddle) = r_0-nM, 0^(i, straddle) according to Equation (<ref>). Then, we rank stocks by their raw momentum scores:

    X_0:t^(i, straddle)   = {+1, -1, 0}

where X_0:t^(i, straddle) = +1 for stocks ranked in the top 10% and -1 for the bottom 10% ranked stocks, and 0 otherwise. We examine both the momentum X_0:t^(i, straddle) (CSHestonMOM) and mean reversion -X_0:t^(i, straddle) (CSHestonMR) strategies.





§ DEEP LEARNING FOR OPTIONS TRADING




 §.§ General End-To-End Framework

We frame the problem of generating optimal trading decisions X_t^(i, straddle) for a portfolio of options with a model f as an end-to-end framework. Given time t and a straddle option of stock i, we have an input 𝐮_t^(i, straddle)∈ℝ^d of option features. We learn a model f with trainable parameters θ:

    X_t^(i, straddle) = f(𝐮_t^(i, straddle) ; θ)


In this framework, trading signals X_t^(i, straddle) are directly computed using the point-in-time snapshot of the option's features, integrating both trend prediction and optimal position sizing within a single function f. Unlike standard supervised learning paradigms, a distinctive characteristic of our end-to-end framework is the unavailability of ground-truth labels for the optimal trading signal X_t^(i, straddle) of a given option at any point in time. This necessitates the learning of a non-trivial mapping from an option's features to optimal trading signals. We discuss the choice of architectures and optimization of these models in the following sections.



 §.§ Network Architectures

Given the problem of learning a mapping from option features to trading signals, it is not immediately clear which choice of architecture would best suit an end-to-end model, warranting the need to consider multiple options. Taking this into account, we examine various choices of neural networks for f.



  
Linear Beginning with the elementary case of a neural network comprising of a single fully connected layer:

    X_t^(i, straddle) 
    = g(𝐖^⊤𝐮_t^(i, straddle) + b)

where 𝐖∈ℝ^m, 𝐮_t^(i, straddle)𝐮_t-τ+1 : t^(i, straddle)∈ℝ^m with m = τ d, b ∈ℝ and g = tanh is the activation function. The model computes a linear combination of the input features prior to the tanh activation function that outputs the trading signal to be within the finite range of [-1, 1]. To factor in the immediate temporal history of observations for making predictions, we concatenate features from the past τ = 5 days from time t into a single input vector. 



  
Multilayer Perceptron (MLP) We include an additional hidden layer to the Linear model, enhancing the model's depth: 

    X_t^(i, straddle) 
    = g 𝐖^ 2 ⊤σ (𝐖^ 1 ⊤𝐮_t^(i, straddle) + b^ 1 ) + b^ 2

with g = σ = tanh corresponding to the activation functions of each layer.



  
Convolutional Neural Networks (CNN) Modified for time series data, CNNs have been designed to incorporate causal convolutions that utilize only past information for forecasting <cit.>, maintaining the autoregressive ordering of temporal features. We consider a 1-D autoregressive CNN:

    𝐡_t^(i, straddle)   = P σ𝐖_c^ 2 ∗σ(𝐖_c^ 1 ∗𝐮_t^(i, straddle) + 𝐛_c^ 1 ) + 𝐛_c^ 2 
    
    X_t^(i, straddle)   = g 𝐖^ 2 ⊤σ (𝐖^ 1 ⊤𝐡_t^(i, straddle) + b^ 1 ) + b^ 2

where 𝐖_c^ l represent convolutional kernels with bias terms 𝐛_c^ l and activation functions g = σ = tanh, and ∗ represents the causal convolution operator. Subsequently, we perform average pooling with P prior to passing the activations 𝐡_t to a fully connected neural network.



  
Long Short-term Memory (LSTM) Recurrent neural networks (RNNs) have traditionally found applications in sequence modelling and time series forecasting <cit.>. Given the sequential nature of our prediction task, it is natural to consider recurrent architectures. We implement a single layer LSTM model <cit.> that takes in an input sequence of option features 𝐮_t^(i, straddle)𝐮_t-τ+1 : t^(i, straddle)∈ℝ^m with m = τ d where τ represents the length of a trajectory, and subdivide the time series into trajectories of τ = 20 during backpropagation. We omit the technical equations for the LSTM architecture for brevity. 



    Γ^i_t    = σ_S(𝐖_i𝐮_t + 𝐕_i𝐡_t-1 + 𝐛_i) 
    Γ^f_t    = σ_S(𝐖_f𝐮_t + 𝐕_f𝐡_t-1 + 𝐛_f) 
    Γ^o_t    = σ_S(𝐖_o𝐮_t + 𝐕_o𝐡_t-1 + 𝐛_o) 
    𝐜̃_t    = g(𝐖_c𝐮_t + 𝐕_c𝐡_t-1 + 𝐛_c) 
    𝐜_t    = Γ^i_t ⊙𝐜̃_t + Γ^f_t ⊙𝐜_t-1
    𝐡_t    = Γ^o_t ⊙ g(𝐜_t) 
    𝐗_t    = g(𝐖_dist𝐡_t + 𝐛)





 §.§ Training Details



  §.§.§ Loss Function


To facilitate the learning of a non-trivial mapping from option features to optimal trading signals that effectively balances both risk and reward, we directly calibrate the models using the Sharpe ratio <cit.>, a risk-adjusted performance metric. Given a set of contemporaneous option features and their respective trading signals 𝒟_Ω = {(𝐮_t^(i, straddle),  X_t^(i, straddle) = f(𝐮_t^(i, straddle) ; θ)) } with Ω = { (i,   t) | i = 1, ⋯, N^t,   t = 1, ⋯, T } denoting all straddle-time pairs, we define the loss ℒ_sharpe(θ) over 𝒟_Ω as the annualized Sharpe ratio: 

    ℒ_sharpe(θ)    = - 1/|Ω|∑_Ω R_i(t) ×√(252)/√(1/|Ω|∑_Ω R_i(t)^2 - [1/|Ω|∑_Ω R_i(t)]^2)
    
    R_i(t)    = X_t^(i, straddle) (σ_tgt/σ_t^(i, straddle)) r_t,t+1^(i, straddle)





  §.§.§ Optimization

Within each in-sample window, we perform a train-validation split with the earlier 90% of data used for calibrating the models and the most recent 10% reserved for validation. To calibrate the models, we perform backpropagation using minibatch stochastic gradient descent with Adam <cit.>, and trigger early stopping with a patience of 25 epochs based on the validation loss. In order to select optimal candidates for each machine learning model, we conduct hyperparameter optimization with 100 iterations of random search. We refer the reader to Appendix <ref> for the detailed description of hyperparameter search ranges. Model calibration was performed on a server equipped with an AMD EPYC7713 CPU and multiple NVIDIA L40 GPUs.




§ PERFORMANCE EVALUATION




 §.§ Backtest Details

Following an expanding window approach, we train all models with every block of 5 additional years. In each block, we fix the weights and hyperparameters of the trained models and evaluate the models out-of-sample in the following 5-year window. We perform model calibration over multiple seeded runs and present the aggregated out-of-sample results in Section <ref>.



 §.§ Option Features


To construct an input of option features 𝐮_t^(i, straddle)∈ℝ^d as described in Equation (<ref>), we include a combination of the predictors used in the strategies outlined in Section <ref>:

    
  * Normalized Returns – we use r^(i, straddle)_t-k, t (σ^(i, straddle)_t √(k)), representing straddle returns normalized by daily volatility estimates scaled to a time scale k ∈{1, 5, 10, 15, 20}, which corresponds to daily, weekly, biweekly, triweekly and monthly returns. 
    
  * MACD Indicators – we take volatility normalised MACD signals Y_t^(i, straddle) (S_k, L_k) from Equation (<ref>) with short and long time scales S_k∈{2, 4, 8} and L_k∈{8, 16, 32}.
  
    
  * Option Momentum Features – we expand our set of predictors to include the momentum features as defined in Equation (<ref>), taking the average returns of straddle options for the stock over lookback periods of n = 1, 3, 6, 12 months.
    
    
  * Core Option Features – to facilitate comparability with the trend-based benchmarks outlined in Section <ref>, and to focus on the predictive power of the features above, we maintain a parsimonious set of core option features which includes the log-moneyness (of both call and put options forming the straddle) and days to expiry (DTE, in years). Given that the moneyness and DTE of a straddle option changes over time, these core contract features are necessary in identifying the option at particular stages of its lifespan. Distinctively, we exclude other features such as option implied volatility or sensitivity measures such as Greeks which would require making an assumption of an underlying option pricing model such as the Black-Scholes or the binomial model. Given our focus on delta-neutral straddles, we also exclude underlying stock characteristics and stock returns from our core set of features.





 §.§ Results and Discussion













We use the following annualized metrics to evaluate the out-of-sample performance of all strategies:

    
  * Profitability Measures – Expected Returns (𝔼[Returns]), Hit Rate
    
  * Risk Measures – Volatility (Vol.), Downside Deviation, Maximum Drawdown (MDD)
    
  * Performance Ratios – Sharpe, Sortino and Calmar Ratios, Average Profit over Loss (Ave. P/Ave. L)



We present the aggregated out-of-sample performance metrics of all strategies computed using the overall returns according to Equation (<ref>). Firstly, we present the performance of all strategies from their raw signal outputs in Table <ref>. We then apply to all strategies (excluding unprofitable strategies) an additional layer of volatility scaling to target an annualized volatility of 15% at the portfolio level and report the performance in Table <ref> and plot their cumulative returns in Figure <ref>. This adjustment at the portfolio level facilitates comparison between individual strategy returns in line with our 15% volatility target. For each Heston portfolio (TSHestonMOM, TSHestonMR, CSHestonMOM, CSHestonMR), we report results for the best performing lookback period for the sake of brevity. In this section, we report the performance of all strategies without factoring in transaction costs to evaluate their raw predictive ability. In Section <ref>, we include an analysis of the impact of transaction costs and the effect of turnover regularization.


From Table <ref>, we find that the Long Only straddle portfolio was a profitable strategy over the backtest period. We find this observation interesting, running contrary to <cit.> who show that retail and institutional investors executing short volatility strategies tend to perform well. Furthermore, a Long Only options portfolio would typically benefit from limited downside exposures as opposed to Short Only. Turning our attention to trend-based strategies, we observe that mean-reversion portfolios (TSMR, MACDMR, TSHestonMR, CSHestonMR) exhibit positive performances compared to their opposite momentum counterparts (TSMOM, MACD, TSHestonMOM, CSHestonMOM), which were generally unprofitable over the backtest period. We note that both TSMR and CSHestonMR exhibited only slight performance improvements over Long Only. We obtain similar results for the Heston portfolios as reported in <cit.>, who document significant reversals in option returns at short-term horizons, and our best performing Heston models correspond mostly with strategies adopting the shortest n = 1 month lookback period. Moving on to our deep learning models, apart from the CNN, we observe that the Linear, MLP and LSTM exhibit a clear disparity in performance above all other strategies as seen from their higher performance ratios.


Referring to Table <ref>, with the implementation of volatility targeting at the portfolio level, we observe modest improvements in performance across all profitable benchmarks. Portfolio volatility targeting resulted in minimal changes to the deep learning models, allowing them to retain a large gap in their performance ratios over the benchmarks. In particular, we see that the Linear and LSTM models exhibited the best performances, outperforming the benchmarks by roughly twice in their Sharpe ratios of 1.290 and 1.329 respectively. We note that the simplest Linear model was able to perform the MLP, most likely due to our introduction of an L1 regularization penalty only for the Linear model during training. Examining the period during the COVID-19 market selloff, we see clearly from Figure <ref> that Long Only exhibited sharp gains during the COVID-19 market selloff at the start of 2020, demonstrating the profitability of long straddles during periods of high market volatility. While we observe that the deep learning models experienced brief drawdowns during the selloff, performance of the models swiftly recovered following the market rebound.




 §.§ Transaction Costs and Turnover Regularization




Some of the key challenges of rebalancing an options portfolio include market microstructure considerations arising from market liquidity and bid-ask spreads, which can result in high transaction costs. To examine the impact of transaction costs on the profitability of all strategies, we first compute Sharpe ratios adjusted for transaction costs by taking into account turnover-adjusted returns: 

    r̃_t,t+1^STRATEGY = 1/N_t∑_i=1^N_t(R_i(t) -  c ·σ_tgt| X_t^(i, straddle)/σ_t^(i, straddle) -  X_t-1^(i, straddle)/σ_t-1^(i, straddle)| )

where c represents a measure of average transaction costs in basis points. Focusing on the LSTM model, we observe from Table <ref> that the model maintains superior risk-adjusted performance over the best performing benchmark - TSMR up to transaction costs of c = 20 bps, deteriorating at higher transaction costs of c = 50 bps.

Following the methodology detailed in <cit.>, we modify the training loss to utilize turnover-adjusted returns as defined in Equation (<ref>), which in effect results in optimizing the Sharpe ratio while regularizing for the turnover generated by the trading signals of the LSTM. Based on Table <ref>, we see that turnover regularization further enhances the performance of the LSTM for high transaction costs of c = 10 to 50 bps, allowing the regularized model to outperform other strategies at prohibitively high levels of transaction costs.






§ CONCLUSIONS


We present a general end-to-end framework for trading options using a highly data-driven machine learning algorithm, adopting the point of view of an active investor seeking to profit from options trading. Departing from conventional approaches that typically rely on specific market dynamics or option pricing models, we train end-to-end neural networks to directly learn mappings from options data to optimal trading positions, removing the need to simulate market processes, price options or predict option returns. Backtesting our approach on portfolios of delta-neutral equity options, our models demonstrate significant improvements in risk-adjusted performance when calibrated using the Sharpe ratio. Crucially, our framework is agnostic to specific underlying market assumptions, potentially allowing for further extensions to a broader set of derivatives or complex instruments where data is available.





We would like to thank the Oxford-Man Institute of Quantitative Finance for providing compute resources. Wee Ling Tan thanks Bryan Lim, Martin Luk, Guillaume Andrieux and Hans Buehler for their insightful comments.







ACM-Reference-Format













§ HYPERPARAMETER OPTIMIZATION







