Paper Title: A Cross-Domain Benchmark for Active Learning.


	









[
    *
    August 5, 2024
==================




§ ABSTRACT 

Active Learning (AL) deals with identifying the most informative samples for
labeling to reduce data annotation costs for supervised learning tasks. AL
research suffers from the fact that lifts from literature generalize poorly and
that only a small number of repetitions of experiments are conducted. To overcome
these obstacles, we propose CDALBench, the first active learning benchmark
which includes tasks in computer vision, natural language processing and tabular
learning. Furthermore, by providing an efficient, greedy oracle, CDALBench
can be evaluated with 50 runs for each experiment. We show, that both the
cross-domain character and a large amount of repetitions are crucial for
sophisticated evaluation of AL research. Concretely, we show that the
superiority of specific methods varies over the different domains, making it
important to evaluate Active Learning with a cross-domain benchmark.
Additionally, we show that having a large amount of runs is crucial. With only
conducting three runs as often done in the literature, the superiority of
specific methods can strongly vary with the specific runs. This effect is so strong, that, depending on the seed, even a well-established method's performance can be significantly better and significantly
worse than random for the same dataset. 






































§ INTRODUCTION

Deep neural networks (NN) have produced state-of-the-art results on many
important supervised learning tasks. Since Deep NNs usually require large
amounts of labeled training data, Active Learning (AL) deals with
selecting the most informative samples out of a large pool of unlabeled data, so
that only these samples need to be labeled. It has been shown that a small
labeled set of this nature can be used to train well-performing models.






In the last decade, many different algorithms for AL have been proposed and
almost every method has reported lifts over all its predecessors. [Out
of all considered algorithms for this paper, only BALD <cit.> did
not claim a new state-of-the-art (SOTA) performance in their result section.] However, real
insights into the current state of AL are hard to draw from these works, due to
the following reasons:


  *  These works do not use a standardized evaluation setting with fixed
datasets and baseline approaches.

  * Due to computational constraints, a lot of works perform only a small amount of experimental
runs, hence it is questionable wether the superiority of a specific approach can
be concluded from the conducted experiments.

  * The works are only evaluated in a specific domain, such as computer vision
or language processing. However, AL is a general principle of supervised
learning, and thus methods should be evaluated in multiple domains to assess
their capabilities.
















While multiple benchmark suites have been proposed to solve
problem <ref>, to the best of
our knowledge, all of them are either limited in the domains they consider or do
not contain enough runs to generate conclusive results.
Hence, the current SOTA in AL is still not well-understood and principled
shortcomings of different algorithms and wether they are domain-independent are currently not identified.

Here we step in with CDALBench, an AL benchmark which covers
multiple application domains and reports a large amount of runs per
experiment, so that the significance of performance differences can be
estimated. Specifically, CDALBench consists of datasets from
computer vision, natural language processing and the tabular domain. We provide
our datasets both in normal format 
as well as “embedded” by a fixed embedding model, enabling evaluation of AL methods in
this semi-supervised setting.
Furthermore, we propose two novel synthetic datasets to highlight general
challenges for AL methods.

The applied evaluation protocol in CDALBench uses 50 runs for
each experiment. By having such a large amount of runs, we can evaluate the
significance of performance gaps and identify the best performing
approaches for each dataset as well as whole domains. 
Furthermore, we show that the small amount of runs other works do, in fact, 
produce misleading results. 
To be more specific, we show that if only 3 restarts are employed for each experiment, 
the performance of specific methods strongly varies. As we will see, even the ranking of
the different methods averaged over many datasets fluctuates with the specific set of runs. 
This effect is so strong, that, depending on the seed, even a well-established method's performance can be significantly better and significantly worse than random for the same dataset. 

To enable the computation of an oracle performance for a protocol with large amounts of restarts,
we propose a greedy oracle algorithm 
which uses only a small amount of search steps to estimate the
optimal solution. While being more time-efficient than established oracle
functions, it possibly underestimates the real upper bound performance. 
However, as our experiments will show, it is still outperforming all current AL methods
by at least 5% and thus is suitable as an upper bound.

Our experimental evaluation shows that there exists no clear SOTA
method for AL. The superiority of methods is strongly dataset- and
domain-dependent with the outstanding observation, that the image domain works
fundamentally different than the tabular and text domain. Here, the best
performing approach for text and tabular data, namely margin sampling, is
significantly outperformed by least confident sampling, which does not belong to
the top performing approaches in any other domain. Thus, using the performance
of AL approaches on the image domain as a proxy of AL
in general, as it is often done <cit.>, is questionable. To further analyze
performance of common methods, we propose Honeypot and Diverging Sine, two
synthetic datasets, designed to be challenging for naive decision-boundary- and clustering-based approaches respectively. Hence, they provide insights in principled shortcomings of AL methods.

In summary, CDALBench is an experimental framework which includes an
efficient oracle approximation, multiple application domains, enough repetitions to
draw valid conclusions and two synthetic tasks to highlight shortcomings of AL methods. 
By being the first benchmark to providing these points in one code-base, 
we believe that CDALBench is a major step forward of assessing the overall state of AL research, independent of
specific application domains. CDALBench is publicly available under 
<https://anonymous.4open.science/r/A-Cross-Domain-Benchmark-for-Active-Learning-CCB1/README.md>

 [1mm]
Our contributions include the following:



  *  We show that the small number of repetitions that previous works have employed is not
sufficient for meaningful conclusions. Sometimes even making it impossible to assess if a performance is above or below random.

  *  We propose an efficient and performant oracle which is constructed iteratively in a greedy fashion,
overcoming major computational hurdles.



  *  We propose CDALBench, the first general benchmark providing tasks
in the domains of image, text and tabular learning. It further contains
synthetic and pre-encoded data to allow for a sophisticated evaluation of AL
methods. Our experiments show, that there is no clear SOTA method
for AL across different domains.

  *  We propose Honeypot and Diverging Sin, two synthetic datasets
designed to hinder AL by naive decision-boundary- or
clustering-based approaches respectively. Thus, they provide an important tool to identify 
shortcomings of existing AL methods.






§ PROBLEM DESCRIPTION

Given two spaces ,, n=l+u data points with l ∈ℕ labeled examples ℒ = {(x_1, y_1),…, (x_l,y_l)}, u ∈ℕ unlabeled examples 𝒰 = {x_l+1,…,x_n}, a model ŷ: →, a budget ℕ∋ b ≤ u and an annotator A: 𝒳→𝒴 that can label x. 
We call x ∈𝒳, y ∈𝒴 predictors and labels respectively where (x,y) are drawn from an unknown distribution ρ. 
Find an AL method Ω: ^(i),^(i)↦ x^(i)∈^(i) that iteratively selects the next unlabeled point x^(i) for labeling

    ^(i+1)   ^(i)∪{(x^(i), A(x^(i)))}
    ^(i+1)   ^(i)∖{x^(i)}

with ^(0) = seed(, s) and ^(0) = (^(0)_i, A(^(0)_i))  i ∈ [1, …, s], where seed(, s) selects s points per class for the initial labeled set ^(0). 

So that the average expected loss ℓ: 𝒴×𝒴→ℝ of a machine learning algorithm fitting ŷ^(i) on the respective labeled set ^(i) is minimal: 

    min  1/B∑_i=0^B 𝔼_(x,y) ∼ρℓ(y, ŷ^(i))







§ RELATED WORK

While multiple benchmark suites have been proposed for AL, none of them provide experiments for more than two domains.
The authors of <cit.>, <cit.>, <cit.>, <cit.> and <cit.> even focus exclusively on the image domain.
Especially the tabular domain is underrepresented in preceding benchmarks, as only <cit.> provides experiments for it.
The interplay between AL and semi-supervised learning is similarly under-researched, as only two works exist <cit.>, both of them only using images.
An oracle algorithm has been proposed by two works <cit.>. 
Both of these algorithms rely on search and are computationally very expensive, while our proposed method efficiently can be constructed sequentially.





The two closest related works to this benchmark are <cit.> and <cit.>, who also place a much higher emphasis on the problem of evaluating AL methods under high variance than their predecessors (indicated in Tab. <ref> by a dashed line).
The authors of <cit.> posed a total of 11 “recommendations" for reliable evaluation of AL methods.
We largely adapt the proposed recommendations and extend their work to multiple domains and query sizes.
For a complete list of the recommendations and our implementation of them, please refer to App. <ref>.
This work also pays attention to the so-called “pitfalls" of AL evaluation proposed in <cit.>.
For a complete list of the pitfalls and our considerations regarding them, please refer to App. <ref>.
To the best of our knowledge, we are the first to extend reliable SOTA (based on <cit.>) experimentation to a total of 5 data domains and a high number of repetitions per experiment.







§ FEW REPETITIONS ARE NOT SUFFICIENT FOR MEANINGFUL RESULTS

To evaluate how many repetitions are necessary to obtain conclusive results in an AL experiment, we computed 100 runs of our top-performing AL method on one dataset.
Our best method is margin sampling and we chose the Splice dataset for its average size and complexity. 

This allows us firstly, to obtain a very strong estimation of the “true" average performance of margin sampling on this dataset and secondly, to draw subsets from this pool of 100 runs.
Setting the size of our draws to α and sampling uniformly, we can approximate a cross-validation process with α repetitions.
Each of these draws can be interpreted as a reported result in AL literature where the authors employed α repetitions.
Figure <ref> shows the “true" mean performance of margin sampling (green) in relation to random sampling (black) and the oracle performance (red).
We display 5 random draws of size α in blue.
We can observe that even for a relatively high number of repetitions the variance between the samples is extremely high, resulting in some performance curves being worse than random and some being significantly better.
When setting α = 50 we observe all samples to converge close to the true mean performance. 

In addition to this motivating example, we carried out our main evaluation (Tab. <ref>) multiple times by sampling 3 from our available runs uniformly at random and comparing the results.
We found significant differences in the performance of AL methods on individual datasets, as well as permutations in the final ranking.
This partly explains the ongoing difficulties in reproducing results for AL experiments and benchmarks.
The details can be found in App. <ref>.
For this benchmark we employ 50 repetitions of every experiment.





 §.§ Seeding vs. Repetitions

Considering the high computational cost of 50 repetitions, another approach to ensure consistency between experiments would be to reduce the amount of variance in the experiment by keeping as many subsystems (weight initialization, data splits, etc.) as possible fixed with specialized seeding. 

We describe a novel seeding strategy in Appendix <ref> that is capable of tightly controlling the amount variance in the experiment.
However, previous works have noted that an actively sampled, labeled set does not generalize well between model architectures or even different initializations of the same model (<cit.>), providing a bad approximation of the quality of an AL method (i.e. measured performances for an AL method might not even transfer to a different model initialization).
Hence, we opt for letting the subsystems vary in controlled way (For details, please refer to App. <ref>) and combine that with a high number of repetitions to obtain a good average of the generalization performance of each AL method. 







§ CDALBENCH: A CROSS-DOMAIN ACTIVE LEARNING BENCHMARK

R0.65
	

	Employed model, chosen budget and available query sizes for each dataset
	

	
	
			    Model      B     1     5     20     50     100     500     1K 

			Semi DNA        Linear     40     o     o                 

			Semi Splice     Linear     100     o     o     o     o          

			TopV2           BiLSTM     200     o     o     o     o           

			Splice          MLP        400     o     o     o     o     o        

			DNA             MLP        300     o     o     o     o     o        

			USPS            MLP        400     o     o     o     o     o        

			Semi Cifar10    Linear     450     o     o     o     o     o        

			Semi FMnist     Linear     500     o     o     o     o     o        

			Semi USPS       Linear     600     o     o     o     o     o        

			News            BiLSTM     3K           o     o     o     o    

			FMnist          ResNet18    10K                    o     o

			Cifar10         ResNet18    10K                    o     o 

		
	
	


A detailed description of the preprocessing of each dataset can be found in Appendix <ref>. 
 [1mm]
Tabular:
AL research conducted on tabular data is sparse (only <cit.> from the considered baseline papers). 
We, therefore, introduce a set of tabular datasets that we selected according to the following criteria:
(i) They should be solvable by medium-sized models in under 1000 samples, (ii) the gap between most AL methods and random sampling should be significant (potential for AL is present) and (iii) the gap between the AL methods and our oracle should also be significant (research on these datasets can produce further lifts).
We use Splice, DNA and USPS from LibSVMTools <cit.>.

Image:
We use FashionMNIST <cit.> and Cifar10 <cit.>, since both are widely used in AL literature.

Text:
We use News Category <cit.> and TopV2 <cit.>.
Text datasets have seen less attention in AL research, but most of the papers that evaluate on text (<cit.>, <cit.>) use at least one of these datasets. 
We use both, as they complement each other in size and complexity. 
 [1mm]

We would like to point out that these datasets are selected for speed of computation (both in terms of the required classifier and the necessary budget to solve the dataset). 
We are solely focused on comparing different AL methods in this paper and do not aim to develop novel classification models on these datasets.
Our assumption is that a well-performing method in our benchmark will also generalize well to larger datasets and classifiers, because we included multiple different data domains, classifier types and sizes in our experiments. 
 [1mm]
Adapting the semi-supervised setting from <cit.>, we offer all our datasets un-encoded (normal) as well as pre-encoded (semi-supervised) by a fixed embedding model that was trained by unsupervised contrastive learning .
The text datasets are an exception to this, as they are only offered in their encoded form.
Pre-encoded datasets enable us to test small query sizes on more complex datasets like Cifar10 and FashionMnist.
They also serve the purpose of investigating the interplay between semi-supervised learning techniques and AL, as well as alleviating the cold-start problem described in <cit.> as they require a way smaller seed set.
The classification model for every encoded dataset is a single linear layer with softmax activation.
The embedding model was trained with the SimCLR <cit.> algorithm adopting the protocol from <cit.>. 

To ensure that enough information from the data is encoded by our embedding model, the quality of embeddings during pretext training was measured after each epoch.
To this end, we attached a linear classification head to the encoder, fine-tuned it to the data and evaluated this classifier for test accuracy. 
The checkpoint of each encoder model will be provided together with the framework. 
 [1mm]
Every dataset has a fixed size for the seed set ^(0) of 1 sample per class, with the only exceptions being un-encoded FashionMnist and Cifar10 with 100 examples per class to alleviate the cold-start problem in these complex domains. 




 §.§ Query Sizes

We selected query sizes for each dataset to accommodate the widest range possible that results in a reasonable runtime for low query sizes and allows for at least 4 round of data acquisition for high query sizes.
The available query sizes per dataset can be found in Table <ref>.




 §.§ Realism vs. Variance

We would like to point out that some design choices for this framework prohibit direct transfer of our results to practical applications. 
This is a conscious choice, as we think that this is a necessary trade-off between realism and experiment variance.
We would like to highlight the following design decisions: 

(i) Creating test and validation splits from the full dataset rather than only the labeled seed set (following <cit.>). Fully fledged test and validation splits are unobtainable in practice, but they provide not only a better approximation of the methods generalization performance, but also a better foundation for hyperparameter tuning, which is bound to reduce variance in the experiment. 

(ii) Choosing smaller classifiers instead of SOTA models. Since we are not interested in archiving a new SOTA in any classification problem, we instead opt to use smaller classifiers for the following reasons:
Smaller classifiers generally exhibit more stable training behavior, on average require fewer sampled datapoints to reach their full-dataset-performance and have faster training times.
For every dataset, the chosen architecture's hyperparameters are optimized to archive maximum full-dataset performance.
Generally, we use MLPs for tabular, RestNet18 for image and BiLSTMs for text datasets.
Every encoded dataset is classified by a single linear layer with softmax activation.
The used model for each dataset can be found in Tab. <ref>.
For a detailed description and employed hyperparameters please refer to Appendix <ref>.





 §.§ Evaluation Protocol

Following <cit.>, the quality of an AL method is evaluated by an “anytime protocol" that incorporates classification performance at every iteration, as opposed to evaluating final performance after the budget is exhausted.
We employ the normalized area under the accuracy curve (AUC):

    AUC(_, ŷ, B) := 1/B∑_i=1^BAcc(_, ŷ^(i))



Since AUC is still influenced by the budget, we define a set of rules to set this hyperparameter upfront, so that we are not favoring a subset of methods by handcrafting a budget.
In this work, we choose the budget per dataset to be the first point at which one of 2 stopping conditions apply: (i) an method (except oracle) manages to reach 99% of the full-dataset-performance (using the smallest query size) or (ii) the best method (except oracle) did not improve the classifier's accuracy by at least 2% in the last 20% of iterations.
The first rule follows <cit.>, while the second rule prevents excessive budgets for cases with diminishing returns in the budget.
The resulting budgets can be found in Tab. <ref>. 
 [1mm]
As described in Sec. <ref>, we repeat each experiment 50 times.
Each repetition retains the train/test split (often given by the dataset itself), but creates a new validation split that is sampled from the entire dataset (not just the seed set ^(0)). 
 [1mm]

Apart from plotting standard performance curves and reporting their AUC values per dataset in App. <ref>, we primarily rely on ranks to aggregate the performance of an AL method across datasets.
For each dataset and query size, the AUC values of all AL methods are sorted and assigned a rank based on position, with the best rank being 1.
These ranks can safely be averages across datasets as they are no longer subjected to scaling differences of each dataset.
Additionally, we employ Critical Difference (CD) diagrams (like Fig. <ref>) for statistical testing.
CD diagrams <cit.> use the Wilcoxon signed-rank test, which is a variant of the paired T-test, to find significant differences of ranks between AL methods.
For a detailed description of how every CD diagram is created, please refer to App. <ref>. 







§ A GREEDY ORACLE ALGORITHM
 
Using additional resources, like excessive training time, or direct access to a labeled test set, an oracle method for AL finds the oracle set 𝒪_b for a given dataset, model, and
training procedure that induces the highest AUC score for a given budget.
However, due to the combinatorial nature of the problem, this is computationally
infeasible for realistic datasets. Hence, previous works have proposed
approximations to this oracle sequence. <cit.> used simulated
annealing to search for a subset with maximal test accuracy and used the best solution
after a fixed time budget. 
Even though their reported performance curves display
a significant lift over all other AL methods, we found the
computational cost of reproducing this oracle for all our datasets to be
prohibitive (The authors reported the search to take several days per dataset on
8 V100 GPUs). In this paper, we propose a greedy oracle algorithm that
constructs an approximation of the optimal set in an iterative fashion. 
Our oracle algorithm
uniformly samples at iteration i a subset 𝒰_S of size τ of the not already
labeled data points 𝒰^(i).
Then it recovers the label y for each of the sampled u ∈𝒰_S and selects the
point u for which the classifier ŷ^(i) trained on
ℒ^(i)∪{u} has maximal performance.







Due to the algorithms greedy nature (considering only the next point to pick),
our oracle frequently encounters situations where every point in u would
incur a negative lift (worsening the test performance). This can happen, for
example, if the oracle picked a labeled set that enables the classifier to
correctly classify a big portion of easy samples in the test set, but now fails
to find the next single unlabeled point that would enable the
classifier to succeed on one of the hard samples. This leads to a situation,
where no point can immediately incur an increase in test performance and
therefore the selected data point can be considered random. To circumvent this
problem, we use our best-performing AL method (margin sampling
<cit.>) as a fallback option for the oracle. Whenever the oracle
does not find an unlabeled point that results in an increase in performance, it
defaults to margin sampling from the entire unlabeled pool 𝒰^(i) in that iteration.
The resulting greedy algorithm
constructs an approximation of the optimal labeled set that consistently
outperforms all other algorithms by a significant margin, while requiring
relatively low computational cost (𝒪(Bτ)).
We fix τ = 20 in this work, as this gives us an average lift of 5% over the
best performing AL method per dataset (which is significant for AL
settings) and we expect diminishing returns for larger τ. The pseudocode
for our oracle can be found in App. <ref>.


Even though our proposed algorithm is more efficient than other approaches, the
computational costs for high budget datasets like Cifar10 and FashionMnist meant
that we could not compute the oracle for all 10000 datapoints. To still provide
an oracle for these two datasets, we select two points per iteration instead of
one and stop the oracle computation at a budget of 2000. The rest of the curve
is forecast with a 2-stage linear regression that asymptotically approaches the
upper bound performance of the dataset. A detailed description can be found in
App. <ref>.






§ EXPERIMENTS




 §.§ Implementation Details

At each iteration i the AL method picks an unlabeled datapoint based on a fixed set of information {ℒ^(i), 𝒰^(i), B, |ℒ^(i)|-|ℒ^(1)|, acc^(i), acc^(1), ŷ^(i), opt_ŷ}, where opt_ŷ is the optimizer used to fit ŷ^(i).
This set grants full access to the labeled and unlabeled set, as well as all parameters of the classifier and the optimizer.
Additionally, we provide meta-information, like the size of the seed set through |ℒ^(i)|-|ℒ^(1)|, the remaining budget though the addition of B and the classifiers potential through acc^(1) and acc^(i).
We allow AL methods to derive information from this set, e.g. predictions of the classifier ŷ^(i)(x);  x ∈𝒰^(i)∪ℒ^(i), clustering, or even training additional models.
However, the method may not incorporate external information e.g. other datasets, queries to recover additional labels, additional training steps for ŷ, or the test/validation set. 

For our study we selected AL methods with good performances reported by multiple different sources that can work with the set of information stated above.
For a list of all AL methods, please refer to Table <ref>, with detailed descriptions being found in Appendix <ref>. 
 [1mm]

The model ŷ^(i) can be trained in two ways. Either the parameters of the model are reset to a fixed initial setting ŷ^(0) after each AL iteration and the classifier is trained from scratch with the updated labeled set ℒ^(i), or the previous state ŷ^(i-1) is retained and the classifier is fine-tuned on ℒ^(i) for a reduced number of epochs.
In this work, we use the fine-tuning method for un-encoded datasets to save computational time, while we use the from-scratch training for encoded datasets since they have very small classifiers and this approach generally produces better results.
Our fine-tuning scheme always trains for at least one epoch and employs an aggressive early stopping with a patience of 2 afterwards.







 §.§ Results on Real-world Data

In Table <ref> we provide the rank of each AL method per dataset. 
Please note, that we are averaging not only over runs, but also over query sizes per dataset, impacting AL methods that do not adapt well to a wide range of query sizes.
For the results per query size, please refer to App. <ref>. 
As mentioned in contribution <ref>, our results on real-world data show significant differences in the performance of methods between data domains:
Not only do some methods overperform on some domains (like least confidence (LC) sampling on images), but the Top-3 of methods (except oracle) does not contain the same three methods for any two domains.
Most interestingly, the image domain, which received most of the attention in benchmarking so far could even be considered an outlier, as this is the only domain where the Top-1 method changes.
This highlights the dire need for diverse data domains in AL benchmarking. 

Results for the semi-supervised domain appear mostly in line with the other 3 domains, but a closer analysis of performances split into encoded images and encoded tabular reveals the need for further research.
For details, please refer to App. <ref>.







§ HONEYPOT AND DIVERGING SINE

AL approaches can be categorized into two types: uncertainty and geometric approaches.
Typical members of the first category are variants of uncertainty sampling like entropy, margin and LC sampling <cit.> as well as BALD <cit.>.
Typical members of the second category are clustering approaches like Coreset <cit.>, BADGE <cit.> and TypiClust <cit.>.
Both types of methods have principled shortcomings in terms of their utilized information that makes them unsuitable for certain data distributions. 
To test for these specific shortcomings, we created two synthetic datasets, namely “Honeypot" and “Diverging Sine", that are hard to solve for methods focused on the classifier's decision boundary or data clustering respectively. 
To avoid methods memorizing these datasets, they are generated from scratch for each experiment. 



Honeypot creates two easy to distinguish clusters and one “honeypot" that represents a noisy region of the dataset with potentially miss-labeled, miss-measured or generally adverse samples.

The honeypot is located on the likely decision boundary of a classifier that is trained on the beneficial samples to maximize its adverse impact on purely uncertainty-based AL methods.
Diverging Sine samples datapoints for each class from two diverging sinusoidal functions that are originating from the same y-intercept.
This creates a challenging region on the left hand side, where a lot of datapoints need to be sampled, and an easy region on the right hand side, where very few datapoints are sufficient. 
The repeating nature of a sine function encourages diversity-based AL methods to equally sample the entire length, drastically oversampling the right hand side of the dataset. 

Both datasets have a budget of B=60 and are tested with query sizes 1 and 5. 



We provide the rank of all AL Methods on Honeypot and Diverging Sine in Fig. <ref>.
Results for the Honeypot dataset reveal expected shortcomings of uncertainty sampling methods like margin, entropy and LC sampling as well as BALD.
In addition, BADGE is underperforming for this dataset compared to real-world data. 
Both margin sampling and BADGE (the two best methods) being vulnerable to adverse samples or simply measurement noise, highlights the need for further research into robust AL methods. 


Results for Diverging Sine also confirm expected behavior, as clustering methods (Coreset, TypiClust) fall behind uncertainty methods (entropy, margin, LC sampling), with the exception of BADGE. 
The fact that BADGE is able to perform well on Diverging Sine highlights the importance of embeddings for the clustering methods, as the gradient embedding from BADGE seems to be able to encode uncertainty information, guiding the selection into the left hand regions of the dataset. 
We provide a small ablation study on the importance of the embeddings by testing a version of Coreset and TypiClust on this dataset that does not use the embeddings produced by the classification model, but rather clusters the data directly.
“Coreset Raw" and “TypiClust Raw" both perform worse than their embedding-based counterpart.



§ CONCLUSION

We strongly advocate to test newly proposed AL methods not only on a wide variety of real data domains, but also to pay close attention to the Honeypot and Diverging Sine datasets to reveal principled shortcomings of the method in question.
Both tasks can be easily carried out by implementing the new AL method into our code base.
For Limitations and Future Work, please refer to App. <ref>.



 

  
Acknowledgement


Funded by the Lower Saxony Ministry of Science and Culture under grant number ZN3492 within the Lower Saxony “Vorab“ of the Volkswagen Foundation and supported by the Center for Digital Innovations (ZDIN).



plain
 


















































§ AL RECOMMENDATIONS FROM JI ET AL.



  
Recommendation 1 Use the backbone architecture with
the community-accepted definition that is best suited
for the dataset at hand and consistently use it across all
experiments. In the image classification domain, we
suggest using ResNet18 for CIFAR-10 and CIFAR-100. 

→ We are using Resnet18 for our image datasets and re-purposed the LSTM model from <cit.> for the text datasets.
For the tabular data, we ran a grid-search (full dataset test accuracy) over MLP architectures.



  
Recommendation 2 Control the type of optimizer across
methods for comparative evaluations to ensure that the
yield performance difference stems from an active learning method itself. As SGD often generalizes better, we
encourage its use for deep active learning.

→ We searched for the best optimizer per dataset via generalization performance (test accuracy). 



  
Recommendation 3 Pragmatically fix the learning rate
to 0.1 for SGD on image datasets. While continuous
hyperparameter tuning can improve overall performance,
a fixed learning rate does not change the ranking of AL
methods from a comparative evaluation’s point of view.

→ We found that for smaller datasets, a learning rate of 0.1 was unsuitable. Ji et al. only used large image datasets, so a learning rate of 0.1 was sufficient for them. The learning rate is part of our hyperparameter grid-search.



  
Recommendation 4 One may use data augmentation if
applied consistently across methods, such that it does
not affect the overall ranking. However, a commonly
accepted baseline is needed, e.g., random horizontal flipping and random cropping for image classification.

→ We did not find a data augmentation technique that could be applied equally on all datasets, so we refrained from it.
The only possibility would be Gaussian noise, but the impact of gaussian noise on pre-encoded is not well-understood.



  
Recommendation 5 Refine model parameters across AL
batch (“warm starts”) to prevent exhaustive reinitialization and feed initialization of the backbone model’s
weights and the “init sets” with fixed inputs over multiple runs to average out the randomness. Moreover, use
identical seeds for all methods under investigation.

→ We employed a novel seeding strategy to closely control the seeding of our experiments (Details in App. <ref>). We applied warm-starts for most datasets, except pre-encoded ones, because we found a generally better performance, if we train the classifier from scratch. 



  
Recommendation 6 Run experiments multiple times to
compensate for non-deterministic operations. If the resulting variance is larger than the gained improvement,
use deterministic operations stringently

→ A study on (non-)deterministic operations has not been conducted in this work, but our large number of repetitions compensate for that.



  
Recommendation 7SW Configure and verify influence parameter in active learning implementations thoroughly. To foster future research, we
provide implementations as part of our framework at:
https://intellisec.de/research/eval-al

→ We provide our own baseline code, since we implemented a novel seeding strategy and unify many additional data domains in the code.



  
Recommendation 7HW Ensure that comparative evaluations are run on identical hardware. While it is not
necessary to execute all experiments on the same physical device, the GPU model, for instance, should be the
same. Do not mix hardware and list hardware details.

→ The large computational cost of our benchmark did not allow us to compute on only one type of hardware (Which would mean to only use part of our cluster). However, our large number of repetitions compensate for that.



  
Recommendation 8 Consider multiple query-batch sizes
in the evaluation. The choice of the sizes needs to be
appropriate for the total number of unlabeled samples.

→ We employed a wide range of batch sizes. For details, please refer to Table <ref>.



  
Recommendation 9 Compare active learning strategies
without sub-sampling, unless one of the approaches uses
it as a fundamental building block. In this case a detailed
analysis of the influence of sub-sampling is necessary.

→ We only carefully employed sub-sampling, when it was absolutely necessary to keep the computation times feasible.



  
Recommendation 10 Evaluate active learning strategies
on multiple benchmark datasets, that comprise balanced,
imbalanced, small-scale, and large-scale datasets to cover
most relevant cases in practice.

→ We deferred the study on (im-)balanced datasets to future work. However, our benchmark contains datasets of many different sizes and we extend this argument to domains as well.



  
Recommendation 11 For a comprehensive analysis of
AL strategies, the overall comparative evaluation should
incorporate as many variables from Section 3 to yield a
summarized PPM that is as expressive as possible. 

→ In our main result (Tab. <ref>) we average the performance over runs, query sizes and datasets.






§ AL PITFALLS FROM LUETH ET AL.



  
P1 Data distribution The proposed evaluation over a diverse selection of dataset distributions
including specific roll-out datasets proved essential for realistic evaluation of QMs as well as the
different training strategies. One main insight is the fact that class distribution is a crucial predictor
for the potential performance gains of AL on a dataset: Performance gains of AL are generally higher
on imbalanced datasets and occur consistently even for ST models with a small starting budget,
which are typically prone to experience cold start problems. This observation is consistent with a few
previous studies [...].

→ We selected our datasets according to their “potential" for AL. We measured this potential by the distance of most AL methods to random and the distance of the best AL method to our oracle. If both distances are >0, we consider the dataset useful.




  
P2 Starting budget The comprehensive study of various starting budgets on all datasets reveals
that AL methods are more robust with regard to small starting budgets than previously reported
[6, 20, 43]. With the exception of Entropy we did not observe cold start problems even for any QM
even in combination with notoriously prone ST models. The described robustness is presumably
enabled by our thorough classifier configuration (P4) and heuristically adapted query sizes (P3). This
finding has great impact potential suggesting that AL can be applied at earlier points in the annotation
process thereby further reducing the labeling cost [...].

→ Our experiments also showed a high resistance against the cold-start problem, which prompted us to use the smallest possible seed set for most datasets (1 point per class), with the only exception being (un-)encoded Cifar10 and FashionMnist. Here we employ a seed set of 100 points per class to avoid a cold-start.



  
P3 Query size Based on our evaluation of the query size we can empirically confirm its importance
with regard to 1) general AL performance and 2) counteracting the cold start problem. The are,
however, surprising findings indicating that the exact interaction between query size and performance
remains an open research question. [...].

→ We generally observed a decreasing performance for larger query sizes. We therefore made sure, that we include the smallest possible query sizes that result in feasible computation times.



  
P4 Classifer configuration Our results show that method configuration on a properly sized validation
set is essential for realistic evaluation in AL. [...] This raises the question of to which extent reported
AL advantages could have been achieved by simple classifier configurations. Further, our models
also generally outperform expensively configured models by Munjal et al.. Thus, we conclude
that manually constraining the search space renders HP optimization feasible in practice without
decreasing performance and ensures performance gains by Active Learning are not overstated. The
importance of the proposed strategy to optimize HPs on the starting budget for each new dataset is
supported by the fact that the resulting configurations change across datasets.

→ We also strongly advocate the use of a fully fledged validation set for HP tuning, as this allows for a higher quality of HPs, which in turn reduces the variance of the experiments.



  
P5 Alternative training paradigms Based on our study benchmarking AL in the context of
both Self-SL and Semi-SL, we see that while Self-SL generally leads to improvements across all
experiments, Semi-SL only leads to considerably improved performance on the simpler datasets
CIFAR-10/100, on which Semi-SL methods are typically developed. Generally, models trained
with either of the two training paradigms receive a lower performance gain from AL (over random
querying) compared to ST. [...] The fact that AL entails multiple training iterations
amplifies the computational burden of Semi-SL, rendering their combination prohibitively expensive
in most practical scenarios. Further, the fact that our Semi-SL models based on Fixmatch do not seem
to generalize to more complex datasets in our setting stands in stark contrast to conclusions drawn by
[...] as to which the emergence of Semi-SL renders AL redundant. Interestingly, the exact settings
where Semi-SL does not provide benefits in our study are the ones where AL proved advantageous.
The described contradiction with the literature underlines the importance of our proposed protocol
testing for a method’s generalizability to unseen datasets. 

→ Due to the high computational costs described by Lueth et al., we opted for the most efficient form of semi-supervised learning, which is to train a fixed encoder-model, pre-encode the datasets and then only train a single linear layer as classifier.





§ DIFFERENCE OF RANKS WITH 3 REPETITIONS

Table <ref> and Table <ref> follow the exact same computation of ranks that created the main result (Table <ref>) with the only difference being a reduced number of runs per AL method.
For each table we sampled 3 runs uniformly at random from the available 50 per AL method. 

We can observe significant differences between the two tables: 

Purple: A multitude of rank differences of AL methods for specific datasets, some as high as 4.7 ranks for TypiClust on the Splice dataset 

Olive: Well separated AL methods in Tab. <ref> (Margin and BADGE) are almost indistinguishable in Tab <ref> 

Red: BALD lost 2 places in the overall ranking and Entropy gained 2 
 [1mm]
Even though the overall ordering of AL methods stayed relatively unchanged due to the averaging across many datasets, each individual dataset was subject to drastic permutations.
This highlights the need for many repetitions in AL experiments.







§ SEEDING STRATEGY

We aim to provide an experimental setup that is fully reproducible independent of the dataset, classification model, or AL method used.
For a fair comparison of two AL methods, both methods need to receive equal starting conditions in terms of train/validation split, initialization of classifier, and even the state of minor systems like the optimizer or mini-batch sampler.
Even though different implementations might have their own solution to some of these problems, only <cit.> has described and implemented a fully reproducible pipeline for AL evaluation.
The term reproducibility in this work is used as a synonym not only for the reproducibility of an experiment (a final result given a seed), but also the reproducibility of all subsystems independent of each other.
The seed for one subsystem should always reproduce the behavior of this subsystem independent of all other subsystems and their seeds.
The main obstacle for ensuring reproducibility is the seeding utility in PyTorch, Tensorflow, and other frameworks, whose default choice is a single global seed.
Since many subsystems draw random numbers from this seed, all of them influence each other to a point where a single additional draw can completely change the model initialization, data split or the order of training batches.
Even though some workarounds exist, e.g. re-setting the seed multiple times, this problem is not limited to the initialization phase, but also extends to the AL iterations and the systems within.
We propose an implementation that creates separate Random Number Generators (RNGs) for each of these systems to ensure equal testing conditions even when the AL method, dataset, or classifier changes.
We hypothesize that the insufficient setup with global seeds contributes to the ongoing problem of inconsistent results of AL methods in different papers. 
 [1mm]
In summary, we introduce three different seeds: s_Ω for the AL method, s_𝒟 for dataset splitting and mini-batch sampling, and s_θ for model initialization and sampling of dropout masks.
Unless stated otherwise, we will keep s_Ω fixed, while s_𝒟 and s_θ are incremented by 1 between repetitions to introduce stochasticity into our framework.
Some methods require a subsample to be drawn from 𝒰 in order to reduce the computational cost in each iteration, while others need access to the full unlabeled pool (e.g. for effective clustering).
If a subsample is required, it will be drawn from s_Ω and therefore will not influence other systems in the experiments.
For each method, we decided if subsampling is required based on our available hardware, but decided against setting a fixed time limit per experiment, since this would introduce unnecessary complexity into the benchmark.
An overview of selected hyperparameters per AL method can be found in Appendix <ref>. 

Note: Even though we decoupled the subsystems via the described seeds, the subsystems can still influence each other in a practical sense. 
For example, keeping s_𝒟 fixed does not mean that always the same sequence of samples from 𝒰 (if subsamples are drawn) are shown to all AL methods. 
This is practically impossible, as different AL methods pick different x^(i).
However, the hypothetical tree of all possible sequences of samples from 𝒰 remains the same, granting every AL methods equal possibilities.




§ HYPERPARAMETERS AND PREPROCESSING PER DATASET

For all our datasets we use the pre-defined train/test splits, if given. 
In the remaining cases, we define test sets upfront and store them into separate files to keep them fixed across all experiments.
The validation set is split in the experiment run itself and depends on the dataset-seed.

Tabular:
We use Splice, DNA and USPS from LibSVMTools <cit.>.
All three datasets are normalized between [0, 1]. 

Image:
We use FashionMNIST <cit.> and Cifar10 <cit.>, since both are widely used in AL literature.
Both datasets are normalized according to their standard protocols. 

Text:
We use News Category <cit.> and TopV2 <cit.>.
For News Category we use  the 15 most common categories as indicated by its Kaggle site.
We additionally drop sentences above 80 words to reduce the padding needed (retaining 99,86% of the data).
For TopV2, we are only using the "alarm" domain.
Both datasets are encoded with pre-trained GloVe (Common Crawl 840B Tokens) embeddings <cit.>.
Since neither dataset provided a fixed test set, we randomly split 7000 datapoints into a test set.










§ PERFORMANCE CURVES PER DATASET





































§ CRITICAL DIFFERENCE DIAGRAMS

We adapted the code for the CD diagrams from <cit.>. 

To compare each AL method, we consider each combination of dataset, query size and run is considered a separate “experiment", i.e. the results of  of an AL Method  is only compared to the results of  of AL method . 

Depending on the use-case, we build the following “experiments":

	
  * Single dataset - single query size: Each AL method has 50 “experiments" in it's 50 repetitions
	
  * Single dataset - all query sizes (Fig. <ref>): Each “experiment" is represented by a string 
	
  * Multiple dataset - all query sizes (Fig. <ref>): Each “experiment" is represented by a string 

Due to the large number of restarts and the wide range of datasets and query sizes, we can provide very accurate significance tests.





§ AL PSEUDOCODE






Alg. <ref> replaces the AL method Ω in the AL loop (Alg. <ref> line 5). 





§ ORACLE CURVE FORECASTING

Unfortunately, the iterative nature of our oracle means that the computational effort scales in the budget B.
For datasets with large budgets, like Cifar10 and FashionMnist (both 10K), we were unable to compute the oracle set for the entire 10K iterations. 

We compromised by (i) picking the two points with highest test accuracy, instead of only one and (ii) only computed until iteration 2000. 

The rest of the curve was forecast using a simple 2-step algorithm, based on linear regression:

	
  * Fit a linear regression model on the second 50% of the existing oracle curve (to accurately capture the trend of the oracle, rather than the intercept) and forecast the oracle performance for the remaining budget.
	
  * Post-process the oracle forecast by letting it asymptotically approach the upper bound performance of the dataset.


    o_i    = min 
    		o_i 
    ϕ(i) * o_i + (1 - ϕ(i)) * upper bound
    ϕ(i)    = e^-i / 0.5









§ AL METHODS

Uncertainty Sampling 
tries to find the sample that the classifier is most uncertain about by computing heuristics of the class probabilities. For our benchmark, we use entropy and margin (a.k.a. best-vs-second-best) sampling.

BALD <cit.>
applies the query-by-committee strategy of model ensembles to a single model by interpreting the classifier's parameters as distributions and then sample multiple outputs from them via Monte-Carlo dropout.

BADGE <cit.> uses gradient embeddings of unlabeled points to select samples where the classifier is expected to change a lot. The higher the magnitude of the gradient the higher the expected improvement in model performance.
BADGE employs a variant to the KMeans++ initialization technique to select batches of points. Even though <cit.> provided pseudocode for this procedure that selects the first point at random, all found implementations of BADGE select the first points instead by maximum gradient magnitude. 

Coreset <cit.>
employs K-Means clustering trying to cover the whole data distribution.
Selects the unlabeled sample that is the furthest away from all cluster centers.
Clustering is done in a semantically meaningful space by encoding the data with the current classifier ŷ.
In this work, we use the greedy variant of Coreset.

TypiClust <cit.>
relies on clustering similar to Coreset, but proposes a new measure called “Typicality" to select unlabeled samples.
It selects points that are in the densest regions of clusters that do not contain labeled samples yet.
Clustering is done in a semantically meaningful space by encoding the data with the current classifier ŷ.
It has to be pointed out that TypiClust was designed for low-budget scenarios, but we think it is still worthwhile to test and compare this method with higher budgets. 

Core-GCN <cit.> trains a Graph-Convolutional-Network (GCN) on embeddings of the unlabeled pool, obtained from the classifier (Similar to Coreset and TypiClust). 
This GCN model propagates uncertainty information through the graph and therefore enhances the nodes uncertainty quantification. Lastly, the node that displays the highest amount of uncertainty is selected for labeling. 

DSA/LSA <cit.> use the metric of test adequacy to construct a set of points that is diverse, ranging from points that are close to points in  and points that are significantly different. DSA and LSA measure the diversity of points by distance in embedding space or likelihood estimation under the given classifier respectively. 
 [2mm]

Excluded Methods

Learning Loss for AL <cit.>
Introduces an updated training of the classification model with an auxiliary loss and therefore cannot be compared fairly against classification models without this boosted training regime.
 [1mm]

Reinforcement Learning Methods 

We postpone the study of learned AL methods to future versions of this benchmark, as reinforcement learning is infamous for being extremely time consuming and itself hard to reproduce .






§ AUCS BY QUERY SIZE

All tables are sorted according to the main result in Table <ref>.










0.47
	
	


0.47
	
	








§ ANALYSIS OF RESULTS FOR THE SEMI-SUPERVISED DOMAIN

Even though the results for the aggregated semi-supervised domain appear in line with our overall ranking of methods, we observe stark differences for the sub-domains of semi-supervised image and semi-supervised tabular. 

While semi-supervised images seem to mostly mirror the results from the normal image domain (with the exception of BALD), semi-supervised tabular data display highly irregular behavior, placing random sampling as second-best method behind the margin sampling.
Our oracle method even falls behind other methods.
Almost all methods a bunched into one region in the CD diagram with many non-significance bars indicating few, if any, significant differences between the methods. 

Both the reasons, for the sub-random performance of most methods, and the bad performance of our oracle are currently unknown and require further research.







§ HYPERPARAMETERS PER AL METHOD








§ AMOUNT OF COMPUTATIONAL RESOURCES INVESTED

For our results we computed a total of 24200 runs (without Oracle runs) over a span of  4 months. 

We used our computational cluster consisting of 30-40 GPUs. 
 [1mm]
Number of runs per dataset and query size : 11 alg. * 50 runs = 550 
 [1mm]
Runs per dataset: 

 Cifar10   x2 = 1100

 Cfr10Enc  x4 = 2200

 DivSin    x2 = 1100

 DNA       x4 = 2200

 DNAEnc    x2 = 1100

 FMnist    x2 = 1100

 FMnistEnc x4 = 2200

 News      x3 = 1650

 Splice    x4 = 2200

 SpliceEnc x3 = 1650

 Honeypot  x2 = 1100

 TopV2     x4 = 2200

 USPS      x4 = 2200

 USPSEnc   x4 = 2200

 = 24200 runs
 
 
 
 
 

§ LIMITATIONS AND FUTURE WORK

 Even though our benchmark includes a wide range of data domains, the number of datasets per domain is still limited.
 It remains untested if our selected datasets are indeed a good representation of their domain, or if additional datasets would skew the results of Fig. <ref>. 
 
 Additionally, since we began working on this benchmark a few new AL Methods have been published.
 We consciously focused on only those methods for which good results have been reported by multiple sources, consequently omitting the newest methods. 

 Most obviously, our future work involves the implementation of more datasets per domain and the newest AL methods. 
 [1mm]
 The choice of ranks for the main result table, like any other choice, has an impact in the interpretation of the results.
 E.g. comparing the rank of BALD in Table <ref> (rank 6.6) with the amount of wins it is able to obtain in the AUC-based tables in Appendix <ref>, suggests that an evaluation purely based on mean AUC values would count BALD to the best methods. 

 We advocate for using ranks in AL evaluations for two reasons: (i) they are more robust to outlier performances in single runs and (ii) they highlight wether an method is able to consistently outperform another method, even if the difference in mean AUC is very small. 

 Nonetheless, we think that the topic of truly fair evaluations for AL needs further research. 
 [1mm]
 Lastly, it remains untested if the differences between domains that we observe, are truly caused by the differences in data, but could also be influenced by the type of model that is common to those domains. 

 An important part of our future work is therefore to test different model archetypes per domain.
 



 Paper Title: Black box meta-learning intrinsic rewards for sparse-reward environments.



Non-radial oscillations mimicking a brown dwarf orbiting the cluster giant NGC 4349 No. 127
    Dane SpaethFellow of the International Max Planck Research School for Astronomy and Cosmic Physics at the University of Heidelberg (IMPRS-HD)
          Sabine Reffert
          Emily L. Hunt
          Adrian Kaminski
          Andreas Quirrenbach
    Received 28 March 2024 / 
   Accepted 09 July 2024
==========================================================================================================================================================================================================================================================



§ ABSTRACT 
Despite the successes and progress of deep reinforcement learning over the last decade, several challenges remain that hinder its broader application. Some fundamental aspects to improve include data efficiency, generalization capability, and ability to learn in sparse-reward environments, which often require human-designed dense rewards. Meta-learning has emerged as a promising approach to address these issues by optimizing components of the learning algorithm to meet desired characteristics. Additionally, a different line of work has extensively studied the use of intrinsic rewards to enhance the exploration capabilities of algorithms. This work investigates how meta-learning can improve the training signal received by RL agents. The focus is on meta-learning intrinsic rewards under a framework that doesn't rely on the use of meta-gradients. We analyze and compare this approach to the use of extrinsic rewards and a meta-learned advantage function. The developed algorithms are evaluated on distributions of continuous control tasks with both parametric and non-parametric variations, and with only sparse rewards accessible for the evaluation tasks.





§ INTRODUCTION


The adoption of neural networks and recent algorithmic advances in reinforcement learning (RL) have enabled its application to complex decision-making problems <cit.>.  However, several challenges persist that must be addressed before RL can be effectively applied to broader range of domains, especially those requiring interaction with the real world.

Several sub-fields of research have emerged out of this need <cit.>. Two main issues they target are poor data efficiency in task learning and the limited generalization capabilities of learned policies when applied to new tasks. Related to data efficiency, another central challenge in RL is devising efficient exploration strategies, and achieving an appropriate trade-off with exploitation.

Reinforcement Learning: 
 In this work, we deal with discrete-time finite-horizon discounted Markov decision processes (MDPs). Reinforcement learning agents learn  a policy π_θ, which maps states to a  probability distribution over actions, through interaction with an environment to maximize expected cumulative reward. The interaction takes place in episodes of length T. The state of the environment at step t is s_t ∈ S, the action the agent takes is a_t ∈ A and the reward it receives is r_t. The trajectory of an agent throughout an episode is denoted by τ=(s_0, a_0, r_1, …, s_T) and the objective function it seeks to maximize is J(π)=𝔼_τ∼ p(τ)[∑_t=1^T γ^t-1 r_t]=𝔼_τ∼ p(τ)[G(τ)], where γ is the MDP's discount factor and G(τ) is the discounted cumulative return attained in an episode of trajectory τ.

Meta Reinforcement Learning: 
An agent's learning algorithm can be viewed as a mapping from the data generated by interacting in an environment, 𝒟, to a policy: θ=f(𝒟), where θ represents the parameters of a parameterized policy π_θ. 
The learning algorithm f itself depends on a variety of decisions. If we parameterize the choice of a subset of this decisions with parameters ϕ, then we can make the dependence explicit, θ=f(𝒟 ; ϕ)= f_ϕ(𝒟).
Meta-RL methods learn ϕ, part of the learning algorithm,  such that it becomes more effective when applied to new tasks <cit.>. 
A common setting for these methods involves a distribution of tasks p(ℳ) and an objective that drives meta-learning to maximize the expected cumulative return an agent attains throughout its lifetime when interacting with sampled tasks ℳ^i ∼ p(ℳ) while learning with f_ϕ. We define an agents lifetime as its whole interaction with an environment (which spans multiple episodes). If the return attained at each episode of interaction with the environment is equally weighted by the objective, then it is described by equation <ref>, but variations with different weightings are possible <cit.>.


    𝒥(ϕ)=𝔼_ℳ^i ∼ p(ℳ)[𝔼_𝒟[∑_τ∈𝒟 G(τ)  |   f_ϕ, ℳ^i]].


The meta-learning process operates at two different levels:
In the inner loop, given a task ℳ^i, the learning algorithm 
f_ϕ learns a task-specific policy π_θ through interaction with the task.
In the outer loop a meta-learning algorithm learns parameters ϕ of f using data from multiple inner loops.

Intrinsic rewards: 
For many RL algorithms, directly maximizing extrinsic rewards can lead to poor exploration of the environment; specially when dealing with sparse-reward environments. The use of intrinsic reward signals, r^i, (that either complement or replace extrinsic rewards r=r^e) has occupied a central role in searching for ways to achieve better exploration and balance it with exploitation <cit.>.


In this work we combine these two approaches by meta-learning an intrinsic reward function. For this purpose, we model the reward function itself as a stochastic agent whose `actions' are the rewards it gives at each step and train it with a standard RL algorithm. 
Our experiments evaluate whether this approach is beneficial  compared to learning with extrinsic
rewards. Additionally, we contrast it against the use of meta-learned advantage function in the training signal. All  experiments are conducted on distributions of continuous control tasks where there is access to shaped extrinsic rewards during meta-learning, but only sparse rewards in the test environments.











§ RELATED WORK



One popular way to optimize for fast task learning is to use a neural network that uses all the data collected up to step t to determine its action distribution at that step. During meta-training, the network is trained to leverage this interaction history to select appropriate actions for the current task. This family of methods was first introduced in <cit.> and <cit.>. 

Instead of meta-training a network to directly output the action taken at each step, another family of methods uses an RL algorithm in the inner loop and learns some component of this inner loop procedure that impacts the final performance. Most work restrict themselves to components that have a differential influence over the policy’s parameters and use meta-gradients for meta-training <cit.>. This framework was introduced in MAML <cit.>, which meta-learns the initial parameters of the policy. Many other components have also been explored <cit.>. Some methods avoid the use of meta-gradients: <cit.> consider first order approximations of meta-gradients , <cit.> uses a value based method, and <cit.> uses an evolutionary algorithm.



Several methods exist for designing intrinsic rewards <cit.>, most of which are heuristic-based. Among these, some approaches aim to reduce the agent's uncertainty about specific  quantities, while others encourage visiting new or varied states. However, these are not the only possibilities. This work studies a different approach, aiming to learn the intrinsic motivation signal <cit.>.

Previous work has learned an intrinsic reward signal using meta-gradients: <cit.> do so in the context of a single task, while <cit.> does this with a distribution of tasks. Additionally, <cit.> explores meta-learning a reward function with a discrete search over a space of programs. Rather than replacing extrinsic rewards, <cit.> learns to shape them into a denser signal. Among these, <cit.> shares the closest algorithmic similarity to our work. The main differences being that they consider a deterministic reward function, train it using meta-gradients, and conduct all experiments in grid world environments. 

Beyond the aforementioned work, several other methods have studied meta-learning different parameterizations of the training signal beyond intrinsic rewards <cit.>.



§ BLACK BOX META-LEARNING INTRINSIC REWARD SIGNALS

This work explores the meta-training of learning signals, which are used to train policies by quantifying their fitness. We consider the inner loop learning algorithm f to be a reinforcement learning algorithm. Instead of training with the standard RL objective, in our approach f uses an objective that is partly determined by a meta-learned neural network. This meta-learned network is itself also trained using RL to maximize the objective defined in equation <ref>. Unlike previous methods that rely on meta-gradients to do so, our approach avoids the computation of second-order gradients. We achieve this by not modelling explicitly the structure of the inner loop learning; instead we let the outer loop treats it as part of the problem's stochasticity. As a result, this approach doesn't require the ability to compute gradients of the policy's parameters with respect to the meta-learned parameters. Due to this considerations we refer to the method as being black box.

We now describe in more detail how we meta-train the intrinsic reward function. The intrinsic reward function is modelled as a stochastic agent π_ϕ^r(r_t^i |𝒟_:t), where 𝒟_:t encodes all the interaction in the MDP up to time-step t. We use a LSTM <cit.> for this purpose. At each step t, the LSTM receives as input the tuple { s_t, a_t , π_θ(a_t | s_t) , r_t^e, r_t-1^i , π_ϕ^r(r_t-1^i|𝒟_:t-1) , d_t } where: π_θ is the policy that is being trained,  r_t^e is the extrinsic reward received from the environment at time-step t ,  d_t ∈{0,1} indicates whether t is the start of a new episode,  r_t-1^i is the intrinsic reward delivered by π_ϕ at time-step t-1 and π_ϕ^r(r_t-1^i|𝒟_:t-1) is its probability density. We train this network with PPO <cit.> to maximize the objective defined in equation <ref>, modified to add per episode discounting (<ref>). Because the network's output influences the inner loop performance, during meta-training it learns to use the history of interaction with the environment to generate outputs that suit the current task. The resulting optimization scheme is similar to that of RL^2 <cit.> but applied to a different space. The data with which it trains comes from running multiple inner loops. For a simplified, high-level overview of the proposed method, refer to the pseudocode in Algorithm <ref>; further implementation details are discussed in later sections. We apply the same framework in section <ref> when learning an advantage function.








As mentioned, a salient feature of our approach compared to previous methods is that it avoids the need to compute second-order gradients by bypassing the explicit modeling of the effect the meta-learned signal has over the policy's parameters. We now discuss some advantages and disadvantages of this black box approach versus the use of meta-gradients. 

The main potential drawback is that, by explicitly considering the role of intrinsic rewards in the inner loop, meta-gradients can provide a less noisy signal for meta-learning. This is discussed in <cit.> in the context of meta-learning policy parameters. By comparison, the proposed method is simpler because it is framed as a standard RL problem and doesn't require the calculation of gradients through an optimization process. Moreover, calculating second-order gradients is a computationally expensive operation compared to first-order gradients, making the black box approach favorable in this regard. Another related advantage is that the method is indifferent to how the inner loop uses the meta-learned component. In particular, it can be used in a non-differentiable manner to affect the choice of actions; whereas meta-gradient methods cannot be applied in such situations.




§ EXPERIMENTAL SETUP



 §.§ Benchmarks


We conduct all our experiments using the MetaWorld benchmarks <cit.>. These consists of distributions of continuous control tasks where a simulated robotic arm must interact with an object to achieve a desired configuration. Variations among tasks can be either non parametric (changes in the class of problem, e.g., opening a window, closing a drawer) or parametric (changes in goal positions or initial positions within the same class of problem):



  * ML1 benchmarks operate within a single class and contain 50 randomly sampled parametric variations for training and another 50 for evaluation.


  * ML10 benchmark consists of a set of 10 classes of problems for training and 5 different ones for evaluation. For each class, 50 sampled parametric variations are considered.


 All tasks share the same 39-dimensional observation space and 4-dimensional action space. Crucially, the agent lacks knowledge of both the task class and the goal position; it must infer these from interaction. All episodes are 500 steps long without the possibility of early termination (even if the task is completed).
Further details of the utilized benchmarks can be found in the supplementary material <ref>.

Availability of Rewards:   It is often easier and more practical to define desired behaviour through sparse rewards (e.g. positive signal upon completion of an objective). However, the absence of frequent rewards makes learning significantly more challenging. This often leads to the manual design of reward signal, which can be a labor-intensive process and increase the risk of reward hacking <cit.>. This work considers a hybrid setting in which there is access to shaped rewards for the training tasks but only to sparse rewards during evaluation. In practice, this means we use shaped rewards during meta-training to compute the objective in equation <ref>, but  exclusively the sparse rewards as inputs to the meta-learned recurrent network. Some prior work that operates in this same setting includes <cit.>. The sparse rewards used are: -0.2 in the last step of episodes where the agent failed, and 1 - 0.7 num. executed steps/T on steps where the agent reaches the goal configuration (and has not done so previously within the episode).



 §.§ Implementation Details

Two different neural network architectures were used in this work: an MLP for the policies and inner loop critics, and an LSTM for the meta-trained networks that output training signals and act as outer loop critics. A learning rate of 3 × 10^-4 was used in the inner loop and 5 × 10^-5 in the outer loop. We used the Adam optimizer, tanh activations, and orthogonal initialization. The outputs of the policies and the training signal generators are modeled as Gaussian distributions, with the policies assuming zero covariance.
During training, each lifetime used only 4000 steps of data for policy learning. Each outer loop update used data from 30 inner loops.

Further implementation details can be found in the supplementary material <ref> and in the project's code[https://github.com/Octavio-Pappalardo/Meta-learning-rewards
Our implementation is available at: https://github.com/Octavio-Pappalardo/Meta-learning-rewards].








 §.§ Evaluation Methodology


During evaluation, the intrinsic reward function was used deterministically by considering the mean of the output Gaussian. In some experiments, making the policy deterministic after in-task training was also found to be beneficial.

The metric used for evaluation is the percentage of episodes in which the agent succeeded (reached a goal configuration).

The values reported in the results section for each algorithm are the average performance obtained from different instances of the method (each trained from scratch with a different seed), along with the standard deviation. For each benchmark, a given seed determines the set of parametric variations used. Five seeds were used for methods that utilize extrinsic rewards, and three for the meta-learning methods; larger numbers were not used due to the high computational demands of meta-RL algorithms and compute constraints.

Furthermore, for each seed, the method's performance on a given set of tasks (train or test set) was obtained as the average performance over multiple evaluation runs. This was done by running the method on each task of the set 10 independent times and averaging the performance across all these runs.
 For example, for the ML10 evaluation set, which has 5 different problem classes, each with 50 parametric variations, a total of 2500 runs were considered to obtain the performance of each seed of a method.





§ EXPERIMENTAL RESULTS

This section presents the main experimental results, divided into two parts. The first part compares training with meta-learned intrinsic rewards to training with extrinsic rewards. The second part investigates the effect of learning a different parameterization of the training signal.



 §.§ Intrinsic vs Extrinsic Rewards

This set of experiments evaluates whether there is benefit to be gained in making an RL agent train with learned intrinsic rewards.
To this end, we compared the success rate of a PPO agent when trained with shaped extrinsic rewards, sparse extrinsic rewards, and intrinsic rewards generated by a meta-learned network (which only has access to the environment's sparse rewards).

Evaluations were conducted over the ML1-reach, ML1-close-door, and ML1-button-press benchmarks, allowing only 4000 steps of learning. Figures <ref> and <ref> illustrate the results.
Both figures demonstrate substantial improvements when training with an intrinsic reward function. Figure <ref> evidences the choice of using an RL algorithm that trains with batches of 4 episodes of data (<ref>). It also reflects that making the policy deterministic after training can improve performance in some environments. Figure <ref> shows that there is no decline in performance when moving from the tasks in the training set to the test set. While this is expected when training with extrinsic rewards (since no meta-learned component is applied), the fact that the same behavior occurs when training with intrinsic rewards indicates that the learned reward network effectively generalizes to new, unseen environments.

However, the observed benefits do not come without costs. The method using intrinsic rewards required having a prior meta-learning phase, which can be computationally costly and is contingent upon having access to training tasks that share structure with those in the test set.

As mentioned, using intrinsic rewards significantly improves performance compared to both types of extrinsic reward signals. It is worth noting that the fairer comparison in the considered evaluation setting, where the meta-learned network does not have access to shaped rewards, is against training with the sparse extrinsic rewards (which showed little to no progress). This makes the gains more notable. Finally, outperforming the shaped extrinsic rewards suggests that learning rewards can also be used for the design or improvement of the shaped reward signals that standard RL benchmarks  typically use <cit.>.














 §.§ Intrinsic Rewards vs Learned Advantages



In this section, we consider whether intrinsic rewards are the right component to meta-learn and we discuss some other options. Focusing on intrinsic rewards is appealing for at least two reasons: 1) as mentioned in section <ref>, the use of intrinsic rewards is a well-studied topic in RL and has consistently shown benefits, 2) all standard RL algorithms assume they receive rewards; thus, meta-learned rewards can be directly integrated into any of them while maintaining the algorithm's structure.

As discussed in section <ref>, it is possible to meta-learn several other parameterizations of the RL objective. In this section, we explore  another such parameterization by meta-learning an advantage function. Instead of learning to assign partial credit to a transition and its preceding transitions, the network learns to evaluate each transition's goodness independently. This set of experiments also introduces non-parametric variations among environments. Results are shown in figure <ref>.






Both methods exhibit similar qualitative behaviour. For the benchmarks considered, using the learned advantage function shows some benefits, which are statistically significant only for ML1-button-press and for the training tasks of ML10. While both methods show good generalization when facing parametric variations, they achieve a very low success rate on the test tasks of ML10 (although some performance improvement occurs with updates).



Meta-learning to predict the training signal can be complementary to learning other components of the inner loop. Training signals hold knowledge on the goodness of observed behaviours <cit.>. Other components can address different loci of knowledge and provide additional benefits. In particular, the most popular approach in meta-learning literature is to meta-learn parameters of a policy. In the supplementary material we show the performance of two such methods when applied to our experimental setting.
One advantage of meta-learning part of the objective function is its applicability to broad distributions of environments and agents. It has also shown to be adequate for the many-shot meta RL setting <cit.>.
Determining the best combination of components to meta-learn remains an open problem and likely to depend on the specific setting.





§ CONCLUSIONS


This paper presented a meta-learned stochastic intrinsic reward function modeled as a recurrent network and trained analogously to a standard reinforcement learning agent. The network was trained through interactions with a distribution of training tasks and evaluated in new, sparse-reward environments. Our experiments demonstrated that a policy that trains using these intrinsic rewards significantly accelerate it's learning compared to one using extrinsic rewards, whether shaped or sparse. The learned reward signal demonstrated effective generalization across parametric task variations, though this was not the case when confronting generalization among a small set of distinct problem types. Additionally, we investigated alternatively learning a policy using a meta-learned advantage function, achieving slightly better results.
Throughout the paper, we discussed the benefits and drawbacks of different features of our approach.

Potential future directions that stem from this work include: conducting a quantitative analysis comparing our black box approach with the use of meta-gradients for the outer loop, extending the method to longer lifetimes and broader task distributions, combining or comparing with the meta-learning of other components, and applying it in settings where only sparse rewards are available during meta-training.
Some straightforward avenues for improvement to our approach include using a network that has access to future interaction steps within the batch of collected data for generating rewards, and computing rewards for off-policy data to enable data reuse.













§ APPENDIX EXPERIMENTAL SETUP


Section <ref> gives more information on the Meta World benchmarks involved in the study. Section <ref> sections provide more detail on the algorithms implementations. 





 §.§ Benchmarks

This section gives more information on the utilized benchmarks <cit.>. They all are from version 2 of MetaWorld.



Observation and action space:
The 4-dimensional action space contains 3 dimensions for the change in 3D space of the robotic arm's end-effector, plus 1 dimension for the gripper's normalized torque. All dimensions are bounded to the range [-1,1]. The 39-dimensional observation space consists of the 3D Cartesian position of the end-effector, a normalized measurement of how
open the gripper is, the 3D position and quaternion of the first and second object of interest, all this same information from the previous time step, and finally the 3D position of the goal. In the benchmarks we utilized, the  3D position of the goal is zeroed out which forces the agent to learn to recognize and adapt to the task through interaction. For tasks where there isn't a second object of interest this information is also zeroed out.



ML1 benchmarks:
The choice of these benchmarks over others in the ML1 category was arbitrary except for the condition that the agent trained with shaped extrinsic rewards exhibited some progress during its lifetime.

    
  * `ML1 reach' Reach a goal position. Randomize the goal positions.
    
  * `ML1 close door' Close a door with a revolving joint. Randomize door positions.
    
  * `ML1 button press'  Press a button. Randomize button positions




ML10 benchmark:

Training environments


    
  * insert peg side: Insert a peg sideways. Randomize peg and goal positions.
    
  * reach-v2: Reach a goal position. Randomize the goal positions.
    
  * door-open-v2: Open a door with a revolving joint. Randomize door positions.
    
  * basketball-v2: Dunk the basketball into the basket. Randomize basketball and basket positions.
    
  * open window: Push and open a window. Randomize window positions.
    
  * pick & place: Pick and place a puck to a goal. Randomize puck and goal positions.
    
  * button-press-topdown-v2: Press a button from the top. Randomize button positions.
    
  * push-v2: Push the puck to a goal. Randomize puck and goal positions.
    
  * close drawer: Push and close a drawer. Randomize the drawer positions.
    
  * sweep-v2: Sweep a puck off the table. Randomize puck positions.

Test environments

    
  * door-close-v2: Close a door with a revolving joint. Randomize door positions.
    
  * place onto shelf: Pick and place a puck onto a shelf. Randomize puck and shelf positions.
    
  * drawer-open-v2: Open a drawer. Randomize drawer positions.
    
  * sweep into hole: Sweep a puck into a hole. Randomize puck positions.
    
  * pull lever: Pull a lever down 90 degrees. Randomize lever positions.











 §.§ Implementation Details


In section <ref> tables with the values of different hyperparameters for each method are presented. Their nomenclature matches that used in the project's code for easier reproducibility. Then, in section <ref> we give a discussion on the influence of various hyperparameters, and provide some additional information.




  §.§.§ Hyperparameter Tables




PPO agent/ Inner loop

Hyperparameters for the inner loop PPO agent. The hyperparameters are the same for all variations of training signals: extrinsic shaped rewards, extrinsic sparse rewards, intrinsic learned rewards, and learned advantage function.



|p4.5cm|p4cm|
Hyperparameter     Value 

Hyperparameter     Value 

Table with the hyperparameter configuration used for task adaptation. Valid both for when the PPO agent was runned on its own. as well as when it used meta-learned training signals. hyperparameters were also the same during meta-training phases. 


      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      






Learned Intrinsic rewards and Advantage function/ Outer loop



|p5.5cm|p4.5cm|
Hyperparameter     Value 

Hyperparameter     Value 

Hyperparameter configuration for training and evaluating the meta-agents providing intrinsic rewards and advantages.  



      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

     Intrinsic r.= Advantages=  

      

      

     "0" ≈( /) 

      

      

         

      

      

      

      








  §.§.§ Hyperparameter Explanations and Further Implementation Details



Architectures:
For the policy and critic we used a (64,64) MLP architecture. For the meta-learned networks we used an LSTM with a hidden dimension of 128. Prior to going into the LSTM, the current step's input was processed by two linear(plus activation) layers (128,32) at which all 1 dimensional inputs were concatenated. The LSTM hidden states were processed by a (512) layer for the outer loop critic, a (128) layer for the gaussian distributions standard deviation, and (128,128) layers with a final arctan activation for its mean. All other activations were tanh. When using the learned advantage function the critic was omitted from the inner loop. All implementations where done in PyTorch.

Hyperparameter details:
Several of the hyperparameters showed in section <ref> are standard in deep RL literature. This section briefly describes those whose influence may be less clear.



 
  * total_timesteps refers to the total number of steps collected in inner loops. The length of lifetimes.  num_steps is the batch size used for PPO updates. Two updates are done during an inner loop.
 
 
  * num_epsiodes_of_validation and num_lifetimes_for_validation control how many of the last lifetimes and episodes whithin them are used to validate performance during training. Meta-learning took place until no appreciable performance improvement was observed for at least 200 outer loop updates.


  * Throughout meta-training, extrinsic rewards obtained in each inner loop were normalized to keep their average value around e_rewards_target_mean. This normalization was done at the level of each environment class. In the ML10 benchmark training, independent averages of extrinsic rewards for each of the 10 problem classes were maintained. This normalization ensures that the meta agent's training signals (extrinsic rewards) stay within the same range during training and assigns similar importance to improvements across different environment classes.


  * Hyperparameters starting with ae and meta_gamma control which variant of Objective <ref> is used and how advantages are estimated for outer loop updates. The most conceptually important are ae.estimation_method and meta_gamma. ae.estimation_method controls whether discounts are applied at the episode or step level and which method is used for estimating advantages. Specifically, `bootstrapping skipping uninfluenced future rewards’ uses episode-wise discounts and estimates advantages with an exponential combination of n-step estimates. This method ignores all extrinsic rewards received by the agent until the generated training signal is used for an update. meta_gamma indicates the discount factor for outer loop learning, either step-wise or episodic. 


  * initial_std controls the initial standard deviation of training signals when meta-training.


  * Hyperparameters starting with ppo for meta-learning have similar but not identical to the standard PPO  hyperparameters. PPO updates for the outer loops use truncated backpropagation. Specifically, ppo.k controls the number of steps before gradient propagation is truncated. ppo.num_minibatches controls the number of processed k-length sequences before a gradient step is taken. The value "0" in the tables indicates a gradient step is taken for each num_inner_loops k-length sequences.


  * Hyperparameters whose values were not mentioned in the tables took the default value assigned in Pytorch.


Some of the values that we found to impact performance the most are: initial_std , entropy_coef , e_rewards_target_mean,num_inner_loops_per_update  and learning rates. Ray was used to run the different inner loops corresponding to an outer loop update in parallel across different processes.















§ APPENDIX EXPERIMENTAL RESULTS




 §.§ Meta-learned Policy Parameters


In this section we show results for two popular meta-learning algorithms whose focus is on policy parameters. MAML (its first order version) <cit.> which meta-learns the initial parameters of a policy and RL^2 <cit.> which meta-learns a recurrent policy that conditions on all past data. They are applied in the same setting described in section <ref>. Results can be found in figure <ref>. It stands out that directly meta-learning policy parameters can lead to informed behaviour from the start of the interaction. MAML showed the best performance of the two methods. Furthermore it attained some generalization when facing non parametric variations. Both methods still have space for improvement and they weren't able to make use of data collected through interaction across episodes to adapt their behaviour (except for MAML in ML1-reach). Further information on their implementation can be found on the work's code repository.





















 Paper Title: SmileyNet -- Towards the Prediction of the Lottery by Reading Tea Leaves with AI.




.jpg,.pdf,.mps,.png

The Bulk Densities of Small Solar System Bodies as a Probe of Planetesimal Formation
    [
    August 5, 2024
====================================================================================



§ ABSTRACT 
We introduce SmileyNet, a novel neural network with psychic abilities. It is inspired by the fact that a positive mood can lead to improved cognitive capabilities including classification tasks. The network is hence presented in a first phase with smileys and an encouraging loss function is defined to bias it into a good mood. SmileyNet is then used to forecast the flipping of a coin based on an established method of Tasseology, namely by reading tea leaves. Training and testing in this second phase are done with a high-fidelity simulation based on real-world pixels sampled from a professional tea-reading cup. SmileyNet has an amazing accuracy of 72% to correctly predict the flip of a coin. Resnet-34, respectively YOLOv5 achieve only 49%, respectively 53%. It is then shown how multiple SmileyNets can be combined to win the lottery.     






§ INTRODUCTION

Modern AI is one of the greatest if not the greatest scientific breakthrough of all times. The publication of the structure of DNA by Watson and Crick in Nature <cit.> has only achieved slightly more than 19,000 citations by summer 2024 according to GoogleScholar. And it has taken 71 years. AI <cit.> has led to many important papers, which received many more citations within a much shorter time-frame; see, e.g., Tab. <ref> for a few examples based on citations according to GoogleScholar.



Given the relevance of AI for our modern world, it is surprising that there are areas of high economic relevance where it has not been applied, yet. We explore here - up to our knowledge for the very first time - the potential of AI for the psychic services industry, which is a multi-billion dollar business. This industry is estimated to have reached a revenue of well above $2 billion in the USA in 2023. And it experiences a steady further growth. This holds especially for psychic online services that managed an exponential growth in recent years according to market-reports.       

We suspect that AI may already be used in some real-world psychic services, especially in the online branches. But the methods and the results are not academically published up to our knowledge - likely due to the large economic potential they possess. While this is somewhat understandable, we believe in Open Science <cit.> and we share our insights due to the large benefits Psychic AI can provide to humanity.

A first medium-term goal is to win the lottery to further fund research in the novel field of Psychic AI. A well-established method of Tasseology (also known as Tasseomancy or Tasseography) <cit.> is used to this end, namely reading tea leaves <cit.>. In the spirit of divide and conquer, the simpler problem of predicting the outcome of flipping a coin is tackled first. We then show how this can be used to predict the numbers for winning the lottery jackpot. 

As an important basic research result, it is proven in this paper, up to our knowledge for the very first time, that a neural network can possess psychic powers. Concretely, it is shown that the novel SmileyNet can predict the flipping of a coin with a precision well above chance. 




     


§ SMILEYNET


SmileyNet is a novel bio-inspired network. It is motivated by the fact that cognitive processes can be positively influenced by a positive mood 
<cit.>.
This includes especially also classification tasks <cit.>. 



Furthermore, it is well-known that visual stimuli are well suited to elicit emotions <cit.>. This includes in particular a smiling face <cit.>. We hence first present the network with smileys as input to bias it into a good mood. More precisely, the Unicode emoticon "U+1F642" for the classical smiley, officially known as "Slightly Smiling Face" is used to this end (Fig.<ref>).

But stress is a negative factor for cognition <cit.>. We do not want to loose the positive effects of the good mood by putting the network under stress in the smiley training phase. Hence, it gets positive feedback <cit.> that it is doing fine, i.e., that it is doing nothing wrong.

More precisely, the training in the smileyfication phase is done as follows. A quadratic loss function L_smile() is used. To avoid putting the network under stress and to give a further positive endorsement, the expected output ŷ is set to be the actual output y whenever a smiley is presented to the net, i.e., ŷ = f_I(y) where f_I() is the identity function. So,


    L_smile()= 1/2 (y - ŷ)^2  with ŷ = f_I(y)


This method of smileyfication can be applied to any network N_old to turn it into a novel one N_new, which is expected to perform better than the previous N_old. 

Often, there may only be small improvements and it can happen that multiple trials are needed until some improvement is seen for the first time, which should then be immediately documented. Also, it can be useful to take additional factors into account. For example, the effects of smileyfication may only be seen on a sunny day as, e.g., the weather has an influence on the mood <cit.>.  



§ RELATED WORK


YOLOv5 <cit.> and Resnet34 <cit.> are well-known networks for image classification. They are used for comparison to SmileyNet in Sec. <ref>. There are also many other networks. But as YOLOv5 and Resnet32 are very popular, we concentrate here on those two. 





YOLOv5 is the fifth iteration of the revolutionary "You Only Look Once" object detection model. It is designed to deliver high-speed, high-accuracy results in real-time. Built on PyTorch, this powerful deep learning framework has garnered immense popularity for its versatility, ease of use, and high performance. Fig. <ref> shows how it works.


ResNet-34 is a convolutional neural network in PyTorch. It is has 34 layers. It addresses the problem of vanishing gradient. This is done with the identity shortcut connection that skips one or more layers. 
Fig. <ref> shows how it works.




§ HIGH-FIDELITY SIMULATION


The constellation of tea leaves and the predicted event are linked by cosmic energy. We use this fact to generate high-fidelity training data. This is not easy in the context of psychic powers and it may be one reason why AI has not been used - up to our knowledge - in this economically highly relevant business up to now.

The core idea of our approach for training data is to use the reverse direction of the link. The coin is flipped in a deterministic way, i.e., it is so to say turned by hand in rounds r from one side to the other. The related constellation of tea-leaves, i.e., the image I_r is then generated afterwards. 

In the software implementation, "heads" is presented with a "1" and "tails" with a "0" for the sake of convenience and wlog. Also wlog, we start with "tails" facing up as initialization of the flipping process. In the very first round r=1, the coin is turned and it hence shows "heads". The coin is then turned again and it now shows "tails" in round r=2, and so on. More formally, it holds for the outcome o_flip(r) in round r that


    o_flip(r) = 
    heads    if  (r+1)  mod  2 = 0
    tails    otherwise



The images I_r in our high-fidelity simulation are based on a real-world image (Fig. <ref>)[wikimedia.org; curid=17659832; user: MochaSwirl; Public Domain] of tea-leaves in a cup used by a fortune-telling professional. From this professional image, real-world pixels p_tea belonging to tea and real-world pixels p_cup belonging to the cup are extracted. This forms the base image I_0 with 100 × 100 pixels (Fig. <ref>). For the sake of simplicity, p_tea-pixels are shown in black and white is used for p_cup-pixels . 




The first image I_1 is then generated right after the coin was flipped for the first time. So, I_1 is linked to producing "head" according to the laws of Tasseology. In general, each image I_r is generated from I_r-1 by randomly selecting k_change coordinates [x,y]_j : x,y ∈{0,...,99} that are altered as follows 


    I_r[x][y] = 
    
        p_tea   if  I_r-1[x][y] = p_cup
    
        p_cup   if  I_r-1[x][y] = p_tea











Each I_r is generated right after doing the flip of the coin in round r, i.e., it is linked to the outcome o_flip(r) of flipping the coin in that round. According to our experience, small to medium values for k_change work best. The process is illustrated in Fig. <ref>. As mentioned, the p_cup, respectively p_tea pixel are shown as white, respectively black pixel in Fig. <ref> for the sake of simplicity. 



§ EXPERIMENTS AND RESULTS

  
The high-fidelity simulation presented in Sec. <ref> is used to generate 500 images. When it is needed by the network, the images are scaled to fit the requirements of PyTorch. The set is split into the first 300 images that are used for training and the following 200 images are used for validation. 


 


Tab. <ref> shows the results for YOLOv5 and Resnet-34. Resnet-34 has only 98 out of 200 cases correctly predicted. YOLOv5 does a bit better and it manages 106 correct predictions on the testing set.

Due to the better performance of YOLOv5 compared to Resnet-34, it is used for a smileyfication to generate SmileyNet version 1. As SmileyNet is expected to have an improved performance, it is trained with less data. Only every fifth image from the data-set is used. Note that this is still an alternating sequence of "heads" and "tails", i.e., it is exactly the same principle as before, but just 100 training images. The remaining 400 images are used for testing.

SmileyNet manages an amazing accuracy of 71.75%, i.e., 287 out of 400 cases are correctly predicted. This is well above the performance of YOLOv5 and Resnet-34 (Tab. <ref>). Most importantly, it is well above chance and we therefore prove - up to our knowledge - for the very first time that AI can possess psychic powers. Note that it can be shown that the improvement is statistically significant, i.e., it is a thorough and sound result.       

Tab. <ref> summarizes the performance gains compared to YOLOv5 and Resnet-34. SmileyNet has an accuracy of 71.75%, i.e., there is an increase by 35.38% from the 53% of YOLOv5. The performance gain is in the comparison with Resnet-34 with 46.43% even higher. SmileyNet requires only 100 training images while YOLOv5 and Resnet-34 had 300 images. So, there is a reduction by 200 images or 66% in comparison to the other two nets. 



§ SMILEYNET FOR LOTTERY FORECASTS



The forecasting of throwing a coin can already be a useful psychic power. But the forecast of lotteries has a very straightforward economical potential. Lotteries exist in most countries around the globe with high jackpots for the correctly predicted numbers. It is shown in the following with the example of the German lottery how SmileyNet can be used to this end.

In the German lottery, six numbers win_1,...,win_6 are drawn out of a pool with the numbers 1,...,49. The psychic power of SmileyNet can forecast the flip of a coin, i.e., 1 bit. We therefore simply combine multiple SmileyNets to encode the numbers 1,...,49. Then, one after the other win_i is predicted. 

More precisely, let b_j denote the forecasting output of SmileyNet N_j and ⟨ b_1,...,b_n ⟩_2 denotes the number x encoded by the bits b_1,...,b_n, i.e.,

    x = ⟨ b_1,...,b_n ⟩_2 = ∑_i=1^nb_i · 2^n-i


The i-th number win_i of a lottery draw is then predicted with ten SmileyNets N_j with outputs b_j as follows 

    win_i = ⟨ b_1,...,b_5 ⟩_2 + ⟨ b_6,...,b_9 ⟩_2 + ⟨ b_10⟩_2


As mentioned, the winning numbers win_1,...,win_6 are simply generated one after the other. It must of course be ensured that fresh images of tea-leaves from a just emptied cup are provided in every round. Nonetheless, SmileyNet does not have perfect psychic power. It may hence happen in rare occasions that a number win_j+k has been predicted before as win_j. But this is not a problem. Then, the current win_j+k is simply discarded and an other round is added to determine a new win_j+k.

Given the predictive power of SmileyNet to correctly forecast a bit with p_ N=0.7175, the exact likelihood p_win of forecasting the German lottery with the above method can be computed as

    p_win   = p_ N^5 ·p_ N^4 · p_ N
       = 0.03615922 
       = 3.615922 · 10^-2
        
This likelihood may look small at first glance, but it is by six orders of magnitude better than just chance with

    p_chance   = 49  6^-1
       = (49!/6!(49-6)!)^-1
       = 1/13,983,816
       = 7.15112 · 10^-8
       ≪ p_win
        

This is an important result that demonstrates the huge potential of the novel field of Psychic AI. Note that it is a thorough result as it can be shown that it is statistically significant.

But before we apply these results in practice, we still want to improve the performance of our approach. This holds first and foremost for the likelihood p_win of predicting the correct numbers for winning the jackpot:

	
  * One option is simply more training data and more computation power for a larger network to turn SmileyNet into a foundation model for psychic AI. 

  * Furthermore, we expect a transformer to be more accurate given that the drawing of lottery numbers is a weekly event in Germany. This is left for future work.

Also, a reduction of the required number of SmileyNets would be nice before applying it in practice as it is a bit inconvenient to drink so much tea. 



§ CONCLUSIONS


SmileyNet version 1 was introduced. It is based on smileyfication, i.e., the fact that a positive mood can have a positive influence on cognitive functions including classification. It was shown how SmileyNet can be used to predict the lottery by reading tea-leaves. First, the flipping of a coin was predicted with a very good accuracy, namely 72%. State of the art networks like YOLOv5 and Resnet-34 perform much worse. Second, it was shown how multiple SmileyNets can be combined for lottery forecasts. In addition to the potential practical implications, it is a major breakthrough to show that AI can have psychic powers.  



§ ACKNOWLEDGMENTS

We thank our anonymous reviewers for their feedback. Special thanks go to reviewer 2, who pointed out that the drawing of lottery numbers tends to be a weekly or at least regular event, and that the use of a transformer should be beneficial. 




§ EPILOG

This is a satirical accumulation of misconceptions, mistakes, and flawed reasoning I have encountered in recent times as a reviewer and sometimes even as a reader of published papers. I hope it is entertaining and useful in the context of the education of BSc, MSc, and PhD students in Machine Learning, Artificial Intelligence, and Cognitive Science. 

For an easy start, here a few questions to reflect upon. First and foremost, there is of course the following puzzle: 

	
  * Where does the "magic" hide that is presented here? How can SmileyNet "predict" the flip of a coin better than chance? 

As a small hint: there is a single conceptual mistake that explains it. 

There are also many other bugs, respectively misconceptions and flawed reasoning in this paper. Here some further food for thought, to hopefully teach a good lesson by setting a bad example:   

	
  * Can AI beat natural laws? What are the pros and cons of a detailed pre-analysis of the possible information content of the data?
	
  * How much academic novelty is in SmileyNet? How much does it differ from the net it is build upon? 
	
  * How do we prevent smileyfication in our own work? How can we properly show that an "extension" or "modification" of an existing approach leads to a substantial improvement and study the related trade-offs?  
	
  * What is a proper assessment of the state of the art? How do we find, understand, and discuss related work?
	
  * What is a proper statistical performance analysis? What is statistical significance and how should it be interpreted?
	
  * When and how do I expect my own work to be of relevance? Why do I publish it? 
	
  * What are Artificial Intelligence and Machine Learning? Which sub-areas within these disciplines exist? What are commonly used methodologies within the sub-areas? 
		
  * What is the role of training data in supervised machine learning? What are the pros and cons of simulation?
	
  * What is Open Science? What are its principles? What does it imply for the quality of the underlying work? What are the pros and cons if others, respectively oneself adheres to it? 
	
  * What is a bio-inspired design? How can insights from Cognitive Science be used and which pitfalls exist?
	
  * What are the pros and cons of bibliometrics? How can they help us to identify relevant work? And where are their constraints and limitations?

This list is a bit concentrated on the "big questions". It is of course also possible to just concentrate on the (many) methodological mistakes in the SmileyNet paper. So, the above list is neither meant to be exhaustive nor to be without dispensable aspects depending on the context. It is just a possible starting point for using the SmileyNet in the class-room, or to just meditate about it on your own, or... 

Enjoy and have fun...

10

url@samestyle








WatsonCrick-DNA-Nature1953
J. D. Watson and F. H. C. Crick, “Molecular structure of nucleic acids: A
  structure for deoxyribose nucleic acid,” Nature, vol. 171, no. 4356,
  pp. 737–738, 1953. [Online]. Available:
  <https://doi.org/10.1038/171737a0>


DeepLearning-Hinton-Nature15
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521,
  no. 7553, pp. 436–444, 2015. [Online]. Available:
  <https://doi.org/10.1038/nature14539>


DeepLearning-Book-MITpress16
I. Goodfellow, Y. Bengio, and A. Courville, Deep learning.1em
  plus 0.5em minus 0.4emMIT press, 2016.

DeepLearning-ResNet-CVPR16
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
  recognition,” in IEEE Conference on Computer Vision and Pattern
  Recognition (CVPR), 2016, Conference Proceedings, pp. 770–778.

Adam-StochasticOptimization-arXiv2014
J. B. Diederik P Kingma, “Adam: A method for stochastic optimization,”
  arXiv preprint, vol. arXiv:1412.6980, 2014.

DeepLearning-AlexNet-NIPS12
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with
  deep convolutional neural networks,” pp. 1106–1114, 2012. [Online].
  Available:
  <http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks>


Breimann-RandomForests-ML01
L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp.
  5–32, 2001. [Online]. Available:
  <https://doi.org/10.1023/A:1010933404324>


DeepCNN-LargeScale-ImageRecognition-2014
K. Simonyan and A. Zisserman, “Very deep convolutional networks for
  large-scale image recognition,” arXiv preprint, vol. arXiv:1409.1556,
  2014.

GAN-NIPS2014
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair,
  A. Courville, and Y. Bengio, “Generative adversarial nets,” 2014. [Online].
  Available:
  <https://proceedings.neurips.cc/paper_files/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf>


FasterR-CNN-NIPS2015
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time
  object detection with region proposal networks,” 2015. [Online]. Available:
  <https://proceedings.neurips.cc/paper_files/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf>


ImageNet-CVPR09
J. Deng, W. Dong, R. Socher, L. J. Li, L. Kai, and F.-F. Li, “Imagenet: A
  large-scale hierarchical image database,” in IEEE Conference on
  Computer Vision and Pattern Recognition (CVPR), 2009, Conference
  Proceedings, pp. 248–255.

DeepLearning-GoogLeNet-CVPR15
C. Szegedy, L. Wei, J. Yangqing, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
  V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
  2015, Conference Proceedings, pp. 1–9.

UNESCO-RecommendationOpenScience2021
UNESCO, “Unesco recommendation on open science,” 2021. [Online]. Available:
  <https://doi.org/10.54677/MNMH8546>


Tasseology-2008
R. E. Guiley, “Tasseomancy,” in The Encyclopedia of Witches, Witchcraft
  and Wicca.1em plus 0.5em minus 0.4emCheckmark Books, 2008.

Tasseology-TeaLeaves-2016
K. Henriott-Jauw, Tea and Tasseomancy: A Guide to Reading Tea
  Leaves.1em plus 0.5em minus 0.4emCreateSpace Independent
  Publishing Platform, 2016.

Tasseology-TeaLeaves-2014
A. O'Reilly, Tea with Annie, a divinitive guide to Tasseomancy and its
  artistry.1em plus 0.5em minus 0.4emMelbourne, Australia:
  Whiteslaw Press, 2014.

PositiveMood-CreativeProblemSolving-Sage2015
J. Politis and J. C. Houtz, “Effects of positive mood on generative and
  evaluative thinking in creative problem solving,” Sage Open, vol. 5,
  no. 2, p. 2158244015592679, 2015. [Online]. Available:
  <https://journals.sagepub.com/doi/abs/10.1177/2158244015592679>


PositiveMood-CognitiveControl-2011
E. A. Martin and J. G. Kerns, “The influence of positive mood on different
  aspects of cognitive control,” Cognition and Emotion, vol. 25, no. 2,
  pp. 265–279, 2011. [Online]. Available:
  <https://doi.org/10.1080/02699931.2010.491652>


PositiveMood-Cognition-Classification-2010
R. T. Nadler, R. Rabi, and J. P. Minda, “Better mood and better
  performance:learning rule-described categories is enhanced by positive
  mood,” Psychological Science, vol. 21, no. 12, pp. 1770–1776, 2010.
  [Online]. Available:
  <https://journals.sagepub.com/doi/abs/10.1177/0956797610387441>


ImagesMoodSurvey2016
M. K. Uhrig, N. Trautmann, U. Baumgärtner, R.-D. Treede, F. Henrich,
  W. Hiller, and S. Marschall, “Emotion elicitation: A comparison of pictures
  and films,” Frontiers in Psychology, vol. 7, 2016. [Online].
  Available:
  <https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2016.00180>


ImagesMoodSurvey2008
J. K. Olofsson, S. Nordin, H. Sequeira, and J. Polich, “Affective picture
  processing: An integrative review of erp findings,” Biological
  Psychology, vol. 77, no. 3, pp. 247–265, 2008. [Online]. Available:
  <https://www.sciencedirect.com/science/article/pii/S0301051107001913>


MoodSmilingFace2019
J. Nikitin and A. M. Freund, “The motivational power of the happy face,”
  Brain Sciences, vol. 9, no. 1, p. 6, 2019. [Online]. Available:
  <https://www.mdpi.com/2076-3425/9/1/6>


CognitionStressNegative2024
G. Picciotto and R. A. Fabio, “Does stress induction affect cognitive
  performance or avoidance of cognitive effort?” Stress and Health,
  vol. 40, no. 1, p. e3280, 2024. [Online]. Available:
  <https://onlinelibrary.wiley.com/doi/abs/10.1002/smi.3280>


CognitionStressNegative2023
E. J. Kim and J. J. Kim, “Neurocognitive effects of stress: a metaparadigm
  perspective,” Molecular Psychiatry, vol. 28, no. 7, pp. 2750–2763,
  2023. [Online]. Available: <https://doi.org/10.1038/s41380-023-01986-4>


StressCognitionFeedback2020
M. Paul, C. Bellebaum, M. Ghio, B. Suchan, and O. T. Wolf, “Stress effects on
  learning and feedback-related neural activity depend on feedback delay,”
  Psychophysiology, vol. 57, no. 2, p. e13471, 2020. [Online].
  Available: <https://onlinelibrary.wiley.com/doi/abs/10.1111/psyp.13471>


MoodWeather2008
J. J. A. Denissen, L. Butalid, L. Penke, and M. A. G. van Aken, “The effects
  of weather on daily mood: A multilevel approach,” Emotion, vol. 8,
  no. 5, pp. 662–667, 2008.

MoodWeather1984
E. Howarth and M. S. Hoffman, “A multidimensional approach to the relationship
  between mood and weather,” British Journal of Psychology, vol. 75,
  no. 1, pp. 15–23, 1984. [Online]. Available:
  <https://bpspsychub.onlinelibrary.wiley.com/doi/abs/10.1111/j.2044-8295.1984.tb02785.x>


YOLOv5-2021
Ultralytics, “YOLOv5: A state-of-the-art real-time object detection
  system,” 2021. [Online]. Available: <https://docs.ultralytics.com>


YOLOv5-architecture-2021
——, “Ultralytics YOLOv5 architecture,” 2021. [Online]. Available:
  <https://docs.ultralytics.com/yolov5/tutorials/architecture_description/>






 Paper Title: Granular-Balls based Fuzzy Twin Support Vector Machine for Classification.














**
Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals








































Granular-Balls based Fuzzy Twin Support Vector Machine for Classification
    Lixi Zhao, Weiping Ding, Senior Member, IEEE, Duoqian Miao, Guangming Lang
Lixi Zhao and Guangming Lang are with the School of Mathematics and Statistics, Changsha University of Science and Technology, Changsha, Hunan 410114, China; Weiping Ding is with the School of Information Science and Technology, Nantong University, Nantong, Jiangsu 226019, China; Duoqian Miao is with the Department of Computer Science and Technology, Tongji University, Shanghai, 201804, China (email: zhaolixi0915@163.com, langguangming1984@126.com, dwp9988@163.com, dqmiao@tongji.edu.cn; Corresponding author: Guangming Lang).

    August 5, 2024
==================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





§ ABSTRACT 
The twin support vector machine (TWSVM) classifier has attracted increasing attention because of its low computational complexity. However, its performance tends to degrade when samples are affected by noise. The granular-ball fuzzy support vector machine (GBFSVM) classifier partly alleviates the adverse effects of noise, but it relies solely on the distance between the granular-ball's center and the class center to design the granular-ball membership function.
In this paper, we first introduce the granular-ball twin support vector machine (GBTWSVM) classifier, which integrates granular-ball computing (GBC) with the twin support vector machine (TWSVM) classifier. By replacing traditional point inputs with granular-balls, we demonstrate how to derive a pair of non-parallel hyperplanes for the GBTWSVM classifier by solving a quadratic programming problem. Subsequently, we design the membership and non-membership functions of granular-balls using Pythagorean fuzzy sets to differentiate the contributions of granular-balls in various regions. Additionally, we develop the granular-ball fuzzy twin support vector machine (GBFTSVM) classifier by incorporating GBC with the fuzzy twin support vector machine (FTSVM) classifier. We demonstrate how to derive a pair of non-parallel hyperplanes for the GBFTSVM classifier by solving a quadratic programming problem. We also design algorithms for the GBTSVM classifier and the GBFTSVM classifier. Finally, the superior classification performance of the GBTWSVM classifier and the GBFTSVM classifier on 20 benchmark datasets underscores their scalability, efficiency, and robustness in tackling classification tasks.



Twin support vector machine, Fuzzy support vector machine, Granular-ball computing, Granular-ball rough sets, Pythagorean fuzzy sets.







§ INTRODUCTION

Support Vector Machine (SVM), developed by Vapnik in 1995 <cit.>, is a machine learning technique for both classification and regression tasks. It is grounded in statistical learning theory and structural risk minimization theory. Up to now, SVM and its variants have shown unique advantages in analyzing high-dimensional and nonlinear datasets <cit.>.
Especially, TWSVM generates two non-parallel hyperplanes instead of a single hyperplane for classification <cit.>. It ensures that each sample point is close to one of the two hyperplanes and far away from the other hyperplane. For classification tasks, a new sample is assigned to the nearest hyperplane class. Moreover, TWSVM determines the classification hyperplane by solving two small-scale quadratic programming problems instead of one large-scale quadratic programming problem. It not only retains the advantage of SVM in handling  high-dimensional and nonlinear classification and regression problems but also achieves a training speed theoretically four times faster than SVM <cit.>.
However, when samples are contaminated by noise, the performance of SVM and its variants will deteriorate <cit.>. To reduce the uncertainty caused by outliers and noise, researchers have assigned a membership degree to each sample based on its confidence level with respect to the native class, and proposed the fuzzy support vector machine (FSVM) classifier <cit.>. While the FSVM classifier improves the robustness, its membership function only considers the distance between the sample and the class center, and leads to confusing support vectors positioned far from the class center as noise.
In response to this problem, numerous scholars have redefined the membership function <cit.>. For instance, Fan et al. defined an entropy-based fuzzy membership function, and proposed the entropy-based fuzzy support vector machine (EFSVM) classifier <cit.>. Rezvani, Wang, and Pourpanah provided the intuitionistic fuzzy twin support vector machine (IFTSVM) classifier by integrating the intuitionistic fuzzy sets with the TWSVM classifier <cit.>.
These studies not only reduce the impact of noise on classification tasks but also distinguish support vectors from noise.


Granular Computing (GC) is a computational paradigm that utilizes information granulation to process imprecise, inaccurate, and incomplete datasets. In 1979, Zadeh took vast amounts of information to achieve intelligent systems and controllers by leveraging the principles of GC <cit.>. Since then, numerous scholars have integrated granular computing with machine learning <cit.>.
In 1982, Chen pointed out that human cognition has the characteristic of large-scale priority <cit.>. Motivated by the cognitive mechanism of the human brain, Wang developed a framework of multi-granularity cognitive computing to handle uncertain information <cit.>. Additionally, Xia et al. provided a framework of granular-ball computing by merging granular computing and granular-balls <cit.>.
Subsequently, GBC has been widely used in various fields such as granular-ball classifiers <cit.>, granular-ball clustering <cit.>, granular-ball rough sets <cit.>, and
feature selection <cit.>.
Recently, Xia et al. introduced the granular-ball support vector machine (GBSVM) classifier by integrating granular-balls and the SVM classifier <cit.>.
Unlike traditional methods that use samples as inputs, the GBSVM classifier employs granular-balls as data inputs, and represents a non-point input approach with remarkable robustness and effectiveness. The robustness of GBC to noise and the efficiency of TWSVM motivate us to design a new classifier by combining granular-balls and the TWSVM classifier.
Additionally, Xue, Shao, and Xia designed the GBFSVM classifier, which exhibits higher robustness compared to the traditional FSVM classifier <cit.>.
However, the GBFSVM classifier only considers the membership degree of granular-balls to the class center, and granular-balls located in the boundary region of two classes have the same membership degree for both classes, it may lead to incorrect classifications and predictions. The current methodology solely determines granular-ball memberships by the Euclidean distance from the granular-ball's center to the class center of samples, and possibly overlooks support granular-balls situated far from class centers yet proximate to the classification boundary. It motivates us to design the GBFTSVM classifier by integrating granular-balls with the FSVM classifier. The contributions of this paper are listed as follows.



(1) We integrate GBC with the TWSVM classifier to construct the GBTWSVM classifier. This classifier uses coarse-grained granular-balls instead of traditional sample points as inputs. It ensures that the hyperplane is positioned as close as possible to one class of granular-balls while being distant from the other class of granular-balls, which enhances the robustness and efficiency of the TWSVM classifier.


(2) We combine GBC, PFS, GBRS, and FTSVM to propose the GBFTSVM classifier. This classifier introduces a new scoring function that assigns different scores to granular-balls within positive and boundary regions, and distinguishes the varied contributions of granular-balls in different regions for classification. It further enhances the performance of the GBTWSVM classifier.


(3) We perform the experiment with the GBTWSVM classifier and the GBFTSVM classifier on twenty benchmark datasets  from the UCI machine learning repository. The experimental results demonstrate that the GBTWSVM classifier and the GBFTSVM classifier outperform five mainstream classifiers in terms of accuracy, precision, recall, and running time. Particularly, the GBTWSVM classifier and the GBFTSVM classifier exhibit higher robustness to noise in the classification task.


The rest of this paper is listed as follows: Section <ref> reviews  GBRS, TWSVM, and FSVM. Section <ref> designs the GBTWSVM classifier. Section <ref> gives the GBFTSVM classifier. Section <ref> conducts the experiment with the GBTWSVM classifier and the GBFTSVM classifier. Section <ref> concludes this paper and shows the future work.





§ PRELIMINARIES


In this section, we review some concepts of GBC <cit.>, TWSVM <cit.>, and FSVM <cit.>.



 §.§ Granular-Ball Computing


The core idea of GBC is to take a family of granular-balls to cover the original set, and use granular-balls instead of sample points as inputs. Given a dataset U={(x_1,l_1),(x_2,l_2),...,(x_n,l_n)}, where X={x_k | x_k∈ℛ^d, k=1,2,...,n} stands for the feature value matrix of U with d features, and ℒ={l_k | l_k∈ℛ, k=1,2,...,n} is the corresponding label vector. The sample universe U is covered by a family of granular-balls 𝒢ℬ={GB_i | i=1,2,...,m}, and
c_i=1/n_i∑_k=1^n_ix_ik stands for the centre of the i-th granular-ball GB_i, where x_ik and n_i stand for the ik-th sample and the number of samples in the i-th granular-ball GB_i, respectively. There are two methods to calculate the radius r_i of the granular-ball GB_i: r_i=max_x_ik∈ GB_i|x_ik-c_i| and  r_i=1/n_i∑_k=1^n_i|x_ik-c_i|.
To eliminate the effect of noisy data within each granular-ball, the overall label y_i =max_l_k∈ℒ|{(x,l) ∈ GB_i| l=l_k}| of GB_i takes the label that appears most frequently within the granular-ball. The purity p_i = | {(x,l)∈ GB_i| l=y_i}|/n_i denotes the proportion of samples with the label y_i in GB_i.


In GBC, the initial primary objective is to generate a family of granular-balls 𝒢ℬ, the objective function of granular-ball generation is shown as follows:

    min     λ _1×n/∑_GB_i∈𝒢ℬ|GB_i|+λ _2× m,
    s.t.      quality(GB_i) ≥ T,

where λ _1 and λ _2 stands for the weight coefficients, m is the number of granular-balls, and quality(GB_i) is the proportion of the majority of samples with the same label in the granular-ball GB_i.
However, the classical method of granular-ball generation faces challenges in adapting to the data distribution of each dataset. This difficulty arises from the purity threshold parameter that struggles to align with individual dataset characteristics.
To address this limitation, Xia et al. proposed a purity-adaptive method of granular-ball generation, and made the granular-ball generation completely parameter-free <cit.>. The objective function can be expressed as follows:

    min    λ _1×n/∑_GB_i∈𝒢ℬ|GB_i|+λ _2× m,
    s.t.     quality(GB_i) ≥ T_0, W(𝒢ℬ_i)>quality(GB_i),
       c_i-c_j>r_i-r_j (i,j∈[1,m],y_i≠ y_j),

where T_0 stands for the initial purity of the granular-ball, 𝒢ℬ_i is the set of the child granular-balls of GB_i, and W(𝒢ℬ_i) signifies the weighted sum of purities of the child granular-balls of GB_i.


Suppose S=(X,C,V,f) is an information system, and 𝒢ℬ is a family of granular-balls towards the attribute set Q⊆ C, the indiscernible granular-ball relation INDGB(Q) with respect to Q is defined as:
INDGB(Q)={(x,z)∈ X^2 |∃ GB ∈𝒢ℬ, x,z∈ GB}, and X/D={X_i | i=1,2,..,N} is the partition of the universe X with respect to the decision attribute set D, the upper and lower approximations of X_i with respect to Q are defined as:
GBR_QX_i=⋃{[x]_Q∈ X/Q |  [x]_Q∩ X_i∅}; GBR_QX_i=⋃{[x]_Q∈ X/Q |  [x]_Q⊆ X_i },
where [x]_Q={z∈ X| (x,z)∈ INDGB(Q)}.
Moreover, the upper approximation and the lower approximation of X/D with respect to Q are defined as:
GBR_QX =⋃_i=1^NGBR_QX_i; GBR_QX =⋃_i=1^NGBR_QX_i.
The positive region and boundary region of X/D with respect to Q are defined as:

    POS_Q(X/D)   =GBR_QX;
    
        BND_Q(X/D)   =GBR_QX-GBR_QX.



















 §.§ TWSVM and FSVM


TWSVM seeks to find a pair of non-parallel hyperplanes, and positions each hyperplane closer to homogeneous samples and somewhat far away from heterogeneous samples. A new sample x∈ℛ^d
is assigned to class +1 or -1
depending on which hyperplane it is closest to. A pair of non-parallel hyperplanes of the TWSVM classifier is obtained by solving the following quadratic programming problems (QPPs):

    ω_1,b_1,ξ_2min     1/2(x_Aω_1 + e_1b_1)^T(x_Aω_1 + e_1b_1) + C_1e^T_2ξ_2,
    s.t.     -(x_Bω_1 + e_2b_1) + ξ_2 ≥ e_2, ξ_2 ≥ 0,

and

    ω_2,b_2,ξ_1min     1/2(x_Bω_2 + e_2b_2)^T(x_Bω_2 + e_2b_2) + C_2e^T_1ξ_1,
    s.t.     (x_Aω_2 + e_1b_2) + ξ_1 ≥ e_1, ξ_1 ≥ 0,

where matrices x_A and x_B stand for the samples of classes +1 and -1, respectively, C_1 and C_2 are penalty parameters, e_1 and e_2 are vectors of ones with suitable dimensions, and ξ_1 and ξ_2 are slack variables.

A pair of non-parallel hyperplanes for the dual model of the TWSVM classifier is obtained by solving the following quadratic programming problems:

    αmax     α^Te_2  - 1/2α^TG(H^TH)^-1G^Tα ,
    s.t.     0 ≤α≤ C_1,


and


    γmax     γ^Te_1   - 1/2γ^TP(Q^TQ)^-1P^Tγ ,
    s.t.     0 ≤γ≤ C_2,


where H = [x_A    e_1], G = [x_B    e_2], P = [x_A    e_1] , Q = [x_B    e_2], u=[ω_1    b_1]^T = (H^TH)^-1G^Tα, and v=[ω_2    b_2]^T = (Q^TQ)^-1P^Tγ.



Assume U={ (x_1,l_1,s_1),(x_2,l_2,s_2),...,(x_n,l_n,s_n)} is a training dataset with fuzzy membership degrees, where s_k∈ (0,1] is the  membership degree that x_k belongs to its label l_k.
The goal of FSVM is to find an optimal hyperplane by solving the following optimization problems:

    min     1/2ω^2 +C∑_k = 1^ns_kξ_k,
    s.t.      l_k(ω ^Tϕ (x_k)+b)≥ 1-ξ _k,
       ξ _k≥ 0, k=1,2,...,n,

where C is a penalty parameter, and ξ_k is a slack variable.





§ GRANULAR-BALL TWIN SUPPORT VECTOR MACHINE



In this section, we give the principle of the GBTWSVM classifier <cit.>.



 §.§ Granular-Ball Twin Support Vector Machine



We define two non-parallel hyperplanes f_1(x): x^T ω _1 + b_1 = 0 and f_2(x): x^T ω _2 + b_2 = 0, where f_1(x) stands for the hyperplane close to the positive-class granular-ball (y_i=+1), and f_2(x) stands for the hyperplane close to the negative-class granular-ball (y_i=-1), ω_t and b_t stand for the normal vector and bias of f_t(x), respectively, where t∈{1,2}.
The two hyperplanes are constrained by two rules: (1) the hyperplane should be as close as possible to the center of the granular-ball belonging to the same class; (2) the distance between the hyperplane and the surface of granular-balls belonging to the other class should be as far as possible.
For binary classification problems, assume that m_1 granular-balls of class +1 and m_2 granular-balls of class -1 are generated, the centers of the granular-balls of classes +1 and -1 are represented by the matrices c_A and c_B, respectively, and the radii of the granular-balls of classes +1 and -1 are represented by the matrices r_A and r_B, respectively. For example,
we show the GBTWSVM classifier by Figure <ref>, where the red and blue circles stand for granular-balls of class +1 and class -1, respectively. A pair of non-parallel hyperplanes of the GBTWSVM classifier is derived by solving the following quadratic programming problems:

    ω_1,b_1,ξ_2min     1/2(c_Aω_1 + e_1b_1)^T(c_Aω_1 + e_1b_1) + C_1e^T_2ξ_2,
    s.t.     -(c_Bω_1 + e_2b_1) - r_B + ξ_2 ≥ e_2, ξ_2 ≥ 0,

and

    ω_2,b_2,ξ_1min     1/2(c_Bω_2 + e_2b_2)^T(c_Bω_2 + e_2b_2) + C_2e^T_1ξ_1,
    s.t.     (c_Aω_2 + e_1b_2) - r_A + ξ_1 ≥ e_1, ξ_1 ≥ 0,

where C_1 and C_2 are constants and both are greater than 0, and e_1 and e_2 are unit vectors of the appropriate dimension.




 §.§ The Dual Model of the GBTWSVM classifier



GBTWSVM minimizes the structural risk by adding the regularization term to the margin maximization objective.
This pair of quadratic programming problems can be achieved by solving the following Lagrange function:

    L(ω_1,b_1,ξ_2,α,β)
       =1/2‖c_Aω_1 + e_1b_1‖^2+C_1e^T_2ξ_2- α^T(-(c_Bω_1
          +e_2b_1)+ξ_2-r_B-e_2)-β ^Tξ _2,

where α and β are Lagrangian multipliers.



According to KKT conditions, we get:

    ∂ L/∂ω_1 = c_A^T(c_Aω _1+e_1b_1) +c_B^Tα =0;
    ∂ L/∂ b_1=e_1^T(c_Aω _1+e_1b_1) +e_2^Tα = 0.


By Equations (<ref>) and (<ref>), we obtain:

    [ c_A^T; e_1^T ][ c_A e_1 ][ ω _1;  b_1 ]+[ c_B^T; e_2^T ]α =0.


By taking E = [c_A    e_1], F = [c_B    e_2], and u = [ω _1    b_1]^T, Equation (<ref>) is reformulated as:

    E^TEu+F^Tα=0.


To improve its generalization capacity, we add the regularization item, and get the expression of u:

    u=-(E^TE+ε I)^-1F^Tα.


Similarly, by taking R = [c_A    e_1], S = [c_B    e_2], and v = [ω _2    b_2]^T, we get the expression of v:

    v=(S^TS+ε I)^-1R^Tγ.


A pair of non-parallel hyperplanes for the dual model of the GBTWSVM classifier is obtained by solving the following quadratic programming problems: 

    αmax     α^T(e_2+r_B) - 1/2α^TF(E^TE)^-1F^Tα ,
    s.t.     0 ≤α≤ C_1,

and

    γmax     γ^T(e_1+r_A)  - 1/2γ^TR(S^TS)^-1R^Tγ,
    s.t.     0 ≤γ≤ C_2.


A new sample x ϵ ℛ ^d is labeled as class t∈{1,2} depending on which of the two hyperplanes is closer:

    Class t = argt∈{1,2}min | ⟨ω _t,x ⟩ +b_t  | /ω _t .




 §.§ Algorithm Design







We give the algorithm for the GBTWSVM classifier as follows:

  
 




































Given a sample size n with approximately n/2 samples in each class, the time complexity of SVM is O(n^3). The time complexity of TWSVM is O(2 × (n/2)^3), which is four times faster than SVM. Assume that n samples generate m granular-balls, where m is less than n, and each class has approximately m/2 granular-balls, the time complexity of GBTWSVM is O(2 × (m/2)^3). If the number of generated granular-balls m is approximately n/2, the computational speed of GBTWSVM is eight times faster than TWSVM.








§ GRANULAR-BALL FUZZY TWIN SUPPORT VECTOR MACHINE



In this section, we define the scoring function for granular-balls by utilizing Pythagorean fuzzy sets <cit.>. We then assign distinct scores to granular-balls situated in the positive and boundary regions. Lastly, we present a detailed discussion on the GBFTSVM classifier.




 §.§ Pythagorean Fuzzy Membership Assignment




Assume X={x_k | k=1,2,...,n} is a dataset with n sample points, we derive a set of granular-balls 𝒢ℬ={GB_i | i=1,2,...,m}, and assign a pair of membership and non-membership degrees (μ_GB_i, ν_GB_i) to each granular-ball GB_i
such that 0≤μ_GB_i,ν_GB_i≤ 1 and μ^2_GB_i+ν^2_GB_i≤ 1, which depicts the relationship between the granular-ball GB_i and a specific class.
If μ_P(x_ik) and ν_P(x_ik) are the membership and non-membership degrees of a sample x_ik in a class, respectively, the membership and non-membership degrees of the granular-ball GB_i in this class are defined by:

    μ_GB_i = 1/n_i∑_k=1^n_iμ_P(x_ik);
    ν_GB_i = 1/n_i∑_k=1^n_iν_P(x_ik),

where x_ik and n_i stand for the ik-th sample and the number of samples in the i-th granular-ball GB_i. In most cases, the membership and non-membership degrees of samples are unknown. So we design the membership and non-membership functions of the granular-balls as follows.

(1) The membership degree: the majority of methods for constructing membership functions are based on the distance from samples to the class center. For binary-class granular-balls 𝒢ℬ={(c_i, r_i, p_i, y_i)| 1≤ i≤ m}, where c_i, r_i, p_i, and y_i represent the center, radius, purity, and label of GB_i, respectively, we define the class center C^+ and the maximum radius R^+ of positive-class granular-balls (y_i=+1), as well as the class center C^- and the maximum radius R^- of negative-class granular-balls (y_i=-1) as:
    
    C^+    = 1/m_+∑_y_i = +1 c_i,
     R^+    = y_i=+1max‖ c_i-C^+ ‖; 
    
        C^-    = 1/m_-∑_y_i = -1 c_i,
     R^-    = y_i=-1max‖ c_i-C^- ‖,

    where m_+ and m_- stand for the numbers of positive and negative granular-balls, respectively. For a granular-ball GB_i, the membership degree can be defined as:
    
    μ_GB_i=
            
                1-‖ c_i-C^+‖/R^+ + ε,    if  y_i = +1; 
    
                1-‖ c_i-C^-‖/R^- + ε,    if  y_i = -1,

    where ε > 0, and ε is a very small number.

(2) The non-membership degree: by using the granular-ball's membership degree and the granular-ball's purity, we give the non-membership function as follows:
    
    ν_GB_i^2=(1-μ_GB_i^2)(1-p_i),

    where p_i stands for the purity of GB_i. For each granular-ball GB_i, the membership degree μ_GB_i and the non-membership degree ν_GB_i satisfy the following conditions: 0<μ_GB_i,ν_GB_i≤1, and 0 ≤μ_GB_i^2+ν_GB_i^2≤ 1.




The impact of samples is not uniform. Especially, the samples in the boundary region play a crucial role in achieving accurate classifications. Conversely, the samples far from the boundary contribute relatively less to the classification. To accurately measure the contribution of granular-balls in different regions to classification, we categorize them into positive and boundary regions based on GBRS, and assign them different scoring functions. Since the overlap between heterogeneous granular-balls reduces the accuracy of classification, we employ an adaptive method of granular-ball generation to ensure that there are no heterogeneous overlaps. Based on the closeness index of Pythagorean fuzzy sets, we define the granular-ball closeness function as follows:

    θ_GB_i=√(1-ν_GB_i^2/2-μ_GB_i^2-ν_GB_i^2).


Granular-balls located in the boundary region are often close to the classification hyperplane, which significantly aids in determining the decision boundary. Conversely, granular-balls in the positive region are typically far from the classification hyperplane and contribute little to determine the decision boundary. According to GBRS, if the purity of granular-ball GB_i is equal to 1, then GB_i belongs to the positive region. If the purity of granular-ball GB_i is not equal to 1, then GB_i belongs to the boundary region. Therefore, we assign a lower score μ_GB_i to the granular-ball in the positive region and a higher score θ_GB_i to that in the boundary region. Thus, the scoring function of the granular-ball GB_i is defined as follows:

    s_GB_i=
        μ_GB_i,    if  p_i=1; 
    θ_GB_i,     if  p_i≠ 1.


In Figure <ref>, dashed granular-balls stand for those belonging to the boundary region, and they are typically positioned close to the separating hyperplane. We assign them a higher score. In contrast, solid granular-balls stand for those belonging to the positive region, and they are often positioned far from the separating hyperplane. This scoring function for granular-balls enables us to clearly distinguish the contribution of granular-balls in different regions.




 §.§ The Linear GBFTSVM Classifier



Consider a set of granular-balls 𝒢ℬ={(c_i, r_i, p_i, y_i)| 1≤ i≤ m}, where c_i is the center of granular-ball GB_i, r_i is the radius of granular-ball GB_i, p_i is the purity of granular-ball GB_i, and y_i∈{-1,+1} signifies the label of granular-ball GB_i. A pair of non-parallel hyperplanes for the linear GBFTSVM classifier is obtained by solving the following quadratic programming problems: 

    ω_1,b_1,ξ_2min     1/2(c_Aω_1 + e_1b_1)^T(c_Aω_1 + e_1b_1) + C_1s^T_Bξ_2,
    s.t.     -(c_Bω_1 + e_2b_1) - r_B + ξ_2 ≥ e_2, ξ_2 ≥ 0,

and

    ω_2,b_2,ξ_1min     1/2(c_Bω_2 + e_2b_2)^T(c_Bω_2 + e_2b_2) + C_2s^T_Aξ_1,
    s.t.     (c_Aω_2 + e_1b_2) - r_A + ξ_1 ≥ e_1, ξ_1 ≥ 0,

where C_1 and C_2 are constants and both are greater than 0, and e_1 and e_2 are unit vectors of the appropriate dimension, s_A∈ℛ and s_B∈ℛ are the score values of positive and negative granular-balls, respectively.





 §.§ The Dual Model of the GBFTSVM Classifier



We tackle the quadratic programming problem of the linear GBFTSVM classifier by incorporating Lagrange multipliers α_1, β_1, α_2, and β_2 as follows.

    L(ω_1,b_1,ξ_2,α_1,β_1)
       =1/2‖c_Aω_1 + e_1b_1‖^2+C_1s^T_Bξ_2- α_1^T(-(c_Bω_1+e_2b_1)
          +ξ_2-r_B-e_2)-β_1 ^Tξ _2,

where α_1 and β_1 are Lagrangian multipliers.

According to KKT conditions, we get:

    ∂ L/∂ω_1   = c_A^T(c_Aω _1+e_1b_1) +c_B^Tα =0;
    ∂ L/∂ b_1   =e_1^T(c_Aω _1+e_1b_1) +e_2^Tα = 0;
    ∂ L/∂ξ_2   = C_2s_B^T-α _1-β _1.


By Equations (<ref>) and (<ref>), we obtain:

    [ c_A^T; e_1^T ][ c_A e_1 ][ ω _1;  b_1 ]+[ c_B^T; e_2^T ]α_1 =0.


By taking E = [c_A    e_1], F = [c_B    e_2], and u = [ω _1    b_1]^T, Equation (<ref>) is changed as:

    E^TEu+F^Tα=0.


Then, by adding the regularization item, we get the expression of u:

    u=-(E^TE+ε I)^-1F^Tα_1.


Similarly, by taking R = [c_A    e_1], S = [c_B    e_2], and v = [ω _2    b_2]^T, we get an expression of v:

    v=(S^TS+ε I)^-1R^Tα_2.


A pair of non-parallel hyperplanes for the dual model of linear GBFTSVM classifier is obtained by solving the following quadratic programming problems: 

    α_1max     α_1^T(e_2+r_B) - 1/2α_1^TF(E^TE)^-1F^Tα_1,
    s.t.     0 ≤α_1 ≤ C_3 s_B,

and

    α_2max     α_2^T(e_1+r_A)  - 1/2α_2^TR(S^TS)^-1R^Tα_2,
    s.t.     0 ≤α_2 ≤ C_4 s_A.


A new input data x ϵ ℛ ^d can be labeled as class t∈{1,2} depending on which of the two hyperplanes is closer:

    Class t = argt∈{1,2}min | ⟨ω _t,x ⟩ +b_t  | /ω _t .




 §.§ Algorithm Design


We give the algorithm for the GBFTSVM classifier as follows:

  
 











































Assume that n samples generate m granular-balls, where m is less than n and each class has approximately m/2 granular-balls, the time complexity of GBFTSVM is O(2×(m/2)^3). If the number of generated granular-balls m is approximately n/2, the computational speed of GBFTSVM is eight times faster than TWSVM.






§ EXPERIMENT ANALYSIS



In this section, we compare GBFTSVM  and GBTWSVM with TWSVM <cit.>, EFSVM <cit.>, IFTSVM <cit.>, GBSVM <cit.> and GBFSVM <cit.>.
All methods are implemented using Python 3.11 on a desktop with an Intel Core^TM i7-10700 CPU @ 2.90 GHz with 16 GB RAM.



 §.§ Experimental Datasets



We download twenty benchmark datasets from the UCI machine learning repository <cit.>, and show the statistics of these benchmark datasets in Table <ref>.
Moreover, we use ten-fold cross-validation to evaluate the performance of the seven methods in terms of Accuracy (Acc), Precision (Prec), Recall (Rec), and Standard deviation of Accuracy (Sd). To ensure a fair comparison of efficiency, the seven models are solved using the optimize library in Python under the optimal parameter settings.





 §.§ Penalty Parameters





We perform a grid search on the penalty parameters that appear in different methods, and the best results obtained are reported as the final results.
The parameters of the TWSVM classifier, the IFTSVM classifier, the GBTWSVM classifier, and the GBFTSVM classifier are set as follows: C_i (i=1,2,3,4) is explored in the grid {2^i | i=-5,-4,...,+4,+5} by setting C_1=C_3, C_2=C_4.
The parameters of the EFSVM classifier, the GBSVM classifier, and the GBFSVM classifier are set as follows: C is explored in the grid {2^i | i=-5,-4,...,+4,+5}.
To investigate the impact of different parameter values on the performance of the GBFTSVM classifier, we collect the experimental results on various datasets. Due to space limitations, we select four datasets, as shown in Figure <ref>. From the graph, it can be observed that the GBFTSVM classifier is highly sensitive to changes in parameter values. We select the parameters corresponding to the highest Acc achieved by seven models on different datasets without noise, as shown in Table <ref>.






 §.§ Experimental Results on Datasets without Noise















  §.§.§ Comparison of Training Times


In the absence of noise, we show the training time of these models across various datasets in Table <ref>. Notably, the GBTWSVM classifier exhibits the shortest training time across all datasets, and achieves remarkable speedups of hundreds of times in certain cases. In addition to the GBTWSM classifier, the GBFTSVM classifier has the shortest training time among the remaining models.
This observation can be attributed to two primary reasons. First, the number of granular-balls generated in the datasets is significantly smaller than the sample size. Second, both GBTWSVM and GBFTSVM derive the hyperplane by two smaller-scale quadratic programming problems instead of a single large-scale one.
Furthermore, although the GBTWSVM classifier performs slightly worse Acc than the GBFTSVM classifier, it exhibits faster. This is because the GBFTSVM classifier requires the computation of a scoring function for each granular-ball to differentiate their contributions to classification, whereas the GBTWSVM classifier does not incorporate this aspect.





  §.§.§ Comparison of Performance

To comprehensively evaluate the performance of the GBTWSVM classifier and the GBFTSVM classifier on multiple datasets without noise, we conduct a rigorous comparison of seven models across 20 benchmark datasets. We evaluate these models based on their Acc, Pre, Rec, and Sd. The results are summarized in Table <ref>. After a thorough analysis, we arrive at the following conclusions:
the GBFTSVM classifier exhibits the highest Acc, Pre, and Rec in 17 out of the 20 datasets, and firmly establishes its superior classification capabilities. Notably, the GBFTSVM classifier also gives a relatively low standard deviation in Acc, which indicates its consistent and reliable performance. While the GBTWSVM classifier trails the GBFTSVM classifier slightly in terms of overall Acc, it achieves high Acc in 14 datasets with a low standard deviation, and emerges as a strong contender among the remaining six models.
Furthermore, we compare the average Acc of each model across all datasets. Evidently, the GBFTSVM classifier achieves the highest average Acc, and outperforms the other models by 2%-12%, which underscores its superior classification performance. In comparison with the GBTWSVM classifier, the notable improvement of the GBFTSVM classifier's average Acc validates the significance of our optimizations and enhancements to the membership function for classification.


To overcome the potential bias caused by a model's high Acc in one dataset and low Accs in others, we calculate the average rank of each model across the datasets. The model with the highest Acc is ranked first, and in cases of equal Acc, the average of the corresponding ranks is taken. The GBFTSVM classifier emerges with the lowest average rank, and firmly establishes its superiority among all the compared models. Similarly, the GBTWSVM classifier's second-lowest average rank demonstrates its strong performance among the remaining models, excluding the GBFTSVM classifier.







 §.§ Experimental Results on Datasets with Noise


To assess the robustness of the GBTWSVM classifier  and the GBFTSVM classifier  to noise, we add 5% and 10% label noise to 20 datasets and compare the Acc of seven models. In Table <ref>, the GBFTSVM classifier  gives the highest Acc on 18 datasets with 5% noise and on 19 datasets with 10% noise, and outperforms the other six models. Notably, in the Messidor-features and WDBC datasets, although the GBFTSVM classifier  does not exhibit optimal performance in the absence of noise, its stability in the face of noise leads to improved classification performance. Especially, it surpasses other models on datasets with 10% noise.
It is noteworthy that classifiers based on granular-balls exhibits a relatively stable Acc fluctuation of approximately 2% with noise addition, compared to a fluctuation of over 2% for classifiers based on point inputs. This suggests that classifiers based on granular-balls offer better robustness to noise in terms of prediction stability.
Furthermore, the GBFTSVM classifier emerges as the top performer in terms of average Acc across all noise levels (0%, 5%, and 10%). Several factors contribute to this performance. First, the coarser size of granular-balls and the assignment of majority point labels reduce the impact of noisy points. Second, the GBFTSVM classifier  employs two symmetric support vectors based on granular-balls, which, through their symmetry, effectively resists noise and outliers, and enhances generalization ability. Third, the assignment of scoring functions to granular-balls effectively distinguishes supportive granular-balls, and minimizes their influence on determining the separation hyperplane.






 §.§ Statistical Analysis



To validate the statistical significance of the GBFTSVM classifier, we conduct Friedman test and Nemenyi post-hoc test on seven models across 20 datasets without noise. Initially, we assume that all models are equivalent under the null hypothesis. Friedman statistic follows a chi-square distribution, which is calculated using the formula:
χ _F ^2 =12N/M(M+1) [ ∑_j=1^MR_j^2-M(M+1)^2/4 ],
where R_j (j=1,2,...7) stands for the average rank of the j^th model, N is the number of datasets, and M is the number of models. Alternatively, when Friedman statistic follows an F-distribution with ((M-1),(M-1)(M-N)) degrees of freedom, the formula is:
F_F=(N-1)χ_F^2/N(M-1)-χ_F^2. Based on Table <ref>, we obtain an F_F value of 28.6 with degrees of freedom (6,114). At a significance level of 0.05, the critical value of F(6,114) is 2.179. Since F_F exceeds this critical value, we reject the null hypothesis.
Subsequently, we ultilize Nemenyi post-hoc test to further distinguish these models. It involves calculating the critical difference (CD):
CD=q_α√(M(M+1)/6N). At a significance level of 0.05, q_α is 2.949. To visualize the differences among seven classifiers, we generate a CD diagram depicted in Figure <ref>. This figure clearly shows that the horizontal line representing the GBFTSVM classifier does not overlap with those of the TWSVM classifier, the EFSVM classifier, the IFTSVM classifier, the GBSVM classifier, and the GBFSVM classifier. This indicates that the GBFTSVM classifier significantly differs from and outperforms other models.





























































§ CONCLUSIONS




Inspired by GBC and TWSVM, we have proposed the GBTWSVM classifier for binary classification problems, which utilizes granular-balls as inputs instead of individual samples, and provides a scalable, efficient, and robust data processing method.
Subsequently, we have introduced the GBFTSVM classifier by integrating GBC, PFS, and FTSVM, in which it defines the granular-ball membership and non-membership functions based on the Pythagorean closeness index. Additionally, the GBFTSVM classifier distinguishes the contribution of granular-balls from different regions to classification. Finally, the experimental results on 20 benchmark datasets have demonstrated that the GBFTSVM classifier exhibits excellent results in terms of classification Acc, while the GBTWSVM classifier shows superior performance in training time.


In real-world applications, there are numerous instances of nonlinear classification problems and multi-classification challenges. Our forthcoming research will be dedicated to proficient classifiers and algorithms specifically designed to tackle these diverse challenges. 



§ ACKNOWLEDGEMENTS


This work is supported by the National Key Research and Development Program (No. 2022YFB3104700), the National Natural Science Foundation of China (No. 62076040, 61976158, 62376198), Hunan Provincial Natural Science Foundation of China (No. 2020JJ3034), and the Scientific Research Fund of Hunan Provincial
Education Department (No. 22A0233).

99
IEEEtran
V.N. Vapnik_1998
V.N. Vapnik, Statistical learning theory, Wiley, vol. 1, Sep. 1998..10in
B. Heisele_2001
B. Heisele, P. Ho, and T. Poggio, Face recognition with support vector machines: Global versus component-based approach, Proceedings Eighth IEEE International Conference on Computer Vision, vol. 2, pp. 688-694, Jul. 2001..10in
O.L. Mangasarian_2006
O.L. Mangasarian and E.W. Wild, Multisurface proximal support vector machine classification via generalized eigenvalues, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 1, pp. 69-74, Jul. 2006..10in
N.Y. Deng_2012
N.Y. Deng, Y.J. Tian, and C.H. Zhang, Support vector machines: optimization based theory, algorithms, and extensions, CRC Press, Nov. 2012..10in
V.N. Vapnik_2017
V.N. Vapnik and R. Izmailov, Knowledge transfer in svm and neural networks, Annals of Mathematics and Artificial Intelligence, vol. 81, pp. 3-19, Feb. 2017..10in
Jayadeva_2007
J.R. Khemchandani and S. Chandra, Twin support vector machines for pattern classification, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 5, pp. 905-910, May 2007..10in
R. Khemchandani_2009
J.R. Khemchandani and S. Chandra, Optimal kernel selection in twin support vector machines, Optimization Letters, vol. 3, no. 1, pp. 77-88, Jan. 2009..10in
Z. Qi_2013
Z. Qi, Y. Tian, and Y. Shi, Robust twin support vector machine for pattern classification, Pattern recognition, vol. 46, no. 1, pp. 305-316, Jan. 2013..10in
X.J. Xie_2017
X.J. Xie and S.L. Sun, PAC-bayes bounds for twin support vector machines, Neurocomputing, vol. 234, pp. 137-143, Apr. 2017..10in
X. Peng_2010
X. Peng, A v-twin support vector machine (v-TSVM) classifier and its geometric algorithms, Information Sciences, vol. 180, pp. 3863-3875, Oct. 2010..10in
B.B. Gao_2015
B.B. Gao, Coordinate descent fuzzy twin support vector machine for classification, 2015 IEEE 14th international conference on machine learning and applications (ICMLA), pp. 7-12, Dec. 2015..10in
C.F. Lin_2002
C.F. Lin and S.D. Wang, Fuzzy support vector machines, IEEE Transactions on Neural Networks, vol. 13, no. 2, pp. 464-471, Mar. 2002..10in
X.W. Yang_2011
X.W. Yang, G.Q. Zhang, J. Lu, and J. Ma, A kernel fuzzy c-means clustering fuzzy support vector machine algorithm for classification problems with outliers or noises, IEEE Transactions on Fuzzy Systems, vol. 19, no. 1, pp. 105-115, Oct. 2011..10in
Y.T. Xu_2012
Y.T. Xu, A rough margin-based linear  support vector regression, Statistics and Probability Letters, vol. 82, no. 3, pp. 528-534, Mar. 2012..10in
A. Shigeo_2015
A. Shigeo, Fuzzy support vector machines for multilabel classification, Pattern Recognition, vol. 48, no. 6, pp. 2110-2117, Jun. 2015..10in

Q. Fan_2017_Jan
Q. Fan, Z. Wang, D.D. Li, D.Q. Gao, and H.Y. Zha, Entropy-based fuzzy support vector machine for imbalanced datasets, Knowledge-Based Systems, vol. 115, pp. 87-99, Jan. 2017..10in
S. Rezvani_2019
S. Rezvani, X. Wang, and F. Pourpanah, Intuitionistic fuzzy twin support vector machines, IEEE Transactions on Fuzzy Systems, vol. 27, no. 11, pp. 2140-2151, Nov. 2019..10in
D. Gupta_2019_Nov
D. Gupta, B. Richhariya, and P. Borah, A fuzzy twin support vector machine based on information entropy for class imbalance learning, Neural Computing and Applications, vol. 31, pp. 7153-7164, Nov. 2019..10in
Z.Z. Liang_2022
Z.Z. Liang  and L. Zhang, Intuitionistic fuzzy twin support vector machines with the insensitive pinball loss, Applied Soft Computing, vol. 115, pp. 108231, Jan. 2022..10in
D.A. Hua_2022
H. Duan, T. Feng, S.N. Liu, Y.L. Zhang, and J.L. Su, Tumor classification of gene expression data by fuzzy hybrid twin SVM, Chinese Journal of Electronics, vol. 31, no. 1, pp. 99-106, Jan. 2022..10in
Z.Z. Liang_2024
Z.Z. Liang  and S.F. Ding, Fuzzy twin support vector machines with distribution inputs, IEEE Transactions on Fuzzy Systems, vol. 32, no. 1, pp. 240-254, Jan. 2024..10in

L.A. Zadeh_1979
L.A. Zadeh, Fuzzy sets and information granularity, Fuzzy sets, fuzzy logic, and fuzzy systems: selected papers, pp. 433-448, Jul. 1979..10in

Y.Y.Yao_2009
Y.Y. Yao, Interpreting concept learning in cognitive informatics and granular computing, IEEE Transactions on Systems, Man, and Cybernetics, vol. 39, no. 4, pp. 855-866, Mar. 2009..10in



W.P.Ding_2024
W.P. Ding, Y. Sun, M. Li, J. Liu, H.R. Ju, J.S. Huang,  and C.T. Lin, A novel spark-based attribute reduction and neighborhood classification for rough evidence, IEEE Transactions on Cybernetics, vol. 54, no. 3, pp. 1470-1483, Mar. 2024..10in

W.P.Ding_2021
W.P. Ding, W. Pedrycz, I. Triguero, Z.H. Cao,  and C.T. Lin, Multigranulation supertrust model for attribute reduction, IEEE Transactions on Fuzzy Systems, vol. 29, no. 6, pp. 1395-1408, Jun. 2021..10in







Y.J. Zhang_2022
Y.J. Zhang, T.N. Zhao, D.Q. Miao,  and W. Pedrycz, Granular multilabel batch active learning with pairwise label correlation, IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 52, no. 5, May 2022..10in


K.H. Yuan_2024
K.H. Yuan, D.Q. Miao, Y.Y. Yao, H.Y. Zhang,  and X.R. Zhao, Feature selection using zentropy-based uncertainty measure, IEEE Transactions on Fuzzy Systems, vol. 32, no. 4, Apr. 2024..10in


S. Ding_2015
S. Ding, H. Huang, J. Yu,  and H. Zhao, Research on the hybrid models of granular computing and support vector machine, Artificial Intelligence Review, vol. 43, pp. 565-577, Apr. 2015..10in

H. Liu_2016
H. Liu and M. Cocea, Granular computing-based approach for classification towards reduction of bias in ensemble learning, Granular Computing, vol. 2, pp. 131-139, Nov. 2016..10in

S. Butenkov_2017
S. Butenkov, A. Zhukov, A. Nagorov,  and N. Krivsha, Granular computing models and methods based on the spatial granulation, Procedia Computer Science, vol. 103, pp. 295-302, Jan. 2017..10in

L. Chen_2021
L. Chen, L. Zhao, Z. Xiao, Y. Liu, and J. Wang, A granular computing based classification method from algebraic granule structure, IEEE Access, vol. 9, pp. 68118-68126, May 2021..10in


L. Chen_1982
L. Chen, Topological structure in visual perception, Science, vol. 218, no. 4573, pp. 699-700, Nov. 1982..10in
G.Y. Wang_2017
G.Y. Wang, DGCC: data-driven granular cognitive computing, Granular Computing, vol. 2, no. 4, pp. 343-355, Jul. 2017..10in



S.Y. Xia_2019
S.Y. Xia, Y.S. Liu, X. Ding, G.Y. Wang, H. Yu,  and Y.G. Luo, Granular ball computing classifiers for efficient, scalable and robust learning, Information Sciences, vol. 483, pp. 136-152, May 2019..10in
Y.F. Xue_2022
Y.F. Xue, Y.B. Shao, and H. Xia, GBFSVM: a robust classification learning method, Scientific Journal of Intelligent Systems Research, vol. 4, no. 1, pp. 238-245, Jan. 2022..10in

S.Y. Xia_2023_Apr
S.Y. Xia, S.Y. Zheng, G.Y. Wang, X.B. Gao, and B.G. Wang, Granular ball sampling for noisy label classification or imbalanced classification, IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 4, pp. 2144-2155, Apr. 2023..10in
S.Y. Xia_2022_Oct
S.Y. Xia, X.C. Dai, G.Y. Wang, X.B. Gao, and E. Giem, An efficient and adaptive granular-ball generation method in classification problem, IEEE Transactions on Neural Networks and Learning Systems, pp. 1-13, Oct. 2022..10in

S.Y. Xia_2022_Jan
S.Y. Xia, D.W. Peng, D.Y. Meng, C.Q. Zhang, G.Y. Wang, E. Giem, W. Wei, and Z.Z. Chen, Ball k-means: Fast adaptive clustering with no bounds, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 87-99, Jan. 2022..10in
J. Xie_2023
J. Xie, W.Y. Kong, S.Y. Xia, G.Y. Wang, and X.B. Gao, An efficient spectral clustering algorithm based on granular-ball, IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 9, pp. 9743-9753, Sep. 2023. .10in


D.D. Cheng_2024
D.D. Cheng, S.S. Liu, S.Y. Xia, and G.Y. Wang, Granular-ball computing-based manifold clustering algorithms for ultra-scalable data, Expert Systems with Applications, vol. 247, no. 1, pp. 123313, Aug. 2024..10in

S.Y. Xia_2022_Mar
S.Y. Xia, H. Zhang, W.H. Li, G.Y. Wang, E. Giem, and Z.Z. Chen, GBNRS: A novel rough set algorithm for fast adaptive attribute reduction in classification, IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 3, pp. 1231-1242, Mar. 2022..10in
Q.H. Zhang_2023
Q.H. Zhang, C.Y. Wu, S.Y. Xia, F. Zhao, M. Gao, Y.L. Cheng, and G.Y. Wang, Incremental learning based on granular ball rough sets for llassification in dynamic mixed-type decision system, IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 9, pp. 9319-9332, Sep. 2023..10in
X. Ji_2023
X. Ji, J.H. Peng, P. Zhao, and S. Yao, Extended rough sets model based on fuzzy granular ball and its attribute reduction, Information Sciences, vol. 640, pp. 119071, Sep. 2023..10in
S.Y. Xia_2023_Nov
S.Y. Xia, C. Wang, G.Y. Wang, X.B. Gao, E. Giem, and J.H. Yu, GBRS: A unified model of Pawlak rough set and neighborhood rough set, IEEE Transactions on Neural Networks and Learning Systems, pp. 1-15, Nov. 2023..10in

X.L. Peng_2022
X.L. Peng, P. Wang, S.Y. Xia, C. Wang, and W.Q. Chen, VPGB: A granular-ball based model for attribute reduction and classification with label noise, Information Sciences, vol. 611, pp. 504-521, Sep. 2022..10in



W.B. Qian_2023_June
W.B. Qian, W.Y. Ruan, Y.H. Li, and J.T. Huang, Granular ball-based label enhancement for dimensionality reduction in multi-label data, Appllied Intelligence, vol. 53, pp. 24008–24033, June 2023..10in
W.B. Qian_2023_Dec
W.B. Qian, F.K. Xu, J. Qian, W.H. Shu,  and W.P. Ding, Multi-label feature selection based on rough granular-ball and label distribution, Information Sciences, vol. 650, pp. 119698, Dec. 2023..10in






L.X.Zhao_2024
L.X. Zhao, Z.F. Zhang, W.J. Liu, and G.M. Lang, GBTWSVM: Granular-ball twin support vector
machine, The 2024 International Joint Conference on Rough Sets, Halifax (Canada), May. 2024..10in


G.M. Lang_2020
G.M. Lang, D.Q. Miao, and H. Fujita, Three-way group conflict analysis based on Pythagorean fuzzy set theory, IEEE Transactions on Fuzzy Systems, vol. 28, no. 3, pp. 447-461, Mar. 2020..10in

M. Kelly_2023
M. Kelly, R. Longjohn, and K. Nottingham, The UCI Machine Learning Repository, 2023, Available: https://archive.ics.uci.edu..10in








§ APPENDIX: ABBREVIATIONS






 Paper Title: The Stochastic Conjugate Subgradient Algorithm For Kernel Support Vector Machines.



Thermalization propagation front and robustness against avalanches in localized systems
    Angelo Russomanno
    
=======================================================================================




§ ABSTRACT 
Stochastic First-Order (SFO) methods have been a cornerstone in addressing a broad spectrum of modern machine learning (ML) challenges. However, their efficacy is increasingly questioned, especially in large-scale applications where empirical evidence indicates potential performance limitations. In response, this paper proposes an innovative method specifically designed for kernel support vector machines (SVMs). This method not only achieves faster convergence per iteration but also exhibits enhanced scalability when compared to conventional SFO techniques. Diverging from traditional sample average approximation strategies that typically frame kernel SVM as an “all-in-one” Quadratic Program (QP), our approach adopts adaptive sampling. This strategy incrementally refines approximation accuracy on an “as-needed” basis. Crucially, this approach also inspires a decomposition-based algorithm, effectively decomposing parameter selection from error estimation, with the latter being independently determined for each data point. To exploit the quadratic nature of the kernel matrix, we introduce a stochastic conjugate subgradient method. This method preserves many benefits of first-order approaches while adeptly handling both nonlinearity and non-smooth aspects of the SVM problem. Thus, it extends beyond the capabilities of standard SFO algorithms for non-smooth convex optimization. The convergence rate of this novel method is thoroughly analyzed within this paper. Our experimental results demonstrate that the proposed algorithm not only maintains but potentially exceeds the scalability of SFO methods. Moreover, it significantly enhances both speed and accuracy of the optimization process.




Kernel SVM, Stochastic Programming, Non-Smooth Convex Optimization




65K05, 90C15, 90C20, 90C25




§ INTRODUCTION




Kernel support vector machines (SVM), a subset of supervised learning models, are instrumental in data-driven classification tasks <cit.>. Consider a dataset of covariates represented as S={ (z_i,w_i) }_i=1^m, where the pairs correspond to features and their associated labels respectively. The primary objective is to predict the label W given a random feature Z utilizing the classifier β. In scenarios where the feature vector Z is exceedingly large or the data deviates from linear separability, it becomes necessary to apply a non-linear transformation to the feature vector Z, commonly referred to as kernel mapping ϕ. Consequently, the kernel SVM problem is formulated as:


    min_β g(β) := 1/2||β||^2+E [max{0,1-W ⟨β, ϕ(Z) ⟩}],


where the expectation E is taken with respect to the joint distribution of (Z,W). Directly addressing the problem as defined in Equation (<ref>) is infeasible due to two significant challenges: (a) the joint distribution of the pair (Z,W) is typically unknown, and (b) the kernel mapping ϕ is not directly accessible. To overcome the first challenge (a), the focus generally shifts towards resolving an empirical risk minimization (ERM) problem, specifically through a sample average approximation (SAA) approach. With regard to the second challenge (b), while direct access to ϕ may not be available, the inner product within the kernel mapping, ⟨ϕ(z_i), ϕ(z_j) ⟩, may be available.  This provides a valuable workaround <cit.>. Essentially, one uses the Representer Theorem<cit.> to express β in the form of β=∑_i=1^mα_i ϕ(z_i). This transformation enables a finite sample kernel SVM approximation as follows:


    min_αf_S(α): = 1/2⟨α, Q α⟩ + 1/m∑_i=1^mmax{0,1-w_i ⟨α, Q_i ⟩},


where α = [α_1, α_2, ..., α_m] is the decision varaible, Q ∈ℝ^m × m is the kernel matrix with Q_ij = ⟨ϕ(z_i), ϕ(z_j) ⟩ and Q_i refers to the i-th row of Q. It is significant to observe that so long as we have access to Q_ij=⟨ϕ(z_i), ϕ(z_j) ⟩, we can solve (<ref>) without having the direct knowledge of the kenel mapping ϕ.  Notably, a variety of kernel mappings, including but not limited to polynomial, Gaussian, and Laplacian kernels, often result in a positive definite matrix Q <cit.>. This characteristic implies that equation (<ref>) constitutes a convex optimization problem.

Equation (<ref>) can be interpreted as an “all-in-one” Quadratic Program (aioQP). However, the computational feasibility of solving the aioQP becomes challenging with extremely large datasets, such as those containing millions of data points. To address this problem, our paper introduces an adaptive sampling method aimed at improving the quality of the approximation locally, while enhancing the accuracy of the global approximation on an “as-needed” basis. The approximation at iteration k, utilizing a significantly smaller sample size |S_k| << m, is expressed as:


    min_αf_k(α) := 1/2⟨α, Q^k α⟩ + 1/|S_k|∑_i=1^|S_k|max{0,1-w_i ⟨α, Q_i^k ⟩},


where α = [α_1, α_2, …, α_|S_k|] is the decision variable, Q^k ∈ℝ^m × m is the kernel matrix with Q^k_ij = ⟨ϕ(z_i), ϕ(z_j) ⟩ and Q^k_i refers to the i-th row of Q^k. We will demonstrate that our proposed approach not only offers enhanced accuracy but also has greater scalability compared to Stochastic First Order (SFO) methods in tackling large-scale kernel SVM problems. To lay the groundwork, we will commence by reviewing a range of classic stochastic methods, many of which are applicable to kernel SVM.















   



 §.§ Stochastic first-order methods

SFO methods such as “stochastic gradient descent" (SGD) <cit.> or dual coordinate ascent <cit.> are very popular for solving machine learning problems. For black-box non-smooth optimization, SGD-type algorithms using secant approximations have been proposed by <cit.>. It is also worth noting that a recent survey <cit.> summarizes this type of optimization method comprehensively. Simple first-order gradient-based methods dominate the field for several convincing reasons: low computational cost, simplicity of implementation, and strong empirical results.   
For example, in order to solve (<ref>), it is common to adopt a specialized version of SGD such as Pegasos <cit.> where the subgradient calculations can be carried out very rapidly and in parallel (for a given α∈ℝ^n). <cit.> shows that the Pegasos algorithm obtains an ε_p-sub-optimal solution in time O(1/(λε_p)), where λ is the condition number of Q. This run-time does not depend on the number of variables n, and therefore is favorable when n is very large.  Finally, asymptotitic analysis via the work of Polyak and Juditsky <cit.> has shown that SFO exhibits asymptotic normality for smooth stochastic optimization and a similar result has been recently obtained in <cit.> for the case of non-smooth stochastic convex optimization problems. 

While the aforementioned methods have low computational requirements in any one iteration, there are some disadvantages: a) the convergence rate of such algorithms depend heavily on the condition number of the Hessian of the quadratic function, b) SFO methods are not known for their numerical accuracy especially when the functions are non-smooth and, c) the convergence rate mentioned in the previous paragraph only applies under the assumption that the component functions are
smooth. Since (<ref>) has non-smooth component functions, such a convergence rate is not guaranteed.   





 §.§ Stochastic methods for general non-linear  optimization

For general non-linear stochastic optimization, there are two broad classes of algorithms, most of which are stochastic  generalizations of smooth deterministic optimization algorithms. For gradient based direction finding,  stochastic analogs were studied by <cit.>, whereas stochastic analogs of trust region methods (STORM) appear in a series of papers <cit.>. From a computational standpoint, direction-finding algorithms may have faster iterations, whereas trust region methods may provide stronger convergence properties <cit.>. However, the aforementioned stochastic methods rely on the existence of gradients which may not be appropriate for non-smooth stochastic problems such as kernel SVM in (<ref>). Readers interested in the non-smooth, non-convex case may refer to <cit.>.



 §.§ Stochastic conjugate gradient method

In contrast to first-order methods, algorithms based on Conjugate Gradients (CG) provide finite termination when the objective function is a linear-quadratic deterministic function <cit.>. Furthermore, CG algorithms are also used for large-scale nonsmooth convex optimization problems <cit.>. However, despite its many strengths, (e.g., fast per-iteration
convergence, frequent explicit regularization on step-size, and better parallelization than first-order
methods), the use of conjugate gradient methods for Stochastic Optimization is uncommon. Although <cit.> proposed some new stochastic conjugate gradient (SCG) algorithms, their assumptions do not support non-smooth objective functions, and hence not directly applicable to kernel SVM problems.
The following assumptions, which are common for kernel SVM models, will be imposed throughout this paper.

 
    
  * The optimization problems (<ref>)-(<ref>) are convex and finite valued. Thus, extended real-valued functions are not permissible in our framework. As a consequence, we assume the function f and f_k possess the Lipschitz constant 0 < L_f < ∞ and their sub-differentials are non-empty for every α. Note that when the sample size is large enough, L_f will represent an upper bound of the Lipschitz constant on both f and f_k. Efficient and accurate estimation of the Lipschitz constant holds significant practical importance. Readers interested in this topic may find valuable insights in the work of <cit.>.
    
    
  * The inner product of the kernel function and the classifier is bounded, i.e., for any kernel mapping ϕ(z) and classifier β,  |⟨ϕ(z), β⟩| < M. Intuitively, this assumption indicates that the classification error can not be unbounded, which is reasonable for kernel SVM. 









In this paper, we adopt a direction-finding approach in which the direction is obtained via a specialized problem of finding a vector with the smallest norm over a line segment. This approach was originally designed for deterministic non-smooth convex optimization problems <cit.>. 

In order to solve stochastic optimization problems when functions are non-smooth (as in  (<ref>)), it is straightforward to imagine that a stochastic subgradient method could be used. In this paper, we treat (<ref>) as a stochastic piecewise linear quadratic optimization problem and propose a new stochastic conjugate subgradient (SCS) algorithm which accommodates both the curvature of a quadratic function, as well as the decomposition of subgradients by data points, as is common in stochastic programming <cit.>. This combination enhances the power of stochastic first-order approaches without the additional burden of second-order approximations. In other words, our method shares the spirit of Hessian Free (HF) optimization, which has attracted some attention in the machine learning community <cit.>. As our theoretical analysis and the computational results reveal, the new method provides a “sweet-spot” at the intersection of speed, accuracy, and scalability of algorithms (see Theorems <ref> and <ref>).

 

  





 §.§ Contributions


(1) Our methodology extends Wolfe's deterministic conjugate subgradient method <cit.> into the stochastic domain. While a straightforward application of <cit.> to stochastic programming (SP) problems using Sample Average Approximation (SAA) <cit.> might be considered, our focus is on solving Kernel SVM problems with extremely large datasets such as Hepmass with 3.5 million samples. Consequently, we employ an adaptive sampling strategy over a deterministic finite sum approximation. This approach bears some resemblance to Stochastic Decomposition (SD) <cit.>, although SD employs a "trust-region" strategy for iterative progress. More notably though, deterministic algorithms for finite-sum SAA can impose significant computational and memory demands, especially with large datasets.  This observation is corroborated by our experimental results. To address the challenges arising from non-smoothness as well as extremely large datasets, our novel algorithm integrates several disparate notions such as adaptive sampling, decomposition, conjugate subgradients, and line search techniques. We demonstrate that this amalgamation effectively approximates solutions for kernel SVM problems with extremely large datasets.

(2) The ensuing analysis of our algorithm establishes a convergence rate of O(1/ε^2) for kernel SVM, where ε denotes the desired accuracy. This contrasts with the convergence rate of O(1/√(k)) reported in <cit.>, applicable to smooth, albeit far more demanding black-box and non-strongly convex problems. 

(3) In contrast to trust-region methods, which necessitate solving a constrained convex optimization problem in each iteration, our algorithm derives a descent direction by solving a one-dimensional convex optimization problem.

(4) Our computational experiments demonstrate that, from an optimization standpoint, the solutions produced by SCS yield lower objective function values compared with SFO methods, maintaining competitive computational times for small to medium-sized datasets. More crucially, as the dataset size increases, the efficiency and accuracy of our Stochastic Conjugate Subgradient (SCS) algorithm surpasses those of algorithms such as Pegasos, a specialized first-order methods for kernel SVM. 

The structure of this paper is as follows.  In <ref>, the discussion evolves from mathematical concepts to computational algorithms. We will discuss the SCS algorithm for solving large scale kernel SVM via a combination of Wolfe's non-smooth conjugate subgradient method <cit.> and adaptive sampling which helps leverage a successive sampling methodology as in Stochastic Decomposition <cit.>. This combination not only introduces some elements of recursive estimation into kernel SVM, but also bypasses the need for constrained QP algorithms, or even the need for solving a large linear system (of equations) rapidly. In <ref>, we will present the convergence analysis of the SCS algorithm by showing that the algorithm generates a super-martingale sequence which converges in expectation. In <ref>, we will provide some technical details about its implementation and compare the computational results with its two progenitors: the Pegasos algorithm (a SFO method) and Wolfe's algorithm <cit.>, which is a deterministic non-smooth optimization method. In the context of this comparison, we will discuss some advantages and disadvantages of adopting the SCS algorithm for solving the kernel SVM problem. Finally, our conclusions are provided in <ref>. The notations used in this paper are summarized in Appendix <ref>.

















  






 
 








 
 
  



§ STOCHASTIC CONJUGATE SUBGRADIENT (SCS) ALGORITHM


In this section, we focus on solving (<ref>) using the SCS  algorithm in which we combine the non-smooth CG method <cit.> with adaptive sampling and decomposition as in stochastic programming <cit.>. In this setting, decomposition refers to a treatment of the piecewise linear (max) function and the quadratic part separately in (<ref>) to find the direction of change for the decision variables. We will first state the SCS algorithm (Algorithm <ref>) and then explain the key concepts underlying this approach.  In the following, we let Nr(G) denote the projection of “0" onto any finite collection of vectors G.  During the course of the algorithm, we will adaptively enlarge the data set which is represented within the SVM problem.  Before the start of the k^th iteration, the set of data available to the algorithm will be denoted S_k-1.  In iteration k we will include additional data points denoted by the set s_k, so that |S_k| = |S_k-1| + |s_k|. The SCS algorithm consists of four important components:


  * Sequential function approximation. In many cases we have a fixed number of data points. However, our focus is on situations where the data can be queried from an oracle sequentially. Thus, we use adaptive sampling <cit.>, 


    min_αf_k(α) = 1/2⟨α, Q^k α⟩ + 1/|S_k|∑_i=1^|S_k|max{0,1-w_i ⟨α, Q_i^k ⟩}.



It is important to note that w_i and Q_i^k are based on realization of random variables (Z,W). Consequently, f_k is a realization of the expectation functional in  (<ref>).
In the remainder of this paper, we will let the kernel matrix Q^k agree with the size of data set (|S_k|) in any iteration.  


  * Direction finding. The idea here is inspired by Wolfe's non-smooth conjugate subgradient method which uses the smallest norm of the convex combination of the previous search direction (-d_k-1) and the current subgradient (g_k). More specifically, we first solve the following one-dimensional QP


    λ_k^* = _λ∈ [0,1]1/2||λ· (-d_k-1) + (1-λ) · g_k||^2.



Then we can set the new search direction as

    d_k=-[λ_k^* · (-d_k-1)+(1-λ_k^*) · g_k] := -Nr(G_k),



where G_k={-d_k-1,g_k}, and λ_k^* denotes the optimal weight for the convex combination.  Clearly if one fixes λ = 0, then the search direction reduces to that of the subgradient method.


  * Choice of step size. This is achieved by combining concepts of deterministic Wolfe's algorithm, stochastic trust region and line search methods <cit.>. The step actually applies Wolfe's line search criteria to the function which is a sampled realization of the expectation functional. Let g(t) ∈∂ f_k-1(α̂_k + t · d_k) and define the intervals L and R.

    L={t>0  |  f_k-1(α̂_k + t · d_k)-f_k-1(α̂_k) ≤ -m_1||d_k||^2 t },
        R ={t>0  |  0 > ⟨ g(t),d_k ⟩≥ -m_2||d_k||^2 }.

The above conditions are consistent with the line-search rules in Wolfe's deterministic algorithm <cit.>. The output of the step size will satisfy two metrics: (i) Identify a set L which includes points which reduce the objective function approximation f_k-1, and (ii) Identify a set R for which the directional derivative estimate for f_k-1 is improved. The algorithm seeks points which belong to L ∩ R. The combination of L and R is called strong Wolfe condition and it has been shown in <cit.> Lemma 1 that L ∩ R is not empty.


  * Termination criteria. The algorithm concludes its process when ||d_k||<ε and δ_k < δ_min. As we will show in Theorem <ref>, a dimunitive value of ||d_k|| indicates a small norm of the subgradient g ∈ f(α̂_k), fulfilling the optimality condition for an unconstrained convex optimization problem. Additionally, a threshold δ_min is established to prevent premature termination in the initial iterations. Without this threshold, the algorithm may halt prematurely if the initial samples are correctly classified by the initial classifier α_0, resulting in ||d_k|| < ε. However, if δ_min is introduced, based on Theorem <ref>, |S_k| should be chosen such that 

    |S_k| ≥ -8log(ε/2)·(M+1)^2/κ^2δ_min^4,
   

where κ is the κ-approximation defined in Definition <ref>. This effectively mitigates any early stopping concern with high probability.



Remark 1: (i) The initial choice of step size δ_0 may require some tuning depending on the data set but it does not affect the convergence property of this algorithm.
 
(ii) According to Theorem <ref>, δ_max := ε/ζ and 

    ζ = max{4nκ,η_2,4κ/C_1(1-η_1)},

 where C_1=m_1/2n, 1< n ∈ℤ^+.

 (iii) Note that replacing d_k with an arbitrary subgradient may not ensure descent unless the function f_k-1 is smooth. Thus, simply substituting d_k with g_k and using SGD-based method may not guarantee the convergence rate shown in Theorem 
 <ref>.

 (iv) In general, optimization problems are stated with the decision variables in a fixed dimension. However, since we have designed an adaptive sampling approach, the dimension of our decision variables grows iteratively according to the increase in sample size.  
 

 



Remark 2: Algorithm <ref> will have two possible outcomes. One is it returns a suitable step size t such that t ∈ L ∩ R and t ||d_k|| ≥δ_k/n. The other is it keeps shrinking the step size until t ||d_k|| < δ_k/n. This case is captured in step 11 of Algorithm 2 and the algorithm will return t = 0. Since α_k = α̂_k-1 + t_k d_k, t=0 means the decision variable will not be updated in this iteration. In the next iteration, Algorithm <ref> will generate more samples and these new samples may improve the search direction. This whole procedure creates a subroutine to first terminate the line-search algorithm and then improve the search direction by further sampling. This phenomenon is similar to retaining the previous incumbent solution in the stochastic decomposition algorithm <cit.>.

 


§ CONVERGENCE PROPERTIES


While the SCS algorithm and the stochastic trust region method share some similarities in their proofs of convergence, the specifics of these arguments are different because the former is based on direction-finding and the latter is based on trust regions. For example, the fully-linear property in stochastic trust region method plays no role in the convergence proof of this paper. Similarly, the sufficient decrease condition (Lemma <ref>) in our algorithm relies on Wolfe's line search algorithm while the condition used for a stochastic trust region method requires the Cauchy decrease property.   

In the following subsections, Theorem <ref> shows the sample complexity of the algorithm, Theorem <ref> and the subsequent corollary show that sufficient descent can be obtained and as a result, the stochastic sequence is a supermartingale. 
In <ref>, we apply the renewal reward theorem and optional stopping time theorem <cit.> to establish the  convergence rate of the SCS algorithm. The specific results are provided in Theorem <ref>. 

Notational Guide. We will adopt the convention that random variables will be denoted by capital letter (e.g., B) while a realization will be denoted by the corresponding lower case letter (e.g., b). Also, to avoid any notational confusion, please note that ℬ will denote a ball and C_i's are ordinary constants.



 §.§ Stochastic Local Approximations
 
In this subsection, we aim to demonstrate that with an adequate number of samples, the function f_k will serve as an “effective” stochastic local approximation of f. The criteria for what constitute “effective" in this context is provided in Definition <ref>, which provides a precise metric. Additionally, Definition <ref> extends this concept to a stochastic framework. 

   A function f_k is a local κ-approximation of f on ℬ(α̂_k,δ_k), if for all α∈ℬ(α̂_k,δ_k),

    |f(α)-f_k(α)| ≤κδ_k^2.











 Let Δ_k, Â_k denote the random counterpart of δ_k,  α̂_k, respectively, and ℱ_k denote the σ-algebra generated by {ℱ_i,F̂_i,
F̂_i+1}_i=0^k-1. Notice that this definition allows us to include the value of the so-called incumbent sequences (of Stochastic Decomposition <cit.>) as part of the σ-algebra. A sequence of random models {ℱ_k} is said to be μ_1-probabilistically κ-approximation of f with respect to {ℬ(Â_k,Δ_k)} if the event 

    I_k Δ={ℱ_k is κ- approximation of f on ℬ(Â_k,Δ_k)}


satisfies the condition 

    ℙ(I_k|ℱ_k-1) ≥μ_1 > 0.














Remark 2: (i) Unlike <cit.> which imposes a generic set of guidelines for stochastic line-search methods including requirements on the gradient approximation, our setup accommodates non-smoothness of the kernel SVM problems. For this reason, our requirements simply focus on sufficiently accurate function estimation without invoking gradient approximations.
(ii) In the interest of clarity, we emphasize that the incumbent decision α̂_k has its random counterpart denoted as Â_k.



Given the relation β = ∑_i=i^|S_k|α_iϕ(z_i), we can define the function f(α) as a transformation of g(β)

    g(β)=f(α):=1/2⟨α, Q^k α⟩ +  E[max{0,1-W ⟨α, Q_i^k(Z) ⟩}],


where Q_i^k(Z) = [⟨ϕ(z_1), ϕ(Z) ⟩, ⟨ϕ(z_2), ϕ(Z) ⟩,...,⟨ϕ(z_|S_k|), ϕ(Z) ⟩]. Similarly, we can also express f_k(α) in terms of g_k(β) as follows.


    f_k(α)=g_k(β):=1/2||β||^2 + 1/|S_k|∑_i=1^|S_k|max{0,1-w_i ⟨β, ϕ(z_i) ⟩}.


  
Assume that f and f_k are Lipschitz continuous with constant L_f, for any 0 < ε <1, κ > 4L_f/δ_min. If for all i, we have |⟨β, ϕ(z_i)⟩| ≤ M and 

    |S_k| ≥ -8log(ε/2)·(M+1)^2/κ^2δ_k^4,

then

    ℙ(|f_k(α)-f(α)| ≤κδ_k^2,  ∀α∈ℬ(α̂_k,δ_k) ) ≥ 1-ε.




First, for the point α̂_k, given that β̂_k = ∑_i=i^|S_k|α̂_k,iϕ(z_i), we have 

    f(α̂_k)=g(β̂_k)=1/2||β̂_k||^2+E [max{0,1-W ⟨β̂_k, ϕ(Z) ⟩}],
        f_k(α̂_k)=g_k(β̂_k)=
            1/2||β̂_k||^2 + 1/|S_k|∑_i=1^|S_k|max{0,1-w_i ⟨β̂_k, ϕ(z_i) ⟩}.


Thus, 


    f_k(α̂_k)-f(α̂_k)     =g_k(β̂_k)-g(β̂_k)
        =1/|S_k|∑_i=1^|S_k|max{0,1-w_i ⟨β̂_k, ϕ(z_i) ⟩}-E [max{0,1-W ⟨β̂_k, ϕ(Z) ⟩}].

Using the assumption that |⟨β, ϕ(z_i)⟩| ≤ M for all i, Hoeffding's inequality implies that

    ℙ(|f_k(α̂_k)-f(α̂_k)| ≤1/2κδ_k^2) ≥ 1-2exp (-κ^2δ_k^4 |S_k|^2/8 · |S_k| · (M+1)^2 ) ≥ 1-ε,

which indicates that when 

   |S_k| ≥-8log(ε/2)·(M+1)^2/κ^2δ_k^4, 
 
we have 

    ℙ(|f_k(α̂_k)-f(α̂_k)| ≤1/2κδ_k^2 ) ≥ 1-ε.


For any other α∈ℬ(α̂_k,δ_k), if |f_k(α̂_k)-f(α̂_k)| ≤1/2κδ_k^2, then


    |f_k(α)-f(α)|    ≤ |f_k(α)-f_k(α_k)|+|f_k(α_k)-f(α_k)|+ |f(α_k)-f(α)| 
       ≤ 2L_f ·δ_k + 1/2κδ_k^2
       ≤κδ_k^2,


where the second last inequality is due to the assumption that f_k and f are Lipschitz continuous and the last inequality is because κ>4L_f/δ_min>4L_f/δ_k. Thus, we conclude that if |S_k| satisfies Equation (<ref>), then 


    ℙ(|f_k(α)-f(α)| ≤κδ_k^2,  ∀α∈ℬ(α̂_k,δ_k) ) ≥ 1-ε.





 §.§ Sufficient Decreases and Supermartingale Property
 
This section establishes the stochastic variant of the sufficient decrease condition, known as the supermartingale property, in Theorem <ref>. The proof is segmented into four distinct cases, each based on whether the conditions of κ-approximation (as defined in Definition <ref>) and ε_F-accuracy (as outlined in Definition <ref>) are met or not. Lemmas <ref> - <ref> detail the outcomes for these various situations. Following these lemmas, the supermartingale property and its comprehensive proof are presented.

 The value estimates f̂_k Δ=f̂_k(α̂_k) and f̂_k+1Δ=f̂_k(α̂_k+td_k) are ε_F-accurate estimates of f(α̂_k) and f(α̂_k+td_k), respectively, for a given δ_k if and only if 

    |f̂_k-f(α̂_k)| ≤ε_F δ_k^2,    |f̂_k+1-f(α̂_k+td_k)| ≤ε_F δ_k^2.



  A sequence of model estimates {F̂_k,F̂_k+1} is said to be μ_2-probabilistically ε_F-accurate with respect to {Â_k,Δ_k,S_k} if the event 

    J_k Δ={F̂_k,F̂_k+1 are ε_F-accurate estimates  for Δ_k }

satisfies the condition 

    ℙ(J_k|ℱ_k-1) ≥μ_2 >0.



 
Suppose that f_k is a κ-approximation of f on ℬ(α̂_k ,δ_k) and C_1 = 1/8n, where n is specified in Algorithm <ref>. If 
    
    δ_k ≤1/16nκ ||d_k||


    and Algorithm <ref> returns a suitable step size t ∈ L and t ||d_k|| > δ_k/n, then we have 
    
    f(α̂_k+t d_k)-f(α̂_k) ≤ -C_1 ||d_k|| δ_k.



Since t ∈ L and t ||d_k|| > δ_k/n, we have 

    f_k(α̂_k+t d_k)-f_k(α̂_k) ≤ -m_1||d_k||^2 t ≤ -m_1/n||d_k||δ_k.

Also, since f̂_k is κ-approximation of f on ℬ(α̂_k,δ_k), we have 

    f(α̂_k+t d_k)-f(α̂_k) 
                 = f(α̂_k+t d_k)-f_k(α̂_k+t d_k)+f_k(α̂_k+t d_k)
       -f_k(α̂_k)+f_k(α̂_k)-f(α̂_k)
       ≤ 2κδ_k^2-m_1/n||d_k||δ_k 
       ≤ -C_1||d_k||δ_k,

which concludes the proof.


  Suppose that f_k is κ-approximation of f on the ball ℬ(α̂_k,δ_k) and the estimates (f̂_k,f̂_k+1) are ε_F-accurate with ε_F ≤κ. Moreover, if 

    δ_k ≤min{(1-η_1)C_1 ||d_k||/2κ, ||d_k||/η_2},


and Algorithm <ref> returns a suitable step size t ∈ L and t ||d_k|| > δ_k/n, then the k-th iteration will be successful (Update α̂_k ←β_k, and δ_k ←min{γδ_k-1,δ_max}).



    Since t ∈ L and t ||d_k|| > δ_k/n, we have
    
    f_k(α̂_k+t d_k)-f_k(α̂_k)  ≤ -m_1/n ||d_k|| δ_k ≤ -2C_1 ||d_k|| δ_k,

    
    where the last inequality holds because m_1 ≥1/4. Also, f_k is κ-approximation of f on ℬ(α̂_k,δ_k) and the estimates (f̂_k,f̂_k+1) are ε_F-accurate with ε_F ≤κ implies that 
    
    
    ρ_k     = f̂_k+1-f̂_k/f_k(α̂_k-t d_k)-f_k(α̂_k)
        = f̂_k+1-f(α̂_k+t d_k)/f_k(α̂_k+t d_k)-f_k(α̂_k)+f(α̂_k+t d_k)-f_k(α̂_k+t d_k)/f_k(α̂_k+t d_k)-f_k(α̂_k)+f_k(α̂_k+t d_k)-f_k(α̂_k)/f_k(α̂_k+t d_k)-f_k(α̂_k)
        + f_k(α̂_k)-f(α̂_k)/f_k(α̂_k+t d_k)-f_k(α̂_k)+f(α̂_k)-f̂_k/f_k(α̂_k+t d_k)-f_k(α̂_k),

    
which indicates that

    1-ρ_k ≤ 2κδ_k/C_1||d_k||≤ 1-η_1.


Thus, we have ρ_k ≥η_1, ||d_k||>η_2δ_k and the iteration is successful. 


  Suppose the function value estimates {(f̂_k,f̂_k+1)} are ε_F-accurate and

    ε_F < η_1 η_2 C_1.

If the kth iteration is successful, then the improvement in f is bounded such that 


    f(α̂^k+1)-f(α̂_k) ≤ -C_2δ_k^2,


where C_2 Δ= 2η_1 η_2 C_1-2ε_F.



    Since t ∈ L and t ||d_k|| > δ_k/n, we have
    
    f_k(α̂_k+t d_k)-f_k(α̂_k) ≤ -2C_1 ||d_k|| δ_k,

    Also, since the iteration is successful, we have ||d_k||>η_2 δ_k. Thus, we have
    
    
    f̂_k-f̂_k+1≥η_1(f(α̂_k)-f(α̂_k+td_k)) ≥ 2η_1 C_1 ||d_k||δ_k ≥ 2η_1 η_2 C_1 δ_k^2.

    Then, since the estimates are ε_F -accurate, we have that the improvement in f can be bounded as  
    
        f(α̂_k+t d_k)-f(α̂_k) ≤f(α̂_k+t d_k)-f̂_k+1+f̂_k+1-f̂_k+f̂_k-f(α̂_k) ≤-C_2 δ_k^2. 
    


  Let the random function V_k Δ= F_k+1-F_k, the corresponding realization be v_k, 


    ζ = max{4nκ,η_2,4κ/C_1(1-η_1)}   and   μ_1μ_2 > 1/2.


Then for any ε>0 such that ||d_k|| ≥ε and ζΔ_k ≤ε, we have 

    𝔼[V_k |  ||d_k|| ≥ε, ζΔ_k ≤ε] ≤ -1/2C_1 ||d_k|| Δ_k ≤ -θεΔ_k,

where θ = 1/2  C_1.
 


First of all, if kth iteration is successful, i.e. α̂_k+1=α̂_k-td_k we have 

    v_k = f(α̂_k+td_k)-f(α̂_k).


If kth iteration is unsuccessful, i.e. α̂_k+1=α̂_k we have


    v_k = f(α̂_k)-f(α̂_k)=0.


Then we will divide the analysis into 4 cases according to the states (true/false) observed for the pair (I_k, J_k).

(a) I_k and J_k are both true. Since the f_k is a κ-approximation of f on ℬ(α̂_k ,δ_k) and condition (<ref>) is satisfied, we have Lemma <ref> holds. Also, since the estimates (f̂_k,f̂_k+1) are ε_F-accurate and condition (<ref>) is satisfied, we have Lemma <ref> holds.  Combining (<ref>) with (<ref>), we have 


    v_k ≤  -C_1 ||d_k||δ_k Δ= b_1.




(b) I_k is true but J_k is false. Since f_k is a κ-approximation of f on ℬ(α̂_k ,δ_k) ∩ X and condition (<ref>) is satisfied, it follows that Lemma <ref> still holds. If the iteration is successful, we have (<ref>), otherwise we have (<ref>). Thus, we have
v_k ≤ 0.


(c) I_k is false but J_k is true. If the iteration is successful, since the estimates (f̂_k,f̂_k+1) are ε_F-accurate and condition (<ref>) is satisfied, Lemma <ref> holds. Hence, 

    v_k ≤ -C_2 δ_k^2.



If the iteration is unsuccessful, we have (<ref>). Thus, we have v_k ≤ 0 whether the iteration is successful or not.
     

(d) I_k and J_k are both false. Since f is convex and t||d_k||<δ_k, for any g(t) ∈∂ f(α̂_k+td_k), with  the assumption in <ref> which implies that ||g(t)|| is bounded, we have 


    v_k=f(α̂_k+td_k)-f(α̂_k) ≤ g(t)^T td_k ≤ C_3δ_k.


If the iteration is successful, then

    v_k ≤ C_3 δ_k Δ= b_2 .


If the iteration is unsuccessful, we have (<ref>). Thus, we have v_k ≤  b_2 whether the iteration is successful or not. 


With the above four cases, we can bound 𝔼[V_k |  ||d_k|| ≥ε, ζΔ_k ≤ε] based on different outcomes of I_k and J_k. Let B_1 and B_2 be the random counterparts of b_1 and b_2.  Then we have 

    𝔼[V_k |  ||d_k|| ≥ε, ζΔ_k ≤ε] 
       ≤μ_1 μ_2 B_1 + (μ_1(1-μ_2)+μ_2(1-μ_1))· 0 + (1-μ_1)(1-μ_2) B_2 
        = μ_1 μ_2  (- C_1 ||d_k||Δ_k) + (1-μ_1)(1-μ_2)C_3 Δ_k.


Choose μ_1 ∈ (1/2,1) and μ_2 ∈ (1/2,1) large enough such that 


    μ_1 μ_2 ≥ 1/2   and   μ_1 μ_2/(1-μ_1)(1-μ_2)≥2 C_3/c_1||d_k||,


we have

    
    𝔼[V_k |  ||d_k|| ≥ε, ζΔ_k ≤ε]     ≤- 1/2 C_1 ||d_k||Δ_k. 
    



A quick observation of the concluding condition reveals that the requirement Δ_k ≤ε/ζ implies that the supermartingale property holds. This prompts us to impose the condition that restricts δ_max = ε/ζ. This property is formalized in the following corollary. 
 Let 

    T_ε = inf{ k ≥ 0: ||d_k|| < ε}.


Then T_ε is a stopping time for the stochastic process Â^k. Moreover, conditioned on T_ε≥ k, F_k is a supermartingale. 





    
    From Theorem <ref>, we have
    
    𝔼[F_k | ℱ_k-1, T_ε  > k] ≤ F_k-1 - ΘεΔ_k.



Hence, F_k is a supermartingale. 




 §.§ Convergence Rate
 

Building upon the results established in Theorem <ref>, where f_k is demonstrated to be a supermartingale and T_ε is identified as a stopping time, we proceed to construct a renewal reward process for analyzing the bound on the expected value of T_ε.
As highlighted in the abstract, Theorem <ref> confirms the rate of convergence to be O(1/ε^2).
To begin with, let us define the renewal process { A_l } as follows: set A_0 = 0, and for each l > 0, define A_l = inf{ m > A_l-1 : ζΔ_m ≥ε}, with ζ being specified in (<ref>). Additionally, we define the inter-arrival times τ_l = A_l - A_l-1. Lastly, we introduce the counting process N(k) = max{n: A_l ≤ k}, representing the number of renewals occurring up to the k^th iteration.


  Let 1/2 < p=μ_1 μ_2 ≤ℙ(I_k ∩ J_k). Then for all l ≥ 1,


    𝔼[τ_l] ≤p/2p-1.



First,
    
    𝔼[τ_l]     = 𝔼[τ_l | ζΔ_A_l-1 > ε] ·ℙ(ζΔ_A_l-1 > ε) +  𝔼[τ_l | ζΔ_A_l-1 = ε] ·ℙ(ζΔ_A_l-1 = ε)
       ≤max{𝔼[τ_l | ζΔ_A_l-1 > ε], 𝔼[τ_n | ζΔ_A_l-1 = ε] }.

    
If ζΔ_A_l-1 > ε, according to algorithm <ref>, 


    𝔼[τ_l | ζΔ_A_l-1 > ε] = 1.


If ζΔ_A_l-1 = ε, by Theorem <ref>, we can treat {Δ_A_l-1,...,Δ_A_l} as a random walk, and we have 


    𝔼[τ_l | ζΔ_A_l-1 = ε] ≤p/2p-1.


Combining (<ref>) and (<ref>) completes the proof. 



  Let ζ and θ be the same as in Theorem <ref> and Δ_max = ε/ζ, then

    𝔼[N(T_ε)] ≤2 ζ F_max/θε^2+ζΔ_max/ε = 2 ζ F_max/θε^2+1,


where F_max=max_i ≤ (k ∧ T_ε)|F_i|.



    We will first show that 
    
    R_k ∧ T_ε =F_k ∧ T_ε + Θε∑_j=0^k ∧ T_εΔ_j

is a supermartingale. Using (<ref>),


    𝔼[R_k ∧ T_ε|ℱ_k-1]     = 𝔼[F_k ∧ T_ε|ℱ_k-1] + 𝔼[Θε∑_j=0^k ∧ T_εΔ_j | ℱ_k-1 ]
       ≤ F_k-1 - ΘεΔ_k + 𝔼[Θε∑_j=0^k ∧ T_εΔ_j | ℱ_k-1 ] 
        = F_k-1 + 𝔼[Θε∑_j=0^(k-1) ∧ T_εΔ_j | ℱ_k-1 ] 
        = F_k-1 + Θε∑_j=0^(k-1) ∧ T_εΔ_j 
        = R_(k-1) ∧ T_ε,


where the summation in the last expectation in (<ref>) is true by moving ΘεΔ_k inside the summation so that it has one less term if k < T_ε.

If k < T_ε, then 


    |R_k ∧ T_ε| = |R_k| ≤ F_max + Θε k Δ_max.


If k ≥ T_ε, then 


    |R_k ∧ T_ε| = |R_ε| ≤  F_max + Θε T_εΔ_max.


This is also bounded almost surely since T_ε is bounded almost surely. Hence, according to (<ref>) and the optional stopping theorem <cit.>, we have


    𝔼[Θε∑_j=0^ T_εΔ_j] ≤𝔼[R_T_ε] + F_max≤𝔼[R_0] + F_max≤ 2 F_max + ΘεΔ_max.


Furthermore, since the renewal A_n happens when ζΔ_j ≥ε and N(T_ε) is a subset of {0,1,2,...,T_ε}, we have 


    Θε(∑_j=0^ T_εζΔ_j) ≥Θε( N(T_ε) ε).


Combining (<ref>) and (<ref>), we have 
    𝔼[N(T_ε)] ≤2 ζF_max + ζΘεΔ_max/Θε^2 ≤2 ζF_max /Θε^2+ζΔ_max/ε.
      



Under conditions enunciated in <ref>, we have
    
    𝔼[T_ε] ≤p/2p-1(2 ζ F_max/Θε^2+2).




First, note that N(T_ε)+1 is a stopping time for the renewal process { A_n: n ≥ 0}. Thus, using Wald's equation (inequality form) <cit.>, we have 
    

    𝔼[A_N(T_ε)+1] ≤𝔼[τ_1] 𝔼[N(T_ε)+1].

    
Moreover, since A_N(T_ε)+1≥ T_ε, we have


    𝔼[T_ε] ≤𝔼[τ_1] 𝔼[N(T_ε)+1].


Hence, by Lemma <ref> and Lemma <ref>

    𝔼[T_ε] ≤p/2p-1(2 ζF_max /Θε^2+2). 





 §.§ Optimality Condition

Theorem <ref> demonstrates that if ||d_k||<ε, the ε-optimality condition for f(x) will be satisfied. The error bound on this optimality is linked to both the sample size and the subgradient of f_k(x). Readers seeking the in-depth proof of Theorem <ref> can refer to Appendix <ref>.

 
    Let d^* = - _g ∈∂ f(α̂_k) ||g||. If k is the smallest index for which ||d_k|| < ε and |S_k| ≥-2(ε')^2/L^2logδ, then 
    
    P(||d^*|| < 4 ε + ε') ≥ 1-δ.





§ PRELIMINARY COMPUTATIONAL RESULTS
 




























    
    
    










 
  










Our computational experiments are based on data sets available at the UCI Machine Learning Repository <cit.>. Our study considers three different methods: Wolfe's algorithm <cit.>, kernel Pegasos algorithm[The computations below are based on our implementation of the Pegasos Algorithm.] and the SCS algorithm. Wolfe's algorithm uses a deterministic objective function (i.e., the SAA) and is treated as a benchmark, while Pegasos and SCS use (<ref>) as the objective function.
However, in order to compare the progress of the objective function  values for different algorithms, we will track the values of (<ref>) for all three algorithms. The values of {f(α̂_k)}_k=1^k=50 and {f(α̂_k)}_k=-50^k=-1 are shown in Figures <ref> and <ref>, where the notation k = 1, …, 50, and k=-50, …, -1 represent the solution value in the first fifty, and last fifty (respectively) iterations of each algorithms. The algorithms were implemented on a MacBook Pro 2019 with a 2.6GHz 6-core Intel Core i7 processor and a 16GB of 2400MHz DDR4 onboard memory.







Remark 3: (i) The objective function values validated for  the SCS algorithm are lower than those for the Pegasos algorithm. (ii) The SCS algorithm also converges faster than the Pegasos Algorithm. (iii) The beauty of the Pegasos algorithm is that it provides a decision rule without providing a solution to (<ref>). This decision rule provides a classifier without any reference to optimality. On the other hand, the SCS algorithm is truly an optimization algorithm  with a stopping rule in which the steps (and stopping) are guided by d_k (||d_k||<ε).

The performance of all three algorithms for different data sets are given in Table <ref>. To make the experiment more persuasive, we divide the data into training and testing parts. We obtain a classifier using the training data and apply such classifiers in the testing data set to estimate the accuracy. Here, accuracy reflects the fraction of total number of data points that have been correctly classified.  
Since Wolfe's algorithm requires the entire data set, instances with very large data sets are beyond its reach and are reported by N/A in Table <ref>. The accuracy and training time reported in the Table <ref> are based on the average of 20 independent runs for each pair (data, algorithm) using different random seeds.  









Remark 4: Conventional wisdom in ML suggests that SFO methods provide the best balance between scalability and accuracy of stochastic optimization problems. However from Table <ref>, we can see that this general impression may not only be questionable but also encourage an over-emphasis on SFO methods. In fact, the SCS algorithm performs better than the Pegasos algorithm both in accuracy and in computational speed, especially for the larger data sets in our experiment. Specifically, for the four largest data sets in our experiment, the computational times and accuracy reported for the SCS algorithm are superior to those obtained using Pegasos. Thus, the SCS algorithm gives an alternative approach which is quite competitive for Kernel SVM problems.    







§ CONCLUSION

Our approach leads to a class of online algorithms that go beyond first-order approximations and incorporates stochastic  analysis of stopping times in a direction-finding method. Specifically, it includes several features of Wolfe's method (e.g., stability and good convergence rate) as well as online aspects of Pegasos. This combination promotes improvements on both deterministic and stochastic methods that have been previously used for SVM. In contrast to the Pegasos algorithm, we find that the optimization performance of the SCS algorithm is more reliable and provides consistently lower objective values (see Figures <ref> and <ref>) in out-of-sample validation. It appears that such reliability may be difficult to achieve using first-order methods without additional fine-tuning. Although our experimental results are based on some fixed data sets, the SCS approach can be effective for classification of streaming data as well. Finally, we should emphasize that although the SCS algorithm in this paper is specifically designed for kernel SVM, it can be generalized to other non-smooth stochastic convex optimization problems, as well as constraimed problems using primal-dual methods similar to <cit.>. We also conjecture that many of the asymptotic properties (e.g., asymptotic normality) of SGD methods may also be studied for the SCS algorithm <cit.>.  


Acknowledgment. This paper is supported, in part, by a grant from AFOSR FA9550-20-1-0006.  The final phase of this paper was funded via ONR grant N00014-20-1-2077.






siamplain






§ PROOF OF OPTIMALITY CONDITION
 
In this section, we want to prove that ||d_k|| < ε implies that the ε-optimality condition of f(α̂_k), as stated in Theorem <ref>. The error bound on optimality is related to the sample size as well as the the subgradient of f_k(α̂_k). First of all, we will define a few well-known notations.


    
  * d^* = - _g ∈∂ f(α̂_k) ||g||: negative of the smallest norm subgradient of f(α̂_k).
    
  * d_k^* = - _g ∈∂ f_k(α̂_k) ||g||: negative of the smallest norm subgradient of f_k(α̂_k).
    
  * d_k = -Nr({g_k,-d_k-1}), where g_k ∈∂ f_k(α̂_k).
    
  * Subdifferential of f at z': ∂ f(z') = {q: f(z) ≥ f(z') + q^T (z-z')}.
    
  * ε-subdifferential of f at z': ∂ f_ε(z') = {q: f(z) ≥ f(z') + q^T (z-z')-ε}.
    
  * Directional derivative of f at z in direction d: f'(z;d)=max{d^Tq: q ∈∂ f(z)}.
    
  * ε-directional derivative of f at z in direction d: f_ε'(z;d)=max{d^Tq: q ∈∂_ε f(z)}.
    
  * 𝒳(z) := -min_||d|| = 1 f'(z;d) and 𝒳_k(z) := -min_||d|| = 1 f_k'(z;d).
    
  * 𝒳_ε(z) := -min_||d|| = 1 f_ε'(z;d) and d̅_k(z) = - _||d|| = 1 f_ε'(z;d).


 
    Let the tuple (m_1, m_2) satisfy the requirement in Algorithm <ref> (1/4≤ m_1 < 1/2 and 1/4≤ m_2 < m_1). If k is the smallest index for which ||d_k|| < ε, then we have ||d_k^*|| < 4 ε.



    Suppose the claim is false, then by the definition of d_k^*, we have ||g_k|| ≥ ||d_k^*|| ≥ 4ε. Thus,
    
    ||d_k||^2     = ||λ_k^* g_k-(1-λ_k^*)d_k-1||^2
        = (λ_k^*)^2||g_k||^2+(1-λ_k^*)^2||d_k-1||^2-2λ_k^*(1-λ_k^*)⟨ g_k,d_k-1⟩
       ≥ (λ_k^*)^2||g_k||^2+(1-λ_k^*)^2||d_k-1||^2


where the inequality holds because Algorithm <ref> ensures the condition R in (<ref>). 

We will now divide the analysis into 2 cases which are examined below: a) ||g_k|| > ||d_k-1|| ≥ε and ||g_k|| ≥ 4 ε  and b) ||d_k-1|| ≥ ||g_k|| ≥ 4 ε. 

(a) In this case from (<ref>) we have 

    ||d_k||^2 ≥(17 (λ_k^*)^2 -2 λ_k^* + 1 ) ε^2


Note that

    λ_k^*=⟨ g_k,d_k-1⟩ + ||g_k||^2/||d_k-1||^2+||g_k||^2+2⟨ g_k,d_k-1⟩≥-m_2||d_k-1||^2+||g_k||^2/||d_k-1||^2+||g_k||^2,


where the inequality holds because the R condition in (<ref>) ensures 0 > ⟨ g_k,d_k-1⟩≥ - m_2 ||d_k-1||^2. Thus, we claim that λ_k^* ≥2/17 because it is suffice to show


    - 17 · m_2||d_k-1||^2+17 · ||g_k||^2 ≥ 2 ||d_k-1||^2 + 2 ||g_k||^2.


This can be verified by observing m_2 < m_1 < 1/2 and ||g_k|| > ||d_k-1||. Thus, based on (<ref>) and λ_k^* ≥2/17, we have
||d_k|| ≥ε. This contradicts the assumptions of the lemma. 

(b) If ||d_k-1|| ≥ ||g_k|| ≥ 4 ε, then we have 

    ||d_k||^2 ≥(32 (λ_k^*)^2 -32 λ_k^* + 16 ) ε^2 ≥ 8 ε^2.


Thus, ||d_k|| ≥ 2 √(2)ε, which also contradicts the assumptions of the lemma.


 
    ||d^*|| = 𝒳(α̂_k) and ||d_k^*|| = 𝒳_k(α̂_k).



    According to the definition of 𝒳(α̂_k),
    
    𝒳(α̂_k)=-min_||d|| = 1 f'(α̂_k;d) = -min_||d|| = 1max_q ∈∂ f(α̂_k)d^Tq


Note that the min-max  problem (<ref>) has a stationary point


    q̅=_q ∈∂ f(α̂_k) ||q||,   d̅ = - q̅/||q̅||.


Thus, 𝒳(α̂_k) = ||q̅|| = ||d^*||. Similarly, we can also prove that 𝒳_k(α̂_k) = ||d_k^*||.



 
    If |S_k| ≥-2(ε')^2/L^2logδ, then P(𝒳_ε(α̂_k)-𝒳_k(α̂_k) ≤ε') ≥ 1-δ.



By definition, d̅_k(α̂_k) = - _||d|| = 1 f_ε'(α̂_k;d). Hence,
    
    𝒳_ε(α̂_k)-𝒳_k(α̂_k)
               ≤1/|S_k|∑_i=1^|S_k| h'(α̂_k;d̅_k(α̂_k),ω_i)-E[h_ε'(α̂_k;d̅_k(α̂_k),ω)]
        = 1/|S_k|∑_i=1^|S_k| [h'(α̂_k;d̅_k(α̂_k),ω_i)-h_ε'(α̂_k;d̅_k(α̂_k),ω_i)]
        + 1/|S_k|∑_i=1^|S_k| h_ε'(α̂_k;d̅_k(α̂_k),ω_i) - E[h_ε'(α̂_k;d̅_k(α̂_k),ω)]

Note that for each ω_i, we have h'(α̂_k;d̅_k(α̂_k),ω_i) ≤ h_ε'(α̂_k;d̅_k(α̂_k),ω_i), since ∂ h(α̂_k,ω_i) ⊆∂ h_ε(α̂_k,ω_i). Also, by Hoeffding's inequality, if |S_k| ≥-2(ε')^2/L^2logδ, then 


    P(1/|S_k|∑_i=1^|S_k| h_ε'(α̂_k;d̅_k(α̂_k),ω_i) - E[h_ε'(α̂_k;d̅_k(α̂_k),ω)] ≤ε') ≥ 1-δ,


which concludes the proof.


 (Theorem <ref>)
    First, by Theorem <ref> and Lemma <ref>, if |S_k| ≥-2(ε')^2/L^2logδ, then with probability at least 1-δ,
    
    ||d^*||-||d_k^*||=lim_ε→ 0𝒳_ε (α̂_k) -𝒳_k(α̂_k) ≤ε'.

    Combining with Lemma <ref>, we have
    
    P (||d^*|| < 4 ε + ε' ) ≥ 1 - δ.






§ NOTATION TABLES











 Paper Title: Decentralized and Uncoordinated Learning of Stable Matchings: A Game-Theoretic Approach.





Decentralized and Uncoordinated Learning of Stable Matchings: A Game-Theoretic Approach
    S. Rasoul Etesami† and R. Srikant

†Department of Industrial and Systems Engineering, Department of Electrical and Computer Engineering, and Coordinated Science Laboratory, University of Illinois Urbana-Champaign,  Urbana, IL 61801 (Email: etesami1@illinois.edu).
Department of Electrical and Computer Engineering and Coordinated Science Laboratory, University of Illinois Urbana-Champaign,  Urbana, IL 61801 (Email: rsrikant@illinois.edu).

    August 5, 2024
=============================================================================================================================================================================================================================================================================================================================================================================================================================================================


  







           
§ ABSTRACT 
We consider the problem of learning stable matchings in a fully decentralized and uncoordinated manner, where decentralized means that players make decisions individually without the influence of a central platform, and uncoordinated means that players do not need to synchronize their decisions using pre-specified rules. In this problem, there are n men and n women, each having preference over the other side. It is assumed that women know their preferences over men, but men are not aware of their preferences over women, and they only learn them if they propose and successfully get matched to women. A matching is called stable if no man and woman prefer each other over their current matches. When all the preferences are known a priori, the celebrated Deferred-Acceptance (DA) algorithm proposed by Gale and Shapley provides a decentralized and uncoordinated algorithm to obtain a stable matching. However, when the preferences are unknown, developing such an algorithm faces major challenges due to a lack of coordination. In this work, we achieve this goal by making a connection between stable matchings and learning Nash equilibria (NE) in noncooperative games. First, we provide a complete information game formulation for the stable matching problem with known preferences such that its set of pure NE coincides with the set of stable matchings, while its mixed NE with full support can be rounded in a decentralized manner to a stable matching. Relying on such a game-theoretic formulation, we show that for hierarchical structured markets, adopting the exponential weight (EXP) learning algorithm for the stable matching game achieves logarithmic regret with polynomial dependence on the number of players, thus answering a question posed in previous literature. Moreover, we show that the same EXP learning algorithm converges locally and exponentially fast to a stable matching in general matching markets. We complement this result by introducing another decentralized and uncoordinated learning algorithm that globally converges to a stable matching with arbitrarily high probability, leveraging the weak acyclicity property of the stable matching game. Since the global convergence of this algorithm can be slow in general matching markets, we investigate conditions under which stable matchings can be learned more quickly. In particular, for general matching markets, we show that if men receive stronger feedback when their proposals are rejected, it is possible to drive the market more rapidly toward an approximate stable matching. Our proposed game-theoretic framework bridges the discrete problem of learning stable matchings with the problem of learning NE in continuous-action games. 


Learning stable matchings, learning Nash equilibrium, weakly acyclic games, monotone games, decentralized learning, uncoordinated learning, online mirror descent, regret minimization. 


Stable matchings constitute one of the fundamental problems in computer science, economics, and engineering that have received considerable attention in the past decades. Stable matchings provide a desirable notion of stability in two-sided matching markets where the agents on each side of the market have preferences over the other side. For instance, in college admission, the applicants have different preferences over colleges and vice versa, and the goal is to match the applicants to the colleges so that no applicant-college pair would prefer to break their current matches and instead get matched to each other. Similar situations arise in other applications, such as kidney exchange programs, assigning jobs to workers, matching in online dating platforms, and scheduling jobs on heterogeneous machines. 

It is known that when all the preferences are known, stable matchings always exist and a simple decentralized and uncoordinated Deferred-Acceptance (DA) algorithm, first proposed by <cit.>, can find one such stable matches in polynomial number of iterations. Here, by a decentralized algorithm we refer to an algorithm in which players make decisions individually without the influence of a central platform, and uncoordinated means that players do not need to synchronize their decisions using pre-specified rules. While the DA algorithm provides a satisfactory solution to many practical applications, there are scenarios where either the information structure of the problem hinders the implementation of the DA algorithm or the stable matching achieved by the DA algorithm is inferior in terms of social optimality, especially when there exist many feasible stable matchings in the market. For instance, there are many recent emergence of online matching markets such as online labor markets (e.g., TaskRabbit, Upwork), online dating markets (e.g., Tinder, Match.com), and online crowdsourcing platforms (e.g., Amazon mechanical turk), where the users do not know their preferences a priori, and can repeatedly interact with the market to improve their matching quality <cit.>. The more a pair on both sides of the market gets matched, the more certain they will be about their preferences. Therefore, a question of interest is how the agents should interact with the market such that in the absence of any coordination, they will learn their preferences as quickly as possible, and their collective behavior results in a stable matching.

In this work, we consider a two-sided matching market posed as a marriage problem, where we refer to the agents of one side as men and to the agents of the other side as women. We assume both men and women have preferences on the other side and that women are aware of their ordinal preferences about men. However, men do not know their preferences about women and only learn those preferences if they propose and get matched, in which case the matched men receive a noisy estimate of their preferences. We assume that men/women cannot observe any other information (e.g., who is rejected/accepted), nor can they coordinate in any sense. The goal for men is to follow a decentralized and uncoordinated proposal strategy such that the entire market converges to a stable matching over time. 

One of the major challenges for learning a stable matching in markets with unknown preferences is the collisions. More precisely, when multiple men propose to the same woman, only one of them gets matched and receives useful information; all others get rejected and receive no information regarding their preferences for that woman. Therefore, resolving such collisions without coordination is a major issue. Moreover, even if a man gets matched to a woman, he receives a noisy utility drawn from an unknown distribution that could differ from his true preference. Thus, a collision-avoidance process should be repeated many times until men can accurately estimate their true preferences. One of our goals in this work is to provide decentralized and uncoordinated algorithms that can effectively learn the true preferences while incurring a minimum number of collisions. Moreover, our work provides an alternative approach for measuring the closeness of the generated dynamics to any stable matching in general matching markets and not necessarily to a specific stable matching in structured markets, which has been the main focus in the past literature. To that end, we make a connection between learning stable matchings in general markets with unknown preferences and learning Nash equilibrium (NE) in noncooperative games and leverage this connection to devise decentralized and uncoordinated algorithms for learning stable matchings.      



 §.§ Related Work


Stable matching was first introduced by <cit.> as a model for college admissions and to study the stability of marriages. Since then, there has been a tremendous effort on generalization and extension of stable matchings to various market settings <cit.>. In their seminal work, Gale and Shapley provided a simple DA algorithm in which men propose to their most preferred women, and the women reject all the received proposals except the most preferred one. They show that such a decentralized algorithm converges to a stable matching in at most O(n^2) steps, where n is the total number of men/women in the market. Unfortunately, when the agents' preferences are unknown, the DA algorithm is no longer guaranteed to converge to a stable matching without extra coordination. More precisely, if men fail to propose to their highest preferred women (due to a lack of knowledge of their true preferences), their subsequent proposals may become unpredictable. One reason is that if a man m is rejected by a woman w at a time instance t, that does not mean that m should stop proposing to w because it could be the case that another man mistakenly proposed to w at time t, hence causing the rejection of m by w. Therefore, in order to extend the DA algorithm to the case of unknown preferences, one requires some sort of coordination among the agents to avoid uninformative acceptance/rejection patterns. To address this issue, the earlier works <cit.> have proposed various coordination-based DA algorithms, where the main idea is to develop a phase-dependent process where in some phases the agents mostly explore, and in other phases, they implement the DA algorithm based on the current estimates of their unknown preferences.  


It is known that stable matchings with known preferences can be characterized using extreme points of a fractional polytope that is totally dual integral <cit.>. Moreover, stable matchings exhibit other nice properties that allow one to characterize their geometry using so-called rotations <cit.>. It was shown in <cit.> that uncoordinated random better response dynamics converge to a stable matching with probability one. Subsequently, <cit.> provided an exponential lower bound for the worst-case convergence time of the uncoordinated random better/best response dynamics to a stable matching. However, all these results are for the case when the market preferences are fully known so that the agents can compute their best/better responses and update their decisions accordingly. 

More recently, there has been a huge interest in matching markets with unknown preferences. However, depending on the information structure and the feedback received by the agents, one would expect a wide range of performance guarantees, which also depends on the type of algorithms followed by the agents (e.g., centralized/decentralized or coordinated/uncoordinated). The work <cit.> devised a randomized polynomial-time centralized algorithm for finding a stable matching with unknown deterministic preferences, where each time, the algorithm proposes a matching and receives feedback in terms of a blocking pair.[A unmatched pair of man-woman (m,w) is called blocking if both m and w prefer each other to their matched partner.] The work <cit.> considers learning a specific stable matching with unknown stochastic preferences by adopting a suitable notion of stable regret that measures the number of time instances that men propose to women other than their stable pair in that stable matching. <cit.> considered the problem of learning stable matchings in hierarchically structured markers[In fact, the hierarchical condition (a.k.a. sequential preference condition <cit.>) that we consider in Section <ref> is weaker than the α-reducible condition considered in   <cit.>, and our results hold for broader matching markets including the ones considered in <cit.>.] in which each submarket has a fixed pair (i.e., a pair of men-women that prefer each other the most in that submarket), and developed a phase-coordinated decentralized algorithm that achieves logarithmic regret with respect to the market's unique stable matching. However, the regret bound in <cit.> depends exponentially on other parameters such as the number of men/women. Furthermore, <cit.> considered learning stable matchings in an uncoordinated and decentralized fashion. However, they require stronger assumptions on the information feedback (e.g., a player observes the actions of other players in the previous round) and use a different performance metric than the one we consider in this work. We refer to <cit.> and <cit.> for other learning algorithms in two-sided matching markets with different performance guarantees. While all these works essentially address the same problem of learning a stable matching with unknown preferences, their algorithms differ substantially due to the information/feedback structure, the type of performance guarantee, and the amount of allowable coordination among the agents. 



 §.§ Contributions and Organization


We consider the problem of learning stable matchings in matching markets, where men do not know their preferences and want to achieve a stable matching in a decentralized and uncoordinated fashion. In particular, we consider the weakest type of feedback that men can receive: a man gets to observe a noisy version of his true preference only if his proposal gets accepted and receives no information otherwise. Our main objective is to provide a novel and principled approach to designing decentralized and uncoordinated learning algorithms for stable matchings through the lens of NE learning in noncooperative games. Our contributions can be summarized as follows:

    
  * We first provide a complete information continuous-action game formulation for the stable matching problem and show that its set of pure NE coincides with the set of stable matchings while its mixed NE points can be rounded in a decentralized way to a stable matching. This connection provides an alternative approach for measuring the closeness of the market dynamics to stable matchings through the lens of NE computation in games, hence extending the conventional notion of regret with respect to a specific stable matching to any arbitrary stable matching in general markets.
    
  * Leveraging this game-theoretic formulation, we provide a simple decentralized and uncoordinated algorithm EXP that converges globally to a stable matching in hierarchical markets and achieves logarithmic regret with only polynomial dependence on the number of players, hence improving the existing bounds in the past literature.
    
  * We then show that the EXP algorithm converges locally exponentially fast to a stable matching in general matching markets. In particular, we show that if EXP converges with positive probability, it converge globally to a stable matching.[Perhaps fast local convergence is the best one can hope for general markets because, for general matching markets, there are strong exponential lower bounds for the number of iterates of the better response dynamics to converge globally to a stable matching.]     
    
  * We also provide an alternative decentralized and uncoordinated algorithm that globally converges to a stable matching in general markets with arbitrarily high probability (albeit at a low convergence rate), thanks to the weakly acyclic property of the stable matching game. 
    
  * Finally, we address an information feedback design problem by seeking conditions in general matching markets that allow players to learn a stable matching quickly. In this regard, we leverage our game-theoretic framework to provide one such information feedback by establishing a connection to monotone games. Our results offer new insights into designing payoff functions that enable faster decentralized and uncoordinated algorithms for driving matching markets to their stable points. 




The paper is organized as follows. In Section <ref>, we formally introduce the problem. In Section <ref>, we provide a complete information game-theoretic formulation for the stable matching problem and prove several characterization results for its set of NE points. In Section <ref>, we provide a decentralized and uncoordinated algorithm (EXP) that achieves logarithmic regret for hierarchical markets with polynomial dependence on the number of players. In Section <ref>, we show that EXP  converges locally exponentially fast to a stable matching in general matching markets. In Section <ref>, we provide an alternative decentralized and uncoordinated algorithm that globally converges to a stable matching in general matching markets with arbitrarily high probability. In Section <ref>, we extend our results to matching markets with stronger feedback and establish a connection between fast global learning of a stable matching and learning a NE in monotone games. We conclude the paper by identifying some future research directions in Section <ref>.




§ PROBLEM FORMULATION




 §.§ Stable Matchings with Known Preferences


Here, we first introduce the stable matching problem with known preferences <cit.>. In this problem, there are a set M of men and a set W of women such that |M|=|W|=n (we use the term “agents" to refer to either men or women).[By introducing dummy agents with appropriate preferences, without loss of generality we may assume |M|=|W| <cit.>.] Each man m∈ M has a cardinal preference for each women w∈ W, denoted by μ_mw∈ (0, 1], such if μ_mw>μ_mw', it means that m prefers w over w'. Moreover, each woman w has an ordinal preference over the men, and we write m>_w m' if woman w strictly prefers m over m'. In this work we assume that no agent has ties in their preferences and we define 

    Δ=min_m, w≠ w'|μ_mw-μ_mw'|,    μ_min=min_m,wμ_mw,    μ_max=max_m,wμ_mw.

In particular, we note that Δ>0 and μ_min>0.

Given a matching and two matched pairs (m,w) and (m',w'), we say that (m,w') forms a blocking pair if μ_mw'>μ_mw and m>_w' m'. In other words, (m,w') is a blocking pair if both m and w' prefer each other over their current matches. A perfect matching is called a stable matching if it does not contain any blocking pair. 

 



 §.§ Stable Matchings with Unknown Preferences


In this work, we consider the problem of finding a stable matching with the main difference that men do not know their true preferences μ_mw, and they only get to learn them through interactions with the market. More precisely, for any m∈ M, w∈ W, we assume that the preference of man m about woman w is in the form of a [0,1]-supported unknown distribution 𝒟_mw with unknown mean μ_mw>0. We assume men and women interact in this market through a discrete-time process, where at any time t=0,1,2,… that a man m proposes to a woman w, he receives a feedback in the following form:

    
  * If the proposal of m gets rejected at time t because woman w has received an offer from a more preferred man m', i.e., m'>_w m, then m receives no information as feedback other than the fact that he was rejected by woman w at time t. 
    
  * If the proposal of m gets accepted by w at time t, then m observes an i.i.d. realization of his sampled preference drawn from 𝒟_mw, denoted by μ̂^t_mw.    


In the stable matching problem with unknown preferences, the agents' goals are to interact with the market through the information feedback structure described above such that the emerging dynamics converge to a stable matching of the market with known preferences {μ_mw}. We note that in the discrete-time process described above, men are the decision makers on whom to propose at each time t;  women merely respond to men's decisions by accepting their most preferred proposal (if they received any) and rejecting all others. 
       





§ A GAME-THEORETIC FORMULATION AND PRELIMINARY RESULTS


In this section, we provide a complete information noncooperative game formulation for the stable matching problem with known preferences whose set of pure Nash equilibrium (NE) points coincides with the set of stable matchings. Later, we will show how to leverage this formulation to extend our results for learning stable matchings with unknown preferences. Such a formulation has three main advantages: i) it reduces the learning task in matching markets to learning NE in continuous-action concave games, ii) it captures the selfish behavior of men (players) and the feedback they receive through their payoff functions, and iii) it simplifies the combinatorial structure inherent in learning stable matchings to learning NE in continuous-action games. While analyzing the agents' decisions directly on the discrete space of matchings seems complicated, which is why some earlier works need to restrict attention to special market structures <cit.>, our game-theoretic reduction simplifies this task to some extent, thanks to the extensive literature on NE learning in concave games.

Consider a complete information noncooperative game in which the action set of each man (player) is given by the set of women W, and the set of (mixed) strategies for each player is given by the probability simplex over the set of women W, i.e., the strategy set of player m is defined by 
    𝒳_m={x_m∈ℝ_+^|W|: ∑_w∈ Wx_mw= 1 }.
 
Moreover, given a strategy profile x=(x_m,x_-m), we define the payoff of player m by [Here, x_-m refers to the strategy vector of all the players except the mth one.]

    u_m(x)=∑_w∈ W(μ_mw∏_k>_w m(1-x_kw))x_mw,

where μ_mw>0 is the utility (true preference) received by man m if he gets matched to women w.


A strategy x_m for player m is called pure if it is zero in all coordinates except one. Otherwise, it is called a mixed strategy. A pure (mixed) strategy profile x^* is called a pure (mixed) NE if for any player m and any pure (mixed) strategy x_m, we have u_m(x_m,x^*_-m)≤ u_m(x^*_m,x^*_-m). 



A strategy x can be represented as a weighted bipartite graph ℋ(x)=(M∪ W, E(x)) between men and women. In this graph, the edges E(x)={(m,w): x_mw>0} are supported over the nonzero entries of x, with the weight of edge (m,w) given by x_mw.


One can interpret u_m(x) as the expected utility that player m would receive by successfully getting matched if each man independently proposes to women according to his mixed strategy distribution x_m. The reason for defining the payoff functions as in (<ref>) is that we want our devised algorithms to be implemented in a fully decentralized and uncoordinated manner among men by relying only on their received feedback (embedded into their payoff functions). While such payoff functions are highly nonlinear, they are essential to eliminate any degrees of coordination among the players. In other words, the cost of devising a fully coordination-free algorithm comes in analyzing more complex payoff functions with higher degrees of nonlinearity. We refer to Section <ref> for a simpler form of payoff functions with less degree of nonlinearities, which in turn requires stronger information feedback. The following are three properties of the payoff functions (<ref>). 


  (i) Player m's strategy does not have any impact on the coefficient terms μ_mw∏_k>_w m(1-x_kw), w∈ W. In particular, the gradient of u_m(·) with respect x_mw is given by 
    v_mw(x):=∇_mwu_m(x)=μ_mw∏_k>_w m(1-x_kw).
 

  (ii) The payoff of each player m is linear with respect to his own strategy x_m.

  (iii) For any fixed strategy of other players x_-m, player m always has a best response among pure strategies that is obtained by setting x_mw=1 for the woman w that achieves the maximum value v_mw(x), and x_mw=0 otherwise.    


In the remainder of this paper, we will refer to the above noncooperative game as the stable matching game and simply denote it by 𝒢=(M, {u_m}_m∈ M, {𝒳_m}_m∈ M). It is worth noting that although we are interested in obtaining a NE of the stable matching game, which is a complete information one-shot game, our goal is to learn one such NE by repeatedly playing the incomplete information game, in which the true preferences μ_mw (and hence the payoff functions) are not fully known. In fact, the set of pure/mixed NE points of the stable matching game has some interesting connections with the set of stable matchings with known preferences. The following theorem establishes one such result.  
 

A pure strategy profile x^*∈{0,1}^n^2 corresponds to the characteristic vector of a stable matching if and only if it is a pure NE for the stable matching game.


Suppose x^* is a pure NE. Then, we cannot have x^*_mw=x^*_m'w=1 for some woman w and m≠ m'. Otherwise, between m and m', the one that woman w prefers less receives zero utility, while he can always deviate to propose to a woman w' that does not have any offer and strictly improve his utility. Therefore, x^* must correspond to the characteristic vector of a perfect matching. To show that x^* is a stable matching, let us, by contradiction, assume that there exists a blocking pair with respect to the matching x^* and denote it by (m,w). Moreover, let m' and w' be the corresponding matches for w and m under x^*, that is x^*_mw'=1 and x^*_m'w=1. Since (m,w) is a blocking pair, we have μ_mw>μ_mw' and m>_w m'. The payoff of player m under NE x^* equals u_m(x^*)=μ_mw'. Now if player m changes its strategy to x_m:=(0,x_mw=1), his payoff becomes 

    u_m(x_m,x^*_-m)=∑_j∈ Wμ_mj∏_k>_j m(1-x^*_kj)x_mj=μ_mw∏_k>_w m(1-x^*_kj)=μ_mw,

where the second equality holds because m'<_wm so that the term 1-x^*_m'w does not appear in the product ∏_k>_w m(1-x^*_kw), and moreover x^*_kw=0 ∀ k≠ m', which implies ∏_k>_w m(1-x^*_kw)=1. This shows that u_m(x_m,x^*_-m)=μ_mw>μ_mw'=u_m(x^*), contradicting the fact that x^* is a NE.  

Conversely, let x^* be the characteristic vector of a stable matching, and by contradiction, assume it is not a NE. Then, there exists at least one player m who can deviate and strictly improve his payoff. In particular, we can consider his pure strategy best response move, which is guaranteed by (iii), and denote his most preferred woman by w. Moreover, let m' and w' be the corresponding matches for w and m under the stable matching x^*, i.e., x^*_m'w=x^*_mw'=1. The current payoff of m under the pure strategy x^* equals u_m(x^*)=μ_mw', while his payoff after such deviation becomes

    u_m(x_m,x^*_-m)=∑_j∈ Wμ_mj∏_k>_j m(1-x^*_kj)x_mj=μ_mw∏_k>_w m(1-x^*_kw)≤μ_mw.
 
Since 0< μ_mw'=u_m(x^*)<u_m(x_m,x^*_-m), we obtain μ_mw'< μ_mw. Moreover, from relation (<ref>) and the fact that u_m(x_m,x^*_-m)>0, we must have ∏_k>_w m(1-x^*_kw)=1, which implies m'<_w m. This together with μ_mw'< μ_mw shows that (m,w) is a blocking pair, contradicting that x^* is stable.              


While the above equilibrium characterization theorem concerns pure NE points, it is possible that the stable matching game admits a mixed NE. The following is one example. 

Let us consider an instance of the stable matching game with three men M={m_1,m_2,m_3} and three women W={w_1,w_2,w_3}. We assume the preferences of men and women are given by

    μ_m_1w_1=2, μ_m_1w_2=1, μ_m_1w_3=ϵ,            m_3<_w_1 m_1<_w_1 m_2,   μ_m_2w_1=3, μ_m_2w_2=4, μ_m_2w_3=5,            m_3<_w_2 m_2<_w_2 m_1,   μ_m_3w_1=ϵ, μ_m_3w_2=12, μ_m_3w_3=6,          m_1<_w_3 m_2<_w_3 m_3,

where ϵ>0 is a small positive constant. Then, the stable matching game has a mixed NE that is supported over seven edges comprising of two cycles of length 4 that share the common edge (m_2,w_2) (see, Remark <ref>). In particular, the seven positive entries of that mixed NE are given by 

    x_m_1w_1=μ_m_2w_1/μ_m_2w_2,              x_m_1w_2=1-μ_m_2w_1/μ_m_2w_2,   x_m_2w_1=1-μ_m_1w_2/μ_m_1w_1,          x_m_2w_2=1-μ_m_2w_2μ_m_3w_3/μ_m_2w_1μ_m_3w_2,       x_m_2w_3=μ_m_1w_2/μ_m_1w_1+μ_m_2w_2μ_m_3w_3/μ_m_2w_1μ_m_3w_2-1,   x_m_3w_2=μ_m_2w_1/μ_m_2w_3,               x_m_3w_3=1-μ_m_2w_1/μ_m_2w_3.
   


As Example <ref> suggests, mixed NE points may represent unpredictable bipartite graphs (Remark <ref>), and it is difficult to make general statements about their structure. However, the following theorem establishes an interesting connection between mixed and pure NE points, implying that obtaining a mixed NE with full support on the women's side is as satisfactory as obtaining a pure NE. 


Let x be any mixed NE in which each woman receives at least one (fractional) proposal.[If this condition does not hold, the theorem statement continues to hold for the induced submarket where each woman has received at least one fractional proposal.] For each man m, let w_m be the least preferred woman among those that he fractionally proposes to them, i.e., w_m=_w∈ W{μ_mw: x_mw>0}. Then {(m,w_m), m∈ M} forms a stable matching (and hence a pure NE) for the stable matching game. 


Please see Appendix <ref> for the proof. 



Theorem <ref> suggests some strategies for obtaining decentralized solutions. We can first design a decentralized algorithm (see Algorithm <ref> in conjunction with the preference estimation oracle of Algorithm <ref>) to learn a mixed NE of the stable matching game and acquire estimated preferences. Subsequently, each man can convert his fractional mixed strategy into a pure strategy in a decentralized manner by proposing only to his least (estimated) preferred woman among those he fractionally proposes to. This process ensures that the resulting pure strategy profile will be a pure NE with high probability, and the probability of error vanishes over time. Importantly, each man has complete access to his own estimated preferences, and the rounding process outlined in Theorem <ref> does not require knowledge of others' mixed strategies. Each man only needs to be aware of his own mixed strategy and estimated preferences. This approach offers a simple decentralized method to convert mixed NE points into stable matchings. 




§ LOGARITHMIC REGRET FOR HIERARCHICAL MARKETS






In this section, we consider the stable matching game with unknown preferences and develop a decentralized and uncoordinated algorithm for learning its NE points. In particular, we focus on matching markets with specific structures because without imposing any assumption on the matching market, there are exponential lower bounds for the number of iterations to find a stable matching through better response dynamics <cit.>. One example of such a structured matching market is the hierarchical matching market, where it was shown in <cit.> that if men follow a certain decentralized (but coordinated) algorithm, the men's expected regret (see Definition <ref>) is at most logarithmic in time but exponential in terms of other parameters such as the number of men n. In particular, the authors in <cit.> raised the question of how to remove the exponential dependence of the regret on n. In this section, we show that Algorithm <ref> achieves this goal whose regret is logarithmic in time with a polynomial factor in terms of other parameters. Moreover, Algorithm <ref> is uncoordinated and, hence, removes the phase-dependent coordination required by the algorithm in <cit.>. Throughout this section, we impose the following hierarchical assumption on the matching market.    


We assume that the sets of men and women can each be ordered as M={m_1,…,m_n} and W={w_1,…,w_n} such that any man and woman with the same rank prefer each other above any other partner with a lower rank; i.e., man m_k prefers woman w_k to women w_k+1, w_k+2,…,w_n, and woman w_k
prefers man m_k to men m_k+1, m_k+2,…,m_n. It is easy to see that under this hierarchical assumption, the matching {(m_k,w_k), k=1,…,n} is a stable matching. 
 


The condition imposed in Assumption <ref> is also known as the Sequential Preference Condition (SPC) <cit.> and is weaker than the α-reducible condition considered in <cit.>, which assumes that each submarket has a fixed-pair: A pair (m,w) is called a fixed pair if both m and w prefer each other the most among anyone else in that submarket.[In fact, one can show that a matching market has a unique stable matching if and only if it satisfies α-reducible condition <cit.>.] Since Assumption <ref> is weaker than the α-reducible condition, all our results immediately applies to α-reducible markets.



Let us assume that players follow a decentralized and uncoordinated algorithm 𝒜 that results in a sequence of proposals α^t=(α^t_m, m∈ M), t=1,…,T. The expected regret of 𝒜 equals

    R(T)=∑_t=1^T 𝔼[1_{∃ k∈ [n]: α^t_m_k≠ w_k}],

which is the expected number of times that the men's proposals do not form a stable matching. 






 §.§ Algorithm Design and Preliminaries


The algorithm that we propose is an adaptation of the online dual mirror descent for adversarial multi-arm bandit problem (see, e.g., EXP3 Algorithm in <cit.>) to the stable matching game, where for simplicity we refer to it as the EXP algorithm. In particular, it uses an entropy regularizer and adaptive stepsize for updating the players' mixed strategy distributions over time.

To describe the EXP algorithm, let us denote the gradient vector of player m by v_m(x)=∇_m u_m(x) such that for any woman w, we have v_mw(x)=μ_mw∏_k>_w m(1-x_kw). In particular, we note that u_m(x)=⟨ x_m, v_m(x)⟩. Therefore, when restricting to a pure strategy (action) profile α^t=(α_1^t,…,α^t_n)∈ W×⋯× W that is played at time t, the w-th coordinate of the gradient vector is given by

    v_mw^t:=μ_mw1_{α^t_k≠ w ∀ k>_w m},

where 1_{·} denotes the indicator function. On the other hand, by playing a pure strategy profile α^t, each man m receives a feedback only for the woman α^t_m that he has proposed to her at time t. In particular, if α_m^t=w, player m observes the payoff μ^t_mw1_{α^t_k≠ w  ∀ k>_w m}1_{α_m^t=w} at time t, where we recall that μ^t_mw is the realization of a random reward drawn independently from an unknown distribution 𝒟_mw with unknown mean μ_mw and bounded variance σ^2_mw∈ [0, 1]. 


During the course of the algorithm, at each time t, each player m holds a mixed strategy X^t_m∈𝒳_m and updates it whenever he receives new information. He then computes his adjusted mixed strategy X̂^t_m=(1-γ^t)X^t_m+γ^t/n1, where γ^t>0 is a mixing parameter and 1 is the vector of all ones. Using ideas from importance sampling for bandit problems <cit.>, and to obtain an unbiased estimator of the actual gradient vector, player m constructs an estimate vector v̂^t_m by normalizing the received feedback with his adjusted mixed strategy probabilities as

    v̂^t_mw=μ^t_mw1_{α^t_k≠ w ∀ k>_w m}/X̂^t_mw·1_{α_m^t=w},     w∈ W.

The reason why we use adjusted mixed strategies X̂^t with additional mixing parameter γ^t is to ensure that the mixed strategies remain bounded away from zero by a positive quantity. That makes the estimators (<ref>) have bounded variance (see Lemma <ref>), which allows us to use martingale concentration results to bound the probability of various events.

Let us denote the (random) score vector of player m at time t-1 by L̂^t-1_m=(L̂_mw,w∈ W). Player m uses L̂^t-1_m to compute his  mixed strategy vector X_m^t using the logit update rule (<ref>). He then proposes to a woman α_m^t chosen independently at random according to his adjusted mixed strategy X̂_m^t, and receives as feedback v̂_m^t with entries given by (<ref>). Player m then updates his score vector by L̂^t_m=L̂^t-1_m+v̂^t_m and proceeds to the next round. The detailed description of the EXP algorithm is summarized in Algorithm <ref>. 





In order to analyze the convergence behavior of Algorithm <ref> to a stable matching, let {ℱ^t-1}_t=1^∞ be the increasing filtration sequence that is adapted to the history of the random processes generated by Algorithm <ref>. More precisely, 
    ℱ^t-1={X^τ,L̂^τ,α^τ,v̂^τ, τ=0,1,…,t-1}∪{X^t},
 contains all the realized events up to time t-1 except the realization of pure strategies α^t and the rewards μ^t at time t. In particular, all relevant processes at time t-1 as well as X^t and X̂^t are ℱ^t-1-measurable, but α^t and v̂^t are not ℱ^t-1-measurable. 

We begin with the following lemma, which shows that the feedback v̂^t_m received by player m at time t is conditionally an unbiased and bounded estimator of the actual gradient vector v_m^t.



The received feedback is a conditionally unbiassed and bounded estimate of the gradients 

    𝔼[v̂_m^t|ℱ^t-1]=v_m(X̂^t)  ∀ m,      𝔼[(v̂_mw^t)^2|ℱ^t-1]≤σ^2/X̂^t_mw≤n/γ^t  ∀ m,w,

where σ^2:=max_m,w𝔼[(μ̂^t_mw)^2]∈ [0, 1].


Consider any pair (m,w). Since X̂^t_mw is ℱ^t-1-measurable and μ̂^t_mw is drawn independently from the algorithm's past decisions (and hence is independent of ℱ^t-1 and α^t), we have

    𝔼[v̂_mw^t|ℱ^t-1]   =𝔼[μ^t_mw]/X̂^t_mw·𝔼[1_{α^t_k≠ w  ∀ k>_w m}·1_{α_m^t=w}|ℱ^t-1]   =μ_mw/X̂^t_mwℙ{α_m^t=w|ℱ^t-1}∏_k>_w mℙ{α^t_k≠ w|ℱ^t-1}   =μ_mw∏_k>_w m(1-X̂^t_kw)=v_mw(X̂^t),

where the second equality holds because conditioned on ℱ^t-1, the random variables {α^t_k, ∀ k} are mutually independent as they are drawn independently from the given distributions {X̂^t_k, ∀ k}. Similarly,

    𝔼[(v̂_mw^t)^2|ℱ^t-1]   =𝔼[(μ̂^t_mw)^2]/X̂^t_mw∏_k>_w mℙ{α^t_k≠ w|ℱ^t-1}≤σ^2/X̂^t_mw≤n/γ^t,   ∀ m,w.
         






 §.§ Performance of Algorithm <ref> for Hierarchical Markets

 
Here, we analyze the performance of Algorithm <ref> for hierarchical matching markets. In the following analysis, we assume that a positive lower bound for c:=1/8min_k∈ [n]{Δ,μ_m_kw_k} is known to all the players so that they can choose their mixing parameters in Algorithm <ref>. However, as we discuss in Appendix <ref>, this condition can be relaxed up to some extent by using an uncoordinated and decentralized UCB gap estimation oracle in which the players use their estimated gaps to tune their mixing parameters. 
The main result of this section is presented in Theorem <ref>, which essentially shows that for hierarchical matching markets, Algorithm <ref> achieves an expected regret that is logarithmic in T and polynomial in terms of other parameters. However, before delving into the analysis, we first provide an overview of the main steps in the proof.


    
  * First, we define a “good" event Ω, which represents the set of circumstances where the difference between the accumulated realized rewards, η^t(L̂^t_mw̃ - L̂^t_mw), for any m, w, w̃, and t, remains close to its conditional expected value. In particular, Lemma <ref> shows that the event Ω occurs with very high probability. This allows us to analyze the performance of Algorithm <ref> under its conditional mean trajectory while incurring only a small, controllable loss. 
    
  * Conditioned on the event Ω, Lemma <ref> shows that, due to the market structure and payoff functions, as time progresses, more pairs in the market are matched according to the underlying hierarchical order with very high probability. Additionally, the size of such hierarchical nested matchings grows with very high probability after a polynomial number of steps. 
    
  * Finally, in Theorem <ref>, we combine the results of Lemma <ref> and Lemma <ref> to bound the expected regret of Algorithm <ref> by conditioning on the event Ω.  






Let us consider the stochastic dynamics of Algorithm <ref> that are given by

    X^t_mw=exp(η^t L̂^t-1_mw)/∑_w'exp(η^t L̂^t-1_mw'),   X̂^t_m=(1-γ^t)X^t_m+γ^t/n1,   L̂^t_mw=L̂^t-1_mw+v̂^t_mw.




Define Y^t_mw=∑_τ=1^t v_mw(X̂^τ), where v_mw(X̂^τ)=μ_mw∏_k>_w m(1-X̂^τ_mw), and consider the event

    Ω:={|η^t(L̂^t-1_mw̃-L̂^t-1_mw)-η^t(Y^t-1_mw̃-Y^t-1_mw)|≤ 2c√(t),  ∀ t≥ 1, ∀ m, w,w̃},
 
where c=1/8min_k∈ [n]{Δ,μ_m_kw_k} is a constant parameter. The following lemma provides a high probability bound for the event Ω to occur under appropriate choice of stepsize and mixing parameter. 


Fix an arbitrary δ∈ (0, 1), and suppose each player follows Algorithm <ref> with stepsize η^t=1/√(t) and mixing parameter γ^t=Mlog t/t, where M=4n/clog1/δ. Then, ℙ{Ω}≥ 1-δ. 


Fix an arbitrary man m. For any w, w̃, as in (<ref>), we can write

    η^t(L̂^t-1_mw̃-L̂^t-1_mw)   =η^t∑_τ=1^t-1(v_mw̃(X̂^τ)-v_mw(X̂^τ))+η^t S^t-1_mw̃-η^t S^t-1_mw   =η^t(Y^t-1_mw̃-Y^t-1_mw)+η^t S^t-1_mw̃-η^t S^t-1_mw,

where S^t-1_mw:=∑_τ=1^t-1(v̂^τ_mw-v_mw(X̂^τ)) ∀ m,w. According to Lemma <ref>, {v̂^τ_mw-v_mw(X̂^τ)} is a martingale difference sequence that satisfies 

    |v̂^τ_mw-v_mw(X̂^τ)|≤n/γ^τ,    𝔼[(v̂^τ_mw-v_mw(X̂^τ))^2|ℱ^τ-1]≤n/γ^τ.

By taking u=n/γ^t-1, v^2=∑_τ=1^t-1n/γ^τ, and x=ct in Lemma <ref> (see Appendix <ref>) and noting that the event {V^t≤ v^2} holds with probability 1, we obtain 

    ℙ{η^t|S_mw^t-1|≥ c√(t)}=ℙ{|S_mw^t-1|≥ ct}≤ 2exp(-Mc^2 t^2/2n(∑_τ=1^t-1τ/logτ+ct(t-1)/log (t-1))).

Since f(τ):=τ/logτ is a concave function for τ≥ 100, using Jensen's inequality, we have

    1/t-1∑_τ=1^t-1τ/logτ=∑_τ=1^t-11/t-1f(τ)≤ f(∑_τ=1^t-11/t-1τ)=∑_τ=1^t-1τ/t-1/log(∑_τ=1^t-1τ/t-1) =t/2logt/2.

Therefore, ∑_τ=1^t-1τ/logτ≤t(t-1)/2logt/2≤t^2/log t. Substituting this relation and t(t-1)/log (t-1)≤t^2/log t into (<ref>) we get

    ℙ{η^t|S_mw^t-1|≥ c√(t)}≤ 2exp(-Mc^2 t^2/2n(1+c)t^2/log t)= 2exp(-Mc^2log t/2n(1+c))≤2δ/(nπ t)^2,

where the last inequality is obtained by choosing M sufficiently large (e.g., M=4n/clog1/δ). Thus, using the union bound, we can write

    ℙ{η^t |S_mw^t-1|< c√(t) ∀ t≥ 1, ∀ m, w}≥ 1-∑_t=1^∞2n^2δ/(nπ t)^2> 1-δ.

Finally, using (<ref>) and (<ref>), one can see that

    ℙ{Ω}   =ℙ{|η^t(L̂^t-1_mw̃-L̂^t-1_mw)-η^t(Y^t-1_mw̃-Y^t-1_mw)|≤ 2c√(t),  ∀ t≥ 1, ∀ m, w, w̃}   =ℙ{|η^t S^t-1_mw̃-η^t S^t-1_mw|≤ c√(t),  ∀ t≥ 1, ∀ m, w, w̃}   ≥ℙ{η^t |S_mw^t-1|< c√(t) ∀ t≥ 1, ∀ m, w}> 1-δ.
 



Before we present the main result of this section, we first prove the following key lemma. 


Assume that each man follows Algorithm <ref> with stepsize η^t=1/√(t) and mixing parameter γ^t=Mlog t/t, where M=4n/clog1/δ. Let a_1≥…≥ a_n be a sequence defined by a_1=Δ/2 and a_k=1/4min_i∈ [k]{Δ,μ_m_iw_i}, ∀ k=2,…,n. Conditioned on the event Ω, there exist t_1≤…≤ t_n=Õ(n^2/c^2log1/δ), such that for any t≥ t_k and any w≠ w_k, we have 

    X̂^t_m_kw_k≥1-γ^t/1+(n-1)e^-η^t a_k t+γ^t/n,    X̂^t_m_kw≤(1-γ^t)e^-η^t a_k t/1+(n-1)e^-η^t a_k t+γ^t/n.
    


We use induction on k∈ [n] to prove the lemma. For the base of the induction k=1, we note that for any X̂∈ [0,1]^n^2, we have

    v_m_1w_1(X̂)=μ_m_1w_1∏_k>_w_1 m_1(1-X̂_kw_1)=μ_m_1w_1,   v_m_1w(X̂)=μ_m_1w∏_k>_w m_1(1-X̂_kw)≤μ_m_1w.
 
Therefore, 

    Y^t-1_m_1w_1-Y^t-1_m_1w   =∑_τ=1^t-1(v_m_1w_1(X̂^τ)-v_m_1w(X̂^τ))   ≥∑_τ=1^t-1(μ_m_1w_1-μ_m_1w)≥Δ (t-1).

Define a_1=Δ/2 and let t_1=4. Conditioned on the event Ω, for any t≥ t_1 and any w≠ w_1, we have

    X^t_m_1w_1/X^t_m_1w   =e^η^t(L̂^t-1_m_1w_1-L̂^t-1_m_1w)≥ e^η^t (Y^t-1_m_1w_1-Y^t-1_m_1w)- 2c√(t)   ≥ e^η^t Δ (t-1)-2c√(t)≥ e^η^t a_1 (2t-2)-1/2η^t a_1 t   = e^η^t a_1 (3/2t-2)≥ e^η^t a_1t,

where the third inequality holds because c≤a_1/4, and the last inequality holds because t≥ t_1. Moreover, 

    1=X^t_m_1w_1+∑_w≠ w_1X^t_m_1w≤(1+(n-1)e^-η^t a_1 t)X^t_m_1w_1.

Combining relations (<ref>) and (<ref>) and using the fact that X̂^t_mw=(1-γ^t)X^t_mw+γ^t/n, for any t≥ t_1 and w≠ w_1, we have

    X̂^t_m_1w_1≥1-γ^t/1+(n-1)e^-η^t a_1 t+γ^t/n,        X̂^t_m_1w≤(1-γ^t)e^-η^t a_1 t/1+(n-1)e^-η^t a_1 t+γ^t/n.
 



Next, by induction hypothesis, suppose that the statement holds for the first k-1 pairs (m_ℓ,w_ℓ), ℓ=1,…,k-1 with corresponding sequences a_1≥ a_2…≥ a_k-1 and t_1≤ t_2≤…≤ t_k-1, and let us consider the kth pair (m_k,w_k) of the market. Notice that for any X̂∈ [0,1]^n^2, we have 

    v_m_kw_k(X̂)=μ_m_kw_k∏_m_ℓ>_w_k m_k(1-X̂_m_ℓw_k)≥μ_m_kw_k∏_ℓ=1^k-1 (1-X̂_m_ℓw_k).

We consider two cases: 

Case I: If w=w_i for some i∈{1,…,k-1}, then

    v_m_kw(X̂)=v_m_kw_i(X̂)=μ_m_kw_i∏_m_ℓ>_w_i m_k(1-X̂_m_ℓw_i)≤μ_m_kw_i(1-X̂_m_iw_i).

Therefore, using (<ref>) and (<ref>) we can write

    Y^t-1_m_kw_k   -Y^t-1_m_kw=∑_τ=1^t-1(v_m_kw_k(X̂^τ)-v_m_kw_i(X̂^τ))   ≥∑_τ=1^t-1(μ_m_kw_k∏_ℓ=1^k-1(1-X̂^τ_m_ℓw_k)-μ_m_kw_i(1-X̂^τ_m_iw_i))   ≥∑_τ=1^t-1(μ_m_kw_k(1-∑_ℓ=1^k-1(γ^t/n+(1-γ^t)e^-η^t a_ℓ t/1+(n-1)e^-η^t a_ℓ t))-μ_m_kw_i(1-γ^t/n-1-γ^t/1+(n-1)e^-η^t a_i t))    ≥∑_τ=1^t-1(μ_m_kw_k-γ^t-∑_ℓ=1^k-1(1-γ^t)e^-η^t a_ℓ t/1+(n-1)e^-η^t a_ℓ t-μ_m_kw_i(n-1)e^-η^t a_i t+γ^t/1+(n-1)e^-η^t a_i t)   ≥∑_τ=1^t-1(μ_m_kw_k-γ^t-∑_ℓ=1^k-1e^-η^t a_ℓ t-μ_m_kw_i(n-1)e^-η^t a_i t-μ_m_kw_iγ^t)   ≥∑_τ=1^t-1(μ_m_kw_k-2γ^t-2(n-1)e^-η^t a_k-1 t),

where the second inequality holds by the induction hypothesis for t≥ t_k-1, and the last inequality holds because a_1≥…≥ a_k-1. Thus, for any time step t≥ t_k-1 that satisfies e^-η^t a_k-1 t≤γ^t and t-2/∑_τ=1^t-1γ^τ≥4n/μ_m_kw_k, we have 

    Y^t-1_m_kw_k-Y^t-1_m_kw   ≥∑_τ=1^t-1(μ_m_kw_k-2nγ^t)= μ_m_kw_k(t-1)-2n∑_τ=1^t-1γ^t≥μ_m_kw_k/2t.


Case II: If w∉{w_1,…,w_k}, then μ_m_kw_k≥μ_m_kw+Δ. Using (<ref>) and the fact that μ_m_kw≥ v_m_kw(X̂) for any X̂∈ [0,1]^n^2, we can write

    Y^t-1_m_kw_k-Y^t-1_m_kw   =∑_τ=1^t-1(v_m_kw_k(X̂^τ)-v_m_kw(X̂^τ))   ≥∑_τ=1^t-1(μ_m_kw_k∏_ℓ=1^k-1(1-X̂^τ_m_ℓw_k)-μ_m_kw)   ≥∑_τ=1^t-1(μ_m_kw_k(1-∑_ℓ=1^k-1X̂^τ_m_ℓw_k)-μ_m_kw)   ≥∑_τ=1^t-1((μ_m_kw_k-μ_m_kw)-∑_ℓ=1^k-1(γ^t/n+(1-γ^t)e^-η^t a_ℓ t/1+(n-1)e^-η^t a_ℓ t))   ≥∑_τ=1^t-1(Δ-γ^t-∑_ℓ=1^k-1e^-η^t a_ℓ t)   ≥∑_τ=1^t-1(Δ-γ^t-(n-1)e^-η^t a_k-1 t).

Thus, for any integer t≥ t_k-1 that satisfies e^-η^t a_k-1 t≤γ^t and t-2/∑_τ=1^t-1γ^τ≥2n/Δ, we have 

    Y^t-1_m_kw_k-Y^t-1_m_kw≥∑_τ=1^t-1(Δ-nγ^t)=Δ(t-1)-n∑_τ=1^t-1γ^τ≥Δ/2t.

As a result, by taking

    t_k:=min{t≥ t_k-1|  tη^t/ln(1/γ^t)≥1/a_k-1,  t-2/∑_τ=1^t-1γ^τ≥max{4n/μ_m_kw_k, 2n/Δ}},   a_k:=1/4min_i∈ [k]{Δ, μ_m_iw_i},

we have a_k-1≥ a_k, t_k≥ t_k-1. In particular, for t≥ t_k both relations (<ref>) and (<ref>) hold and we have 

    Y^t-1_m_kw_k-Y^t-1_m_kw≥min{Δ/2, μ_m_kw_k/2} t≥ 2a_k t, ∀ t≥ t_k, ∀ w≠ w_k.
 
Now, using a similar argument as in the base case, for any t≥ t_k and w≠ w_k, we have

    X^t_m_kw_k/X^t_m_kw=e^η^t(L̂^t-1_m_kw_k-L̂^t-1_m_kw)≥ e^η^t (Y^t-1_m_kw_k-Y^t-1_m_kw)- 2c√(t)≥ e^η^t 2a_k t-2c√(t)≥ e^η^t 2a_k t-η^t a_k t= e^η^t a_kt,

where the second inequality uses (<ref>), and the last inequality holds because c=min_k∈ [n]{Δ/8,μ_m_kw_k/8}≤a_k/2. The above relation, together with similar arguments as in deriving (<ref>) and (<ref>) in which a_1 is replaced by a_k, gives us the desired result (<ref>). 

Finally, using the stepsize and mixing parameters η^t=1/√(t) and γ^t=Mlog t/t in relation (<ref>), one can obtain an explicit upper bound for t_n as

    t_n≤min{t∈ℤ_+|  √(t)/log(t/Mlog t)≥1/a_n-1,  t-2/Mlog^2 t≥max{4n/μ_m_nw_n, 2n/Δ}},

for which it suffices to take t_n=Õ(n^2/c^2log1/δ), where Õ(·) hides the logarithmic factors.




Assume that each man follows Algorithm <ref> with η^t=1/√(t) and γ^t=Mlog t/t, where M=4n/clog T and c=1/8min_k∈ [n]{Δ,μ_m_kw_k}. Then, the expected regret of Algorithm <ref> in hierarchical matching markets is at most R(T)=O(n^2/clog^3 T).  


Conditioned on the event Ω, which according to Lemma <ref> it occurs with probability at least 1-δ, the expected regret can be upper-bounded as 

    𝔼[R(T)|Ω]   = ∑_t=1^T 𝔼[1_{∃ k: α^t_m_k≠ w_k}|Ω]≤∑_t=1^T∑_k=1^n 𝔼[1_{α^t_m_k≠ w_k}|Ω]   ≤∑_t=1^T∑_k=1^n 𝔼[𝔼[1_{α^t_m_k≠ w_k}|X̂^t_m_k]|Ω]=∑_t=1^T∑_k=1^n 𝔼[ℙ{α^t_m_k≠ w_k|X̂^t_m_k}|Ω]   =∑_t=1^T∑_k=1^n 𝔼[1-X̂^t_m_kw_k|Ω]≤ nt_n+∑_t=t_n+1^T∑_k=1^n 𝔼[1-X̂^t_m_kw_k|Ω]   ≤Õ(n^3/c^2log1/δ)+∑_t=t_n+1^T∑_k=1^n (1-γ^t/n-1-γ^t/1+(n-1)e^-η^t a_k t)   ≤Õ(n^3/c^2log1/δ)+∑_t=t_n+1^T∑_k=1^n(n-1)e^-η^t a_k t+γ_t/1+(n-1)e^-η^t a_k t   ≤Õ(n^3/c^2log1/δ)+n^2∑_t=1^T e^-η^t a_n t+n∑_t=1^Tγ_t   ≤Õ(n^3/c^2log1/δ)+2n^2/a_n^2+nMlog^2 T   =O(n^3/c^2log1/δ+n^2/clog1/δlog^2 T),

where the third inequality uses Lemma <ref>, the fifth inequality uses the fact that a_n:=min_k∈ [n]{a_k}, and the last inequality is obtained by choosing γ^t=Mlog t/t, η^t=1/√(t), M=4n/clog1/δ, and noting that c≤ a_n.

Finally, since δ∈ (0,1) can be chosen arbitrary, by choosing δ=1/T, we can write

    𝔼[R(T)]   =ℙ{Ω}𝔼[R(T)|Ω]+ℙ{Ω̅}𝔼[R(T)|Ω̅]   ≤ (1-δ)O(n^3/c^2log1/δ+n^2/clog1/δlog^2 T)+δ n^2T    =O(n^2/clog^3 T).



It is worth noting that the regret of Algorithm <ref> is logarithmic in T, with a polynomial factor n^2/c in terms of the problem parameters, thereby addressing a question raised in <cit.>. Furthermore, Algorithm <ref> is both decentralized and uncoordinated, overcoming the limitations of the algorithm proposed in <cit.>, which is notably more complex and requires phase-dependent coordination among men according to the Deferred-Acceptance algorithm by Gale and Shapley. 






§ EXPONENTIAL LOCAL CONVERGENCE FOR GENERAL MATCHING MARKETS
 

In this section, we consider learning a stable matching in general matching markets (i.e., without any hierarchical assumption) and show that the same decentralized and uncoordinated algorithm EXP (Algorithm <ref>) converges locally at an exponential rate to a stable matching when players' strategies get sufficiently close to one of the stable matchings. To that end, we first consider the following technical lemma, which shows that if a strategy profile is sufficiently close to a stable matching, then the reward of choosing an action according to that stable matching is strictly the best decision.    

Let X^* be a pure NE of the stable matching game and set c=1/8min{Δ,μ_min}. For any strategy profile x that satisfies x-X^*_1≤c/μ_max, we have v_mw^*(x)-v_mw(x)>c ∀ w≠ w^*, where w^* is the woman that m is matched to her under the pure NE X^*. 


Please see Appendix <ref> for the proof.


Next, we consider the following lemma that will be used to capture the local behavior of the dynamics of Algorithm <ref> around a pure NE.  


Suppose X^* is a pure NE and {X^t} be the sequence of iterates generated by Algorithm <ref>. Moreover, let A^t-1= ∑_τ=1^t-1(v_mw^*(X^τ)-v_mw(X^τ)), where w^* is the woman that man m is matched to her under the pure NE X^*. Then, the following statements hold:


  * i) If X^t-X^*_1≤c/250n^2, then η^tL̂^t-1_mw^*-η^tL̂^t-1_mw≥ln (n^2/2c)+6  ∀ m,w≠ w^*.

  * ii) If η^tL̂^t-1_mw^*-η^tL̂^t-1_mw≥ln(n^2/2c) ∀ m,w≠ w^*, then X^t-X^*_1≤ c.

  * iii) Let t_0 be the first time such that 1/η^t_0+1-1/η^t_0≤ c/(ln(n^2/2c)+3). If for some t≥ t_0 we have η^t A^t-1≥ln(n^2/2c)+3 and v_mw^*(X^t)-v_mw(X^t)≥ c ∀ m,w≠ w^*, then η^t+1 A^t≥ln(n^2/2c)+3.
 


Please see Appendix <ref> for the proof. 


Now we are ready to state the main result of this section. 

Fix an arbitrary δ∈ (0, 1), and suppose each player follows Algorithm <ref> with some mixing sequence {γ^t} and decreasing stepsize sequence {η^t} that satisfie the following conditions:

    (∑_τ=1^t-1γ^t-1/γ^τ)^-1≤η^t≤min{(2n^2∑_τ=1^t-1γ^τ)^-1, (8nln(nπ/√(δ)t)∑_τ=1^t-11/γ^τ)^-1/2}.

Let t_0 be the first time such that 1/η^t_0+1-1/η^t_0≤ c/(ln(n^2/2c)+3) and X^t_0-X^*≤c/250n^2. Then with probability at least 1-δ, the sequence {X^t} converges to the pure NE X^* exponentially fast, i.e.,

    ℙ{X^t+1-X^*_1≤ 41 nexp(-ctη^t+1) ∀ t≥ t_0 | X^t_0-X^*_1≤c/250n^2}≥ 1-δ.
  


Fix an arbitrary man m, and assume X^*_mw^*=1. For any w≠ w^*, we have

    η^tL̂^t-1_mw^*   -η^tL̂^t-1_mw=η^t∑_τ=1^t-1(v̂^τ_mw^*-v̂^τ_mw)   =η^t∑_τ=1^t-1(v_mw^*(X̂^τ)-v_mw(X̂^τ))+η^t∑_τ=1^t-1(v̂^τ_mw^*-v_mw^*(X̂^τ))-η^t∑_τ=1^t-1(v̂^τ_mw-v_mw(X̂^τ))   =η^t∑_τ=1^t-1(v_mw^*(X̂^τ)-v_mw(X̂^τ))+η^t S^t-1_mw^*-η^t S^t-1_mw,
 
where S^t-1_mw:=∑_τ=1^t-1(v̂^τ_mw-v_mw(X̂^τ)). Moreover, for any m and w, using the mean-value theorem, there exists a θ=λX̂^τ+(1-λ)X^τ such that 

    |v_mw(X̂^τ)-v_mw(X^τ)|   =|∇ v_mw(θ)· (X̂^τ-X^τ)|   ≤∇ v_mw(θ)_∞X̂^τ-X^τ_1   ≤ n X̂^τ-X^τ_1   =nγ^τ∑_mX_m^τ-1/n1_1 ≤ n^2γ^τ.

Substituting this relation into (<ref>) and using the triangle inequality, we can write

    |η^tL̂^t-1_mw^*-η^tL̂^t-1_mw-η^t∑_τ=1^t-1(v_mw^*(X^τ)-v_mw(X^τ))|   ≤η^t |S^t-1_mw^*|+η^t |S^t-1_mw|+2n^2η^t∑_τ=1^t-1γ^τ   ≤η^t |S^t-1_mw^*|+η^t |S^t-1_mw|+1,

where the last inequality follows from the upper bound on the choice of stepsize η^t. 

Next, notice that according to Lemma <ref>, {v̂^τ_mw-v_mw(X̂^τ)} is a martingale difference sequence that almost surely satisfies 

    |v̂^τ_mw-v_mw(X̂^τ)|≤n/γ^τ,   𝔼[(v̂^τ_mw-v_mw(X̂^τ))^2|ℱ^τ-1]≤𝔼[(v̂_mw^τ)^2|ℱ^τ-1]≤n/γ^τ.

Therefore, by taking u=n/γ^t-1, v^2=∑_τ=1^t-1n/γ^τ, and x=1/η^t in Lemma <ref>, we obtain

    ℙ{η^t |S_mw^t-1|≥ 1}≤ 2exp(-(1/η^t)^2/2n(∑_τ=1^t-11/γ^τ+1/η^tγ^t-1))≤ 2exp(-(1/η^t)^2)/4n∑_τ=1^t-11/γ^τ),
 
where the second inequality holds because using the lower bound on η^t we have 1/η^tγ^t-1≤∑_τ=1^t-11/γ^τ. Since by the choice of stepsize (1/η^t)^2/∑_τ=1^t-1 1/γ^τ≥ 8nln(nπ/√(δ)t), we get ℙ{η^t |S_mw^t-1|≥ 1}≤2δ/(nπ t)^2. In particular, using the union bound, the probability that η^t |S_mw^t-1|< 1 for any t≥ 1 and any m, w, is at least

    ℙ{η^t |S_mw^t-1|< 1 ∀ t≥ 1, ∀ m, w}≥ 1-∑_t=1^∞2n^2δ/(nπ t)^2> 1-δ.

Let us define A^t-1= ∑_τ=1^t-1(v_mw^*(X^τ)-v_mw(X^τ)). Using (<ref>) and (<ref>), with probability at least 1-δ, we have

    |η^tL̂^t-1_mw^*-η^tL̂^t-1_mw-η^tA^t-1|≤ 3,  ∀ t≥ 1, ∀ m, w.
    

Next, let us assume X^t-X^*_1∈c/250n^2. Then, for any m, w≠ w^*, we have v_mw^*(X^t)-v_mw(X^t)≥ c and η^tL̂^t-1_mw^*-η^tL̂^t-1_mw≥ln(n^2/2c)+6, by Lemma <ref> and Lemma <ref> (part i), respectively. Using relation (<ref>), it means that η^tA^t-1≥ln(n^2/2c)+3, and thus by Lemma <ref> (part iii), we have η^t+1 A^t≥ln(n^2/2c)+3. This in view of Lemma <ref> (part ii) shows that X^t+1-X^*_1≤ c. Similarly, Lemma <ref> implies v_mw^*(X^t+1)-v_mw(X^t+1)≥ c ∀ m,w≠ w^*, which in view of Lemma <ref> (part iii) shows that η^t+2 A^t+1≥ln(n^2/2c)+3. Hence, using relation (<ref>), η^t+2L̂^t+1_mw^*-η^t+2L̂^t+1_mw≥ln(n^2/2c) ∀ m,w≠ w^*, and thus X^t+2-X^*_1≤ c by Lemma <ref> (part ii). 

The above inductive argument shows that if X^t_0-X^*≤2c/n^2 for some t_0≥ 1 such that 1/η^t_0+1-1/η^t_0≤ c/(ln(n^2/2c)+3), then with probability at least 1-δ, we have X^t-X^*≤ c ∀ t≥ t_0, and in particular, using Lemma <ref>, v_mw^*(X^t)-v_mw(X^t)≥ c ∀ m,w, t≥ t_0 . Therefore, with probability at least 1-δ, for any m, w≠ w^*, t≥ t_0, we have 

    X^t+1_mw^*/X^t+1_mw=exp(η^t+1L̂^t_mw^*-η^t+1L̂^t_mw)≥exp(η^t+1A^t-3)≥exp(ctη^t+1-3).

Therefore, X^t+1_m-X^*_m_1=2(1-X^t+1_mw^*)≤ 2nexp(-ctη^t+1+3), which shows that

    ℙ{X^t+1-X^*_1≤ 2nexp(-ctη^t+1+3) ∀ t≥ t_0| X^t_0-X^*_1≤c/250n^2}≥ 1-δ.




    In the statement of Theorem <ref>, the condition that t≥ t_0  is not restrictive because if Algorithm <ref> converges to a pure NE, then eventually for some t_0, we have 1/η^t_0+1-1/η^t_0≤ c/(ln (n^2/2c)+3) and  X^t_0-X^*≤c/250n^2. In particular, t_0 is a constant that scales polynomially in terms of other parameters (see, e.g., Remark <ref>). We note that it is possible to further relax this condition by a constant scaling of η^t, i.e., by setting η^t←1/c(ln (n^2/2c)+3)η^t and rescaling all other parameters such as the mixing sequence γ^t accordingly. However, that requires the players to know c=1/8min{Δ,μ_min} to choose their stepsize η^t, while knowing c is not required in the stepsize condition given in (<ref>).    



In fact, there exist feasible choices for η^t and γ^t that satisfy the stepsize condition (<ref>) in Theorem <ref>. For instance, if for some α,β∈ (0,1),  α> max{1-β, 1+β/2}, we choose 

    η^t=Θ(1/ln( t/√(δ))t^α),      γ^t=Θ(1/t^β),

then condition (<ref>) is satisfied for any t_0≥ (α b/c)^1/(1-α), where b=ln(n^2/2c)+3. More specifically, by choosing γ^t=1/t^1/3 and η^t=1/t^3/4, the exponential convergence rate of Theorem <ref> would be in the order of exp(-ct^1/4) for any t_0≥ (b/c)^4. We note that condition (<ref>) provides only a sufficient condition to achieve local exponential convergence and we did not attempt to fully characterize all feasible choices of stepsize/mixing parameters. However, there could be other choices of stepsize that do not satisfy condition (<ref>) while still achieving local exponential convergence.


Although Theorem <ref> shows that Algorithm <ref> achieves an exponential convergence rate, its convergence guarantee holds in a local sense, i.e., when the dynamics get sufficiently close to a neighborhood of a stable matching.
In the following theorem, we complement this result by showing that if the dynamics of Algorithm <ref> converge with positive probability, it must converge to a mixed NE of the stable matching game. 


Under the same stepsize assumptions of Theorem <ref>, if Algorithm <ref> converges to a limit point X^* with positive probability, then X^* must be a (possibly mixed) NE of the matching game. 


Please see Appendix <ref> for the proof.    







§ GLOBAL LEARNING OF STABLE MATCHINGS IN GENERAL MARKETS


In the previous section, we showed that the decentralized and uncoordinated EXP algorithm converges locally to a stable matching at an exponential rate in general matching markets. This prompts the question of whether there is an uncoordinated and decentralized algorithm capable of globally learning a stable matching in general markets, regardless of the convergence rate. In this section, we answer this question affirmatively by introducing an alternative algorithm that leverages the weak acyclic property of the stable matching game. To that end, we first present several useful properties of the stable matching game, which will be instrumental in establishing our main convergence results. 



 
A pure strategy profile x is a good state if all the men matched under x are at their best responses.[In a pure strategy profile x=(x_m, m∈ M), a matched man is the one whose pure strategy x_m has exactly one coordinate equal to 1 (say wth coordinate), and all other coordinates are zero, and no other man of higher preference proposes to w, i.e., x_kw=0 ∀ k>_w m.] In other words, a good state x refers to a partial matching in which the matched men are at their best responses.
 


The following theorem shows that if the stable matching game with known preferences starts from a good state, then any sequence of best response dynamics by the players reaches a pure NE. Unless stated otherwise, we assume that each player m is aware of both his own true preferences and those of all the women, enabling him to compute his payoff function. This assumption will be relaxed in Theorem <ref>, where players seek to learn these unknown preferences through interactions with others. 


Consider the stable matching game that starts from a good initial state x^0. Consider any sequence of updates where at each time t, an arbitrary subset of players that includes at least one unsatisfied player updates their actions by playing their best responses. Then, the sequence of updates converges to a pure NE in no more than |M||W| iterations.


Let us encode the ordinal preferences of each woman w into cardinal values λ_wm∈ℝ^+ such that m_1>_w m_2 if and only if λ_wm_1>λ_wm_2. Given a  pure strategy profile x, let r_w(x)∈{λ_wm, m∈ M} be woman w's reward for her most preferred man who is matched to her under x. By convention, we set r_w(x)=0 if w is not matched to any man under the pure-strategy profile x. We show that r_w(x) is nondecreasing for each woman w (and for at least one w it is strictly increasing) such that the potential function Φ(x)=∑_wr_w(x) strictly increases after each iteration. Since each r_w(x) can increase at most |M| times, the sequence must terminate at a pure NE after no more than |M||W| iterations. We show this by arguing that during the sequence of updates, no matched man m wants to deviate unless he is rejected by his partner w because she has received a better offer, implying that r_w(x) must have increased. Note that this statement holds initially because x^0 is a good state where no matched man wants to deviate.    

More precisely, let us by contradiction assume that t is the first iteration at which some woman w loses her more preferred man during the sequence of updates, i.e., r_w(x^t)<r_w(x^t-1). This can happen only if man m that is matched to w at time t-1 (i.e., x^t-1_mw=1) is selected to update at time t whose best response is to leave w for another women w' (i.e., x^t_mw'=1). Otherwise, r_w(x^t-1) can only increase or remain the same because women reject any proposal other than their most preferred one. Note that since the game starts from a good state x^0, we must have t≥ 2.  

According to the payoff structures, the best response of player m at time t is to propose to his most preferred achievable woman w', i.e., among all women who rank m higher or equal to their match under x^t-1, m proposes to the woman w' which brings him the highest utility μ_mw'. Note that in particular μ_mw'>μ_mw. Thus, the first time τ≤ t-1 when man m proposed and got matched to woman w, the woman w' was his more preferred choice and the reason why m did not propose to w' at time τ was because w' was not achievable, i.e., x^τ-1_m'w'=1 for some man m'>_w' m. Since w' was not achievable for m in x^τ-1 but she is achievable in x^t-1 implies that r_w'(x^t-1)<r_w'(x^τ-1). Therefore, r_w'(x^τ')<r_w'(x^τ'-1) for some τ≤τ'≤ t-1, contradicting the choice of t. 



In fact, one can view the DA algorithm of <cit.> as a special version of the above lemma in which all the players are selected to best respond at each round, and the initial good state is the empty matching. Lemma <ref> shows that the stable matching game admits an ordinal potential function if the game starts from a good state (we refer to Appendix <ref> for another interpretation of this potential function). However, that does not imply that the stable matching game is an ordinal potential game <cit.>, which requires the existence of a potential function whose changes are aligned with players' payoff changes from any initial state. Nonetheless, the existence of such a “weak” potential function allows us to show the weak acyclicity of the stable matching game as defined next.


A better response path in a finite action noncoopertaive game is a sequence of pure strategy profiles x^1,x^2,…,x^L such that for each ℓ=1,…,L-1, x^ℓ+1 is obtained from x^ℓ by letting some player i_ℓ to play a better response. A game is called weakly acyclic if from any pure strategy profile x^0, there is a finite length better response path to a pure NE.



A pure NE x^*=(x^*_m,x^*_-m) of the stable matching game is called strict if each player has a unique best response at the equilibrium x^*, i.e., for every m and any pure strategy x_m≠ x^*_m, we have u_m(x_m,x^*_-m)<u_m(x^*_m,x^*_-m).   


Using the above definitions, we are now ready to state the main result of this section, which provides a decentralized and uncoordinated algorithm for learning stable matchings in general markets with unknown preferences. The algorithm is an adaptation of the sample experimentation algorithm for weakly acyclic games <cit.> to the stable matching game. Intuitively, each player m goes through a sequence of phase-dependent exploration/exploitation. At the end of each episode, player m evaluates his payoff for each action he has taken during the last episode. He then updates his baseline action (an action that brings him the highest reward in the past episode up to some tolerance) and sticks to that baseline action most of the time in the next episode while still exploring new actions with a small probability. The detailed description of the algorithm is summarized in Algorithm <ref>.   






Let p,ω∈ (0,1) be arbitrary probabilities and ϵ≤min{1-p/n, δ/4n, Δ-δ/4n}, where δ∈ (0, Δ). For each man m, there exists a sufficiently large episode length τ_m=τ_m(p,ϵ), such that if each player m follows Algorithm <ref> with episode length τ_m, for all sufficiently large times t>0, the pure strategy profile α^t will be a (fixed) stable matching with probability at least p.
   

First, we show that the stable matching game is a weakly acyclic game. If the game starts from a good state, by Lemma <ref>, any sequence of best responses by the players is a better response path that leads to a pure NE in |M||W| iterations. Thus, we only need to show that from any initial state, there is a better response path to a good state. This is also true by sequentially letting a matched man who wants to deviate (if there is such a man) to play his best response. In this way, either the number of matched men decreases due to collisions (which can happen at most |M| times), or between every two consecutive collisions, there could be at most |M| best response updates. Therefore, |M|^2 steps are enough to bring us from any arbitrary state to a good state, and the overall number of best response iterates to obtain a pure NE starting from any initial state is at most |M||N|+|M|^2.



Next, we note that any pure NE in the stable matching game is a strict NE.[In a weakly acyclic game, an arbitrarily high probability convergence can be guaranteed to a strict NE <cit.>. If we do not have strictness property, the dynamics may oscillate between two nonstrict NE points with a positive probability.] This follows directly by the fact that there are no ties between men's preferences in the stable matching game. More precisely, let α^* be a pure NE (stable matching) and assume u_m(α^*)=μ_mw. For any α_m≠α^*_m, either u_m(α_m,α^*_-m) equals 0 or μ_mw' for some w'≠ w. Because α^*_m is the best response of player m at the NE and there are no ties between man m's preferences, we must have μ_mw'<μ_mw. Since 0<μ_mw, in either case we have u_m(α_m,α^*_-m)<u_m(α^*_m,α^*_-m).

Finally, it was shown in <cit.> that the class of sample experimentation dynamics in weakly acyclic games converges with arbitrarily high probability to a strict NE if for any joint baseline action b^s at the start of an episode s, u^s_mw is an arbitrarily close estimation of u_m(w,b^s_-m) for sufficiently large episode length τ_m=τ_m(p,ϵ). Since Algorithm <ref> is an adaptation of the sample experimentation dynamics to the stable matching game, the convergence result follows from <cit.> if we can show that for any δ^*=min{δ/4, Δ-δ/4}>0, p<1, and 1-(1-ϵ)^n-1<δ^*/2, for sufficiently large τ_m we have[This is an equivalent version of Claim 3.4 in <cit.> to the stable matching game.]  

    ℙ{|u_mw^s-u_m(w,b^s_-m)|≥δ^*}≤ 1-p.

This statement is also true because during each episode, players sample their baseline actions (or explore a new action) independently of others, and moreover, μ^t_mw are i.i.d. realizations of a random variable with mean μ_mw. Thus, relation (<ref>) holds using the law of large numbers, which completes the proof.   


While Algorithm <ref> provides a decentralized and uncoordinated algorithm for learning stable matchings in general markets with arbitrarily high probability, the convergence time to a NE may take a long time. This seems inevitable to some extent in view of the exponential lower bound in terms of n such that the best response dynamics in general matching markets converge to a stable matching <cit.>. 






§ MATCHING MARKETS WITH STRONGER INFORMATION FEEDBACK


So far, we have assumed that men receive the minimum amount of information feedback as they interact with the market; namely, men only get to observe whether their proposal gets rejected or, if matched, they observe a random realization of their preference. In this final section, we consider the learning problem from a new perspective by addressing the following question: Is there stronger information feedback that will allow us to derive the market faster globally to a stable matching in a decentralized and uncoordinated fashion?  Therefore, we address the problem of learning 
a stable matching from an information feedback design perspective by showing that if the players are allowed to receive a bit more information as feedback, then the stable matching game will become a monotone game, hence allowing a wide range of decentralized and uncoordinated learning algorithms for obtaining its NE points. 


Let us consider a simplified version of the stable matching game with the same set of players M and strategy spaces 𝒳_m, m∈ M, except that the payoff of player m is now given by   

    u_m(x)=∑_w∈ Wμ_mw(1-∑_k>_w mx_kw)x_mw.




The simplified payoff function (<ref>) can be viewed as the expected perceived reward for player m if he randomly and independently of others proposes to women according to his mixed strategy x_m∈𝒳_m but under slightly stronger feedback: if player m's proposal gets rejected from a woman w, he also gets to know how many men higher ranked than him proposed to w. In this way, each man m learns how many more preferred men compete with him to get matched to woman w. Such information feedback can be implemented by using a waiting list order for each woman, which shows the ranking of a rejected man m in that list. Henceforth, we refer to such information feedback as the waiting list feedback. 

Let α_k∈ W denote the random woman that player k proposes to her, which is drawn independently from his mixed strategy distribution x_k. Under the waiting list feedback information, player m has access to the information ∑_k>_w m1_{α_k=w} if he proposes to w, and hence can estimate his perceived payoff by proposing to a woman w as μ_mw(1-∑_k>_w m1_{α_k=w}),  Therefore, the expected perceived payoff of player m under the waiting list feedback equals

    𝔼[∑_w∈ Wμ_mw(1-∑_k>_w m1_{α_k=w})1_{α_m=w}]   =∑_w∈ W𝔼[μ_mw(1-∑_k>_w m1_{α_k=w})|α_m=w]ℙ(α_m=w)   =∑_w∈ Wμ_mw(1-∑_k>_w mx_kw)x_mw=u_m(x),
    
where the second equality holds by independency of α_k ∀ k∈ M. 

The motivation for waiting list feedback and the payoff function (<ref>) is that if we view 1-∑_k>_w mx_kw as the remaining fractional capacity of women w for being matched, using the approximation 1-∑_k>_w mx_kw∼ e^-∑_k>_w mx_kw, one can think that man m loses his interest to offer women w exponentially in terms of the occupied capacity of woman w by higher preferred men. One immediate consequence of such simplified payoff functions is the following proposition whose proof follows nearly verbatim arguments as those in Theorems <ref> and <ref>, and is omitted from here for the sake of brevity.


Consider the stable matching game with payoff functions (<ref>). Then, x^* is a pure NE if and only if it is the characteristic vector of a stable matching. Moreover, any mixed NE of this game with full support can be rounded in a decentralized manner to a pure NE.  



Next, we consider the following definition of monotone games <cit.>. 

A continuous-action game (M,{U_m}_m∈ M, {𝒳}_m∈ M) is called monotone if for any two action profiles x,x'∈𝒳_1×⋯×𝒳_|M|, we have (x'-x)^T(F(x')-F(x))≤ 0, where 
    F(x)=(∇_m U_m(x), m∈ M)^T.
  


An important property of monotone games is that they admit a variety of decentralized learning algorithms that converge globally to a NE, which, under further assumptions, it can be shown that the converge rate could be exponential or polynomial <cit.>. The following theorem shows that the regularized stable matching game with waiting list feedback is indeed a monotone game under a suitable assumption on the preferences. 
 


Consider the regularized stable matching game with list feedback, where each player aims to maximize its regularized payoff function given by U_m(x_m,x_-m):=u_m(x_m,x_-m)-β/2x_m^2. Then, for any two strategy profiles x',x, we have

    (x'-x)^T(F(x')-F(x))=-1/2∑_w∈ W(x_w'-x_w)^TQ^w(x_w'-x_w),
 
where x_w is the vector x_w=(x_mw, m∈ M)^T, and Q^w is a symmetric matrix whose diagonal entries are 2β and all the entries in row m up to the diagonal entry in that row equal μ_mw.  In particular, if β>nμ_max/2, the regularized stable matching game with waiting list feedback is a monotone game.  


Note that due to the linearity of U_m(x) with respect to player m's strategy x_m, we have 

    ∇_mw U_m(x)=μ_mw(1-∑_k>_w mx_kw)-β x_mw, ∀ m,w,

where ∇_mw U_m(x) denotes the gradient of U_m(x) with respect to x_mw. Thus, we can write

    (x'   -x)^T(F(x')-F(x))=∑_m (x_m'-x_m)^T (∇_m U_m(x')-∇_m U_m(x))   =∑_m∑_w (x_mw'-x_mw)(∇_mw U_m(x')-∇_mw U_m(x))   =∑_m∑_w -μ_mw(x_mw'-x_mw)(∑_k>_w mx'_kw-∑_k>_w mx_kw+β(x'_mw-x_mw))   =-∑_m∑_w∑_k>_w mμ_mw(x_mw'-x_mw)(x'_kw-x_kw)-β∑_m∑_w (x'_mw-x_mw)^2   =-∑_w(∑_m∑_k>_w mμ_mw(x_mw'-x_mw)(x'_kw-x_kw))-β∑_m∑_w (x'_mw-x_mw)^2   =-∑_w(∑_m∑_kμ_mw1_{k>_w m}(x_mw'-x_mw)(x'_kw-x_kw))-β∑_w∑_m (x'_mw-x_mw)^2.

For each woman w, let us define a vector x_w=(x_mw, m∈ [M])^T, and an n× n incidence preference matrix A^w whose entries are given by A^w_mk=μ_mw1_{k>_w m}, ∀ m,k∈ [n]. Then, using (<ref>) we have

    (x'-x)^T(F(x')-F(x))   =-∑_w(∑_m∑_k A^w_mk(x_mw'-x_mw)(x'_kw-x_kw))       -∑_w (x'_w-x_w)^T(β I)(x'_w-x_w)   =-∑_w(x_w'-x_w)^T(A^w+β I) (x_w'-x_w)   =-1/2∑_w(x_w'-x_w)^TQ^w(x_w'-x_w),

where Q^w=(A^w)^T+A^w+2β I. Next, we note that permuting the rows and columns of a matrix by the same permutation matrix preserves the semi-definiteness property as it only changes the basis of its eigenvectors. Thus, without loss of generality, we may assume 1>_w 2>_w …>_w n. However, in that case, we may assume that for any w, the matrix A^w will be a lower triangular matrix whose diagonal entries are all zero. In particular, Q^w=(A^w)^T+A^w+2β I would be a symmetric matrix whose elements in each row m up to the diagonal entry are identical and equal to μ_mw. Finally, for sufficiently large β>nμ_max/2, the matrix Q^w is positive semidefinite for all w, and we have -(x_w'-x_w)^TQ^w(x_w'-x_w)≤ 0  ∀ w. This shows that (x'-x)^T(F(x')-F(x))≤ 0, and completes the proof.  



The monotonicity property in Theorem <ref> holds when the regularizer parameter β is sufficiently large. The regularizer implies that players tend to maximize their approximate payoff functions. Thus, instead of learning a NE for the payoff functions (<ref>), they learn an ϵ-NE where the approximation error ϵ grows as β increases. As a result, applying decentralized learning algorithms to learn a NE of the regularized game only guarantees an ϵ-stable matching in the sense that the players will achieve a solution in which each player would be ϵ worse off than what he could achieve in a stable matching. Therefore, the regularizer parameter trades off between the market's convergence speed and the accuracy of the learned stable matching.  



Finally, we note that without imposing any assumption on the agents' preferences, one cannot expect the stable matching game to be a monotone game. The reason is that monotone games imply the global uniqueness of a NE <cit.>. However, by Proposition <ref>, any stable matching is a pure NE, and for general matching markets, there could be many stable matchings under arbitrary preferences. Moreover, the assumption of waiting list feedback is not necessary for faster convergence in any general market; it is merely one choice that we consider in this work, and there could be other types of information feedback that allow faster convergence. Nevertheless, our game-theoretic approach to the stable matching problem provides valuable insights on how to properly design the utilities and the information feedback to devise faster decentralized and uncoordinated algorithms for learning stable matchings.   









§ CONCLUSIONS


In this work, we considered the problem of learning stable matchings with unknown preferences in a fully decentralized and uncoordinated manner. Using a game-theoretic formulation, we showed how to leverage the rich literature for learning NE in noncooperative games to derive the matching market to its stable points in a principled way. In particular, we established several global and local convergence results with performance guarantees for learning stable matchings in hierarchical markets, general markets, and under weak/strong information feedback. Our results provide new insights for learning stable matchings through NE learning in noncooperative games and bridge the discrete problem of learning a stable matching with learning NE in continuous-action games.  

This work opens several avenues for future research. One interesting direction is to characterize an information-theoretic limit below and above which efficient learning of a stable matching in a fully decentralized and uncoordinated fashion may or may not be possible. Moreover, our waiting list feedback is merely an initial attempt to design faster decentralized and uncoordinated algorithms under richer information feedback. Extending our results to include other types of information feedback and studying their effect on the convergence speed of the market to its stable points would be another interesting research direction.      





IEEEtran








§ SUPPLEMENTARY MATERIALS





 §.§ Freedman's Concentration 
Inequality for Martingales
 

Let {ζ_τ}_τ=1^t be a martingale difference sequence with max_τ∈ [t]|ζ_τ|≤ u, and consider the corresponding martingale S^t=∑_τ=1^tζ_τ with S^1=0. Let V^t=∑_τ=1^t𝔼[ζ^2_τ|ℱ^t-1] be the sum of the conditional variances. Then, for any x>0 and v∈ℝ, we have 
    ℙ{|S^t|≥ x, V^t≤ v^2}≤ 2exp(-x^2/2(v^2+ux)).



The proof can be found in <cit.>.   




 §.§ Proof of Theorem <ref>



Given a mixed NE x, let ℋ(x)=(M∪ W, E(x)) be the induced bipartite graph by x, where E(x)={(m,w): x_mw>0} denotes all the edges with fractionally positive proposals. Moreover, let us denote the neighboring nodes of m and w in ℋ(x) by N_m(x) and N_w(x), respectively. Since x is a NE, the value v_mw(x)=μ_mw∏_k>_w m(1-x_kw) should be the same for all w∈ N_m(x) and equals to u_m(x). Otherwise, using properties (i) and (ii), player m could strictly increase his payoff by putting his entire strategy mass on the woman w^*=_w∈ N_m(x)v_mw(x). The fact that u_m(x)=v_mw(x) ∀ w∈ N_m(x) follows because x_m is a probability distribution that is supported over N_m(x).

Next, we note that a man m cannot be the most preferred choice for more than one woman in ℋ(x). Otherwise, let w_1,w_2∈ N_m(x) be two different women who rank m the highest among their fractional matches in N_w_1(x) and N_w_2(x), respectively. Then, we have 
    ∏_k>_w_1 m(1-x_kw_1)= ∏_k>_w_2 m(1-x_kw_2)=1,
 which together with the above argument implies that

    μ_mw_1=μ_mw_1∏_k>_w_1 m(1-x_kw_1)=v_mw_1(x)=u_m(x)=v_mw_2(x)=μ_mw_2∏_k>_w_2 m(1-x_kw_2)=μ_mw_2.
  
This contradicts the fact that there are no ties between the preferences of man m.    

Since each woman receives at least one fractional proposal, N_w(x)≠∅ ∀ w. Let m_w be the unique most favorite man of w in N_w(x). Using the above argument, the elements of {m_w, w∈ W} must be distinct such that ℳ_1={(m_w,w), w∈ W} forms a matching. Viewing this matching from men's side, one can represent ℳ_1 in the form of ℳ_2={(m,w_m), m∈ M}, where w_m=_w∈ N_m(x)μ_mw is the least preferred woman among those that man m fractionally proposed to. To see this equivalence, fix a man m and let w_0 be his match in M_1, i.e., m_w_0=m. Observe that each w'∈ N_m(x)∖{w_0} has a unique distinct most favorite man m_w'≠ m. That means that 

    v_mw'(x)=μ_mw'∏_k>_w' m(1-x_kw')≤μ_mw'(1-x_m_w'w')<μ_mw'.

On the other hand, since w_0 ranks m the highest among those who propose to her, we have v_mw_0(x)=μ_mw_0∏_k>_w_0 m(1-x_kw')=μ_mw_0. Since at a NE v_mw_0(x)=v_mw'(x), we must have μ_mw_0<μ_mw', ∀ w'∈ N_m(x)∖{w_0}, which implies w_0=w_m. Thus, for any pair (m_w_0,w_0)∈ℳ_1 there is a pair (m,w_m)∈ M_2 such that (m_w_0,w_0)=(m,w_m), which shows that ℳ_1=ℳ_2. 

Finally, we will show that the binary characteristic vector of the matching {(m,w_m), m∈ M}, denoted by x̂, is indeed a pure NE. We will show that by arguing that any strictly better deviation in x̂ is also a strictly better deviation in x, contradicting the fact that x is a NE. On the contrary, let m_w be a man who wants to deviate from his pair w to a woman w'≠ w. Such a deviation is beneficial to player m_w only if m_w'<_w' m_w. Otherwise, his payoff will reduce from u_m_w(x̂)=μ_m_ww to 0. Henceforth, we may assume m_w'<_w' m_w. The fact that the payoff of player m_w is currently u_m_w(x̂)=μ_m_ww and he wants to deviate to w' implies that μ_m_ww<μ_m_ww'. As m_w is the most preferred man for woman w in ℋ(x), we also have u_m_w(x)=μ_m_ww. Since m_w'<_w' m_w, the woman w' would prefer m_w over all the men in N_w'(x). Thus, in the mixed NE x, man m_w could also deviate by solely proposing to w', in which case his payoff would strictly increase from u_m_w(x)=μ_m_w,w to μ_m_ww', which is a contradiction.    






 §.§ Proof of Lemma <ref>



Fix a player m and assume X^*_mw^*=1. Since X^* is a pure NE, for any w≠ w^* we have

    v_mw^*(X^*)=μ_mw^*> v_mw(X^*)=μ_mw∏_k>_w m(1-X^*_kw)∈{0, μ_mw}.

On the other hand, we have 

    v_mw^*(x)-v_mw^*(X^*)   =μ_mw^*∏_k>_w^* m(1-x_kw^*)-μ_mw^*   ≥ -μ_mw^*∑_k>_w^* mx_kw^*   ≥ -μ_mw^*/2x-X^*_1,

where the second inequality holds because X^*_mw^*=1 implies X^*_kw^*=0 ∀ k≠ m, and hence 1/2x-X^*_1≥∑_k>_w^* mx_kw^*. Now we consider two cases: 

i) If v_mw(X^*)=μ_mw, then v_mw(x)≤μ_mw< μ_mw^* ∀ x, which implies v_mw^*(X^*)-v_mw(x)≥Δ. Combining this relation with (<ref>) and because x-X^*_1≤Δ/μ_max, we get v_mw^*(x)-v_mw(x)>Δ/2>c. 

ii) If v_mw(X^*)=0, there exists k>_w m such that X^*_kw=1. Then, v_mw(x)≤μ_mw(1-x_kw)= μ_mw(X^*_kw-x_kw) ≤1/2μ_mwX^*-x_1, and we have v_mw^*(X^*)-v_mw(x)≥μ_mw^*-1/2μ_mwX^*-x_1. Combining this relation with (<ref>) and because x-X^*_1≤μ_min/μ_max, we get v_mw^*(x)-v_mw(x)>μ_min/2> c.    




 §.§ Proof of Lemma <ref>


i) Suppose X^t-X^*_1≤c/250n^2, then X^t_m-X^*_m_1=2(1-X^t_mw^*)≤c/250n^2 ∀ m. Since X^t_m is a probability distribution, we can write 
    X^t_mw^*/X^t_mw≥1-c/500n^2/c/500n^2= 500n^2/c-1, ∀ m,w≠ w^*.
 
Therefore, for any m,w≠ w^*, we have η^tL̂^t-1_mw^*-η^tL̂^t-1_mw≥ln (500n^2/c-1)≥ln (n^2/2c)+6.

ii) Suppose η^tL̂^t-1_mw^*-η^tL̂^t-1_mw≥ln(n^2/2c) ∀ m,w. Then, we can write

    X^t_mw^*/X^t_mw=exp(η^tL̂^t-1_mw^*-η^tL̂^t-1_mw)≥n^2/2c ∀ m,w≠ w^*,

which implies X^t_mw^*≥1/1+2c/n≥ 1-2c/n. Thus, X^t_m-X^*_m_1=2(1-X^t_mw^*)≤4c/n ∀ m, and hence X^t-X^*_1≤ c.

iii) Suppose η^tA^t-1=b+β, where b=ln(n^2/2c)+3 and β≥ 0. Then, we have

    η^t+1A^t   =η^t+1A^t-1+η^t+1(v_mw^*(X^t)-v_mw(X^t))   ≥η^t+1A^t-1+η^t+1 c   = η^t+1/η^t(b+β)+η^t+1 c   =b+η^t+1/η^tβ+(η^t+1 c-(1-η^t+1/η^t)b)   ≥ b+η^t+1(c-(1/η^t+1-1/η^t)b)   ≥ b,
 
where the last inequality holds by the fact that 1/η^t+1-1/η^t≤c/b ∀ t≥ t_0.




 §.§ Proof of Theorem <ref>



Using identical arguments as in the proof of Theorem <ref>, with probability at least 1-δ,

    |η^tL̂^t-1_mw'-η^tL̂^t-1_mw-η^tA^t-1|≤ 3,  ∀ t≥ 1, ∀ m, w w',

where we recall that A^t-1= ∑_τ=1^t-1(v_mw'(X^τ)-v_mw(X^τ)). Now let Ω be the event such that X^τ→ X^*, and assume X^*_mw>0 for some m,w. Conditioned on the event Ω, we have

    η^tA^t-1≤η^tL̂^t-1_mw'-η^tL̂^t-1_mw+3=ln(X^t_mw'/X^t_mw)+3,  ∀ t≥ 0.

For any w'≠ w, by taking limit from the above relation as t→∞, we get
lim sup_t→∞η^tA^t-1≤ln(X^*_mw/X^*_mw)+3≤ O(1).[Here, if X^*_mw'=0, by convention we define ln(X^*_mw'/X^*_mw)=-∞.] Now let κ:=v_mw'(X^*)-v_mw(X^*). We claim κ≤ 0. Otherwise, if κ>0, then for any sufficiently large t>t̅ we must have A^t-1>tκ/2. Given the choice of stepsize, we obtain lim_t→∞η^t A^t-1=lim_t→∞tη^tκ/2=∞, which contradicts the fact that lim sup_t→∞η^tA^t-1≤ O(1). Thus, κ≤ 0, which shows that if X^*_mw>0, then v_mw(X^*)=max_w'v_mw'(X^*), i.e., X^* is a NE.          




 §.§ An Alternative Form for the Potential Function in Lemma <ref>

One can represent the potential function in Lemma <ref> directly for womens' ordinal preferences. Given a mixed strategy profile x, let us associate a vector p_w(x)∈ℝ_+^|M| to each woman w whose coordinates are sorted in the order of w's preferences, such that if m_1>_wm_2>_w…>_w m_n, we let the jth coordinate of p_w(x) be given by 
    p_jw(x)=x_m_jw∏_k>_w m_j(1-x_kw),
 which is the probability that woman w is matched to her jth favorite man, assuming that the players propose to women independently and according to their mixed strategies x. We note that for a pure-strategy profile x∈{0,1}^|M||W|, p_w(x) would be a binary vector with all entries being zero except the jth one being one, where j is the index of the highest ranked man proposing to w. Therefore, Lemma <ref> asserts that after each best response move, the vectors {p_w(x), w∈ W} increase lexicographically with at least one of the vectors increases strictly. Now, if we again encode woman w's preferences in a cardinal vector λ_w=(λ_wm_1,…,λ_wm_n), then for a pure strategy profile x∈{0,1}^|M||W|, we have 
    r_w(x)=⟨ p_w(x),λ_w ⟩=∑_mλ_wmx_mw∏_k>_w m(1-x_kw),
 which coincides with the definition of r_w(x) in Theorem <ref> that picks the reward of the highest-ranked man proposing to w under x. Therefore, in the space of cardinal preferences and for any pure-strategy profile x∈{0,1}^|M||W|, one can rewrite the potential function in Theorem <ref> in a compact as 

    Φ(x)=∑_w r_w(x)=∑_m,wλ_wmx_mw∏_k>_w m(1-x_kw),

whose value increases after every best response update by a player. We note the similarity between the potential function (<ref>) and sum of players' payoffs ∑_mu_m(x)=∑_m,wμ_mwx_mw∏_k>_w m(1-x_kw), where one is written using mens' preferences {μ_mw} and the other using the womens' preferences {λ_wm}.



 §.§ A Gap Estimation Oracle for Unknown Δ.


Motivated by the Upper Confidence Bound (UCB) for stochastic bandits <cit.>, here we provide a gap estimation oracle, which, in combination with Algorithm <ref>, allows each player to adaptively adjust his mixing parameter based on the estimated gap he has obtained so far. The integration of this oracle with Algorithm <ref> allows the algorithm to be more practical to the situation when a lower bound for the gap Δ is unknown. In particular, using standard concentration bounds for the UCB algorithm, one can show that the event that the estimated gap is larger than the true gap decays fast as time increases, hence only introducing a small error term to the overall performance of the algorithm.     

 








 Paper Title: Mobility-Aware Federated Self-supervised Learning in Vehicular Network.



Article Title]Mobility-Aware Federated Self-supervised Learning in Vehicular Network










1]Xueying Guxueyinggu@stu.jiangnan.edu.cn


[1,2]Qiong Wuqiongwu@jiangnan.edu.cn


3]Pingyi Fanfpy@tsinghua.edu.cn


4]Qiang Fanqf9898@gmail.com




[1]School of Internet of Things Engineering, Jiangnan University, Wuxi, 214122,  China

[2]State Key Laboratory of Integrated Services Networks, Xidian University, Xi'an , 710071, China


[3]Department of Electronic Engineering, Beijing
		National Research Center for Information Science and Technology, Tsinghua
		University, Beijing, 100084, China

[4]Qualcomm, San Jose CA, 95110, USA




Federated Learning (FL) is an advanced distributed machine learning approach, that protects the privacy of each vehicle by allowing the model to be trained on multiple devices simultaneously without the need to upload all data to a road side unit (RSU). This enables FL to handle scenarios with sensitive or widely distributed data. However, in these fields, it is well known that the labeling costs can be a significant expense, and models relying on labels are not suitable for these rapidly evolving fields especially in vehicular networks, or mobile internet of things (MIoT), where new data emerges constantly. To handle this issue, the self-supervised learning paves the way for training without labels. Additionally, for vehicles with high velocity, owing to blurred images, simple aggregation not only impacts the accuracy of the aggregated model but also reduces the convergence speed of FL. This paper proposes a FL algorithm based on image blur level to aggregation, called FLSimCo, which does not require labels and serves as a pre-training stage for self-supervised learning in the vehicular environment. Simulation results demonstrate that the proposed algorithm exhibits fast and stable convergence.



















[
    [
    Accepted for publication in Astrophysics & Space Sciences (ApSS)
====================================================================





§ INTRODUCTION


The development of Internet of Things (IoT) makes many practical applications available to people, such as automatic navigation, weather forecast and self-driving systems, which not only reduces the probability of accidents, but also improves the happiness of life <cit.>. Training a good model can make the practical application more robust, which requires a lot of data. Moving vehicles can constantly collect new data by their devices, most of which can be captured in the form of images via camera mounted on the roof of the vehicle. Compared with the traditional method of collecting image data with onboard radar, this way greatly reduces the cost and technical requirements, making the shift of image collection paradigm to vision-based measurement (VBM) has broad promising <cit.>.

After the acquisition is completed, the vehicle process the image data to realize image recognition and classification. It provides the necessary information for self-driving and driver assistance systems <cit.>, and helps drivers perceive and understand their surroundings. Especially in nighttime or bad weather conditions, image recognition and classification models can be critical valuable in identifying road signs, other vehicles, and obstacles, providing specific object classification, improving driver visibility, and improving driving safety. Therefore, image recognition and classification in self-driving plays a key role in reducing the risk of traffic accidents and improving road safety.

As mentioned earlier, training a robust model requires a lot of data. However, the data stored locally on each vehicle may be private and sensitive, and drivers are reluctant to share it with each other in order to protect their privacy. If the vehicles do not share information, they can only train the local model using their own data <cit.>. Due to the single driving environment, the image categories will be skewed, so that the trained model does not have good transference. Compared to single vehicle training, FL can solve this issue by aggregating different vehicles' local trained models with local data. This approach can take a large data set into account in the training process without the need for vehicles to share local data, resulting in superior performance and enhanced generalization <cit.>.

However, this approach still faces the following two issues: low image quality and missing or incorrect labels. The first one is due to the motion blur caused by the movement of vehicles, and the other is because of the high cost and incorrectness of labeling.

In a vehicular network, a high velocity usually leads to insufficient exposure time for the camera sensor, causing motion blur. In this paper, we consider the motion blur caused by different vehicle velocities in the training process, and adjust corresponding weights of the model parameters for FL model aggregation. 

Self-supervised learning <cit.> can abandon labels for pre-training, and thus mitigates the impact of incorrect labels on the model and removes the cost of the labeling process, which makes it suitable for IoT, where new data is constantly appearing.

To the best of our knowledge, few research works have taken into account both the privacy protection of vehicles and the blurring of images in real scenes, as well as the cost of labels during the training of models.

In this paper, we build a framework, called FLSimco[The source code has been released at: https://github.com/qiongwu86/FLSimCo], to train the vehicle side local model by self-supervised learning at dual temperatures. Only upload the local models of the vehicles, after the completion of local training, and then  the RSU through the FL aggregated the received local models, both to protect the privacy of the vehicle, and training a generation model.

The remaining of this paper is organized as follows. In Section <ref>, we will review related works. The Section <ref> details the system model we designed. Section <ref> will showcase and discuss the experimental results obtained in the simulation environment. Finally, we will summarize the research findings and present conclusions in the Section <ref>.




§ REVIEW OF RELATED WORKS


In recent years, the emergence of deep neural networks, particularly convolutional neural network (CNN), has facilitated significant advances in computer vision benchmarks. Self-supervised learning is a special form of unsupervised learning, and its main feature is that it uses information inherent in the data itself to help machine learning models get a better representation, thereby improving performance across a variety of tasks<cit.>.

Until now, research on unsupervised learning has focused on mining shared features between pre-trained tasks and downstream tasks <cit.>. Wu et al. <cit.> introduced noise contrast estimation (NCE) loss as an objective function to distinguish between different instances. Each image was treated as a positive sample and the others as negative samples, effectively treating each image as a category, constituting an instance-level classification task. SimCLR <cit.> has maximized the sample characteristics of similarity to study representations. It abandoned the memory library <cit.> and generated more negative samples by increasing the batch size. The core idea of these methods is to encourage models to place similar data representations(positive sample pairs) together in the embedding space, while separating dissimilar representations(negative sample pairs) to learn feature extraction.
Since these methods do not consider the fact that the distance between samples of the same category should be smaller than that of different categories' samples, but treat all pairs equally. In this way, a large number of negative samples are required. It will pose significant challenges to storage and computing power in the vehicle environment, increasing the requirements of hardware configuration levels.

MoCo <cit.>, as a self-supervised learning method for CNN <cit.>, has made an important contribution by introducing queue and momentum updating techniques to create a large and consistent dictionary conducive to contrast learning. It replaced the original memory library with a queue as an additional data structure to store these negative samples. Momentum encoder k is used to replace the original loss constraint term. The key advantage of the MoCo family is the use of the queue as negative samples and the momentum update of the queue, which greatly reduces the need for vehicle storage capacity and computing power in local training. <cit.> used MoCo for the pre-training model weights on ImageNet and achieves excellent results in downstream fine-tuning tasks. <cit.> employed MoCo for unsupervised learning with large amounts of unlabeled data on remote servers. The local vehicles downloaded the trained model and used it as the initialization model.


These methods are characterized by the presence of a large default datasets that can be used for unsupervised learning, and the recognition accuracy increases with the expansion of the dictionary size<cit.>. Simultaneously, as the dictionary size increases, the recognition accuracy gradually improves. However, it is important to note that, larger dictionaries also requires larger memory size, increasing the storage cost and the computing capability of hardness. SimCo <cit.> attributed the need for a large dictionary in MoCo to hardness awareness between anchors, and the author believed that the consistency between positive and negative keys was more crucial than that between negative keys. Therefore, it utilized dual temperatures to differentiate inter-anchor samples and intra-anchor samples, as shown in Fig. <ref>, which eliminates the need for a dictionary of negative samples.

Due to the imbalance of data categories, the data distribution faces the challenge of non-independent identically distributed (non-IID) characteristics <cit.>, requiring more training rounds and interaction frequency to improve accuracy. However, increased interaction means require more computing and network resources <cit.>. To protect the privacy of local data and its distribution, FedCo proposed to employ FL to aggregate local models. Each vehicle passed its own k value to the RSU for forming a bigger new queue. However, a new queue consisting of the k values of different vehicles violates MoCo's inherent requirement for consistency in negative key pairs. When the vehicle uploaded the trained model and the corresponding k value, it can, to some extent, reconstruct the original input. As such, it does not ensure privacy protection and defeats the original purpose of using FL. <cit.> used federated self-supervised learning for a single-event classification, adding a binary classifier for each new event by adhering to a one-to-many paradigm. However, these FL algorithms do not take into account the effect of image blurring. As a result, they cannot effectively simulate real-world scenarios.

In this paper, we will train the local model on the vehicles side by self-supervised learning with dual temperatures. After the training is completed, only the local models of vehicles are uploaded, and then the received local model is aggregated in RSU by  FL, which not only protects the privacy of vehicles, but also produces a model with better transform through distributed training. At the same time, considering that the images collected in the actual vehicle environment may be blur, the blur level is used as the weight during aggregation, which makes the aggregated model more reliable and stable.







§ SYSTEM MODEL




 §.§ System design



As shown in Fig. <ref>, let's consider a scenario where an RSU is deployed at the intersection and there are several vehicles driving in coverage of the RSU. Each vehicle driving straightly when cross the intersection. We firstly build a truncated Gaussian distribution model as the mobility model for each vehicle. At the same time, each vehicle is equipped with a large mount of images, called local data, captured by camera mounted in the roof before entering the coverage of the RSU. It is worth noting that vehicles coming from different directions may capture different categories of images from each other.

In each FL round,  each vehicle downloads the global model from the RSU as it enters the coverage of the RSU. Then each vehicle adopts the global model for the pre-training of the self-supervised learning to classify each category of captured local image data. During this process, the vehicle randomly selects a specified number of images from its local data <cit.>. When the local training is complete, the vehicle uploads its local model to the RSU. Then, the RSU  aggregates the local models from different vehicles to obtain a new global model. After that, FL moves on to the next round until the maximum round R^max is reached. Next, we will describe the models for each round in the system.



 §.§ Mobility model

Similar with <cit.>, we adopt the following mobility model to reflect the real vehicle mobility. We consider the velocities of different vehicles are independent identically distributed (IID). The velocity of each vehicle follows the truncated Gaussian distribution. Let N_r be the number of vehicles traveling within the coverage area of RSU in the round r, v_n_r be the velocity of vehicle n_r∈[1, N_r], and v_min and v_max be the minimum and maximum velocity of vehicles, respectively, namely velocity v_n_r∈[v_min,  v_max], and the probability density function of v_n_r is expressed as  <cit.>

    f(v_n_r) = {e^ - 1/2σ ^2(v_n_r - μ )^2/√(2πσ ^2) [erf(v_max - μ/σ√(2)) - erf(v_min - μ/σ√(2))]
    v_min≤v_n_r≤v_max,
    
    		0                    otherwise.
    	.
where erf(μ, σ^2) is the Gaussian error function of velocity v_n_r with mean μ and variance σ^2. 




 §.§ FLSimCo algorithm

Self-supervised learning requires a large amount of unlabeled data. In the vehicle scene, it is easy for the vehicles to collect enough image data during driving. However, due to the limitations of the environment, storage and camera perspective, the categories of image data of each vehicle are limited, which may not meet the IID <cit.> requirement. To be specific, when the vehicle uses the image data stored by itself for local training, a model shewed to its own image categories is obtained, namely a model with poor generation. After the training is completed, the vehicle uploads the parameters of local model, and the RSU aggregates the received local model parameters from different vehicles, making the aggregated model more general and effectively solving the data non-IID problem.

We establish a novel algorithm called FLSimCo, which means simplified MoCo with no momentum in FL. Typically, the specific FLSimCo process is as follows:

Step 1, initialization: The RSU initializes a global model θ^0 for an encoder f^0 with random initial parameters.

Step 2, local training: For round r, N_r vehicles take part in the FL, where each vehicle n_r downloads the model θ^r of encoder f^r from the RSU. 

The blur level L_n_r of local image data in the vehicle n_r can be represented as <cit.>:

    L_n_r=Hs/Qv_n_r.

In this context, v_n_r is the velocity of vehicle n_r. Hs/Q is the parameter of the camera, in which H represents the exposure time interval, s is the focal length, and Q represents pixel units.

For each image x_i∈{x_1, x_2, … x_i,… x_M}, after applying two different data augmentation methods π_1(·) and π_2(·),  and passing through the encoder f^r, we obtain anchor sample q_n_r^i, positive sample k_n_r^i and negative samples k_n_r^j:

    q_n_r^i=f^r[π_1(x_i)]=ResNet[π_1(x_i)]|_r, i∈[1,M],


    k_n_r^i=f^r[π_2(x_i)]=ResNet[π_2(x_i)]|_r, i∈[1,M],


    k_n_r^j=f^r(x_j)=ResNet(x_j)|_r, j∈[1,M] and j≠ i.


According to <cit.>, the loss can be transformed into


    ℒ_q_n_r^i^DT=   -sg[(W_β)_n_r^i/(W_α)_n_r^i]×
       logexp(q_n_r^i· k_n_r^i/τ_α)/exp(q_n_r^i· k_n_r^i/τ_α)+∑_j=1^Kexp(q_n_r^i· k_n_r^j/τ_α),

and

    (W_β)_n_r^i=1-exp(q_n_r^i· k_n_r^i/τ_β)/exp(q_n_r^i· k_n_r^i/τ_β)+∑_j=1^Kexp(q_n_r^i· k_n_r^j/τ_β),


    (W_α)_n_r^i=1-exp(q_n_r^i· k_n_r^i/τ_α)/exp(q_n_r^i· k_n_r^i/τ_α)+∑_j=1^Kexp(q_n_r^i· k_n_r^j/τ_α),
where ℒ_q_n_r^i^DT is the dual-temperature (DT) loss of the i-th image of the vehicle's anchor sample in the round r. K represents the queue length and sg[·] indicates the stop gradient. The denominator in the above equations consists of one positive sample and K negative samples. It is noting that '·' in Eq. (<ref>) - Eq.  (<ref>) means dot product. τ_α and τ_β are different temperature hyper-parameters <cit.>, and controls the shape of the samples distribution.  Based on the different requirements for dictionary size by inter-anchor and intra-anchor, and the temperature's ability to control the feature distribution, different temperatures will be used to control the distance between different samples, thus eliminating MoCo's dependency on a large dictionary. This will remove the inter-anchor's reliance on a large dictionary. 

The objective function can be defined as minimizing the loss function <cit.>

    θ̂_n_r=θ_n_r argmin1/M∑_i=1^Mℒ_q_n_r^i^DT(θ_n_r,  q_n_r^i, k_n_r^i, k_n_r^j),
where θ_n_r represents the parameter of local model of vehicle n_r in round r. Each vehicle performs the local training to approach θ̂_n_r according to the following algorithm:

    θ_n_r θ_n_r-η^r∇ℒ^DT(θ_n_r, q_n_r^i, k_n_r^i,  k_n_r^j),
where η^r represents the learning rate for the  round r.

Step 3, Upload model: N_r vehicles upload the parameter {θ_1, θ_2…θ_n_r…θ_N_r} of local models after their local training finished, along with the velocity {v_1,v_2,… v_n_r… v_N_r} of each vehicle.

Step 4, Aggregation and Update: After receiving the trained models from N_r vehicles, the RSU employs a weighted federated algorithm to aggregate the parameters of N_r models based on the blur level L_n_r. The expression for the aggregated model is


    θ^r+1=∑_n_r=1^N_r[(∑_n_r=1^N_rL_n_r-L_n_r)θ_n_r/∑_n_r=1^N_r(∑_n_r=1^N_rL_n_r-L_n_r)].








Repeat the above step 2 to step 4 until reaching max round R^max.



§ RESULTS


In this section, we will introduce the setup of experiments, show the results, and give a brief explanation.



 §.§ Experimental setup


We adopt an improved ResNet-18 with a fixed dimension of 128-D as the backbone model and employ SGD (Stochastic Gradient Descent) as the optimizer. In addition, inspired by the concept of cosine annealing, we gradually reduce the learning rate at different stages of training to improve the training efficiency of the model <cit.>.

Datasets: This paper uses the datasets CIFAR-10 <cit.> with 50,000 unlabeled images as the training datasets, which is distributed in 10 different categories, i.e., each category consists of 5,000 images.




(1) IID: Samples that follow IID are independent of each other and share the same distribution. The appearance of each sample is not affected by the other samples. We uniformly assigned 10 categories from 50,000 images in CIFAR-10 to 95 vehicles, ensuring that each vehicle have at least 520 images available for training. As can be seen from the Fig. <ref>, the categories on each vehicle are evenly distributed.

(2) Non-IID: IID provides theoretical convenience, but in a practical scenario, there is very little data that meets the IID requirement. Therefore, it is importance to use data under non-IID conditions to transfer the model to the real scenario. As shown in Fig. <ref> and <ref>, the Dirichlet distribution parameter α is 0.1 and 1, respectively <cit.>. Clearly, the smaller the α, the larger the gap between data categories. We set the Dirichlet distribution parameter α to 0.1 to simulate the Non-IID data in the vehicular scene, in order to simulate the uneven distribution of the  image categories collected by each vehicle due to the limited viewing perspective and environmental constraints. In order to ensure that there is enough data for local training, for CIFAR-10 we ensure that there are at least 520 images per vehicle.




 §.§ Results


According to FedCo <cit.>, in the round r of the training process, we set each vehicle uploads all stored k-values (with a batch size set to 512 in the experiment) to the RSU (global queue set to 4096) to update the global queue. 

As shown in Fig. <ref>, we consider the FedCo simulation result as baseline1. In addition, our proposed method FLSimCo is represented by a red line in the same diagram. It can be observed that FLSimCo outperforms FedCo given the same number of rounds. From FedCo's perspective, the update queues with k values from different vehicles has compromised the Negative-Negative consistency requirement in MoCo, resulting in a less accurate approach. Meanwhile, FedCo enables the vehicle's own k value to be uploaded to the RSU, which also goes against FL's purpose of protecting user privacy. In addition, we also conducted experiments on Non-IID data sets, and the results show that the training performance of Non-IID data sets is slightly lower than that of IID data sets, but still better than FedCo algorithm.

 

For Fig. <ref>, we assume that some images will be blurred when the vehicle velocity exceeds 100 km/h and conduct three comparative experiments.To better demonstrate its performance, we also introduce two baseline algorithm.  FedAvg <cit.> is employed as baseline1, which averaging the model parameters accordingly. Baseline2 indicates that RSU will discard the model trained by the vehicle velocity exceeding 100 km/h, that is, discard the local model trained with the blurred images, and then use the FedAvg aggregation model parameters.

From the results of baseline1 and baseline2, it is obvious that it is not beneficial to drop the model trained with motion blurred images(baseline2). Because it drops some models during the aggregation process, baseline2 has the lowest number of models used in the global training. This suggests that the number of models involved in aggregation is more critical than the quality of the models. Finally, our proposed aggregation method assigns smaller weights to models trained by faster vehicles. It can be seen that the proposed algorithm can effectively reduce the fluctuation of the loss function, while facilitating the loss function and converge to a smaller value faster. This demonstrated that proposed approach can reduce the effect of image blur and also increase the global model training speed.




§ CONCLUSION


In this paper, we propose a FLSimCo algorithm. First, on the basis of self-supervised learning, the advantages of FL are used. The main goal is to solve the problem of data category skew and to train a  model with well generation without sharing private data. By integrating self-supervised learning and FL, the algorithm can improve the model's adaptability to data heterogeneity to achieve a more robust pre-training model. More importantly, considering the data blur caused by vehicle movement, the blur level is used as the weight of federated aggregation, which makes the algorithm more suitable for real scenarios.

In summary, the FLSimCo algorithm ensures the protection of vehicle privacy, and requires no labeled image data, while addressing the aggregation challenges in vehicular environments characterized by significant data shew and image blurring due to vehicle movement. 



§ DECLARATIONS
 

Competing interests: The author declare that there is no competing interest.

Open Access: This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article?s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licen ses/by/4.0/.

00

wsy01
Wu Q, Wang S, Ge H, Fan P, Fan Q, Letaief K (2023) Delay-sensitive Task Offloading in Vehicular Fog Computing-Assisted Platoons. IEEE Transactions on Network and Service Management.https://doi.org/10.1109/TNSM.2023.3322881. 

ss01
Wu Q, Shi S, Wan Z, Fan Q, Fan P, Zhang C (2023) Towards V2I Age-aware Fairness Access: A DQN Based Intelligent Vehicular Node Training and Test Method. Chinese Journal of Electronics. pp. 1230-1244. 

ldx01
Long D, Wu Q, Fan Q, Fan P, Li Z, Fan J (2023) A Power Allocation Scheme for MIMO-NOMA and D2D Vehicular Edge Computing Based on Decentralized DRL. Sensors.



cmi2021
Mi C, Huang Y, Fu C, Zhang Z, Postolache O (2021) Vision-Based Measurement: Actualities and Developing Trends in Automated Container Terminals. IEEE Instrumentation and Measurement Magazine. pp. 65-76, https://doi.org/10.1109/MIM.2021.9448257.


zzhang2023
Zhang Z, Ma S, Yang Z, Xiong Z, Kang J, Wu Y, Zhang K, Niyato D (2023) Robust Semi-supervised Federated Learning for Images Automatic Recognition in Internet of Drones. IEEE internet of things journal. pp. 5733-5745.


dye2020
Ye D, Yu R, Pan M, Han Z (2020) Federated learning in vehicular edge computing: A selective model aggregation approach. IEEE Access. pp. 23920-23935, https://doi.org/10.1109/ACCESS.2020.2968399.


canagnostopoulos2023
Anagnostopoulos C, Gkillas A, Piperigkos N, Lalos S (2023)  Federated Deep Feature Extraction-based SLAM for Autonomous Vehicles. 2023 24th International Conference on Digital Signal Processing (DSP). https://doi.org/10.1109/DSP58604.2023.10167897.


yzhao2023
Wu Q, Zhao Y, Fan Q, Fan P, Wang J, and Zhang C (2023) Mobility-Aware Cooperative Caching in Vehicular Edge Computing Based on Asynchronous Federated and Deep Reinforcement Learning, IEEE Journal of Selected Topics in Signal Processing, vol. 17, no. 1, pp. 66-81.

sshi2023
Wu Q, Shi S, Wan Z, Fan Q, Fan P, and Zhang C (2023) Towards V2I Age-aware Fairness Access: A DQN Based Intelligent Vehicular Node Training and Test Method, Chinese Journal of Electronics, vol. 32, no. 6, pp. 1230-1244.






wxb01
Wu Q, Wang X, Fan Q, Fan P, Zhang C, Li Z (2023) High Stable and Accurate Vehicle Selection Scheme based on Federated Edge Learning in Vehicular Networks. China Communications. pp. 1-17. 

xsy01
Wu Q, Xia S, Fan Q, Li Z (2019) Performance Analysis of IEEE 802.11p for Continuous Backoff Freezing in IoV. Electronics. 


ryan2023
Yan R, Qu L, Wei Q, Huang S, Shen L, Rubin D, Xing L,  Zhou Y (2023) Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imaging. IEEE Transactions on Medical Imaging. https://doi.org/10.1109/TMI.2022.3233574.

kwang2019
Wang K, Yu F, Wang L, Li J, Zhao N, Guan Q, Li B, and Wu Q (2019) Interference Alignment with Adaptive Power Allocation in Full-Duplex-Enabled Small Cell Networks, IEEE Transactions on Vehicular Technology, vol. 68, No. 3.

sxia2018
Wu Q, Xia S, Fan P, Fan Q, and Li Z (2018) Velocity-Adaptive V2I Fair Access Scheme Based on IEEE 802.11 DCF for Platooning Vehicles, Sensors, vol. 18, no. 12, Art no. 4198.

jfan2010
Fan J, Yin S, Wu Q, and Gao F (2010) Study on Refined Deployment of Wireless Mesh Sensor Network, in Proc. of IEEE International Conference on Wireless Communications, Networking and Mobile Computing (WICOM?10), Chengdu, China, pp. 370-375.



czhang2022dual temp
Zhang C, Zhang K, Pham T, Niu A, Qiao Z, Yoo C, Kweon I (2022) Dual temperature helps contrastive learning without many negative samples: towards understanding and simplifying MoCo. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).  https://doi.org/10.1109/CVPR52688.2022.01404.


swang2024
Wu Q, Wang S, Ge H, Fan P, Fan Q, and Letaief K. B. (2024) Delay-sensitive Task Offloading in Vehicular Fog Computing-Assisted Platoons, IEEE Transactions on Network and Service Management, vol. 21, no. 2, pp. 2012-2026. 

hliu2019
Wu Q, Liu H, Zhang C, Fan Q, Li Z, and Wang K (2019) Trajectory Protection Schemes Based on a Gravity Mobility Model in IoT, Electronics, vol. 8, no. 148.

snie2018
Wu Q, Nie S, Fan P, Liu H, Fan Q, and Li Z (2018) A Swarming Approach to Optimize the One-Hop Delay in Smart Driving Inter-Platoon Communications, Sensors, vol. 18, no. 10, Art. no. 3307.


jzheng2015
Wu Q, and Zheng J (2015) Performance Modeling and Analysis of the ADHOC MAC Protocol for VANETs, in Proc. of IEEE International Conference on Communication (ICC?15), London, UK, pp. 3646-3652.

jfan2010
Fan J, Wu Q, and Hao J (2010) Optimal Deployment of Wireless Mesh Sensor Networks based on Delaunay Triangulations, in Proc. of IEEE International Conference on Information, Networking and Automation (ICINA?10), Kunming, China, pp. 1-5. 

xwang2023
Wu Q, Wang X, Fan Q, Fan P, Zhang C, and Li Z (2023) High Stable and Accurate Vehicle Selection Scheme based on Federated Edge Learning in Vehicular Networks, China Communications, vol. 20, no. 3, pp. 1-17.






cdoersch2016
Doersch C, Gupta A, Efros A (2016) Unsupervised visual representation learning by context prediction. 2015 IEEE International Conference on Computer Vision (ICCV). https://doi.org/10.1109/ICCV.2015.167.

zwu
Wu Z, Xiong Y, Yu S, Lin D. Unsupervised Feature Learning via Non-Parametric Instance Discrimination. https://arxiv.org/pdf/1805.01978.pdf.


tchen2020SimCLR
Chen T, Kornblith S, Norouzi M, Hinton G (2020) A Simple Framework for Contrastive Learning of Visual Representations. https://arxiv.org/abs/2002.05709. 


skong2023
Kong S (2023) Self-supervised Image Classification Using Convolutional Neural Network. 2023 IEEE 3rd International Conference on Power, Electronics and Computer Applications (ICPECA). https://doi.org/10.1109/ICPECA56706.2023.10075949.


rhjelm
Hjelm R, Fedorov A, Laavoie-Marchildon S, Grewal K, Bachman P, Trischler A, Bengio Y. Learning deep representations by mutual information estimation and maximization. https://arxiv.org/pdf/1808.06670.pdf.


khe2020
He K, Fan H, Wu Y, Xie S, Girshick R (2020) Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR42600.2020.00975.


xchen2020MoCoV2
Chen X, Fan H, Girshick R, He K (2020) Improved Baselines with Momentum Contrastive Learning. https://arxiv.org/pdf/2003.04297.pdf.

jzhao01
Zhao J, Xiong X, Zhang Q, Wang D (2023) Extended Multi-Component Gated Recurrent Graph Convolutional Network for Traffic Flow Prediction. IEEE Transactions on Intelligent Transportation Systems. https://doi.org/10.1109/TITS.2023.3322745.




tcai2022
Cai T, Gan H, Peng B, Huang Q, Zou Z (2022) Real-Time Classification of Disaster Images from Social Media with a Self-Supervised Learning Framework. IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium. https://doi.org/10.1109/IGARSS46834.2022.9883129.


jzhao2021Hotfed
Zhao J, Li R, Wang H, Xu Z (2021) HotFed: Hot Start through Self-Supervised Learning in Federated Learning. 2021 IEEE 23rd Int Conf on High Performance Computing and Communications; 7th Int Conf on Data Science and Systems; 19th Int Conf on Smart City; 7th Int Conf on Dependability in Sensor, Cloud and Big Data Systems and Application. https://doi.org/10.1109/HPCC-DSS-SMARTCITY-DEPENDSYS53884.2021.00046.

jzheng2016
Wu Q, and Zheng J (2016) Performance Modeling and Analysis of the ADHOC MAC Protocol for Vehicular Networks, Wireless Networks, vol. 22, no. 3, pp. 799-812. 

jzheng2014
Wu Q, and Zheng J (2014) Performance Modeling of the IEEE 802.11p EDCA Mechanism for VANET, in Proc. of IEEE Global Communications Conference (Globecom?14), Austin, USA, pp.57-63.

dlong2023
Long D, Wu Q, Fan Q, Fan P, Li Z, and Fan J (2023) A Power Allocation Scheme for MIMO-NOMA and D2D Vehicular Edge Computing Based on Decentralized DRL, Sensors, vol. 23, no. 7, Art. no. 3449.


jzhao00
Zhao J, Li Q, Ma X, Yu FR. Computation Offloading for Edge Intelligence in Two-Tier Heterogeneous Networks. IEEE Transactions on Network Science and Engineering. https://doi.org/10.1109/TNSE.2023.3332949.



jzheng20141
Wu Q, and Zheng J (2014) Performance Modeling of IEEE 802.11 DCF Based Fair Channel Access for Vehicular-to-Roadside Communication in a Non-Saturated State, in Proc. of IEEE International Conference on Communication (ICC?14), Syndey, Australia, pp. 2575-2580.

sxia2019
Wu Q, Xia S, Fan Q, and Li Z (2019) Performance Analysis of IEEE 802.11p for Continuous Backoff Freezing in IoV, Electronics, vol. 8, no. 12, Art. no. 1404. 

wwang2024
Wu Q, Wang W, Fan P, Fan Q, Wang J, and Letaief K. B. (2024) URLLC-Awared Resource Allocation for Heterogeneous Vehicular Edge Computing, IEEE Transactions on Vehicular Technology, doi: 10.1109/TVT.2024.3370196.




jzhao02
Zhao J, Quan H, Xia M, Wang D (2023) Adaptive Resource Allocation for Mobile Edge Computing in Internet of Vehicles: A Deep Reinforcement Learning Approach.IEEE Transactions on Vehicular Technology. https://doi.org/10.1109/TVT.2023.3335663.


qwu2024
Wu Q, Wang W, Fan P, Fan Q, Wang J, Letaief K (2024) URLLC-Awared Resource Allocation for Heterogeneous Vehicular Edge Computing. IEEE Trans. Veh. Technol. https://doi.org/10.1109/TVT.2024.3370196.


mfeng2022
Feng M, Kao C, Tang Q, Sun M, Rozgic V, Matsoukas S, Wang C (2022) Federated self-supervised learning for acoustic event classification. ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP43922.2022.9747472.

 wwang20241
Wu Q, Wang W, Fan P, Fan Q, Zhu H, and Letaief K. B. (2024) Cooperative Edge Caching Based on Elastic Federated and Multi-Agent Deep Reinforcement Learning in Next-Generation Networks, IEEE Transactions on Network and Service Management, doi: 10.1109/TNSM.2024.3403842.

ssong2024
Song S, Zhang Z, Wu Q, Fan P, and Fan Q (2024) Joint Optimization of Age of Information and Energy Consumption in NR-V2X System Based on Deep Reinforcement Learning, Sensors, vol. 24, no. 13, Art. no. 3448.

swan2021
Wan S, Lu J, Fan P, Shao Y, Peng C, and Letaief K. B. (2021) Convergence Analysis and System Design for Federated Learning Over Wireless Networks, IEEE Journal on Selected Areas in Communications, vol. 39, issue. 12. pp. 3622-3639, doi: 10.1109/JSAC.2021.3118351.







qwu2023

Wu Q, Zhao Y, Fan Q, Fan P, Wang J, Zhang C (2023) Mobility-Aware Cooperative Caching in Vehicular Edge Computing Based on Asynchronous Federated and Deep Reinforcement Learning. IEEE journal of selected topics in signal processing. pp. 66-81.






zyu2021
Yu Z, Hu J, Min G, Zhao Z, Miao W, Hossain MS (2021) Mobility-Aware Proactive Edge Caching for Connected Values using Federated Learning. IEEE Trans. Intell. Transp. Syst. pp. 5341-5351. https://doi.org/10.1109/TITS.2020.3017474.


khsieh2020
Hsieh K, Phanishayee A, Mutlu O, Gibbons P (2020) The Non-IID Data Quagmire of Decentralized Machine Learning. https://arxiv.org/abs/1910.00189.


yzhao2022
Zhao Y, Li M, Lai L, Suda N, Civin D, Chandra V (2022) Federated Learning with Non-IID Data. https://arxiv.org/abs/1806.00582.


sshirmohammadi
Shirmohammadi S, Ferrero A. Camera as the instrument: the rising trend of vision based measurement. IEEE Instrumentation & Measurement Magazine. https://doi.org/10.1109/MIM.2014.6825388.


jacorts-Osorio2018
Corts-Osorio JA, Gmez-Mendoza JB, Riao-Rojas JC (2018) Velocity estimation from a single linear motion blurred image using discrete cosine transform. IEEE Transactions on Instrumentation and Measurement. pp. 4038-4050.


khou2020
Hou K, Lv X, Zhang W (2020) An adaptive fusion panoramic image mosaic algorithm based on circular LBP feature and HSV color system. 2020 IEEE International Conference on Information Technology,Big Data and Artificial Intelligence (ICIBA). https://doi.org/10.1109/ICIBA50161.2020.9277348.

kxiong2014
Xiong K, Fan P, Xu Z, Yang H, and Letaief K. B. (2014) Optimal Cooperative Beamforming Design for MIMO Decode-and-Forward Relay Channels, IEEE Transactions on Signal Processing, vol. 62, issue. 6, pp. 1476 ? 1489, doi: 10.1109/TSP.2014.2298380. 

xchen2017
Chen X, Lu J, Fan P, and Letaief K. B. (2017) Massive MIMO Beamforming With Transmit Diversity for High Mobility Wireless Communications, IEEE Access, vol. 5, pp. 23032 ? 23045, doi: 10.1109/ACCESS.2017.2766157. 

jliu2020
Liu J, Xiong K, Ng D, Fan P, Zhong Z, and Letaief K. B. (2020) Max-Min Energy Balance in Wireless-Powered Hierarchical Fog-Cloud Computing Networks, IEEE Transactions on Wireless Communications, vol. 19, issue.11, pp. 7064 ? 7080, doi: 10.1109/TWC.2020.3007805. 

pfan2002
Fan P, Feng C, Wang Y, and Ge N (2002) Investigation of the time-offset-based QoS support with optical burst switching in WDM networks, 2002 IEEE International Conference on Communications. Conference Proceedings. ICC 2002, doi: 10.1109/ICC.2002.997330.

rjiang2019
Jiang R, Xiong K, Fan P, Zhang Y, and Zhong Z (2019) Power Minimization in SWIPT Networks With Coexisting Power-Splitting and Time-Switching Users Under Nonlinear EH Model, IEEE Internet of Things Journal, vol. 6, issue. 5, pp. 8853-8869, doi: 10.1109/JIOT.2019.2923977.

yguo2021
Guo Y, Xiong K, Lu Y, Wang D, Fan P, and Letaief, K. B. (2021) Achievable Information Rate in Hybrid VLC-RF Networks With Lighting Energy Harvesting, IEEE Transactions on Communications, vol. 69, issue. 10, pp. 6852-6864, doi: 10.1109/TCOMM.2021.3098030.





CIFAR-10
Krizhevsky A (2009) Learning Multiple Layers of Features from Tiny Images. https://www.cs.toronto.edu/ kriz/learning-features-2009-TR.pdf


hzheng2021
Zheng H, Xiong K, Fan P, Zhong Z, and Letaief K. B. (2021) Age of Information-Based Wireless Powered Communication Networks With Selfish Charging Nodes, IEEE Journal on Selected Areas in Communications, vol. 39, issue. 5, pp. 1393 ? 1411, doi: 10.1109/JSAC.2021.3065038.



sweiFedCo
Wei S, Cao G, Dai C, Dai S, Guo B (2023) FedCo: self-supervised Learning in Federated Learning with Momentum Contrast. 2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys). https://doi.org/10.1109/HPCC-DSS-SmartCity-DependSys57074.2022.00192.

hmcmahan2017
Mcmahan H, Moore E, Ramage D, Hampson S, Arcas B (2017)  Communication-Efficient Learning of Deep Networks from Decentralized Data. Proceedings of the 20 th International Conference on Artificial Intelligence and Statistics (AISTATS). https://arxiv.org/abs/1602.05629.





 Paper Title: Zero Shot Health Trajectory Prediction Using Transformer.


Data-driven Modeling for Grid Edge IBRs: A Digital Twin Perspective of User-Defined Models 





    Kaveri Mahapatra*, Bhaskar Mitra, and Soumya Kundu
Pacific Northwest National Laboratory, United States 



{kaveri.mahapatra, bhaskar.mitra, soumya.kundu}@pnnl.gov































    
=======================================================================================================================================================================================================



§ ABSTRACT 
Integrating modern machine learning and clinical decision-making has great promise for mitigating healthcare's increasing cost and complexity. We introduce the Enhanced Transformer for Health Outcome Simulation (ETHOS), a novel application of the transformer deep-learning architecture for analyzing high-dimensional, heterogeneous, and episodic health data. ETHOS is trained using  Patient Health Timelines (PHTs)—detailed, tokenized records of health events—to predict future health trajectories, leveraging a zero-shot learning approach. ETHOS represents a significant advancement in foundation model development for healthcare analytics, eliminating the need for labeled data and model fine-tuning. Its ability to simulate various treatment pathways and consider patient-specific factors positions ETHOS as a tool for care optimization and addressing biases in healthcare delivery. Future developments will expand ETHOS’ capabilities to incorporate a wider range of data types and data sources. Our work demonstrates a pathway toward accelerated AI development and deployment in healthcare.








§ INTRODUCTION

Healthcare in the U.S. is the world’s most expensive, and the quality and safety of care do not compare well to other developed countries <cit.>. While electronic healthcare records are now ubiquitous in the U.S., and decision-support technologies are widely implemented, most are rule-based, and their effectiveness so far has been limited <cit.>. Artificial intelligence has emerged as a technique with great potential for improving care, but most organizations are not using it to any major degree. Two major limiting factors have been (1)  the lack of large, labeled datasets, which are expensive and time-consuming to develop; and (2) limited system capacity to deliver recommendations to the appropriate clinician at the optimal time.  In this manuscript, we describe a novel method called the Enhanced Transformer for Health Outcome Simulation (ETHOS), which we believe can help address many of the limitations that have prevented widespread AI adoption.

ETHOS is a novel application of the transformer deep-learning architecture, originally conceptualized for natural language processing <cit.>. This architecture, a cornerstone in large language model (LLM) development, is repurposed in ETHOS to analyze health-related data, moving beyond the textual focus of traditional LLMs. ETHOS is designed to process Patient Health Timelines (PHTs)—detailed tokenized chronological records of health-related events—to predict future health timelines. In PHTs, a token serves as the fundamental unit of information, encapsulating diverse data types such as patient admissions, administered medications, or time intervals. We elaborate on this pivotal aspect of our methodology in the Methods section. Our model takes the patient's health history, as represented by PHT, and subsequently forecasts future PHT (fPHT) on a token-by-token basis (refer to Figure <ref>).



ETHOS’s generative capabilities are gained in unsupervised learning. Once trained, ETHOS can forecast future health events without requiring task-specific training. This is done through a zero-shot learning approach, making ETHOS a versatile foundation model for numerous healthcare applications. With appropriate modifications, ETHOS can be adapted to a broad range of data types, including but not limited to medical images, clinical and discharge notes, monitoring data, data from wearables, or omics data.

In this research, we leverage the recently released MIMIC-IV v.2.2 dataset <cit.>, a rich open-source repository accompanied by our code, allowing others to replicate our findings. MIMIC-IV is expansive, chronicling more than 400,000 hospitalizations in more than 200,000 patients. Although relatively large, we anticipate that the performance of our system will further improve as we expand the dataset with additional patient histories and data types. 

Importantly, we utilize the MIMIC-IV dataset in its original noisy form without any data modifications, cleaning, or targeted imputation for missing entries. The information is retained in the face of large data inconsistencies, such as discharge dates noted before admission dates. We operated under the assumption that, within large enough datasets and appropriate tokenization and training methods, ETHOS would be robust enough to handle the noisy input and automatically manage the noise/anomalies in the input data. The resilience of ETHOS to data inaccuracies and missing information has important implications for the efficiency of downstream model development. Healthcare data inevitably contains errors, some of which may not be immediately apparent or easily rectifiable. Attempts to clean large datasets can be impractical and may inadvertently introduce biases and errors. Our approach highlights the vital need for algorithms adept at managing these challenges, a prerequisite for the large-scale development of reliable and robust healthcare AI applications. 

Our research showcases the zero-shot learning capabilities of ETHOS in predicting inpatient and ICU mortality, estimating ICU length of stay (LOS), and determining readmission probabilities. Additionally, we illustrate the model's versatility by performing a regression task to estimate the first-day Sequential Organ Failure Assessment (SOFA) score <cit.> at the time of ICU admission using information before admission (see example in Figure <ref>d). The SOFA score is a critical tool for monitoring a patient's condition in the ICU, evaluating organ function or failure across six systems—respiratory, cardiovascular, hepatic, coagulation, renal, and neurological—with each system scored from 0 to 4, culminating in a total possible minimum score of 0 and maximum score of 24. Furthermore, we predict Diagnostic-Related Group (DRG) classifications, encompassing over 771 categories, at the time of hospital discharge. The DRG system categorizes hospital cases into standardized case complexity-based Medicare and Medicaid payment groups, encouraging efficient patient care without compromising quality. The diversity of tasks ETHOS can perform, from mortality predictions and LOS estimation to SOFA scoring and DRG classification, highlights its broad applicability and zero-shot learning efficiency. 

ETHOS is a foundation model <cit.>, introducing a novel approach in the landscape of data analysis within the healthcare domain. The other foundational models developed recently have fallen into two broad categories. The first of these categories encompasses Clinical Language Models (CLaMs), a specialized subset of large language models (LLMs) <cit.> tailored for processing clinical and biomedical text data. These models are typically trained on extensive datasets containing clinical notes, biomedical literature, and other healthcare-related text sources. CLaMs are proficient in various clinical tasks such as extracting drug names, summarizing medical dialogues, predicting clinical outcomes, and responding to patient queries <cit.>. The second category comprises Foundation Models for Electronic Medical Records (FEMRs), representing another class of clinical foundation models tailored specifically for EMR data analysis. FEMRs undergo training on the extensive medical histories of patients, covering both structured data (such as demographics and lab results) and unstructured data (including progress notes and radiology reports). Unlike CLaMs, FEMRs are not designed to generate clinical text. Instead, they produce machine-understandable representations of patient data, facilitating tasks such as patient phenotyping and outcome prediction <cit.>. Similarly, data that chronicles human lives, akin to EMR, can also be modeled effectively in this manner. 

The primary distinction between ETHOS and previously published methods lies in our approach, which eliminates the need for fine-tuning or labeled data to produce accurate inferences or predictions. We demonstrate inference across a wide array of tasks without task-specific training. Moreover, the ability of ETHOS to forecast future PHTs opens the door to a wide array of bespoke and innovative applications, facilitating its use in unique scenarios in healthcare, some of them explored in the discussion section. Unlike many studies, which often apply specific criteria for selecting data for training and testing, our methodology imposes no such limitations. This feature is crucial for considering the scalability of the ETHOS approach to data sets comprising millions or even hundreds of millions of patients.



§ RESULTS




 §.§ Tokenization of MIMIC data and training of ETHOS


Figure <ref>a summarizes some statistics of the tokenization process, including the number of tokens generated and other details. Figure <ref>b presents visualizations of the 768-dimensional embeddings reduced to a 2D plane using Principal Component Analysis (PCA) for quantile tokens, which encode all quantitative values in the data. The tokens are arranged from Q1 (the lowest quantile) to Q10 (the highest quantile). This suggests that the transformer model has learned a sequential relationship between the tokens that mirrors their natural order, ascertaining this order from the data during the training process. The proximity between points could reflect the model's differentiation among the quantiles. We observe that the gaps between Q4, Q5, and Q6 are narrower than those between Q9 and Q10. This may suggest that the model deems the variance between population-average values to be less substantial than that of extremely high values. For example, the difference in clinical significance between a blood pressure reading of 110 mmHg (Q5) and one of 130 mmHg (Q6) is less pronounced than the difference between 140 mmHg (Q9) and 160 mmHg (Q10), which could account for the greater disparity in the embedding vectors of high quantiles.

The embeddings for time-interval tokens, representing the approximate durations between different tokenized events in PHT, are illustrated in Figure <ref>b. These embeddings display a pattern analogous to that observed for Q tokens, where ETHOS systematically arranged them according to the actual time values they represent. Remarkably, the model perceives the two shortest (5m-15m, 15m-1h), and two longest (3m-6m, 6m) intervals as relatively similar. 





 §.§ ETHOS inferences


In our study, we conducted zero-shot inferences for a diverse array of classification tasks, including readmission to the ICU, inpatient mortality, ICU mortality, combined inpatient and ICU mortality in patients with sepsis, readmission to the ICU for patients with intracerebral hemorrhage, assignment of DRG class assessed at inpatient discharge. We also demonstrate regression of first-day SOFA score at the time of ICU admission and regression of the length of stay in ICU in days assessed upon admission. The results corresponding to these tasks are summarized in Figure <ref>. We also provide precision-recall curves of the corresponding results in Figure <ref>.



To situate our results within the broader scientific discourse, we conducted a literature review, concentrating on contemporary studies that utilized the MIMIC-III and MIMIC-IV datasets for similar tasks and reported their outcomes. A notable observation from our review is that many of these studies either lacked publicly available source code or implemented specific exclusion criteria for their data selection. Such practices pose challenges for directly comparing their results with our approach. Nonetheless, we posit that the numerical outcomes reported in these works provide a valuable benchmark for assessing the performance of ETHOS. Furthermore, we conducted a direct comparative analysis of ETHOS against specialized algorithms developed in-house, with these findings detailed in the supplementary materials.

We conducted an analysis focusing on risk estimation for inpatient and ICU mortality, calculated at the respective points of patient admission to the hospital and ICU. The test set comprised 43,309 hospital admissions with a 2.0% mortality and 7,483 ICU admissions with a 7.0% mortality. The ETHOS model demonstrated robust performance, achieving an AUC of 0.921 (95% CI: 0.908-0.931) for hospital mortality and 0.927 (95% CI: 0.914-0.938) for ICU mortality. Comparatively, in the ICU mortality risk prediction domain, the highest performance identified in our literature review was an AUC of 0.918 (95% CI: 0.915-0.922) reported by <cit.> using the XGBoost model. On the lower end, <cit.> reported an AUC of 0.642 ± 0.101. Within a specific subgroup of the test set of 3,324 patients with sepsis with 10.8% mortality prevalence, ETHOS's prediction of ICU mortality exhibited an AUC of 0.889 (95% CI: 0.870-0.906), which is a better performance than obtained in a study by <cit.>, which estimated ICU mortality in adult sepsis patients using SOFA and additional features, achieving an AUC of 0.762 ± 0.006. We also estimated performance for a task of ICU mortality estimation for patients remaining in ICU for at least 24 hours in which we obtained an AUC of 0.928 (95% CI: 0.916-0.939). 

Furthermore, ETHOS estimated the length of stay (LoS) in the ICU with a mean absolute error (MAE) of 2.262 days (95% CI: 2.161-2.355 days). These results paralleled those of18, who reported an MAE of 2.42 ± 0.10 days. ICU LoS prediction and mortality risk, underscoring the competitive zero-shot performance of ETHOS across multiple key healthcare metrics.

For the ICU readmission task, ETHOS’ AUC of 0.807 (95% CI: 0.786-0.827) is slightly smaller than the AUC of 0.82 obtained using knowledge graph embeddings <cit.> and is higher than the AUC of 0.791 (95% CI, 0.782–0.800) using LSTMs based on MIMIC-III data <cit.>. Additionally, we applied our method to a task characterized by a relatively low prevalence, specifically focusing on only 174 cases of patients with hemorrhage admitted to the ICU present within our test set. The prediction of readmission by ETHOS yielded an AUC of 0.667 (95% CI: 0.402-0.839), comparable to the AUC of 0.736 (95% CI: 0.668-0.801) achieved by previous studies <cit.> using LightGBM. For hospital readmission, ETHOS achieved an AUC of 0.749 (95% CI: 0.743-0.755), lower than the AUC of  0.791 [0.766-0.816] obtained by <cit.>. It's important to recognize that although MIMIC offers a wealth of data on acute care, it might not encompass all the subtleties necessary for readmission research, including comprehensive post-discharge outcomes or data on readmissions to various hospitals. Consequently, the accuracy of results for tasks related to readmission may be limited, regardless of the method employed.



We explored the task of predicting the first-day SOFA score at the time of admission (Figure <ref>). Given that the SOFA score is a critical indicator of survival, particularly in sepsis <cit.>, this prediction can serve as a valuable indirect prognostic marker of ICU patient health status. We achieved a SOFA score estimation with an MAE of 1.502 (95% CI: 1.475-1.534). To our knowledge, no prior literature predicts first-day SOFA at the time of admission.

For the DRG assignment, we observed a top-1 (out of 771 classes) accuracy rate of 84.8% (95% CI: 84.4%-85.2%) in 28,932 hospitalizations using our methodology, a significant improvement over the 52% reported by <cit.>, who explored DRG estimation using LLMs from discharge notes. This marked enhancement in performance can be attributed to the comprehensive nature of ETHOS, which incorporates a wide array of clinical events leading up to discharge within the PHT. In contrast, the approach taken by <cit.> relies solely on discharge notes, which may not encompass the breadth of information captured by PHT, thus potentially explaining the disparity in accuracy rates. 

We want to reiterate an important point: all comparisons presented in this section are made between ETHOS, trained indiscriminately on the entire test population and task-specific algorithms developed using much smaller MIMIC data subsets obtained after data curation. In addition to the results in this section, in supplementary materials, we benchmark the performance of ETHOS against XGBoost <cit.>, recurrent neural networks, and logistic regression. 



§ DISCUSSION


This work introduces an innovative approach to developing a Foundation Model for medical data derived from EMRs, designed to execute zero-shot inferences across a diverse range of tasks. Our model generates interpretable, causally forecasted future patient health timelines. By "causal," we mean that predictions are made solely based on information that occurred in the past. We applied and evaluated this model using the MIMIC-IV EMR datasets, comparing its performance with the results of methods published in the literature for the same tasks. Our objective, however, was not merely to surpass the performance of these specialized SOTA implementations. Instead, we aimed to demonstrate that ETHOS, a single foundation model trained just once with zero-shot derived inference, can achieve performance levels comparable to that of multiple models optimized for various tasks. This underscores the potential of ETHOS to streamline the application of AI in healthcare by leveraging a single unified model development architecture and set of methods for multiple prediction tasks, thereby greatly enhancing medical data model development efficiency and scalability.

The application of patient timelines for generating insights has been established in existing research <cit.>, as has the implementation of foundation models <cit.>. Our methodology sets itself apart by integrating a zero-shot capability, obviating the need for additional training beyond the initial model. The design of ETHOS accommodates various approaches to inference, including few-shot predictions, although this necessitates fine-tuning for specific downstream tasks. Notably, the zero-shot prediction methodology introduces capabilities absent in few-shot prediction. To forecast future outcomes, ETHOS generates multiple health timelines representing possible future scenarios. This functionality exploits the model's capacity to explore and evaluate potential future events, thereby potentially estimating uncertainties. Future work will undoubtedly concentrate on refining this aspect of ETHOS. Moreover, ETHOS is specifically engineered to produce causal predictions in the form of future timelines, ensuring they are inherently comprehensible to human users. This is achieved through a novel tokenization process for medical data, a distinctive feature of our work.

The generation of multiple scenarios using the zero-shot approach places significant demands on time and computational resources. We estimated the inference time based on the average duration required to generate 1,000 tokens. On a single Nvidia A100 GPU, this process took approximately 15 seconds. Given that computations are executed in batches and are thus highly parallelizable, we anticipate that in a potential production environment, response times could vary between 1 to 30 seconds. This variation is contingent upon the complexity of the downstream task at hand.

Another highly distinctive capability of ETHOS is the potential to generate individualized care-integrated PHT-based projected healthcare expenditures. This capability is exemplified through the prediction of Diagnosis-Related Group (DRG) codes but is not limited to this application. Specifically, ETHOS can model future PHTs at critical decision-making junctures in patient care. For instance, ETHOS can model outcomes for administering either drug A or B, considering the patient's unique conditions (such as sex, age, race, gender, income, etc.) to determine which path might yield better clinical and cost outcomes. In this regard, ETHOS has the potential to revolutionize medical decision-analytic modeling science by incorporating a level of personalization previously unavailable in conventional decision-analytic models. This has the potential to enhance clinical decision-making and incorporate individualized real-time quantitatively robust value-based care policies into clinical care.  This is a potentially transformative change, radically  unlike current evidence-based medicine practices, which  rely on high-quality data obtained from and averaged across patient populations <cit.>.

In designing ETHOS, we have considered explainability, fairness, and transparency. These are vital aspects of our ongoing research. In future work, we plan to implement and test advanced visualization attention layers of the transformer <cit.> to gain insights into the model's reasoning process. Additionally, a dedicated interface for decision-making is envisaged further to enhance the usability of ETHOS in clinical settings.

Envisioning the development of a robust AI method that offers fully personalized advice on a wide range of medical questions necessitates learning from an extensive dataset of patients. Such a model must assimilate as much data as possible and be adaptable to a vast array of medical tasks. ETHOS represents a significant stride in this direction. Built on a transformer architecture, it is inherently scalable and, as a zero-shot learner, is versatile enough to address numerous key medical prediction tasks without task-specific training. Currently, ETHOS does not incorporate various types of critical information, including clinical and discharge notes, medical imaging and pathology images, genetic data, socioeconomic factors, lifestyle considerations, and monitoring signals. Nonetheless, the conceptual framework for incorporating these diverse data types is relatively straightforward. This can be done by leveraging the encoder and cross-attention mechanisms inherent in the transformer architecture; we anticipate the potential for integrating a nearly limitless amount of information during training. This expansion of ETHOS's capabilities forms the cornerstone of our future work, promising to enhance its applicability and efficacy in personalized medical advice and diagnostics.

We aim to modify further and train ETHOS to apply it across diverse data sources. This capability is currently hindered by variations in data collection methodologies, disparities in data quality, and the presence or absence of certain data types across different sources. Additionally, non-overlapping populations present significant challenges, rendering ETHOS not yet generalizable. To mitigate some of these compatibility issues, we propose the development of a universal tokenization format <cit.>. While this approach may resolve certain discrepancies, it does not address all underlying compatibility concerns. The ultimate solution, we believe, lies in a system capable of transforming tokenized data from one healthcare system to another, akin to text translation between languages. Specifically, for ETHOS, this would mean converting the patient journey, as encapsulated by the PHT, from one system's format to another. This conversion would not only facilitate a consistent and unified representation of patient histories across different systems but also offer insights into the operational nuances of these systems. Pursuing such a translation strategy represents a vital direction for our future research endeavors, alongside evaluating the methodologies introduced in this paper through analysis of prospectively collected data.

ETHOS and LLMs such as GPT-4o, Claude 3 Opus, and Gemini 1.5 Ultra, although built upon similar AI principles, serve different purposes and exhibit distinct capabilities. ETHOS is specifically designed to predict fPHTs through explicit modeling of quantitative values and temporal sequences. This approach allows ETHOS to leverage structured patient data to generate predictions. In contrast, LLMs are general-purpose models optimized for tasks involving knowledge integration, reasoning, and interactive conversation. They do not explicitly model quantitative values and time sequences, which are important for accurate clinical decision support. Studies, such as those by <cit.> and <cit.>, highlight the limitations of LLMs in handling temporal information and decision support tasks, emphasizing the potential need for specialized models like ETHOS. There is a potential of ETHOS to be used in conjunction with LLMs through retrieval-augmented generation (RAG) mechanisms, offering a promising direction for future AI applications in healthcare. In supplementary material,  we present a comparison in predictive performance of ETHOS and LLM (GPT-4o). Furthermore, while LLMs excel in processing vast amounts of unstructured text, their computational performance in generating detailed and contextually accurate patient predictions remains suboptimal compared to ETHOS because of efficient representation of information in ETHOS tokenized PHTs.

This work has limitations.  We utilized the MIMIC dataset, which may be cleaner than many routine clinical datasets. Performance and usability should be tested prospectively in diverse datasets and in real-time. The transformer model in the current version of ETHOS is relatively simple and uses only 2048 PHT tokens for predictions. When token density per time is large, this may not contain sufficient information for optimal performance. Mitigation of the limitation is expected with additional computational infrastructure.

In conclusion, ETHOS presents a promising approach to deriving insights from massive clinical datasets without labor-intensive labeling or distinct model creation for each prediction task. This approach has the potential to significantly lower the costs and complexities associated with AI model development, thereby accelerating the development and implementation of healthcare AI.



§ METHODS




 §.§ Data


In this study, the Medical Information Mart for Intensive Care (MIMIC-IV) database served as a data source, providing a rich and comprehensive collection of de-identified health-related information4. Managed collaboratively by the Massachusetts Institute of Technology (MIT), Beth Israel Deaconess Medical Center (BIDMC), and Philips Healthcare, MIMIC-IV encompasses detailed records for more than 200,000 patients who were admitted to hospital and critical care units at BIDMC in Boston, Massachusetts, between 2008 and 2019. The following tables from the MIMIC-IV were used: 1) Patients, which contains static information about the patients, such as gender, date of birth, and date of death; 2) Admissions, which holds information about patient admissions to the hospital, including admission and discharge times, as well as information related to the hospital stay; 3) Icustays, which is specifically related to intensive care unit (ICU) stays, including the timings and type of ICU; 4) Labevents, which contains laboratory test results for patients. We used the 200 most frequent tests covering 95% of tests completed; 5) Prescriptions, which holds information on medications prescribed to patients during their stay, with each drug converted to ATC code [<https://www.whocc.no>]. We converted GSN codes in MIMIC-IV to ATC codes using conversion tables <cit.>; 6) Procedures which contains information about procedures performed on patients, coded using ICD10-PCS codes; 7) Diagnoses which contains diagnostic information, typically coded using ICD10-CM codes. We converted ICD9 to ICD10-CM if needed using conversion table [<https://www.cms.gov/medicare/coding-billing/icd-10-codes>]; 8) Emar, which holds information related to the documentation and administration of medications to patients; 9) Omr with information about measurements taken from a patient, such as blood pressure or BMI; 10) Services with information about the clinical service under which a patient is managed during their hospital stay; 11) drgcodes DRG codes which are a classification system used in the healthcare industry to categorize hospital cases into groups that are expected to have similar hospital resource use;  12) SOFA, taken from the derived tables in MIMIC. The remaining tables were not used in the current ETHOS implementation as they will require additional processing. For example, clinical notes require natural language processing to be converted to meaningful tokenized information.





 §.§ Patient health timelines (PHTs), tokenization


The core concept behind ETHOS is the Patient Health Timeline (PHT), as depicted in Figure <ref>. The fundamental component of the PHT is the token, which represents a distinct unit of information occurring within the patient's health timeline. To construct the PHT, we gathered all pertinent data from tables 1 to 12 of the MIMIC-IV database, as detailed in the Data section. We arranged this data chronologically based on timestamps, as shown in Figure <ref>, into a chronological sequence of health-related events for each patient. These events were timestamped with a floating-point number in 64-bit precision to denote the patient's age at the time of occurrence of the event. Subsequently, events from the MIMIC-IV tables were converted into tokens. Each event was represented by 1 to 7 tokens to encapsulate information about the event, as illustrated in Figure <ref>a. We crafted this encoding process to ensure each token conveys specific, meaningful information, with examples in Figure <ref>c-f. A comprehensive list of token encodings within the PHT is available in the supplementary material. The final step of tokenization involved the insertion of time-interval tokens to represent the intervals between events, depicted in Figure <ref>c. We employed 13 different time-interval tokens to represent the intervals. No interval token was inserted if the duration between tokens was less than 5 minutes. Typically, a single time-interval token was placed between other types of tokens unless the interval exceeded one year. In such cases, multiple 6-month tokens were used to approximate the actual interval. For example, an interval of 1.4 years was represented by three 6-month tokens, while four 6-month tokens represented 1.76 years. One interval-tokens were inserted the exact time of events was dropped from PHTs.

The patient's age and the commencement date of the PHT were represented using the same token set. We used 20 distinct tokens to denote age intervals such as 0-5 years, 5-10 years, and so forth. For instance, to encode information about a 46-year-old patient with PHT beginning in 1982, we inserted a "45_50 years" token at the 4th position in the PHT. To signify the year 1982, we used a "15_20 years" token at the 5th position of the PHT, considering 1970 as the baseline year. We emphasize that age and the commencement of the PHT are encoded in five-year intervals, given that health status typically does not undergo rapid changes with age, making finer granularity unnecessary. However, we plan to scrutinize these assumptions in subsequent research. The token denoting the commencement of the PHT delineates the temporal context of the medical data—identifying whether it corresponds to earlier medical practices (e.g., 1990s), contemporary practices, or periods in between. Using tokens with a precision of five years is done under the premise that technological and methodological progress within the medical field does not advance at a pace that justifies the necessity for time intervals more granular than five-year spans. Pertinent to the MIMIC dataset, the obfuscation of actual dates through uniform random adjustments for each patient—a measure implemented to safeguard privacy—compromises the utility of this temporal information for ETHOS, as it obscures the precise date of the start of PHT. However, the absence of precise reference dates is less critical, given that the entire dataset was collected over a relatively brief period, from 2008 to 2019 (<cit.>).

As mentioned previously, token locations within the timeline are contingent upon the temporal occurrence of events. Nonetheless, certain data elements are temporally invariant, or at least presented as such within the MIMIC-IV database. In our implementation, we designate six static tokens to encapsulate the information encoded in these static data elements. Although, in reality, some of these variables may change over time, they are represented as invariable constants in the MIMIC database. We encoded this information in the six static tokens exactly as recorded in the MIMIC dataset. These include gender, marital status, race, body mass index (BMI), birth date, and the start date of the timeline. While PHTs have the potential to extend to hundreds of thousands of tokens, our current methodology utilizes a maximum of 2048 subsequent tokens within the transformer model context, as elaborated in the "Methods: ETHOS Training" section. To accommodate invariant data, we substitute the initial six tokens of the 2048-token context with static information tokens, where the sixth token demarcates the temporal juncture of the seventh token, which is the first token of the actual timeline. Although the transformer architecture inherently facilitates the inclusion of static data via its encoder component and cross attention module3, we opted for a more streamlined approach as described, deferring the integration of an encoder implementation to future endeavors where more substantial time-invariant data like genetics is used.

Medical encounters yield a plethora of numerical data. We employ a quantile-based tokenization strategy to process continuous numerical values, such as blood pressure readings or cholesterol levels. Specifically, all numerical values are transformed into integers representing the quantile to which each value corresponds. Quantile ranges were determined using the training dataset, where histograms of all numerical values were generated and subsequently divided into quantiles. We chose to utilize ten quantiles, a decision aimed at striking a balance between the need for precise representation of numerical data and the clinical reality that significant changes in health indicators often manifest as relatively large variations, such as shifts of 10 or 20 percent (Figure <ref>). This rationale underpins our selection of ten quantiles for tokenization.

In our study, Diagnosis-Related Group (DRG) codes for each inpatient stay were utilized, despite the absence of assigned times when they were created in the MIMIC tables. Given that a DRG code is assigned after or during discharge, we positioned it after a trio of tokens representing discharge-related information: the discharge token, a quantile token indicating the length of the hospital stay, and a token specifying the discharge destination (e.g., home). Additionally, we incorporated data from MIMIC regarding the initial SOFA score for ICU patients, placing this token after the patient's admission-to-the-ICU token, along with a token denoting the ICU type. Given that the SOFA score in the dataset ranges from 0 to 23 (with the score of 24 never appearing), we uniformly map scores from 0-23 across 1-10 quantiles. Consequently, in quantile Q1, SOFA scores of 0, 1, and 2 (average of 1) are included, while quantile Q2 encompasses SOFA scores of 3 and 4 (average of 3.5), and this pattern continues accordingly.

ETHOS operates as a causal network. It relies solely on information available up to the time being considered in making predictions. Consequently, to ensure causality, actual values of DRG codes and SOFA scores are not employed during inference; instead, predictions of these values are used. This principle ensures that future-obtained information does not influence the prediction of yet-to-occur events. In essence, if tokens are integrated into the timeline based on their approximate occurrence time, their actual values must not be utilized for inference purposes, or they are placed in the timeline far in the future to ensure they are inserted after they occurred.

For the tokenization of drugs, whether administered or prescribed, we utilized the ATC classification system due to its hierarchical, tree-like structure (Figure <ref>). Each ATC code, comprising up to seven characters, was encoded using up to three sequential tokens: the first token for the initial three characters, the second for the subsequent character, and the third optional token, for the remaining suffix. Similarly, ICD-10-CM codes were encoded with three tokens: the first representing the first three characters of the code, the next two by the second token, and the final token capturing the code's remaining suffix. For ICD-10-PCS codes, each character in the seven-character code was represented by a distinct token. The rationale behind such tokenization is that the initial characters in those coding schemes denote specific classes of drugs and diseases or procedures, which are interpretable and have distinct meanings which we anticipated to be important for the network's self-attention mechanisms. Looking ahead, our approach, which assigns well-defined meanings to each token, will be crucial for refining attention mechanisms and enhancing the model's explainability. This method ensures that individual tokens contribute significantly to the interpretability of the network's outcomes. For more information on the tokenization process applied to MIMIC data in our analysis, as well as examples of Patient Health Timelines (PHTs), readers are directed to Table <ref> and Table <ref> where we present real PHTs used in this work with annotations. A summary of all tokenized components of the MIMIC dataset is in Table <ref>.



 §.§ ETHOS training


We employ a model inspired by the decoder architecture of the transformer <cit.>, drawing parallels between tokenized text in Natural Language Processing (NLP) and our approach to tokenizing PHTs. We based our model development on Andrej Kapathy’s implementation of GPT-2 [<https://github.com/karpathy/nanoGPT>]. The design choice slightly varies from the original transformer paper, because instead of using fixed sinusoidal positional encodings, it utilizes learneable position embeddings that are added to the token embeddings at the stage where tokens are converted to their corresponding embeddings. The ETHOS model's training begins by synthesizing a dataset from existing patient records. Each patient's PHT is ended with a "End of timeline" token, and then they are concatenated, creating a single long sequence of tokens for the training. Similarly to generative LLM, ETHOS is trained to predict a single token based on the context of preceding ones. Given the large data scale and model complexity, this phase is resource-intensive similar to methods for training used for NLP transformers used in LLMs (<cit.>). We estimated that the size of the network training task that we face with ETHOS is similar to GPT-2 <cit.>, and therefore we used the size of the transformer used in that network as a starting point (details on the hyperparameter search and choice can be seen in Figure <ref>). We made heuristic adjustments to the size of the network to optimize the value of the loss function. Further details on our training methodology of transformers are provided in <cit.> and for our implementation in supplementary material and full complete code published on GitHub [<https://github.com/ipolharvard/ethos-paper>].



 §.§ Evaluation of Clinical Outcomes and Tasks Using ETHOS


The experiments were chosen so the results can be compared to the work of others in terms of the estimation of inpatient mortality and readmission on MIMIC data. Patients in the MIMIC were randomly divided into training and testing groups, with splits of 90%/10% (Table <ref>).

The chance of inpatient mortality was assessed at the time of admission for all inpatient stays for patients in the test set unless the discharge day was unknown. This was performed by the generative process that began with the admission token and ended upon generating a discharge or death token, repeating this cycle 20 times. The 'N', representing the number of times a death token was generated first, was divided by 20 to estimate the chance of inpatient mortality. Similarly, the likelihood of ICU mortality was computed for the MIMIC dataset, with an additional experiment conducted where predictions were made starting 24 hours after ICU admission, rather than at the point of ICU admission. In the same simulation, the LOS in the ICU was estimated by aggregating the time-interval tokens generated in the simulated timeline until the discharge token appeared. Instances where the patient died in the ICU during the simulation were excluded from the LOS calculation. We opted for 20 repetitions, yielding 21 unique probability estimators, which were adequate for constructing robust Receiver Operating Characteristic (ROC) curves yielding excellent Gaussian fits (Figure <ref>). Nevertheless, alternative repetition counts may also be employed.

To calculate the probability of 30-day inpatient readmission, the generation of fPHTs commenced at the discharge token from inpatient stays and ceased upon the appearance of either a new admission or death token or when the cumulative time tokens generated exceeded 30 days. The simulation was repeated 20 times. The probability of 30-day readmission was then derived as M/20, where 'M' is the count of terminations occurring because of patient new admission tokens across the 20 repetitions.

In our approach, tasks are accomplished by simulating future patient health timelines. Yet, ETHOS offers additional methods for deriving insights, two of which we illustrate here. For instance, in the construction of PHTs following each ICU admission, a sequence is created starting with a token that identifies the type of ICU, followed by a SOFA score token, and then by a Q token that signifies the actual SOFA score on the first day. We predict the SOFA score using SOFA Q node probabilities as generated by ETHOS and the mean SOFA score per quantile as assigned during tokenization (Figure <ref>a).

The exact timing of the 1-day SOFA score assessment is not specified in the dataset, leading to a potential causality issue by inserting the SOFA score immediately after admission, as it relies on data acquired subsequently. During the model's training phase, ETHOS permits this apparent causality violation. However, such true values of 1-day SOFA scores, not available at the moment of ICU admission, are not used for simulating future timelines during inference to prevent causality violation during inference. Instead, these scores are predicted from prior information, as demonstrated in our study. This feature of ETHOS enables the inclusion of information with indeterminate timing.

Another distinctive inference capability facilitated by ETHOS is DRG class estimation. As illustrated in Figure <ref>c, the token denoting the DRG class is consistently positioned following the discharge token and a Q token specifying the length of hospital stay. With 771 unique tokens available for this purpose, we infer the actual class by generating a probability array in the final network layer of the transformer for the DRG token. This array is then utilized to predict the classification's top-1 and top-2 accuracy metrics.



 §.§ Statistical Analysis


The performance of classification algorithms of binary tasks was assessed using Receiver Operating Curve Analysis (ROC). The ROC curves were fitted to experimental points using Gaussian models with unequal variances for binary hypotheses (code provided). Values of Areas Under Curves (AUCs) and 95% confidence intervals (CI) were calculated using bootstrapping (code provided). For multiclass classification (DRG task), we used top-1 and top-2 accuracy. We used mean absolute error (MEA) for the regression tasks to indicate prediction fidelity with 95% confidence intervals estimated using bootstrapping. Python numpy and scikit-learn were used.



 §.§ Comparison of ETHOS to existing methods


Employing the data segmentation as detailed in Table <ref>, we evaluated traditional algorithms for predicting 30-day hospital readmission rates and juxtaposed these outcomes with those obtained via ETHOS. The features used in Figure <ref> were culled from data accrued during the patient's hospitalization, adhering to the feature derivation methodology outlined by <cit.>. Attempting to apply the algorithm devised by the authors to our dataset presented challenges, notably due to the Graph Neural Network (GNN) implementation by Tang et al., which necessitates the computation of a similarity score for each pair of admissions. Given the significantly larger volume of admissions in our dataset—approximately 400,000, in stark contrast to the 14,500 reported by Tang et al.—this task proved impractical on a compute node with 2TB of RAM, defying all efforts to achieve it within a reasonable timeframe. Consequently, we limited our application to the data preprocessing and feature extraction segments of Tang et al.’s methodology. The adapted and modified code from Tang et al.’s repository, which we cloned for feature extraction, is accessible on GitHub [<https://github.com/ipolharvard/readmit-stgnn>]. For models unsupportive of temporal sequence analysis, such as Logistic Regression and XGBoost, we modified the approach to handle time-varying features by consolidating them over time. This entailed distilling the minimum, first quartile, median, third quartile, and maximum values of dynamically changing features. Furthermore, we integrated the day of admission as a unique feature to retain an element of temporal dimension within the dataset. In Figure <ref> ETHOS was compared to one of the leading proprietary LLM models - GPT-4o in two temperature variants: 0.3 and 0.5. We constructed a comprehensive prompt that directs the model to analyze a timeline of 2048 tokens and calculate the probability of patient readmission for 2000 cases from the test set. This prompt is structured into four distinct parts: task instructions, a basic patient description corresponding to ETHOS's static information, PHT and a detailed description of subgroups of tokens and their identification. The complete codebase for this experiment including the prompt design is accessible on GitHub [<https://github.com/ipolharvard/ethos-paper/blob/master/notebooks/llm_readmission_task.ipynb>]. ETHOS significantly outperforms both variants of GPT-4o for the same subset of testing samples.



§ ADDITIONAL INFORMATION




  
Data Availability The MIMIC-IV dataset is publicly available at <https://physionet.org/content/mimiciv/2.2>.



  
Code Availability The code, ETHOS model weights used for all inferences, results of inferences, scripts to generate numerical results for all aspects of this study for the MIMIC-IV dataset are made publicly available at <https://github.com/ipolharvard/ethos-paper>. In our experiments, we used Python 3.10, and the following open-source libraries: torch=2.3.0, joblib=1.4.2, tqdm=4.66.4, colorlog=6.8.2, h5py=3.11.0, pandas=2.2.2, numpy=1.26.4, pyarrow=16.1.0, click=8.1.7.



  
Author Contribution AS and PR conceptualized the work. AS, PR, YJ, AES designed the study. PR, AS performed the coding and the experiments. YJ, AS conducted the literature search. DB, AES, QL, JW provided advisory support for the project. PR and AS prepared the initial draft of the manuscript, with all authors actively participating in the refinement and finalization of the manuscript through comprehensive review and contributions. AS supervised the project.



  
Competing Interests YJ is currently also affiliated with Verily life science, SSF, CA. The other authors declare no competing interests.

unsrtnat




§ SUPPLEMENTARY INFORMATION






















ll
    Example of a tokenized timeline with a hospital admission

    Patient Health Timeline     Notes 

    

    2c
      – Continued from previous page 

    Patient Health Timeline     Notes 

    

    2rContinued on next page 

    

    ED_ADMISSION_START     Admission to Emergency Department 

    _2h-6h     Time interval between 2h and 6h 

    LAB_pH_units     Lab test of pH and the unit is units 

    _Q4     Quantile token referring to the lab test of pH 

    LAB_Protein_mg/dL     

    _Q4     

    LAB_RBC_#/hpf     

    _Q1     

    LAB_Specific Gravity     

    _Q2     

    LAB_WBC_#/hpf     

    _Q1     

    LAB_Alanine Aminotransferase (ALT)_IU/L     

    _Q3     

    LAB_Albumin_g/dL     

    _Q3     

    LAB_Alkaline Phosphatase_IU/L     

    _Q1     

    LAB_Anion Gap_mEq/L     

    _Q5     

    LAB_Asparate Aminotransferase (AST)_IU/L     

    _Q8     

    LAB_MCHC_g/dL     

    _Q9     

    LAB_MCV_fL     

    _Q7     

    LAB_Monocytes_%     

    _Q7     

    LAB_Neutrophils_%     

    _Q3     

    LAB_Platelet Count_K/uL     

    _Q3     

    LAB_RDW_%     

    _Q1     

    LAB_Red Blood Cells_m/uL     

    _Q8     

    LAB_White Blood Cells_K/uL     

    _Q5     

    LAB_Absolute Basophil Count_K/uL     

    _Q5     

    LAB_Absolute Eosinophil Count_K/uL     

    _Q4     

    LAB_Absolute Monocyte Count_K/uL     

    _Q6     

    _6h-12h     

    INPATIENT_ADMISSION_START     Token indicating an admission to the hospital 

    TYPE_OBSERVATION     The type of the admission is observation 

    INSURANCE_MEDICARE     The patient’s insurance is MEDICARE 

    [l]ICD_Other symptoms and signs involving cognitive 
 functions and awareness     [l]The primary diagnosis at the beginning of the hospital stay
 is Altered mental status, unspecified (R4182), which is 
 broken down into two tokens: R41 and 82 

    ICD_4-5_82     

    ICD_Dorsalgia     

    ICD_4-5_16     

    ICD_Malaise and fatigue     

    ICD_4-5_1     

    ICD_Personal history of certain other diseases     

    ICD_4-5_73     

    ICD_Other hypothyroidism     

    ICD_4-5_9     

    ICD_Type 2 diabetes mellitus     

    ICD_4-5_21     

    ICD_Dorsalgia     

    ICD_4-5_5     

    ICD_Abnormalities of gait and mobility     

    ICD_4-5_2     

    TRANSFER_MED     Transfer to a different care unit - MED 

    _6h-12h     

    ATC_stomatological preparations     

    ATC_4_A     

    ATC_SUFFIX_D05     

    ATC_diuretics ATC_4_A     

    ATC_4_A     

    ATC_SUFFIX_A03     

    ATC_agents acting on the renin-angiotensin system     

    ATC_4_A     

    ATC_SUFFIX_A03     

    _6h-12h     

    ED_ADMISSION_END     Discharge from Emergency Department 

    _Q10     [l]Quantile referring to the length of the stay in Emergency 
 Department based on all stays in the data 

    INPATIENT_ADMISSION_END     Discharge from the hospital 

    _Q2     Quantile referring to the length of the stay in the hospital 
 based on all stays in the data 

    DISCHARGED_UNKNOWN     The reason of discharge 

    UNKNOWN_DRG     DRG assigned to the hospital stay 

    _=6mt     Time interval of 6 months 

    

ll
    Example of a tokenized timeline with an admission to ICU

    Patient Health Timeline     Notes 

    

    2c
      – Continued from previous page 

    Patient Health Timeline     Notes 

    

    2rContinued on next page 

    

    ICU_STAY_START     Admission to ICU token 

    Surgical Intensive Care Unit (SICU)     Type of ICU 

    SOFA     Indicates SOFA quantile follows 

    _Q1     Q indicating quantile of SOFA score 

    _15m-1h     time-interval 

    LAB_Sodium_mEq/L     Lab test results 

    _Q7     

    LAB_Urea Nitrogen_mg/dL     

    _Q3     

    LAB_L_no_unit     

    _Q4     

    _15m-1h     time-interval 

    LAB_Base Excess_mEq/L     

    _Q6     

    LAB_Calculated Total CO2_mEq/L     

    _Q8     

    LAB_pCO2_mm Hg     

    _Q7     

    LAB_pH_units     

    _Q7     

    LAB_pO2_mm Hg     

    _Q5     

    _2h-6h     

    ATC_anti-asthmatics     Medications start and administered with specified time intervals 

    ATC_4_A     

    ATC_SUFFIX_C02     

    _5m-15m     

    ATC_analgesics     

    ATC_4_B     

    ATC_SUFFIX_E01     

    _2h-6h     

    ATC_drugs for acid related disorders     

    ATC_4_B     

    ATC_SUFFIX_C02     

    _1h-2h     

    ATC_antithrombotic agents     

    ATC_4_A     

    ATC_SUFFIX_B01     

    _15m-1h     

    ATC_beta blocking agents     

    ATC_4_A     

    ATC_SUFFIX_B02     

    _1h-2h     

    ATC_mineral supplements     Patient is medicated until the end of ICU stay 

    ATC_4_C     

    ATC_SUFFIX_C02     

    _15m-1h     

    ATC_laxatives     

    ATC_4_    

    A ATC_SUFFIX_D04     

    _1h-2h     

    ATC_analgesics     

    ATC_4_B     

    ATC_SUFFIX_E01     

    _15m-1h     

    ATC_antipruritics, anesthetics, etc.     

    ATC_4_    

    A ATC_SUFFIX_A32     

    LAB_Hematocrit_%     Here lab test results are reported 

    _Q7     

    LAB_Hemoglobin_g/dL     

    _Q6     

    LAB_MCH_pg     

    _Q8     

    LAB_MCHC_g/dL     

    _Q5     

    LAB_MCV_fL     

    _Q9     

    LAB_Platelet Count_K/uL     

    _Q3     

    LAB_RDW_%     

    _Q4     

    LAB_Red Blood Cells_m/uL     

    _Q5     

    LAB_White Blood Cells_K/uL     

    _Q4     

    LAB_RDW-SD_fL     

    _Q6     

    LAB_INR(PT)_no_unit     

    _Q4     

    LAB_PT_sec     

    _Q5     

    LAB_PTT_sec     

    _Q2     

    LAB_Anion Gap_mEq/L     

    _Q4     

    LAB_Bicarbonate_mEq/L     

    _Q4     

    LAB_Calcium, Total_mg/dL     

    _Q3     

    LAB_Chloride_mEq/L     

    _Q5     

    LAB_Creatinine_mg/dL     

    _Q2     

    LAB_Glucose_mg/dL     

    _Q8     

    LAB_H_no_unit     

    _Q9     

    LAB_I_no_unit     

    _Q2     

    LAB_Magnesium_mg/dL     

    _Q2     

    LAB_Phosphate_mg/dL     

    _Q1     

    LAB_Potassium_mEq/L     

    _Q6     

    LAB_Sodium_mEq/L     

    _Q6     

    LAB_Urea Nitrogen_mg/dL     

    _Q2     

    LAB_L_no_unit     

    _Q2     

    _1h-2h     

    ATC_antithrombotic agents     

    ATC_4_A     

    ATC_SUFFIX_B01     

    ICU_STAY_END     Patient is discharged from ICU 

    _Q5     [l]Quantile token indicates the length of stay in ICU. 
 Here it is 5, indicating about average 





 Paper Title: Discovering Car-following Dynamics from Trajectory Data through Deep Learning.


Diff3DETR: Agent-based Diffusion Model for Semi-supervised 3D Object Detection
    Jiacheng Deng10000-0003-2838-0378 Jiahao Lu1 0009-0006-3280-9447 Tianzhu Zhang1, 2 † 0000-0003-1856-9564
    
============================================================================================================



§ ABSTRACT 
This study aims to discover the governing mathematical expressions of car-following dynamics from trajectory data directly using deep learning techniques. We propose an expression exploration framework based on deep symbolic regression (DSR) integrated with a variable intersection selection (VIS) method to find variable combinations that encourage interpretable and parsimonious mathematical expressions. In the exploration learning process, two penalty terms are added to improve the reward function: (i) a complexity penalty to regulate the complexity of the explored expressions to be parsimonious, and (ii) a variable interaction penalty to encourage the expression exploration to focus on variable combinations that can best describe the data. 
We show the performance of the proposed method to learn several car-following dynamics models and discuss its limitations and future research directions. 










§ INTRODUCTION







There are two recent trends in transportation and the broader science/engineering fields, which make the headlines almost every day. The first one is the emergence of connected/automated vehicles (CAVs) that i) may introduce new, complex traffic dynamics and interactions in the current and future traffic streams, and ii) generate increasingly available and massive datasets from both vehicles and the infrastructure. The second trend is the rapid development and application of deep learning techniques that seem to revolutionize almost every aspect of technology, science, engineering, and the entire society. While there have been numerous studies and applications of deep learning in transportation, in the paper, we are interested in the question of whether deep learning can help discover traffic dynamics (car-following models in particular) from data directly with no or little human involvement. An affirmative answer to this question will not only help discover/develop traffic dynamics models in this era but also have important implications for other science/engineering fields where dynamical systems and their governing equations are widely used and studied.

Car-following depicts the driving behavior of how a vehicle (driver) follows and interacts with the vehicle in front of it. It is one of the basic traffic models in revealing traffic dynamics characteristics at the microscopic traffic flow level <cit.>. Car-following studies can be traced back to the 1950s and 1960s when <cit.>, and <cit.> initiated an era of modeling car-following and traffic dynamics. <cit.> modeled car-following dynamics on the assumption that a following vehicle (the follower) is required to maintain a certain distance from the preceding vehicle (the leader). <cit.> further analyzed vehicle movements by a linear relationship between the acceleration of the follower and the velocity gap of the leader-follower pair. The model is given in Eq. (<ref>) below. Here a_f(t+δ t), v_l (t), and v_f (t), respectively, represent the follower's acceleration at the next time step, and the velocity of the leader and the velocity of the follower at the current time step, with c the so-called sensitivity term that may depend on the system states. 


    a_f (t+δ t) = c[v_l (t) - v_f (t)].


The model listed in Eq.(<ref>) is named the General Motors (GM) model and is widely regarded as the foundational model based on which many later car-following models were developed, e.g., by specifying various forms of the sensitivity term c. It states that the follower will accelerate or decelerate depending on the speed difference between the leader and the follower, while the action is with a time delay δ t representing the reaction time of the follower. The model is simple and intuitively makes sense, and was also validated by real-world data <cit.>. It can thus be considered as the "governing equation" of the underlying dynamical system of car-following. The example of the above GM model also illustrates the unique feature of existing, physics-based car-following models: they are concise, explicit analytical models that have clear physical meanings and generalize well with different datasets/sites. 


The development of existing car-following models can be generally summarized as a scientific process <cit.>, which includes key steps such as problem definition, background research, experimental design, data collection, data analysis, hypothesis, and testing/validation. This is often an involved, iterative process that is time and resource-consuming. The data analysis and hypothesis steps are arguably the most intellectually challenging and most creative in the entire process. This is because the data analysis step requires innovative ways with intuition/knowledge to interpret the data, properly visualize, reveal salient patterns, and discover the right variables and parameters to build suitable car-following relations, as well as to apply knowledge to compare/contrast the patterns with existing models in the same or similar fields. This data analysis step draws useful insights that serve as the input to the hypothesis step. This later step then proposes a suitable dynamical model to best describe the discovered patterns and relations in the data analysis step, while at the same time ensuring that the proposed model is tractable for analysis and computation. This data analysis-hypothesis process often involves many iterations for trial and error, with heavy human involvement.

With the advances in deep learning and emerging technologies in transportation, we wonder if the data analysis-hypothesis steps in the existing scientific process can be replaced by a deep learning approach with minimum human interaction, for discovering the governing equations of car-following dynamics. The motivation for this is multifaceted. First, while predictive models <cit.>, including those predicting the car-following behavior <cit.>, have been increasingly developed in transportation based on deep learning techniques and outperformed traditional physics-based models, they are often a black box that lacks interpretations. We believe that concise models with clear physical interpretations, as traditionally done in transportation and other fields, are still crucial as "human knowledge" and in understanding and modeling new phenomena with emerging technologies and systems. This leads to the second motivation, i.e., important practical needs to develop such learning-based methods. Existing car-following models were developed mainly for human-driven vehicles (HDVs). With the emergence and rapid deployment of CAVs, it is imperative to understand the interactions, including car-following, among CAVs and between CAVs and HDVs <cit.>. Distinct to CAVs, CAV and HDV data may be collected by CAV themselves or from the infrastructure <cit.>, which may make the classical scientific process cumbersome to discover the governing equations for new car-following and other traffic dynamics. Deep learning models may be a promising alternative method. Third, it is an intriguing research topic to develop a fully data-driven approach integrated with deep learning, with no or little human involvement, to directly produce a car-following model that shares the same features with the existing models: concise, interpretable, and generalizable. In the meantime, a successful development of such learning-based methods can provide useful insights into the interpretability of deep learning techniques, an open question that is drawing much attention nowadays.





Developing the aforementioned learning techniques for car-following from data can be viewed as a data-driven discovery problem <cit.>. Early research has been primarily investigated on how to draw physical laws from ideal scientific experiments using classical methods such as regression models. In the early 2000s, <cit.> and <cit.> proposed an approach named "symbolic regression (SR)" for identifying the underlying governing equation of a dynamical system directly from data. The purpose of the SR approach is to determine a mathematical expression f: ℝ^d →ℝ, X_i ∈ℝ^d, y_i∈ℝ, and a dataset (X, y), such that f is a short mathematical expression and y ≈ f(X). The SR model was solved by Genetic Programming (GP) <cit.> based approaches, which were later widely extended <cit.>. We refer the readers to Section <ref> for more detailed reviews.


Despite the above progress of learning-based discovery methods, discovering the underlying governing expressions from real-world traffic data that often involves noise and uncertainties can be challenging. Further, integrating with GP to solve SR problems may struggle with the downside of GP, such as sub-optimal and complex solutions. This study applies deep symbolic regression (DSR) <cit.> and GP to uncover the underlying car-following mathematical models from trajectory data. In particular, we make two improvements to the DSR-GP framework. First, DSR-GP is given some prior knowledge before learning the dynamics. The prior knowledge is a set of feature (variable) combinations extracted from the trajectory data through the variable intersection selection (VIS) algorithm proposed by <cit.>. Through this prior knowledge, the framework is guided to mathematical expressions that have reasonable physical meanings. Second, we propose a new reward function to encourage exploring low-complexity mathematical expressions. These two improvements can lead to expressions that are parsimonious and interpretable. We call the proposed learning framework the VIS-enhanced DSR.

In the next section, we briefly review the DSR and VIS methods. The VIS-enhanced DSR framework is presented in Section <ref>. In this framework, we first apply VIS to discover variable combinations from a trajectory dataset and then use DSR to learn the dynamics based on the discovered variable combinations. The expression reconstruction performance of the proposed framework is compared with the performance using only DSR and DSR with GP. Further, we investigate the impact of data noise and the performance of the method on different types of car-following dynamics. The results and comparisons are included in Section <ref>. We discuss the limitation of the proposed method in Section <ref>, with conclusions and future research directions presented in Section <ref>.


The main contributions of this study are summarized as follows.

  
  * Deep learning models are proposed for identifying car-following dynamics from trajectory data with minimum human involvement.
  
  * VIS-enhanced DSR framework is developed to find mathematical expressions that tend to be parsimonious and interpretable.
  
  * Reward function in DSR is improved by new penalty terms to regulate the complexity of the explored expressions and to search expressions with desirable feature combinations. 







§ RELATED WORKS



In this section, we briefly review car-following models and the data-driven models in solving SR problems, including the GP methods and the DSR framework <cit.>. We also briefly review the VIS approach <cit.>. 




 §.§ Car-following Models


Most of the pioneering car-following models <cit.> between fifties to seventies modeled car-following dynamics as an acceleration function with the differences of positions and speeds between the leader and the follower. Later on, car-following models were extended to consider multiple scenarios including the follower's emergency braking, following the leader, approaching the leader, free moving, and accelerating from rest, where notable works include  <cit.>. In particular, the Krauss model <cit.> considers that the follower selects its action from its desired speed, its comfortable acceleration, or the speed that is able to maintain a safe distance from the leader. The Intelligent Driver Model (IDM) <cit.> further considers a "soft braking strategy" when the follower approaches a slow-moving leader. It has been shown that the Krauss model performs better over IDM when modeling unsteady traffic conditions, whereas IDM performs better under steady traffic conditions <cit.>. In the numerical experiments of this paper, we use a traffic simulation model that integrates the Krauss model for car following. The model is defined by Eq. (<ref>), where t_react, v_f(t), v_l(t), s_f(t),  and  s_des respectively represent follower's reaction time, the velocities of the follower and the leader at the current time step, the spacing at the current time step, and the desired spacing (s_des) that is defined by measuring the distance that the leader is estimated to drive forward in the next time interval, i.e., s_des=v_l(t)δ t. The equation describes that given the current velocities of the leader and the follower, and the spacing between the pair, the follower selects its velocity from the choices of its current velocity plus its maximum acceleration v_f (t)+a_maxδ t, safe velocity v_s, and the maximum velocity that this vehicle is capable of driving at, i.e., v_max that is a known parameter. Other parameters that are pre-defined and assumed fixed in this model include the maximum acceleration a_max, comfortable deceleration b, driver's reaction time t_react, driving uncertainty ε, and time step δ t. 


    v_f (t+δ t)   =max(0, v_d-ε a_max) 
    
        v_d   =min(v_f (t)+a_maxδ t, v_s, v_max) 
    
        v_s   =v_l (t)+s_f(t)-s_des(t)/v_f (t)+v_l (t)/2b+t_react.


As this paper concerns about discovering existing car-following models using deep learning methods, we do not aim to conduct a comprehensive review of car-following models; interested readers may refer to <cit.> for more recent reviews.



 §.§ Data-driven Discovery Methods


GP <cit.> was used to solve SR problems <cit.>, which was shown to be effective by leveraging the evolutionary features of GP to improve fitting functions <cit.>. Many later approaches for solving SR problems were based on the concept of GP, including the epsilon-lexicase selection (EPLEX) <cit.>, the feature engineering automation tool (FEAT) <cit.>, semantic backpropagation GP (SBP-GP) <cit.>, the GP-based gene-pool optimal mixing evolutionary algorithm (GP-GOMEA) <cit.>, Operon <cit.>, and the interaction-transformation evolutionary algorithm (ITEA) <cit.>. Despite the powerful solving capabilities from the nature of evolution, GP methods suffer from generating appropriate expressions that are parsimonious and general <cit.>, and fitting may take a long time due to the large size of search space.

<cit.> developed the sparse identification of nonlinear dynamics (SINDy) framework that describes the expression discovery problem by a sparse regression problem. The framework is based on an underlying assumption that the dynamics of a physical system can be represented by a linear combination of a limited number of mathematical terms. The SINDy approach was integrated with autoencoders by <cit.> and further with Bayesian approaches by <cit.>. However, SINDy users need to provide all the possible feature combinations for the framework to discover important terms. <cit.> incorporated Bayesian approaches in solving SR problems. In the deep learning domain, the AIFeynman approach <cit.> is also worth mentioning. It is inspired by the divide-and-conquer method powered by neural networks. <cit.> and <cit.> applied graph neural networks (GNNs) and transformers to solve SR problems, respectively. Similar to the works by <cit.> and <cit.>, which discovered the governing behaviors from the intrinsic coordinates, <cit.> proposed a two-stage autoencoder framework to estimate the intrinsic variables and use the variables to predict the behaviors at the next time step. The intrinsic patterns are however hidden in the autoencoders and thus are not interpretable. 

DSR <cit.> is a recent framework that uses a recurrent neural network (RNN) with the proposed risk-seeking policy gradient approach for solving SR problems. DSR leverages the conditional probability feature in RNNs to increase the parsimony and accelerate the exploration efficiency. It was developed on the basis of the fact that a mathematical expression can be represented by an expression tree <cit.> composed of mathematical tokens, where the tokens are sampled by RNNs from a pre-defined token pool. The sampling is restricted by fixed constraints to restrict token combinations and lengths of expressions, i.e., expression complexities. Given a range of expression complexity as an exploration constraint, the RNN will stop sampling if the expression has enough complexity and the expression tree is complete. This mechanism may result in expressions with a high level of complexity that lose parsimony and interpretability, which is also the main issue of GP. Data noise is another challenge: results showed that 10% of data noise affected over 10% of expression reconstruction, although it outperformed benchmark solvers, including GP, Eureqa <cit.>, etc. Expression searching in the DSR framework relies on training RNNs or long-short-term-memory (LSTM) neural networks, a special structure of RNNs, whose training can sometimes be computationally expensive. Combining GP with DSR has also been proposed <cit.>, for which GP is an option to assist the search. 



 §.§ Variable Interaction Selection (VIS)


To encourage the exploration to search for expressions that have reasonable lengths and physical meanings, we add a VIS step prior to DSR to provide knowledge to the exploration. This VIS step produces potential variable combinations extracted from data. The VIS algorithm was introduced in <cit.> in their proposed sparse interaction additive networks (SIANs). SIANs leverages the generalized additive function, which speculates that a dependent variable is linearly related to an addition of a set of shape functions. A shape function can be any specific function, e.g., a polynomial function of a single variable or variable combinations that have a strong interaction, where the interaction strength is quantified by Archipelago <cit.>. 
The algorithm first fits a deep neural network to the data, where the deep neural network acts as the reference function. The algorithm detects all the possible variable combination candidates and the selection is based on several key criteria including a pre-defined threshold of the interaction strength. That is, a variable combination is determined as a strong interaction if its interaction strength is greater than the threshold. We next present the VIS-enhanced DSR method to discover car-following models from data.



§ VIS-ENHANCED DSR METHOD



Fig. <ref> depicts the flowchart of our proposed VIS-enhanced DSR method. On the left of the flowchart, we use a VIS process to detect potential variable combinations from the trajectory data to increase the interpretability of the explored expressions. On the right of the flowchart, we use an expression checker to detect whether the expression determined by the RNN sampling has reasonable physical meanings based on the recommended variable combinations. The RNN training is governed by our proposed reward function that encourages parsimonious expressions for better interpretation. Fig. <ref> illustrates a pipeline example of the VIS process. Given a set of trajectory data, the process trains a deep neural network to match the data and then produces a set of variable combinations that can capture the salient patterns in the data. The identified variable combinations can guide DSR to search for mathematical expressions that contain such variables. To introduce the framework and the reward function in detail, we first define the problem and notations as follows. 



Define the same mathematical expression f(X) and dataset (X, y) introduced in Section <ref>, i.e., 


    y=f(X)+ϵ,

where ϵ represents the approximation error term. We aim to discover the exact form of f(X) from the dataset. Given a set of mathematical tokens 𝒯, which can include mathematical operations (+, -, sin, constant placeholders, etc.) and variables, the DSR framework samples each mathematical token τ_i from 𝒯 according to the probabilities determined by the RNN. The token in 𝒯 with the highest probability will be sampled and stored in the latest node spot. Each mathematical term is sampled without replacement. We denote the resulting expression by f_s(τ), where τ = {τ_i }_i=1 ^n_s,  ∀τ_i ∈𝒯, and n_s represents the number of sampling occasions, i.e., expression complexity. We apply the DSR approach to find f_s(τ) through sampling tokens from the given 𝒯 such that ϵ is small. As an example, f_s(τ) can be α x+sinx if the sample token set is {+, ×, α, x, sin, x}, where x is the only variable and α is a constant.

As shown in Fig. <ref>, we first feed the collected trajectory dataset, (X, y), into both the VIS and DSR blocks. VIS starts to find the variable interactions from the trajectory data, as shown in Fig. <ref>, by first fitting a deep neural network to the data, calculating the interaction strength for variables or variable combinations, and determining whether a variable or combination is significant to the data relationship. In the meantime, DSR starts to explore potential expressions that best describe the relationship between X and y. The potential expressions are then fed into the expression-checking process, which will check each expression candidate term by term for reward evaluation. The checking process will also assess whether the expression has recommended variable combinations by VIS and tag those without any recommended combinations (so that a penalty may be imposed as shown in Section <ref>). The DSR process will then continue evaluating the reward of the retained expression candidates, training the RNN, and exploring potential expressions.



 §.§ Variable Interaction Selection (VIS)


In the VIS process, we consider an extended generalized additive model <cit.>, in which the dependent variable linearly depends on some functions consisting of single variables and/or groups of variable combinations, as defined in Eq.(<ref>). In the equation, f_∅ is a constant term. Each h_i(X_i) function is a shape function that represents a specific function, e.g., a polynomial function. X_i is an element of the variable set, i.e., X_i ∈ℐ⊆{X_i}_i=1^d. The last term considers the variable combinations, where Z_j (e.g., Z_j=(X_1, X_2)) represents an element in the set of variable combinations. The variable combination set is defined as 𝒥, Z_j ∈𝒥.


    y = f_∅ + ∑_X_i ∈ℐ h_i(X_i) + ∑_Z_j ∈𝒥 h_j(Z_j).


In VIS, a deep neural network is first trained to fit the data. We use f^dnn(X) to denote the deep neural network. The interaction strengths of variables (in set ℐ) or variable combinations (in set 𝒥), denoted as ψ(ℐ) and ψ(𝒥) respectively, can be measured by the Archipelago <cit.>, which uses the approximation of the Hessian of f^dnn(X) as shown in Eq. (<ref>). The VIS algorithm proposed in <cit.> is then used to detect strong variable interactions performed in the data based on f^dnn(X) and Eq. (<ref>). 


    ψ(ℐ) := 𝔼_X [ ∂^|ℐ| f^dnn(X)/∂ X_i_1∂ X_i_2⋯∂ X_i_|ℐ|]^2 > 0. 
       ψ(𝒥) := 𝔼_X [ ∂^|𝒥| f^dnn(X)/∂ Z_j_1∂ Z_j_2⋯∂ Z_j_|𝒥|]^2 > 0.




 §.§ Improved DSR for Discovering Car-following Dynamics


In the DSR process, we use an LSTM neural network to sample tokens. The LSTM is trained through the risk-seeking policy gradient algorithm proposed by <cit.>, which updates the LSTM weights through the top-performing cases (instead of the average-performing cases as normally done). We redesign the original reward function in <cit.> to encourage the LSTM to (i) explore expressions consisting of the recommended variable combinations generated by VIS, and (ii) explore expressions with reasonable expression complexity. The new reward function is shown in Eq. (<ref>), where R_c represents the combined reward function, L_e represents an error-based loss function, p is the complexity of an expression f_s, and β is a penalty parameter that penalizes expressions that do not contain the recommended variable combinations from VIS. The penalty only steps in for the first EPOCH epochs to initially guide the exploring direction and prevent the exploration from being time-consuming, as checking each term of an expression needs extra time.


    R   = 
        
            R_c(1-β),   if no potential interactions and epoch≤EPOCH
    
            R_c,                 otherwise,
        , 
    
        R_c   = 2/(1+L_e)+(1+norm(p(f_s))),





As shown in Eq. (<ref>), the combined reward function R_c considers the error-based loss and the complexity of the expression candidate. R_c is squashed to a range between 0 to 1. R_c → 1.0 when the error loss and the loss of complexity are both close to zero, i.e., L_e → 0 and norm (p(f_s)) → 0. R_c goes down when the predicted error and/or the complexity of an expression increase. To encourage the LSTM to explore expressions with reasonable complexities, we design the loss of expression complexity as a normalization function. The normalization relies on human knowledge to provide a reasonable range, as defined in Eq. (<ref>), where p_min and p_max represent the minimum expression complexity and maximum expression complexity, respectively, which are pre-defined based on human knowledge.


    norm(p(f_s))   = 
        p(f_s) - p_min/p_max - p_min,   if  p_min≤ p(f_s)
    p_max - p(f_s)/p_max - p_min,                 otherwise.


The equation's expression complexity is scaled into a base range of 0 to 1. When the complexity is greater than or equal to the minimum p_min, it is scaled by the top conversion of the equation. To prevent the same conversion from scaling the complexity to a negative number when the complexity is smaller than the minimum, we use the bottom conversion of the equation to scale the complexity. The bottom conversion comes from the idea of treating the smaller complexity (smaller than the desired range) as the greater complexity (greater than the desired range) to prevent producing negative losses. That is, if p(f_s) < p_min, then norm(p(f_s)) = ( p_max + (p_min - p) ) - p_min/p_max - p_min, and rewriting ( p_max + (p_min - p) ) - p_min/p_max - p_min = ( p_max - p )/p_max - p_min obtains the bottom conversion of Eq. (<ref>).

L_e here captures the "accuracy" of the expression in describing the data. It can be any predicted loss function, such as the root-mean-squared error (RMSE), mean-squared-error (MSE), normalized MSE (NMSE), NRMSE, log MSE, etc. This study uses NRMSE as L_e. Given the expression formed by sampled tokens f_s(τ) and dataset (X, y), L_e is defined as Eq. (<ref>), where σ_y is the standard deviation of y.


    L_e = √(1/nΣ^n_i=1 (y_i - f_s(τ, X_i))^2)/σ_y.


Before concluding this section, we summarize the human involvements/knowledge that are required by the proposed VIS-enhanced DRS method. A set of tokens is first required, including variables (X) and mathematical operations. The variables can be relatively easily obtained as the key elements in the data: e.g., for trajectory data, this can be speeds, accelerations, locations, etc. Operations are a bit trickier and require certain level of human knowledge, e.g., the operations used in existing car-following models if new car-following models are to be developed. The good news is that we do not need to provide the exact set of operations, e.g., multiplication and subtraction for the car-following model in Eq. (<ref>). Rather a superset of the needed operations is sufficient, e.g., the set of operations that contain ×, ÷, +, - for Eq. (<ref>). Other human inputs include the min and max complexities in Eq (<ref>) and the β parameter in Eq (<ref>). These may be treated as hyberparameters for machine leanring, which may be pre-defined based on knowledge or learned by trial and error. Compared with traditional methods of deriving a car-following model from observed data, we can clearly see that the human involvement in the proposed method is marginal since the data analysis and hypothesis steps are largely automated via the VIS and DSR processes.




§ NUMERICAL RESULTS



We now use the proposed VIS-enhanced DSR framework to help discover the potential governing mathematical expressions from car-following trajectory datasets. We first present how the datasets are prepared. 



 §.§ Dataset and Problem Statement



The trajectory data is collected through numerical simulation with car-following pairs using the Simulation of Urban MObility (SUMO). To enhance the variety of the car-following behaviors, we control the leading vehicle by randomly selecting the velocity between 0 to 30 m/s, whereas the decisions of the following vehicle are described by the Krauss model <cit.>, as defined in Eq. (<ref>), with predefined vehicle parameters: a_max=2.6  m/s^2, v_max=55.55  m/s, b=4.5  m/s^2, and 5-m vehicle length. The follower is first assumed to have zero driving uncertainty, i.e., ε = 0.0, and their reaction time is a second, i.e., t_react=1.0 second. Later, we also impose data noises and test how the proposed method can deal with such noises. The trajectory dataset has in total ∼3.6k car-following samples (pairs). Given the parameters, the Krauss model can be simplified as Eq. <ref>. The function will be the target expression in the following experiments. From the simulation, we collect the velocities of the follower and the leader at the current time step, the spacing at the current time step, the following velocity at the next time step, and the difference between current spacing and the desired spacing, i.e., , v_f(t), v_l(t), s_f(t), v_f(t+δ t), and ds(t)=s_f(t)-s_des(t), where s_des(t)=v_l(t)δ t.

The data is organized into independent variables X={v_f(t), v_l(t), s_f(t), ds(t)}, whereas the dependent variable is the velocity of the follower at the next time step, i.e., y=v_f(t+δ t). The problem is to develop a mathematical expression that describes the relationship between the collected features and the next velocity of the follower, i.e., v_f(t+δ t) = f(v_f(t), v_l(t), s_f(t), ds(t)).


    v_f (t+δ t)   =min(v_f (t)+2.6, v_s) 
    
        v_s   =v_l (t)+9.0 ( s_f(t)-s_des (t) )/v_f (t)+v_l (t)+9.0.


Note that certain simplifications are performed to derive (<ref>), including only vehicles from a single class (cars), setting the reaction time to be 1s, as well as fixing the maximum acceleration and speed. In Section <ref>, we discuss these issues and possible ways to relax them.



 §.§ Variable Interactions


The interaction selection process starts by training a 3-layer (in-32-64-32-out) neural network. The trajectory dataset is split into an 80% training set and a 20% validation set. After training, the mean squared error of the deep neural network estimation drops to 7.5E-04, 7.1E-04, and 4.6E-04 after 50, 100, and 300 epochs of training, as shown in Fig. <ref>. The training takes 5.8, 16.9, and 60.6 seconds at 50, 100, and 300 epochs on an Intel Core i7-12700 CPU. 



Given variables v_f(t), v_l(t), s_f(t), and ds(t), the potential variable combinations along with their interaction strengths after 300 epochs of training are listed in Table <ref>, where variable combinations are listed in ascending order by the interaction strength presented in Fig. <ref>. In this figure, a few elbow points can be observed, including strengths of v_l(t), (ds(t), v_l(t), v_f(t)), (v_l(t), v_f(t)), (ds(t), v_f(t)), and (ds(t), s_f(t)). Based on these elbow points, five sets of variables or variable combinations are recommended as follows. 


    
  Set 0 {v_f(t), v_l(t)},
    
  Set 1 {ds(t), (ds(t), v_l(t), v_f(t))},
    
  Set 2 {(ds(t), v_l(t)), (v_l(t), v_f(t))},
    
  Set 3 {(ds(t), v_f(t))},
    
  Set 4 {(ds(t), s_f(t))}.


The two variables in Set 0 are unlikely to be the only features (variables) describing the car-following behavior since the spacing difference may also be an important feature. We therefore combine Set 0-1 as the first possible recommended set of variable combinations, denoted as Scenario 1. The rest sets are respectively coded up as Scenario 2 to Scenario 4. Table <ref> presents all variable combinations, their interaction strengths, and the scenarios. For instance, Scenario #1 uses Set 0 and Set 1, and Scenario #2 uses Set 0 - Set 2. 







 §.§ Expression Exploration

Based on the literature and analysis in the past <cit.>, car-following behaviors are related to the velocity and gap differences with +, -, ×, ÷, min and parameters a_max, and b. We thus provide +, -, ×, ÷, min, v_f(t), v_l(t), ds(t), s_f(t), a_max, and b as the set of tokens to the DSR process. If we allow exploration to focus solely on fitting accuracy without constraints and complexity regulations, we may end up with a complex expression that lacks physical meaning, e.g., the expression below, which may have a high level of accuracy but also a high level of expression complexity. Such expression is certainly not desirable.


    (s_f(t)(-v_l(t)/((v_f(t) - a_max/(v_f(t) s_f(t) min{-2b, v_l(t)})) min{v_f(t), v_l(t)}) - 2b) + 
     s_f(t) -min{a_max, s_f(t)} -min{v_f(t), a_max} -min{s_f(t), -2b v_l(t)})/(v_f(t) +v_l(t)).


In the following expression exploration, min is limited to only being sampled once and should be the first token sampled. We configure a size of 300 to be the batch size of exploration. The reward penalty to encourage recommended variable interactions will be turned off after 10 epochs of exploration, i.e., EPOCH=10 in Eq. (<ref>), to prevent too much running time while gaining initial guidance. The classic GP method is set to be allowed to assist expression exploration. In the following, we use DSR based on the variable combinations recommended by the VIS method with the proposed complexity penalty reward strategy. All of the cases in the remainder of this subsection are tested using a noise-free dataset. We compare the performances of the pure DSR framework <cit.>, the DSR framework with GP assistance <cit.>, and our proposed VIS-enhanced DSR framework with GP assistance. We use DSR-only, DSR-GP, and VIS-DSR-GP in the following to refer to the three methods, respectively. The DSR-GP method is configured with the same constraints of limiting min to only being sampled once and first. Additionally, DSR-GP is configured with a constraint of limiting the length of explored expressions to a range of 10 to 40. 



  §.§.§ Expression Exploration with Penalty

We first compare explorations using different penalty parameters of the complexity penalty strategy, i.e., β in Eq. (<ref>). In the experiment, βs are tested with values from 0.00 to 1.0, with a 0.05 increment from 0.00 to 0.20 and then with a 0.20 increment from 0.20 to 1.0. All the tests are based on the recommended variable combinations in Scenario #1 shown in Table <ref>. Note that β=0 indicates R=R_c, i.e., there is no penalty if DSR explores expressions with variable combinations not recommended by the VIS.



We compare exploration epochs until converging using the VIS-DSR-GP and DSR-GP methods with different β's in Fig. <ref>, where each result is averaged over 10 random seeds with the standard errors shown by the error bars in red. All have discovered the target expression as Eq. (<ref>). DSR-only is excluded from the comparison, as it does not recover the target expression.  Fig. <ref> and Fig. <ref> compare the numbers of learning epochs and running times over different βs, where each value is also the average of the results from running the test over 10 random seeds. The comparisons show that 0.15 is a sweet spot of β (VIS-DSR-GP), as it takes the fewest number of epochs and the shortest running time to discover the target expression. Even adding the running time of VIS (60.6 seconds from Section <ref>), VIS-DSR-GP with β=0.15 is still faster than using DSR-GP, which takes more than 150 epochs and over 200 seconds to discover the target expression. Choosing β=0.00 takes more epochs to learn the dynamics, but the running time is relatively short. It is because in the test of β=0.00, the exploration does not check each expression term, and the reward takes only accuracy and parsimony into account, i.e., R=R_c in Eq. (<ref>). Using DSR-GP is found to tend to look for better expressions through higher complexity, as shown in Fig. <ref> that the high range of expression complexities (20 to 40 given the desired range from 10 to 40) dominates the exploration compared to the range of low complexities (10 to 20). As shown in Fig. <ref>, running multiple tests with the DSR-GP method, we found that this method tends to try exploring better expressions through a high level of expression complexity. Our proposed method, on the other hand, encourages the exploration of parsimonious expressions, as presented in Fig. <ref>, where the expression complexities explored are mostly in a range of 10 to 20.






  §.§.§ Expression Exploration with Recommended Variable Combinations

The above experiments indicate that using β=0.15 performs the best in terms of the number of exploration epochs and running times. We adopt β=0.15 and apply the same constraints to test explorations with different recommended variable combinations listed in Table <ref>. The following tests still use the noise-free dataset. Fig. <ref> presents a performance comparison using different sets of recommended variable combinations, where blue bars show the average of running 10 random seeds, and error bars in red demonstrate the standard errors. The target expression is discovered by all the test cases. The results show that a good set of variable combinations can help reduce the number of exploration epochs and further improve the running time. With that said, providing too many conditions to regulate expression terms may result in long running times, as the elapsed times of Scenarios #2 - #4 shown in Fig. <ref>. In particular, Scenarios #3 and #4 take a longer time to find the target expression than DSR-GP.





 §.§ Noise Impact Investigation



The above experiments are based on perfectly simulated data described by only one governing equation. Here we test how data noises may impact the performance of the proposed method. The following tests in this subsection are given the set of variable combinations recommended in Scenario #1 (see Table <ref>) with β=0.15 and the same constraint configuration, and a recommended expression complexity ranging from 10 to 40.

In the investigation, the dataset is contaminated 100% by different noise levels, which increase from 0 to 10% with a 1% increment of the standard deviation of the clean data. Fig. <ref> shows an example of the clean data, data with 5%-level noise, and data with 10%-level noise. The VIS algorithm yields the same recommended variable combination sets as listed in Table <ref> at all noise levels. Thus, the following focuses on comparing the performances of the DSR process under different noise levels. The performances in terms of mean percent errors (MPE) are presented in Fig. <ref>, where we compare the MPEs between using DSR-only, DSR-GP, and VIS-DSR-GP. 








This figure shows MPEs (with respect to the ground truth) over different noise levels, where each result is the average of running 10 random seeds with error bars (vertical lines) indicating standard errors. The results demonstrate that DSR-only fails to discover a suitable expression that can adequately capture the behaviors performed in the data for all cases, even on clean data (0% noise level). The average MPEs are at around 3% given the noise levels below 7%. DSR-GP can reconstruct the target expression for all random seeds on clean data. The averaged MPEs drop below 2% for noise levels from 0% to 9%, while MPEs range 2-5% when the dataset is imposed 10% of noise. The results clearly show that GP helps reduce the impact of data noise. On the other hand, with prior knowledge of potential variable combinations, VIS-DSR-GP can uncover the target expression for all tests when noise levels are at or below 3%, and produce expressions with the highest MPEs below 2% (at 10% of noise level). The best expressions (out of the 10 random seeds) recovered using these methods are listed in Table <ref>, where we only document the component of the second term in the min function, as all the methods can recover the first term (v_f(t)+2.6). Here the expression is chosen as the best if it has the highest accuracy and/or the most similar expression structure as the target expression. The results show that DSR-only is not able to find the correct expression at all noise levels. It falls into a local optimum, where the expression structure of each term is limited. With the help of GP, DSR-GP can recover the target expression at or below 6%. On the other hand, VIS-DSR-GP can recover the target expression for all tests at least once at all noise levels. The results also show that learning the exact structure of the expression (a division of two terms in this case) and the correct values of the constant (e.g., 9.0) are challenging for DSR and DSR-GP. 




 §.§ Testing Performance on Other Car-following Models


In this subsection, we test the expression reconstruction capability of the proposed VIS-enhanced DSR framework (VIS-DSR-GP) on other car-following models. We choose the GM model <cit.> and the Gazis-Herman-Rothery (GHR) model <cit.> as the target expressions in the following experiments. The two expressions are respectively listed in Eq. (<ref>) and Eq. (<ref>). Here we simplify the sensitivity term c in the GM model as a constant.


    v_f (t+δ t) = v_f (t) + c[v_l (t) - v_f (t)].











    v_f (t+δ t) = v_f (t) + k_1v_f^k_2(t)
        v_l(t-δ t) - v_f(t-δ t)/s^k_3_f(t-δ t).


Each car-following dataset is generated through the SUMO numerical simulation with the corresponding car-following model sharing the same vehicle parameters, including δ t, a_max, v_max, b, and t_react, as the Krauss model mentioned in Section <ref>. For constants in the tested car-following models, we use the recommended c=0.368 <cit.> for the GM model, and k_1=1.2, k_2=1.0, k_3=1.1 for the GHR model <cit.>. While collecting the GHR car-following data, we simplify the velocity difference as a term, i.e., Δ v(t-δ t)= v_l (t-δ t) - v_f (t-δ t). The simplified car-following models with pre-determined parameters are as Eq. (<ref>) and Eq. (<ref>) for GM and GHR, respectively.


    v_f (t+1) = v_f (t) + 0.368 [v_l (t) - v_f (t)].



    v_f (t+1) = v_f (t) + 1.2v_f(t)
        Δ v(t-1)/s^1.1_f(t-1).


The collected GM dataset shares the same variables as the Krauss model dataset, i.e., features in the GM dataset include v_f(t), v_l(t), s_f(t), and ds(t). The collected GHR dataset, on the other hand, comprises variables v_f(t), v_l(t), s_f(t-1), and Δ v(t-1), as it has more specific features. Feeding both datasets respectively in the VIS algorithm obtains interaction strengths as shown in Fig. <ref>. We apply the elbow method to determine recommended combinations and document them in Table <ref>. 

Now, we use the VIS-enhanced DSR framework to reconstruct the target expressions. After being simplified, the GM and GHR expressions have one and two constants, respectively. We reconstruct the two models by giving the framework the following mathematical token pool: {+, -, ×, ÷, min, power, constant},  parameters {a_max, v_max, and b}, and the variables as shown in Table <ref>. The hard constraints include: i) min and constants cannot occur more than twice, and ii) min should be the first sampled token. Soft regulations, such as the reward penalty factor and recommended complexity, are configured the same as the values in the previous subsections.






By imposing noise from level 0% to 10%, we document the best-reconstructed expressions in Table <ref>, to which we document only the second term in each expression since the first term can be correctly learned by the method for all noise levels. From the table, VIS-DSR-GP is able to reconstruct the GM expression with the given token pool and constraints for the zero-noise dataset. Although the decimal values of the explored constants between noise levels 1% to 7% do not exactly match the target value, they are quite close (less than 0.5%). However, the constant starts to differ significantly from the target value when the noise level exceeds 8%. For the GHR, on the other hand, the best expression the framework can explore is v_f (t+1) = v_f (t) + 1.01v_f(t) v_l(t-1) - v_f(t-1)/s_f(t-1)-0.16. Although the resulting expression has the same structure as the GHR: v_f(t), plus the product of a constant and a fraction term, the constant and the power term (also a constant) in the denominator is the most challenging to learn and discover. This becomes even more challenging when the data has a higher level of noise. Starting at 5% noise level, an additional term appears beside the primary expression structure. When the noise level exceeds 9%, the structure of the explored expression begins to noticeably differ from the target expression of GHR. The results here further confirms that learning the exact structure of the model and the constants (especially those with decimal values) are challening even for the proposed VIS-DSR-GP method.






§ DISCUSSIONS


The results of the above numerical experiments illustrate that under relatively simple scenarios - Krauss or GM model, single vehicle class, noise-free or low-noise level (less than 8% as shown in Table <ref> and Table <ref>) - the proposed VIS-enhanced DSR method can discover the target car-following models exactly or almost exactly from data directly with minimum human involvement. We think this is quite impressive considering the various structures/terms, math operations, variables, and especially constants in the expressions of the car-following models. This is especially so when compared with traditional methods for developing car-following models, for which extensive, time-consuming, and iterative processes of data analysis, visualization, hypothesis, validation, and testing are essential steps. The results, however, also demonstrate that the proposed framework faces significant challenges in exploring expressions correctly for dynamics with complex expression structures, constants, and data noises. Target expressions with intricate structures, such as the power term in the GHR model, can make it more difficult for the proposed method to find the correct expression. When there are many constants to be fitted, finding the exact expressions can also be challenging. Another challenge is data noises, which can have a significant impact on the exploration process, as they can obscure the desired predictions and thus make it difficult to obtain the target expression terms correctly. Furthermore, real-world traffic streams often contain multiple vehicle classes, such as cars, buses, and trucks, which may have different key model parameters (maximum speed, acceleration, etc.) even if they follow the same car-following dynamics. Our initial testing shows that this can bring great challenge to the proposed learning method, due to the large number of constants and the multimodal nature of the problem. As a result, there is much room for improvements in VIS-enhanced DSR to address these challenges.

More advanced deep learning techniques, such as large language model (LLM) based methods, need to be explored to address these challenges; see Section <ref> for more discussions on this aspect. On the other hand, clearly defining objectives and developing measures that can properly capture such objectives are also crucial. With the goal of learning the governing equations of dynamical systems, such as the car-following dynamics in this paper, the learned mathematical expressions should accurately capture the observed data, which should also have proper physical meanings and can be generalized to different sites and traffic conditions. Learning accuracy is not the only objective, which is starkly different from other DL-based applications. This prompts the definition of four key measures that help evaluate the quality of the derived expressions: i) accuracy, ii) interpretability, iii) parsimony, and iv) generalizability, where interpretability, parsimony (compactness), and generalizability are also discussed in <cit.>.

Accuracy measures the “goodness of fit” of the learned expressions with respect to the observed data, which is the basic measure to evaluate the quality of the learned expression. A good expression should provide reasonable accuracy in fitting the observed data. Metrics such as MSE, NMSR, or MPE have been widely used to measure accuracy; for example, DSR uses NMSE as one measure for the performance of the resulting expression. In addition, other mathematical properties of the model, such as stability, are also studied and used as metrics to evaluate the performance of the models. 

Interpretability means that a learned expression should have human-understandable justifications about the “physical meaning” of the model. It is important as it implies “trust” – people often have more confidence in an interpretable model that can then be more easily adopted and used <cit.>.  Interpretability is an important objective for learning governing equations, as the main goal is to add new knowledge to our understanding of the physical systems by developing such models. This can be particularly essential (and challenging) when we learn the potentially new dynamics for mixed traffic flow of CAVs and HDVs, as discussed in the Conclusion Section below. As we expect that explicit analytical expressions are produced in this research, we focus on two aspects of interpretability: i) the variables in the model represent clear physical phenomena or characteristics of the traffic system; and ii) the relations of the variables described in a model have clear, concise (see parsimony below) explanations. In this paper, we applied VIS to find the key (strong) interactions among variables, with the help of constraining the model complexity, which helps produce expressions that have reasonable physical meanings. As interpretability and the above two aspects are more subjective, it is imperative to develop specific measures for interpretability that can be properly integrated into the DSR learning process (e.g., the design of the reward function). 


Parsimony is “a good principle to explain the phenomena by the simplest hypotheses possible” <cit.> or the simplest possible solution to a problem, which has been widely used as a guiding principle for daily life and scientific investigations <cit.>. In our specific context, parsimony means that the learned expressions are concise and straightforward, given all others, e.g., the metrics for other objectives being equal or very similar. In this paper, we use the complexity of the expression, i.e., the number of variables and operations included in the expression, as the measure to define and evaluate its parsimony. However, parsimony is also a subjective measure requiring substantial human judgment of the learned expression. Specific procedures/measures need to be developed to help evaluate the parsimony of an expression so that it is not too complex or over-parsimonious (i.e. too simple), and also help reach a proper balance with other objectives. 

Generalizability means that the learned governing equations can be applied to data collected from broader traffic systems at various sites with different traffic mixes and conditions, which indicates that the learned expression can capture the underlying dynamics of the traffic system rather than only fitting to a specific dataset. Generalizability can be measured by the model performance (e.g., accuracy) across different datasets and different sites. An expression that can describe all scenarios better is considered a more generalizable model than those that only work well for certain datasets but not on others. This may be measured by the metrics such as their averages and variances of other three objectives discussed above across different datasets and test sites. Furthermore, expressions with no or minimum calibration but can well describe the dynamics of different datasets and/or sites also have better generalizability properties.
 








§ CONCLUSION



In this study, we proposed an expression exploration framework based on the DSR approach while encouraging parsimonious mathematical expressions that include potential variable combinations detected through a VIS algorithm. The variable combinations are used to guide DSR in focusing on expressions that involve the identified combinations, which can also increase the interpretability of the explored mathematical expressions. For this, we designed two penalty terms for the reward function of training RNNs in DSR: (i) a complexity penalty to encourage the exploration to crawl toward the desired range of expression complexity, and (ii) a variable interaction penalty to encourage the exploration of expressions consisting of the recommended variable combinations. The following results were discovered. First, the recommended variable combinations were found useful for directing the exploration to identify suitable mathematical expressions. Secondly, the proposed complexity penalty strategy was shown to be better for regulating the complexity range searched than the traditional complexity regulation method in DSR. Thirdly, the exploration based on the recommended variable combinations with the complexity penalty strategy helped identify the governing equations more efficiently than the traditional DSR. From the data noise investigation experiments, we recommended turning on the exploration assistance of GP, as it enables a wider search space for exploration. 

The methods and results in this paper present the first step of discovering governing equations of car-following dynamics using deep learning. Limitations exist which motivate future research. We need to improve the current learning framework to address the limitations summarized in Section <ref>, and apply the improved framework to more diversified, real trajectory data. In particular, the proposed framework can be applied to multimodal traffic systems, e.g., to capture the car-following dynamics among CAVs and between HDVs and CAVs, and to help develop interpretable dynamical models for the interactions between vehicles and vulnerable road users. This may be done by integrating multiple dynamic exploration frameworks as a coordinated learning system in which each exploration framework learns an interaction type in a coordinated way from the collected multimodal data. 

There are two main directions that can help improve the current learning framework and address the challenges. First, as discussed in the previous section, accurately fitting the observed data is NOT the only objective for deriving proper governing equations. One has to balance with other objectives, including interpretability, parsimony, and generalizability. Proper metrics for each of these objectives need to be defined and integrated into the learning process, e.g., to design a comprehensive reward function that includes metrics for measuring parsimony, interpretability, generalization, and accuracy. An exploration incorporated with a reward that takes these features into account may also help find appropriate expressions from data with noise and complicated underlying dynamics. However, developing such a reward function is challenging, as there may be conflicts between accuracy and other objectives. Despite the challenges, this is an important research topic that has the potential to advance the capability of using deep learning models to "automatically" discover the underlying expressions from collected data. Second, given the advance of deep learning especially recent LLMs, one should leverage the ability of interaction and memory of LLMs to enhance search efficiency and expression quality in discovering the governing equations. Research has started in this direction <cit.>, which needs to be further improved and tailored to car-following and other engineering applications.


unsrtnat
  





























