Paper Title: Long-Term Forecasts of Failures in Wind Turbines.



On the temporal bandwidth of consecutive polariton amplification
    Mikhail Misko^1,∙, Anton D. Putintsev^1,∙, Denis Sannikov^1, Anton V. Zasedatelev^1, Ullrich Scherf^2, and Pavlos G. Lagoudakis^1
    August 5, 2024
=====================================================================================================================================




§ ABSTRACT 
We collect papers forecasting wind turbine failures at least two days in advance. We examine the prediction time, methods, failed components, and dataset size. 

We investigate the effect of using standard SCADA data and data from additional sensors, such as those measuring vibration.  

We observe a positive correlation between dataset size and prediction time. 

In the considered cases, one may roughly expect a forecast for at least two days using a dataset of one turbine year and a forecast for two hundred days from a dataset of a hundred turbine years. 



Keywords: long range forecast, degradation time, remaining useful lifetime.




§ INTRODUCTION


This study aims at systematizing approaches that predict failures in wind turbines at least two days in advance. 

This requirement is common among wind turbine operators, see, e.g., the wind energy challenge raised by Energias de Portugal <cit.>. 

This is in line with the fact that powerful wind turbines are big and complicated structures with high repair and downtime costs, and require skilled personnel and special equipment for maintenance. 

Therefore, it takes some time to decide which repair team is needed and which spare parts need to be found and delivered.

Moreover, the minimal required prediction time may further increase because there is a trend to have bigger and more powerful wind turbines <cit.>.  

Apart from the prediction time, the size of the available dataset may also influence the outcome. Some methods work better for small datasets, while others work for large datasets, and a minimal size of the dataset required for analysis may exist. 

Unfortunately, it is difficult to obtain and curate large datasets with a consistent and good quality <cit.>. 

Therefore, it is interesting to estimate to what extent the dataset size influences the prediction time of fault detection methods. 

We propose using turbine years (TY) to measure the size of a dataset. One obtains TY by multiplying the number of working turbines by the number of years they worked. The term TY is similar to labor hour, machine hour, and horsepower, and its purpose is the same – standardization and the possibility to compare. The term TY exists and is used, for example, to compare the size of datasets <cit.> or to quantify the harm of wind turbines to birds <cit.>, but we see a margin for increasing the popularity of this term.

The paper is organized as follows: Section <ref> describes the methods we used to find relevant papers. Section <ref> shows our results and findings, while Section <ref> concludes the paper. 





§ METHODOLOGY




The number of research articles on wind turbines is enormous and growing rapidly. A search for ”wind turbine” in Scopus in July 2024 gave us more than 100,000 indexed papers. Moreover, this number grows with acceleration. There were 5,000 papers indexed only in one year of 2013, while in 2023 this number of indexed papers was about 9,000. Limiting the search results only to the words ”wind” and ”turbine” found one after another with zero words in between (wind w/0 turbine), only to articles published in journals (no books and no proceedings), and only in the English language reduces the mentioned numbers to about 47,000, 1,600 and 4,800 correspondingly, with the most important factor being the proceedings exclusion. This abundance means that it is practically impossible even to read every article published within five recent years, and one must rely on some filtering. Therefore, we set our aim not to cover every paper but rather show popular solutions.

Our approach is twofold. On the one hand, we collected the papers from the literature that were relevant for a research project on fault prediction of wind turbines in the context of a specific case study with an open dataset, see Ref. <cit.>. These papers we found using Google Scholar. On the other hand, we used Scopus to make a thorough "scan" of articles published within the last five years that we found for a search phrase "wind w/0 turbine and SCADA and ", where SCADA means Supervisory Control and Data Acquisition system, which is the name for standard installed sensors, and  should be replaced with a term meaning a specific signal, for example, vibration. 

The use of Google Scholar is motivated by the fact that it is free of charge, counts citations well, thus showing approaches that are relevant to others, and is widely available outside academic institutions. 

The use of Scopus is motivated by the fact that it is a curated proprietary database that allows complicated searches, counts citations as well, and sorts by year even better. 

The choice of SCADA and a  was to limit the number of papers for deep reading and to check a popular hypothesis that modern techniques allow making good predictions using only SCADA data without dedicated and typically expensive additional sensors. The  measures that we used in the Scopus search include the mentioned "vibration" and also "strain", "torque", "LiDAR", and "acoustic". We also tried to search other signals, see Ref. <cit.> for a comprehensive list of possibilities. However, they give either too rare results in combination with SCADA, e.g. "thermography", or too many false positives as "electric".   

The most common is SCADA data with a 10-minute acquisition frequency. In principle, 5-, 1-minute, and other SCADA frequencies exist, but they are rare. The change from 10- to 1-minute frequency increases the amount of data 10 times but still does not allow us to monitor fast processes. 

A radically different amount of data appears if one considers vibrations because even a 1-minute frequency is only 1/(60 s) = 0.017 Hz, while vibration sensors measure values of the order of kHz. 

Therefore, we do not multiply the amount of more frequent SCADA data by a corresponding factor, but treat the SCADA data, for which vibration measurements are available, separately.

A turbine consists of many interrelated elements that form different groups. To classify failures into a small number of categories, we generalize compound terms based on our understanding. For example, we call a generator cooling failure discussed in Ref. <cit.> a generator failure because it addresses overheating of the generator, which has a different construction, sensors, and working modes than, e.g., a transformer, which also has a cooling. Another example is generator bearing and gearbox bearing failures discussed in Ref. <cit.>, which we classify as bearing failures because these failures are rooted in bearings. 

We analyze only the best predictions of the authors and disregard false alarms, failures that have not been found, absence of failure decision mechanism, validation or test, or both, and also accept the prediction if authors at least see a failure with their approach. The latter is particularly common for small datasets with TY<10. 

If several predictions are available, then we take an arithmetic mean. 

These are subjective choices. However, we provide citations to the discussed papers, and a reader can make his own classifications of the papers with long-term predictions that we found.







§ RESULTS




The main result of this work is the list of found articles shown in tables <ref> and <ref> with a corresponding list of acronyms in Appendix, and figure <ref>, which shows the relation between the size of a dataset and prediction time. Tables are sorted according to the prediction time. Prediction days and the number of TY larger than one are rounded to integers. 










We found 42 articles, that give forecasts for 2 days and more:


    
  * ≥ 2 and <10 days – 6 articles <cit.>,
    
  * ≥ 10 and <100 days – 24 articles <cit.>,
    
  * ≥ 100 days – 12 Articles <cit.>.
 


An overview gives Figure <ref>. One can see a positive correlation between dataset size and prediction time –  more data allows better prediction. In other words, long-term predictions are not possible for small datasets – the upper left corner is empty while having large datasets, one predicts at least for 10 days – the down right corner is empty. 

A large number of predictions for 10 TY indicates a freely available dataset that was analysed by many authors including us, see Ref. <cit.> for details. 

Excluding the unavailable regions and the anomaly at 10 TY, one can see that the points are distributed relatively homogeneously in the double-log scale. This means that the number of both large analyzed datasets and long-term predictions drops exponentially. This means that it is very hard to find large datasets and make long-term predictions. 

The approximate scaling is that having the number of turbine years 1≲ TY ≲ 100, one may expect[We have not studied the minimal required number of turbines or years to calculate the amount of TY. However, it seems reasonable to assume that one needs to have measurements from at least one turbine for at least a few months.] a forecast period of 2*TY days.

NBM is the group of methods that best follows the scaling. We assign a point to the group if authors build a model of normal behavior and then analyze the accumulated deviations from it. 

The group that does not follow the scaling is Vibration. The reason is that vibration gives access to fundamentally different information with much better time resolution than SCADA. Another reason is that we do not take into account the size of the vibration dataset, which is hard to estimate because authors often do not specify vibration frequencies and the time cuts they applied to reduce the amount of vibration data. 

The groups Specific and ML/Statistics are not strictly defined, and authors from these groups sometimes use the same or similar methods which may also overlap with NBM. However, we see a tendency to use advanced mathematical methods and sophisticated modeling for datasets with TY<10, while for larger datasets authors more often use the "out of the box" machine learning methods for classification and have enough data to apply filters.  

The examples are the usage of Wiener process <cit.> and Gaussian mixture copula model <cit.> for small datasets, while for large datasets typical methods include direct classification using isolation forest <cit.> or a perceptron <cit.> for differentiating between normal and abnormal states. Small datasets are also more often analyzed with special features designed by authors only for these datasets and not found in other works. These features are also called scores, factors or indicators, see the exact author's namings in the list of Acronyms.  

One may notice that the list of acronyms is larger than the number of papers and only a few acronyms repeat. This means that a comparison of the performance of different methods is difficult, while there are not enough statistics for the methods. 

It is hard to find more papers because of the variety of possible methods and their combinations. Authors also do not report their prediction time in a unified format that can be searched. The size of the dataset one has to calculate from the authors' notice of the start and end of measures, which is also not unified or even not mentioned, and one has to calculate it from their figures. Therefore, one can not systematically search by these parameters. 

Our systematic Scopus search for "wind w/0 turbine and SCADA and vibration" brought us mainly the articles that we already have seen using Google Scholar, with the exception of Ref. <cit.>. 

The Scopus search for other sensors brought us only false positives. 

We found that the keywords strain and torque are used to examine the main structural elements such as foundation and tower of the turbine. These elements are designed for much longer services than other components of a turbine. The typical time scale is 20 years and there is also a large safety margin. Therefore, corresponding fatigue life evaluations can predict 20, as well as 200 years remaining life <cit.>. 

The "LiDAR" as a searched signal gives articles with predictions no longer than 5 minutes, but clear clustering by amount of TY. Among 21 found articles, half analyze datasets with less than 1 TY and discuss mainly yaw and alignment, while the rest discuss mainly power reduction due to wake and aging. 



The "acoustic" gives articles with only instant measures without prediction and for typically very small datasets starting from 0.026 TY. This particular number of TY originates from 4 turbines and 2 hours of measurements every night in February <cit.>. The predicted value is not the condition of the turbine, but the noise it makes. 

Answering the question of whether SCADA and a dedicated sensor are always better than SCADA is cumbersome. 

First of all, there are no openly available datasets with SCADA and vibration data. Therefore, one has only a solution given by the analyzers of the dataset, which is not possible to check independently. The analyzers can be over-presenting positive results and under-publishing negative results. 

One can see from Fig. <ref> how rare the points with TY>10, in other words, how difficult it is to get a large dataset, especially with vibration data. The owners of such datasets may be interested in proving that vibration sensors are worth the money already paid for them. The authors also did not ask this question themselves and were just showing the result of the analysis without checking whether one can get the same prediction with SCADA only. 

However, we found one paper, which shows that both SCADA and vibration data give comparable results <cit.>. 
Our Figure <ref> also suggests that having vibration data can be beneficial, but with enough data and some effort, one can get similarly good results using only SCADA data.

One can advocate the usage of rated power and production year with arguments similar to those for using TY. 

Turbines of a similar power should have a similar construction, hence similar failure modes and rates. 

There could also be a dependence on the calendar year when a dataset was analyzed. 

The rapid growth of computing power, the development of machine learning algorithms, and growing interest in condition monitoring may also be important factors. A fashion for using certain models may exist and change with time. 

Authors of Ref. <cit.> observe strong fluctuations and kinds of waves in the distribution of publications per model and in keywords. 

We do not have enough data to check this hypothesis. However, we see that the most common rated power we find in papers with our search methods is 2 MW, while a few papers discuss turbines with other rated power in the range from 600kW to 4 MW. 

The notable effect is that the keywords vibration, torque, and strain allow finding papers talking about structural elements like towers and foundations of large offshore wind turbines with the rated power of the order of 10 MW, see, e.g., Ref. <cit.>. 







§ CONCLUSIONS





We found and classified papers that make long-term forecasts of wind turbine failures by prediction time, size of the dataset, failed component, and methods used for the solution. 

We see a positive correlation between the size of a dataset and prediction time with approximate linear scaling. With datasets of 1-100 TY, one may expect a 2-200 day forecast. 


We do not observe any grouping by failed components in the found articles with long-term predictions. However, we see that most studies consider short-term predictions, with the exception of those focused on structural elements like towers and foundations, which are forecast for many years. We note also that vibration is measured for rotating elements like gearbox, generator, and bearing, which represent a significant amount but not all turbine components. 

The methods used for the forecasts are very versatile. However, we see a tendency to use specific methods and advanced modeling for datasets smaller than 10 TY, and more machine learning and statistical approaches with filtering for larger datasets. 


We propose using TY to indicate the size of the used dataset for better search by this parameter and because of the development of different machine learning methods. Some methods require more data, while others work better on small datasets. Separation by TY allows looking at the models that work better for either of the cases, thus giving access to measuring performance and selection of the best suitable methods. 












§ APPENDIX: LIST OF ACRONYMS


2



ANFISadaptive neuro-fuzzy interference system
AInanomaly index
AOInanomaly operation index
ARauto regression
AEautoencoder
BagRbagging regression
BayCBayesian classifier
BayRBayesian regularization
bi-LSTMbidirectional LSTM
BGraphBond Graph
CombPcombination prediction
CNNconvolutional neural network
CUSUMcumulative sum
DI-CUSUMdecision interval CUSUM
DTdecision tree
dAEdeep autoencoder
DBNdeep belief network
DDPG-SOMdeep deterministic policy gradient-self-organizing map
DAEdenoising auto-encoder
DBSCANdensity-based spatial clustering of applications with noise
DegrVdegradation velocity
EWMAexponentially weighted moving average
XGBOOSTextreme gradient boosting
FIfeature importance
FVfeature vector
FLfuzzy logic
GRUgated recurrent unit
GMCgaussian mixture copula model
GMIglobal mahalanobis indicator
GBDTgradient boosting DT
GCA-MTSgraph convolutional autoencoder for multivariate time series
HInhealth index
HCFhigh correlation filter
ICAindependent component analysis
IEinformation entropy
IFisolation forest
kNNk nearest neighbours
KFKalman filter
KCPDkernel-based change-point detection
kMCk-means clustering
LASSOleast absolute shrinkage and selection operator
LARSleast-angle regression
LinRlinear regression
L-SVClinear supporct vector machine classifier
LoMSTlocal minimum spanning tree
LOFlocal outlier factor
LRIlocal residual indicator
LogRlogistic regression
LSTMlong short-term memory neural network
MDMahalanobis distance
MAmoving average
MLRmultivariate linear regression
MRmultivariate regression
MImutual information
NNneural network that does not have a specific name
NARXnonlinear autoregressive NN with exogenous inputs
PFparticle filtering
PPMCPearson prod-moment correlation
Physphysics mased model
PPSpower predictive score
PCAprincipal component analysis
QFquantile filtering
RBFNNradial basis function neural network
RFrandom forest
RNNrecurrent neural network
RBMrestricted Boltzmann machine
SVMsupport vector machine
SVRsupport vector regression
SMOTEsynthetic minority over-sampling technique
WHCWard hierarchical clustering
WTwavelet transform
WDWeibull distribution
WAweighted average
WPWiener process













 Paper Title: CXSimulator: A User Behavior Simulation using LLM Embeddings for Web-Marketing Campaign Assessment.







0009-0003-8870-8010

  CyberAgent, Inc.
  Tokyo
  Japan

kasuga_akira@cyberagent.co.jp


0000-0002-2724-6233

  CyberAgent, Inc.
  Tokyo
  Japan

yonetani_ryo@cyberagent.co.jp



e.g.
c.f.
i.e.
et al. 
etc

§ ABSTRACT 
This paper presents the Customer Experience (CX) Simulator, a novel framework designed to assess the effects of untested web-marketing campaigns through user behavior simulations. The proposed framework leverages large language models (LLMs) to represent various events in a user's behavioral history, such as viewing an item, applying a coupon, or purchasing an item, as semantic embedding vectors. We train a model to predict transitions between events from their LLM embeddings, which can even generalize to unseen events by learning from diverse training data. In web-marketing applications, we leverage this transition prediction model to simulate how users might react differently when new campaigns or products are presented to them. This allows us to eliminate the need for costly online testing and enhance the marketers' abilities to reveal insights. Our numerical evaluation and user study, utilizing BigQuery Public Datasets from the Google Merchandise Store, demonstrate the effectiveness of our framework. 



<ccs2012>
   <concept>
       <concept_id>10002951.10003227.10003241.10003243</concept_id>
       <concept_desc>Information systems Expert systems</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010341.10010370</concept_id>
       <concept_desc>Computing methodologies Simulation evaluation</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Simulation evaluation
[500]Information systems Expert systems



CXSimulator: A User Behavior Simulation using LLM Embeddings for Web-Marketing Campaign Assessment
    Ryo Yonetani
    August 5, 2024
==================================================================================================



§ INTRODUCTION



Sophisticated web marketing tools, such as Google Analytics <cit.> and Braze <cit.>, have been used to measure the behavior of users and assess marketing strategies. However, only fully developed web services can use these tools optimally. A/B testing, though commonly employed, is expensive to execute repeatedly and challenging to ensure a sufficient number of target users. Expert marketing knowledge and experience are necessary to draw useful insights from user behavior. As a consequence, it is not trivial to benefit from such tools in the early stages of service or in situations where there is a shortage of experts.

We focus on the potential of LLMs to solve this issue. LLMs have been applied not only for natural language processing tasks <cit.> but also for common sense reasoning in multi-modal data <cit.>. We believe that the ability of LLMs, especially to represent the high-level semantics of complex event descriptions with compact embedded vectors (, LLM embeddings) <cit.>, can also be advantageous for web marketing applications

In this work, we propose a novel framework named CXSimulator (Figure <ref>). CXSimulator can facilitate a fast offline assessment of the effects of marketing campaigns via LLM-based user behavior simulation, in the absence of costly A/B testing and expert knowledge. The proposed framework describes user behaviors on e-commerce sites with an event transition graph, where nodes represent individual events such as viewing an item and clicking a coupon, while edges indicate the transition from one event to another. In the graph, we embed the detailed description of each event into a semantic latent vector using LLMs, and train transition prediction models that use the LLM embeddings to predict connections and transition probabilities between events. The trained models can be employed to simulate plausible user behaviors, even when the graph includes a new node absent in the training data. This way, our approach can assess the effectiveness of new, untested campaigns by comparing simulated user behaviors with and without the nodes and measuring the difference in conversion rates.

We evaluate the effectiveness of the proposed CXSimulator using BigQuery Public Datasets <cit.> from GA360 <cit.>, which comprises real user behavior histories on the Google Merchandise Store. The experimental results reveal that our transition prediction models outperform relevant link prediction methods <cit.>, as well as other LLM-based solutions using GPT-3.5/4 <cit.>. Furthermore, our user study with five domain experts validates a high correlation between the CXSimulator's assessments and expert marketers' judgments regarding untested campaign effectiveness. The contributions of this paper are summarized as follows:



  * We present CXSimulator, a framework that models user behaviors on e-commerce sites using event transition graphs.

  * We demonstrate the effectiveness of using LLM embeddings to predict transition probabilities between events, which allows us to simulate user behaviors even in the presence of a new and untested campaign.

  * We evaluate the effectiveness of a new campaign between the proposed method and domain experts.





§ RELATED WORK



User Behavior Modeling and Simulation.
The recent growth in e-commerce has highlighted the importance of analyzing extensive user behavior logs <cit.>. User behavior modeling is employed in search <cit.> and recommendation systems <cit.>. Recent work has investigated methods that employ graph neural networks <cit.> and long short-term memory <cit.> to session-based user behavior histories. Moreover, LLM technologies have been advantageous for web content generation <cit.> and creative image generation <cit.>. However, online evaluation methods present challenges due to their high costs and scalability issues <cit.>. Conversely, there has recently been increasing interest in offline evaluation through simulation. Simulators have been used for evaluating conversational recommendation systems <cit.> and information retrieval methods <cit.>. Despite such developments, research has been limited in scope. This background motivates us to employ marketing campaign simulation to address this research gap.

Link Prediction, which aims to anticipate relationships between nodes, serves various purposes such as network reconstruction, recommendation systems, and knowledge graph completion <cit.>. In e-commerce applications, the primary focus is on predicting purchase and viewing relationships between users and items <cit.>. Nevertheless, studies on predicting transition probabilities for behavior transition graphs are scarce. Existing methods use Node2Vec <cit.> or graph auto-encoder <cit.> for link prediction <cit.>, but these approaches have limitations in representing knowledge outside of the training data. Recently, methods have been developed to directly predict links by fine-tuning node features and graph structures as prompts in knowledge graphs <cit.>, as well as techniques for direct link prediction in zero-shot scenarios <cit.>. Consequently, this paper leverages the potential of LLMs for common sense reasoning <cit.>, and employs LLM embeddings of user behavior for link prediction.


§ CXSIMULATOR FRAMEWORK



As illustrated in Figure <ref>, the CXSimulator incorporates three main components: a) user behavior modeling based on an event transition graph, b) prediction of transition probabilities through LLM embeddings, and c) user behavior simulation for marketing campaign assessment. Each of these components is detailed below.



 §.§ User Behavior Modeling

Consider users who are interacting with an e-commerce service. Their behavior history is represented as a sequence of events such as “viewing page A”, “obtaining coupon B”, “adding item C to the cart”, and “checking out the item”. A weighted directed graph naturally serves to represent the transitions among these events. Specifically, let G=(V, E) be a digraph, referred to as the event transition graph, where V denotes the nodes each corresponding to individual events. E=V × V indicates the edges that stand for transitions between events. Each edge carries a transition probability, p_i, j∈ (0, 1] for a pair of vertices v_i, v_j ∈ V, {i,j | 1,2,…,m}. The existence of edges and probabilities for linked nodes can be acquired from a corpus of user behavior data in the same segment such as cities and devices. Once this is achieved, a plausible user behavior history can be simulated by traversing the event transition graph according to these transition probabilities.



 §.§ Transition Probability Prediction

Our key question arises here: how can we simulate user behaviors if we add a new event to the event transition graph? Such situations would frequently occur in web-marketing scenarios, for example, when a new item is put on sale or a new discount campaign is launched. The effects of events, , whether these events contribute to the conversion rate, can vary based on user behaviors within a given segment. Ideally, we expect to simulate such effects before the items/campaigns are presented to real users. 

This situation can be viewed as a variant of link prediction problems <cit.>, where appropriate feature extraction for nodes is essential for high prediction performance. In this work, we propose to leverage the reasoning ability of LLMs to represent the semantic aspects of events and user segments with compact embeddings. Google Analytics provides detailed events in a well-structured format, such as . Likewise, some attributes of user segments are readily available in a similar format, such as . We transform such events and segment descriptions into LLM embeddings, and employ them as node features. 

Formally, we solve two-fold tasks: 1) binary classification for the existence of edges between a pair of nodes, and 2) regression for the transition probabilities of the edges when they exist. Let f_i, f_j, f_seg∈ F be the LLM embedding vectors from nodes v_i, v_j as well as the segment attribute, where F∈ℝ^d is the d-dimensional embedding space. We concatenate these vectors to use as the input to classifier 𝐶𝑙𝑠:F× F× F →{0, 1} and regressor 𝑅𝑒𝑔:F× F× F → [0, 1]. Note that filtering the input with the classifier before regressing the transition probabilities yields better performance than directly performing the regression for all possible pairs of nodes, regardless of the existence of edges. This is due to the wide variety of nodes, the sparsity of edges, and the resulting data imbalance.



 §.§ Simulating Campaign Effects

This section describes how to simulate user behavior for a new event and how to utilize these simulations to assess web-marketing campaigns. To achieve this, we consider two event transition graphs: the Control G_C=(V_C, E_C), where the edge connectivity and weights are derived solely from user behavior histories, and the Treatment G_T=(V_T, E_T), which has an expanded node and edge collection with a new event v_new. While it is simply V_T=V_C+{v_new}, we span edges between v_i∈ V_C and  v_new based on the output from the trained models of 𝐶𝑙𝑠(f_i, f_new, f_seg) and 𝐶𝑙𝑠(f_new, f_i, f_seg), and also estimate the transition probability with the trained 𝑅𝑒𝑔(f_i, f_new, f_seg) and 𝑅𝑒𝑔(f_new, f_i, f_seg) for the spanned bidirectional edges, where f_new is the LLM embedding vector on v_new.

Beginning from the same event node (, “session start”), we sample multiple event sequences from both G_C and G_T until we reach the ending node (, “session end”) or exceed the maximum sequence length. Each sequence signifies plausible user behavior with or without the intervention of the new event v_new. A collection of these sequences allows us to predict the potential difference in conversion rates using standard marketing analysis methods.


§ EXPERIMENT



We employed the BigQuery Public Datasets <cit.> from GA360 <cit.> to conduct numerical experiments on transition probability prediction and a user study for assessment of new campaigns.



 §.§ Numerical Evaluation

Datasets.
The dataset <cit.> includes e-commerce events (, click through of product lists, product detail views, add product(s) to cart, check out, and completed purchase) and detailed descriptions (, pageTitle, pagePath, productName, localProductPrice,…). It spans a total period of 366 days, from 2016/08/01 to 2017/08/01, amounting to 4,153,675 records. We designated the training, validation, and testing periods as 2016/08, 2016/09, and 2016/10, respectively, ensuring no seasonal differences in the data during these periods.[The limited granularity of event descriptions (due to ID mapping) prevented the utilization of other popular datasets, including CIKM2016EComm <cit.> and REES46 <cit.>.]

Implementation Details.
We implemented 𝐶𝑙𝑠 and 𝑅𝑒𝑔 using LightGBM <cit.>, a lightweight and fast model to ensure timely execution that preserves user experience. In the training data, the total number of nodes amounted to 1,350, with 19,126 linked edges out of total 1,822,500 edges. We applied Laplace smoothing α=5 to mitigate the zero frequency issue due to data sparsity. The class weight in Cls was considered based on edge sparsity and imbalance. The event description was vectorized to d=128 by Azure OpenAI Service  <cit.>.
The main parameters are, for 𝐶𝑙𝑠, {objective: binary,  scale_pos_weight: 5.0, num_iteration: 1500, metric: auc}, and for 𝑅𝑒𝑔, {objective: regression, num_iteration: 1500, metric: rmse}. Both models have the early_stopping_round parameter set to 50. We validated prediction accuracy for transition probabilities p_i,j of all node pairs v_i,v_j in the testing period. Furthermore, we compare accuracy using all combinations of text-embedding-3-small (v3-small) <cit.> and text-embedding-3-large (v3-large) <cit.>, as well as either 𝐶𝑙𝑠 or 𝑅𝑒𝑔 solely.

Baselines.
As baseline methods, we predict all transition probabilities p_i,j using Node2Vec <cit.> based on the graph structure, and graph auto-encoder (GAE) <cit.> that considers both the graph structure and node descriptions. With Node2Vec and GAE, an embedding vector of d=16 effectively described a node feature. 
Moreover, we adopted GPT-3.5/4 <cit.> via few/zero-shot <cit.> with prompts[The prompt begins as follows: Given a src node description and dst node description, your task is to predict transition probability(0.0-1.0) of the edge between 2 nodes.] consisting of detailed event description to directly predict p_i,j.

Evaluation Metrics.
Our evaluation metrics include: 
RMSE minimizes the error of p_i,j, a continuous value. 
SMAPE <cit.> evaluates relative error in probability regression p_i,j. 
F1-Score represents the harmonic mean of precision and recall. 
ROC-AUC measures a model's ability to correctly classify between positive and negative cases.
PR-AUC provides a more appropriate metric in situations with class imbalance than ROC-AUC.
We adjusted the decision threshold <cit.> in 𝑅𝑒𝑔 and predicted probability in 𝐶𝑙𝑠 to align with their objectives and evaluate all metrics consistently across all methods. The optimized decision threshold and predicted probability were also 0.1 in this experiment.

Results.
Table <ref> demonstrates that our models outperform the baseline methods. Using 𝐶𝑙𝑠 and 𝑅𝑒𝑔 with v3-small is deemed more practical due to cost efficiency, since the difference in metrics between v3-small and v3-large is not significant. The GPT-3.5/4 baseline showed limited performances because it could only receive 20-30 samples in a prompt.



 §.§ User Study

We conducted a user study with marketers to validate the practical utility of the CXSimulator in actual business environments.

Participants. We recruited five marketers where three are senior (over 3 years of experience) and two are junior ones (1-2 years of experience). They participated in the two studies described below.

Study 1: Validity of simulating the difference in conversion rates.
In this study, we assess the concordance between the difference in conversion rates simulated using the CXSimulator framework and the judgements of marketers. Each participant was asked to assess the effectiveness of 35 campaigns which have never been implemented in the store, for example, Enjoy 1 month Free of YouTube Premium for YouTube related Product!, by analyzing the GA data from 2016/08/01 to 08/31 on BigQuery within 40 minutes. The 35 campaigns consisted of 5 variations each in 7 categories, and comprehensiveness of the assessment is reasonable. Each campaign was scored on a four-point scale: effective (4), somewhat effective (3), not very effective (2), and not effective (1). For when it was difficult to judge, an option of unable to judge was also provided. On the other hand, we simulated 10,000 user sessions from each of G_C and G_T during the same period in our framework and measured conversion rates. Finally, we evaluate the correlation between the difference in conversion rates and human assessments.

Study 2: Usefulness of proposed simulator UI.
Furthermore, we developed a UI to experience the CXSimulator. As shown in Figure <ref>, our UI consists of five components: 1) data view and analysis, 2) event input forms (, campaignTitle, actionType), 3) Simulation run button, 4) Control/Treatment CVR comparison, 5) Control/Treatment user behavior data view and analysis.
Participants were asked to input their own campaigns, for example, {actionType: Click a campaign page, campaignTitle: Recommend the top 5 best-selling...}, and simulate the effect on the UI. After sufficient interactions with the UI, they evaluated planning efficiency (Eff), output data validity (Val), system operability (Ops), function expandability (Ext), and recommendation to another user (Rec) on a 10-point scale from No to Yes within 20 minutes. We also collected responses about the selection rationale through a questionnaire.




Results.
All the results are summarized in Table <ref>.
The correlation coefficient for evaluations from senior participants was 0.6220, while it was 0.3811 for junior participants. This indicates that the simulator's assessments could be more similar to those of senior marketers. In the questionnaire, the senior participants expressed difficulty in assessing campaigns within a restricted time frame, as they often relied on their intuition rather than conducting a thorough analysis. This reliance on intuition can explain why a higher correlation exists between the seniors and the simulator compared to the juniors. The overall evaluations on the usefulness of our system were also high, pointing to potential contributions to marketing work. Interestingly, junior marketers gave higher ratings to the system than the seniors. These results indicate the potential of our simulator to be a training tool.


 §.§ Limitations and Possible Extensions

While our system worked reasonably well, we found several failure cases through the user study. For example, the assessments for the campaign Rank up to silver member with an additional $30 purchase this month! were extremely high with 4,3,3,4,4 across all participants, while the simulator outputs a low conversion rate uplift with or without the campaign event.
The observed difficulty might stem from the absence of a real membership system in the store and in the data. This situation is outside the scope of the LLM's common-sense reasoning. Potential extensions to mitigate this problem can be fine-tuning of LLMs to obtain domain-specific contexts.  Moreover, one senior participant rated 4/10 on the validity of developed UI, due to unclear rationale for the conversion rate presented by the simulator. Enhancing interpretability by illustrating variations in statistical trends or extracting significant user behavior in a control and treatment group would be an interesting future direction.









§ CONCLUSION



In this work, we propose a novel framework, CXSimulator, designed for offline assessments of untested campaigns through user behavior simulations based on LLM embeddings. 
Our experimental results demonstrate the effectiveness of our transition probability prediction based on LLM embeddings compared to existing link prediction methods. We also confirm a correlation between the assessments from the CXSimulator and those from marketers, thereby indicating the practicability of the proposed framework in real-world marketing operations. In future research, we aim to examine the effectiveness of our approach within the domain and format constraints of private data from actual retail stores.




ACM-Reference-Format





 Paper Title: Decentralized and Uncoordinated Learning of Stable Matchings: A Game-Theoretic Approach.





Decentralized and Uncoordinated Learning of Stable Matchings: A Game-Theoretic Approach
    S. Rasoul Etesami† and R. Srikant

†Department of Industrial and Systems Engineering, Department of Electrical and Computer Engineering, and Coordinated Science Laboratory, University of Illinois Urbana-Champaign,  Urbana, IL 61801 (Email: etesami1@illinois.edu).
Department of Electrical and Computer Engineering and Coordinated Science Laboratory, University of Illinois Urbana-Champaign,  Urbana, IL 61801 (Email: rsrikant@illinois.edu).

    August 5, 2024
=============================================================================================================================================================================================================================================================================================================================================================================================================================================================


  







           
§ ABSTRACT 
We consider the problem of learning stable matchings in a fully decentralized and uncoordinated manner, where decentralized means that players make decisions individually without the influence of a central platform, and uncoordinated means that players do not need to synchronize their decisions using pre-specified rules. In this problem, there are n men and n women, each having preference over the other side. It is assumed that women know their preferences over men, but men are not aware of their preferences over women, and they only learn them if they propose and successfully get matched to women. A matching is called stable if no man and woman prefer each other over their current matches. When all the preferences are known a priori, the celebrated Deferred-Acceptance (DA) algorithm proposed by Gale and Shapley provides a decentralized and uncoordinated algorithm to obtain a stable matching. However, when the preferences are unknown, developing such an algorithm faces major challenges due to a lack of coordination. In this work, we achieve this goal by making a connection between stable matchings and learning Nash equilibria (NE) in noncooperative games. First, we provide a complete information game formulation for the stable matching problem with known preferences such that its set of pure NE coincides with the set of stable matchings, while its mixed NE with full support can be rounded in a decentralized manner to a stable matching. Relying on such a game-theoretic formulation, we show that for hierarchical structured markets, adopting the exponential weight (EXP) learning algorithm for the stable matching game achieves logarithmic regret with polynomial dependence on the number of players, thus answering a question posed in previous literature. Moreover, we show that the same EXP learning algorithm converges locally and exponentially fast to a stable matching in general matching markets. We complement this result by introducing another decentralized and uncoordinated learning algorithm that globally converges to a stable matching with arbitrarily high probability, leveraging the weak acyclicity property of the stable matching game. Since the global convergence of this algorithm can be slow in general matching markets, we investigate conditions under which stable matchings can be learned more quickly. In particular, for general matching markets, we show that if men receive stronger feedback when their proposals are rejected, it is possible to drive the market more rapidly toward an approximate stable matching. Our proposed game-theoretic framework bridges the discrete problem of learning stable matchings with the problem of learning NE in continuous-action games. 


Learning stable matchings, learning Nash equilibrium, weakly acyclic games, monotone games, decentralized learning, uncoordinated learning, online mirror descent, regret minimization. 


Stable matchings constitute one of the fundamental problems in computer science, economics, and engineering that have received considerable attention in the past decades. Stable matchings provide a desirable notion of stability in two-sided matching markets where the agents on each side of the market have preferences over the other side. For instance, in college admission, the applicants have different preferences over colleges and vice versa, and the goal is to match the applicants to the colleges so that no applicant-college pair would prefer to break their current matches and instead get matched to each other. Similar situations arise in other applications, such as kidney exchange programs, assigning jobs to workers, matching in online dating platforms, and scheduling jobs on heterogeneous machines. 

It is known that when all the preferences are known, stable matchings always exist and a simple decentralized and uncoordinated Deferred-Acceptance (DA) algorithm, first proposed by <cit.>, can find one such stable matches in polynomial number of iterations. Here, by a decentralized algorithm we refer to an algorithm in which players make decisions individually without the influence of a central platform, and uncoordinated means that players do not need to synchronize their decisions using pre-specified rules. While the DA algorithm provides a satisfactory solution to many practical applications, there are scenarios where either the information structure of the problem hinders the implementation of the DA algorithm or the stable matching achieved by the DA algorithm is inferior in terms of social optimality, especially when there exist many feasible stable matchings in the market. For instance, there are many recent emergence of online matching markets such as online labor markets (e.g., TaskRabbit, Upwork), online dating markets (e.g., Tinder, Match.com), and online crowdsourcing platforms (e.g., Amazon mechanical turk), where the users do not know their preferences a priori, and can repeatedly interact with the market to improve their matching quality <cit.>. The more a pair on both sides of the market gets matched, the more certain they will be about their preferences. Therefore, a question of interest is how the agents should interact with the market such that in the absence of any coordination, they will learn their preferences as quickly as possible, and their collective behavior results in a stable matching.

In this work, we consider a two-sided matching market posed as a marriage problem, where we refer to the agents of one side as men and to the agents of the other side as women. We assume both men and women have preferences on the other side and that women are aware of their ordinal preferences about men. However, men do not know their preferences about women and only learn those preferences if they propose and get matched, in which case the matched men receive a noisy estimate of their preferences. We assume that men/women cannot observe any other information (e.g., who is rejected/accepted), nor can they coordinate in any sense. The goal for men is to follow a decentralized and uncoordinated proposal strategy such that the entire market converges to a stable matching over time. 

One of the major challenges for learning a stable matching in markets with unknown preferences is the collisions. More precisely, when multiple men propose to the same woman, only one of them gets matched and receives useful information; all others get rejected and receive no information regarding their preferences for that woman. Therefore, resolving such collisions without coordination is a major issue. Moreover, even if a man gets matched to a woman, he receives a noisy utility drawn from an unknown distribution that could differ from his true preference. Thus, a collision-avoidance process should be repeated many times until men can accurately estimate their true preferences. One of our goals in this work is to provide decentralized and uncoordinated algorithms that can effectively learn the true preferences while incurring a minimum number of collisions. Moreover, our work provides an alternative approach for measuring the closeness of the generated dynamics to any stable matching in general matching markets and not necessarily to a specific stable matching in structured markets, which has been the main focus in the past literature. To that end, we make a connection between learning stable matchings in general markets with unknown preferences and learning Nash equilibrium (NE) in noncooperative games and leverage this connection to devise decentralized and uncoordinated algorithms for learning stable matchings.      



 §.§ Related Work


Stable matching was first introduced by <cit.> as a model for college admissions and to study the stability of marriages. Since then, there has been a tremendous effort on generalization and extension of stable matchings to various market settings <cit.>. In their seminal work, Gale and Shapley provided a simple DA algorithm in which men propose to their most preferred women, and the women reject all the received proposals except the most preferred one. They show that such a decentralized algorithm converges to a stable matching in at most O(n^2) steps, where n is the total number of men/women in the market. Unfortunately, when the agents' preferences are unknown, the DA algorithm is no longer guaranteed to converge to a stable matching without extra coordination. More precisely, if men fail to propose to their highest preferred women (due to a lack of knowledge of their true preferences), their subsequent proposals may become unpredictable. One reason is that if a man m is rejected by a woman w at a time instance t, that does not mean that m should stop proposing to w because it could be the case that another man mistakenly proposed to w at time t, hence causing the rejection of m by w. Therefore, in order to extend the DA algorithm to the case of unknown preferences, one requires some sort of coordination among the agents to avoid uninformative acceptance/rejection patterns. To address this issue, the earlier works <cit.> have proposed various coordination-based DA algorithms, where the main idea is to develop a phase-dependent process where in some phases the agents mostly explore, and in other phases, they implement the DA algorithm based on the current estimates of their unknown preferences.  


It is known that stable matchings with known preferences can be characterized using extreme points of a fractional polytope that is totally dual integral <cit.>. Moreover, stable matchings exhibit other nice properties that allow one to characterize their geometry using so-called rotations <cit.>. It was shown in <cit.> that uncoordinated random better response dynamics converge to a stable matching with probability one. Subsequently, <cit.> provided an exponential lower bound for the worst-case convergence time of the uncoordinated random better/best response dynamics to a stable matching. However, all these results are for the case when the market preferences are fully known so that the agents can compute their best/better responses and update their decisions accordingly. 

More recently, there has been a huge interest in matching markets with unknown preferences. However, depending on the information structure and the feedback received by the agents, one would expect a wide range of performance guarantees, which also depends on the type of algorithms followed by the agents (e.g., centralized/decentralized or coordinated/uncoordinated). The work <cit.> devised a randomized polynomial-time centralized algorithm for finding a stable matching with unknown deterministic preferences, where each time, the algorithm proposes a matching and receives feedback in terms of a blocking pair.[A unmatched pair of man-woman (m,w) is called blocking if both m and w prefer each other to their matched partner.] The work <cit.> considers learning a specific stable matching with unknown stochastic preferences by adopting a suitable notion of stable regret that measures the number of time instances that men propose to women other than their stable pair in that stable matching. <cit.> considered the problem of learning stable matchings in hierarchically structured markers[In fact, the hierarchical condition (a.k.a. sequential preference condition <cit.>) that we consider in Section <ref> is weaker than the α-reducible condition considered in   <cit.>, and our results hold for broader matching markets including the ones considered in <cit.>.] in which each submarket has a fixed pair (i.e., a pair of men-women that prefer each other the most in that submarket), and developed a phase-coordinated decentralized algorithm that achieves logarithmic regret with respect to the market's unique stable matching. However, the regret bound in <cit.> depends exponentially on other parameters such as the number of men/women. Furthermore, <cit.> considered learning stable matchings in an uncoordinated and decentralized fashion. However, they require stronger assumptions on the information feedback (e.g., a player observes the actions of other players in the previous round) and use a different performance metric than the one we consider in this work. We refer to <cit.> and <cit.> for other learning algorithms in two-sided matching markets with different performance guarantees. While all these works essentially address the same problem of learning a stable matching with unknown preferences, their algorithms differ substantially due to the information/feedback structure, the type of performance guarantee, and the amount of allowable coordination among the agents. 



 §.§ Contributions and Organization


We consider the problem of learning stable matchings in matching markets, where men do not know their preferences and want to achieve a stable matching in a decentralized and uncoordinated fashion. In particular, we consider the weakest type of feedback that men can receive: a man gets to observe a noisy version of his true preference only if his proposal gets accepted and receives no information otherwise. Our main objective is to provide a novel and principled approach to designing decentralized and uncoordinated learning algorithms for stable matchings through the lens of NE learning in noncooperative games. Our contributions can be summarized as follows:

    
  * We first provide a complete information continuous-action game formulation for the stable matching problem and show that its set of pure NE coincides with the set of stable matchings while its mixed NE points can be rounded in a decentralized way to a stable matching. This connection provides an alternative approach for measuring the closeness of the market dynamics to stable matchings through the lens of NE computation in games, hence extending the conventional notion of regret with respect to a specific stable matching to any arbitrary stable matching in general markets.
    
  * Leveraging this game-theoretic formulation, we provide a simple decentralized and uncoordinated algorithm EXP that converges globally to a stable matching in hierarchical markets and achieves logarithmic regret with only polynomial dependence on the number of players, hence improving the existing bounds in the past literature.
    
  * We then show that the EXP algorithm converges locally exponentially fast to a stable matching in general matching markets. In particular, we show that if EXP converges with positive probability, it converge globally to a stable matching.[Perhaps fast local convergence is the best one can hope for general markets because, for general matching markets, there are strong exponential lower bounds for the number of iterates of the better response dynamics to converge globally to a stable matching.]     
    
  * We also provide an alternative decentralized and uncoordinated algorithm that globally converges to a stable matching in general markets with arbitrarily high probability (albeit at a low convergence rate), thanks to the weakly acyclic property of the stable matching game. 
    
  * Finally, we address an information feedback design problem by seeking conditions in general matching markets that allow players to learn a stable matching quickly. In this regard, we leverage our game-theoretic framework to provide one such information feedback by establishing a connection to monotone games. Our results offer new insights into designing payoff functions that enable faster decentralized and uncoordinated algorithms for driving matching markets to their stable points. 




The paper is organized as follows. In Section <ref>, we formally introduce the problem. In Section <ref>, we provide a complete information game-theoretic formulation for the stable matching problem and prove several characterization results for its set of NE points. In Section <ref>, we provide a decentralized and uncoordinated algorithm (EXP) that achieves logarithmic regret for hierarchical markets with polynomial dependence on the number of players. In Section <ref>, we show that EXP  converges locally exponentially fast to a stable matching in general matching markets. In Section <ref>, we provide an alternative decentralized and uncoordinated algorithm that globally converges to a stable matching in general matching markets with arbitrarily high probability. In Section <ref>, we extend our results to matching markets with stronger feedback and establish a connection between fast global learning of a stable matching and learning a NE in monotone games. We conclude the paper by identifying some future research directions in Section <ref>.




§ PROBLEM FORMULATION




 §.§ Stable Matchings with Known Preferences


Here, we first introduce the stable matching problem with known preferences <cit.>. In this problem, there are a set M of men and a set W of women such that |M|=|W|=n (we use the term “agents" to refer to either men or women).[By introducing dummy agents with appropriate preferences, without loss of generality we may assume |M|=|W| <cit.>.] Each man m∈ M has a cardinal preference for each women w∈ W, denoted by μ_mw∈ (0, 1], such if μ_mw>μ_mw', it means that m prefers w over w'. Moreover, each woman w has an ordinal preference over the men, and we write m>_w m' if woman w strictly prefers m over m'. In this work we assume that no agent has ties in their preferences and we define 

    Δ=min_m, w≠ w'|μ_mw-μ_mw'|,    μ_min=min_m,wμ_mw,    μ_max=max_m,wμ_mw.

In particular, we note that Δ>0 and μ_min>0.

Given a matching and two matched pairs (m,w) and (m',w'), we say that (m,w') forms a blocking pair if μ_mw'>μ_mw and m>_w' m'. In other words, (m,w') is a blocking pair if both m and w' prefer each other over their current matches. A perfect matching is called a stable matching if it does not contain any blocking pair. 

 



 §.§ Stable Matchings with Unknown Preferences


In this work, we consider the problem of finding a stable matching with the main difference that men do not know their true preferences μ_mw, and they only get to learn them through interactions with the market. More precisely, for any m∈ M, w∈ W, we assume that the preference of man m about woman w is in the form of a [0,1]-supported unknown distribution 𝒟_mw with unknown mean μ_mw>0. We assume men and women interact in this market through a discrete-time process, where at any time t=0,1,2,… that a man m proposes to a woman w, he receives a feedback in the following form:

    
  * If the proposal of m gets rejected at time t because woman w has received an offer from a more preferred man m', i.e., m'>_w m, then m receives no information as feedback other than the fact that he was rejected by woman w at time t. 
    
  * If the proposal of m gets accepted by w at time t, then m observes an i.i.d. realization of his sampled preference drawn from 𝒟_mw, denoted by μ̂^t_mw.    


In the stable matching problem with unknown preferences, the agents' goals are to interact with the market through the information feedback structure described above such that the emerging dynamics converge to a stable matching of the market with known preferences {μ_mw}. We note that in the discrete-time process described above, men are the decision makers on whom to propose at each time t;  women merely respond to men's decisions by accepting their most preferred proposal (if they received any) and rejecting all others. 
       





§ A GAME-THEORETIC FORMULATION AND PRELIMINARY RESULTS


In this section, we provide a complete information noncooperative game formulation for the stable matching problem with known preferences whose set of pure Nash equilibrium (NE) points coincides with the set of stable matchings. Later, we will show how to leverage this formulation to extend our results for learning stable matchings with unknown preferences. Such a formulation has three main advantages: i) it reduces the learning task in matching markets to learning NE in continuous-action concave games, ii) it captures the selfish behavior of men (players) and the feedback they receive through their payoff functions, and iii) it simplifies the combinatorial structure inherent in learning stable matchings to learning NE in continuous-action games. While analyzing the agents' decisions directly on the discrete space of matchings seems complicated, which is why some earlier works need to restrict attention to special market structures <cit.>, our game-theoretic reduction simplifies this task to some extent, thanks to the extensive literature on NE learning in concave games.

Consider a complete information noncooperative game in which the action set of each man (player) is given by the set of women W, and the set of (mixed) strategies for each player is given by the probability simplex over the set of women W, i.e., the strategy set of player m is defined by 
    𝒳_m={x_m∈ℝ_+^|W|: ∑_w∈ Wx_mw= 1 }.
 
Moreover, given a strategy profile x=(x_m,x_-m), we define the payoff of player m by [Here, x_-m refers to the strategy vector of all the players except the mth one.]

    u_m(x)=∑_w∈ W(μ_mw∏_k>_w m(1-x_kw))x_mw,

where μ_mw>0 is the utility (true preference) received by man m if he gets matched to women w.


A strategy x_m for player m is called pure if it is zero in all coordinates except one. Otherwise, it is called a mixed strategy. A pure (mixed) strategy profile x^* is called a pure (mixed) NE if for any player m and any pure (mixed) strategy x_m, we have u_m(x_m,x^*_-m)≤ u_m(x^*_m,x^*_-m). 



A strategy x can be represented as a weighted bipartite graph ℋ(x)=(M∪ W, E(x)) between men and women. In this graph, the edges E(x)={(m,w): x_mw>0} are supported over the nonzero entries of x, with the weight of edge (m,w) given by x_mw.


One can interpret u_m(x) as the expected utility that player m would receive by successfully getting matched if each man independently proposes to women according to his mixed strategy distribution x_m. The reason for defining the payoff functions as in (<ref>) is that we want our devised algorithms to be implemented in a fully decentralized and uncoordinated manner among men by relying only on their received feedback (embedded into their payoff functions). While such payoff functions are highly nonlinear, they are essential to eliminate any degrees of coordination among the players. In other words, the cost of devising a fully coordination-free algorithm comes in analyzing more complex payoff functions with higher degrees of nonlinearity. We refer to Section <ref> for a simpler form of payoff functions with less degree of nonlinearities, which in turn requires stronger information feedback. The following are three properties of the payoff functions (<ref>). 


  (i) Player m's strategy does not have any impact on the coefficient terms μ_mw∏_k>_w m(1-x_kw), w∈ W. In particular, the gradient of u_m(·) with respect x_mw is given by 
    v_mw(x):=∇_mwu_m(x)=μ_mw∏_k>_w m(1-x_kw).
 

  (ii) The payoff of each player m is linear with respect to his own strategy x_m.

  (iii) For any fixed strategy of other players x_-m, player m always has a best response among pure strategies that is obtained by setting x_mw=1 for the woman w that achieves the maximum value v_mw(x), and x_mw=0 otherwise.    


In the remainder of this paper, we will refer to the above noncooperative game as the stable matching game and simply denote it by 𝒢=(M, {u_m}_m∈ M, {𝒳_m}_m∈ M). It is worth noting that although we are interested in obtaining a NE of the stable matching game, which is a complete information one-shot game, our goal is to learn one such NE by repeatedly playing the incomplete information game, in which the true preferences μ_mw (and hence the payoff functions) are not fully known. In fact, the set of pure/mixed NE points of the stable matching game has some interesting connections with the set of stable matchings with known preferences. The following theorem establishes one such result.  
 

A pure strategy profile x^*∈{0,1}^n^2 corresponds to the characteristic vector of a stable matching if and only if it is a pure NE for the stable matching game.


Suppose x^* is a pure NE. Then, we cannot have x^*_mw=x^*_m'w=1 for some woman w and m≠ m'. Otherwise, between m and m', the one that woman w prefers less receives zero utility, while he can always deviate to propose to a woman w' that does not have any offer and strictly improve his utility. Therefore, x^* must correspond to the characteristic vector of a perfect matching. To show that x^* is a stable matching, let us, by contradiction, assume that there exists a blocking pair with respect to the matching x^* and denote it by (m,w). Moreover, let m' and w' be the corresponding matches for w and m under x^*, that is x^*_mw'=1 and x^*_m'w=1. Since (m,w) is a blocking pair, we have μ_mw>μ_mw' and m>_w m'. The payoff of player m under NE x^* equals u_m(x^*)=μ_mw'. Now if player m changes its strategy to x_m:=(0,x_mw=1), his payoff becomes 

    u_m(x_m,x^*_-m)=∑_j∈ Wμ_mj∏_k>_j m(1-x^*_kj)x_mj=μ_mw∏_k>_w m(1-x^*_kj)=μ_mw,

where the second equality holds because m'<_wm so that the term 1-x^*_m'w does not appear in the product ∏_k>_w m(1-x^*_kw), and moreover x^*_kw=0 ∀ k≠ m', which implies ∏_k>_w m(1-x^*_kw)=1. This shows that u_m(x_m,x^*_-m)=μ_mw>μ_mw'=u_m(x^*), contradicting the fact that x^* is a NE.  

Conversely, let x^* be the characteristic vector of a stable matching, and by contradiction, assume it is not a NE. Then, there exists at least one player m who can deviate and strictly improve his payoff. In particular, we can consider his pure strategy best response move, which is guaranteed by (iii), and denote his most preferred woman by w. Moreover, let m' and w' be the corresponding matches for w and m under the stable matching x^*, i.e., x^*_m'w=x^*_mw'=1. The current payoff of m under the pure strategy x^* equals u_m(x^*)=μ_mw', while his payoff after such deviation becomes

    u_m(x_m,x^*_-m)=∑_j∈ Wμ_mj∏_k>_j m(1-x^*_kj)x_mj=μ_mw∏_k>_w m(1-x^*_kw)≤μ_mw.
 
Since 0< μ_mw'=u_m(x^*)<u_m(x_m,x^*_-m), we obtain μ_mw'< μ_mw. Moreover, from relation (<ref>) and the fact that u_m(x_m,x^*_-m)>0, we must have ∏_k>_w m(1-x^*_kw)=1, which implies m'<_w m. This together with μ_mw'< μ_mw shows that (m,w) is a blocking pair, contradicting that x^* is stable.              


While the above equilibrium characterization theorem concerns pure NE points, it is possible that the stable matching game admits a mixed NE. The following is one example. 

Let us consider an instance of the stable matching game with three men M={m_1,m_2,m_3} and three women W={w_1,w_2,w_3}. We assume the preferences of men and women are given by

    μ_m_1w_1=2, μ_m_1w_2=1, μ_m_1w_3=ϵ,            m_3<_w_1 m_1<_w_1 m_2,   μ_m_2w_1=3, μ_m_2w_2=4, μ_m_2w_3=5,            m_3<_w_2 m_2<_w_2 m_1,   μ_m_3w_1=ϵ, μ_m_3w_2=12, μ_m_3w_3=6,          m_1<_w_3 m_2<_w_3 m_3,

where ϵ>0 is a small positive constant. Then, the stable matching game has a mixed NE that is supported over seven edges comprising of two cycles of length 4 that share the common edge (m_2,w_2) (see, Remark <ref>). In particular, the seven positive entries of that mixed NE are given by 

    x_m_1w_1=μ_m_2w_1/μ_m_2w_2,              x_m_1w_2=1-μ_m_2w_1/μ_m_2w_2,   x_m_2w_1=1-μ_m_1w_2/μ_m_1w_1,          x_m_2w_2=1-μ_m_2w_2μ_m_3w_3/μ_m_2w_1μ_m_3w_2,       x_m_2w_3=μ_m_1w_2/μ_m_1w_1+μ_m_2w_2μ_m_3w_3/μ_m_2w_1μ_m_3w_2-1,   x_m_3w_2=μ_m_2w_1/μ_m_2w_3,               x_m_3w_3=1-μ_m_2w_1/μ_m_2w_3.
   


As Example <ref> suggests, mixed NE points may represent unpredictable bipartite graphs (Remark <ref>), and it is difficult to make general statements about their structure. However, the following theorem establishes an interesting connection between mixed and pure NE points, implying that obtaining a mixed NE with full support on the women's side is as satisfactory as obtaining a pure NE. 


Let x be any mixed NE in which each woman receives at least one (fractional) proposal.[If this condition does not hold, the theorem statement continues to hold for the induced submarket where each woman has received at least one fractional proposal.] For each man m, let w_m be the least preferred woman among those that he fractionally proposes to them, i.e., w_m=_w∈ W{μ_mw: x_mw>0}. Then {(m,w_m), m∈ M} forms a stable matching (and hence a pure NE) for the stable matching game. 


Please see Appendix <ref> for the proof. 



Theorem <ref> suggests some strategies for obtaining decentralized solutions. We can first design a decentralized algorithm (see Algorithm <ref> in conjunction with the preference estimation oracle of Algorithm <ref>) to learn a mixed NE of the stable matching game and acquire estimated preferences. Subsequently, each man can convert his fractional mixed strategy into a pure strategy in a decentralized manner by proposing only to his least (estimated) preferred woman among those he fractionally proposes to. This process ensures that the resulting pure strategy profile will be a pure NE with high probability, and the probability of error vanishes over time. Importantly, each man has complete access to his own estimated preferences, and the rounding process outlined in Theorem <ref> does not require knowledge of others' mixed strategies. Each man only needs to be aware of his own mixed strategy and estimated preferences. This approach offers a simple decentralized method to convert mixed NE points into stable matchings. 




§ LOGARITHMIC REGRET FOR HIERARCHICAL MARKETS






In this section, we consider the stable matching game with unknown preferences and develop a decentralized and uncoordinated algorithm for learning its NE points. In particular, we focus on matching markets with specific structures because without imposing any assumption on the matching market, there are exponential lower bounds for the number of iterations to find a stable matching through better response dynamics <cit.>. One example of such a structured matching market is the hierarchical matching market, where it was shown in <cit.> that if men follow a certain decentralized (but coordinated) algorithm, the men's expected regret (see Definition <ref>) is at most logarithmic in time but exponential in terms of other parameters such as the number of men n. In particular, the authors in <cit.> raised the question of how to remove the exponential dependence of the regret on n. In this section, we show that Algorithm <ref> achieves this goal whose regret is logarithmic in time with a polynomial factor in terms of other parameters. Moreover, Algorithm <ref> is uncoordinated and, hence, removes the phase-dependent coordination required by the algorithm in <cit.>. Throughout this section, we impose the following hierarchical assumption on the matching market.    


We assume that the sets of men and women can each be ordered as M={m_1,…,m_n} and W={w_1,…,w_n} such that any man and woman with the same rank prefer each other above any other partner with a lower rank; i.e., man m_k prefers woman w_k to women w_k+1, w_k+2,…,w_n, and woman w_k
prefers man m_k to men m_k+1, m_k+2,…,m_n. It is easy to see that under this hierarchical assumption, the matching {(m_k,w_k), k=1,…,n} is a stable matching. 
 


The condition imposed in Assumption <ref> is also known as the Sequential Preference Condition (SPC) <cit.> and is weaker than the α-reducible condition considered in <cit.>, which assumes that each submarket has a fixed-pair: A pair (m,w) is called a fixed pair if both m and w prefer each other the most among anyone else in that submarket.[In fact, one can show that a matching market has a unique stable matching if and only if it satisfies α-reducible condition <cit.>.] Since Assumption <ref> is weaker than the α-reducible condition, all our results immediately applies to α-reducible markets.



Let us assume that players follow a decentralized and uncoordinated algorithm 𝒜 that results in a sequence of proposals α^t=(α^t_m, m∈ M), t=1,…,T. The expected regret of 𝒜 equals

    R(T)=∑_t=1^T 𝔼[1_{∃ k∈ [n]: α^t_m_k≠ w_k}],

which is the expected number of times that the men's proposals do not form a stable matching. 






 §.§ Algorithm Design and Preliminaries


The algorithm that we propose is an adaptation of the online dual mirror descent for adversarial multi-arm bandit problem (see, e.g., EXP3 Algorithm in <cit.>) to the stable matching game, where for simplicity we refer to it as the EXP algorithm. In particular, it uses an entropy regularizer and adaptive stepsize for updating the players' mixed strategy distributions over time.

To describe the EXP algorithm, let us denote the gradient vector of player m by v_m(x)=∇_m u_m(x) such that for any woman w, we have v_mw(x)=μ_mw∏_k>_w m(1-x_kw). In particular, we note that u_m(x)=⟨ x_m, v_m(x)⟩. Therefore, when restricting to a pure strategy (action) profile α^t=(α_1^t,…,α^t_n)∈ W×⋯× W that is played at time t, the w-th coordinate of the gradient vector is given by

    v_mw^t:=μ_mw1_{α^t_k≠ w ∀ k>_w m},

where 1_{·} denotes the indicator function. On the other hand, by playing a pure strategy profile α^t, each man m receives a feedback only for the woman α^t_m that he has proposed to her at time t. In particular, if α_m^t=w, player m observes the payoff μ^t_mw1_{α^t_k≠ w  ∀ k>_w m}1_{α_m^t=w} at time t, where we recall that μ^t_mw is the realization of a random reward drawn independently from an unknown distribution 𝒟_mw with unknown mean μ_mw and bounded variance σ^2_mw∈ [0, 1]. 


During the course of the algorithm, at each time t, each player m holds a mixed strategy X^t_m∈𝒳_m and updates it whenever he receives new information. He then computes his adjusted mixed strategy X̂^t_m=(1-γ^t)X^t_m+γ^t/n1, where γ^t>0 is a mixing parameter and 1 is the vector of all ones. Using ideas from importance sampling for bandit problems <cit.>, and to obtain an unbiased estimator of the actual gradient vector, player m constructs an estimate vector v̂^t_m by normalizing the received feedback with his adjusted mixed strategy probabilities as

    v̂^t_mw=μ^t_mw1_{α^t_k≠ w ∀ k>_w m}/X̂^t_mw·1_{α_m^t=w},     w∈ W.

The reason why we use adjusted mixed strategies X̂^t with additional mixing parameter γ^t is to ensure that the mixed strategies remain bounded away from zero by a positive quantity. That makes the estimators (<ref>) have bounded variance (see Lemma <ref>), which allows us to use martingale concentration results to bound the probability of various events.

Let us denote the (random) score vector of player m at time t-1 by L̂^t-1_m=(L̂_mw,w∈ W). Player m uses L̂^t-1_m to compute his  mixed strategy vector X_m^t using the logit update rule (<ref>). He then proposes to a woman α_m^t chosen independently at random according to his adjusted mixed strategy X̂_m^t, and receives as feedback v̂_m^t with entries given by (<ref>). Player m then updates his score vector by L̂^t_m=L̂^t-1_m+v̂^t_m and proceeds to the next round. The detailed description of the EXP algorithm is summarized in Algorithm <ref>. 





In order to analyze the convergence behavior of Algorithm <ref> to a stable matching, let {ℱ^t-1}_t=1^∞ be the increasing filtration sequence that is adapted to the history of the random processes generated by Algorithm <ref>. More precisely, 
    ℱ^t-1={X^τ,L̂^τ,α^τ,v̂^τ, τ=0,1,…,t-1}∪{X^t},
 contains all the realized events up to time t-1 except the realization of pure strategies α^t and the rewards μ^t at time t. In particular, all relevant processes at time t-1 as well as X^t and X̂^t are ℱ^t-1-measurable, but α^t and v̂^t are not ℱ^t-1-measurable. 

We begin with the following lemma, which shows that the feedback v̂^t_m received by player m at time t is conditionally an unbiased and bounded estimator of the actual gradient vector v_m^t.



The received feedback is a conditionally unbiassed and bounded estimate of the gradients 

    𝔼[v̂_m^t|ℱ^t-1]=v_m(X̂^t)  ∀ m,      𝔼[(v̂_mw^t)^2|ℱ^t-1]≤σ^2/X̂^t_mw≤n/γ^t  ∀ m,w,

where σ^2:=max_m,w𝔼[(μ̂^t_mw)^2]∈ [0, 1].


Consider any pair (m,w). Since X̂^t_mw is ℱ^t-1-measurable and μ̂^t_mw is drawn independently from the algorithm's past decisions (and hence is independent of ℱ^t-1 and α^t), we have

    𝔼[v̂_mw^t|ℱ^t-1]   =𝔼[μ^t_mw]/X̂^t_mw·𝔼[1_{α^t_k≠ w  ∀ k>_w m}·1_{α_m^t=w}|ℱ^t-1]   =μ_mw/X̂^t_mwℙ{α_m^t=w|ℱ^t-1}∏_k>_w mℙ{α^t_k≠ w|ℱ^t-1}   =μ_mw∏_k>_w m(1-X̂^t_kw)=v_mw(X̂^t),

where the second equality holds because conditioned on ℱ^t-1, the random variables {α^t_k, ∀ k} are mutually independent as they are drawn independently from the given distributions {X̂^t_k, ∀ k}. Similarly,

    𝔼[(v̂_mw^t)^2|ℱ^t-1]   =𝔼[(μ̂^t_mw)^2]/X̂^t_mw∏_k>_w mℙ{α^t_k≠ w|ℱ^t-1}≤σ^2/X̂^t_mw≤n/γ^t,   ∀ m,w.
         






 §.§ Performance of Algorithm <ref> for Hierarchical Markets

 
Here, we analyze the performance of Algorithm <ref> for hierarchical matching markets. In the following analysis, we assume that a positive lower bound for c:=1/8min_k∈ [n]{Δ,μ_m_kw_k} is known to all the players so that they can choose their mixing parameters in Algorithm <ref>. However, as we discuss in Appendix <ref>, this condition can be relaxed up to some extent by using an uncoordinated and decentralized UCB gap estimation oracle in which the players use their estimated gaps to tune their mixing parameters. 
The main result of this section is presented in Theorem <ref>, which essentially shows that for hierarchical matching markets, Algorithm <ref> achieves an expected regret that is logarithmic in T and polynomial in terms of other parameters. However, before delving into the analysis, we first provide an overview of the main steps in the proof.


    
  * First, we define a “good" event Ω, which represents the set of circumstances where the difference between the accumulated realized rewards, η^t(L̂^t_mw̃ - L̂^t_mw), for any m, w, w̃, and t, remains close to its conditional expected value. In particular, Lemma <ref> shows that the event Ω occurs with very high probability. This allows us to analyze the performance of Algorithm <ref> under its conditional mean trajectory while incurring only a small, controllable loss. 
    
  * Conditioned on the event Ω, Lemma <ref> shows that, due to the market structure and payoff functions, as time progresses, more pairs in the market are matched according to the underlying hierarchical order with very high probability. Additionally, the size of such hierarchical nested matchings grows with very high probability after a polynomial number of steps. 
    
  * Finally, in Theorem <ref>, we combine the results of Lemma <ref> and Lemma <ref> to bound the expected regret of Algorithm <ref> by conditioning on the event Ω.  






Let us consider the stochastic dynamics of Algorithm <ref> that are given by

    X^t_mw=exp(η^t L̂^t-1_mw)/∑_w'exp(η^t L̂^t-1_mw'),   X̂^t_m=(1-γ^t)X^t_m+γ^t/n1,   L̂^t_mw=L̂^t-1_mw+v̂^t_mw.




Define Y^t_mw=∑_τ=1^t v_mw(X̂^τ), where v_mw(X̂^τ)=μ_mw∏_k>_w m(1-X̂^τ_mw), and consider the event

    Ω:={|η^t(L̂^t-1_mw̃-L̂^t-1_mw)-η^t(Y^t-1_mw̃-Y^t-1_mw)|≤ 2c√(t),  ∀ t≥ 1, ∀ m, w,w̃},
 
where c=1/8min_k∈ [n]{Δ,μ_m_kw_k} is a constant parameter. The following lemma provides a high probability bound for the event Ω to occur under appropriate choice of stepsize and mixing parameter. 


Fix an arbitrary δ∈ (0, 1), and suppose each player follows Algorithm <ref> with stepsize η^t=1/√(t) and mixing parameter γ^t=Mlog t/t, where M=4n/clog1/δ. Then, ℙ{Ω}≥ 1-δ. 


Fix an arbitrary man m. For any w, w̃, as in (<ref>), we can write

    η^t(L̂^t-1_mw̃-L̂^t-1_mw)   =η^t∑_τ=1^t-1(v_mw̃(X̂^τ)-v_mw(X̂^τ))+η^t S^t-1_mw̃-η^t S^t-1_mw   =η^t(Y^t-1_mw̃-Y^t-1_mw)+η^t S^t-1_mw̃-η^t S^t-1_mw,

where S^t-1_mw:=∑_τ=1^t-1(v̂^τ_mw-v_mw(X̂^τ)) ∀ m,w. According to Lemma <ref>, {v̂^τ_mw-v_mw(X̂^τ)} is a martingale difference sequence that satisfies 

    |v̂^τ_mw-v_mw(X̂^τ)|≤n/γ^τ,    𝔼[(v̂^τ_mw-v_mw(X̂^τ))^2|ℱ^τ-1]≤n/γ^τ.

By taking u=n/γ^t-1, v^2=∑_τ=1^t-1n/γ^τ, and x=ct in Lemma <ref> (see Appendix <ref>) and noting that the event {V^t≤ v^2} holds with probability 1, we obtain 

    ℙ{η^t|S_mw^t-1|≥ c√(t)}=ℙ{|S_mw^t-1|≥ ct}≤ 2exp(-Mc^2 t^2/2n(∑_τ=1^t-1τ/logτ+ct(t-1)/log (t-1))).

Since f(τ):=τ/logτ is a concave function for τ≥ 100, using Jensen's inequality, we have

    1/t-1∑_τ=1^t-1τ/logτ=∑_τ=1^t-11/t-1f(τ)≤ f(∑_τ=1^t-11/t-1τ)=∑_τ=1^t-1τ/t-1/log(∑_τ=1^t-1τ/t-1) =t/2logt/2.

Therefore, ∑_τ=1^t-1τ/logτ≤t(t-1)/2logt/2≤t^2/log t. Substituting this relation and t(t-1)/log (t-1)≤t^2/log t into (<ref>) we get

    ℙ{η^t|S_mw^t-1|≥ c√(t)}≤ 2exp(-Mc^2 t^2/2n(1+c)t^2/log t)= 2exp(-Mc^2log t/2n(1+c))≤2δ/(nπ t)^2,

where the last inequality is obtained by choosing M sufficiently large (e.g., M=4n/clog1/δ). Thus, using the union bound, we can write

    ℙ{η^t |S_mw^t-1|< c√(t) ∀ t≥ 1, ∀ m, w}≥ 1-∑_t=1^∞2n^2δ/(nπ t)^2> 1-δ.

Finally, using (<ref>) and (<ref>), one can see that

    ℙ{Ω}   =ℙ{|η^t(L̂^t-1_mw̃-L̂^t-1_mw)-η^t(Y^t-1_mw̃-Y^t-1_mw)|≤ 2c√(t),  ∀ t≥ 1, ∀ m, w, w̃}   =ℙ{|η^t S^t-1_mw̃-η^t S^t-1_mw|≤ c√(t),  ∀ t≥ 1, ∀ m, w, w̃}   ≥ℙ{η^t |S_mw^t-1|< c√(t) ∀ t≥ 1, ∀ m, w}> 1-δ.
 



Before we present the main result of this section, we first prove the following key lemma. 


Assume that each man follows Algorithm <ref> with stepsize η^t=1/√(t) and mixing parameter γ^t=Mlog t/t, where M=4n/clog1/δ. Let a_1≥…≥ a_n be a sequence defined by a_1=Δ/2 and a_k=1/4min_i∈ [k]{Δ,μ_m_iw_i}, ∀ k=2,…,n. Conditioned on the event Ω, there exist t_1≤…≤ t_n=Õ(n^2/c^2log1/δ), such that for any t≥ t_k and any w≠ w_k, we have 

    X̂^t_m_kw_k≥1-γ^t/1+(n-1)e^-η^t a_k t+γ^t/n,    X̂^t_m_kw≤(1-γ^t)e^-η^t a_k t/1+(n-1)e^-η^t a_k t+γ^t/n.
    


We use induction on k∈ [n] to prove the lemma. For the base of the induction k=1, we note that for any X̂∈ [0,1]^n^2, we have

    v_m_1w_1(X̂)=μ_m_1w_1∏_k>_w_1 m_1(1-X̂_kw_1)=μ_m_1w_1,   v_m_1w(X̂)=μ_m_1w∏_k>_w m_1(1-X̂_kw)≤μ_m_1w.
 
Therefore, 

    Y^t-1_m_1w_1-Y^t-1_m_1w   =∑_τ=1^t-1(v_m_1w_1(X̂^τ)-v_m_1w(X̂^τ))   ≥∑_τ=1^t-1(μ_m_1w_1-μ_m_1w)≥Δ (t-1).

Define a_1=Δ/2 and let t_1=4. Conditioned on the event Ω, for any t≥ t_1 and any w≠ w_1, we have

    X^t_m_1w_1/X^t_m_1w   =e^η^t(L̂^t-1_m_1w_1-L̂^t-1_m_1w)≥ e^η^t (Y^t-1_m_1w_1-Y^t-1_m_1w)- 2c√(t)   ≥ e^η^t Δ (t-1)-2c√(t)≥ e^η^t a_1 (2t-2)-1/2η^t a_1 t   = e^η^t a_1 (3/2t-2)≥ e^η^t a_1t,

where the third inequality holds because c≤a_1/4, and the last inequality holds because t≥ t_1. Moreover, 

    1=X^t_m_1w_1+∑_w≠ w_1X^t_m_1w≤(1+(n-1)e^-η^t a_1 t)X^t_m_1w_1.

Combining relations (<ref>) and (<ref>) and using the fact that X̂^t_mw=(1-γ^t)X^t_mw+γ^t/n, for any t≥ t_1 and w≠ w_1, we have

    X̂^t_m_1w_1≥1-γ^t/1+(n-1)e^-η^t a_1 t+γ^t/n,        X̂^t_m_1w≤(1-γ^t)e^-η^t a_1 t/1+(n-1)e^-η^t a_1 t+γ^t/n.
 



Next, by induction hypothesis, suppose that the statement holds for the first k-1 pairs (m_ℓ,w_ℓ), ℓ=1,…,k-1 with corresponding sequences a_1≥ a_2…≥ a_k-1 and t_1≤ t_2≤…≤ t_k-1, and let us consider the kth pair (m_k,w_k) of the market. Notice that for any X̂∈ [0,1]^n^2, we have 

    v_m_kw_k(X̂)=μ_m_kw_k∏_m_ℓ>_w_k m_k(1-X̂_m_ℓw_k)≥μ_m_kw_k∏_ℓ=1^k-1 (1-X̂_m_ℓw_k).

We consider two cases: 

Case I: If w=w_i for some i∈{1,…,k-1}, then

    v_m_kw(X̂)=v_m_kw_i(X̂)=μ_m_kw_i∏_m_ℓ>_w_i m_k(1-X̂_m_ℓw_i)≤μ_m_kw_i(1-X̂_m_iw_i).

Therefore, using (<ref>) and (<ref>) we can write

    Y^t-1_m_kw_k   -Y^t-1_m_kw=∑_τ=1^t-1(v_m_kw_k(X̂^τ)-v_m_kw_i(X̂^τ))   ≥∑_τ=1^t-1(μ_m_kw_k∏_ℓ=1^k-1(1-X̂^τ_m_ℓw_k)-μ_m_kw_i(1-X̂^τ_m_iw_i))   ≥∑_τ=1^t-1(μ_m_kw_k(1-∑_ℓ=1^k-1(γ^t/n+(1-γ^t)e^-η^t a_ℓ t/1+(n-1)e^-η^t a_ℓ t))-μ_m_kw_i(1-γ^t/n-1-γ^t/1+(n-1)e^-η^t a_i t))    ≥∑_τ=1^t-1(μ_m_kw_k-γ^t-∑_ℓ=1^k-1(1-γ^t)e^-η^t a_ℓ t/1+(n-1)e^-η^t a_ℓ t-μ_m_kw_i(n-1)e^-η^t a_i t+γ^t/1+(n-1)e^-η^t a_i t)   ≥∑_τ=1^t-1(μ_m_kw_k-γ^t-∑_ℓ=1^k-1e^-η^t a_ℓ t-μ_m_kw_i(n-1)e^-η^t a_i t-μ_m_kw_iγ^t)   ≥∑_τ=1^t-1(μ_m_kw_k-2γ^t-2(n-1)e^-η^t a_k-1 t),

where the second inequality holds by the induction hypothesis for t≥ t_k-1, and the last inequality holds because a_1≥…≥ a_k-1. Thus, for any time step t≥ t_k-1 that satisfies e^-η^t a_k-1 t≤γ^t and t-2/∑_τ=1^t-1γ^τ≥4n/μ_m_kw_k, we have 

    Y^t-1_m_kw_k-Y^t-1_m_kw   ≥∑_τ=1^t-1(μ_m_kw_k-2nγ^t)= μ_m_kw_k(t-1)-2n∑_τ=1^t-1γ^t≥μ_m_kw_k/2t.


Case II: If w∉{w_1,…,w_k}, then μ_m_kw_k≥μ_m_kw+Δ. Using (<ref>) and the fact that μ_m_kw≥ v_m_kw(X̂) for any X̂∈ [0,1]^n^2, we can write

    Y^t-1_m_kw_k-Y^t-1_m_kw   =∑_τ=1^t-1(v_m_kw_k(X̂^τ)-v_m_kw(X̂^τ))   ≥∑_τ=1^t-1(μ_m_kw_k∏_ℓ=1^k-1(1-X̂^τ_m_ℓw_k)-μ_m_kw)   ≥∑_τ=1^t-1(μ_m_kw_k(1-∑_ℓ=1^k-1X̂^τ_m_ℓw_k)-μ_m_kw)   ≥∑_τ=1^t-1((μ_m_kw_k-μ_m_kw)-∑_ℓ=1^k-1(γ^t/n+(1-γ^t)e^-η^t a_ℓ t/1+(n-1)e^-η^t a_ℓ t))   ≥∑_τ=1^t-1(Δ-γ^t-∑_ℓ=1^k-1e^-η^t a_ℓ t)   ≥∑_τ=1^t-1(Δ-γ^t-(n-1)e^-η^t a_k-1 t).

Thus, for any integer t≥ t_k-1 that satisfies e^-η^t a_k-1 t≤γ^t and t-2/∑_τ=1^t-1γ^τ≥2n/Δ, we have 

    Y^t-1_m_kw_k-Y^t-1_m_kw≥∑_τ=1^t-1(Δ-nγ^t)=Δ(t-1)-n∑_τ=1^t-1γ^τ≥Δ/2t.

As a result, by taking

    t_k:=min{t≥ t_k-1|  tη^t/ln(1/γ^t)≥1/a_k-1,  t-2/∑_τ=1^t-1γ^τ≥max{4n/μ_m_kw_k, 2n/Δ}},   a_k:=1/4min_i∈ [k]{Δ, μ_m_iw_i},

we have a_k-1≥ a_k, t_k≥ t_k-1. In particular, for t≥ t_k both relations (<ref>) and (<ref>) hold and we have 

    Y^t-1_m_kw_k-Y^t-1_m_kw≥min{Δ/2, μ_m_kw_k/2} t≥ 2a_k t, ∀ t≥ t_k, ∀ w≠ w_k.
 
Now, using a similar argument as in the base case, for any t≥ t_k and w≠ w_k, we have

    X^t_m_kw_k/X^t_m_kw=e^η^t(L̂^t-1_m_kw_k-L̂^t-1_m_kw)≥ e^η^t (Y^t-1_m_kw_k-Y^t-1_m_kw)- 2c√(t)≥ e^η^t 2a_k t-2c√(t)≥ e^η^t 2a_k t-η^t a_k t= e^η^t a_kt,

where the second inequality uses (<ref>), and the last inequality holds because c=min_k∈ [n]{Δ/8,μ_m_kw_k/8}≤a_k/2. The above relation, together with similar arguments as in deriving (<ref>) and (<ref>) in which a_1 is replaced by a_k, gives us the desired result (<ref>). 

Finally, using the stepsize and mixing parameters η^t=1/√(t) and γ^t=Mlog t/t in relation (<ref>), one can obtain an explicit upper bound for t_n as

    t_n≤min{t∈ℤ_+|  √(t)/log(t/Mlog t)≥1/a_n-1,  t-2/Mlog^2 t≥max{4n/μ_m_nw_n, 2n/Δ}},

for which it suffices to take t_n=Õ(n^2/c^2log1/δ), where Õ(·) hides the logarithmic factors.




Assume that each man follows Algorithm <ref> with η^t=1/√(t) and γ^t=Mlog t/t, where M=4n/clog T and c=1/8min_k∈ [n]{Δ,μ_m_kw_k}. Then, the expected regret of Algorithm <ref> in hierarchical matching markets is at most R(T)=O(n^2/clog^3 T).  


Conditioned on the event Ω, which according to Lemma <ref> it occurs with probability at least 1-δ, the expected regret can be upper-bounded as 

    𝔼[R(T)|Ω]   = ∑_t=1^T 𝔼[1_{∃ k: α^t_m_k≠ w_k}|Ω]≤∑_t=1^T∑_k=1^n 𝔼[1_{α^t_m_k≠ w_k}|Ω]   ≤∑_t=1^T∑_k=1^n 𝔼[𝔼[1_{α^t_m_k≠ w_k}|X̂^t_m_k]|Ω]=∑_t=1^T∑_k=1^n 𝔼[ℙ{α^t_m_k≠ w_k|X̂^t_m_k}|Ω]   =∑_t=1^T∑_k=1^n 𝔼[1-X̂^t_m_kw_k|Ω]≤ nt_n+∑_t=t_n+1^T∑_k=1^n 𝔼[1-X̂^t_m_kw_k|Ω]   ≤Õ(n^3/c^2log1/δ)+∑_t=t_n+1^T∑_k=1^n (1-γ^t/n-1-γ^t/1+(n-1)e^-η^t a_k t)   ≤Õ(n^3/c^2log1/δ)+∑_t=t_n+1^T∑_k=1^n(n-1)e^-η^t a_k t+γ_t/1+(n-1)e^-η^t a_k t   ≤Õ(n^3/c^2log1/δ)+n^2∑_t=1^T e^-η^t a_n t+n∑_t=1^Tγ_t   ≤Õ(n^3/c^2log1/δ)+2n^2/a_n^2+nMlog^2 T   =O(n^3/c^2log1/δ+n^2/clog1/δlog^2 T),

where the third inequality uses Lemma <ref>, the fifth inequality uses the fact that a_n:=min_k∈ [n]{a_k}, and the last inequality is obtained by choosing γ^t=Mlog t/t, η^t=1/√(t), M=4n/clog1/δ, and noting that c≤ a_n.

Finally, since δ∈ (0,1) can be chosen arbitrary, by choosing δ=1/T, we can write

    𝔼[R(T)]   =ℙ{Ω}𝔼[R(T)|Ω]+ℙ{Ω̅}𝔼[R(T)|Ω̅]   ≤ (1-δ)O(n^3/c^2log1/δ+n^2/clog1/δlog^2 T)+δ n^2T    =O(n^2/clog^3 T).



It is worth noting that the regret of Algorithm <ref> is logarithmic in T, with a polynomial factor n^2/c in terms of the problem parameters, thereby addressing a question raised in <cit.>. Furthermore, Algorithm <ref> is both decentralized and uncoordinated, overcoming the limitations of the algorithm proposed in <cit.>, which is notably more complex and requires phase-dependent coordination among men according to the Deferred-Acceptance algorithm by Gale and Shapley. 






§ EXPONENTIAL LOCAL CONVERGENCE FOR GENERAL MATCHING MARKETS
 

In this section, we consider learning a stable matching in general matching markets (i.e., without any hierarchical assumption) and show that the same decentralized and uncoordinated algorithm EXP (Algorithm <ref>) converges locally at an exponential rate to a stable matching when players' strategies get sufficiently close to one of the stable matchings. To that end, we first consider the following technical lemma, which shows that if a strategy profile is sufficiently close to a stable matching, then the reward of choosing an action according to that stable matching is strictly the best decision.    

Let X^* be a pure NE of the stable matching game and set c=1/8min{Δ,μ_min}. For any strategy profile x that satisfies x-X^*_1≤c/μ_max, we have v_mw^*(x)-v_mw(x)>c ∀ w≠ w^*, where w^* is the woman that m is matched to her under the pure NE X^*. 


Please see Appendix <ref> for the proof.


Next, we consider the following lemma that will be used to capture the local behavior of the dynamics of Algorithm <ref> around a pure NE.  


Suppose X^* is a pure NE and {X^t} be the sequence of iterates generated by Algorithm <ref>. Moreover, let A^t-1= ∑_τ=1^t-1(v_mw^*(X^τ)-v_mw(X^τ)), where w^* is the woman that man m is matched to her under the pure NE X^*. Then, the following statements hold:


  * i) If X^t-X^*_1≤c/250n^2, then η^tL̂^t-1_mw^*-η^tL̂^t-1_mw≥ln (n^2/2c)+6  ∀ m,w≠ w^*.

  * ii) If η^tL̂^t-1_mw^*-η^tL̂^t-1_mw≥ln(n^2/2c) ∀ m,w≠ w^*, then X^t-X^*_1≤ c.

  * iii) Let t_0 be the first time such that 1/η^t_0+1-1/η^t_0≤ c/(ln(n^2/2c)+3). If for some t≥ t_0 we have η^t A^t-1≥ln(n^2/2c)+3 and v_mw^*(X^t)-v_mw(X^t)≥ c ∀ m,w≠ w^*, then η^t+1 A^t≥ln(n^2/2c)+3.
 


Please see Appendix <ref> for the proof. 


Now we are ready to state the main result of this section. 

Fix an arbitrary δ∈ (0, 1), and suppose each player follows Algorithm <ref> with some mixing sequence {γ^t} and decreasing stepsize sequence {η^t} that satisfie the following conditions:

    (∑_τ=1^t-1γ^t-1/γ^τ)^-1≤η^t≤min{(2n^2∑_τ=1^t-1γ^τ)^-1, (8nln(nπ/√(δ)t)∑_τ=1^t-11/γ^τ)^-1/2}.

Let t_0 be the first time such that 1/η^t_0+1-1/η^t_0≤ c/(ln(n^2/2c)+3) and X^t_0-X^*≤c/250n^2. Then with probability at least 1-δ, the sequence {X^t} converges to the pure NE X^* exponentially fast, i.e.,

    ℙ{X^t+1-X^*_1≤ 41 nexp(-ctη^t+1) ∀ t≥ t_0 | X^t_0-X^*_1≤c/250n^2}≥ 1-δ.
  


Fix an arbitrary man m, and assume X^*_mw^*=1. For any w≠ w^*, we have

    η^tL̂^t-1_mw^*   -η^tL̂^t-1_mw=η^t∑_τ=1^t-1(v̂^τ_mw^*-v̂^τ_mw)   =η^t∑_τ=1^t-1(v_mw^*(X̂^τ)-v_mw(X̂^τ))+η^t∑_τ=1^t-1(v̂^τ_mw^*-v_mw^*(X̂^τ))-η^t∑_τ=1^t-1(v̂^τ_mw-v_mw(X̂^τ))   =η^t∑_τ=1^t-1(v_mw^*(X̂^τ)-v_mw(X̂^τ))+η^t S^t-1_mw^*-η^t S^t-1_mw,
 
where S^t-1_mw:=∑_τ=1^t-1(v̂^τ_mw-v_mw(X̂^τ)). Moreover, for any m and w, using the mean-value theorem, there exists a θ=λX̂^τ+(1-λ)X^τ such that 

    |v_mw(X̂^τ)-v_mw(X^τ)|   =|∇ v_mw(θ)· (X̂^τ-X^τ)|   ≤∇ v_mw(θ)_∞X̂^τ-X^τ_1   ≤ n X̂^τ-X^τ_1   =nγ^τ∑_mX_m^τ-1/n1_1 ≤ n^2γ^τ.

Substituting this relation into (<ref>) and using the triangle inequality, we can write

    |η^tL̂^t-1_mw^*-η^tL̂^t-1_mw-η^t∑_τ=1^t-1(v_mw^*(X^τ)-v_mw(X^τ))|   ≤η^t |S^t-1_mw^*|+η^t |S^t-1_mw|+2n^2η^t∑_τ=1^t-1γ^τ   ≤η^t |S^t-1_mw^*|+η^t |S^t-1_mw|+1,

where the last inequality follows from the upper bound on the choice of stepsize η^t. 

Next, notice that according to Lemma <ref>, {v̂^τ_mw-v_mw(X̂^τ)} is a martingale difference sequence that almost surely satisfies 

    |v̂^τ_mw-v_mw(X̂^τ)|≤n/γ^τ,   𝔼[(v̂^τ_mw-v_mw(X̂^τ))^2|ℱ^τ-1]≤𝔼[(v̂_mw^τ)^2|ℱ^τ-1]≤n/γ^τ.

Therefore, by taking u=n/γ^t-1, v^2=∑_τ=1^t-1n/γ^τ, and x=1/η^t in Lemma <ref>, we obtain

    ℙ{η^t |S_mw^t-1|≥ 1}≤ 2exp(-(1/η^t)^2/2n(∑_τ=1^t-11/γ^τ+1/η^tγ^t-1))≤ 2exp(-(1/η^t)^2)/4n∑_τ=1^t-11/γ^τ),
 
where the second inequality holds because using the lower bound on η^t we have 1/η^tγ^t-1≤∑_τ=1^t-11/γ^τ. Since by the choice of stepsize (1/η^t)^2/∑_τ=1^t-1 1/γ^τ≥ 8nln(nπ/√(δ)t), we get ℙ{η^t |S_mw^t-1|≥ 1}≤2δ/(nπ t)^2. In particular, using the union bound, the probability that η^t |S_mw^t-1|< 1 for any t≥ 1 and any m, w, is at least

    ℙ{η^t |S_mw^t-1|< 1 ∀ t≥ 1, ∀ m, w}≥ 1-∑_t=1^∞2n^2δ/(nπ t)^2> 1-δ.

Let us define A^t-1= ∑_τ=1^t-1(v_mw^*(X^τ)-v_mw(X^τ)). Using (<ref>) and (<ref>), with probability at least 1-δ, we have

    |η^tL̂^t-1_mw^*-η^tL̂^t-1_mw-η^tA^t-1|≤ 3,  ∀ t≥ 1, ∀ m, w.
    

Next, let us assume X^t-X^*_1∈c/250n^2. Then, for any m, w≠ w^*, we have v_mw^*(X^t)-v_mw(X^t)≥ c and η^tL̂^t-1_mw^*-η^tL̂^t-1_mw≥ln(n^2/2c)+6, by Lemma <ref> and Lemma <ref> (part i), respectively. Using relation (<ref>), it means that η^tA^t-1≥ln(n^2/2c)+3, and thus by Lemma <ref> (part iii), we have η^t+1 A^t≥ln(n^2/2c)+3. This in view of Lemma <ref> (part ii) shows that X^t+1-X^*_1≤ c. Similarly, Lemma <ref> implies v_mw^*(X^t+1)-v_mw(X^t+1)≥ c ∀ m,w≠ w^*, which in view of Lemma <ref> (part iii) shows that η^t+2 A^t+1≥ln(n^2/2c)+3. Hence, using relation (<ref>), η^t+2L̂^t+1_mw^*-η^t+2L̂^t+1_mw≥ln(n^2/2c) ∀ m,w≠ w^*, and thus X^t+2-X^*_1≤ c by Lemma <ref> (part ii). 

The above inductive argument shows that if X^t_0-X^*≤2c/n^2 for some t_0≥ 1 such that 1/η^t_0+1-1/η^t_0≤ c/(ln(n^2/2c)+3), then with probability at least 1-δ, we have X^t-X^*≤ c ∀ t≥ t_0, and in particular, using Lemma <ref>, v_mw^*(X^t)-v_mw(X^t)≥ c ∀ m,w, t≥ t_0 . Therefore, with probability at least 1-δ, for any m, w≠ w^*, t≥ t_0, we have 

    X^t+1_mw^*/X^t+1_mw=exp(η^t+1L̂^t_mw^*-η^t+1L̂^t_mw)≥exp(η^t+1A^t-3)≥exp(ctη^t+1-3).

Therefore, X^t+1_m-X^*_m_1=2(1-X^t+1_mw^*)≤ 2nexp(-ctη^t+1+3), which shows that

    ℙ{X^t+1-X^*_1≤ 2nexp(-ctη^t+1+3) ∀ t≥ t_0| X^t_0-X^*_1≤c/250n^2}≥ 1-δ.




    In the statement of Theorem <ref>, the condition that t≥ t_0  is not restrictive because if Algorithm <ref> converges to a pure NE, then eventually for some t_0, we have 1/η^t_0+1-1/η^t_0≤ c/(ln (n^2/2c)+3) and  X^t_0-X^*≤c/250n^2. In particular, t_0 is a constant that scales polynomially in terms of other parameters (see, e.g., Remark <ref>). We note that it is possible to further relax this condition by a constant scaling of η^t, i.e., by setting η^t←1/c(ln (n^2/2c)+3)η^t and rescaling all other parameters such as the mixing sequence γ^t accordingly. However, that requires the players to know c=1/8min{Δ,μ_min} to choose their stepsize η^t, while knowing c is not required in the stepsize condition given in (<ref>).    



In fact, there exist feasible choices for η^t and γ^t that satisfy the stepsize condition (<ref>) in Theorem <ref>. For instance, if for some α,β∈ (0,1),  α> max{1-β, 1+β/2}, we choose 

    η^t=Θ(1/ln( t/√(δ))t^α),      γ^t=Θ(1/t^β),

then condition (<ref>) is satisfied for any t_0≥ (α b/c)^1/(1-α), where b=ln(n^2/2c)+3. More specifically, by choosing γ^t=1/t^1/3 and η^t=1/t^3/4, the exponential convergence rate of Theorem <ref> would be in the order of exp(-ct^1/4) for any t_0≥ (b/c)^4. We note that condition (<ref>) provides only a sufficient condition to achieve local exponential convergence and we did not attempt to fully characterize all feasible choices of stepsize/mixing parameters. However, there could be other choices of stepsize that do not satisfy condition (<ref>) while still achieving local exponential convergence.


Although Theorem <ref> shows that Algorithm <ref> achieves an exponential convergence rate, its convergence guarantee holds in a local sense, i.e., when the dynamics get sufficiently close to a neighborhood of a stable matching.
In the following theorem, we complement this result by showing that if the dynamics of Algorithm <ref> converge with positive probability, it must converge to a mixed NE of the stable matching game. 


Under the same stepsize assumptions of Theorem <ref>, if Algorithm <ref> converges to a limit point X^* with positive probability, then X^* must be a (possibly mixed) NE of the matching game. 


Please see Appendix <ref> for the proof.    







§ GLOBAL LEARNING OF STABLE MATCHINGS IN GENERAL MARKETS


In the previous section, we showed that the decentralized and uncoordinated EXP algorithm converges locally to a stable matching at an exponential rate in general matching markets. This prompts the question of whether there is an uncoordinated and decentralized algorithm capable of globally learning a stable matching in general markets, regardless of the convergence rate. In this section, we answer this question affirmatively by introducing an alternative algorithm that leverages the weak acyclic property of the stable matching game. To that end, we first present several useful properties of the stable matching game, which will be instrumental in establishing our main convergence results. 



 
A pure strategy profile x is a good state if all the men matched under x are at their best responses.[In a pure strategy profile x=(x_m, m∈ M), a matched man is the one whose pure strategy x_m has exactly one coordinate equal to 1 (say wth coordinate), and all other coordinates are zero, and no other man of higher preference proposes to w, i.e., x_kw=0 ∀ k>_w m.] In other words, a good state x refers to a partial matching in which the matched men are at their best responses.
 


The following theorem shows that if the stable matching game with known preferences starts from a good state, then any sequence of best response dynamics by the players reaches a pure NE. Unless stated otherwise, we assume that each player m is aware of both his own true preferences and those of all the women, enabling him to compute his payoff function. This assumption will be relaxed in Theorem <ref>, where players seek to learn these unknown preferences through interactions with others. 


Consider the stable matching game that starts from a good initial state x^0. Consider any sequence of updates where at each time t, an arbitrary subset of players that includes at least one unsatisfied player updates their actions by playing their best responses. Then, the sequence of updates converges to a pure NE in no more than |M||W| iterations.


Let us encode the ordinal preferences of each woman w into cardinal values λ_wm∈ℝ^+ such that m_1>_w m_2 if and only if λ_wm_1>λ_wm_2. Given a  pure strategy profile x, let r_w(x)∈{λ_wm, m∈ M} be woman w's reward for her most preferred man who is matched to her under x. By convention, we set r_w(x)=0 if w is not matched to any man under the pure-strategy profile x. We show that r_w(x) is nondecreasing for each woman w (and for at least one w it is strictly increasing) such that the potential function Φ(x)=∑_wr_w(x) strictly increases after each iteration. Since each r_w(x) can increase at most |M| times, the sequence must terminate at a pure NE after no more than |M||W| iterations. We show this by arguing that during the sequence of updates, no matched man m wants to deviate unless he is rejected by his partner w because she has received a better offer, implying that r_w(x) must have increased. Note that this statement holds initially because x^0 is a good state where no matched man wants to deviate.    

More precisely, let us by contradiction assume that t is the first iteration at which some woman w loses her more preferred man during the sequence of updates, i.e., r_w(x^t)<r_w(x^t-1). This can happen only if man m that is matched to w at time t-1 (i.e., x^t-1_mw=1) is selected to update at time t whose best response is to leave w for another women w' (i.e., x^t_mw'=1). Otherwise, r_w(x^t-1) can only increase or remain the same because women reject any proposal other than their most preferred one. Note that since the game starts from a good state x^0, we must have t≥ 2.  

According to the payoff structures, the best response of player m at time t is to propose to his most preferred achievable woman w', i.e., among all women who rank m higher or equal to their match under x^t-1, m proposes to the woman w' which brings him the highest utility μ_mw'. Note that in particular μ_mw'>μ_mw. Thus, the first time τ≤ t-1 when man m proposed and got matched to woman w, the woman w' was his more preferred choice and the reason why m did not propose to w' at time τ was because w' was not achievable, i.e., x^τ-1_m'w'=1 for some man m'>_w' m. Since w' was not achievable for m in x^τ-1 but she is achievable in x^t-1 implies that r_w'(x^t-1)<r_w'(x^τ-1). Therefore, r_w'(x^τ')<r_w'(x^τ'-1) for some τ≤τ'≤ t-1, contradicting the choice of t. 



In fact, one can view the DA algorithm of <cit.> as a special version of the above lemma in which all the players are selected to best respond at each round, and the initial good state is the empty matching. Lemma <ref> shows that the stable matching game admits an ordinal potential function if the game starts from a good state (we refer to Appendix <ref> for another interpretation of this potential function). However, that does not imply that the stable matching game is an ordinal potential game <cit.>, which requires the existence of a potential function whose changes are aligned with players' payoff changes from any initial state. Nonetheless, the existence of such a “weak” potential function allows us to show the weak acyclicity of the stable matching game as defined next.


A better response path in a finite action noncoopertaive game is a sequence of pure strategy profiles x^1,x^2,…,x^L such that for each ℓ=1,…,L-1, x^ℓ+1 is obtained from x^ℓ by letting some player i_ℓ to play a better response. A game is called weakly acyclic if from any pure strategy profile x^0, there is a finite length better response path to a pure NE.



A pure NE x^*=(x^*_m,x^*_-m) of the stable matching game is called strict if each player has a unique best response at the equilibrium x^*, i.e., for every m and any pure strategy x_m≠ x^*_m, we have u_m(x_m,x^*_-m)<u_m(x^*_m,x^*_-m).   


Using the above definitions, we are now ready to state the main result of this section, which provides a decentralized and uncoordinated algorithm for learning stable matchings in general markets with unknown preferences. The algorithm is an adaptation of the sample experimentation algorithm for weakly acyclic games <cit.> to the stable matching game. Intuitively, each player m goes through a sequence of phase-dependent exploration/exploitation. At the end of each episode, player m evaluates his payoff for each action he has taken during the last episode. He then updates his baseline action (an action that brings him the highest reward in the past episode up to some tolerance) and sticks to that baseline action most of the time in the next episode while still exploring new actions with a small probability. The detailed description of the algorithm is summarized in Algorithm <ref>.   






Let p,ω∈ (0,1) be arbitrary probabilities and ϵ≤min{1-p/n, δ/4n, Δ-δ/4n}, where δ∈ (0, Δ). For each man m, there exists a sufficiently large episode length τ_m=τ_m(p,ϵ), such that if each player m follows Algorithm <ref> with episode length τ_m, for all sufficiently large times t>0, the pure strategy profile α^t will be a (fixed) stable matching with probability at least p.
   

First, we show that the stable matching game is a weakly acyclic game. If the game starts from a good state, by Lemma <ref>, any sequence of best responses by the players is a better response path that leads to a pure NE in |M||W| iterations. Thus, we only need to show that from any initial state, there is a better response path to a good state. This is also true by sequentially letting a matched man who wants to deviate (if there is such a man) to play his best response. In this way, either the number of matched men decreases due to collisions (which can happen at most |M| times), or between every two consecutive collisions, there could be at most |M| best response updates. Therefore, |M|^2 steps are enough to bring us from any arbitrary state to a good state, and the overall number of best response iterates to obtain a pure NE starting from any initial state is at most |M||N|+|M|^2.



Next, we note that any pure NE in the stable matching game is a strict NE.[In a weakly acyclic game, an arbitrarily high probability convergence can be guaranteed to a strict NE <cit.>. If we do not have strictness property, the dynamics may oscillate between two nonstrict NE points with a positive probability.] This follows directly by the fact that there are no ties between men's preferences in the stable matching game. More precisely, let α^* be a pure NE (stable matching) and assume u_m(α^*)=μ_mw. For any α_m≠α^*_m, either u_m(α_m,α^*_-m) equals 0 or μ_mw' for some w'≠ w. Because α^*_m is the best response of player m at the NE and there are no ties between man m's preferences, we must have μ_mw'<μ_mw. Since 0<μ_mw, in either case we have u_m(α_m,α^*_-m)<u_m(α^*_m,α^*_-m).

Finally, it was shown in <cit.> that the class of sample experimentation dynamics in weakly acyclic games converges with arbitrarily high probability to a strict NE if for any joint baseline action b^s at the start of an episode s, u^s_mw is an arbitrarily close estimation of u_m(w,b^s_-m) for sufficiently large episode length τ_m=τ_m(p,ϵ). Since Algorithm <ref> is an adaptation of the sample experimentation dynamics to the stable matching game, the convergence result follows from <cit.> if we can show that for any δ^*=min{δ/4, Δ-δ/4}>0, p<1, and 1-(1-ϵ)^n-1<δ^*/2, for sufficiently large τ_m we have[This is an equivalent version of Claim 3.4 in <cit.> to the stable matching game.]  

    ℙ{|u_mw^s-u_m(w,b^s_-m)|≥δ^*}≤ 1-p.

This statement is also true because during each episode, players sample their baseline actions (or explore a new action) independently of others, and moreover, μ^t_mw are i.i.d. realizations of a random variable with mean μ_mw. Thus, relation (<ref>) holds using the law of large numbers, which completes the proof.   


While Algorithm <ref> provides a decentralized and uncoordinated algorithm for learning stable matchings in general markets with arbitrarily high probability, the convergence time to a NE may take a long time. This seems inevitable to some extent in view of the exponential lower bound in terms of n such that the best response dynamics in general matching markets converge to a stable matching <cit.>. 






§ MATCHING MARKETS WITH STRONGER INFORMATION FEEDBACK


So far, we have assumed that men receive the minimum amount of information feedback as they interact with the market; namely, men only get to observe whether their proposal gets rejected or, if matched, they observe a random realization of their preference. In this final section, we consider the learning problem from a new perspective by addressing the following question: Is there stronger information feedback that will allow us to derive the market faster globally to a stable matching in a decentralized and uncoordinated fashion?  Therefore, we address the problem of learning 
a stable matching from an information feedback design perspective by showing that if the players are allowed to receive a bit more information as feedback, then the stable matching game will become a monotone game, hence allowing a wide range of decentralized and uncoordinated learning algorithms for obtaining its NE points. 


Let us consider a simplified version of the stable matching game with the same set of players M and strategy spaces 𝒳_m, m∈ M, except that the payoff of player m is now given by   

    u_m(x)=∑_w∈ Wμ_mw(1-∑_k>_w mx_kw)x_mw.




The simplified payoff function (<ref>) can be viewed as the expected perceived reward for player m if he randomly and independently of others proposes to women according to his mixed strategy x_m∈𝒳_m but under slightly stronger feedback: if player m's proposal gets rejected from a woman w, he also gets to know how many men higher ranked than him proposed to w. In this way, each man m learns how many more preferred men compete with him to get matched to woman w. Such information feedback can be implemented by using a waiting list order for each woman, which shows the ranking of a rejected man m in that list. Henceforth, we refer to such information feedback as the waiting list feedback. 

Let α_k∈ W denote the random woman that player k proposes to her, which is drawn independently from his mixed strategy distribution x_k. Under the waiting list feedback information, player m has access to the information ∑_k>_w m1_{α_k=w} if he proposes to w, and hence can estimate his perceived payoff by proposing to a woman w as μ_mw(1-∑_k>_w m1_{α_k=w}),  Therefore, the expected perceived payoff of player m under the waiting list feedback equals

    𝔼[∑_w∈ Wμ_mw(1-∑_k>_w m1_{α_k=w})1_{α_m=w}]   =∑_w∈ W𝔼[μ_mw(1-∑_k>_w m1_{α_k=w})|α_m=w]ℙ(α_m=w)   =∑_w∈ Wμ_mw(1-∑_k>_w mx_kw)x_mw=u_m(x),
    
where the second equality holds by independency of α_k ∀ k∈ M. 

The motivation for waiting list feedback and the payoff function (<ref>) is that if we view 1-∑_k>_w mx_kw as the remaining fractional capacity of women w for being matched, using the approximation 1-∑_k>_w mx_kw∼ e^-∑_k>_w mx_kw, one can think that man m loses his interest to offer women w exponentially in terms of the occupied capacity of woman w by higher preferred men. One immediate consequence of such simplified payoff functions is the following proposition whose proof follows nearly verbatim arguments as those in Theorems <ref> and <ref>, and is omitted from here for the sake of brevity.


Consider the stable matching game with payoff functions (<ref>). Then, x^* is a pure NE if and only if it is the characteristic vector of a stable matching. Moreover, any mixed NE of this game with full support can be rounded in a decentralized manner to a pure NE.  



Next, we consider the following definition of monotone games <cit.>. 

A continuous-action game (M,{U_m}_m∈ M, {𝒳}_m∈ M) is called monotone if for any two action profiles x,x'∈𝒳_1×⋯×𝒳_|M|, we have (x'-x)^T(F(x')-F(x))≤ 0, where 
    F(x)=(∇_m U_m(x), m∈ M)^T.
  


An important property of monotone games is that they admit a variety of decentralized learning algorithms that converge globally to a NE, which, under further assumptions, it can be shown that the converge rate could be exponential or polynomial <cit.>. The following theorem shows that the regularized stable matching game with waiting list feedback is indeed a monotone game under a suitable assumption on the preferences. 
 


Consider the regularized stable matching game with list feedback, where each player aims to maximize its regularized payoff function given by U_m(x_m,x_-m):=u_m(x_m,x_-m)-β/2x_m^2. Then, for any two strategy profiles x',x, we have

    (x'-x)^T(F(x')-F(x))=-1/2∑_w∈ W(x_w'-x_w)^TQ^w(x_w'-x_w),
 
where x_w is the vector x_w=(x_mw, m∈ M)^T, and Q^w is a symmetric matrix whose diagonal entries are 2β and all the entries in row m up to the diagonal entry in that row equal μ_mw.  In particular, if β>nμ_max/2, the regularized stable matching game with waiting list feedback is a monotone game.  


Note that due to the linearity of U_m(x) with respect to player m's strategy x_m, we have 

    ∇_mw U_m(x)=μ_mw(1-∑_k>_w mx_kw)-β x_mw, ∀ m,w,

where ∇_mw U_m(x) denotes the gradient of U_m(x) with respect to x_mw. Thus, we can write

    (x'   -x)^T(F(x')-F(x))=∑_m (x_m'-x_m)^T (∇_m U_m(x')-∇_m U_m(x))   =∑_m∑_w (x_mw'-x_mw)(∇_mw U_m(x')-∇_mw U_m(x))   =∑_m∑_w -μ_mw(x_mw'-x_mw)(∑_k>_w mx'_kw-∑_k>_w mx_kw+β(x'_mw-x_mw))   =-∑_m∑_w∑_k>_w mμ_mw(x_mw'-x_mw)(x'_kw-x_kw)-β∑_m∑_w (x'_mw-x_mw)^2   =-∑_w(∑_m∑_k>_w mμ_mw(x_mw'-x_mw)(x'_kw-x_kw))-β∑_m∑_w (x'_mw-x_mw)^2   =-∑_w(∑_m∑_kμ_mw1_{k>_w m}(x_mw'-x_mw)(x'_kw-x_kw))-β∑_w∑_m (x'_mw-x_mw)^2.

For each woman w, let us define a vector x_w=(x_mw, m∈ [M])^T, and an n× n incidence preference matrix A^w whose entries are given by A^w_mk=μ_mw1_{k>_w m}, ∀ m,k∈ [n]. Then, using (<ref>) we have

    (x'-x)^T(F(x')-F(x))   =-∑_w(∑_m∑_k A^w_mk(x_mw'-x_mw)(x'_kw-x_kw))       -∑_w (x'_w-x_w)^T(β I)(x'_w-x_w)   =-∑_w(x_w'-x_w)^T(A^w+β I) (x_w'-x_w)   =-1/2∑_w(x_w'-x_w)^TQ^w(x_w'-x_w),

where Q^w=(A^w)^T+A^w+2β I. Next, we note that permuting the rows and columns of a matrix by the same permutation matrix preserves the semi-definiteness property as it only changes the basis of its eigenvectors. Thus, without loss of generality, we may assume 1>_w 2>_w …>_w n. However, in that case, we may assume that for any w, the matrix A^w will be a lower triangular matrix whose diagonal entries are all zero. In particular, Q^w=(A^w)^T+A^w+2β I would be a symmetric matrix whose elements in each row m up to the diagonal entry are identical and equal to μ_mw. Finally, for sufficiently large β>nμ_max/2, the matrix Q^w is positive semidefinite for all w, and we have -(x_w'-x_w)^TQ^w(x_w'-x_w)≤ 0  ∀ w. This shows that (x'-x)^T(F(x')-F(x))≤ 0, and completes the proof.  



The monotonicity property in Theorem <ref> holds when the regularizer parameter β is sufficiently large. The regularizer implies that players tend to maximize their approximate payoff functions. Thus, instead of learning a NE for the payoff functions (<ref>), they learn an ϵ-NE where the approximation error ϵ grows as β increases. As a result, applying decentralized learning algorithms to learn a NE of the regularized game only guarantees an ϵ-stable matching in the sense that the players will achieve a solution in which each player would be ϵ worse off than what he could achieve in a stable matching. Therefore, the regularizer parameter trades off between the market's convergence speed and the accuracy of the learned stable matching.  



Finally, we note that without imposing any assumption on the agents' preferences, one cannot expect the stable matching game to be a monotone game. The reason is that monotone games imply the global uniqueness of a NE <cit.>. However, by Proposition <ref>, any stable matching is a pure NE, and for general matching markets, there could be many stable matchings under arbitrary preferences. Moreover, the assumption of waiting list feedback is not necessary for faster convergence in any general market; it is merely one choice that we consider in this work, and there could be other types of information feedback that allow faster convergence. Nevertheless, our game-theoretic approach to the stable matching problem provides valuable insights on how to properly design the utilities and the information feedback to devise faster decentralized and uncoordinated algorithms for learning stable matchings.   









§ CONCLUSIONS


In this work, we considered the problem of learning stable matchings with unknown preferences in a fully decentralized and uncoordinated manner. Using a game-theoretic formulation, we showed how to leverage the rich literature for learning NE in noncooperative games to derive the matching market to its stable points in a principled way. In particular, we established several global and local convergence results with performance guarantees for learning stable matchings in hierarchical markets, general markets, and under weak/strong information feedback. Our results provide new insights for learning stable matchings through NE learning in noncooperative games and bridge the discrete problem of learning a stable matching with learning NE in continuous-action games.  

This work opens several avenues for future research. One interesting direction is to characterize an information-theoretic limit below and above which efficient learning of a stable matching in a fully decentralized and uncoordinated fashion may or may not be possible. Moreover, our waiting list feedback is merely an initial attempt to design faster decentralized and uncoordinated algorithms under richer information feedback. Extending our results to include other types of information feedback and studying their effect on the convergence speed of the market to its stable points would be another interesting research direction.      





IEEEtran








§ SUPPLEMENTARY MATERIALS





 §.§ Freedman's Concentration 
Inequality for Martingales
 

Let {ζ_τ}_τ=1^t be a martingale difference sequence with max_τ∈ [t]|ζ_τ|≤ u, and consider the corresponding martingale S^t=∑_τ=1^tζ_τ with S^1=0. Let V^t=∑_τ=1^t𝔼[ζ^2_τ|ℱ^t-1] be the sum of the conditional variances. Then, for any x>0 and v∈ℝ, we have 
    ℙ{|S^t|≥ x, V^t≤ v^2}≤ 2exp(-x^2/2(v^2+ux)).



The proof can be found in <cit.>.   




 §.§ Proof of Theorem <ref>



Given a mixed NE x, let ℋ(x)=(M∪ W, E(x)) be the induced bipartite graph by x, where E(x)={(m,w): x_mw>0} denotes all the edges with fractionally positive proposals. Moreover, let us denote the neighboring nodes of m and w in ℋ(x) by N_m(x) and N_w(x), respectively. Since x is a NE, the value v_mw(x)=μ_mw∏_k>_w m(1-x_kw) should be the same for all w∈ N_m(x) and equals to u_m(x). Otherwise, using properties (i) and (ii), player m could strictly increase his payoff by putting his entire strategy mass on the woman w^*=_w∈ N_m(x)v_mw(x). The fact that u_m(x)=v_mw(x) ∀ w∈ N_m(x) follows because x_m is a probability distribution that is supported over N_m(x).

Next, we note that a man m cannot be the most preferred choice for more than one woman in ℋ(x). Otherwise, let w_1,w_2∈ N_m(x) be two different women who rank m the highest among their fractional matches in N_w_1(x) and N_w_2(x), respectively. Then, we have 
    ∏_k>_w_1 m(1-x_kw_1)= ∏_k>_w_2 m(1-x_kw_2)=1,
 which together with the above argument implies that

    μ_mw_1=μ_mw_1∏_k>_w_1 m(1-x_kw_1)=v_mw_1(x)=u_m(x)=v_mw_2(x)=μ_mw_2∏_k>_w_2 m(1-x_kw_2)=μ_mw_2.
  
This contradicts the fact that there are no ties between the preferences of man m.    

Since each woman receives at least one fractional proposal, N_w(x)≠∅ ∀ w. Let m_w be the unique most favorite man of w in N_w(x). Using the above argument, the elements of {m_w, w∈ W} must be distinct such that ℳ_1={(m_w,w), w∈ W} forms a matching. Viewing this matching from men's side, one can represent ℳ_1 in the form of ℳ_2={(m,w_m), m∈ M}, where w_m=_w∈ N_m(x)μ_mw is the least preferred woman among those that man m fractionally proposed to. To see this equivalence, fix a man m and let w_0 be his match in M_1, i.e., m_w_0=m. Observe that each w'∈ N_m(x)∖{w_0} has a unique distinct most favorite man m_w'≠ m. That means that 

    v_mw'(x)=μ_mw'∏_k>_w' m(1-x_kw')≤μ_mw'(1-x_m_w'w')<μ_mw'.

On the other hand, since w_0 ranks m the highest among those who propose to her, we have v_mw_0(x)=μ_mw_0∏_k>_w_0 m(1-x_kw')=μ_mw_0. Since at a NE v_mw_0(x)=v_mw'(x), we must have μ_mw_0<μ_mw', ∀ w'∈ N_m(x)∖{w_0}, which implies w_0=w_m. Thus, for any pair (m_w_0,w_0)∈ℳ_1 there is a pair (m,w_m)∈ M_2 such that (m_w_0,w_0)=(m,w_m), which shows that ℳ_1=ℳ_2. 

Finally, we will show that the binary characteristic vector of the matching {(m,w_m), m∈ M}, denoted by x̂, is indeed a pure NE. We will show that by arguing that any strictly better deviation in x̂ is also a strictly better deviation in x, contradicting the fact that x is a NE. On the contrary, let m_w be a man who wants to deviate from his pair w to a woman w'≠ w. Such a deviation is beneficial to player m_w only if m_w'<_w' m_w. Otherwise, his payoff will reduce from u_m_w(x̂)=μ_m_ww to 0. Henceforth, we may assume m_w'<_w' m_w. The fact that the payoff of player m_w is currently u_m_w(x̂)=μ_m_ww and he wants to deviate to w' implies that μ_m_ww<μ_m_ww'. As m_w is the most preferred man for woman w in ℋ(x), we also have u_m_w(x)=μ_m_ww. Since m_w'<_w' m_w, the woman w' would prefer m_w over all the men in N_w'(x). Thus, in the mixed NE x, man m_w could also deviate by solely proposing to w', in which case his payoff would strictly increase from u_m_w(x)=μ_m_w,w to μ_m_ww', which is a contradiction.    






 §.§ Proof of Lemma <ref>



Fix a player m and assume X^*_mw^*=1. Since X^* is a pure NE, for any w≠ w^* we have

    v_mw^*(X^*)=μ_mw^*> v_mw(X^*)=μ_mw∏_k>_w m(1-X^*_kw)∈{0, μ_mw}.

On the other hand, we have 

    v_mw^*(x)-v_mw^*(X^*)   =μ_mw^*∏_k>_w^* m(1-x_kw^*)-μ_mw^*   ≥ -μ_mw^*∑_k>_w^* mx_kw^*   ≥ -μ_mw^*/2x-X^*_1,

where the second inequality holds because X^*_mw^*=1 implies X^*_kw^*=0 ∀ k≠ m, and hence 1/2x-X^*_1≥∑_k>_w^* mx_kw^*. Now we consider two cases: 

i) If v_mw(X^*)=μ_mw, then v_mw(x)≤μ_mw< μ_mw^* ∀ x, which implies v_mw^*(X^*)-v_mw(x)≥Δ. Combining this relation with (<ref>) and because x-X^*_1≤Δ/μ_max, we get v_mw^*(x)-v_mw(x)>Δ/2>c. 

ii) If v_mw(X^*)=0, there exists k>_w m such that X^*_kw=1. Then, v_mw(x)≤μ_mw(1-x_kw)= μ_mw(X^*_kw-x_kw) ≤1/2μ_mwX^*-x_1, and we have v_mw^*(X^*)-v_mw(x)≥μ_mw^*-1/2μ_mwX^*-x_1. Combining this relation with (<ref>) and because x-X^*_1≤μ_min/μ_max, we get v_mw^*(x)-v_mw(x)>μ_min/2> c.    




 §.§ Proof of Lemma <ref>


i) Suppose X^t-X^*_1≤c/250n^2, then X^t_m-X^*_m_1=2(1-X^t_mw^*)≤c/250n^2 ∀ m. Since X^t_m is a probability distribution, we can write 
    X^t_mw^*/X^t_mw≥1-c/500n^2/c/500n^2= 500n^2/c-1, ∀ m,w≠ w^*.
 
Therefore, for any m,w≠ w^*, we have η^tL̂^t-1_mw^*-η^tL̂^t-1_mw≥ln (500n^2/c-1)≥ln (n^2/2c)+6.

ii) Suppose η^tL̂^t-1_mw^*-η^tL̂^t-1_mw≥ln(n^2/2c) ∀ m,w. Then, we can write

    X^t_mw^*/X^t_mw=exp(η^tL̂^t-1_mw^*-η^tL̂^t-1_mw)≥n^2/2c ∀ m,w≠ w^*,

which implies X^t_mw^*≥1/1+2c/n≥ 1-2c/n. Thus, X^t_m-X^*_m_1=2(1-X^t_mw^*)≤4c/n ∀ m, and hence X^t-X^*_1≤ c.

iii) Suppose η^tA^t-1=b+β, where b=ln(n^2/2c)+3 and β≥ 0. Then, we have

    η^t+1A^t   =η^t+1A^t-1+η^t+1(v_mw^*(X^t)-v_mw(X^t))   ≥η^t+1A^t-1+η^t+1 c   = η^t+1/η^t(b+β)+η^t+1 c   =b+η^t+1/η^tβ+(η^t+1 c-(1-η^t+1/η^t)b)   ≥ b+η^t+1(c-(1/η^t+1-1/η^t)b)   ≥ b,
 
where the last inequality holds by the fact that 1/η^t+1-1/η^t≤c/b ∀ t≥ t_0.




 §.§ Proof of Theorem <ref>



Using identical arguments as in the proof of Theorem <ref>, with probability at least 1-δ,

    |η^tL̂^t-1_mw'-η^tL̂^t-1_mw-η^tA^t-1|≤ 3,  ∀ t≥ 1, ∀ m, w w',

where we recall that A^t-1= ∑_τ=1^t-1(v_mw'(X^τ)-v_mw(X^τ)). Now let Ω be the event such that X^τ→ X^*, and assume X^*_mw>0 for some m,w. Conditioned on the event Ω, we have

    η^tA^t-1≤η^tL̂^t-1_mw'-η^tL̂^t-1_mw+3=ln(X^t_mw'/X^t_mw)+3,  ∀ t≥ 0.

For any w'≠ w, by taking limit from the above relation as t→∞, we get
lim sup_t→∞η^tA^t-1≤ln(X^*_mw/X^*_mw)+3≤ O(1).[Here, if X^*_mw'=0, by convention we define ln(X^*_mw'/X^*_mw)=-∞.] Now let κ:=v_mw'(X^*)-v_mw(X^*). We claim κ≤ 0. Otherwise, if κ>0, then for any sufficiently large t>t̅ we must have A^t-1>tκ/2. Given the choice of stepsize, we obtain lim_t→∞η^t A^t-1=lim_t→∞tη^tκ/2=∞, which contradicts the fact that lim sup_t→∞η^tA^t-1≤ O(1). Thus, κ≤ 0, which shows that if X^*_mw>0, then v_mw(X^*)=max_w'v_mw'(X^*), i.e., X^* is a NE.          




 §.§ An Alternative Form for the Potential Function in Lemma <ref>

One can represent the potential function in Lemma <ref> directly for womens' ordinal preferences. Given a mixed strategy profile x, let us associate a vector p_w(x)∈ℝ_+^|M| to each woman w whose coordinates are sorted in the order of w's preferences, such that if m_1>_wm_2>_w…>_w m_n, we let the jth coordinate of p_w(x) be given by 
    p_jw(x)=x_m_jw∏_k>_w m_j(1-x_kw),
 which is the probability that woman w is matched to her jth favorite man, assuming that the players propose to women independently and according to their mixed strategies x. We note that for a pure-strategy profile x∈{0,1}^|M||W|, p_w(x) would be a binary vector with all entries being zero except the jth one being one, where j is the index of the highest ranked man proposing to w. Therefore, Lemma <ref> asserts that after each best response move, the vectors {p_w(x), w∈ W} increase lexicographically with at least one of the vectors increases strictly. Now, if we again encode woman w's preferences in a cardinal vector λ_w=(λ_wm_1,…,λ_wm_n), then for a pure strategy profile x∈{0,1}^|M||W|, we have 
    r_w(x)=⟨ p_w(x),λ_w ⟩=∑_mλ_wmx_mw∏_k>_w m(1-x_kw),
 which coincides with the definition of r_w(x) in Theorem <ref> that picks the reward of the highest-ranked man proposing to w under x. Therefore, in the space of cardinal preferences and for any pure-strategy profile x∈{0,1}^|M||W|, one can rewrite the potential function in Theorem <ref> in a compact as 

    Φ(x)=∑_w r_w(x)=∑_m,wλ_wmx_mw∏_k>_w m(1-x_kw),

whose value increases after every best response update by a player. We note the similarity between the potential function (<ref>) and sum of players' payoffs ∑_mu_m(x)=∑_m,wμ_mwx_mw∏_k>_w m(1-x_kw), where one is written using mens' preferences {μ_mw} and the other using the womens' preferences {λ_wm}.



 §.§ A Gap Estimation Oracle for Unknown Δ.


Motivated by the Upper Confidence Bound (UCB) for stochastic bandits <cit.>, here we provide a gap estimation oracle, which, in combination with Algorithm <ref>, allows each player to adaptively adjust his mixing parameter based on the estimated gap he has obtained so far. The integration of this oracle with Algorithm <ref> allows the algorithm to be more practical to the situation when a lower bound for the gap Δ is unknown. In particular, using standard concentration bounds for the UCB algorithm, one can show that the event that the estimated gap is larger than the true gap decays fast as time increases, hence only introducing a small error term to the overall performance of the algorithm.     

 








 Paper Title: Comparative Study of Data-driven Area Inertia Estimation Approaches on WECC Power Systems.









IEEE Power Energy Society General Meeting, 2024
Shell et al.: Bare Demo of IEEEtran.cls for Journals

Comparative Study of Data-driven Area Inertia Estimation Approaches on WECC Power Systems
    Bendong Tan,  Jiangkai Peng, Ningchao Gao, Junbo Zhao, Jin Tan
  This work was authored in part by the National Renewable Energy Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. This material is based upon work supported by the U.S. Department of Energy's Office of Energy Efficiency and Renewable Energy (EERE) under the Solar Energy Technologies Office Award Number 37772. B. Tan and J. Zhao are with University of Connecticut, Storrs, CT 06269. This work was based on Bendong Tan's summer internship at NREL. J. Peng, N. Gao and J. Tan are with National Renewable Energy Laboratory, Golden, CO 80401. (e-mail:  jin.tan@nrel.gov)
    ^1 Department of Mathematics, Politecnico di Milano, p.za Leonardo da Vinci 32, 20133 Milano, Italy

^2 IFP Energies nouvelles, 1 et 4 avenue de Bois-Préau, 92852 Rueil-Malmaison, France

^3 Department of Engineering Mechanics, KTH Royal Institute of Technology, Brinellvägen 8, 114 28 Stockholm, Sweden

======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================


§ ABSTRACT 
  With the increasing integration of inverter-based resources into the power grid, there has been a notable reduction in system inertia, potentially compromising frequency stability. To assess the suitability of existing area inertia estimation techniques for real-world power systems, this paper presents a rigorous comparative analysis of system identification, measurement reconstruction, and electromechanical oscillation-based area inertia estimation methodologies, specifically applied to the large-scale and multi-area WECC 240-bus power system. Comprehensive results show that the system identification-based approach exhibits superior robustness and accuracy relative to its counterparts.




Area inertia estimation, WECC 240-bus power system, system identification, measurement reconstruction, electromechanical oscillation.






§ INTRODUCTION

The role of inertia in ensuring the frequency stability of power systems is pivotal <cit.>. Nonetheless, the substantial integration of inverter-based resources (IBRs), such as wind energy and solar, has led to a marked decline in system inertia.  Consequently, obtaining an accurate understanding of inertia distribution becomes imperative for the secure operation of power systems <cit.>.

Extensive research has been dedicated to inertia estimation at the system level, focusing on quantifying the cumulative inertia within an entire power system. This approach conceptualizes all generators as a singular equivalent entity, the dynamics of which are articulated through the aggregated swing equation, framed under the center of inertia (COI) coordinate system <cit.>. Regarding the system-level inertia estimation, both the total disturbed active power and the COI frequency are requisite <cit.>. System inertia can be estimated via the dynamics of the aggregated swing equation or the definition of inertia. The COI frequency can be depicted as the weighted average frequency derived from phasor measurement units (PMUs) positioned at various locations <cit.>. On the other hand, the total disturbed power can be obtained via recorded large disturbance <cit.> or the total power variation at each generator <cit.>. 

However, acquiring the total disturbed power poses challenges, primarily attributed to the complexities in disturbance power quantification and insufficient PMU placements. Furthermore, in the context of systems with a significant presence of IBRs, inertia might be unevenly distributed throughout the system <cit.>. This introduces spatial variances, nuances that system-level inertia estimation fails to capture. While the overall system inertia might seem satisfactory, specific regions may experience an inertia deficit, potentially giving rise to stability concerns.

Hence, if system operators are inclined towards understanding inertia distribution, area inertia estimation emerges as a more favorable approach. Analogous to system-level inertia estimation, area inertia determination relies on the total disturbed power within the area and the corresponding area COI frequency. However, the cumulative disturbed power within a specific area can be ascertained by aggregating the changes in tie-line power <cit.>. Approaches for area inertia estimation can be categorized into three distinct categories: (1) System identification-based methods; (2) Measurement reconstruction-based methods; (3) Electromechanical oscillation-based methods. Within the realm of system identification-based methodologies, both <cit.> and <cit.> employ a reduced-order model, i.e., the transfer function, to emulate the dynamic input-output relationship between the total disturbed area power and the area COI frequency. This results in a straightforward computation of area inertia, anchored on the inertia's fundamental definition. Transitioning to the measurement reconstruction-based techniques, <cit.> deploys polynomial expansion to reconstruct both power deviation and frequency. Concurrently, <cit.> harnesses dynamic mode decomposition (DMD), yielding an analytical framework for deducing area inertia. Delving into the electromechanical oscillation-based strategies, both <cit.> and <cit.> establish the relationship between the electromechanical oscillation mode and area inertia, facilitating the quantification of the latter.

The previously discussed inertia estimation methodologies, tested on benchmark systems with coherent generator dynamics, have yet to be comprehensively compared in the same benchmark system or real-world systems under practical testing scenarios where areas are not strictly defined by generator coherency. The goal of this paper is to understand the pros and cons of each method, i.e, on the WECC 240-bus system, a reduced model of the US western grid. Here, frequency stability is maintained through dispatch in balanced areas, which may not always correspond to generator coherency <cit.>.




§ PROBLEM FORMULATION

In a specific regional power system, as shown in Fig. <ref>, the dynamics between the area's tie-line power and its center of inertia (COI) frequency can be effectively characterized by representing the system with an equivalent synchronous generator. This representation yields an aggregated swing equation given by:

    ω̇_COI,i = 1/2 H_i(P_m, i-P_e, i-D_iΔω_COI,i)

where ω_COI,i denotes the COI frequency for the i-th area: 

    ω_COI,i=m ∈Area  i∑ω_mS_m/m ∈Area  i∑S_m

where ω_m and S_m are the generator m's rotor speed and rated power respectively.  P_m, i and P_e, i are representative of the tie-line total mechanical and electrical power for the i-th area, respectively. Additionally, H_i signifies the equivalent inertia and D_i the damping factor pertinent to the i-th area. Since primary frequency control can be formulated as  K_iΔω_COI,i, where K_i is the primary frequency control coefficient, the perturbations effects associated with P_m, i can be included into term D_iΔω_COI,i in (<ref>). Therefore, (<ref>) can be further reformulated in an incremental form:

    Δω̇_COI,i = 1/2 H_i(-Δ P_e, i-D_iΔω_COI,i)

where Δ P_e, i signifies the deviation in electrical power from its steady-state value.  
When P_m, i undergoes dynamic changes, as observed during the initiation of primary frequency control, (<ref>) yields a higher estimate for the equivalent damping factor.

To estimate H_i, the primary challenge lies in accurately determining Δω̇_COI,i. Relying solely on the finite difference method for frequency measurements can introduce numerical errors. Therefore, existing state-of-art methods try to estimate inertia without directly calculating Δω̇_COI,i.





§ AREA INERTIA ESTIMATION APPROACHES

In the following section, we benchmark cutting-edge methodologies from each category of area inertia estimation on a large-scale practical power system. Specifically, we adopt the techniques proposed in <cit.>, <cit.>, and <cit.> which represent the system identification-based, measurement reconstruction-based, and electromechanical oscillation-based methods, respectively. It is worth mentioning that enhancements have been incorporated into the techniques outlined in <cit.> and <cit.> to address and mitigate numerical challenges.




 §.§ System Identification-based Method

The Laplace transformation applied to both sides of (<ref>) results in the relation Δω_COI, i(s)/Δ P_e, i(s)≈-1/2 H_i s+D_i. Acknowledging the influences of additional control components linked with the generator, the central premise of <cit.> is to employ a higher-order transfer function, denoted as G_i(s), to act as a surrogate model elucidating the relationship between ω_COI, i and Δ P_e, i. The form of G_i(s) is given by:

    G_i(s) = 1/a_n s^n+⋯ +a_1 s+a_0,

where a_n signifies the coefficient of the nth-order term. Given the presence of other control modules linked to generators, n = 2 is deemed sufficient to approximate the relationship between ω_COI, i and Δ P_e, i. The parameters intrinsic to G_i(s) can be discerned through the N4SID (Numerical Algorithm for Subspace State Space System Identification) method, leveraging the input Δ P_e, i and output Δω_COI, i <cit.>.

Subjecting G_i(s) to a step function (emulating a scenario where Δ P_e, i=1 pu) yields the step response g_step(t). This step response can be approximated using an N_p-order polynomial:

    g_step(t) = c_N_pt^N_p+⋯ +c_1 t+c_0

where t is the temporal variable and c_N_p is the coefficient of the N_p-order term of g_step(t). Given the small magnitude of D_i, which can be approximated as negligible, we propose to estimate H_i with (<ref>):

    H_i   =-. Δ P_e, i(t)/2 ω̇_COI, i(t)|_t=0 =-. 1/2ġ_step(t)|_t=0 = - 1/2c_1

where H_i is estimated equivalent inertia for the i-th area. In
the swing equation, inertia values of synchronous generator
range from 1.75 s to approximately 10 s, and the damping
factor varies from the magnitude of 10^-2 to 10^-1 <cit.>.
Hence, the damping effect is small enough to be neglected.
The exclusion of D_i in expression (6) is expected to result in a slightly biased estimate of inertia. Generators with similar rotor angle response following a disturbance are called
coherent generators. The coherency between generators, particularly over large electrical distances, may not be significant, potentially leading to substantial errors in ω_COI as an aggregate frequency representation. However, since the system identification-based method solely utilizes dynamics at the initial stage (t=0), the coherency of area generators exerts minimal influence on its performance.




 §.§ Measurement Reconstruction-based Method

A typical approach for measurement reconstruction-based method is using DMD to reconstruct the measurements of X∈ℝ^2N_a× M, where X = [Δω_COI, 1, Δω_COI, 2  ,⋯, Δω_COI, N_a ,ΔP_e, 1 ΔP_e, 2 ⋯,ΔP_e, N_a]^𝖳  <cit.>. N_a is the number of areas in the power system, while M is the number of measurements.

    A_c[[ φ_k^Δω_COI;          –;   φ_k^ΔP_e ]]=λ_k[[ φ_k^Δω_COI;          –;   φ_k^ΔP_e ]]

where A_c = ln(A_d)/ Δ t;  
Δ t is the sampling time interval; A_d is the state transition matrix that leads to X_2:M = A_dX_1:M-1; X_2:M denotes the measurements from time instants 1 to M-1; φ_k^ΔP_e and φ_k^ΔP_e denote the the k-th DMD modes with respect to
Δω_COI and ΔP_e; λ_k is the k-th eigenvalue. As a result, measurements can be reconstructed as:

    [[ Δω_COI;   ΔP_e ]]=∑_k=1^2N_a e^λ_k t b_k[[ φ_k^Δω_COI;   φ_k^ΔP_e ]]

where b_k is the initial value coefficient corresponding to
k-th eigenvalues. By applying (<ref>) to (<ref>), (<ref>) can be derived:

    [ Re( φ ^Δω_COIλb)   Re( φ ^Δω_COIb)
    Im( φ ^Δω_COIλb)   Im( φ ^Δω_COIb)
    ] [ [ 2H;  D;    ]] =-[ [ Re( φ ^ΔP_eb); Im( φ ^ΔP_eb);               ]]


In the work of <cit.>, inertia is inferred through the resolution of (<ref>). Nonetheless, our observations suggest potential inaccuracies in scenarios where the aforementioned equation admits multiple solutions. In an attempt to constrain and refine the solution space, we introduce additional constraints:

    0 ≤D≤ 1

By jointly solving (<ref>) and (<ref>), we posit that the inertia estimation converges to a more pragmatic and precise value.





 §.§ Electromechanical Oscillation-based Method

The study by <cit.> establishes a connection between area inertia and electromechanical oscillations, facilitating the inference of area inertia directly from these oscillations. Taking into account the frequency bandwidth B of the electromechanical oscillation, the Fourier transform of (<ref>) can be derived as:

    2H_i∑_γ=0^γ=BΔω̇_COI,i(γ)+D_i∑_γ=0^γ=BΔω_COI,i(γ)+∑_γ=0^γ=BΔ P_e, i(γ)=0

where γ is the frequency in the oscillation band. According to (<ref>), H_i can be estimated via (<ref>).









 §.§ Comparisons among Various Methods

From the comprehensive analysis of the methodologies presented, it is evident that none necessitate the computation of COI frequency derivatives, a process known to encounter numerical challenges, particularly in the presence of noise or outliers.

(1) The pros of each method are listed as follows:

    
  * System Identification: This method offers ease of use due to its limited hyperparameters, which minimally impact its performance.

    
  * Measurement Reconstruction: Beyond inertia estimation, this method also furnishes damping estimation, thereby yielding additional dynamic information.
    
  * Electromechanical Oscillation: Similar to measurement reconstruction-based method, this method can also provide damping estimation.



(2) The cons of each method and corresponding measures are summarized as follows:

    
  * System Identification: Despite yielding a biased estimation due to the omission of the damping factor, the resultant error is deemed acceptable, considering the relatively minor magnitude of the damping factor.
    
  * Measurement Reconstruction: Given that b, the initial value coefficient associated with eigenvalues in measurement reconstruction, is determined by the first data point in response measurements, noise considerably influences the final inertia estimation. Consequently, noise processing and careful choice of the start point of the response measurements is essential.
    
  * Electromechanical Oscillation: As delineated in (<ref>), inertia estimation accuracy is contingent upon the bandwidth B, underscoring its sensitivity to B. Thus, precise estimation of B is crucial.







§ NUMERICAL RESULTS

To comprehensively compare the pros and cons of each method, extensive validations have been conducted on the WECC 240-bus power system, where a total of 13 regional power grids have been delineated according to balanced areas, as shown in Fig. <ref>. Every scenario was examined in the presence of a disturbance, characterized by a three-phase short-circuit fault at Bus 1002, initiated at t = 1s and cleared by t = 1.1s. The entirety of the simulation spanned 10 seconds.






 §.§ Parameter Sensitivity Analysis for Different Methods


Upon reviewing various methodologies, it becomes evident that the polynomial order N_p, the initial value b, and the oscillation bandwidth B serve as hyperparameters for system identification, measurement reconstruction, and the electromechanical oscillation-based method, respectively. For b, it is imperative to designate the specific instant at which at which the starting point of the measurement response is selected. To evaluate the efficacy of each method, we introduce two indices: the maximum estimation error (MEE) and the estimation error (EE):

    MEE = 1≤ i ≤ N_amax( | H_i-H_i/H_i |   × 100%)   ,  EE =  | H_i-H_i/H_i |  × 100%


From Fig. <ref>, it is evident that an increased N_p enhances the fitting quality for g_step(t), thereby improving the accuracy of area inertia estimation. Specifically, N_p = 4 emerges as an optimal choice, yielding the minimal MEE for the system identification-based method. For the measurement reconstruction-based method, the initial values b are chosen at several time steps post-disturbance clearing. As depicted in Fig. <ref>(a), selecting b shortly after disturbance clearing results in a MEE exceeding 400%. This can be attributed to the fact that while DMD offers a linear approximation of the dynamic system, measurements taken shortly post-disturbance exhibit pronounced nonlinear characteristics. Consequently, b is determined using the measurement from the 14-th time step post-disturbance to minimize the estimation error. Regarding the electromechanical oscillation-based method, its elevated MEE, reaching up to 100%, stems from the formulation of (<ref>) through the continuous domain's Fourier transform. This necessitates approximation via the discrete Fourier transform, given that measurements are discretely acquired <cit.>. Owing to the substantial estimation error associated with the electromechanical oscillation-based method, it is excluded from subsequent comparative analyses.




 §.§ Area Inertia Estimation Results Under Noise




In this section, the performance of system identification and measurement reconstruction-based methods are validated under measurement noise. The measurement noise is postulated to follow a zero-mean Gaussian distribution with a standard deviation of σ = 4 × 10^-3 <cit.>. Notably, this is greater than the realistic noise encountered in measurement data, which has an upper limit of 1% total vector error (TVE). To mimic the data preprocessing typically encountered in practical operations, noisy measurements undergo smoothing using a low-pass filter with a cut-off frequency of 5Hz <cit.>. Fig. <ref> illustrates the inertia estimation errors for each area as determined by both the system identification and measurement reconstruction-based methods. In both noise-free and noisy conditions, the system identification-based method consistently yields estimation errors below 5% for all areas, with the exception of AZ. This underscores the method's robustness. In contrast, the measurement reconstruction-based method exhibits area inertia estimation errors exceeding 100%. Such discrepancies might arise from noise-induced alterations in the initial values b, which in turn precipitate significant estimation errors.




 §.§ Estimation Results under Different Operation Conditions



This study extends the validation of both the system identification and measurement reconstruction-based methods across diverse operational conditions. The two-week hourly operational data, spanning January 1-14, 2019, was obtained using NREL's Multi-timescale Integrated Dynamic and Scheduling Model (MIDAS)<cit.>. For each operational condition, a consistent disturbance and measurement noise, as previously described, is introduced. Owing to page constraints, we present the estimation results exclusively for states CO, OR, WA, CA1, and CA2, as these exhibit notable inertia fluctuations over time, as shown in Figs. <ref> and <ref>. From the data, it is evident that the total system load undergoes periodic fluctuations, leading to the disconnection or connection of generators. As a result, the total inertia in each area exhibits time-varying characteristics. The system identification-based method showcases consistent estimation performance, with the maximum estimation error remaining below 15%, as depicted in Fig. <ref>. While its estimation outcomes tend to be lower than the actual values, indicating a biased approach, its performance is nonetheless reliable. This further illustrates its suitability for regional power systems where generator coherency might not necessarily be maintained. Conversely, due to the influence of operational conditions on the initial value b, the measurement reconstruction-based method registers errors exceeding 100% for states OR, WA, CA1, and CA2, even though its estimations for CO appear accurate, as shown in Fig. <ref>. This underscores the limited robustness of the measurement reconstruction-based method in estimating area inertia. In summary, the system identification-based method emerges as the more favorable approach for practical power system area inertia estimation.




§ CONCLUSIONS

In this paper, we undertook an exhaustive comparative analysis to assess the efficacy of system identification, measurement reconstruction, and electromechanical oscillation-based methods for area inertia estimation in a large-scale practical power system, specifically the WECC 240-bus power system. Numerical evaluations under conditions of noise and time-varying operations underscore that the system identification-based method offers superior robustness and accuracy in area inertia estimation, despite its slight estimation bias,  compared to the other methods.



99




Kundur_1994
 P. Kundur, Power System Stability and Control. McGraw-Hill, 1994.

Tan_2022
B. Tan, et al., “Power system inertia estimation: Review of methods and the impacts
of converter-interfaced generations,” Int. J. Elect. Power Energy Syst.,
vol. 134, p. 107362, 2022.

Wall_2018
P. Wall and V. Terzija, “Simultaneous estimation of the time of disturbance and inertia in power systems,” IEEE Trans. Power Syst., vol. 29, no. 4, pp. 2018–2031, 2014.

Schiffer_2019
J. Schiffer, et al., “Online estimation of power
system inertia using dynamic regressor extension and mixing,” IEEE
Trans. Power Syst., vol. 34, no. 6, pp. 4993–5001, 2019.

Wang_2019
B. Wang, et al., “Power system inertia estimation method based on maximum frequency
deviation,” IET Renew. Power Gener., vol. 16, no. 3, pp. 622–633, 2022.


Phurailatpam_2019
C. Phurailatpam, et al., “Measurement
based estimation of inertia in ac microgrids,” IEEE Trans. Sustain. Energy, pp. 1–1, 2019.

Makolo_2021
P. Makolo, et al., “Online inertia estimation for power systems with high penetration of res using recursive parameters
estimation,” IET Renew. Power Gener., vol. 15, no. 12, pp.
2571–2585, 2021.

Zeng_2020
F. Zeng, et al., “Online identification of inertia distribution in normal operating power system,” IEEE Trans. Power Syst., vol. 35, no. 4, pp. 3301–3304, 2020.

Tuttelberg_2018
K. Tuttelberg, et al., “Estimation of power
system inertia from ambient wide area measurements,” IEEE Trans. Power Syst., vol. 33, no. 6, pp. 7249–7257, 2018.

Li_2018
D. Li, et al., “Area inertia estimation
of power system containing wind power considering dispersion of frequency response based on measured area frequency,” IET Gener. Transm. Distrib., vol. 16, no. 22, pp. 4640–4651, 2022.

Zeng_2020
F. Zeng, et al., “Online estimation of power system inertia constant under normal operating conditions," IEEE Access, vol. 8, pp. 101426-101436, 2020.

Phurailatpam_2020
C. Phurailatpam, et al., “Measurement-based estimation of inertia in AC microgrids," IEEE Trans. Sustain. Energy, vol. 11, no. 3, pp. 1975-1984, July 2020.

Yang_2021 D. Yang, et al., “Data-driven estimation of inertia for multiarea interconnected power systems using dynamic mode decomposition," IEEE Trans. Ind. Inform., vol. 17, no. 4, pp. 2686-2695, April 2021.

Cai_2019
G. Cai, et al., “Inertia estimation based on observed electromechanicaloscillation response for power systems," IEEE Trans. Power Syst., vol. 34, no. 6, pp. 4291-4299, Nov. 2019..

Wang_2022
B. Wang, et al., “An improved electromechanical oscillation-based inertia estimation method," IEEE Trans. Power Syst., vol. 37, no. 3, pp. 2479-2482, May 2022.

Yuan_2020
H. Yuan, et al., “Developing a reduced 240- busWECC dynamic model for frequency response study of high renewable integration,” in Proc. IEEE PES Transmiss. Distrib. Conf. Expo., 2020, pp. 1–5.

WECC_2020 
National Renewable Energy Laboratory, “Test case repository for high renewable study,” 2020. [Online]. Available: https://www.nrel.gov/grid/test-case-repository.html.

Panda_2020 R. K. Panda, et al., “Online estimation of system inertia in a power network utilizing synchrophasor Measurements," IEEE Trans. Power Syst., vol. 35, no. 4, pp. 3122-3132, July 2020.




Singh_2009
R. Singh, et al., “Choice of estimator for distribution system state estimation”. IET Gener. Transm. Distrib., vol. 3, no. 7, pp. 666, 2009.

Tan_2023
J. Tan, et al., “Multi-timescale integrated dynamics and scheduling for solar (MIDAS-solar) (final technical report)," United States, 2023. [Online]. Available: https://doi.org/10.2172/1972321.






 Paper Title: H-Watch: An Open, Connected Platform for AI-Enhanced COVID19 Infection Symptoms Monitoring and Contact Tracing.





1]
Tommaso Polonelli

1]
Lukas Schulthess

2]
Philipp Mayer

1]
Michele Magno

2]
Luca Benini

[1]
Project Based Learning, ETH Zurich, Zurich, Switzerland

[2]
Integrated Systems Laboratory, ETH Zurich, Zurich, Switzerland


H-Watch: An Open, Connected Platform for AI-Enhanced COVID19 Infection Symptoms Monitoring and Contact Tracing

    [
    August 5, 2024
===============================================================================================================



§ ABSTRACT 
The novel COVID-19 disease has been declared a pandemic event. Early detection of infection symptoms and contact tracing are playing a vital role in containing COVID-19 spread. As demonstrated by recent literature, multi-sensor and connected wearable devices might enable symptom detection and help tracing contacts, while also acquiring useful epidemiological information.
This paper presents the design and implementation of a fully open-source wearable platform called H-Watch. It has been designed to include several sensors for COVID-19 early detection, multi-radio for wireless transmission and tracking, a microcontroller for processing data on-board, and finally, an energy harvester to extend the battery lifetime. Experimental results demonstrated only 5.9 mW of average power consumption, leading to a lifetime of 9 days on a small watch battery. Finally, all the hardware and the software, including a machine learning on MCU toolkit, are provided open-source, allowing the research community to build and use the H-Watch.  



Wearable Device, COVID-19, Smart Sensors, Low Power Design, Tiny Machine Learning, Wireless Sensors Networks.




§ INTRODUCTION



COVID-19 has been declared a pandemic by the World Health Organization (WHO) and poses a significant challenge for healthcare infrastructure around the world. Continuous vital sign monitoring for symptoms of severe pneumonia and sepsis, such as blood <cit.> oxygen saturation level (SpO2 <93%), respiratory rate (>30 breaths/minute), heart rate, body temperature, fatigue, coughing detection, and blood pressure can assist in the early recognition of high-risk patients <cit.>. For example, saturation values below 95% are a symptom of hypoxemia (reduction in the presence of oxygen in the blood), usually due to a decrease in gas exchange at the pulmonary alveoli level, a typical symptom of the worsening of some viral pneumonia <cit.>. Continuously monitoring those parameters and tracking users in their movements is crucial not only to early detect infections but also to follow its diffusion <cit.>. However, continuous patient monitoring and tracking are ultra-challenging for many key issues, such as privacy <cit.>, complexity of the monitoring, early recognition <cit.>, long-term operation with mobile battery-operated devices, and discontinuous connectivity, among others. 




Thanks to technology advancements in low-power integrated circuits (ICs), sensors, and wireless protocols have enabled the practical and daily use of light-weighted and unobtrusive wearable devices <cit.>, where electronics are worn on the human body or hidden into clothes. Among other products, smartwatches are a massive commercial reality with hundreds of products specifically designed for fitness and entertainment applications. As a containment measure for the COVID-19 pandemic, back-tracing contacts and personal interactions via smartphones is becoming increasingly important. Its active contribution in identifying potential outbreaks has been demonstrated <cit.>. While GPS tracking, in addition to non-negligible privacy and security implications <cit.>, poses serious limits to adequate indoor coverage, the use of BLE technology is considered as a good compromise <cit.> between the tracing precision (in a range from 10 cm to 10 m) and user privacy <cit.>; for this reason, it is the most widely adopted technology in consumer devices. However, today most of commercial systems are not ready to be used in a pandemic situation as the COVID-19. The most critical weaknesses are the limited energy supply due to the battery small size and limited computational resources.



A new recent trend is to couple low power design with machine learning on microcontrollers (MCUs), energy-efficient wireless communication, and energy harvesting to accurately re-design smartwatches to not only help the management of chronic diseases but to potentially play a key role in providing early infection warnings <cit.>.



This paper presents the design and implementation of a hardware-firmware open-source smartwatch called Health Watch (H-Watch). H-Watch combines multi-sensors for health monitoring, an ARM-Cortex-M4F for data acquisition and processing, wireless communication, and energy harvesting to achieve a long-lasting intelligent device. The main H-Watch features are low-power in the range of few mW peak, sub mW average, the capability to run artificial neural networks on-board, a multi-source energy harvesting, high integration resulting in a small form factor, and its novel 5G Narrow Band Internet of Things (NB-IoT) communication. Due to the ultra-low power design and the aggressive power management, H-Watch can automatically and continuously measure dissolved oxygen, heart rate, temperature, respiration rate, motion, and audio signals requiring an average power of 5.9 mW. Thanks to the advanced low-power design and photovoltaic energy harvesting, the device lifetime reaches up to 10 working days with 500 of average light. Anyway, it lasts 9 days with a single battery charge. 

H-Watch supports a wide range of connectivity options that can also be used to track and exchange alarms and data with local devices, for example, a smartphone, through the BLE 5.0 connection. Direct global connectivity is also available, using NB-IoT for tracking. NB-IoT is a novel protocol standardized by 3GPP. It is also known as LTE Cat-NB1(NB2) and belongs to Low Power Wide Area Network (LPWAN) technologies that could work virtually anywhere if the 4G (or 5G) infrastructure is present. It can send alarms and data directly to secure servers, such as time traces from on-board sensors. 

To allow researchers and engineers to exploit the features and the hardware-firmware co-design of H-Watch, this paper open-sources[<https://github.com/ETH-PBL/H-Watch>] all the hardware schematics and the layout as well as software libraries for sensors and peripherals, providing a platform to analyze, classify and study a broad infected population sample helping for remote health assistance and diagnosis. Moreover, the repository will provide a library and tools to run Artificial Intelligence (AI) algorithms for on-board data analysis. Indeed, the H-Watch aims to run pre-trained AI models, such as neural networks, for in-situ feature extraction and classification. In this work, we also provide system energy consumption to estimate the battery life with and without the energy harvester. 







§ SYSTEM ARCHITECTURE


Figure <ref> shows the H-Watch architecture. From left to right, the figure illustrates the power management (PM), which is specifically designed to achieve extremely low power consumption and to handle both a Li-Ion 370 mAh battery and a 7 cm^2 solar panel. The sub-system includes the STM32WB55RGV6 SoC (STM32 hereafter) from ST Microelectronics that manages the sensor acquisition and the wireless connectivity. In parallel with four internal sensors, namely 6-axes IMU (LSM6DSM), 6-axes magnetometer (LSM303AGR), a MEMS Microphone (MP34DT05TR), skin temperature and pressure sensors (LPS22HB), the H-Watch features a low power LCD display (LS012B7DH02). In addition, it features the MAX30101, an integrated SoC for pulse oximetry and heart rate monitoring at low power consumption, 5.5 mW.  H-Watch can communicate with local devices, such as smartphones and laptops, and directly to the cloud through an embedded NB-IoT transceiver (Quectel BC95-G) and the BLE interface.







 §.§ Power supply and Energy Harvesting

The power supply sub-system is designed around the BQ25570 and the TPS63031 from Texas Instruments.

BQ25570 is today the most efficient energy harvester chip on the market. It is used to recharge the Li-Ion battery with the flexible solar panel wrapped around the wrist. The BQ25570 achieves high conversion efficiency (90%) as it periodically adjusts its internal input-impedance to handle Maximum Power Point (MPP) of the energy source; indeed, this parameter changes with the illumination environment.

The BQ25570 has the function of charging the batteries using an integrated boost-converter, and, simultaneously, TPS63031 supplies the system exploiting an integrated high-efficiency buck-boost converter (Table <ref> - V_DD), specifically selected to compensate input voltage oscillations coming from the discontinuous energy source.



The whole supply voltage has been chosen to be 3.3 V, providing a single supply cluster for ICs. This was a specific design choice to decrease the PCB size, as fewer external components are required. Furthermore, it is the lowest voltage supported by the majority of ICs used in the circuit, minimizing the system energy consumption. 

From a wearable application point of view, the effective light spectrum, the size, the flexibility of the cells, and output power, are essential requirements. 

We selected a flexible panel, SP3-12 by Flexsolarcells[http://www.flexexsolarcells.com], that measures only 7 cm^2, which is directly used as H-Watch wrist band.

The energy harvesting capability of the small-sized cell is shown in Figure <ref>. For the measurement, a Roline RO1332 lux meter has been placed with the solar cell in a darkened chamber artificially illuminated by a controlled broadband light source. The cell output power in matched conditions is measured for illumination from 0 to 2000, which corresponds with the recommended lighting conditions for offices or workspaces. The importance of maximum power point tracking is highlighted in the inset of Figure <ref> measured at the static illumination of 1900.

In a typical indoor scenario of 500 with artificial lighting, the circuit allows to harvest 73 with a conversion efficiency of 92% and a storage voltage of 3.7. In outdoor light conditions of 10  more than 15 can be achieved. 

The power generation of each energy source and the power consumption of each internal sub-system (Table <ref>) have been evaluated separately under controlled conditions. We measured the power intake through the Keysight B2900A Series Precision Source/Measure Unit (SMU). 





 §.§ Smart Module  and on-board Processing


As demonstrated in previous works <cit.>, the IMU can be beneficial to improve the quality of the measurement when the user is moving <cit.>. The pulse oximeter non-invasively measures the blood oxygenation by shining light at two different wavelengths into the wrist and by analyzing the pulsatile component of the reflected
signal <cit.>. The oximeter sensor controls two LEDs and converts the analog signal from its photodiode into a digital representation for the connected microcontroller. The case form ensures adequate contact with the skin.

The on-board processing capability consists of the STM32 that integrates multiple hardware accelerators (data processing, controlling sensing, communication) in a miniaturized (8 mm × 8 mm) single IC. The microcontroller has a dual-core architecture with an ARM Cortex M4F for processing and an ARM Cortex M0+ dedicated to the Bluetooth stack.

The power-optimized ARM Cortex M0+ runs autonomously, allowing the rest of the processor to stay in standby mode. Its radio controller is a further reason for selecting this MCU, the patient contact tracing from the BLE advertising mode can be acquired and stored in memory without involving the ARM Cortex M4, improving the energy efficiency of the whole system.  

On-board classification and feature extraction are available through two different tools: X-CUBE-AI from ST Microelectronics is an expansion package that extends the standard STM32 capabilities with automatic conversion of pre-trained Neural Network and ANSI C code generation, and FANN-ON-MCU, ARM Cortex-M, and low power micro-controllers <cit.>. The latter is a free open source neural network library, which implements instruction optimized (exploiting ARM CMSIS-NN) artificial neural networks in fixed and floating-point computations. It supports C code for both fully connected and sparsely connected networks.



In the GitHub repository[<https://github.com/ETH-PBL/H-Watch>] we release two machine learning examples using both tools. Classification and feature extraction go beyond this paper's scope, which focuses on the hardware and energy profile description.



From our experience, temporal convolutional networks, based on 1D-Convolutional layers, and more widely used MLP feed-forward fully connected networks, need an execution time between 21 ms and 500 ms on the STM32 @ 64 MHz. These values are defined by sampling and classifying heart rate, SpO2, and accelerometer data to extract the patient's health condition.



 §.§ Wireless Connectivity


H-Watch hosts a dual-radio sub-system, including BLE 5.0 and NB-IoT. In particular, NB-IoT can stream at 170 Kbps with 164 dB of link budget, enabling a wide signal coverage. Despite the long-range communication capabilities, the transceiver needs just 31 uJ per bit when transmitting <cit.>.

H-Watch supports BLE 5.0 for privacy-preserving contact tracing <cit.>, to be compliant with standard protocols used by smartphone Apps, and the NB-IoT for optional geo-tracking and bridge-less communication to secure remote servers, to stream results from on-board processing. 
However, the NB-IoT features a non-negligible energy overhead due to the cellular complex infrastructure. NB-IoT has the dual advantage of communicating through the cellular network like regular smartphones and transferring sensitive data by connecting directly to secure servers, even for low-cost miniaturized devices without the need to go through complex software stacks.
In <cit.>, authors extensively studied and characterized the NB-IoT protocol, where the energy is lightly dependent on the packed size, and heavily affected by the signal strength indicator (RSSI). 
The average energy per packet is consequently modeled in Table <ref>, where three different I^BC95G_active coverage conditions are provided. 



§ EXPERIMENTAL RESULTS


 H-Watch has been developed to evaluate functionalities and performances in terms of low power and lifetime.  Figure <ref> shows the H-Watch mechanical cross-section in which it presents the flexible solar panel, PCB board, the display, and the clock case. The proposed solution is fully wearable, plug & play, and it is comfortable to wear, allowing long term measurements without annoying the observed patient.



Table <ref> collects and presents the sub-system power consumption for each H-Watch ICs. In this paper, we consider four operation modes. Sleep mode in which all the sensors and radios are off, but the real-time clock and the display are active to enable periodic wake-up, it needs 97 μ W. Advertising mode, it is similar to Sleep mode, but the BLE is advertising at 1 Hz and 0 dBm. It is dedicated to contact tracing and consumes only 226 μ W. In motion detection mode, with the accelerometer and skin temperature enabled, the H-Watch needs 1.75 mW on average for collecting and processing the data. Lastly, with the full operation mode, in which the watch performs human activity and health classification (oximeter and Heart-rate enabled), the power consumption increases up to 10 mW. These results consider the DC/DC efficiency, named V_DD and V^MAX30101_DD in Table <ref>.

In full operation mode, the battery can support the H-Watch for 5 days, while in advertising mode, it exceeds 1 month of operations. Duty cycling between full and motion at 50%, the H-Watch lifetime is 9 days. One NB-IoT packet per day is considered in these conditions, which is used to send highly compressed information to secure servers after local processing.

In the worst case, when the application layer requires long-range connectivity, the battery longevity is heavily affected. With RSSI > -95 dBm, each uplink needs approximately the equivalent amount of energy to run the full operation mode for 1 minute, and with RSSI < -110 dBm it grows up to 6 minutes. With good coverage (RSSI > -95 dBm), sending 2.2 MB of data requires approximately 50% of the battery capacity, while, in the worst case, the uplink volume decreases to 0.6 MB. It is clear that the NB-IoT supports only few packets per day, mainly streaming alarms or pre-extracted features from on-board processing. 



With the help of the energy harvester, H-Watch extends the monitoring time.

Considering an indoor scenario with 500, the captured power support sleep and advertising modes, whose life can reach up to two months (8 hours at 500 - 16 hours no energy). However, considering outdoor activities, sleep and advertising mode are fully covered by solar panel, while the 50% duty-cycled mode between full and motion reaches up to 20 days (6 hours at 10 and 4 hours at 500, 14 hours no energy). 







§ CONCLUSIONS


This paper proposed the design and implementation of an open-source wearable long-lasting smart monitoring platform for health monitoring and tracking, which can provide direct cloud connectivity through state-of-the-art NB-IoT cellular technology. 

The H-Watch is based on widely available off-the-shelf components; however, it is designed with low power and on-board intelligence in mind. By continuously measuring the blood oxygenation and heart rate with a sampling rate of 50 Hz, accurate results can be achieved with a battery life of 9 days or 20 days, respectively non-using and using the solar energy harvester. 





Finally, H-Wach is also a low-cost platform. The MCU and the energy harvester cost approximatively 10 1KQty, while the BC95-G, the NB-IoT radio transceiver, is the most expensive part, 15. Both external sensors are commonly used in wearable devices, which cost just 2 each. The total cost of off-the-shelf electronics and battery is below 40. The solar panel costs 4 and the battery 2, while the display is included in the watch frame: 35. 



Future works will focus on the sensor data acquisition to verify the accuracy versus a medical device and, on machine learning, to validate early detection. 



§ ACKNOWLEDGMENT

The authors recognise and thank the European Open Science Cloud (EOSC) for financial support under COVID-19 research to open access grant agreements, program call H2020-INFRAEOSC-2018-4 grant number 831644.




unsrt




 Paper Title: Large Language Model (LLM)-enabled In-context Learning for Wireless Network Optimization: A Case Study of Power Control.



IEEEexample:BSTcontrol






 Large Language Model (LLM)-enabled In-context Learning for Wireless Network Optimization: A Case Study of Power Control 
Hao Zhou, Chengming Hu, Dun Yuan, Ye Yuan, and Xue Liu are with the School of Computer Science, McGill University, Montreal, QC H3A 0E9, Canada. (mails:hao.zhou4, chengming.hu, dun.yuan, ye.yuan3@mail.mcgill.ca, xueliu@cs.mcgill.ca). Di Wu is with the School of Electrical and Computer Engineering, McGill University, Montreal, QC H3A 0E9, Canada. (email: di.wu5@mcgill.ca).
Charlie Zhang is with Samsung Research America, Plano, Texas, TX 75023, USA. (email: jianzhong.z@samsung.com).

     Hao Zhou, Chengming Hu, Dun Yuan, Ye Yuan, Di Wu, 
 Xue Liu, Fellow, IEEE, and Charlie Zhang, Fellow, IEEE. 
    
================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================







§ ABSTRACT 
Large language model (LLM) has recently been considered a promising technique for many fields. This work explores LLM-based wireless network optimization via in-context learning.

To showcase the potential of LLM technologies, we consider the base station (BS) power control as a case study, a fundamental but crucial technique that is widely investigated in wireless networks.    

Different from existing machine learning (ML) methods, our proposed in-context learning algorithm relies on LLM's inference capabilities. It avoids the complexity of tedious model training and hyper-parameter fine-tuning, which is a well-known bottleneck of many ML algorithms.

Specifically, the proposed algorithm first describes the target task via formatted natural language, and then designs the in-context learning framework and demonstration examples. 

After that, it considers two cases, namely discrete-state and continuous-state problems, and proposes state-based and ranking-based methods to select appropriate examples for these two cases, respectively.   

Finally, the simulations demonstrate that the proposed algorithm can achieve comparable performance as conventional deep reinforcement learning (DRL) techniques without dedicated model training or fine-tuning. 

Such an efficient and low-complexity approach has great potential for future wireless network optimization.




Large language model, in-context learning, network optimization, transmission power control






§ INTRODUCTION


From LTE and 5G to envisioned 6G, wireless networks are increasingly complicated with diverse application scenarios and novel signal processing and transmission techniques, e.g., unmanned aerial vehicle (UAV), vehicle-to-everything (V2X), mmWave and THz networks, reconfigurable intelligent surface, etc <cit.>.   

The constantly evolving network architecture requires more efficient management schemes, and most existing network optimization methods can be summarized into two main approaches: convex optimization and machine learning (ML) algorithms.

Specifically, convex optimization usually needs dedicated problem formulation for each specific task, then transforms the objective function or constraints into convex forms. 

By contrast, ML algorithms, such as reinforcement learning, have lower requirements for problem formulations, but the tedious model training and fine-tuning indicate a large number of iterations <cit.>.

Therefore, these potential issues, e.g., problem-specific transformation and relaxation, hyperparameter tuning, and long training iterations, have become obstacles to further improve the efficiency of next-generation networks.  


Large language models (LLMs) are recently considered revolutionary technologies that have been successfully applied to education, finance, healthcare, biology, etc <cit.>.

The great potential of LLM technologies also provides promising opportunities for network management and optimization <cit.>.

For instance, a promising feature of LLMs is learning from language-based descriptions and demonstrations, which is known as in-context learning.

Compared with convex optimization or conventional ML approaches, in-context learning has multiple advantages <cit.>: 
1) in-context learning relies on LLM's inference process, and it avoids the complexity of dedicated model training and fine-tuning, which is a well-known bottleneck for many ML techniques; 

2) in-context learning allows natural language-based task design and implementation, and the operator can easily formulate the target task using human language and instructions. Such a user-friendly approach can also significantly lower the requirements for professional knowledge when solving specific tasks.  


LLM-enabled wireless networks have recently attracted considerable interest, e.g., 6G edge intelligence<cit.>, network intrusion detection and LLM-enhanced reconfigurable intelligent surface for internet of vehicles (IoV)<cit.>.  

By contrast, this work focuses on optimization problems, and it considers base station (BS) power control as a case study, exploring the potential of using LLM to solve optimization problems.
The power control of cellular networks has been extensively studied with diverse objectives and algorithms, i.e., convex optimization, game theory, reinforcement learning <cit.>, etc.
These studies prove that power control is a fundamental and critical technique to improve energy efficiency, reduce interference, and save power consumption <cit.>.  

Therefore, given such crucial importance, we select BS power control as a case study, and explore the potential of state-of-the-art LLM technologies for wireless network optimization.



In particular, our proposed LLM-enabled in-context learning algorithm first designs a natural language-based task description, i.e., task goal, definition, and rules. 
The formatted task description, along with a set of selected examples, will become the prompt input for the LLM model.

Then, the LLM model can utilize the task description and advisable examples to generate a decision based on the current environment state.

After that, the output decision, environment state, and corresponding system performance will become a new example and be stored in an experience pool. 

Given the next environment state, we will select new advisable examples from the experience pool, serving as references for the next LLM decision-making. 

In addition, we further consider two scenarios, namely discrete-state and continuous-state problems, and propose state-based and ranking-based example selection methods for these two cases, respectively.   

Finally, we evaluate the proposed algorithm with various LLMs, e.g.,  Llama3-8b-instruct, Llama3-70b-instruct, and GPT-3.5 turbo, 
and the simulations prove that the proposed algorithm can achieve satisfactory performance.

The main contribution of this work is that we explored LLM-enabled 
in-context learning for wireless network optimization problems. Specifically, it proposed an in-context optimization technique that learns from the environment interactions without model training and fine-tuning.

The simulation results show that our proposed algorithm enables LLM to learn from previous explorations and examples, and constantly improve the performance on target tasks.

Such an efficient and training-free algorithm has great potential for wireless network optimization and management.




§ SYSTEM MODEL

This section introduces a BS power minimization problem, serving as a case study of the proposed in-context learning algorithm[Various objectives and constraints can be defined for power control problems as summarized in <cit.>, and here we select power minimization as a specific problem formulation.].
Considering a BS with U_b users, the achievable data rate C_b,u between BS b and user u is defined by <cit.>

    0.91!C_b,u=∑_k=1^K_bd_klog(1+ p_b,kh_b,k,uγ_b,k,u/∑_b'∈ B_-bp_b',k'h_b',k',u'γ_b',k',u'+d_kN_0),

where K_b is the total number of resource blocks (RBs) in BS b, d_k is the bandwidth of RB k, 
p_b,k indicates the transmission power of BS b on RB k, 
h_b,k,u defines the channel gain between BS b and user u on RB k, and N_0 is the noise power density. 
For the RB allocation, γ_b,k,u∈{0,1} indicates whether RB k is allocated to the transmission for user u.  
For the interference, B_-b represent the set of adjacent BSs except for BS b, p_b',k'h_b',k',u'γ_b',k',u' defines the inter-cell interference, and we assume orthogonal frequency-division multiplexing is applied to eliminate intra-cell interference.

This work aims to minimize the BS transmission power and meanwhile satisfy the average data rate constraint <cit.>:


    min_P_b   ∑_b∈ B P_b   <ref>
    s.t.    0 ≤ P_b≤ P_max,     
         P_b=∑_k=1^K_b p_b,k,     
       ∑_u=1^U_bC_b,u/U_b≥ C_min,


where P_b is the total transmission power of BS b and P_b=∑_k=1^K_b p_b,k, p_b,k has been defined in equation (<ref>) as the transmission power of RB k, 
P_max is the maximum power, 
U_b is the total number of users, and C_min is the average achievable data rate constraint.
We assume P_b is equally allocated to all RBs, and a proportional fairness method is used for RB allocation, which has been widely used as a classic approach. Then we can better focus on LLM features. 

The control variable in equation (<ref>) is the total BS transmission power P_b, which needs to be dynamically adjusted based on the wireless environment, e.g., current user numbers or user-BS distances, to save power consumption and meanwhile maintain the average data rate.  

Problem (<ref>) has been extensively investigated in existing studies, but this work differs from previous works by presenting a unique view from the perspective of natural language-based network optimization. Specifically, we propose a novel LLM-enabled in-context learning algorithm in the following Section <ref>.  








§ IN-CONTEXT LEARNING-BASED OPTIMIZATION ALGORITHM


In-context learning refers to the process that LLMs can learn from formatted natural language such as task descriptions and task solution demonstrations, to improve the performance on target tasks.

In-context learning can be defined as <cit.>

    D_task×ℰ_t× s_t×ℒℒℳ⇒ a_t,

where D_task is the task description, ℰ_t is the set of examples at time t, s_t is the environment state at time t that is associated with the target task

, ℒℒℳ indicates the LLM model, and a_t is the LLM output. 

For a sequential decision-making problem, we expect the LLM can utilize the initial task description D_task, learn from the example set  ℰ_t, and then make decision a_t based on current environment state s_t of the target task. In the following, we will introduce the design of task description D_task and the selection of example set ℰ_t. 




 §.§ Language-based Task Description

D_task is crucial to provide target task information to the LLM model. In particular, it involves “Task_goal”, “Task_definition”, and extra “Rules”. The following is a detailed task description we designed to prompt the LLM.
[title = Task description for BS transmission power control]

Task goal: You have a decision-making task for base station power control, and you need to select between 4 power levels from 1 to 4.

Task definition: You have to consider the specific user number of each case, which is the “base station user number”.

Following are some examples {Example_set}.

Now I will give you a new condition to solve, the current BS user number is {Num_BS_user}. 

Rules: Now please select from “level 1”, “level 2”, “level 3”, and “level 4” based on the above examples.


In particular, the Task_goal first specifies a “decision-making task for base station power control”, and the goal is to “select between 4 power levels”[Here we select 4 power levels as an example, which can be changed to any number of levels]. Then the Task_definition introduces the environment states we need to consider. For example, this work assumes the total user numbers may change dynamically, and then the LLM has to consider the “base station user number” of each case. In addition, it means that the environment state s_t in equation (<ref>) refers to the total user number U_b in problem (<ref>).

After that, the example set ℰ_t is included by “Following are some examples....”, and we provide a new condition for the LLM to solve, which is associated with the current user number U_b. 

Finally, we set extra reply rules such as “select from ... based on the above examples”, indicating the LLM to focus on the decision-making process.

The above task description provides a template to define optimization tasks by formatted natural language, avoiding the complexity of dedicated optimization model design. It is also user-friendly since the operator can easily add or remove task descriptions without requiring any professional knowledge of optimization techniques.    





 §.§ In-context Learning Framework and Example Design


Examples are of great importance in in-context learning, and they must be carefully selected because: 1) examples serve as crucial references for LLM decision-making, which means the LLM relies on examples to justify its decision; 2) due to the LLM context window size constraint[The context window size indicates the largest number of tokens that can be sent to the LLM], it is impractical to send a large number of examples to the LLM. 

Moreover, there are many optimization problems with continuous environment states, which are very common in wireless networks, e.g., adjusting the BS transmission power based on user-BS distance. Such cases mean that there may be an infinite number of examples, and therefore identifying the most relevant and useful examples becomes challenging.
This work defines an example E by

    E={s, a, r(s,a)}, E ∈ℰ

where s and a are environment state and decision, respectively. Inspired by reinforcement learning, we further define a reward value to evaluate the decision a by

    r=P_target-P_b-β

where P_target is a target power consumption, and P_b has been defined in problem (<ref>) as the total power consumption of BS b. β is a penalty term, which is only applied when constraint  (<ref>) is not satisfied. Then, r provides a comprehensive metric to evaluate the selected decision a under environment state s. 


Fig.<ref> shows the overall design of the proposed in-context learning algorithm for transmission power control. 
Specifically, the above task description D_task, current environment state s_t, and selected examples ℰ_t are integrated as input prompt as defined in equation (<ref>), and then the LLM model will generate a power control decision a_t based on s_t and the experiences in ℰ_t.

Then, the decision a_t is implemented, the achieved data rate C_b,u is collected, and the reward r_t is calculated as equation (<ref>). E_t={s_t, a_t, r_t(s_t,a_t)} becomes a new example in the accumulated experience pool ℰ_pool in Fig.<ref>.   

After that, based on the next environment state s_t+1, a new example set ℰ_t+1 is selected, and the selected examples are inserted into the task description with s_t+1, becoming a new prompt for the LLM model to generate a_t+1. 





 §.§ State-based Example Selection for Discrete State Problems


Selecting appropriate examples is critical for in-context learning since the LLM model learns from existing demonstrations to handle the target task.

For problems with discrete environment states, relevant demonstrations can be easily identified by finding existing examples with the same states in the accumulated experience pool ℰ_pool.  

Considering a target task with environment state value s_target, the set of relevant examples can be identified by

    ℰ_relevant= {E{s,a,r(s,a)}| s=s_target, E ∈ℰ_pool}

where ℰ_pool is the accumulated experience pool in Fig. <ref>. 
Given ℰ_relevant, we can easily select recommended examples with high reward, i.e., top-K examples, and inadvisable examples, e.g., examples with lower reward or violating the minimum data rate constraint.   

In addition, we include a well-known epsilon-greedy policy to balance exploration and exploitation. 

    a={[   Random action selection,             if   rand < ϵ;; LLM-based decision-making,                      else, ].

where ϵ is a predefined value, and rand is a random number between 0 and 1. 
Therefore, the random exploration in equation (<ref>) can constantly explore new examples, and then the LLM model can learn from better relevant examples ℰ_relevant to improve the performance.




 §.§ Ranking-based Example Selection for Continuous State Problems

Compared with discrete-state problems, environments with continuous states can be much more complicated. 
For instance, when using average user-BS distance as an environment state for BS transmission power control with a target task s_target, it is unlikely to find a specific existing example E{s,a,r(s,a)} with s=s_target, since s_target is a random number within the BS maximum coverage distance. This problem may be solved by discretizing the continuous states into some discrete values, but this may still lead to a large number of states or extra errors. 

To this end, we define a new metric ℒ for example selection with continuous states:

    ℒ(E,s_target)=r(s,a)-τ||s-s_target||,


where ℒ(E,s_target) is a comprehensive metric to evaluate the usefulness of E={s, a, r(s,a)} to the decision-making of s_target, 

and ||s-s_target|| is the L^2 norm to define the distance between s and s_target. 

Equation (<ref>) aims to jointly consider the reward and states of example E, and τ is a weighting factor to balance the importance of higher reward r(s,a) and more similar states between s and s_target. 

Specifically, a higher reward r(s,a) indicates that E includes a good action selection a under environment state s, 
and meanwhile lower ||s-s_target|| value means the environment state s in E is more similar to s_target.

Therefore, we use ℒ(E, s_target) as a comprehensive metric, and then the recommended and inadvisable examples can be selected similarly as in Section <ref> by using top-K methods. 





 §.§ Baseline Algorithm


We consider deep reinforcement learning (DRL) as a baseline since it is one of the most widely used ML algorithms to solve network optimization problems <cit.>.
DRL can usually produce satisfactory optimization results, but it requires proper parameter fine-tuning and dedicated model training. 
This work investigates two scenarios, and the Markov decision processes (MDPs) are defined by: 1) for discrete-state problems, the state is defined by user numbers associated with the BS; 2) for continuous-state problems, we use average user-BS distance as a continuous-changing state.
The action is defined by the BS power level, and the reward is shown as equation (<ref>).





§ PERFORMANCE EVALUATION








 §.§ Simulation Settings

We consider three adjacent small base stations (SBSs); the user number of each SBS randomly changes from 5 to 15, and the SBS's coverage is 20 meters. The channel gain applies 3GPP urban network models, and

2 cases are evaluated: 

Case I: Discrete states defined by user numbers of each SBS;
 
Case II: Continuous states defined by average user-SBS distance, which represents 2 kinds of network optimization problems.
Then, the simulation considers two main approaches: 
 
1) LLM-based method includes 3 models: Llama3-8b-instruct, Llama3-70b-instruct, and GPT-3.5 turbo.
Llama3-8b is a small-scale LLM, while Llama3-70b and GPT-3.5 turbo are large models.
Using LLM models with various sizes can better evaluate the capabilities of our proposed algorithms.


2) DRL-based method as introduced in Section <ref>. With dedicated model training, here we consider DRL as an optimal baseline since its capability has been demonstrated in many existing studies <cit.>.






 §.§ Simulation Results

Fig. <ref> shows the simulation results and comparisons, and the metrics include average reward, power consumption, and service quality (indicating the probability of satisfying the minimum average data rate constraint defined in equation (<ref>)).

Firstly, Fig. <ref> presents the system reward and service quality of different LLMs under discrete state space. 

One can observe that both Llama3 LLMs achieve a comparable reward and service quality as the DRL baseline, while GPT-3.5 shows a lower reward and service quality.

Fig. <ref> demonstrates that the proposed in-context learning algorithm and state-based example selection method can provide satisfactory performance for problems with a limited number of environment states.

Then, we consider more complicated scenarios with continuous states defined by the average user-BS distance. Fig. <ref> and <ref> present the reward and average power consumption, respectively. 
All LLM models achieve higher rewards and lower power consumption as the number of episodes increases and finally converge to stable values.

The results demonstrate that LLMs can learn from previous examples and explorations and then improve the performance on target tasks.


In addition, we observe the algorithm performance under different minimum data rate constraints. Fig. <ref>, <ref>, and <ref> present the average reward, power consumption, and service quality, respectively. Here, every value in the results is obtained by taking the average performance of converged episodes of corresponding LLMs as in Fig. <ref> and <ref>. 

As expected, the simulation results show that increasing the minimum data rate constraint leads to lower reward, lower service quality, and higher power consumption. 

Therefore, Fig. <ref>, <ref>, and <ref> demonstrate that the proposed algorithms can adapt to different optimization settings and then adjust their policies to improve the performance on target tasks.

Note that the algorithm performance is also related to specific LLMs. Llama3 represents state-of-the-art LLM designs, while GPT-3.5 is an early LLM model.  

Therefore, it is reasonable that Llama3-8b and Llama3-70b maintain comparable performance as the DRL baseline, while GPT-3.5 turbo presents a worse performance in different tasks. 

For instance, when the minimum data rate constraint is 2 Mbps per user, GPT-3.5 has a 25% lower reward and 20% lower service quality than other LLMs.

In summary, the simulations in Fig. <ref> demonstrate that our proposed in-context learning algorithm can achieve comparable performance as conventional DRL algorithms without dedicated model training and fine-tuning. 



§ CONCLUSION

LLM is a promising technique for future wireless networks, and this work proposes an LLM-enabled in-context learning algorithm for BS transmission power control.

The proposed algorithm can handle both discrete and continuous state problems, and the simulations show that it achieves comparable performance as conventional DRL algorithms. This work demonstrates the great potential of in-context learning for handling network management and optimization problems. 





IEEEtran







 Paper Title: Task-oriented and Semantics-aware Communications for Augmented Reality.




 






 


Task-oriented and Semantics-aware Communications for 
 Augmented Reality
    Zhe Wang,  Yansha Deng1
 Department of Engineering, King's College London, London, UK

 {tylor.wang, yansha.deng}@kcl.ac.uk 
 

 This work was supported in part by UKRI under the UK government’s Horizon Europe funding guarantee (grant number 10061781), as part of the European Commission-funded collaborative project VERGE, under SNS JU program (grant number 101096034). This work is also a contribution by Project REASON, a UK Government funded project under the FONRC sponsored by the DSIT.
 
    August 5, 2024
==================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 

Upon the advent of the emerging metaverse and its related applications in Augmented Reality (AR), the current bit-oriented network struggles to support real-time changes for the vast amount of associated information, creating a significant bottleneck in its development. To address the above problem, we present a novel task-oriented and semantics-aware communication framework for augmented reality (TSAR) to enhance communication efficiency and effectiveness significantly. We first present an analysis of traditional wireless AR point cloud communication framework, followed by a detailed summary of our proposed semantic information extraction within the end-to-end communication. Then, we detail the components of the TSAR framework, incorporating semantics extraction with deep learning, task-oriented base knowledge selection, and avatar pose recovery. Through rigorous experimentation, we demonstrate that our proposed TSAR framework considerably outperforms traditional point cloud communication framework, reducing wireless AR application transmission latency by 95.6% and improving communication effectiveness in geometry and color aspects by up to 82.4% and 20.4%, respectively.




 Metaverse, augmented reality, semantic communication, semantics extraction, point cloud.













§ INTRODUCTION

The metaverse, as an expansive digital universe, is poised to revolutionize various aspects of individual's daily life, primarily through applications such as Augmented Reality (AR), Virtual Reality (VR), and other immersive technologies. These technologies are reshaping areas from virtual conferencing to online gaming, and thus garnering attention from both industry and academia <cit.>.
However, real-time transmission of complex data such as avatars and point clouds poses considerable challenges, especially for avatar-centered applications that demand high bandwidth and rapid interaction. In contrast to earlier video-related applications dealing with images and text, the data-intensive nature of AR applications necessitates numerous packet transmissions, thereby intensifying bandwidth demands <cit.>. In general, a swift response time of less than 15 ms is crucial, which is substantially lower than the delay tolerated in video communication <cit.>. Therefore, redesign the communication framework is pivotal to mitigating time delays and bandwidth limitations in AR applications.


To address the high bandwidth challenges in AR applications, semantic communication has been proposed, aiming to facilitate more efficient communication by prioritizing task-relevant data <cit.>. Early studies in this domain have focused on extracting semantic content within traditional data like text and images <cit.>, using Age of Information (AoI) as a metric to assess the timeliness of data <cit.>. Despite these advancements, the practical application of semantic communication within AR remains unexplored. Moreover, current AoI-based strategies often fail to account for the importance of data sufficiency within emerging AR datasets, revealing a notable research gap. Hence, it is crucial to devise new techniques that effectively integrate semantic communication in AR applications, considering not only the timeliness but also the relevance and sufficiency of the data.







Current AR research mainly uses Head-Mounted Displays (HMDs), focusing on avatar-based applications to reduce computational and transmission loads while ensuring user privacy <cit.>. Social media platforms, such as TikTok and Instagram, also use avatars for AR effects. Interestingly, current research demonstrate that avatars don't hinder social behaviors and even speed up tasks in gaming <cit.>. Virtual fitness and games like Pokemon Go also employ avatars to boost interactions. However, the efficiency of avatar transmission isn't optimal, and bandwidth issues persist. Lagging AR experiences are frequent in weak signal areas, demonstrating the current AR communication model's limitations. Various data types, including avatar skeleton and point cloud, are explored for avatar representation in AR wireless communication <cit.>. Besides, the lack of a universal standard for avatar transmission in AR indicates a research gap, emphasizing the need to develop task-focused and semantics-aware communication for avatar representation in wireless AR.


Inspired by the 3D keypoints extraction method presented in <cit.>, we propose a task-oriented and semantics-aware communication framework in AR (TSAR) for avatar-centric end-to-end communication. In contrast to traditional point cloud AR communication framework that rely solely on point cloud input, our proposed TSAR extracts and transmits only essential semantic information. The contributions of our research can be summarized as follows: 





























  * We propose a task-oriented and semantics-aware communication framework for augmented reality (TSAR) in an avatar-centric conferencing and gaming AR application,

which significantly enhances the efficiency and effectiveness of wireless communication compared to the traditional point cloud communication framework.



  * We innovatively initiate rigorous ablation experiments and define base knowledge and semantic information within our proposed TSAR framework, specifically for avatar-centric conferencing and interactive gaming AR applications for bolstering communication efficiency and enriching the AR experience.


  * 


Compared with the point cloud communication framework, our proposed TSAR framework achieves better client side viewing in terms of color quality, geometry quality, and transmission delay with improvements of up to 20.4%, 82.4%, and 95.6%, respectively. 


The rest of the paper is organized as follows: Section II presents the system model and problem formation, encompassing both the traditional point cloud and the TSAR framework. Section III elaborates on the proposed methods for both semantics and task-level optimization, including semantics extraction with deep learning, task-oriented base knowledge selection, and avatar pose recovery. Section IV outlines the evaluation indicators and experiment performance. Finally, Section V concludes the paper.




§ SYSTEM MODEL AND PROBLEM FORMATION



In this section, we first present the traditional point cloud communication framework for AR applications. Then, we subsequently introduce our novel TSAR framework that incorporates both semantics and task levels, as depicted in Fig. <ref>. Finally, we present the problem formation associated with our proposed framework.





 §.§ System Model



  §.§.§ Traditional Point Cloud Communication Framework

We focus on an avatar-centric AR application in conferencing and gaming, which is the prominent scenery in the metaverse <cit.>. These applications need real-time transmissions of avatar and background model for HMD display in a spatial volume defined by dimensions L, H, and W. For a smooth AR experience, high-resolution point clouds of both avatar and background models are generated, compression and transmitted. Utilizing plugins on the Unity3D platform, such as FM POINTS, allows real-time point cloud transformation of AR scenery. The generated point clouds 𝐏_ar consist of numerous points v_i, which could be written as 𝐏_𝐚𝐫=[.v_i|_0 ^N_pc]^T, where N_pc is the total number of point clouds. Typically, over 1,500 thousand point clouds per frame is needed to represent each 3D object for a satisfactory QoS <cit.>, with each point contains location and color information.
These point clouds then follow a transmission process of point cloud communication framework, as represented in the Fig. <ref> (a). The procedure commences with point cloud downsampling to reduce the volume of data, followed by wireless transmission. Upon reception, point cloud upsampling is performed to restore the full detail of the 3D objects. Finally, these upsampled point clouds are displayed on the Unity3D platform.









  §.§.§ Semantic Information Extraction

Unlike the traditional point cloud communication, which heavily relies on raw data sensing and acquisition, our proposed TSAR framework incorporates a comprehensive method that simultaneously processes and compresses point cloud data at both task and semantics levels. This approach significantly reduces the data size since only the semantics and task-related data are extracted and transmitted.
The process begins with point cloud sensing data, 𝐏_ar, which encapsulates all models within the AR scenery. Here, only the moving avatar's position, requiring a refresh in the subsequent frame, is considered crucial. As a result, the semantics extraction process yields the skeleton information of the avatar, which is crucial for avatar pose representation.

To effectively recover the avatar pose at the client side, the skeleton information, I_i, needs to encompass both the avatar model's position and quaternion rotation <cit.>, which can be symbolically expressed as

    I_i=(l_x, l_y, l_z,r_x, r_y, r_z, r_w)  i ∈ [1,N_a],

where, N_a denotes the total number of avatar skeletons, l_i=(l_x, l_y, l_z) represents a three-dimensional location, and r_i=(r_x, r_y, r_z, r_w) denotes the quaternion rotation. The semantics extraction process, 𝒮(·), is defined as

    𝐃_tsar= 𝒮(𝐃_pc, θ_s)= [I_1, I_2,...,I_N_a]^T,

where θ_s represents all the neural network and setup parameters utilized in the semantics extraction, and 𝐃_tsar symbolizes the entire semantic information encapsulated in the moving avatar's skeleton I_i.

































































































 §.§ Wireless Channel Model

Our wireless communication model is characterized by a Rayleigh fading channel, impacted by additive white Gaussian noise and utilizing an Orthogonal Frequency-division Multiplexing (OFDM) scheme. The OFDM approach divides the physical channel into multiple parallel subchannels. Each subchannel experiences varying levels of noise, leading to different Signal-to-Noise Ratios (SNRs).

Before wireless transmission, Binary Phase-shift Keying (BPSK), a widely adopted modulation technique, transforms analog signals into digital bits. This can be done by modifying the phase of a carrier signal in response to the data values inputted into the system. Once BPSK processing is complete, the resulting bits, denoted as s_n, are ready for transmission.
The multi-path channel within the OFDM scheme can be described as H_c=(.h_n|_0 ^N_c)



where N_c stands for the total number of subchannels in H_c, and h_n signifies the channel gain of the n-th subchannel.

Considering the characteristics of each subchannel, the cumulative SNR of the communication process within channel H_c is expressed as

    SNR=∑_n=1^N_ch_n · s_n^2/∑_n=1^N_cσ_n^2,

where, σ_n^2 represents the noise within the n-th subchannel. Various semantic information is transmitted through different subchannels in the OFDM wireless channel. 















 §.§ Problem Formation

In summary, the overall TSAR framework aims to achieve efficient data transmission for better avatar representation in wireless AR applications and improve the client's QoS through semantics extraction. The primary objective of the framework is to minimize the transmitted data package in wireless communication and maximize the client-side AR viewing experience based on the transmitted semantic information. The objective function of the entire TSAR framework can be expressed as




    𝒫: min _(θ_s)lim _T →+∞1/T∑_t=0^T ∑_i=0^N_a(I_i^t-I_i^t')
        i ∈[1, N_a],

where I_i^t represents the semantic information of the skeleton at time t, and I_i^t' is the received semantic information after the wireless channel, which is detailed in Eq. (<ref>). This equation formulates the problem of minimizing the error in avatar representation during wireless communication.



































































§ PROPOSED METHODS

In this section, we will demonstrate the design principles of TSAR optimization at the semantics and task levels in detail. 


 §.§ Semantics Extraction with Deep Learning








































To efficiently extract semantic information from point cloud, we design a semantics-aware network named SANet to extract the skeleton keypoint information of a moving avatar from the point clouds of AR scenery. 

The SANet operates by using point cloud data 𝐃_pc as input, which represents the 3D coordinates of both the background environment and the moving avatar. This data is then processed by the SANet to extract accurate the avatar skeleton information, crucial for faithfully reproducing the avatar's movements and interactions in the virtual environment.
The design objective of the SANet is to minimize the Euclidean distance (ℒ2) between the semantic information according to predicted skeleton index, denoted as 𝒮(𝐃_pc), and the labeled semantic information of the skeleton location, represented as 𝐃^l_tsar. The interplay between these variables is defined as

    Loss=min(θ_s.)ℒ2(𝒮(𝐃_pc), 𝐃^l_tsar),


where the symbol θ_s stands for all the neural network parameters in the SANet. Training the SANet entails optimizing these neural network parameters in θ_s to minimize the ℒ2 loss.









 §.§ Task-oriented Base Knowledge Selection

In contrast to traditional point cloud wireless communication framework, the TSAR performs the avatar pose recovery differently with the transmission of the base knowledge B at the beginning of AR application. As illustrated in the base knowledge part in Fig. <ref>, the base knowledge encompasses different types of information: avatar skeleton graph 𝒢, moving avatar model, avatar location l_0, avatar model 𝒜_o, stationary background model 𝒮_o, and their respective appearance meshes, ℳ_a and ℳ_s. Whenever a new model appears in the AR scene, the base knowledge at both transmitter and receiver need to be updated synchronously.

Apart from quaternion rotation, current research also employs euler angles to represent rotations in AR scenery. In comparison to quaternion, euler angles offer a simpler and more information-efficient method for representing rotation and calculating root node position when skeleton graph 𝒢 is available to represent node connection. Euler angles define pitch e_p, roll e_y, and yaw e_r to represent rotations around the three primary axes with an associated root point. This approach necessitates less information to reconstruct the avatar's skeleton pose compared to quaternion <cit.>, resulting in smaller data packets and potentially more efficient communication. The transformation from rotation to euler angles can be expressed as

    [[ e_p; e_r; e_y ]]=[[ arctan2(r_y r_z+r_w r_x)/1-2(r_x^2+r_y^2);                arcsin(2(r_w r_y-r_x r_z)); arctan2(r_x r_y+r_w r_z)/1-2(r_y^2+r_z^2) ]] *180/π.


 
To better explore the most suitable base knowledge, we have designed the following ablation experiments for semantic communication with different shared base knowledge and semantic information definitions[Semantic information, as shown in Fig. <ref>, consists of the skeleton information that need to be transmitted in every frame. Conversely, base knowledge encompasses information used primarily in the first frame.], which include the base TSAR framework (TSAR) and Euler angle based TSAR framework (E-TSAR).

TSAR: 

In the base TSAR framework, semantic information for each skeleton is defined as the data pertaining to position and quaternion rotation, as illustrated in Eq. (<ref>). The shared base knowledge comprises the background model, moving avatar model, and their corresponding appearance meshes, which could be represented as

    B={𝒜_o, 𝒮_o, ℳ_a, ℳ_s}.


E-TSAR: 

Based on the TSAR, the semantic information in each skeleton is defined as the position and euler angle in E-TSAR, according to Eq. (<ref>) which could be defined as

    I^e_i = (e_r, e_y, e_p),  i ∈ [1,N_a],

where I^e_i=(e_r, e_y, e_p) denotes the euler angle. The shared base knowledge B^e of E-TSAR encompasses the avatar skeleton graph, shared background model, moving avatar model, avatar location l_0, and their appearance meshes, which is defined as 

    B^e={𝒜_o, 𝒮_o, ℳ_a, ℳ_s, l_0, 𝒢}.









 §.§ Avatar Pose Recovery

The details of the avatar pose recovery involve using the skeleton graph 𝒢 in the base knowledge and the received semantic information to reconstruct the avatar position. The entire avatar position recovery process is shown in Alg. <ref>. Specifically, a recursive algorithm is employed to traverse and assign all skeleton information to the avatar skeleton graph with initialized parameters. However, due to differences in the definition of the semantic information and the shared base knowledge, the avatar poses recovery process has variations between the base TSAR and E-TSAR framework.

On the one hand, the base TSAR framework employs a simple avatar pose recovery method, assigning the avatar model with value based on the skeleton point identity using the received position and quaternion rotation in the semantic information. On the other hand, the E-TSAR framework, which only transmits the euler angle of each skeleton point as semantic information, 

reconstructs the avatar pose by first determining the relationships between the skeleton points in the 𝒢. It then computes the position of each skeleton point by considering its euler angle, avatar location l_0, and the position of its root skeleton within the 𝒢. The relative distance Δ l_(i,i-1) between skeleton node I_i and its root node I_i-1, which can be represented as

    Δ l_(i,i-1) = r_i × l_i-1,

where r_i represents the eular angle of skeleton node I_i, and the actual position of skeleton node I_i will be calculated by combining Δ l_(i,i-1) and the root position of l_i-1, which can be expressed as

    l_i= l_i-1 + Δ l_(i,i-1),

where l_i represents the position of the i-th skeleton node in the avatar skeleton graph 𝒢.






§ EXPERIMENTS

In this section, we compare our proposed TSAR with traditional point cloud framework and conduct numerous experiments to verify the superiority of the proposed TSAR framework.



 §.§ Point Cloud Communication Framework

Adapting the method from <cit.>, we propose a baseline point cloud framework named the Point Cloud. The framework, as depicted in Fig. <ref> (a), incorporates point cloud downsampling with the farthest point sampling algorithm and upsampling with linear interpolation algorithm. 







 §.§ Semantics Extraction

To determine the effectiveness of the designed SANet, and achieve outstanding performance, we train the SANet with various backbone networks, including ResNet, RsCNN, PointNet, SpiderCNN, PointConv, and DGCNN. Similar to <cit.>, we use the mean Average Precision (mAP) as the performance evaluation metric to assess the prediction accuracy of the predicted keypoint probabilities in relation to the ground truth semantic information labels. Fig. <ref> (a) demonstrates that the SpiderCNN-based SANet attains the highest accuracy in skeleton information extraction, exceeding the accuracy of over 96% among the same training epochs. This result demonstrates the SANet achieves significantly more reliable and stable prediction compared to other backbone-based networks.





 §.§ Evaluation Indicators

To thoroughly evaluate the performance of our TSAR framework, we implement metrics to verify the communication efficiency and effectiveness in AR application. These metrics include Point-to-Point (P2Point), Peak Signal-to-Noise Ratio of luminance component (PSNR_y), Mean Per Joint Position Error (MPJPE), and transmission delay. 

P2Point: The P2Point metric is employed to assess the alignment of point cloud data between the transmitter 𝐏_t and receiver 𝐏_r. P2Point quantifies the geometric discrepancy relative to the transmitted and received points acts as a benchmark for gauging the quality of the point cloud.

PSNR_y: The PSNR_y metric facilitates the evaluation of the luminance component of the point cloud. It works by mapping the color of each point in the 𝐏_t to the nearest color in the 𝐏_r and computing the PSNR for the luminance component error.

MPJPE: The MPJPE metric is used to estimate human poses, serving as a tool to evaluate the discrepancy in avatar pose between the transmitter and receiver.

Transmission Delay: The transmission delay metric is used to estimate and ensure service quality. The delay in the entire TSAR comprises different components, such as semantics extraction, wireless communication, and avatar pose recovery.









 §.§ Performance Evaluation
















Fig. <ref> (b) illustrates the significant reduction in transmission delay achieved by our TSAR and E-TSAR frameworks in comparison to the traditional point cloud framework. Despite the inclusion of an additional semantics extraction step, the TSAR and E-TSAR frameworks manage to curtail transmission time by approximately 95.6% and 97% compared with point cloud framework, respectively. This remarkable efficiency is due to the TSAR framework transmitting a mere 25 skeleton points, as opposed to the 2,048 points required by point cloud communication. This considerable reduction in transmission packages significantly cuts down bandwidth usage. These impressive outcomes underscore the viability of our TSAR in facilitating high bandwidth AR applications in dynamic wireless environments.


Fig. <ref> (a) and Fig. <ref> (b) plot the P2Point and PSNR_y results, respectively. The P2Point metric assesses geometric discrepancies throughout the complete AR scene, not limited to the avatar or any specific model. As the SNR increases, the P2Point results is ordered as E-TSARTSARPoint Cloud, and the pattern becomes more pronounced with SNR decrease. Specifically, the E-STAR decrease the P2Point for about 82.4% compared with point cloud communication, indicating significant distortion in avatar positions when using the point cloud framework.

Conversely, the PSNR_y metric measures the differences in the color aspect between the transmitter and the receiver. Here, both TSAR and E-TSAR surpass the traditional point cloud framework by over 20.4% at the best SNR scenery. With the PSNR_y value ordering as E-TSARTSARPoint Cloud, it is evident that TSAR and E-TSAR, due to their ability to create compact and grouped point clouds, demonstrate their ability in minimizing distortion effectively. This demonstrates TSAR's high effectiveness in preserving color quality and its potential to improve wireless AR applications.

Fig. <ref> (c) plots the MPJPE results which measure the variance in avatar skeleton position between receiver and transmitter. The TSAR and E-TSAR notably surpasses the point cloud communication framework across all SNR scenarios. Particularly, TSAR achieves higher avatar pose recovery in higher SNR scenarios with an 83.3% decrease in MPJPE compared to point cloud framework, highlighting its superior effectiveness in avatar transmission. However, in lower SNR conditions, E-TSAR outperforms TSAR in MPJPE difference, suggesting that utilizing shared avatar model as base knowledge helps in preserving movements within the avatar's capabilities, thereby enhancing AR avatar representation from distortion.








§ CONCLUSION

This paper has proposed a task-oriented and semantics-aware communication framework for Augmented Reality that aims to achieve reliable and low-latency wireless communication in the AR application. By defining the semantic information and base knowledge in the AR, our proposed TSAR have successfully 
reducing wireless AR application transmission latency by 95.6% and improving communication effectiveness 

by up to 82.4% and 20.4%, respectively.




Our future work includes exploring the scalability of our framework to support larger virtual environments and investigating the integration of other semantic features to improve the accuracy and efficiency of communication.




ieeetr




 Paper Title: Analysis of Functional Insufficiencies and Triggering Conditions to Improve the SOTIF of an MPC-based Trajectory Planner.
















Analysis of Functional Insufficiencies and Triggering Conditions to Improve the SOTIF of an MPC-based Trajectory Planner
    Mirko Conrad[samoconsult GmbH, Französische Str. 13, 10117 Berlin, Germany], Georg Schildbach[University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany corresponding author ()]
    July 2024
=============================================================================================================================================================================================





§ ABSTRACT

Automated and autonomous driving has made a significant technological leap over the past decade. In this process, the complexity of algorithms used for vehicle control has grown significantly. Model Predictive Control (MPC) is a prominent example, which has gained enormous popularity and is now widely used for vehicle motion planning and control. 
However, safety concerns constrain its practical application, especially since traditional procedures of functional safety (FS), with its universal standard ISO 26262, reach their limits. Concomitantly, the new aspect of safety-of-the-intended-function (SOTIF) has moved into the center of attention, whose standard, ISO 21448, has only been released in 2022.  Thus, experience with SOTIF is low and few case studies are available in industry and research. 
Hence this paper aims to make two main contributions: (1) an analysis of the SOTIF for a generic MPC-based trajectory planner and (2) an interpretation and concrete application of the generic procedures described in ISO 21448 for determining functional insufficiencies (FIs) and triggering conditions (TCs). Particular novelties of the paper include an approach for the out-of-context development of SOTIF-related elements (SOTIF-EooC), a compilation of important FIs and TCs for a MPC-based trajectory planner, and an optimized safety concept based on the identified FIs and TCs for the MPC-based trajectory planner.



§ KEYWORDS

Automated Driving System (ADS) ,  
Autonomous Vehicles ,  
Trajectory Planning ,  
Model Predictive Control (MPC) ,  
Functional Safety (FS) ,  
Safety of the Intended Functionality (SOTIF) ,  
Functional Insufficiencies (FIs) ,  
Triggering Conditions (TCs) ,  
SOTIF-related Element out of Context (SOTIF-EooC)




§ INTRODUCTION


Since the beginning of the 21st century, automated and autonomous vehicles have become a major subject of research across the globe, in academia and industry <cit.>. The technological progress is fueled by ever more powerful hardware for computation, communication, and sensing. Another relevant factor is the rapid advances in mathematical algorithms, especially in Artificial Intelligence (AI), Machine Learning (ML), and Numerical Optimization (NO). A plethora of new and powerful methods have been developed and successfully demonstrated in actual vehicles on the road, among them Model Predictive Control (MPC). 

MPC is a popular computer-based control method <cit.> that has been frequently tested in automated driving system (ADS) prototypes <cit.>. Thus far, however, it has not been considered a viable option for series production. The main reason is its relatively high complexity, causing relatively high efforts in the development process and increasing the computational hardware requirements. Arguably, an even bigger challenge results from safety concerns related to MPC. These concerns are similar to other state-of-the-art methods of AI, ML, and NO. 

Due to the complexity of these methods, and the complexity of the corresponding automated driving tasks, Functional Safety (FS), which is concerned with safety in the presence of malfunctions, is no longer the single most important aspect of safety. Instead, Safety of the Intended Function (SOTIF), which is concerned with the  safety of the nominal function even if all system elements work as intended, becomes equally important. The intention of this paper is hence to present a contribution towards a viable SOTIF concept for an MPC-based motion planning module in automated driving systems (ADSs).



 §.§ Model Predictive Control


Over the past decades, MPC has developed into a mainstream approach for the control of complex and safety-critical systems, including ADSs. The basic idea is to formulate and solve a finite-horizon optimal control problem (FHOCP) as a numerical optimization program in real time <cit.>. The FHOCP contains a model of the system and naturally integrates constraints on the model states and control inputs. It can be used for motion planning <cit.> and trajectory tracking control <cit.>, where feedback is introduced by a receding-horizon implementation. It can also be used as a hybrid module, covering some or all aspects of planning and tracking control <cit.>. It has even been used to cover some aspects of tactical planning, such as lane change decisions <cit.>.

The current trend is to use MPC as a motion planner, such as in the commercial product ProDriver by Embotech <cit.>. The goal is to integrate as many aspects as possible into the motion planning module, in order to slim down the module interfaces. Interfaces always require additional engineering efforts and they are a common source of functional, and thus also safety problems. The MPC module should thus be capable of making tactical decisions, by introducing integer variables, and ideally should also provide a basic level of vehicle and actuator control.

Recent research in the context of MPC for ADSs has begun to focus on interaction-aware motion planning <cit.>. The basic idea is to consider the predictions of dynamic objects in the environment not as invariably fixed, but as interdependent with the other objects and also dependent on the actions of the ego vehicle. Hence the future trajectories of dynamic objects are included in the model-based predictions in the MPC. The interactions themselves can be modeled by different approaches, such as game theory <cit.> or neural networks <cit.>. 

A common idea is to represent the intention of other drivers as modes <cit.>, corresponding to lane keeping or a lane change, for example. The remaining uncertainty regarding the exact trajectory of the dynamic object can be considered as stochastic noise. Another noteworthy approach is Branch MPC <cit.>, where the modes of each dynamic object and the ego vehicle may change at several specific instances over the prediction horizon.



 §.§ Safety of the Intended Function (SOTIF)


SOTIF is a comparatively new branch of safety that is becoming increasingly important for SAE driving automation levels 3 or higher. The first edition of the applicable SOTIF standard ISO 21448 has only been published in 2022 <cit.>. 
Due to its relatively recent introduction, scientific literature on the application of SOTIF to ADSs is still  sparse. 
Zhu et. al. <cit.> emphasize that the research on the SOTIF of automated and autonomous driving systems is still in its infancy. Their paper focuses on safety analysis methods that could be applied to SOTIF, but it does not cover specifically the topic of identifying and analyzing functional insufficiencies (FIs) or triggering conditions (TCs).


Qidong et. al. <cit.> state that the identification and systematic analysis of FIs and TCs remains a major challenge for the SOTIF. Using an Adaptive Cruise Control (ACC) system as an example, the authors propose a scenario-based analysis method of SOTIF triggering conditions. In this approach, the TCs are analyzed according to elements of SOTIF scenarios that are formed in view of FIs. 
The approach is then applied to the 'Sense' portion of the ACC system. The 'Plan' part is not addressed in this paper.



Junfeng et. al. <cit.> apply the System-Theoretic Process Analysis (SPTA) method to study the SOTIF of a Lane Keep Assistance (LKA) system. The study intends to verify the feasibility of STPA as part of the SOTIF lifecycle.
The paper proposes a set of TCs for the LKA, but not all of them seem to be TCs in the sense of ISO 21448. A subset of these TCs will be used as input for the SOTIF analysis of the MPC-TP. 
In the analysis, there is also no specific focus on the 'Plan' part.



 §.§ Content and Contributions


This paper describes a possible approach for the analysis of FIs and TCs, with the goal of improving the SOTIF of a generic planning module for an ADS. The trajectory planning module is based on the Model Predictive Control (MPC) approach. It is assumed that the MPC-based trajectory planner (MPC-TP) can be integrated into different vehicle-level functions and that it is re-usable for different vehicles within a specified vehicle class. Hence it is treated as a SOTIF-related element out of Context (SOTIF-EooC), which is a particular novelty compared to the existing literature.

The main objective of this paper is to demonstrate how the framework of ISO 21448 <cit.> can be used for improving the SOTIF of a modern, state-of-the-art planning algorithm. The system architecture and functional specifications are outlined in Section <ref> and the details of the MPC-TP are provided in Section <ref>. The relevant SOTIF background is outlined in Section <ref>. The main part of this paper is the SOTIF analysis in Section <ref>, which describes the application of various methods and techniques from ISO 21448 <cit.> towards the identification of FIs and TCs for the MPC-TP. The analysis leads to a refined specification and design of the MPC-TP, as described in Section <ref>. The conclusions of this work are presented in Section <ref>. The main contributions of this paper can be summarized as follows:


  * Application of the abstract / generic framework of the new SOTIF standard ISO 21448 to a realistic case study, with a focus on early parts of the SOTIF lifecycle;

  * Interpretation and application of generic methods in ISO 21448 for the determination of FIs / TCs, in the context of a specific use case;

  * Derivation of a structured list of FIs / TCs as a consolidated result of the SOTIF analysis (in a tabular format);

  * A refinement of the notion of an out-of-context development in the scope of SOTIF.










§ INITIAL CONCEPT FOR VEHICLE LEVEL CONTROL SYSTEM






The starting point for the design of the planning module as a SOTIF-SEooC are assumptions about the encompassing vehicle control system into which the module is to be embedded, as well as the related interfaces. Furthermore, specifications are needed about the overall functionality and the operational design domain (ODD). 



 §.§ Functional Architecture


The functional architecture of a generic ADS is  shown in Figure <ref>. The MPC-TP is shown in the middle (red). The task of the MPC-TP is to compute a reference trajectory, which is then used by the trajectory controller (blue) and lower-level modules to control the vehicle's actuators. The actuators are an integral part of the chassis, including, in particular, the drivetrain and braking systems for longitudinal motion control and the steering system for lateral motion control. 

The MPC-TP receives information from centralized estimation modules (green), which may serve also other functions and/or other modules in the architecture <cit.>. The key point is that they are developed independently of the MPC-TP. These modules provide a local map, including the topology and road geometry and information about the infrastructure, such as traffic signs (Map Fusion). They also include the relative position of the vehicle on the map, the vehicle's dynamic states, a list of relevant vehicle parameters, and diagnostic information about the vehicle's sensors and actuators (Ego Motion Fusion). Furthermore, a list of relevant traffic objects in the vicinity of the vehicle is provided, including their classification, position, speed, and other states, as well as a situation analysis and prediction(s) about (possible) future evolution of the situation (Object Fusion and Prediction). The sensors also deliver important information about the environment, such as ambient temperature or road surface (Environment Fusion). Finally, the MPC-TP may receive high-level commands by the human driver or the ADS, such as the reference speed, which lane or road to use, or where to take a turn (Tactical Planner).









	






 §.§ Encompassing Vehicle Control System


The MPC-TP is treated as a SOTIF-related element out of context (SOTIF-EooC), as detailed in Section <ref>. Hence it shall be possible to integrate the MPC-TP into different the SOTIF-related systems as well as different vehicle types. For this paper, it is assumed that the MPC-TP can be used in the context of the following systems for a given class of passenger cars:

    
  F1) Energy Efficient Lane Centering and Adaptive Cruise Control (E2LCACC): E2LCACC is a SAE level 2 function that combines the classic Lane Centering (LC) and Adaptive Cruise Control (ACC). It performs the lateral and longitudinal control of the vehicle, in order to keep the vehicle on a reference path in its lane and to follow the preceding vehicle. E2LCACC contains AI-based elements for energy efficient driving, in particular to determine the optimal speed profile. 
    
  F2) Active Lane Change Assist (ALCA): ALCA is SAE level 2 or 3 function designed to help drivers changing lanes safely by automatically steering the vehicle into the adjacent lane. ALCA operates in combination with E2LCACC and also provides longitudinal and lateral control of the vehicle during overtaking. The driver can issue lane change requests to the left or the right, e.g., by operating the indicator lever. The ALCA then checks whether a safe lane change is possible within 10 seconds. If it is possible, it performs the lane change; if it is not possible, it keeps the vehicle in the current lane and continues with E2LCACC.
    
  F3) Highway Pilot (HP): The HP is a SAE level 4 function that takes full control of the vehicle during highway driving. Only high level objectives are provided by the user, such as the destination, the desired route, the targeted arrival time, or the driving style. The driving itself, including all decisions, is performed autonomously by the HP.





 §.§ Operational Design Domain (ODD)


The vehicle-level functions F1, F2, F3 are only to be used within the conditions and limits of the ODD () specified in Table <ref>. The ODD documents the specific conditions under which the ADS is designed to function <cit.>. Its description is based on the structure provided in DOT HS 812 623 <cit.>.





 §.§ Assumptions about the Functionality and the Vehicle Class


Since the MPC-TP is considered as a SOTIF-EooC, specific assumptions are required about its integration into the vehicle and the vehicle-level functions F1, F2, F3, as well as its interfaces to other modules. An overview of the upstream and downstream modules can be gathered from Figure <ref>. The perception modules make use of appropriate sensors and communication infrastructure. The control modules are connected with the relevant vehicle actuators. It is beyond the scope of this paper to specify these modules here in detail. A list of the most relevant specifications for the ego vehicle (EgoV) and the module interfaces is provided below.


    
  Spec:ODDj) The EgoV belongs to a specific class of passenger vehicles, consisting of a finite number of vehicle types.
    
  Spec:ODDk) All possible EgoV types are known at the time of the MPC-TP development process, including their physical design and specifications, chassis and suspension systems, and chassis control systems (Anti-lock Braking System, Electronic Stability Control, etc.).
    
  Spec:ODDl) Further parameters of EgoV (such as the vehicle mass, location of the center of gravity, moments of inertia, and tire parameters) are known to be within specified value ranges.
    
  Spec:ODDm) The loads inside the EgoV are within the limits specified by the respective vehicle type.
    
  Spec:ODDn) There are no external loads attached to the EgoV, such as a trailers, roof racks, or any type of load carriers. 




§ INITIAL CONCEPT OF THE MPC-TP


The specifications for the MPC-TP are based on the overall architecture in Figure <ref>. The MPC-TP is (initially) given by the single block displayed in Figure <ref>. The specifications concern the input signals, the output signals, and the contents of the block itself, and they are presented along this structure in the remainder of this section.





 §.§ Input Interface / Perception


For the interface to the map fusion module, the 6 layers road model specified in ISO 34503 <cit.> is considered. See Figure <ref> for an illustration.






  ) Map Fusion: The MPC-TP receives a layered map every 50 ms, including

    
  Spec:MapFusiona) geometry and topology of the road, lane and road boundaries (layer 1),
    
  Spec:MapFusionb) traffic signs, speed limits, traffic guidance, construction barriers (layer 2),
    
  Spec:MapFusionc) temporal modifications, e.g., due to construction sites (layer 3).


For the sake of the SOTIF-EooC, it is assumed that all of this map data is always fully accurate. The MPC-TP must plan its trajectories such that they conform to the applicable traffic rules (e.g., speed limit, no passing zone).



  ) Object Fusion and Prediction: The MPC-TP receives a complete list of dynamic objects in its relevant vicinity every 50 ms, including

    
  Spec:ObjectFusiona) other traffic participants (layer 4),
    
  Spec:ObjectFusionb) animals and other temporary obstacles, like dropped cargo, rocks, etc. (layer 4).


All dynamic objects include additional information about their classification (vehicle, truck, motorcycle, pedestrian, animal type, obstacle, etc.), position, velocity, and heading direction, if applicable. Furthermore, a reliable prediction for all of these objects is provided over the next 6 seconds. The uncertainty in these predictions is accounted for by the use of intervals (e.g., for the velocity) or bounding boxes (e.g., for the position).

No assumptions are made on specific sensors, their field-of-view, or their operational status. All assumptions are based on the interfaces to the sensor fusion modules. If the information flow on these interfaces cannot be guaranteed, the vehicle-level function will hand over to the driver (SAE levels 1,2,3) or transfer the vehicle to a safe state (SAE levels 4,5).



  ) Ego Motion Fusion: The MPC-TP receives relevant information about the states and parameters of the ego vehicle every 50 ms, along with a with maximum estimation error (marked with a “Δ”):

    
  Spec:EgoFusiona) the relative position of the ego vehicle on the map x,y (Δ x, Δ y),
    
  Spec:EgoFusionb) the ego vehicle velocity v_lon,v_lat (Δ v_lon,Δ v_lat),
    
  Spec:EgoFusionc) the ego vehicle's global heading angle ψ (Δψ),
    
  Spec:EgoFusiond) the ego vehicle's acceleration a_lon,a_lat (Δ a_lon,Δ a_lat),
    
  Spec:EgoFusione) the vehicle's mass m (Δ m),
    
  Spec:EgoFusionf) the vehicle's moment of inertia around the vertical axis I_z (Δ I_z),
    
  Spec:EgoFusiong) the position of the center of gravity relative to the midpoint of the rear axle x_cog, y_cog, z_cog (Δ x_cog, Δ y_cog, Δ z_cog),
    
  Spec:EgoFusionh) the cornering stiffness of the front and the the rear tires C_α,f, C_α,r (Δ C_α,f, Δ C_α,r).



  ) Tracking Controller: Further information about the ego vehicle, in particular about the states of the lower-level actuators, are provided to the MPC-TP through the Tracking Controller every 50 ms:

    
  Spec:TrackControla) the front wheels' steering angles δ_f,l, δ_f,r,
    
  Spec:TrackControlb) the current engine torque T_eng,
    
  Spec:TrackControlc) the total gear ratio i_gear,
    
  Spec:TrackControld) the effective brake torque T_brake.
    

Actuator failures, controller failures or performance degradations are not considered in this paper, as the focus is on SOTIF instead of functional safety.


  ) Environment Fusion: The MPC-TP receives accurate information about the environmental conditions every 50 ms, including

    
  Spec:EnvironmentFusiona) conditions of the road surface, in particular the friction coefficient μ (Δμ) (layer 1),
    
  Spec:EnvironmentFusionb) weather conditions and lighting conditions, in particular the side wind speed v_wind (Δ v_wind) (layer 4).


In reality, environment parameters, such as the friction coefficient or the side wind speed, are variable over space and/or time. For simplicity, however, the architectures presumes the communication of constant values, which characterize the entire drivable area of the ego vehicle over the relevant prediction time, i.e., in this case 6 seconds. The tolerance thus includes the measurement uncertainty as well as the true temporal and/or spacial variations, if applicable.



  ) Tactical Planner: The MPC-TP receives tactical choices from the Tactical Planner every 50 ms, containing

  
  Spec:TacticalPlanninga) the desired target lane (e.g., the current lane, the left or right lane), 
  
  Spec:TacticalPlanningb) a desired reference speed (e.g., because the vehicle is about to leave the highway on the next exit),
  
  Spec:TacticalPlanningc) optional information about the desired lateral position on the target lane (e.g., deviation from the lane center line),
  
  Spec:TacticalPlanningd) optional information about desired distances to other vehicles or objects (e.g., the desired gap to the leading or trailing vehicle).



All requests by the Tactical Planner are facultative. For example, if the Tactical Planner requests a lane change, it is the task of the MPC-TP to evaluate if this lane change is feasible and safe, given the dynamics of the vehicle and the traffic situation (dynamic and static obstacles, traffic rules, etc.). If it is not feasible or not safe, the MPC-TP resorts to a trajectory that keeps the EgoV on its current lane. Similarly, the desired distances in (Spec:TacticalPlanningc,d) are for performance purposes only, and not required for safety.

The tactical choices are generated by the active driving function. For example, for the E^2LCACC the unique command is always to keep the current lane, for the ALCA the lane change commands are issued by the driver, and for the HP the commands are issued by a dedicated software module.





 §.§ Output Interface / Reference Trajectory


The main output of the MPC-TP is a reference trajectory, as shown in the architecture in Figure <ref>. The reference trajectory consists of (i) a trajectory header and (ii) a trajectory body. The trajectory header contains a sequential number, the generation time, a validity time interval, and a checksum.



  ) Trajectory Body: The main part of the reference trajectory is the trajectory body. It is represented by N_p-1 linear segments, as illustrated in Figure <ref>. Each segment, numbered with i=1,…,N_p-1, carries the following data:

    
  Spec:TrajectoryBodya) x_i,y_i: the planar coordinates of the starting point of the segment,
    
  Spec:TrajectoryBodyb) ψ_i: the heading angle of the segment,
    
  Spec:TrajectoryBodyc) v_i: the reference speed of the vehicle along the segment,
    
  Spec:TrajectoryBodyd) δ_i: the reference steering angle along the segment,
    
  Spec:TrajectoryBodye) l_i: the length of the segment.
    
  Spec:TrajectoryBodyf) In addition, the final point of the trajectory is given in terms of its planar coordinates x_N_p,y_N_p.


Note that the trajectory body data are partially redundant. This is mainly to avoid unnecessary and mirrored re-calculations within the MPC-TP.





 §.§ Specification and Design of the MPC-TP


This case study considers a trajectory planner based on Model Predictive Control (MPC). A generic MPC formulation with a mainstream application to highway driving is used, in order to make the SOTIF concept in this paper as generally applicable as possible. It should be remarked, however, that numerous variations of MPC-based planning algorithms have been proposed in the recent literature <cit.>. Moreover, commercial modules for trajectory planning with MPC have been made available, such as the software ProDriver by Embotech <cit.>. It assumes a similar functional architecture as the MPC-TP in this paper, with a sampling time of 100 ms. 

The computations of the MPC-TP are based on an internal prediction model. Following <cit.>, a  dynamic bicycle model with a linear tire model (LDBM) is selected and discretizes with an appropriate numerical integration method (e.g., Runge-Kutta):

    𝐱_t+1=f(𝐱_t,𝐮_t) .

The model is augmented with a suitable patch to circumvent the singularity of the tire forces at the origin. Here t=0,1,2,… denote the discrete time steps, corresponding to multiples of the fundamental sampling time t_s=50 ms. The vector 𝐱_t∈ℝ^6 includes the (model) states at step t, namely

    
  * s_t,e_t: coordinates of the vehicle's reference point in road-aligned (Frenét) coordinates, 
    
  * ψ_t,ω_t: yaw angle and yaw rate of the vehicle,
    
  * v_t,u_t: longitudinal and lateral velocity of the vehicle.

The vector 𝐮_t∈ℝ^2 contains the (control) inputs at step t, namely

    
  * δ_t: the steering angle of the front wheel,
    
  * M_t: the torque of the drivetrain at the wheels, used to accelerate or decelerate the vehicle.

Furthermore, a variable for the reference lane n_t is used to indicate the lane decision for the vehicle at time t. It is assumed that the lanes are numbered from right to left, starting with 1 and counting up to the total number of lanes. The set of available lanes at step t is denoted ℕ_t.

Based on the prediction, the concept of the MPC-TP is to set up and solve the following Optimal Control Problem (OCP) in each time step, using an appropriate numerical optimization solver:

    n_t,𝐮_t,𝐱_tminimize      ℓ_s(s_N_p)+∑_t=1^N_pℓ_e(e_t,n_t)+ℓ_o(s_t,e_t)+ℓ_u(δ_t,M_t) ,
    s.t.      𝐱_t+1=f(𝐱_t,𝐮_t)    ∀ t=0,1,…,N_p-1 ,
       𝐱_0=𝐱̅_0 ,
       𝐧_t∈ℕ_t    ∀ t=1,2,…,N_p ,
       𝐮_t∈𝕌    ∀ t=0,1,…,N_p-1 ,
       𝐱_t∈𝕏^(s)_t ,  𝐱_t∈𝕏^(t)_t    ∀ t=1,2,…,N_p-1 ,
       𝐱_N_p∈𝕏_f .

The objective function (<ref>a) comprises the following cost terms:

    
  * ℓ_s(s_N_p): rewards the progress along the road made by the vehicle over the prediction horizon;
    
  * ℓ_e(e_t,n_t): penalizes the lateral deviation of the vehicle from the center line of the corresponding reference lane;
    
  * ℓ_o(s_t,e_t): repels the vehicle from static and dynamic obstacles, e.g., by means of a potential field;
    
  * ℓ_u(δ_t,M_t): favors a low input usage and smooth changes in the inputs, e.g., for driver comfort and energy efficient driving.

These cost terms are multiplied by individual tuning weights, which are selected during the control design process.

The initial state 𝐱̅_0 of the LDBM (<ref>b,c) is obtained from the module for Ego Motion Fusion. The lane variables n_t over the prediction horizon in (<ref>d) are either a decision variable of the OCP (as for the HP), or they can be pre-determined before solving the OCP (as for E2LCACC and ALCA). Equations (<ref>e,f,g) represent the input constraints, state constraints, and terminal constraints, respectively. The state constraints are split into two sets:

    
  * The safety constraints 𝕏^(s)_t avoid collisions of the EgoV with static and dynamic objects, most importantly other vehicles, called target vehicles (TgtVs). Furthermore, the constraints ensure the validity of the tire model, by restricting the tire forces to remain within the admissible bounds of the linear tire regime (i.e., no slip in the tire-road contact patch).
    
  * The traffic rule and comfort constraints 𝕏^(t)_t represent the constraints from traffic rules (e.g., the EgoV must stay within its lane, the EgoV must respect the speed limit) and passenger comfort (e.g., no excessive acceleration or jerk).

The safety constraints are essential and cannot be relaxed, whereas the traffic rule and comfort constraints can be relaxed in emergency situations. Furthermore, in emergency situations, the LDBM in (<ref>) can be replaced with a nonlinear dynamic bicycle model (NDBM), which uses a Pacejka tire model instead of the linear tire model. By these modifications, the MPC-TP may temporarily operate the vehicle close to its dynamic limits and against the traffic rules, e.g., driving in between two lanes, or without considering passenger comfort.



  ) Further specifications related to the MPC formulation include the following:

    
  Spec:MPCa) The fundamental sampling time is t_s=50 ms.
    
  Spec:MPCb) The planning horizon is N_p=120 steps, which corresponds to a prediction time of T_p=6 s.
    
  Spec:MPCc) The input constraint set 𝕌 represents all limits of the steering, engine and braking systems, as well as the relevant limits of their rate-of-change.
    
  Spec:MPCd) The state constraint sets 𝕏_t are time-dependent and prevent collisions with static and dynamic obstacles, with the exception of vehicles driving behind the ego vehicle on the same lane. Moreover, the state constraints keep the vehicle away from its dynamic limits and ensure satisfaction of the traffic rules, e.g., by preventing forbidden lane changes or violation of the speed limit.
    
  Spec:MPCe) The terminal constraint 𝕏_f corresponds to a safe state of the vehicle in which it can remain indefinitely (e.g., at standstill), or it represents a safe exit strategy (e.g., maximum deceleration until standstill). Its purpose is to ensure the recursive feasibility of the OCP.



The numerical computations of the MPC-TP comprise two main steps:

    
  * setup of the OCP problem,
    
  * solution of the OCP problem.

Step 1 is to parse the OCP into the standard form of a nonlinear programming problem, which is realized by handwritten code. Step 2 is to solve the nonlinear program using a generic optimization solver, for which a generous selection of software packages is available <cit.>. 


    
  ) Besides the optimal trajectory, the optimization solver returns an exit flag q_exit in exactly one of four classes: 

    
    
  q_exit=1: A (local) minimum is found with an acceptable tolerance level.
    
  q_exit=2: A feasible point is found, but without an optimality property (e.g., because the gradient / search direction becomes too small or the solver has reached the maximum number of iterations / solution time).
    
  q_exit=3: The solver detects that no feasible point exists (`certificate of infeasibility').
    
  q_exit=4: A feasible point is not found, but without a certificate of infeasibility (e.g., because the solver has reached the maximum number of iterations / solution time), or an identified or unidentified solver error occurred (e.g., due to insufficient memory).



In reality, most solvers provide further diagnostics of the results, i.e., their exit flag is more detailed than the four classes above. The requirement (Spec:ExitFlag) states that it must be possible to map the true exit flag of the solver uniquely to one of the indicated classes. Moreover, it is assumed that the exit flag correctly characterizes the result of the solver. This can be verified by a separate solution checker. Cases where the exit flag is incorrect are not considered further as part of the SOTIF analysis. They represent software fault and hence belong to the realm of FS.



 §.§ Hazards at the Vehicle Level, Safety Goals, and SOTIF Requirements


For the SOTIF analysis in this paper, the following hazards at the vehicle level are considered:


    
    
  ) The ego vehicle leaves the assigned lane.
    
  ) The ego vehicle collides with other vehicles or obstacles.

These hazards lead directly to the specification of the following safety goals:


    
    
  ) The ego vehicle shall not leave the assigned lane(s).
    
  ) The ego vehicle shall not collide with other vehicles or obstacles.


As part of a top-down safety lifecycle, these safety goals would be broken down recursively into more and more detailed safety requirements until one arrives at safety requirements for the component in question. Safety requirements for an ADS comprise both functional safety requirements and SOTIF-related safety requirements (or SOTIF  requirements in short).
Such a refinement of safety requirements does not take place when developing a SOTIF-EooC. Instead, safety requirements must be assumed for the out-of-context element under consideration. The SOTIF-SEooC's fulfillment of these assumed safety requirements represents its contribution to achieving the original safety goals.

For the MPC-TP, we assume the following SOTIF-related safety requirements: 


    
  ) The MPC-TP shall provide a reference trajectory within t_c=30 ms.[The computation time is less than the sampling time t_s=50 ms, in order to account for additional communication delays.]
    
  ) The reference trajectory provided by the MPC-TP has to be consistent regarding the redundant information in the trajectory body. 
    
  ) The reference trajectory provided by the MPC-TP has to be admissible, in the sense that it must satisfy all relevant actuator constraints:

    
  SR:Admissiblea) The minimum / maximum longitudinal acceleration of the vehicle is a_lon,min=-10 m/s^2, a_lon,max=+3 m/s^2.
    
  SR:Admissibleb) The minimum / maximum velocity of the vehicle is v_min=0 km/h, v_max=130 km/h.[The minimum velocity assumption means that the EgoV may not drive in reverse. A maximum velocity is required by the limited forward view of the vehicle's sensor suite.]
    
  SR:Admissiblec) The maximum front wheel steering angle of the vehicle (in each direction) is δ_max=34^∘.
    
  SR:Admissibled) The maximum front wheel steering rate (in each direction) is γ_max=68^∘ 1/s.
    
  SR:Admissiblee) The maximum lateral acceleration of the vehicle (in each direction) is a_lat,max=5 m/s^2.



Similar to the upstream perception modules, no detailed technical specifications have made for the downstream Tracking Controller (TC). As it is developed independently, responsibilities have to be clearly allocated and distributed between the MPC-TP and the TC. Thus requirements have to be specified for the interface, i.e., in this case, the reference trajectory. The approach proposed in this case study is to require “drivability” based on a nonlinear dynamic double-track model (NDDM):

    
  ) The reference trajectory provided by the MPC-TP shall be drivable, i.e., it must be possible to track it with a NDDM using some arbitrary controller, such that all of the following conditions are satisfied:

    
  SR:Drivablea) The initial condition of the NDDM corresponds to the measured initial condition of the vehicle.
    
  SR:Drivableb) The parameters of the NDDM correspond to those of the vehicle (mass m, moment of inertia I_z, position of the center of gravity x_cog and y_cog, front and rear cornering stiffness C_α,f and C_α,r).
    
  SR:Drivablec) The tires of the NDDM do not saturate, i.e., there is no slip in the tire-road contact patch, for the worst-case friction coeffient μ-Δμ. This means that the tire forces remain inside the linear regime at all times.
    
  SR:Drivabled) The bank angle is between β-Δβ and β+Δβ.
    
  SR:Drivablee) The side wind speed is between v_wind-Δ v_wind and v_wind+Δ v_wind.
    
  SR:Drivablef) All existing actuator constraints are respected: communication delays, actuator dynamics, actuator limits, and limits of the rates of change.
    
  SR:Drivableg) The maximum lateral offset from the reference trajectory is e_max=5 cm.
    
  SR:Drivableh) The length of each reference trajectory is such that it takes at least 5 s to reach the end.



The set of all consistent, admissible and drivable trajectories is denoted as 𝒯. Additionally, the trajectory should be such that it does not lead to collisions of the vehicle.


  ) The trajectory shall be collision-free, i.e., it must stay within the boundaries of the road and it must avoid collisions with all dynamic obstacles. 

    
  SR:CollisionFreea) For lateral collsition avoidance, the MPC-TP may rely on a maximum lateral offset e_max^TC produced by the TC:
    
    
  * for v≤ 30 km/h: e_max^TC=10 cm,
    
  * for v≤ 80 km/h: e_max^TC=15 cm,
    
  * for v≤ 130 km/h: e_max^TC=20 cm.
    
    
  SR:CollisionFreeb) The MPC-TP shall plan its trajectories such that it is always safe for the ego vehicle to stay in its current lane, in particular by maintaining an appropriate safety distance to the leading vehicle.

 
To this end, the MPC-TP may assume that all of the input information it receives is up-to-date and fully correct. 

Note that the scope of the paper is limited with regards to the considered vehicle level hazards and SOTIF requirements. In reality, there may be further hazards and derived requirements, which may involve also other aspects of the lateral motion, a such as acceleration and jerk, as well as the longitudinal motion of the vehicle, e.g., for overtaking maneuvers.



§ SOTIF BACKGROUND




 §.§ Aspects of Safety


The safety of a technical system is commonly defined as the absence of unreasonable risk for the health and life of humans <cit.>. A reasonable level of risk has to account for the fact that most technological systems, including ADSs, can never be entirely safe. Safety risks of E/E systems can be categorized according to the source of potential hazards into

    
  1) hazards that occur due to malfunctioning behavior (Functional Safety), addressed by the ISO 26262 <cit.> standard, and
    
  2) hazards that occur due to functional insufficiencies (Safety of the Intended Functionality, SOTIF), addressed by the ISO 21448 <cit.> standard.

Functional safety and SOTIF are distinct and complementary aspects of safety that both need to be accounted for in the development of an ADS <cit.>. The exclusive focus of this paper, however, is SOTIF.
 



 §.§ SOTIF


The international standard ISO 21448 'Road vehicles - Safety of the intended functionality' published in 2022 defines SOTIF as the absence of unreasonable risk due to hazards resulting from functional insufficiencies of the intended functionality, or its implementation <cit.>. As a basic framework, it defines an iterative lifecycle with several activities that are deemed adequate for developing an ADS that satisfies the SOTIF, as shown in Figure <ref>.
















 §.§ SOTIF-related Element out of Context (SOTIF-EooC)
 

As a consequence of the collaborative development typical for the automotive industry, a supplier company often provides a component that will be integrated by the OEM into a SOTIF-related system. Under this scenario, at the time of developing the component, the the supplier may only know certain aspects of the overarching system. One way of facilitating the achievement of SOTIF in such a context is to treat the component as a SOTIF-related Element out of Context (SOTIF-EooC), according to ISO 21448 <cit.>.   

For a component treated as a SOTIF-EooC, assumptions can be made regarding its use within the whole system and its contribution to the intended functionality. Based on this, assumptions about the SOTIF-related functional insufficiencies (FIs) and their allowed target rate of occurrence can be derived. These assumptions need to be documented. They are used as inputs for the subsequent development of the SOTIF-EooC by the supplier. For the SOTIF-EooC, the identified triggering conditions (TCs) of the component and their resulting FIs are documented together with the assumptions of use for the component (a.k.a. integration requirements).
When the OEM integrates the SOTIF-EooC, the validity of the assumptions or integration requirments is established by  activities in the context of whole vehicle-level functionalities, according to ISO 21448, Clause 4.4.3 <cit.>.

In this paper, the MPC-TP is treated as a SOTIF-EooC. 
In fact, the MPC-TP may be used as part of different driving automation systems, namely E^2LCACC, ALCA, and HP. Assumptions regarding the overall systems, the integration of the MPC-TP and the contributions of the MPC-TP to the intended functionality have been discussed in Sections <ref> and <ref>.
The identification of TCs of the MPC and their resulting FIs are discussed in Section <ref>. Naturally, their analysis leads to an improved architecture and refined specifications, which are presented in Section <ref>.





 §.§ Functional Insufficiencies


r0.55
	

	
    < g r a p h i c s >

	

SOTIF scenario categories.
	



FIs lead to hazards only in specific (driving) scenarios, called the hazardous scenarios, as opposed to non-hazardous scenarios. Conceptually, these scenarios are further sub-divided into known and unknown scenarios. This categorization of scenarios is illustrated in Figure <ref>. Hazardous scenarios may be known or unknown at the start of the  analysis. The overall goal of SOTIF is to improve the specification and the design of the system, or the considered element, with respect to reducing the number of (known and unknown) hazardous scenarios. 









FIs may exist at the system level, e.g., the E2LCACC system, or at the element level, e.g., the 'Sense', 'Plan', or 'Act' parts of the system. Since FIs can lead to hazardous behaviour, their identification and the evaluation of their effects are key activities in the development of a SOTIF related item <cit.>. ISO 21448 subdivides FIs into

    
  * specification insufficiencies, which refer to a (possibly incomplete) specification contributing to hazardous behavior, and
    
  * performance insufficiencies, which refer to limitations of the technical capabilities contributing to hazardous behavior <cit.>.
 



Examples of specification insufficiencies include an incomplete specification of the E2LCACC's headway distance resulting in the ego vehicle not keeping a safe distance to the vehicle in front or the inability of the HP to handle uncommon freeway road signs that were not part of its specification and thus cannot be processed appropriately by the system.
Examples of performance insufficiencies include limitations of technical capabilities of the ALCA such as limited calculation performance, limited perception range of its front radar sensor, or limited actuation capabilities of the foundational steering system  <cit.>.

As this paper is concerned with an MPC-TP for different driving functions, the focus is on FIs at element level, in this case, of a “Plan” component. 





 §.§ Triggering Conditions


Hazardous scenarios can be decomposed into the following elements: (1) A traffic scenario with a triggering condition, (2) which leads to a hazardous behavior of the ego vehicle, (3) which may cause a hazard in combination with the surrounding traffic or environment. Hence a TC is defined as a specific condition of a scenario that initiates a subsequent system reaction contributing to hazardous behavior <cit.>(simplified). 


TCs and FIs are interrelated: A TC of a scenario activates a FI, which results in a subsequent system reaction. Figure <ref> illustrates the activation of FI at the element level by a TC and the resulting hazardous behavior. In general, the SOTIF standard also addresses reasonably foreseeable misuse. This aspect however, remains outside the scope of this paper.



For an example, assume an activated E2LCACC system that uses a radar sensor with an insufficient field of view in cased of curved roads. The upper part of Figure <ref> shows a scenario without a TC. A FI is not being activated. The E2LCACC system works as intended, i.e., the scenario is considered not hazardous.  The lower part of Figure <ref> shows a second, hazardous scenario. Here, the TC is a slower moving vehicle ahead in lane on a curved road. In this case, the assumed FI at the sensor is activated, resulting in a late detection of the leading vehicle. As a result, the system brakes too late, resulting in a rear-end collision with the leading vehicle.   








§ SOTIF ANALYSIS FOR THE MPC-TP




 §.§ Identification and Evaluation of Functional Insufficiencies and Triggering Conditions



ISO 21448 suggests a list of specific methods to analyze potential FIs and TCs <cit.>. Table <ref> shows the list of methods that are selected as appropriate to derive FIs and/or TCs of the MPC-TP, and the ones that are considered further in the remainder of this paper.



For the analysis of the “Plan” component of a system, ISO 21448 <cit.> provides an additional, specific list of methods that can be used for finding FIs and TCs. This list is provided in Table <ref>. The categories considered for the SOTIF analysis in this paper are shown in black, the ones not considered further are shown in gray.



This section describes the application of the selected analysis methods, and the FIs and potential TCs that have been identified. Multiple methods may lead to the same results, in which case the corresponding FIs and TCs are identified with the same number.



 §.§ Analysis of ODD, Use Cases, and Scenarios


This analysis method involves a close examination of the ODD specifications in (Spec:ODD), as well as predictable use cases and scenarios that may lead to potential TCs and FIs of the MPC-TP. Using this approach, the following items have been identified for our case study:  


In congestion situations (TC:AOUCS01), the MPC-TP does not consider the formation of a rescue alley for police or ambulances (FI:AOUCS01). Conversely, the formation of a rescue alley by other vehicles or the presence  of police or ambulances in congestion situations may lead the MPC-TP to perform unusual, illegal, counter-intuitive or even dangerous maneuvers (FI:DangerousManeuver).


In obstacle avoidance scenarios (TC:AOUCS02), the MPC-TP does not allow emergency maneuvers without a proper lane change. For example, it is not possible to avoid an obstacle by driving in between two lanes or by using the shoulder (FI:AOUCS02). Conversely, the emergency obstacle avoidance by other vehicles may lead the MPC-TP to perform counter-intuitive or even dangerous maneuvers (FI:DangerousManeuver).


There is no speed limit on certain sections of the Autobahn. Suppose there is blocking obstacle on the road, such as the rear end of a congestion. If the obstacle is at standstill shortly beyond the prediction horizon of the MPC-TP (TC:AOUCS03) and an obstacle avoidance maneuver is not possible, e.g., by performing a lane change. If the EgoV exceeds a certain speed v_max, this can lead to a rear-end collision because the prediction horizon N_p is not long enough for the EgoV to come to a full stop within it (FI:AOUCS03). 






The design of the current MPC-TP is based on clearly defined traffic rules. It assumes, that surrounding vehicles either have the right-of-way, and are thus considered as obstacles, or they have to yield, and can thus be ignored. However, there are situations where the right-of-way is unclear, such as at the end of two merging lanes (TC:AOUCS04a). Without further measures, this may lead to an overly conservative driving behavior (FI:AOUCS04a), and even a complete deadlock for the EgoV (FI:AOUCS04b).


On freeways, cars typically only drive forwards. However, in exceptional situations, e.g., during congestion or due to police or ambulance operations, it may be necessary to perform reverse maneuvers. Moreover, surrounding vehicles may drive backwards (TC:AOUCS05), in which case the MPC-TP may become infeasible (FI:Infeasible) if the EgoV is at rest or slowly moving forwards.


In the case of a wrong way driver (TC:AOUCS06), the MPC-TP may be unable to compute a feasible trajectory (FI:Infeasible). Or it may lead the MPC-TP to perform counter-intuitive or even dangerous maneuvers (FI:DangerousManeuver).


The MPC-TP shall be able handle situations with an unexpected, sudden change of road conditions. Important examples include a dropping friction coefficient, e.g., due to water / hydroplaning (TC:AOUCS07a), snow / ice (TC:AOUCS07b), or oil (TC:AOUCS07c), or an irregular surface, e.g., due to gravel or stones (TC:AOUCS08a), pot holes (TC:AOUCS08b), or road bumps (TC:AOUCS08c). In such situations, the trajectory generated by the MPC-TP may actually be not drivable, in the sense of (SR:Drivable), e.g., because of the limitations of the LDBM using a linear tire model (FI:Drivable).





 §.§ Analysis of System Architecture (Including Redundancies)


This analysis is based on the system architecture, as depicted in Figure <ref>. The goal is to identify potential FIs and TCs based on the overall behavior of the entire system.

The receding-horizon implementation causes the MPC-TP to act also as a feedback controller for the vehicle, which is a well-known feature of MPC <cit.>. The interoperation of the MPC-TP with the tracking controller (TC) may thus lead to undesired bahavior, such as oscillations (FI:ASA01), or even instability of the EgoV motion. This may be triggered, for example, by external disturbances such as a side wind gust, e.g., on a bridge deck (TC:ASA01), or irregular, non-continuous lane markings, e.g., due to suddenly disappearing (TC:ASA02a), dirty (TC:ASA02b), broken (TC:ASA02c), or covered (TC:ASA02d) lane markings.

Difficulties may also appear in situations where the right-of-way is unclear (TC:AOUCS04), or when other traffic participants do not strictly adhere to the traffic rules (TC:ASA05). Their interaction with the MPC-TP may then lead to an overly conservative behavior of the EgoV (FI:AOUCS04), possibly resulting in a poor progress on the road (FI:AOUCS04a), or even a deadlock (FI:AOUCS04b).

The cut-in of a faster TgtV in front of the EgoV (TC:ASA04) may lead to a violation of the front safety distance, and hence the inability of the MPC-TP to generate a feasible solution (FI:Infeasible). This can be resolved by relaxing the minimum safety distance constraint in case of a leading vehicle with a higher velocity. Even if the safety distance is not violated, the cut-in of a faster TgtV (TC:ASA04) may still lead to an overly conservative braking maneuver on behalf of the EgoV (FI:AOUCS04c).


The MPC-TP does not consider a trailing vehicle driving closely behind the EgoV in the same lane (TC:ASA06). According to (Spec:MPCd), even though it is the responsibility of the trailing vehicle to keep a sufficient safety distance, sudden braking maneuvers may lead to a collision with the trailing vehicle and are thus unsafe (FI:ASA03).




 §.§ Analysis of Functional Dependencies


In the context of this case study, functional dependencies are interpreted as other modules that are actually or potentially connected with the MPC-TP, be it upstream or downstream. The MPC-TP needs to cope with challenges arising from the limitations of these modules. With this interpretation of functional dependencies, TCs and FIs identified with this analysis method may overlap with those identified by analyzing the system architecture (in Section <ref>), in the context of the entire system.


Dynamic obstacles suddenly appearing in the field of view of the system (TC:AFD01), may overwhelm the ability of the MPC-TP to solve the OCP in order to compute a new trajectory in due time (FI:InsufficientTime), violating (SR:Timing), or to find a feasible trajectory at all (FI:Infeasible), violating (SR:CollisionFree). Examples of this group of FIs are a previously shadowed traffic participant (TC:AFD01a), an object dropping from a bridge or a truck (TC:AFD01b), or a large animal (e.g., a wild boar) crossing the freeway (TC:AFD01c).  


At points of sudden traffic rule changes (TC:AFD02), such as the beginning of a speed limit area or a no passing zone, the MPC-TP may also unable to compute a feasible trajectory (FI:Infeasible), or may be unable to solve the OCP within the allotted time (FI:InsufficientTime). 





 §.§ Analysis of Algorithms and Their Outputs or Decisions


In the context of the MPC-TP, this analysis pertains to the formulation of the OCP (<ref>) and the numerical optimization algorithm, the decisions taken by the MPC-TP, as well as the reference trajectory and the exit flag as its outputs.

The OCP is based on the LDBM, whose linear tire model may represent a poor approximation of a real vehicle if essential model assumptions are violated (FI:AAOD01). The validity of the linear tire model is already supported by corresponding constraints 𝕏^(s)_t (see Section  <ref>). The other main assumption of the LDBM is a non-negligible roll motion of the vehicle. It can be violated by an extremely asymmetric load (TC:AAOD01a), a highly dynamic cornering maneuver (TC:AAOD01b), or a high position of the CoG (TC:AAOD01c). 


Potential problems arising from the interoperation of the MPC-TP with the TC and from the lack of interaction-awareness have already been discussed (see Section <ref>). In addition, the MPC-TP may cause a lack of string stability in a chain of vehicles (TC:StringStability). If the leading brakes  abruptly, this can lead to the build-up of an increasing wave of braking maneuvers by all following vehicles. This string instability may be fostered by the MPC-TP in the middle of a chain of human-driven or automated vehicle, which may lead to collisions (FI:StringStability), if the MPC-TP is not properly tuned. 


The outputs of the MPC-TP include the reference trajectory and the solver exit flag (see Figure <ref>). 


Without software or hardware faults, the computed reference trajectory will be consistent, admissible, drivable, and collision-free, if the set 𝒯 of such trajectories is non-empty. However, due to the finite prediction time of T_p=6 s, the MPC-TP may lead the vehicle to an (unsafe) situation that cannot be resolved by a future trajectory, i.e., where 𝒯 becomes empty. In MPC language, this is known as the lack of recursive feasibility (FI:RecursiveFeasibility). A possible TC is the fast approach of a slowly driving TgtV in the front of the EgoV, where the TgtV cannot be avoided by a lateral maneuver (TC:SlowLeadingTgtV).

The solver exit flag itself might provide an indication of a triggered FI. In particular, all exit flag values q_exit≠ 1 should be considered carefully:

    
  * q_exit=2 indicates that the MPC may exceed the allotted computation time (FI:InsufficientTime), but produces a feasible trajectory. This could be triggered by rapidly changing traffic situations (TC:AAOD02) or by suddenly appearing obstacles (TC:AFD01). 
    
  * q_exit=3 means that the MPC-TP is unable to generate a feasible (collision-free) trajectory for the given constraints, because 𝒯 is empty (FI:Infeasible). For example, this can be triggered by (TC:AOUCS05), (TC:AOUCS06), (TC:AOUCS07), (TC:AOUCS08), (TC:ASA04), (TC:AFD01),(TC:AFD02).
    
  * q_exit=4 means that the solver is unable to find a solution (FI:NoSolution), even though 𝒯 may be non-empty. As software and hardware faults are out of scope, this must be due to unpredictable numerical problems (e.g., ill-conditioning, floating point arithmetics). To the best of the authors' knowledge, there are no known specific triggering conditions for this case.






 §.§ Analysis of Boundary Values


For the analysis of boundary values, the MPC-TP is examined in the context of scenarios where relevant parameters take on extreme values. Parameters considered in the analysis include

    
  * ODD and road parameters, i.e., the road bank angle, the friction coefficient μ, the road curvature, the change of curvature, and the wind speed;
    
  * maneuver parameters, i.e., the reference velocity, maximum lateral motion (e.g., due to a sudden lane change or obstacle avoidance), maximum longitudinal motion (e.g., due to emergency braking or maximum acceleration);
    
  * vehicle parameters, i.e., the vehicle mass (m), the moment of inertia (I_z), the position of the CoG (x_cog, y_cog, z_cog), the cornering stiffness of the front and the the rear tires (C_α,f, C_α,r), and all chassis and suspension parameters of the specified vehicle class.


The analysis is based on extensive simulations using a high-fidelity vehicle model, e.g, in CarMaker. The simulations scenarios aim to cover possible combinations of extreme values of the above parameters. The results are checked against the requirements (SR:Timing) - (SR:CollisionFree).

To this end, a database of artificial test scenarios covering corner cases, is generated, based on different incarnations of the maneuver parameters. Specifically for the reference velocity, scenarios are separated into low-speed (v≤ v_low=5 km/h), medium-speed (v≤ v_med=50 km/h), and high-speed (v≤ v_high=130 km/h) categories. For road curvature and change of curvature, only scenarios that (at least partially) involve maximum values are considered. The generation process of corner cases can also be automated, as proposed in Lubiniecki et al. <cit.>.

The database of corner cases is then simulated for extreme values of the ODD and road parameters and the vehicle parameters. Due to the combinatorial nature, the number of simulation scenarios thus becomes intractable. To regain tractability, only pairs of extreme parameter values are included in the boundary value analysis, while all other parameters are set to their default or regular values. For some parameters, such as the height of the CoG z_cog, instead of a default or regular values, it is more reasonable to consider only the maximum value in all simulations. Critical cases are expected to result from certain parameter combinations (in some or all corner cases), such as

    
  * the maximum side wind speed v_wind,max = 8 bft and the maximum bank angle β_max = 8^∘ (TC:CommonTrigger1),
    
  * the minimum friction value μ_min=0.15 and the minimum tire cornering stiffness C_α,f,min=60 kN/rad (TC:CommonTrigger2),
    
  * maximum front tire cornering stiffness C_α,f,max=200 kN/rad and the minimum rear tire cornering stiffness C_α,r,min=60 kN/rad, and vice versa (TC:CommonTrigger3).


These TCs may lead the MPC-TP to provide trajectories that are not drivable by a specific vehicle model or configuration (FI:Drivable). However, these are potential TCs and FIs, and their existence and relevance has to be confirmed by actually performing a large-scale, high-fidelity simulation study.



 §.§ Analysis of Common Triggering Conditions


Generally, common TCs are collected from prior experiences with the development of similar systems. They may be available, for example, by internal reports, industry standards, or other publicly documents. The TCs identified in this way may, of course, overlap with TCs identified using other methods.

The authors are currently not aware of a database or any other source for common TCs for a trajectory planner. However, applicable TCs for other ADSs from the relevant literature have been taken into account in this case study. In particular, selected TCs for a LKA system from <cit.> can analyzed with respect to their ability to challenge the MPC-TP and to trigger potential FIs:

    
  * lane keeping / lane changing scenario with suddenly disappearing lane markings (TC:ASA02a);
    
  * lane keeping / lane changing scenario with dirty, broken, or covered lane markings (TC:ASA02b,c,d);
    
  * lane keeping / lane changing scenario with unrecognizable lane markings due to direct sunlight or change of lighting conditions, e.g., when entering a tunnel (TC:ACTC03a,b);
    
  * insufficient visibility of lane markings due to low visibility, e.g., due to snow / ice or fog (TC:ACTC04a,b);
    
  * infrastructure items that could be misinterpreted as lane markings, e.g., curbs or dividers (TC:ACTC05a,b).

Other common TCs for a trajectory planner that can been identified from the existing literature include

    
  * side wind gusts, e.g., on a bridge deck (TC:ASA01a),
    
  * an uneven road surface, due to potholes (TC:AOUCS08b) or road bumps (TC:AOUCS08c),
    
  * a slippery road surface, e.g., due to a wet road with foilage (TC:ACTC09a) or a snowy / icy road (TC:ACTC09b).


For many of the common TCs from literature it remains unclear however, whether they activate FIs in the MCP-TP or elsewhere in the system. This needs to be verified in simulations or real-world experiments.



 §.§ Environment and Location


The systematic analysis of the environment and location aims to identify relevant freeway scenarios and warning signs that may compromise the safety of the MPC-TP. By means of brainstorming, the following scenarios in the ODD can be identified as potential TCs in this context.

An increase or decrease in the number of lanes on the freeway may present a particularly challenging situation for the MPC-TP, including:

    
  * entering the freeway from an on-ramp (TC:EAL01),
    
  * exiting the freeway via an off-ramp (TC:EAL02),
    
  * lane ending / lane merging situations (TC:AOUCS04a),
    
  * lane adding situations (TC:EAL04).


Similarly, situations with special vehicles or vehicles operating beyond traffic rules may pose particular difficulties for the MPC-TP, such as: 

    
  * a wrong way driver (TC:AOUCS06), 
    
  * a reckless driver or person driving under-the-influence (TC:EAL08),
    
  * an approaching emergency vehicle (TC:EAL05),
    
  * a heavy transport or oversized vehicles (TC:EAL06),
    
  * a vehicle parking on the freeway shoulder (TC:EAL07).



Furthermore, rare situations and uncommon road signs on the German Autobahn can be considered in the context of this analysis, leading to the following list of TCs:

    
  * uneven road surface (road sign 112), e.g., due to ruts or blow-ups (TC:AOUCS08d,e),
    
  * an invisible / indetectable substance is substantially lowering the friction coefficient, e.g., due to an oil spillage (road sign 114) (TC:AOUCS07c),
    
  * the freeway shoulder is not available (TC:EAL09) or the shoulder is to be used as an additional lane (road signs 223.1, 223.2, 223.3) (TC:EAL10),
    
  * end of the freeway (road sign 330.1) (TC:EAL11) or a freeway closure (TC:EAL12),
    
  * an unusually steep slope (road signs 108, 110) (TC:EAL13) or an usually high curvature (road signs 103-10, 103-20, 105-10, 105-20) of the road (TC:EAL14),
    
  * dangerous side wind gusts (road sign 117) (TC:ASA01),
    
    
  * potentially unknown traffic scenarios or traffic signs in other European countries (TC:EAL15).


All of the above TCs may affect the ability of the MPC-TP to solve the OCP and compute a trajectory in due time (FI:InsufficientTime), or to find a feasible (collision-free) trajectory at all (FI:Infeasible). Furthermore, they may cause the MPC-TP to provide trajectories which are not drivable and thus lead the EgoV to leave its assigned lane and/or collide with other vehicles or obstacles (FI:Drivable) or which may cause the vehicle to perform unusual, illegal, counter-intuitive or even dangerous maneuvers (FI:DangerousManeuver).



 §.§ Summary of Identified Triggering Conditions and Functional Insuffciencies


Appendix <ref> provides a table with an overview of all triggering conditions and functional insuffciencies that have been identified for the MPC-TP in this case study.



§ REFINED CONCEPT FOR THE VEHICLE LEVEL CONTROL SYSTEM AND THE MPC-TP


As per the SOTIF lifecycle, the SOTIF-EooC analysis in Section <ref> leads to a revision of the overall concept of the vehicle level control system and the MPC-TP, in particular. The results include a refinement of the MPC-TP specifications and design as well as the ODD, as explained in Sections <ref> and <ref>). As a further consequence, the integration requirements to the vehicle level control system are updated, as detailed in Section <ref>. Some situations that could not be (fully) resolved as part of this case study are described in Sections  and .



 §.§ Refined Specification and Design of the MPC-TP


The refined system architecture based on the SOTIF analysis in Section <ref> is depicted in Figure <ref>. The particular refinements of the specification and the design of the MPC-TP are discussed below.





  §.§.§ Design Improvements















The SOTIF-EooC analysis has led to the following concepts to improve the architecture and algorithmic design of the MPC-TP.

An overly conservative behavior of the EgoV (FI:AOUCS04c) may result, e.g., due to the cut-in of a faster TgtV (TC:ASA04). This potential issue can be mitigated by a proper tuning of the MPC-TP.

A trailing TgtV not keeping a proper safety distance to the EgoV (TC:ASA06) may cause a collision in case of a sudden braking maneuvers of the EgoV (FI:ASA03). 

a*) Even though this lies within the responsibility of the TgtV,  the EgoV shall perform sudden braking maneuvers only when strictly necessary and only to an appropriate extent. This can be accomplished by adding a strong penalty on the usage of high braking inputs in the OCP.

Spec:SuddenBrakingb*) Furthermore, the distance to the trailing vehicle shall aslo be included in the tuning of the MPC-TP.

The interaction with other traffic participants if the right-of-way is unclear (TC:AOUCS04) or if they do not strictly adhere to the traffic rules (TC:ASA05) may lead to an overly conservative behavior of the EgoV (FI:AOUCS04). 

*) This should be properly resolved by using an interaction-aware MPC approach, which is the subject of ongoing research <cit.>. For the given architecture, the interaction-aware MPC also has to be designed in coordination with the Tactical Planner.

Finally, a lack of string stability may be fostered by the MPC-TP in a chain of human-driven or automated vehicles (TC:StringStability), which may potentially lead to collisions (FI:StringStability). The problem, however, is not MPC-specific and has been analyzed in the previous literature, including ACC systems <cit.>. It shall be resolved by a proper tuning of the MPC-TP, which should be verified in extensive simulations and/or on-road tests.



  §.§.§ Adaptations to the Reference Trajectory


The interoperation of the MPC-TP with the lower-level TC may lead to oscillations and even unstable behavior (FI:ASA01). 
Spec:TrajectoryBodyg*) As a solution to this, the first N_r=4 steps of the previously computed trajectory remain unchanged, i.e., only the subsequent inputs are newly computed. N_r is called the reaction horizon. The concept is illustrated in Figure <ref>. The initial condition for the OCP in (<ref>c) thus becomes the predicted state x̂_N_r from the previous OCP solution.



As a negative consequence of this modification, the reaction time of the MPC-TP is increased. This is relevant, for instance, if the EgoV has to react to sudden events in its environment, such as in an emergency braking or obstacle avoidance scenario. However, a reaction time of T_d=N_rt_s=200 ms is still superior to that of an average human driver, which is typically between 600 ms and 1,000 ms, according to recent studies <cit.>. Thus a reaction time of 200 ms is deemed to be sufficient for common road traffic.



  §.§.§ Potential Extension of Computation Time


The regular MPC is designed for a maximum computation time of t_c=30 ms. This is less than the given sampling time, t_s=50 ms, in order to account for additional latencies, e.g., due to communication and actuation delays. Obviously, the choice of t_c should be based on generous computational margins and extensive testing. However, it is generally not possible to guarantee that the solver always provides a definite answer within this allotted computation time, i.e., that it finishes with an exit flag q_exit = 1,2,3 in due time (FI:InsufficientTime). Several TCs have been identified as part of the SOTIF analysis in Section <ref> that may represent a potential cause for an excess computation time, such as a suddenly appearing obstacle (TC:AFD01), sudden changes of traffic rules (TC:AFD02), or rapidly changing traffic situations (TC:AAOD02). A strategy to remedy this issue may look as follows:

Due to the concept of the reaction horizon, introduced in Section <ref>, the computation time can be extended by multiple (in this case: N_r=4) time steps. In occasional peaks, it can reach up to a theoretical maximum time of t̅_c=t_c+(N_r-1)t_s=180 ms. After that, a new trajectory must definitely be provided. If multiple time steps are used to solve the OCP, the new decisions (shown in red in Figure <ref>) comprise a corresponding number of time steps, which are now more than a single one.

Furthermore, the solver settings are adjusted such that the iterations can be halted after t_c, if they have not terminated; or again after each t_c+k t_s, for k=1,2,…,N_r-1. After each instance, the status of the current iterate can be checked. If the current exit flag equals to q_exit = 4, the solver may continue its computations from the current iterate, until finally the ultimate maximum solver time t̅_c is reached.





  §.§.§ Backup MPC


The Backup MPC (MPC-TP2) is a redundant implementation of the Primary MPC (MPC-TP1), which has been discussed thus far. The Backup MPC has the same inputs and outputs as the Primary MPC, as shown in Figure <ref>). It is designed according to the same specifications and requirements, as described in Sections <ref> and <ref>, except for the following differences:

    
  * The Backup MPC is utilized for safety and not passenger comfort. It has the ability to perform any maneuver that is physically possible. This means a different tuning of the weight parameters in the cost function and the traffic rule and comfort constraints 𝕏^(t)_t are removed from the OCP in (<ref>f).
    
  * Instead of a LDBM, which has a linear tire model, the Backup MPC shall include a NDBM with a Pacejka tire model. The NDBM provides a higher fidelity for the dynamics of the vehicle at the limits of handling, even though it may have a longer solution time.
    
  * For additional redundancy, the Backup MPC shall apply a different numerical solver (different algorithm and/or software implementation) and it shall be running on an independent computational platform.


The Backup MPC is constantly running in a shadow mode. During normal operation, when the Primary MPC is active, the solutions of the Backup MPC solutions are simply discarded. The trajectories of the Backup MPC are activated, or subsequently deactivated, according to a new module called Infeasibility Analysis. Its logic is based on the exit flag of the Primary MPC q_exit^(P), the exit flag of the Backup MPC q_exit^(B), and various information from the perception modules; see Section <ref> for further details. In particular, the Backup MPC shall be activated if the Primary MPC is unable to find a feasible solution (FI:Infeasible) under the following circumstances:

    
  * The friction coefficient suddenly drops by a value of μ_d>0 within a specified time interval t_d>0, due to water, snow / ice, or oil (TC:AOUCS07), or an irregular surface, e.g., due to gravel or stones, pot holes, or road bumps (TC:AOUCS08).
    
  * Static or dynamic obstacles suddenly appear close to the vehicle, e.g., due to a previously shadowed traffic participant (TC:AFD01a), a lost cargo or other object dropping from a bridge or a truck (TC:AFD01b), or a large animal (e.g., a wild boar) crossing the freeway (TC:AFD01c).
    
  * There is a sudden change of traffic rules (TC:AFD02), such as the beginning of a speed limit zone or a no passing zone.


The Backup MPC remains active until the Primary MPC produces a optimal solution again (case q_exit=1) and it is safe to switch back. For the latter condition, it is important to carefully avoid problems that may arise from a possible periodic switching behavior between the two planners.



  §.§.§ Infeasibility Analysis


The desirable outcome of the Primary MPC is a (local) minimum with an acceptable tolerance (q_exit = 1). The SOTIF analysis in Section <ref>, however, has revealed multiple scenarios where this may not be the case. Then a new module called the Infeasibility Analysis is activated. Its logic is designed as follows:

If the Primary MPC has not converged to such a minimum and returns only a feasible trajectory (q_exit = 2), the sub-optimal solution is acceptable on an exceptional basis. Because the sub-optimal solution satisfies also the terminal constraints, the recursive feasibility of the trajectory planning is not jeopardized. However, situations where the solver detects the definite infeasibility of the OCP (q_exit = 3) or where it is unable to find a feasible solution (q_exit = 4) require further attention. Here a switching logic, as presented in Table <ref>, is proposed as a basis for the Infeasibility Analysis.



Before even activating the switching logic, however, the Infeasibility Analysis filters a few special situations that will obviously or possibly lead to an infeasibility of the Primary MPC (FI:Infeasible). This includes 

    
  * (TC:AOUCS05) a leading TgtV driving in reverse, while the EgoV is at rest or slowly moving forward; 
    
  * (TC:ASA04) at higher speeds, the cut-in of a faster TgtV in front of the EgoV leading to a violation of the safety distance in the present time step.


For (TC:AOUCS05), a special planning mode shall be designed to check if (a) the EgoV is driving with a speed less than or equal to some threshold v_low (e.g., v_low=5 km/h) and (b) the minimum safety distance to the leading TgtV is already violated at the current time step. If this is the case, a stopping trajectory is automatically issued that transitions the EgoV to, or keeps it in, a complete standstill. When the Primary MPC becomes feasible again, the EgoV resumes its normal operation and moves forward. 

For (TC:ASA04), the situation can be resolved by an additional logic that relaxes the minimum safety distance constraint to a leading TgtV in the Primary MPC, if it has a higher velocity than the EgoV. Note that, even if the safety distance is not violated, the cut-in of a faster TgtV (TC:ASA04) may still lead to an overly conservative braking maneuver on behalf of the EgoV (FI:AOUCS04c). This is a separate issue that has to be resolved by a different measure.

In situations with a slippery road surface (TC:ACTC09) or a sudden drop of the friction value μ within a time interval of t_d for more than μ_d (TC:AOUCS07), in a first step the linear tire regime of the LDBM in (<ref>b) shall be exchanged with a (nonlinear) Pacejka tire model. This measure is taken pre-emptively in order to avoid a lack of drivability of the computed trajectory (FI:Drivablea). The Pacejka tire model is not used as the standard tire model of the Primary MPC because of its additional computational complexity.



According to Table <ref>, if the Primary MPC does not produce a feasible solution (q_exit^(P) = 3,4), the Backup MPC is activated already after 2 steps. It should be remarked that this could be extended, if the reaction horizon were increased beyond N_r=4, which is used in this paper. This may allow for further attempts to modify or update the constraints of the OCP in an acceptable way, similar to the steps described above. Or additional attempts may be taken to modify the algorithm or the solver settings in order to achieve an optimal, or at least a feasible solution, of the OCP.







  §.§.§ MPC-TP Driving Modes


To adequately adapt to specific traffic or environmental situations, the MPC-TP needs to introduce a switchable behavior. *) In particular, (at least)the specialized driving modes
listed in Table <ref> shall be introduced.




The proper driving mode needs to be selected by the overarching system and provided to the primary as well as the back-up MPC-TP as additional mode inputs. The MPC-TPs can then adjust their behaviors considering the detected situation. This goes hand in hand with integration requirements to detect the various driving modes.




 §.§ Refined ODD


Spec:ODDi*) To address issues, where the prediction horizon N_p of the EgoV is not large enough for the EgoV to come to a full stop if needed (cf. FI:AOUCS03 in combination with (TC:AOUCS03), we limit the maximum permissible speed in the ODD: 

    v_max=N_p a_lon,min t_s .

As the prediction horizon N_p is static,  v_max must be chosen in accordance with the maximum deceleration a_lon,min.
a_lon,min≤ 0 is the worst-case (highest) deceleration, depending on current road conditions. It depends on the worst-case values of other ODD parameters, in particular the friction value μ.  


Spec:ODDo*) To cope with (FI005) in combination with (TC015c) the ODD will be limited to freeway sections with a game fence. 

(Spec:ODDf*,h*) To mitigate (FI:Infeasible) in combination with (TC:AOUCS07), in particular with (TC:AOUCS07b), we propose to limit the minimum ambient temperature to +4^∘C  (Spec:ODDf*) and the permissible minimum road friction value (Spec:ODDh*). This will be combined with an integration requirement to conservatively estimate the friction value (IR:FrictionEstimation), cf. Section <ref>.  

The refined ODD is provided in in Table <ref>. 





 §.§ Integration Requirements


A key concept to facilitate the out-of-context development of SOTIF-related elements are assumptions on the encompassing vehicle control system. As these need to be ensured during the integration of the SOTIF-EooC into the ADS, they shall be designated as integration requirements. The new integration requirements of the SOTIF analysis are listed in Table <ref>.







Cases where neither the Primary MPC nor the Back-Up MPC is able to generate a permissible trajectory shall be handled by the encompassing system as follows:

(IR:EmergencyOperationRequest) Situations where the MPC-TP requests an emergency operation shall be handled by the encompassing system utilizing a dedicated emergency planner that is independent of the MPC-TP.

Situations involving irregular road surfaces (TC:AOUCS08) where the MPC-TP would be unable to generate a feasible (collision-free) trajectory (FI:Infeasible) shall be identified and handled by the encompassing system. Here, the Tactical Planner needs to decide what to in such cases, e.g., depending on the size of the stones or potholes and provide an appropriate tactical decision to the MPC-TP or handle the situation without the MPC-TP.
This leads to integration requirement (IR:IrregularRoadSurface): Situations involving irregular road surfaces shall be detected and handled by the encompassing system utilizing the Tactical Planner.





 §.§ Situations Requiring Further Analysis


(Further analysis required) The OCP is based on the LDBM, assuming a linear tire model and no roll motion, which may represent a poor approximation of a real vehicle if these assumptions are violated (FI:AAOD01). The non-negligible roll motion may be violated by an extremely asymmetric load (TC:AAOD01a), a highly dynamic cornering maneuver (TC:AAOD01b), or a high position of the CoG (TC:AAOD01c). A detailed simulation analysis is required to examine whether the LDBM is sufficient for the targeted use cases (especially with respect to SR:Drivable), or whether a two-track model is required instead.

Situations encompassing side wind gusts (TC:ASA01) are deemed as adequately covered by the wind limitation in the ODD (Spec:ODDg).

Potentially unknown traffic scenarios or traffic signs, e.g., due to deviating regulations in other European countries, are potentially problematic as well (TC:EAL15). However, the existing ODD restriction limits the permissible roadway types to the German Autobahn (Spec:ODDa) seems sufficient to mitigate this concern.



 §.§ Interaction-aware MPC


Particular combinations of TCs and FIs cannot be adressed without extending the architecture of the encompassing vehicle, e.g. by adding additional senors and/or capabilities. An example is the combination of (TC:AOUCS04a) and (FI:AOUCS04).
As per our analysis the OCP in (<ref>) may lead to an overly conservative behavior of the EgoV (FI:AOUCS04), possibly leading to poor progress on the road (FI:AOUCS04a), or even a deadlock (FI:AOUCS04b). The only safe way to reduce this conservatively is to internalize the relevant TgtVs in the MPC predictions and to reflect the interactions between the TgtVs and the EgoV by appropriate behavioral models. Interaction-aware MPC approaches, however, are the subject of ongoing and future research <cit.> and are beyond the scope of this paper.



§ CONCLUSION


In this paper, multiple aspects of the general SOTIF framework provided in ISO 21448 have been applied to a generic MPC-based trajectory planner, which is part of an ADS. The generic MPC-TP is treated as an SOTIF-EooC, elaborating this concept that is only sketched in the SOTIF standard. In a first step, the initial concept of a generic MPC-TP has been analyzed in order to identify relevant FIs and corresponding TCs. To this end, the analysis techniques that are described in ISO 21448 only in very general terms have to be interpreted and applied to the case study. In a second step, the resulting comprehensive list of relevant FIs and TCs informed an SOTIF-optimized concept of the MPC-TP. The revised safety architecture includes, in particular, an infeasibility analysis as a component and a back-up trajectory planner, a switchable behavior of the MPC-TP to support special driving modes, and adaptations to the reference trajectory.  

An important aspect of the results is the identification of important research topics on MPC for path planning in the context of automated and autonomous driving, from a safety perspective:


  (i) interaction-awareness with other (human) traffic participants;

  (ii) ability to avoid obstacles and drive safely even at low friction values;

  (iii) design of a recursively feasible MPC-based path planner with a reaction horizon;

  (iv) string stability in a chain of vehicles;

  (v) examination of a safe switching logic between a Primary MPC and a Backup MPC. 


In future work, an extensive simulation study would be beneficial for a more detailed understanding of the potential shortcomings of the MPC approach itself, or the respective vehicle model to be used for its predictions in extreme driving situations.



§ ACKNOWLEDGMENT


The research leading to these results was funded by the German Federal Ministry for Economic Affairs and Climate Action (BMWK) through the project EEMotion.










    10cm(4.5cm,2cm)
        

§ APPENDIX A: OVERVIEW OF TCS AND FIS

    
    [c]|p6cm|p6cm|p7cm|
    TC     Triggered FI     Notes



() traffic congestion
    
() no capability to form a rescue alley for police or ambulances() generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(refined specification/design) introduce switchable behavior of the MPC (driving mode: Rescue Alley)
(integration requirement) congestion situations shall be detected and handled by the encompassing system (a,b)



() obstacle avoidance scenario 
    
() no capability to carry out emergency maneuvers without a proper lane change, driving in between two lanes, or by using the shoulder(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(refined specification/design) introduce switchable behavior of the MPC (driving mode: Obstacle Avoidance)(integration requirement) obstacle avoidance situations shall be detected and handled by the encompassing system (a,b)



() blocking obstacle shortly beyond the prediction horizon N_p of the MPC-TP and no obstacle avoidance is possible 
    
() prediction horizon N_p of the MPC-TP is too short to come to a full stop within it
    
(ODD restriction) limit permissible max. speed to
v_max=|N_p a_lon,min t_s| (Spec:ODDi*)



(a) right-of-way is unclear: at the end of two merging lanes (TC:AOUCS04b) right-of-way is unclear: special situations during general lane changes 
   
(a) overly conservative behavior of the MPC-TP: inability to ensure sufficient progress on the road(FI:AOUCS04b) overly conservative behavior of the MPC-TP: inability to avoid deadlocks in certain situations (FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(Interaction-aware MPC) interaction-aware MPC approach, in coordination with the Tactical Planner (Spec:InteractionAwareMPC*)



() Leading TgtV driving in reverse, in combination with a stopping or slowly moving EgoV 
    
() inability to generate a feasible (collision-free) trajectory in certain situations 
    
(refined specification/design) infeasibility analysis: selection of stopping trajectory in situations with low speed and safety distance violated



() wrong way driver 
    
(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations (FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(refined specification/design) introduce switchable behavior of the MPC (driving mode: Wrong Way Driver)(integration requirement) if infeasibility still occurs, switch to emergency operation of the encompassing system ()(integration requirement) situations with wrong way drivers shall be detected and handled by dedicated mode of the encompassing system ()



(a) sudden drop of friction coefficient: water (hydroplaning) (TC:AOUCS07b) sudden drop of friction coefficient: snow / ice (TC:AOUCS07c) sudden drop of friction coefficient: oil spillage 
    
(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations 
   
(ODD restriction) limit permissible minimum temperature to exclude snowy/icy conditions (Spec:ODDf*) 
(ODD restriction) limit permissible minimum friction value (Spec:ODDh*)(integration requirement) conservative estimation of the friction value (a)(refined specification/design) avoid small areas filled with water by evasive maneuvers within the lane boundaries




(TC:AOUCS07a) sudden drop of friction coefficient: water (hydroplaning)(TC:AOUCS07b) sudden drop of friction coefficient: snow, ice (TC:AOUCS07c) sudden drop of friction coefficient: oil spillage 
    
(a) inability to generate a drivable trajectory in certain situations: due to linear tire model in LDBM
    (refined specification/design) infeasibility analysis: temporarily relax the linear tire regime if the friction value drops by more than μ_d within t_d



(a) irregular road surface: gravel or stones(TC:AOUCS08b) irregular road surface: potholes(TC:AOUCS08c) irregular road surface: road bumps
     
 (FI:Drivableb) inability to generate a drivable trajectory in certain situations: due to irregular road surface
    
(integration requirement) situations with irregular road surface shall be detected and handled by the encompassing system in combination with the Tactical Planner ()



(a) side wind gusts: bridge deck(TC:ASA02a) irregular lane markings: suddenly disappearing lane markings(b) irregular lane markings: dirty lane markings(TC:ASA02c) irregular lane markings: broken lane markings(TC:ASA02d) irregular lane markings: covered lane markings 
    
() receding-horizon planning of the MPC-TP in combination with tracking controller could lead to oscillations, or even an unstable behavior of the EgoV 
    
(refined specification/design) MPC-TP to keep the first N_r=4 steps (reaction horizon) of the trajectory unchanged (Spec:TrajectoryBodyg*) 



(TC:AOUCS04) right-of-way is unclear() other traffic participants not strictly adhering to the traffic rules 
    (FI:AOUCS04a) overly conservative behavior of the MPC-TP: inability to ensure sufficient progress on the road(FI:AOUCS04b) overly conservative behavior of the MPC-TP: inability to avoid deadlocks in certain situations(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
   
(Interaction-aware MPC) interaction-aware MPC approach, in coordination with the Tactical Planner (Spec:InteractionAwareMPC*)



26cm() cut-in of a faster TgtV in front of the EgoV
    
(FI:AOUCS04c) overly conservative behavior of the MPC-TP: overly conservative braking in cut-in scenarios 
    
(refined specification/design) tuning of MPC-TP such that it avoids overly conservative braking to a leading TgtV with a positive relative velocity

2-3
    
(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations
    
(refined specification/design) infeasibility analysis: if TgtV cuts into the EgoV's lane and is faster than EgoV, then temporarily relax linear minimum distance to this TgtV



() trailing vehicle driving closely behind the EgoV in the same lane, not keeping a proper safety distance 
    () inability of the MPC-TP to consider trailing traffic in the lane of the EgoV, where a sudden braking maneuver of the EgoV may lead to a collision 
   
(refined specification/design) MPC-TP to brake only when strictly necessary and only to an appropriate extent (Spec:SuddenBrakinga*) (refined specification/design) MPC-TP to account for distance to trailing vehicle by means of tuning the weight of ℓ_o(·,·) in the cost function (Spec:SuddenBrakingb*) 



26cm(a) suddenly appearing obstacle: previously shadowed traffic participant(TC:AFD01b) suddenly appearing obstacle: object dropping from a bridge or truck(TC:AFD01c) suddenly appearing obstacle: large animal (e.g., wild boar) crossing the freeway
    
(FI:Infeasible) inability to compute a feasible (collision-free) trajectory in certain situations () insufficient time to solve the OCP, in order to compute a new trajectory
   
(refined specification/design) infeasibility analysis(refined specification/design) potential extension of computation time to multiple time steps based on the concept of a reaction horizon

2-3
   
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(ODD restriction) limit ODD to freeway sections with a game fence (Spec:ODDo*) (refined specification/design) introduce switchable behavior of the MPC (driving mode: Defensive Driving)



(a) sudden traffic rule change: start of a speed limit area(TC:AFD02b) sudden traffic rule change: start of a no passing zone 
    
(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory 
   
(ODD restriction) limit permissible max. speed to
v_max=|N_p a_lon,min t_s| (Spec:ODDi*)(refined specification/design) infeasibility analysis(refined specification/design) potential extension of computation time to multiple time steps based on the concept of a reaction horizon




(a) LDBM assumption violation: asymmetric load(TC:AAOD01b) LDBM assumption violation: dynamic cornering(TC:AAOD01c) LDBM assumption violation: high position of the CoG 
    
()  poor approximation of real vehicle behavior by the LDBM in the OCP
    
(further analysis required) experimental analysis: evaluate if MPC-TP behaves safely in these situations; otherwise consider usage of a more complex model (e.g., a two track model) in the OCP



() chain of vehicles, EgoV in the middle, first vehicle brakes suddenly 
    
() lack of string stability of MPC-TP in a chain of vehicles, potentially leading to collisions
    
(refined specification/design) proper tuning of MPC-TP for string stability, to be verified in simulations and/or on-road tests



() fast approach of a slowly driving TgtV in the front of the EgoV, where the TgtV cannot be avoided by a lateral maneuver

   
(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations() lack of recursive feasibility 
    
(refined specification/design) infeasibility analysis(refined specification/design) if collision is unavoidable, activation of a collision mitigation mode in the encompassing system



() rapidly changing traffic situations 

   
(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory 
   
(refined specification/design) potential extension of computation time to multiple time steps based on the concept of a reaction horizon



no specific TCs known
   
() inability to generate a solution in certain situations at all 
   
(refined specification/design) infeasibility analysis



Pairs of extreme parameter values in some or all corner cases:() max. side wind speed v_wind,max = 8 bft and max. bank angle β_max = 8^∘ () min. friction value μ_min=0.15 and min. tire cornering stiffness C_α,f,min=60 kN/rad(a) max. front tire cornering stiffness C_α,f,max=200 kN/rad × and min. rear tire cornering stiffness C_α,r,min=60 kN/rad(TC:CommonTrigger3b) min. front tire cornering stiffness C_α,f,min=60 kN/rad × and max. rear tire cornering stiffness C_α,r,max=200 kN/rad...
   
(FI:Drivablea) inability to generate a drivable trajectory in certain situations: due to linear tire model in LDBM(FI:AAOD01) poor approximation of real vehicle behavior by the LDBM in the OCP
    
(further analysis needed) experimental analysis for pairs of boundary values, conduct simulations to confirm/discard relevance of LDBM for MPC-TP; otherwise consider usage of a more complex model (e.g., a two track model) in the OCP



(TC:ASA02a) irregular lane markings: suddenly disappearing lane markings (TC:ASA02b) irregular lane markings: dirty lane markings(TC:ASA02c) irregular lane markings: broken lane markings(TC:ASA02d) irregular lane markings: covered lane markings 
    
    
(further analysis needed) common TCs, conduct simulations to confirm/discard relevance for MPC-TP 



(a) unrecognizable lane markings: direct sunlight (TC:ACTC03b) unrecognizable lane markings: change of lighting conditions, e.g., when entering a tunnel 
    
    
(further analysis needed) common TCs, conduct simulations to confirm/discard relevance for MPC-TP 



(a) insufficient visibility of lane markings: snow / ice  (TC:ACTC04b) insufficient visibility of lane markings: fog 
    
    
(further analysis needed) common TCs, conduct simulations to confirm/discard relevance for MPC-TP 



(a) infrastructure items that could be misinterpreted as lane markings: curbs (TC:ACTC05b) infrastructure items that could be misinterpreted as lane markings: dividers 
    
    
(further analysis needed) common TCs, conduct simulations to confirm/discard relevance for MPC-TP 



(TC:AOUCS08b) irregular road surface: potholes(TC:AOUCS08c) irregular road surface: road bumps 
    
    
(further analysis needed) common TCs, conduct simulations to confirm/discard relevance for MPC-TP 



(a) slippery road surface: wet road with foilage(TC:ACTC09b) slippery road surface: snowy / icy road
    
(FI:Drivablea) inability to generate a drivable trajectory in certain situations: due to linear tire model in LDBM
   
(ODD restriction) limit permissible minimum temperature to exclude snowy/icy conditions (Spec:ODDf*)(refined specification/design) infeasibility analysis: temporarily relax the linear tire regime if the friction value drops by more than μ_d within t_d(further analysis needed) common TCs, conduct simulations to confirm/discard relevance for MPC-TP 












26cm() entering the freeway from an on-ramp () exiting the freeway via an off-ramp () lane adding situations
    
(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations
    
(refined specification/design) potential extension of computation time to multiple time steps based on the concept of a reaction horizon(refined specification/design) infeasibility analysis

2-3
    
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(further analysis needed) conduct simulations and/or real experiments



26cmsituations with special vehicles or vehicles operating beyond traffic rules:(TC:AOUCS06) wrong way driver () approaching emergency vehicle () heavy transport or oversized vehicles () vehicle parking on the freeway shoulder() reckless driver / person driving under-the-influence
    
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(refined specification/design) introduce specialized driving modes for particular situations (Spec:SpecialDrivingModes*)(further analysis needed) conduct simulations and/or real experiments

2-3
    
(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations    
(refined specification/design) potential extension of computation time to multiple time steps based on the concept of a reaction horizon(refined specification/design) infeasibility analysis



26cm(TC:AOUCS08d) irregular road surface: ruts(TC:AOUCS08e) irregular road surface: blow-ups
   
(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations
    
(refined specification/design) potential extension of computation time to multiple time steps based on the concept of a reaction horizon(refined specification/design) infeasibility analysis


2-3
   
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(further analysis) extension of perception required




() freeway shoulder not available () freeway shoulder to be used as additional lane 
   
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
   
(refined specification/design) introduce specialized driving modes for particular situations (Spec:SpecialDrivingModes*)(further analysis needed) conduct simulations and/or real experiments




() end of freeway () freeway closure
    
 (FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
   
(further analysis needed) conduct simulations and/or real experiments

() steep road slope () high road curvature 
   
(FI:Drivablea) inability to generate a drivable trajectory in certain situations: due to linear tire model in LDBM(FI:AAOD01) poor approximation of real vehicle behavior by the LDBM in the OCP
   
(further analysis required) experimental analysis: evaluate if MPC-TP behaves safely in these situations; otherwise consider usage of a more complex model (e.g., a two track model) in the OCP



(TC:ASA01) side wind gusts
   
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory
    
(no change needed) covered by wind limitation in the ODD (Spec:ODDg)



() potentially unknown traffic scenarios / traffic signs in other European countries
   
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory
    
(no change needed) covered by permissible road type restriction in the ODD (Spec:ODDa) 



    
    









 Paper Title: Who should I trust? A Visual Analytics Approach for Comparing Net Load Forecasting Models.
































Who should I trust?A Visual Analytics Approach for Comparing Net Load Forecasting Models



Parts of this work were supported by the U.S. Department of Energy Solar Energy Technologies Office and the Office of Electricity Sensors Program, and performed jointly at the Pacific Northwest National Laboratory under Contract DE-AC05-76RL01830 and at the Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344).

    Kaustav Bhattacharjee1, Soumya Kundu2, Indrasis Chakraborty3, and Aritra Dasgupta1

1Department of Data Science, New Jersey Institute of Technology, USA. Email: {kb526, aritra.dasgupta}@njit.edu 
2Optimization and Control Group, Pacific Northwest National Laboratory, USA. Email: soumya.kundu@pnnl.gov
3Center for Applied Scientific Computing, Lawrence Livermore National Laboratory, USA. Email: chakraborty3@llnl.gov 

    August 5, 2024
========================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Net load forecasting is crucial for energy planning and facilitating informed decision-making regarding trade and load distributions. However, evaluating forecasting models' performance against benchmark models remains challenging, thereby impeding experts' trust in the model's performance. 

In this context, there is a demand for technological interventions that allow scientists to compare models across various timeframes and solar penetration levels. 

This paper introduces a visual analytics-based application designed to compare the performance of deep-learning-based net load forecasting models with other models for probabilistic net load forecasting. 

This application employs carefully selected visual analytic interventions, enabling users to discern differences in model performance across different solar penetration levels, dataset resolutions, and hours of the day over multiple months. We also present observations made using our application through a case study, demonstrating the effectiveness of visualizations in aiding scientists in making informed decisions and enhancing trust in net load forecasting models.



visual analytics, AI/ML, net load forecasting




§ INTRODUCTION



Comparing multiple computational models for their performance is crucial for enhancing trust in model outcomes because it provides a basis for evaluating the reliability and consistency of each model <cit.>. By assessing how different models perform under various conditions and scenarios, stakeholders can better understand their strengths, weaknesses, and overall effectiveness. Such comparative analysis helps to identify the most suitable model for specific tasks or applications, thereby instilling confidence in the reliability of the chosen model. This becomes more important while predicting the net load of an electric grid, which is defined as the difference between total electricity demand and generation from behind-the-meter resources like solar and distributed generators, and is influenced by various factors such as weather conditions and time of day <cit.>. 

Accurate net load forecasting enables grid operators, policymakers, and energy providers to make informed decisions regarding energy trade, load distribution, and resource allocation. However, the rise of solar energy generation sources in residential settings has significantly impacted the performance of traditional net load forecasting models, highlighting the need for robust time-series forecasting techniques <cit.>. Collaborating with scientists, we developed a deep-learning model that integrates variables such as temperature, humidity, apparent power, and solar irradiance to achieve strong predictive performance and resilience in the face of missing data <cit.>. However, the model's sensitivity to seasonal variations and noisy inputs underscores the need for further exploration with domain experts. Hence, we developed an interactive tool that empowers users to investigate the model's performance across diverse time periods and input scenarios <cit.>.





However, while we initially recognized the importance of comparing the model's performance with traditional models, feedback from domain experts further emphasized its significance in building trust in model outcomes. Visual analytics can play a pivotal role here, as evidenced by prior research demonstrating its importance in enhancing trust in machine learning models <cit.>.

This is exemplified by the interactive tool we developed, which enabled domain experts to extract valuable insights concerning the model's sensitivity toward temperature and humidity. Moreover, recent discourse, as highlighted in <cit.>, emphasizes the critical role of visual analytics in fostering trust-augmented applications of artificial intelligence and machine learning (AI/ML) within the energy sector. Building upon this, we designed  , incorporating carefully selected visual analytic interventions. These interventions facilitate the comparison of multiple models across various parameters, including solar penetration levels, dataset resolutions, and different hours of the day, enhancing stakeholders' confidence in model performance.



The aim of our application is to build trust through model comparison, primarily comparing our net load forecasting model with a reference model. We enhanced our model to process inputs at varying resolutions and then devised the reference model, which generates predictions by averaging net load ground truths for the last 30 days at the same time point. Despite lacking predictive ML components, this reference model serves as a benchmark in various net load forecasting competitions, including the recent Net Load Forecasting Prize by the National Renewable Energy Laboratory (NREL) and the U.S. Department of Energy Solar Technologies Office (SETO) <cit.>. Through the web interface of , we were able to uncover patterns in the performance that help improve trust in the model outcomes. In this work, we identify the visual analytic tasks that are required to compare these models. These tasks can be extended to compare other similar net load forecasting models, enabling users to make well-informed decisions based on the outcomes of these models.

While previous research has predominantly focused on developing interfaces during the model development phase, tailored to assist model developers, our emphasis lies in the post-hoc evaluation of model performance, specifically addressing the needs of energy scientists and grid operators <cit.>. Other studies have explored the performance of probabilistic net load forecasting models through different visualization charts but have often lacked an integrated interactive interface <cit.>. In contrast, our interactive visual analytic tool provides carefully designed visual cues for comparing model performance across different factors like dataset resolutions and solar penetration levels, thus offering a novel approach in this domain.

In this paper, we first introduce the different models used in this work and the rationale behind choosing them. Subsequently, we outline the identified visual analytic tasks and detail the design decisions guiding the development of our interactive interface. This is followed by some of the observations made by our power scientist collaborator through that demonstrate its efficacy in comparing model performance across multiple facets. Finally, we conclude by sharing insights gained from this development and discussing some of the future research opportunities in this domain. Additionally, a short demonstration video is available https://bit.ly/trustvishere.



§ MODEL DESCRIPTION




We start with a deep learning-based probabilistic model tailored for net load forecasting in high behind-the-meter solar scenarios <cit.>. This model has three key components: a kernelized probabilistic forecasting (kPF) module, an autoencoder (AE), and a long short-term memory (LSTM) network. This model effectively captured complex temporal dependencies and uncertainties inherent in net load data, which is crucial for reliable forecasting in environments with high solar penetration. By incorporating kernel methods into probabilistic forecasting, the model handled non-linear relationships and captured subtle variations in net load influenced by solar energy fluctuations. The autoencoder component enhanced feature extraction and dimensionality reduction, facilitating the LSTM network's ability to capture long-term dependencies and predict future net load values accurately. Experimental results demonstrated the superior performance of this model compared to traditional forecasting models, showcasing its efficacy in addressing the challenges posed by high solar scenarios and advancing the state-of-the-art in net-load forecasting methodologies. 

This model was used in the Net Load Forecasting Prize competition hosted by NREL and SETO <cit.>. However, during this competition, we observed its underperformance on the data provided by the organizers, which had lower resolutions. We discovered that while the autoencoder component excelled with high-resolution datasets (such as 15-minute intervals), it struggled to effectively capture temporal dependencies in lower-resolution datasets (e.g., 1-hour intervals provided by the organizers). Consequently, this limitation led to subpar outcomes generated by the LSTM component. In light of this, we opted to remove the kPF and autoencoder components and instead developed a version of the model solely utilizing the LSTM component. Additionally, fine-tuning the number of layers in the LSTM component yielded significantly improved results in the competition.

The competition employed a reference model to assess model performance across various probability levels. As previously mentioned, this reference model simply utilizes historical input data from the past 30 days to generate probabilistic forecasts for a specific time point. This model can serve as the initial benchmark for assessing the effectiveness of other models. Therefore, in this work, we developed a reference model following the same principles. Since these forecasts are probabilistic in nature, we calculated the Continuous Ranked Probability Score (CRPS) for both the reference model and our model. Subsequently, we computed the Continuous Ranked Probability Skill Score (CRPSS) based on these CRPS scores, evaluating whether our forecast presents an improvement or deterioration compared to the reference forecast <cit.>. A positive CRPSS indicates that the forecast outperforms the reference forecast, whereas a negative value suggests inferior performance. Utilizing these CRPSS values, we compare the performance of our model against the reference model across multiple dates throughout the year, and present these values in . 



§ VISUAL ANALYTICS-BASED DESIGN


Our application employs a visual analytics-based design featuring coordinated views enhanced with visual cues to help users compare model performance. It combines interactive visualization with comparison metrics like CRPSS to improve trust in the model outcomes and also allows the users to probe the model and understand its performance across different solar penetration levels and months. In this section, we highlight the two tasks performed by and how its visual analytics-based design aids in executing these tasks:
T1: Compare model performance across different solar penetration levels and data resolutions: Model performance may vary across different solar penetration levels due to the increased variability in net load data caused by intermittent solar generation. LSTM-based models might struggle to capture and predict these dynamic behaviors accurately. Furthermore, datasets with higher resolutions, such as sub-hourly intervals, enable models to more effectively capture short-term fluctuations and dependencies. Hence, this task essentially involves comparing model performance at different solar penetration levels and data resolutions.
T2: Identify patterns across different timeframes: By analyzing performance across multiple timeframes, power scientists can assess the net load forecasting models' robustness and consistency in capturing both short-term fluctuations and long-term trends. This evaluation helps to identify whether a model's performance is consistent across various temporal scales or exhibits variability or biases at specific time periods. This task relates to identifying patterns in the model's performance across different months, various hours of the day, and different time periods.

Our application implements multiple coordinated views and components in order to fulfill these tasks. Next, we discuss the design of these views and components along with the rationale behind them:
Comparison View: As comparing model performance is the main objective of , we begin with the Comparison View, which utilizes modified box plots to compare models. Figure <ref>a depicts CRPSS values on the y-axis and different data resolutions on the x-axis. The box plots illustrate the distribution of CRPSS values, with the median typically exceeding zero, indicating superior performance of our model over the reference in most cases. Users can utilize the solar penetration level filter to compare performance across various levels (20%, 30%, 50%) (Figure <ref>c) (T1). However, based on initial feedback from domain experts, we recognized the importance of displaying the distribution of CRPSS values. Therefore, we integrated dots representing the CRPSS values, jittered along the x-axis to illustrate their distribution over the entire year. These dots align with insights from the box plot and additionally reveal instances where our model performs worse than the reference on certain dates. This led to the development of the Patterns View, where we can identify the timeframes where the model underperforms compared to the reference.

Patterns View: In this view, utilizes a heatmap to depict performance patterns for each month across different hours of the day (T2). The x-axis represents the 24 hours of the day (0-23), while the y-axis displays the months of the year (Feb - Dec) (Figure <ref>b). Each box in the heatmap denotes the average CRPSS value for each month at each hour, indicated by the color. Darker blues signify more positive CRPSS values, indicating the superior performance of our model compared to the reference at that time. Conversely, darker reds indicate more negative CRPSS values, signifying poorer performance of our model compared to the reference at that time. This diverging color scale aids in easily identifying performance patterns across different time points and facilitates the identification of instances where our model does not outperform the reference. Users can filter specific months to focus on particular time periods and analyze performance accordingly (Figure <ref>e). Additionally, based on feedback from domain experts, we implemented the Sidebar component to allow users to select specific date ranges and analyze performance patterns within those ranges.
Sidebar: The Sidebar facilitates user selection of start and end dates to filter results across all views, enabling focus on specific date ranges (T2) for comparing model performance within those periods (Figure <ref>d). Additionally, based on feedback from domain experts expressing interest in comparing model performance across all solar penetration levels simultaneously, offers a comparison mode toggle in the Sidebar. Enabling this mode updates both views to display box plots and heatmaps for all solar penetration levels side by side, aiding in point-to-point comparison and identification of performance patterns across different solar penetration levels.



§ RESULTS FROM A CASE STUDY

The efficacy of an application can be validated if the application is able to perform the intended tasks effectively. In this section, we show the results through a case study that demonstrates how can be used to compare and select net load forecasting models effectively.

This case study involved a power scientist with over 10 years of experience in power and grid systems. With expertise in nonlinear dynamics, large-scale networks, and distributed control, he played a crucial role in developing the model. His main objective was to assess the model's performance relative to the reference model across various time points and solar penetration levels, which are critical factors to consider before deploying it for a project. We informed him that we had integrated CRPSS values for both the model and the reference model for all dates throughout the year. He then accessed through a browser and examined the distribution of CRPSS values across different data frequencies at a 20% solar penetration level (Figure <ref>a). Notably, he discovered that the model performed better with the lower resolution dataset (1-hour), contrary to expectations for an LSTM-based model (T1). Upon further examination, he noted a marginal difference in the median CRPSS between high and low-resolution datasets (0.85 and 0.89, respectively). Therefore, he enabled the comparison mode through the Sidebar, enabling him to assess the model's performance as solar penetration increased. He noted that the model consistently performed well with the higher resolution dataset (15-min) across all other solar penetration levels, confirming the notion that our model excels with high-resolution datasets in high solar penetration scenarios (Figures <ref>b and  <ref>c). Next, he observed that although the median CRPSS was significantly above zero, the minimum value was negative. Upon inspecting the dots adjacent to the box plots in the Comparison View, he observed that while most dots clustered around the median line, there were a few outliers with negative CRPSS values.

Hence, the scientist sought to understand temporal patterns to identify if any specific hour of the day contributed to the negative values. Consequently, he navigated to the Patterns View within to analyze the CRPSS value distribution in the heatmap at a 20% solar penetration level (Figure <ref>d). Consistent with observations from the Comparison View, most heatmap boxes displayed varying shades of blue, indicating superior model performance compared to the reference across most months and hours. On closer look, the scientist noted a box-like pattern in the heatmap, revealing enhanced performance during morning hours (8 am to 10 am) from April to September, followed by a decline during midday (11 am to 4 pm), and another spike in the evening (4 pm to 6 pm) during these months. This pattern suggested that the model effectively captured diurnal variations in net load data and adjusted predictions accordingly (T2). While similar box-like patterns were evident across other solar penetration levels, the intensity of model performance varied with increased solar penetration. This insight holds practical significance for model selection during deployment, as it suggests the potential use of different models or model ensembles tailored to distinct times of the day, leveraging their respective performances on diurnal patterns. Thus, the power scientist was satisfied that could yield valuable insights regarding the model, aiding informed decision-making. A short demonstration video is available at https://bit.ly/trustvisthis link.


§ CONCLUSION

Efficient model selection for net load forecasting plays a pivotal role in energy planning and grid operations. In this work, we delve into this process, integrating visual analytics with input from domain experts to identify key tasks for comparative model selection. Subsequently, we translate these tasks into an interactive interface, enabling users to assess model behavior across various factors such as solar penetration levels, data resolution, and time of day.


Throughout this endeavor, we gained valuable insights into the model behavior and the challenges posed during the multi-way comparison of models. This collaborative effort underscored the need for interactive tools that facilitate the seamless translation of model insights into actionable decisions. We can argue that our application is a first step towards this direction. As a next step, our plan is to incorporate multiple net load forecasting models into the application and integrate additional metrics for effective comparison of their performance.

Looking ahead, we aim to enhance our application in several ways. Incorporating economic planning and analysis options will allow stakeholders to assess the cost-benefit ratio before model selection, thereby enhancing trust in the outcomes. Additionally, as the energy landscape evolves, our application's flexibility will be pivotal in effectively comparing and selecting forecasting models for real-world applications. In summary, our collaborative endeavor has produced a robust tool for multi-faceted model comparison and has paved the way for informed decision-making through visual analytics in energy planning.








abbrv-doi




 Paper Title: MAARS: Multi-Rate Attack-Aware Randomized Scheduling for Securing Real-time Systems.





























MAARS: Multi-Rate Attack-Aware Randomized Scheduling for Securing Real-time Systems
    Arkaprava Sain, Sunandan Adhikary, Ipsita Koley, Soumyajit Dey

Department of Computer Science and Engineering, Indian Institute of Technology, Kharagpur, India

    
=====================================================================================================================================================================



§ ABSTRACT 
    Modern Cyber-Physical Systems (CPSs) consist of numerous control units interconnected by communication networks. Each control unit executes multiple safety-critical and non-critical tasks in real-time. 
    
    Most of the safety-critical tasks are executed with a fixed sampling period to ensure deterministic timing behaviour that helps in its safety and performance analysis. 
    However, adversaries can exploit this deterministic behaviour of safety-critical tasks to launch inference-based-based attacks on them. This paper aims to prevent and minimize the possibility of such timing inference or schedule-based attacks to compromise the control units. This is done by switching between strategically chosen execution rates of the safety-critical control tasks such that their performance remains unhampered. Thereafter, we present a novel schedule vulnerability analysis methodology to switch between valid schedules generated for these multiple periodicities of the control tasks in run time. Utilizing these strategies, we introduce a novel Multi-Rate Attack-Aware Randomized Scheduling (MAARS) framework for preemptive fixed-priority schedulers that minimize the success rate of timing-inference-based attacks on safety-critical real-time systems. To our knowledge, this is the first work to propose a schedule randomization method with attack awareness that preserves both the control and scheduling aspects.
    
    
    The efficacy of the framework in terms of attack prevention is finally evaluated on several  automotive benchmarks in a Hardware-in-loop (HiL) environment.



Schedule-Based Attacks, Multi-Rate Control Execution, CPS Security




§ INTRODUCTION



Real-time systems (RTSs) in CPS domains like automotive, avionics, power grids, etc. are mostly safety-critical. 

The feature-rich implementation of such modern CPSs makes the system design mixed-critical, for which tasks with different criticality levels are executed in each processor <cit.>. 




The safe operation of these RTSs relies highly on the timely execution of the safety-critical control tasks. 



For achieving predictable execution behaviour, fixed priority based scheduling algorithms remain popular for running tasks on embedded control units in CPSs, with higher priority typically being assigned to hard real time safety-critical tasks for ensuring deadline satisfaction even under platform uncertainties like sensing/actuation/communication delays. Also, the deterministic nature of static priority scheduling enables the designers to analyse the worst-case response time (WCRT) of every safety-critical control loop and design delay-aware control strategies for these loops <cit.>.

However, this predictability of fixed-priority scheduling exposes the timing information of safety-critical task executions. The attacker first finds out the periodicity of a safety-critical task by observing its data communication in the network or by observing the changes in physical system states of the controlled plant  <cit.>. By compromising a relatively lower priority task (mostly non-safety-critical) that periodically executes immediately before or after this victim task, the attacker observes its execution timestamps for multiple hyper-periods. 


Since a static fixed-priority schedule is followed, the execution sequence repeats every hyper-period, which helps the attacker infer possible initial offsets of the safety-critical task and predict their future arrival times. 

The attacker can manipulate data inside the victim task's shared I/O device buffer within a time window around those inferred future arrival time instances <cit.>. We term such vulnerable time windows as attack effective window (AEW), a quantity which is implementation-dependent and can be measured experimentally by the attacker <cit.>.


Such schedule-based attacks (SBAs) can be categorised into the following four models depending on the timing relation between the victim's arrival and the attacker task's arrival within AEW <cit.>, i.e., Posterior Attack (attacker task executes after victim task), Anterior Attack (attacker task executes before the victim task), Concurrent Attack (attacker task executes while the victim task runs) and, Pincer Attack (combines both posterior and anterior attack). Since the non-critical attacker tasks are of lower priority, the corresponding jobs are mostly scheduled/completed after higher-priority victim task instances in case of nearby arrival times. This makes posterior attacks more prevalent compared to other attack models in embedded controllers <cit.>. In this paper, we consider the posterior attack model, analyse the drawbacks of state-of-the-art defence mechanisms, and discuss how the proposed defensive solution outdoes them.

Researchers have proposed several solutions to prevent and defend against such SBAs. Among these, schedule randomization-based defence approaches are widely explored. One of the initial works in this area is TaskShuffler <cit.>, which randomizes job execution by allowing lower-priority jobs in the ready queue to run before higher-priority jobs. 

In <cit.>, the authors discuss the following two slot-level randomization techniques to mitigate directed schedule-based attacks. (i) Slot-level online randomization of schedules, for selecting the next job at runtime meeting task deadline constraints and (ii) Offline schedule-diversiﬁcation, where offline precomputed schedules are stored beforehand, and their selection is made online. Such schedule randomization methods obfuscate the schedule information, which makes it difficult for the attacker to predict the future arrival times of the victim task.

However, randomizing without any insight into the attacker's strategy produces some vulnerable schedules during randomization, which can make a random attack attempt successful <cit.>. The authors in <cit.> devise a privacy-preserved strategy that samples a suitable execution offset for critical tasks from a Laplace distribution and switches between them to keep the execution sequences from getting exposed. However, they do not ensure the performance and schedulability criteria of the control tasks while doing so. In another line of work, the authors in <cit.> suggested a temporal protection mechanism for multi-core Linux platforms. In this method, all the non-safety critical tasks, mostly considered to be untrusted, are blocked within the AEW of all the safety-critical or trusted tasks, eliminating all schedule vulnerabilities. However, this significantly delays the execution of untrusted tasks in uniprocessor systems, and they frequently miss their deadlines. If the trusted or critical task utilization is considerably high, such a protection strategy can impose an unbounded delay on the execution of other tasks that hampers the overall system  performance <cit.>. In this work, we propose a schedule randomization policy that is attack-aware and ensures meeting the deadlines for all tasks in the system, by carefully balancing the trade-offs between performance and security. 



One of the crucial pieces of information for any schedule-based attack model is the future arrival instances of a critical task, which is derived by finding out its periodicity <cit.>. If the sampling period is dynamically switched, the attacker will find it difficult to infer the exact schedule even after doing schedule analysis. With this motivation, we utilise multi-rate controller scheduling strategy to ensure security in real-time task schedules by reducing its inferability,  i.e. identifying start times of victim task instances. 

As shown in the state-of-the-art research works, dynamic switching between a predefined set of sampling periods, coupled with optimally designed controllers, significantly reduces resource utilization, ensuring performance, compared to conventional fixed sampling rate static scheduling <cit.>. 

There exist state-of-the-art works that analyse the performance of such systems by analysing their Lyapunov stability <cit.>. Such works mostly analyse whether a Common Quadratic Lyapunov Function (CQLF) exists among the control loops with different sampling rates. The existence of such a function that abides by a given performance criteria guarantees stability when the system is subject to arbitrary or fast switching.



There are works that derive a Multiple Lyapunov functions (MLF)-based <cit.> or dwell time-based switching rules for slowly switching between different modes to ensure stability. Most of these research works focus on optimizing resources or control costs utilizing multi-rate switching strategies <cit.>. In this work, we intend to switch between different controller sampling rates in runtime to reduce the inferability and vulnerability of a deployed schedule towards posterior SBAs. To enable this arbitrary switching, we utilize the CQLF-based fast-switching rules that ensure stability.


We propose a novel  Multi-Rate Attack-Aware Schedule Randomization (MAARS) Framework to secure all the safety-critical control tasks that are scheduled to execute in a uni-processor embedded platform against schedule-based posterior attacks <cit.>. This essentially combines (I) an offline schedule diversification technique that chooses a set of schedules with minimum inferability/predictability for a schedule-based attacker and (II) an online schedule selection technique that suitably deploys schedules to minimize the vulnerability of an attack-affected safety-critical task. 










To the best of our knowledge, this is the first work that proposes an attack-aware schedule randomization policy. The following are the main contributions of this work:

1. We present a novel approach for judiciously choosing performance-aware sampling rates for a safety-critical control task such that the inferability of its future arrival instances are minimized, thereby effectively thwarting the leakage of critical task information (such as periodicity, initial task offset, etc.)
2. We propose a novel methodology that generates a set of schedules offline using slot-level randomization techniques for the chosen set of sampling rates for each control task. We also develop a runtime schedule selection algorithm that intelligently deploys schedules from this set by observing the posterior attacks' effect on a control loop 

such that the vulnerability of the corresponding control task is minimized in that schedule. 





3. We experimentally evaluate how well the proposed MAARS framework balances the trade-off between security and system performance compared to state-of-the-art techniques by applying it to a synthetic automotive task set.
We implement the same task set in a Hardware-in-Loop (HIL) testbed setup to judge the real-time applicability of the MAARS framework. 



§ SYSTEM MODEL




 §.§ Task and Scheduler Model

Let  Γ denote a set of N periodic tasks s.t. Γ = {τ_1, τ_2, ... τ_N  }, that are scheduled on a uniprocessor system by a static priority scheduler with fixed priority. Each task has three parameters (e_i, p_i, d_i), where e_i is the worst-case execution time (WCET), p_i is the sampling period, and d_i is the relative deadline. We assume d_i = p_i, ∀ i≤ N and the WCET of a task includes jitter and context-switch overheads. We consider a mixed-criticality system, i.e., it comprises control tasks of varying criticality levels in order of their priority. 

Following the work in <cit.>, we consider all the safety-critical control tasks as trusted and hard to compromise. Whereas other tasks are considered to be untrusted and can be compromised by a schedule-based attacker. 

Any safety-critical control task from the trusted set Γ_t
⊆Γ can be a victim to a schedule-based attack (SBA) launched by an adversary who has compromised a non-safety-critical task from the untrusted set Γ_u ⊆Γ. Due to the high criticality level, the trusted tasks are mostly assigned with higher priority than the untrusted tasks. Hence, their executions are not delayed by the non-critical tasks as they are promptly scheduled as they arrive by a fixed-priority preemptive scheduler. In this work, we consider these trusted safety-critical control tasks to be designed with different sampling rates, which changes their periodicities of execution. We denote the set of periodicities for i^th control task as P_i={p1_i,p2_i,⋯,pl_i}. We assume that the task set Γ is schedulable by a fixed-priority preemptive scheduler such that the worst-case response time (WCRT)  wcrt_i

for any task τ_i ∈Γ satisfies,


  wcrt_i = e_i + ∑_τ_j ∈ hp(τ_i)⌈r_i^k/p_j⌉ e_j ≤ d_i. 

Here r_i^0 = e_i, wcrt_i = r_i^k+1 = r_i^k for some k≥ 0 and for every τ_i∈Γ_t, p_i= minimum periodicity from the set P_i. This criteria ensures that the tasks are schedulable for their maximum execution rates and assuming no precedence constraints among the tasks,  valid schedules can be generated without any deadline misses. 





 §.§ Control Task Model


We express the plant model as a Linear time-invariant (LTI) system having dynamics as follows.

    ẋ= A_c x(t) + B_cu(t),  y(t)= Cx(t)+v(t)
    
    
       x̂[k + 1] = (A-LC) x̂ [k] + B u[k] + L y[k]
       u[k+1] = K x̂ [k+1]


Here, the vectors x ∈ℝ^n, x̂∈ℝ^n, y ∈ℝ^m, and u ∈ℝ^p describe the plant state, the estimated plant state, output and the control input, respectively. The matrices A_c, and B_c, are continuous time matrices that define the continuous-time state and input-to-state transition matrices, respectively. C is the output transition matrix. At the k^th sampling instance, the real-time is t = kh, where h is the sampling period. We can, therefore, define the discrete-time matrices corresponding to their continuous-time counterparts as A= e^A_c h,  B= ∫_0^h e^A_c tB_c · dt. K and L are Linear Quadratic Regulator (LQR) controller gain and Kalman estimator gain, respectively. Each control task samples the sensed plant measurement vector y at each sampling iteration, using which future states of the plant are estimated. Based on the estimated plant state, a control task calculates the required control input u to transmit it to the actuator to actuate and control the plant output.





  §.§.§ Closed Loop Under Sampling Period Variation


As discussed above, the discrete-time system characteristic matrices depend on the sampling period h. Since the LQR gain is designed to minimize a user-defined quadratic cost function, which is sampled once in every h time interval, the LQR control gain is also dependent on the sampling period. So is the estimator Kalman gain L, as it is also designed to minimize the variance of the estimation error, i.e., the difference between the estimated plant state x̂ and actual plant state x at every sampling iteration.

Therefore, the discrete-time matrices A, B, K, L in Eq. <ref> can be replaced with their sampling period-dependent versions, i.e., A_h,B_h,K_h,L_h. By discretizing the continuous-time plant equation in Eq. <ref> we get x[k+1] = A_hx[k] + B_hu[k]. Since we intend to capture the change in the progression of the control loop w.r.t. the change in its sampling period,

we define an extended augmented state vector of the control task as X = [x^T, x̂^T]^T that captures both plant and estimator/controller dynamics. By replacing u with -K_hx̂ and y with Cx, the overall closed-loop system evolves as follows.


    X[k + 1] = 𝔸_h X[k], 𝔸_h = [             A_h         -B_hK_h;            L_hC A_h-L_hC-B_hK_h ]

For two different sampling periods p1 and p2, we denote the discrete-time augmented system matrices by 𝔸_p1 and 𝔸_p2. Our methodology uses this augmented system representation in Eq. <ref> to ensure a desired performance while varying periodicities of control tasks for security.





  §.§.§ Residue-based Anomaly Detector


Each of the safety-critical control loops is equipped with a residue-based detector that observes changes in the system residue (i.e., the difference between the sensed and estimated outputs) for anomaly detection. We consider an integrated implementation of controller and detector functionalities as part of the control task to avoid data manipulation. We use a χ^2-based detector in this work. The χ^2 detector utilises a normalized quadratic function of the residual to amplify and easily detect minute variations in system residue. For system residue res[k] at k^th sampling iteration, its chi-square measure is z[k] = res[k]^T Σ_res^-1 res[k] where Σ_res is the variance of system residue. Considering the measurement noise to be a zero-mean Gaussian noise, res[k] ∼𝒩(0,Σ_res)⇒ z[k]∼χ^2(m,2m), where m is the number of output measurements, considered as the degree of freedom of the χ^2 distribution. We employ a windowed chi-square detector that compares the average value of the chi-square statistic of system residue over a pre-defined time window (N), i.e., g[k]=1/N∑_i=0^N-1z[k-i] and compares it with a pre-defined threshold Th. The threshold is calculated to maintain a desired false alarm rate <cit.>. The chi-squared detector raises the alarm, denoting an attack attempt on a certain closed loop when g[k]> Th at any k^th sampling period of that closed loop.



























§ THREAT MODEL


In this section, we discuss in-depth how we model the schedule-based attacker and explain our assumptions about the attacker's objectives and capabilities. The objective of the attacker is to utilize the timing information exposed by the processor-level task execution schedules to compromise tasks with higher criticality. 




Since tasks with higher criticality are mostly developed by OEMs or trusted vendors and go through thorough security scans and functionality checks, they are usually very difficult to compromise <cit.>. Hence, an attacker is more likely to attempt compromising less critical tasks, that are often assigned lower priorities. We denote such compromisable tasks as untrusted tasks. Since these untrusted tasks are part of the software that comes from a range of third-party vendors<cit.>, compromising them using a man-in-the-middle-type attack is possible. As mentioned in Sec. <ref>, depending on the timing relationship between the victim and attacker task, SBAs can be categorised into 4 types <cit.>. 

(1) Anterior Attack: An SBA launched before the victim task starts its execution is known as an anterior attack.  e.g., false-data injection type attacks <cit.>. (2) Posterior Attack: An SBA launched after the victim task completes its execution is known as a posterior attack, e.g., replay attack, zero-dynamic attack etc. <cit.> (3) Pincer attack model: Attack is executed both before and after the victim task <cit.>. (4) Concurrent attack: Attack is performed while the victim control task is running and can be launched by executing between the execution window of the victim task’s job <cit.>.

Embedded controllers are mostly scheduled with fixed-priority preemptive schedulers, which schedule the lower-priority tasks after the higher-priority tasks. 

Since untrusted tasks are mostly of lower priority, CPSs with embedded control software systems are more vulnerable to posterior attacks <cit.>. 

This work considers a posterior SBA model and provides a mitigation framework against it. The following are assumptions regarding the attacker's ability, objectives, and attack methods.



  §.§.§ Attacker's Capabilities

We make the following assumptions about the attacker's capabilities: (1) The scheduler is trustworthy and can not be compromised by the attacker. Consequently, the attacker lacks direct control over task execution and is restricted to leveraging compromised tasks only when the scheduler permits.

[8]l0.42
    


    < g r a p h i c s >



   Attack Effective Window
   
        


(2)  The adversary knows the scheduling policy used, and it can compromise one task among all the lower-priority untrusted tasks (Γ_u). This enables the attacker to observe executions of this compromised untrusted task for several hyper-periods to find out partial information about its intended victim control task(∈Γ_t). (3) The attacker can partially deduce the sampling rates of any control task (∈Γ_t) either by observing the message data packets in the network or physical system states.

(4) Attacker can modify the data written by the victim control task inside an I/O buffer or cache only if it executes within an Attack Effective Window <cit.>. AEW is the specific time interval within which the attacker task must be executed for successful data manipulation. This duration is completely implementation-dependent and can be derived by experiments <cit.>. The attacker must tamper the data in the I/O buffer before the transmission task reads and transmits it for actuation. Hence, AEW is the duration after which the transmission task transmits the control program-generated actuation data and is scheduled to execute with the same periodicity as the control task. In Fig. <ref>, we illustrate this where the AEW of the control task has been shaded in red. AEW starts when the control task (in blue shade) finishes execution, writes its data to the buffer and ends once the transmission task (in green stripe) starts.

We denote the AEW length for a control task τ_i ∈Γ_t by Ω_i.




  §.§.§ Attack Execution
 

We assume an attacker's objective is to manipulate control inputs by performing a schedule-based posterior attack on control tasks. To implement such an attack, the attacker must infer the exact future arrival instances of the victim task. Having compromised a lower-priority untrusted task, the attacker can log that task's start and end time as part of the overall schedule. Utilising this information, it attempts to predict the start and end times of other trusted tasks in the schedule. The attacker can achieve this by using a schedule ladder-based analysis proposed in the work <cit.>. We briefly discuss how the schedule ladder is constructed, and the attacker uses it to derive the future arrival instances of the victim task.
[5]l0.30

font=scriptsize
    
        Tasks     P     e

        τ_1     {4,5}     1

        τ_2     4     1

        τ_3     5     2

        
    
    Example Task-Set 1









A schedule ladder represents a task schedule arranged vertically with adjacent timelines of equal row size. Each column in the ladder represents an atomic observation period, or unit time, of real-time duration δ. The attacker takes the row size of the ladder as equal to the victim task's sampling period measured in time units of size δ. Hence, for a given victim task τ_i executing with a sampling rate p_i time units, the row length of the ladder becomes δ p_i. The motive behind taking such a row length is that the victim task will arrive once in every row of the ladder and will occupy δ e_i columns. Moreover, all arrivals of τ_i will be located in the same column. We illustrate a schedule ladder diagram for the task set in Tab. <ref> in  Fig. <ref>. The row length of the ladder is 4 since τ_1(victim) sampling rate = 4. The first row represents the timeline [0,4], second row [4,8] and so on. The blue arrow shows the arrival column of the victim task that arrives once in every row and occupies column 1 of the ladder.  

While performing an analysis, the starting point of analysis (observation) of the top row can be arbitrarily chosen by the attacker, which is decided by the start of its observation window. The attacker starts observing its own start and end times using the compromised untrusted task. Note the attacker task is a lower priority untrusted task (w.r.t victim). Hence, if the victim task arrives in the same column, it preempts the attacker task. This is shown in Fig. <ref>, where the attacker task τ_3 arrives at instances t=0,20 in the first column (since its sampling rate is 5) but gets preempted by τ_1. In column 2 it gets preempted by τ_2. Therefore, it always executes in the third and fourth column of the ladder.

[13]l0.41





    < g r a p h i c s >

 Ladder diagram of Task-Set-1 illustrated along with their corresponding AEI and AAI.






After the end of observation, the attacker finds out Attack Arrival Instances (AAI). The AAI is the set of ladder columns where the attacker task has arrived at least once during its entire observation window. For e.g. in Fig. <ref>, the AAI is [0,4], shown below the ladder diagram. The set of ladder columns where the attacker task has been executed at least once during its entire observation window are termed as Attack Execution Instances (AEI). Note that the ladder columns where the attacker task arrives (∈ AAI) but never gets executed (∉ AEI since it is preempted by higher priority victim tasks) are the possible candidates for the columns where the victim task arrives. Hence, the attacker easily infers these ladder columns, which are in AAIs but not AEIs, as the victim task's arrival column (offset), predicting them as the victim's future arrival instances. We illustrated this in Fig. <ref> for task-set Tab. <ref>, where using τ_2 as an attacker task results in AAI=4 and AEI=2. One immediate conclusion we draw from this discussion is that a longer execution time for the attacker task make it easier for an attacker to infer the arrival column of the victim task. This is because the attacker task execution will span a greater proportion of ladder columns, making it easier to guess the correct victim task arrival column.




Consider the task set in Tab. <ref>, where the trusted control task set Γ_t ={τ_1, τ_2} and the untrusted task set Γ_u ={τ_3} (Fig.<ref>). After determining the arrival time of the victim task τ_1, the attacker uses untrusted task τ_3 to launch a posterior SBA on τ_1. We assume the AEW of τ_1 is Ω_1 = 1 time unit (highlighted in the shaded region of the timeframe). Therefore, for a successful SBA, τ_3 must execute within this timeframe after τ_1 finishes executing. Using this threat model, we demonstrate how state-of-the-art secure scheduling policies can mitigate such attacks and discuss their shortcomings in balancing performance and security, which motivates our novel methodology.














§ MOTIVATING EXAMPLE





We motivate our proposed methodology with an example by discussing how (i) the conventional protection window-based secure scheduling methods, such as <cit.>, fail to ensure system performance, optimal resource utilisation, etc., and (ii) the randomization-based scheduling methods, such as <cit.>, fail to guarantee security against SBAs. This, in turn, motivates our methodology, which utilises variable sampling period-based implementation of control tasks for balancing the trade-offs between security and performance without compromising the schedulability of the task set.
[5]l0.38

font=scriptsize
    

    
        Tasks     P     e

        τ_1     {2,3}     1

        τ_2     4     1

        τ_3     4     1

        
    
    Example Task Set-2
    


We refer the task set in Tab. <ref>, where the trusted control tasks Γ_t ={τ_1, τ_2} and the untrusted tasks Γ_u ={τ_3}. The control task τ_3 executes at two different sampling rates 2 and 3 time units (δ time unit=10 ms), respectively. We assume the attacker has compromised the untrusted task τ_3 to launch a posterior attack on the victim task τ_1. The AEW duration for τ_1 is Ω_1=1 time unit. 

[7]l0.55



    < g r a p h i c s >


AEW-based scheduling of Taskset-2





Protection window-based approaches such as in <cit.> propose restricting the execution of untrusted tasks' job instances within the AEW of all the trusted tasks. 

Fig. <ref> demonstrates how a task schedule generated following <cit.> (for the task set in Tab. <ref>) leads to deadline misses of untrusted tasks and wastage of CPU resources. The first job of τ_1 executes at [0,1], In the next interval at [1,2], job instance of τ_2 executes. In [2,3], second job of task τ_1 arrives and executes. However, the job instance of τ_3 will not be allowed to execute at [3,4] and hence will miss its deadline, even though the CPU remains idle. This leads to inefficient CPU utilisation.
On the other hand, the schedule randomization methods are 

more suitable since they make sure every task meets its deadlines (irrespective of trust). However, the state-of-the-art randomization policies do not consider the attacker's model into consideration and hence are often more vulnerable to SBAs while randomly changing the schedules <cit.>.

Earlier works <cit.> have demonstrated that selecting schedules using such attack-unaware randomization policies can still lead to successful attacks. 



[12]l0.6




    < g r a p h i c s >

Feasible Schedules Generated using Task-Set 2 (Considering p_1=2)






For example, if we apply such a schedule randomization method <cit.> on the same task set for a single sampling rate of p_1=2, for control task τ_1, we can generate a total of 8 unique feasible task schedules of hyper-period length of 4 time units (see Fig. <ref>). But among them, 4 schedules s_4,s_5,s_6 and s_7 are vulnerable for τ_1, since instances of τ_3 appears within the AEW of the victim task τ_1. However, if we use two different sampling periods for the control task τ_1,  P_1={2,3}, we can form two possible task specifications Γ_1 and Γ_2 with unique sets of sampling period, i.e., {p_1=2,p_2=4,p_3=4} and {p_1=3,p_2=4,p_3=4}. For the task specification Γ_2, we can generate 36 unique and feasible schedules (using TaskShuffler) with a hyper-period length of 12 time units. By analysing the vulnerability of τ_1 for all 36 schedules, 12 schedules are found to be safe (no posterior attackable instances within AEW) for victim task τ_1. Thus, multi-rate execution enables the generation of a higher number of randomized schedules with minimal vulnerability.


[14]l0.36
font=scriptsize
    


    < g r a p h i c s >

     Schedule Ladder Diagram for Task Set-1
    


Moreover, multi-rate task execution also hampers the analysis of the attacker task. We refer to the Example Task Set-1. After corrupting an untrusted task τ_3, the attacker observes its execution using a schedule ladder diagram. In the case of a single sampling rate, the attacker can figure out the future arrival instances of τ_1. Since the attacker never executes in the first column, i.e., within [0,1] time units, hence the attacker infers that the victim task must arrive in the second column, i.e., within [0,1] time units. This has been illustrated in case-1 of Fig. <ref>. However, due to a change in the victim task's (τ_1) sampling rate after one t=15 (sampling rate changes to 5 from 3), the attacker can execute in all three columns, i.e., within [0,1],[1,2] and [2,3] time unit column. This has been illustrated in Fig. <ref>. Therefore, the attacker cannot infer the exact column where τ_1 arrives from the schedule analysis. 

From these examples, we draw a conclusion that utilising multi-rate for trusted tasks along with imparting attack awareness to the task schedules (w.r.t trusted tasks) can 
hamper a schedule-based posterior attacker's 

 success rate. 




§ PROPOSED METHODOLOGY




In this section, we present the methodology for generating multi-rate attack-aware randomized schedules. 








A step-wise overview of the MAARS methodology is shown in Fig. <ref>. During the design time, our proposed framework takes the following as inputs: (i) a task set Γ consisting of both trusted and untrusted set of tasks, Γ_t,Γ_u (⊆Γ) respectively, (ii) trusted task specifications, i.e., criticality level/ priorities c_i, a controllable, schedulable set of periodicities P_i, WCRT e_i of each trusted task τ_i∈Γ_t, (iii) trusted task specifications, i.e., criticality level or priority c_j, periodicity p_j, WCRT e_j of each untrusted task τ_j∈Γ_u, (iv) physical system specifications corresponding to each control task τ_i∈Γ_t, its performance specification, (v) other tunable design parameters related to MAARS, like Tolerable Attack Probability, Maximum Schedule Vulnerability Index, etc. to constrain the vulnerability of MAARS as defined in Sec. <ref>




As motivated in Sec. <ref> switching between the allowable set of sampling periods P_i reduces the determinism (explained as inferability in Sec. <ref>) in the real-time task execution pattern. The first step in our methodology is to prune a set of periodicities P_i for a given critical/trusted task such that the system, in closed loop with this control task, delivers its desired performance (see Fig. <ref>).

In the second step, we intelligently select a subset of secure sampling periods from this performance-aware set of periodicities, such that the attacker's schedule analysis is maximally hampered. We theoretically prove that the devised secure periodicity selection policy maximizes the non-determinism in the execution pattern of the victim control task, by preventing the attacker from inferring critical task parameters. The third step uses these performance-aware and secure set of sampling rates to generate all possible valid task schedules by following the schedule randomization strategy given in <cit.>.

These valid schedules are 
 generated by ensuring that no task misses their deadlines while switching between the periodicities. In the fourth step, we analyse the probability of posterior schedule-based attack (SBA) on each victim task for each schedule. By quantifying the vulnerability of each schedule, we rank and suitably store the schedules for using them in runtime to prevent posterior SBAs.

During run-time, MAARS deploys a secure schedule selection algorithm. As discussed earlier, each control task is implemented with a windowed χ^2-detector that flags any attack attempt on its corresponding closed-loop (box with black-dotted outline in Fig. <ref>, refer to Sec. <ref>). Under no attack, MAARS randomly selects and deploys different schedules from a designer-selected set of valid schedules with minimum vulnerability. We term this mode as Normal Mode (green-dotted outlined box in Fig. <ref>). MAARS switches to an Alert Mode when the detector flags an attack on a certain victim task τ_i∈Γ_t(red-outlined box in Fig. <ref>). In this mode, the schedule selection algorithm switches to the least vulnerable schedule for victim task τ_i to ensure minimum posterior attack success. In the following sections, we discuss each design time step and run-time algorithm in detail.







 §.§ Performance-aware multi-rate Period Selection








To ensure a desired performance while arbitrarily switching between the closed loops with a schedulable sampling periods to reduce the inferability of victim task parameters, we must constrain our switching choices.

Let us consider, for the augmented closed loop system (as mentioned in Eq. <ref>) corresponding to a control task τ_i∈Γ_t, this sampling period switching signal is σ_i:ℕ↦ P_i, where P_i={p1_i,p2_i,⋯,pn_i} represents the chosen set of non-zero sampling periods. In presence of this switching signal, we can express Eq. <ref> for the control task τ_i as

 
    X^(i)(k + 1) = 𝔸^(i)_σ_i(k)(X^(i)(k))

 
the switching signal σ_i(k)=pj signifies that at k^th sampling instance, this augmented system switches to the controller designed with sampling period pj. For a switched system like Eq. <ref>, each such switchable closed loop systems are considered as its subsystems. As stated in <cit.>, the following condition must be satisfied for maintaining a desired performance while arbitrarily switching between the closed loop subsystems.


    In the case of a switched system like Eq. <ref>, in order to maintain a desired global uniform exponential stability (GUES) decay rate of γ while arbitrarily switching among different sub-systems, there must exist a common Lyapunov function (CLF) for all its arbitrarily switchable subsystems that satisfy the following set of equations
    
    
    κ_1(‖ X^(i)[k]‖)≤ V_i(X^(i)[k])≤κ_2(‖ X^(i)[k]‖)
        V_i(X^(i)[k])≤α_i V_i((X^(i)[k]), for α_i=g(𝔸^(i)_pj) s.t. σ_i(k)=pj


Here, κ_1,κ_2 are class 𝒦_∞ functions, and g:{𝔸^(i)_p1,⋯, 𝔸^(i)_pn}↦ ℝ∖{0}. Global uniform exponential stability (GUES) with a decay rate γ<0 signifies that at k^th sampling iteration ‖ X(k)‖≤ M e^γ k ‖ x[0]‖, where ‖.‖ is vector norm and M >0. We consider a common quadratic Lyapunov function (CQLF) V_i(X^(i))=X^(i)𝒫^(i)X^(i)^T. We validate the existence of this CQLF for all switchable closed loops by solving the following linear matrix inequalities (LMIs) and finding out a positive definite matrix 𝒫^(i)>0 such that for given α_j, 𝔸^(i)_pj^T𝒫^(i)𝔸^(i)_pj-𝒫^(i)≤α_j𝒫^(i) ∀ pj∈ P_i. Given a large set of periodicities, we only consider the set of sampling periods for which their corresponding closed loops have a CQLF that follows Claim <ref> for a given performance criteria. This ensures that the desired performance is guaranteed while we devise an arbitrary switching strategy between these closed loops to minimize the vulnerabilities of the task execution schedules against posterior SBAs.



 §.§ Secure Sampling Period Selection and Schedule Generation


We devise a novel periodicity selection policy to further prune the set of performance-aware sampling periodicities for a victim control task to maximally reduce the leakage of its critical task parameters (eg,. arrival times, execution offsets).

As discussed earlier, the attacker is capable of figuring out the victim task's sampling rate by observing the network traffic or characteristics of the physical plant <cit.>. 

The attacker chooses the smallest periodicity of the victim task to construct its schedule ladder as the smallest row length of the ladder increases the cardinality of Attack Arrival Instances (AAI) to infer the correct future arrival instances of the victim (refer Fig. <ref>). 

As discussed earlier, an attacker with a longer execution time is a more suitable choice since it increases the chance of getting preempted by a victim instance and, thereby, increasing 
the cardinality of Attack Execution Instances (AEI). 


 

We defend against such an attacker by suitably choosing a set of sampling periods for the higher-priority victim tasks so that the chances of preemption are reduced (recall that an attacker infers the timing information of a victim task based on its preemption by the victim). To quantify the inferability of victim task arrivals, considering an attacker task in a schedule, we introduce a metric inferability ratio that represents the number of preemptions by the victim task. 
 










(Inferability Ratio)
    The Inferability Ratio (IR) of a schedule s for a victim task, τ_i having minimum sampling rate p_i and an attacker task τ_j, is defined as the ratio between the total number of columns in the schedule ladder where the attacker task τ_j arrives and gets executed
    
    and the total number of columns where the attacker task τ_j arrives. Mathematically,  Inferability Ratio IR = | AEI|%| AAI |/| AAI|, where |·| denotes cardinality of a set. The numerator uses the modulo operation to reset the numerator when the attacker executes wherever it arrives, indicating no inferability due to no preemptions by higher-priority tasks.
 
If an attacker task is executed in more number of columns in a schedule ladder designed for a victim task, the IR of that schedule, w.r.t. the victim, and its attacker choice is high. This limits the number of columns where the attacker task arrives but does not get executed due to preemption by higher-priority victim tasks. Therefore, it is easier for the attacker to guess the arrival column of the victim task. Intelligent selection of sampling rates can reduce the IR by reducing the number of columns where the attacker task is preempted by the victim task, i.e., the attacker arrives but does not execute. 

We claim the following to ensure reduced preemption of the attacker task by suitable selection of victim periodicities: 














Consider an attacker task τ_j ∈Γ_u and a victim control task τ_i ∈Γ_t. For τ_i, let the set of periodicities allowed be P_i with minimum sampling period p_i = min{p | p ∈ P_i }. In this scenario, the generated fixed-priority preemptive schedules shall have the least Inferability Ratio (IR) if for each p' ∈ P_i ∖{p_i}: p' = n p_i + e_j, n ∈ℕ^+.


 We consider that an attacker constructs the schedule ladder with a row length equal to the minimum sampling period p_i of the victim task τ_i∈Γ_t and observes the AAI and AEI of an attacker task τ_j∈Γ_u. 

The duration between the attacker task's arrival 

and the subsequent victim task's arrival 

repeats after every LCM(p_i,p_j) unit of time. Note that the e_j time units

are covered by every execution instance of the attacker task. As long as this 

the duration between the attacker's and the subsequent victim's arrival is smaller than e_j 

time units, the attacker task will be preempted by the victim task, and the same preemption will repeat 

after every multiple of LCM(p_i,p_j) in the timeline. 


Now, let us consider that at k-th instance of the victim task, there is a predicted chance of preemption while the victim is running with some np_i periodicity where n, k∈ℕ^+. Therefore, this k-th victim instance, that arrives at knp_i time  may preempt the ⌊knp_i/p_j⌋-th 

instance of the attacker task. Therefore, for successful preemption, the k-th victim task instance must arrive before the execution of  ⌊knp_i/p_j⌋-th attacker instance is finished, i.e., ⌊knp_i/p_j⌋ p_j+e_j > knp_i

Now, consider changing the periodicity of the victim task to some p^'_i, to avoid this preemption. Therefore p^'_i must ensure that the 

k-th victim instance arrives after the ⌊kp^'_i/p_j⌋-th attacker instance finishes, i.e., ⌊kp^'_i/p_j⌋ p_j+e_j ≤ kp^'_i. By substituting p^'_i with np_i+e_j, this equation becomes as follows.

    k(np_i+e_j)    ≥⌊k(np_i+e_j)/p_j⌋ p_j+e_j = ⌊k(np_i)/p_j+k(e_j)/p_j⌋ p_j+e_j
    
       ≥⌊k(np_i)/p_j+ ⌊ke_j/p_j⌋ + ke_j % p_j⌋ p_j+e_j, a/b = ⌊a/b⌋ + a%b, 
       ≥⌊k(np_i)/p_j⌋ p_j+ k e_j + ke_j % p_j+e_j,  since k, e_j, p_j ∈ℕ^+
       ≥ knp_i-knp_i%p_j+ke_j-ke_j%p_j
       ≥ k(np_i+e_j)-k(np_i%p_j-e_j%p_j)




Therefore, the (LCM(p_i,p_j)/p_i-1)-th instance of the victim's arrival must be shifted by k= e_j number of columns to avoid this preemption. Changing the periodicity of the victim task to p^'_i=np_i+k after (LCM(p_i,p_j)/p_i-1)-th execution achieves this shift avoiding preemption. 
Guiding Example: In case-2 of Fig. <ref> the sampling period of the victim task τ_1(execution marked with blue) is p_1=3 (execution time e_1=1), and the attacker task τ_3 (execution marked with red) is p_3=5 (e_3=2, see the task set in Tab. <ref>). The arrival distance between the victim and the attacker repeats after LCM(3,5)= 15 time units. The victim preempts (preemptions marked with red stripes) the attacker at  3 and 12 unit time at the 1st column and is expected to preempt again in the same column at 18, 27 unit time. This is because the attacker task τ_3 shifts by p_3%p_1=5%3 columns in every consecutive arrival, but the victim task τ_1 always arrives at the 1st column.

Since the schedule ladder has a row length, same as the minimum periodicity p_i of victim task τ_i, every other arrival instance of victim task with p^'_i periodicity will now shift by p^'_i% p_i or k≥ e_j columns. We know that the attacker task shifts p_j% p_i columns in every other arrival instance. To avoid preemptions in the following arrival instances, the arrival of the victim task must shift as much as the attacker's arrival shifts, i.e., k= p_j% p_i. 


This always holds true since k(np_i%p_j-e_j%p_j)>0, because for τ_j to be schedulable p_j>e_j. Note that, e_j, p_j are considered as whole numbers since we express them in multiples of δ, and the analysis assumes δ as the notion of unit time. Therefore, this choice of periodicity p^'_i = np_i+e_j ensures no preemption of the attacker by any k-th victim instance. This, in turn, proves the claim that scheduling the victim with a set of periodicities P_i such that ∀ p^'∈ P_i{p_i}, p^'_i= np_i+e_j ensures the least IR for an attacker task τ_j∈Γ_u in any execution schedule.





The periodicity selection criteria used in Claim <ref> heavily restricts the periodicity choices since higher values of n∈ℕ^+ are often not supported by Claim <ref>. We update the criteria as follows in order to get multiple periodicity choices for a victim task with reduced IR. 


For a victim task τ_i, let the set of periodicities allowed be P_i with minimum sampling period p_i = min{p | p ∈ P_i }. In this scenario, the generated fixed-priority preemptive schedules shall have a reduced Inferability Ratio if for each p' ∈ P_i ∖{p_i}: p' = n p_i + k', n ∈ℕ^+, such that k'∈ [e_j,p_i-1] considering an attacker task τ_j with periodicity p_j and execution time e_j. Since, k'≥ e_j, it ensures zero preemption against any attacker, i.e., IR=0. Since, k≤ (p_i-1), the predicted preemption after every LCM(p_i,p_j) time is not repeated and delayed. This clearly reduces the cardinality of AEI, ensuring the reduction in IR compared to the situation when the victim task was scheduled with p_i.



We thus prune the performance-aware set of periodicities P_i to make it security-aware. For each victim task τ_i, we eliminate the periodicities from the set P_i that do not satisfy the criteria given in Remark <ref>. 





















Once all the possible sets of P_i for each victim control task τ_i are pruned, we utilise TaskShuffler  <cit.>, a schedule randomization technique. 

As defined in <cit.>


    the worst-case maximum inversion budget (WMIB) of task τ_i, denoted by WMIB_i, is the maximum amount of time for which all tasks with priority lower than task τ_i (denoted by lp(τ_i)) are allowed to execute while an instance of τ_i is still in the ready queue, while assured of meeting it's deadline in worst-case scenario. 
    It is calculated by,
    
    
    WMIB_i = d_i - [ e_i+ ∑_τ_j ϵ hp(τ_i)( ⌈d_i/p_j⌉  +1 ) e_j].





This randomization methodology shuffles job instances while ensuring that each job deadline is met by constraining unbounded priority inversions within specified limits (WMIB_i). 

Note that after previous steps our task set Γ=Γ_t ∪Γ_u, has a set of period choices for tasks in Γ_t. Each control task τ_i ∈Γ_t is assigned a performance-aware and secure set of sampling periodicities, i.e. P_i= {p_i^1,p_i^2...,p_i^n_1}. 
For |Γ_t | = q, the number of task specifications for Γ is given by: |P_1| ×⋯× |P_q|, as the periods of tasks ∈Γ_u are assumed as unique.  Each task specification is given as input to the TaskShuffler algorithm for generating the possible set of randomized as well as feasible schedules where no job instance misses the deadline. We denote the overall set of randomized schedules generated considering all task-specifications by 𝒮.

























 §.§ Schedule Vulnerability Analysis




In order to characterize each valid/feasible fixed-priority preemptive schedule's vulnerability level, we first formalise the task schedule s as an array, s = {s[1], s[2],⋯,s[l]}, ∀ s[j] ∈{0,1,⋯, N}. Here, l is the number of time units (each time unit of size δ, same as the column sizes of the schedule ladder, refer to Sec. <ref>) in the schedule hyper-period (the time duration after which a static task schedule repeats itself). The j^th element s[j] in this array denotes the priority of the task executed at j^th time unit, i.e., s[j] = i⇒τ_i∈Γ is executed at δ j time. We use 0 to denote an idle period in the schedule, and i ∈{0,1,...N} denotes the fixed priority assigned to the executed task at a certain time unit. For example, s={1,2,1,3} refers to a fixed-priority preemptive schedule for a task set <ref>. Job 1 of task 1 with priority 1 executes first in the execution order, then job 1 of task 2 with priority 2, and so on. 

Given any schedule s∈𝒮, executing multiple trusted and untrusted tasks from the task set Γ, first we count the total number of possible posterior attacks on s. Given a victim task τ_i with an AEW length of Ω_i, Eq.<ref> gives a total count of all possible posterior attacks on task τ_i denoted by C_p(τ_i) in a schedule s of length l. 


    C_p(τ_i) = ∑_j=1^l I( (s[j]==i) ⋁_∀ k ∈ [1, Ω_i]∃τ_p∈Γ_u, (s[j+k]==p) )

Here, I(x) is an indicator function defined as follows: I(x)=1 if x is true, else I(x)=0 for a proposition x that returns true/false. The indicator function returns 1 if any job from τ_p ∈Γ_u executes in any of the AEW Ω_i of task τ_i∈Γ_t. 
Note that even though the attacker can compromise a single untrusted task, the system designer analyses the vulnerability of a posterior attack on each trusted task τ_i∈Γ_t by assuming every untrusted task τ_j∈Γ_u as a potential attacker.  


Since sampling rates of all the control tasks τ_i ∈Γ_u across different schedules vary, the number of job executions of the task that arrive in each schedule hyper-period will also vary. Therefore, it is necessary to express this posterior attack counts on a trusted task τ_i for a schedule s_k∈𝒮 in probabilistic terms. 

We denote this as the Attack Probability, which is defined as follows:

(Attack Probability)

     Given a valid schedule s_k(∈𝒮) generated for a task set Γ, the Attack Probability AP_<τ_i,s_k> of a victim task τ_i ∈Γ_t⊆Γ is the ratio between the total number of attacker tasks getting executed within the AEW length Ω_i of victim control task τ_i,  and the total number of job-arrivals of the task τ_i in the schedule hyper-period of length l_k. Mathematically, AP_<τ_i,s_k> =  C_p(τ_i)  × p_i/l_k. 
     



For each trusted control task  τ_i∈Γ_t, the AP_<τ_i,s_k> is different in a schedule s_k.

For each safety-critical control task τ_i ∈Γ_t, we define its Tolerable Attack Probability (TAP) as the maximum attack probability on a schedule s that can be tolerated by the closed-loop system, associated with τ_i such that the plant operates in a safe and preferable operating region. For a control task τ_i, we denote its TAP by TAP_i,s_k.

In a mixed-criticality system, the system designer chooses the criticality values of the tasks based on their importance to the system's operations and how their compromise would affect the system's overall performance <cit.>. Considering tasks with a higher index have a lower priority value, the q safety-critical tasks {τ_1, τ_2,.., τ_q } are arranged in decreasing order of priorities.  We assign them with their criticality values {q, (q-1),⋯,2, 1} respectively. We normalize the criticality values by their total sum of criticality values to derive a criticality level of each task. Therefore, we denote the normalized criticality levels for a task τ_i as cl_i = c_i/ ∑_j=1^kc_j,  ∀τ_j∈Γ_t. Since AP is task-specific, a schedule may have a minimum attack probability associated with one safety-critical task, whereas it might have the highest AP for the other safety-critical tasks. To secure the schedule w.r.t. all victim tasks, we use these criticality levels to decide the APs of which higher criticality tasks should be given more importance while quantifying the level of vulnerability of the overall task execution schedule. We define a Schedule Vulnerability Index (SVI) for each schedule that is expressed as a weighted (with criticality levels) average of the APs of all high-criticality tasks.

 (Schedule Vulnerability Index)
Given a schedule s_k ∈𝒮 generated by a task set Γ, its Schedule Vulnerability Index, denoted by the symbol SVI_k, is the weighted sum of attack probability AP_<τ_i,s_k>, weighted with normalized task criticality levels c_i for each control task τ_i.
Mathematically, we can write 


SVI_k= ∑_∀τ∈Γ_t AP_<τ_i,s_k>× cl_i  ∀τ_j∈Γ_t


Similar to TAP, we intend to find a tolerable upper limit of SVI. We term this as Schedule Vulnerability Threshold (SVT). The schedules having SVI above SVT are not normally (we shall define what we mean by normally in the next section) deployed in order to reduce vulnerability against posterior SBAs. This ensures that the attack probability thresholds of the higher-criticality (here same as priority) trusted tasks are taken more into consideration during this vulnerable schedule elimination process than that of the lower-criticality victim tasks. Therefore, the Schedule Vulnerability Threshold denoted by SVT can be determined by replacing the Attack Probability with TAP as SVT_k = ∑_i=1^q TAP_i × cl_i.






The attack-aware schedules generated using the methodology mentioned in Sec. <ref> are pre-computed and stored along with their SVIs and victim task-wise AP values after this vulnerability analysis (See Fig. <ref>). This process eliminates the need for complex online computations in real-time, thereby minimizing the runtime overhead associated with schedule randomization. 




All valid fixed-priority schedules are stored in an array 𝒜, sorted in the order of their respective SVIs. At the K-th index, the schedule with SVT value is stored. Another data structure, Schedule Lookup Table (denoted with LUT),  stores task-wise an array of schedule indices (from 𝒜) which show less attack probabilities w.r.t. a victim task than its TAP. An array of schedule indices stored in the j-th row of LUT is sorted in the increasing order of AP w.r.t. task τ_j. For example, if the i_1, i_2-th schedules from 𝒜 (i.e., 𝒜[i_1]=s_i_1, 𝒜[i_2]=s_i_2) are stored at k, k^'-th positions of LUT[j] (j-th row in LUT) respectively, with k< k^', then AP_⟨τ_j,s_i_1⟩<AP_<τ_j,s_i_2>≤ TAP_j.  

In the next section, we discuss the online counterpart of the MAARS framework that uses these stored data structures after the design time analysis for randomized deployment of task execution schedules in an attack-aware manner.






 §.§ Schedule Selection Algorithm
















The runtime algorithm in the MAARS framework 

is responsible for randomly deploying schedules in an attack-aware fashion. The runtime schedule selection function SchedSel in Algo. <ref> takes the following inputs, (i)  The lookup table LUT and the stored schedule array 𝒜, (ii) the χ^2-detector flag Atkflag, (iii) the schedule vulnerability threshold SVT and (iv) the schedule deployed in the current hyper-period s_k. 
The algorithm starts by storing the schedule index in 𝒜, which has the same SVI as the input SVT. We use the IndexOfSVI method, which takes the sorted schedule array 𝒜 and the SVT as inputs to find out this index and store in K (see line <ref>). In line <ref>, by using IndexOf method we find out the index k of the input schedule s_k that is currently deployed. Another variable, k', is also initialised (with -1) to store the index of the next schedule. 


The algorithm has two distinct modes of schedule selection and deployment operations depending on the detector flag Atkflag. 
(1) Normal mode:  When there are no attack flags raised, i.e., Atkflag=0 denoting no attack on any control task, the algorithm operates in Normal Mode (lines <ref> to <ref>). In this mode, schedules with SVI less than SVT are randomly selected from the array 𝒜. Line <ref> of the algorithm checks if the AtkFlag value is zero. If it is zero, then the algorithm selects a random number in the range [0,K-1] and stores it in k' (see line <ref>). Since MAARS randomly deploys new non-vulnerable schedules at every hyper-period, the random schedule index k' must not match with the currently deployed schedule index k. The algorithm keeps picking another random number until k≠ k' for this purpose.

(see the while loop in line <ref>). The next schedule s_k^' is selected from the k'-th row of the schedule array 𝒜 (see line <ref>).  

(2) Alert Mode:  If the χ^2-detector running inside the control task raises an alarm by detecting an attack (g[k]>Th⇒ AtkFlag>0, see Sec. <ref>) on a control task τ_i, the algorithm runs in Alert mode. In this mode, schedules are selected from the schedule array 𝒜 such that the attack probability for the victim control task τ_i is lower than its tolerable attack probability TAP_i. In line <ref>, if AtkFlag is non-zero, we first store its value in i. The value of the Atkflg denotes a potential attack on task τ_i. In line <ref>, we find out the length of the i-th row in LUT and store it in M. This is used to find a random number ki in the range [0,M-1] (see line <ref>). This number is then used to get the index of the next schedule k' from the ki-th column and i-th row of LUT. Note that each such schedule index picked from LUT satisfies AP_<τ_i,𝒜_k'> < TAP_i. As stated earlier, this random number selection is repeated if the index of the current schedule k matches with the randomly picked next schedule index k' (see the while loop in line <ref>). We select the schedule s_k^' from the k'(≠ k)-th row of 𝒜 (see  line <ref>). 

Finally, SchedSel returns the new schedule s_k' for deployment. Algo <ref> is run once every hyper-period in order to select and deploy new schedules randomly. This online schedule selection is done in an attack-informed way. As a result, the vulnerability level of the task execution schedule is minimized while maintaining the system's performance. 


Overhead Analysis:
The time complexity of Algo. <ref> depends on the complexity of selecting a random number so that it does not match the current value. Since we store the schedules in the sorted orders of SVI and AP in arrays, the algorithm can access them in O(1).  



The total memory cost for storing 𝒜 and LUT can also be calculated, assuming that each element(of the arrays) takes b data bytes. Each schedule, therefore, comprises rows equal to the length of its hyper-period, say l_k for the k-th schedule. Considering a total of M schedules, the total cost is (∑_k=1^M l_k) · b bytes.

LUT, is stored as a 2-D array of q rows (number of trusted control tasks) and M columns, the cost is qM. Hence, the total memory cost is  [ qM + ( ∑_k=1^M l_k ) · M ] · b bytes.
In the next section, we demonstrate the efficacy of this runtime schedule selection algorithm guided by the design-time analysis in real-time system setup.





















§ EXPERIMENTAL EVALUATION



Experimental Setup: We experimentally evaluate our proposed MAARS framework in a Hardware-in-Loop (HIL) testbed and compare it with state-of-the-art (SOTA) approaches against posterior SBAs. 

The first step of the experiment involves design-time schedule generation and analysis for inferability and vulnerability of safety-critical trusted control tasks. This is performed using MATLAB and Python on an 8-core 7th generation Intel i7 CPU with 16 GB of RAM. The implementations can be found in <cit.>. In the next step, we perform a run-time analysis of the control tasks under posterior SBA. For this purpose, we employ an ARM-based 32-bit Infineon Tricore Aurix-397 ECU, connected to ETAS Labcar, a Real-time PC (RTPC), via Controller Area Network (CAN). The implementations can be found in <cit.>.




The physical system/plant is implemented on the RTPC, which sends the sensor measurement to the ECU via CAN, where the control tasks are co-scheduled. Following AUTOSAR mandates<cit.>, we design separate reception tasks to filter and receive sensor IDs transmitted by the ETAS Labcar RTPC over CAN. After processing these sensor data


the control tasks (scheduled by fixed-priority preemptive scheduler in the ECU) are executed. Subsequently, corresponding transmission tasks send computed control input through CAN to the RTPC facilitating plant actuation. Our proposed run-time schedule selection process SchedSel 

is scheduled to run once during each hyper-period in the same core as the other control tasks on the ECUs. Note that SchedSel doesn't use data from the buffer; rather, it receives the detector signal in an interrupt-driven manner and provides a new schedule to the scheduler for the next hyper-period. So the attacker cannot compromise its operations.

[7]l0.53



Task     p_i(ms)     e_i(ms)     c]@l@Utilization
 (Low/High) 
 τ_5     10     1     3*Low, High 
 1-3
τ_6     30     1      
 1-3
τ_7     20     1      
 τ_8     30     5     2*High 
 1-3
τ_9     10     2      
 
Experiment: Untrusted Task Set





We implement four distinct automotive controllers 

similar to the experimental setup in <cit.>. These control tasks include i) an electronic stability program (ESP), responsible for maintaining yaw stability, ii) trajectory tracking control (TTC) system that regulates deviation from a desired longitudinal trajectory, iii) cruise control (CC) system that ensures 

a desired vehicle speed, and iv) suspension control (SC) system that maintains vehicle suspension across varied roads and driving conditions. 

All four control tasks' parameters, their respective AEWs, criticality levels, and TAPs are tabulated in Tab. <ref>. The parameters of the untrusted task set are listed in Tab. <ref>. We have performed the analysis on two different processor utilization scenarios: Low Utilization (LU) (≤ 60% with less no. of untrusted tasks, refer to utilization column in Tab. <ref>) and High utilization (HU) (≥ 90%). 


(1) Design-time  Analysis: 

To generate a set of security-aware task schedules, first, it is required to synthesize a set of periodicities for the control tasks (Tab. <ref>) that ensures their control performance. To achieve this, we solve the LMIs given in Claim <ref>

using 

YALMIP <cit.> along with Gurobi  solver <cit.>. 

The performance-preserving set of periodicities of each control task is further pruned following Claim <ref> and Remark <ref>,

to derive the secure sampling rates for all the control tasks. The final pruned set of security-aware performance-preserving periodicities P_i of each control task are presented in the 3-rd column of Tab. <ref>). 




Next, we extend the TaskShuffler algorithm for multi-rate scheduling, 

as described in Sec. <ref> to generate a set 𝒮 of valid and randomly deployable schedules using P_i's in both HU and LU scenarios.





For each victim task in Tab. <ref>, we find the number of schedules for all victim tasks with attack probability under their respective TAPs. 
The last column of Tab <ref> shows the number of such schedules. Now, using our experimental results, we discuss and show that our proposed randomized schedule generation method outperforms the SOTA in terms of inferability ratio, attack probability, and schedule vulnerability. As SOTA, we consider two cases: i) processor with static priority schedules, and ii) attack-unware randomized schedules are deployed where the set of randomized schedules is generated by TaskShuffler, only considering the minimum sampling rates of each control task.


(A) Inferability Ratio (IR): 
IR of 1,000 randomized schedules generated by our proposed MAARS framework are presented in Fig. <ref>, <ref>, <ref>, <ref> for LU and Fig. <ref>, <ref> <ref>,  <ref> for HU. The above plots depict the IRs of a certain victim-attacker combination among all possible ones that exhibit the maximum IR.


For example, the schedules of τ_1,τ_2, and τ_4 show highest inferability for the untrusted task τ_5 and schedules of τ_3 show maximum inferability for the untrusted task τ_6 in case of LU.

Fig. <ref> and Fig. <ref> (the red plots) represents the highest IR among all the victim-attacker combinations when attack-unaware randomized schedules are deployed.

We can observe that in the case of attack-unaware schedule randomization, most of the randomized schedules show higher IR values compared to the schedules generated by MAARS framework. It implies that MAARS is much better at  hiding timing information of safety-critical control tasks than SOTA.



(B) Average Attack Probability: 
For each schedule in 𝒮, we determine the attack probability of each trusted task using Def. <ref>. For each trusted task τ_i, we compute the attack probability AP_i^j considering the untrusted task τ_j. The average attack probability A̅P̅_̅i̅= ∑_j=1^MAP_⟨τ_i,s_j⟩/M is computed for each trusted task τ_i where M is the total number of schedules generated by MAARS. Fig. <ref> shows the average AP for each of the four trusted tasks in Tab. <ref> for both LU and HU. The blue, grey, and brown plots in Fig. <ref> represent the average APs in case of the schedules generated by static priority schedule, attack-unaware randomized schedule, and the MAARS framework.


We can observe in Fig. <ref>, under LU, schedules generated by MAARS (dark brown plot) show 88% and 38.5% reduction in average AP compared to the static (blue) and SOTA (light grey) respectively. Fig. <ref> shows that under HU, schedules generated by MAARS (dark brown plot) show 83.7% and 47.5% reduction in average AP compared to the static (blue) and SOTA (light grey), respectively.

[8]l0.45
    


    < g r a p h i c s >

    % of schedules under SVT: attack-unaware Randomization (grey), MAARS (brown) 
        


(C) Schedule Vulnerability Index: 
In Fig. <ref>, we plot the percentage of schedules out of all the schedules generated by MAARS (in brown) and SOTAs (in grey) that have SVIs below the schedule vulnerability threshold or SVT. The SVT value calculated with the TAP values corresponding to our control task set (given in Tab. <ref>) is 0.14 (refer to Sec. <ref>).
Our results show that in the LU scenario, 68% of MAARS schedules have SVI<SVT compared to only 31% for attack-unaware randomization. Similarly, in the HU scenario, our results show that 52% and 20% schedules have SVI below SVT for MAARS and attack-unaware scheduling, respectively. Therefore, by using MAARS, we can generate more schedules with lower vulnerability against posterior SBAs.

From the above analysis, we can state that our proposed MAARS framework reduces the inferability ratio and vulnerability index of the schedules and attack probabilities of the tasks compared to the SOTAs.


(2) Runtime Deployment in HIL: 
Since τ_2 exhibits the highest average attack probability (Fig. <ref>) and τ_8 has the highest schedule inference capability (Fig. <ref>) in HU scenario, henceforth we consider τ_2 to be victim task and τ_8 to be the attacker task.


We consider settling time (t_s) and GUES decay rate (λ) metrics to evaluate the plant's performance. To launch the attack, τ_8 executes a malicious code on the same ECU in which τ_2 is implemented.

The malicious code overrides the control input data sent to ETAS RTPC via CAN. A windowed χ^2-detector (window length=1) is implemented along with τ_2 to detect any unnatural deviation in the plant states. The detector's threshold is set to 4 for the false alarm rate (FAR) to be less than 2%. We compare the performance of the TTC system under posterior SBA in Fig. <ref> when schedules generated by MAARS are deployed and when a static fixed priority schedule is deployed.

In Fig. <ref>, the green, blue, red, and magenta plots respectively represent the control input acceleration, plant state deviation from the reference trajectory, the detector's threshold, and χ^2 statistics of the detector. Under no SBA, we note t_s= 4.3s (<desired t_s=10), and λ =95% when a fixed priority schedule is used in Fig. <ref>. However, we can observe in Fig. <ref> that the plant becomes unstable due to posterior SBA under the static fixed priority schedule.


On the other hand, Fig. <ref> shows the behavior of the TTC system under no attack when MAARS is deployed.
We can observe t_s=6s and λ=93%. Under posterior SBA, the scheduler switches from normal mode to alert mode once χ^2 residue>4 (red highlighted area in Fig. <ref>). In alert mode, we note t_s=9.2s and λ=89%. This implies that the system is resilient against posterior attack when MAARS is deployed and subsequently the performance of the system is preserved.











































§ CONCLUSION


Existing schedule randomization methodologies for secureing real-time task-schedules are attack-unaware, hence susceptible to schedule-based attacks. In this paper we present MAARS framework, a novel attack-aware schedule randomization approach that selects performance-aware multiple sampling rates such that the exposure of task parameters is reduced and the vulnerability against posterior schedule-based attacks are mitigated. Our results show that MAARS framework is effective in reducing inferability of critical trusted task parameters, while also reducing posterior attack probability. In future, we aim to extend this work for multiprocessor systems to provide security against other practical schedule-based attacks. 



ieeetr


 Paper Title: A Novel Edge Laplacian-based Approach for Adaptive Formation Control of Uncertain Multi-agent Systems with Unified Relative Error Performance.


	



	
A Novel Edge Laplacian-based Approach for Adaptive Formation Control of Uncertain Multi-agent Systems with Unified Relative Error Performance
    Kun Li, Kai Zhao, Member, IEEE, Yongduan Song, Fellow, IEEE, and Lihua Xie, Fellow, IEEE
K. Li, K. Zhao, and Y. D. Song are with School of Automation, Chongqing University, Chongqing 400044 China (e-mail: likun/zhaokai/ydsong@cqu.edu.cn).
L. H. Xie is with School of Electrical and Electronic Engineering, Nanyang Technological University, 639798 Singapore (e-mail: elhxie@ntu.edu.sg).
    August 5, 2024
=====================================================================================================================================================================================================================================================================================================================================================================================================



	
§ ABSTRACT 
	For most existing prescribed performance formation control methods, performance requirements are not directly imposed on the relative states between agents but on the consensus error, which lacks a clear physical interpretation of their solution. In this paper, we propose a novel adaptive prescribed performance formation control strategy, capable of guaranteeing prescribed performance on the relative errors, for uncertain high-order multi-agent systems under a class of directed graphs. Due to the consideration of performance constraints for relative errors, a coupled nonlinear interaction term that contains global graphic information among agents is involved in the error dynamics, leading to a fully distributed control design more difficult and challenging. Here by proposing a series of nonlinear mappings and utilizing the edge Laplacian along with Lyapunov stability theory, the presented formation control scheme exhibits the following appealing features when compared to existing results: 1) different performance requirements can be guaranteed in a unified way by solely tuning the design parameters a priori, without the need for control redesign and stability reanalysis under the proposed fixed control protocol, making the design more user-friendly and the implementation less demanding; 2) the complex and burdensome verification process for the initial constraint, often encountered in existing prescribed performance controls, is completely obviated if the performance requirements are global; and 3) nonlinear interaction is completely decoupled and the asymptotic stability of the formation manifold is ensured via using the adaptive parameter estimate technique. Finally, simulations of various performance behaviors are performed to show the efficiency of the theoretical results.

	

	Adaptive control, edge Laplacian, formation control, prescribed performance.

	
=2500



§ INTRODUCTION

Cooperative control of multi-agent systems (MASs) has attracted considerable attention in recent decades due to its broad application potential. Representative problems include consensus <cit.>, formation control <cit.>, and converge control <cit.>, to name a few. Wherein, formation control, the goal of which is to coordinate each agent using local information so that the whole team forms a prescribed spatial geometric pattern <cit.>, has been widely studied as it is applicable to various practical tasks, such as transporting loads, exploring resources, and environmental monitoring <cit.>.

In addition to the classical formation control objectives, MASs may also need to meet some additional steady and transient behaviors due to performance requirements or safety constraints in physical, e.g., completion time, maximum allowable overshoot, tolerable range of tracking accuracy, maintaining connectivity, and collision avoidance <cit.>. If we ignore such performance requirements, it may lead to the failure of cooperative formation tasks. Let us provide two specific examples to illustrate the necessity of performance and safety constraints in formation control. Consider the multi-robot hand used in neurosurgery, a large overshoot or slow response can present a risk to life. Therefore, the controller used in the robotic hand should not only have a desired steady-state response but also a desired transient response. In addition, for a platoon of vehicles to cooperatively transport a common load, the distances between any two vehicles in the team must be neither too small nor too large, as this can lead to collisions between the vehicles or a breakdown of the link. 

Current efforts on such issues are primarily based on the artificial potential function <cit.>, barrier Lyapunov function <cit.>, funnel control (FC) <cit.>, and prescribed performance bound (PPB) control <cit.>. Notably, only the FC and PPB methodologies can be used to guarantee the prescribed performance behavior of the MASs in terms of overshoot, convergence rate and steady-state error. Yet, it is essential to emphasize that, whereas such performance specifications are obtained, these FC and PPB-based works only take into account the performance of the sum of relative errors between agent i and all its neighbors (consensus errors) and not the relative errors between agent i and each neighbor, which does not reflect the inter-agent physical restrictions. Taking the cooperative formation control as an example, it is indeed more practical to specify the performance constraints on the relative position of agents, so that the interest of collision avoidance and connectivity maintenance can be tackled. On the other hand, from the user's perspective, it is more intuitive to define constraint functions based on the relative errors between agents, as the constraint requirements are normally shaped by the environmental boundaries.

In recent years, some typical results on the performance characteristics of relative errors between agents have been developed. In <cit.> and <cit.>, consensus protocols with certain prescribed performance guarantees were proposed for single- and double-integrator dynamics, respectively. In <cit.>, a decentralized formation control protocol with connectivity maintenance was presented for first-order nonlinear MASs. In <cit.>, the relative position-based formation control for leader-follower MASs with prescribed performances was studied. In <cit.>, a distributed dynamic average consensus algorithm was proposed for MASs under undirected communication topologies, so that the consensus with predefined and arbitrary convergence rate is guaranteed. Recently, a novel connectivity preservation and collision-free formation control algorithm for double-integrator dynamics was proposed in <cit.>. 

By reviewing the above performance-constrained works <cit.>, we can conclude that 1) the results only consider the single- or double-integrator dynamics under undirected graphs, which is impractical in the real applications since many engineering systems suffer from the high-order dynamics and modeling uncertainties; and 2) there exists an initial constraint on the relative errors in the algorithm implementation, which implies that the aforementioned methods only guarantee the semi-global results. To achieve the performance requirements, it is necessary to look for suitable initial values of system states and then to verify the initial constraint. However, due to the complexity and large-scale network structure of MASs, the verification process will become more and more demanding and grow explosively as the number of agents increases (the detailed discussion can be found in Section <ref>). Moreover, when the considered system is restarted or the target formation is changed, we have to check whether the original design parameters are applicable to the new initial data. If such a condition is not met, the corresponding control is no longer available, leading to the inflexibility of the control method. These limitations pose numerous obstacles to their application, which in turn motivates this work.
	
Motivated by the above observations, we will investigate the relative-error-performance formation problem for uncertain high-order MASs under directed graphs. As mentioned earlier, the difficulties in solving these problems are mainly in two aspects, 1) how to design a fully distributed formation protocol with prescribed performance guarantees for uncertain high-order MASs under directed graphs, and 2) how to ensure different performance behaviors between an agent and its neighbors within a fixed control framework according to its own environmental constraints. To address the above challenges, we present a unified prescribed performance solution for the formation control problem of uncertain high-order MASs. Compared with the existing results, the main contributions of this paper can be listed as follows:
	
		
  * Plug and Play Function: The proposed framework features that different performance requirements for the relative errors can be guaranteed by tuning the design parameters a priori and are completely decoupled from the system model, topology graph, and control gain selection. It provides a new formulation that achieves different prescribed performance for relative position errors of uncertain high-order nonlinear MASs in a unified way, without changing the control structure. Especially, the complex and explosive verification process for the initial constraints in existing prescribed performance controls is completely obviated if the performance requirement is in the global form;
		
  * User-Oriented Feature: Compared with the consensus error-based PPB schemes <cit.>, the proposed solution enables user to establish performance constraint boundaries directly based on environmental constraints, making it more favorable in practical applications. This physics-based performance control design approach provides a physical interpretation of the proposed approach; and
		
  * Broad Applicability: We propose an adaptive formation control scheme for uncertain high-order nonlinear MASs under a class of directed and undirected graphs. In this respect, existing methods in <cit.> that consider simple dynamics under undirected graphs are contained as special cases of our work.
	
	
	Notations: In this paper, ||·|| is the Euclidean norm. The sets of real, positive real, and n-dimensional Euclidean space are ℝ, ℝ^+, and ℝ^n, respectively. Let 1_n be unit column vector with n-dimensional. X^⊤ denotes the transpose of a matrix X. For X_i ∈ℝ^m × n, i=1,…,q, we denote diag[X_i]= blockdiag{X_1,…, X_q}∈ℝ^qm × qn. For a set of numbers V, |V| is its cardinality. Null(A) stands for the null space of matrix A. λ_min(A) and λ_max(A) denote the minimum and maximum eigenvalue of matrix A respectively. 
	
	

§ PRELIMINARIES

	

 §.§ Graph & Edge Laplacian

	We consider a team of N agents whose information exchange is described by a directed graph 𝒢≜(𝒱,ℰ), where 𝒱={1,...,N} is the set of nodes and ℰ⊆𝒱×𝒱 with cardinality m is the set of edges. An edge (i, j) ∈ℰ indicates that agent j can obtain information from agent i, but not necessarily vice versa. A directed path in a directed graph is a sequence of edges, for example, (i_1, i_2), (i_2, i_3), …, (i_N-1, i_N), and a directed cycle is a directed path where the initial and final nodes are the same. If (i, j) ∈ℰ implies (j, i) ∈ℰ, the graph is said to be undirected. The adjacency matrix 𝒜=[a_ij] ∈ℝ^N × N with a_ij=1, if (i,j)∈ℰ, and a_ij=0, otherwise. We denote the set of neighbors for agent i as 𝒩_i={j∈𝒱| (i,j)∈ℰ}. The degree matrix is =diag[_i]∈ℝ^N × N with _i=∑_j∈𝒩_i a_ij being the sum of the ith row in 𝒜. Then, the graph Laplacian matrix of 𝒢 is defined as L= - 𝒜. 
	
	To ensure the specified performance for relative errors in the formation control, we choose an alternative graph-theoretic tool, i.e., the edge Laplacian<cit.>. Now we recall this concept as follows.
	
	
	Label all edges of the digraph 𝒢 with e_1,e_2,…,e_m. Suppose edge (i, j) in 𝒢 corresponds to the kth directed edge, where k∈ I= {1, …, m}. The incidence matrix E(𝒢)∈ℝ^N× m for the digraph is a {0,±1}-matrix with rows and columns indexed by nodes and edges of 𝒢, respectively, such that
	
    [E(𝒢)]_ik:=
    	+1,    if i is the initial node of edge e_k,
     -1,    if i is the terminal node of edge e_k, 
     0,    otherwise.

	From the definition of the incidence matrix, it follows that the null space of its transpose, Null(E(𝒢)^⊤), contains span{1_N}.
	Moreover, for the purpose of analysis, it is necessary to partition the incidence matrix as E(𝒢)=E_⊙(𝒢)+E_⊗(𝒢), where E_⊙(𝒢)∈ℝ^N× m corresponds to the so-called in-incidence matrix, whose elements are defined as
	
    [E_⊙]_ik:=
    	-1,    if i is the terminal node of edge e_k, 
     0,    otherwise.

	and E_⊗(𝒢)∈ℝ^N× m corresponds to the so-called out-incidence matrix, whose elements are defined as
	
    [E_⊗]_ik:=
    	+1,    if i is the initial node of edge e_k,
     0,    otherwise.

	
	According to the definitions of incidence matrix and in-incidence matrix, the edge Laplacian matrix L_e∈ℝ^m× m and graph Laplacian L∈ℝ^N× N of a digraph 𝒢 can be defined in terms of the incidence and in-incidence matrices as L_e=E^⊤ E_⊙ and L=E_⊙E^⊤. For an undirected graph, L_e=E^⊤ E and L=EE^⊤, respectively. 	Moreover, using an appropriate permutation of the edge ordering, the incidence matrix is expressed as E=[E_t   E_c], where E_t∈ℝ^N×(N-1) denotes the full-column-rank incidence matrix corresponding to an arbitrary spanning tree 𝒢_t⊂𝒢 and E_c∈ℝ^N×(m-N+1) represents the incidence matrix corresponding to the remaining edges not contained in 𝒢_t. Defining R:=[I_N-1   T], where T:=(E^⊤_tE_t)^-1E^⊤_tE_c, one obtains representation of the incidence matrix of the graph, given by E=E_tR. Furthermore, inspired by <cit.>, the following facts will play a crucial role in the controller design and stability analysis presented later in this paper.
	<cit.>
	For a digraph that is a spanning tree, it holds that L^s_e=1/2(E^⊤ E_⊙+E^⊤_⊙E) is positive definite; For a digraph that is a directed cycle, it holds that E^⊤_tE_t is positive definite.
	
	
	
	

 §.§ Problem Statement

	In this paper, we consider a group of N nonlinear agents that can be described as:
	
	
    ẋ_i,q= x_i,q+1, q=1,…,n-1, 
    ẋ_i,n= u_i+φ^⊤_i(x_i)θ_i, i=1,…,N,

	where x_i,q∈ℝ and u_i∈ℝ are the system state and control input of agent i, respectively, where we only consider the one dimension case in the theoretical analysis without loss of generality. The discussions in this work can be generalized to higher dimensions cases with Kronecker product. x_i:=[x^⊤_i,1,x^⊤_i,2,⋯,x^⊤_i,n]^⊤∈ℝ^n, φ_i: ℝ^n→ℝ^ν_i × n is a known smooth nonlinear function, and θ_i∈ℝ^ν_i represents an unknown constant vector, where ν_i∈ℝ^+. 
	
	
	
	
	
	Let e_k:=x_i,1-x_j,1∈ℝ be the relative position between a pair of neighboring agents i and j for (i,j)∈ℰ, and let e^d_k∈ℝ denote the desired relative position for e_k. The  target relative position-based formation is assumed to be admissible and is described as ℱ:={e|e_k=e^d_k, k∈ℐ}, where e =E^⊤ x_1∈ℝ^m denotes the stack vector of the relative states, and x_1=[x_1,1,x_2,1,…,x_N,1]^⊤∈ℝ^N denotes the stack vector of the first-order components. Then we define the formation errors as ẽ_k=e_k-e^d_k∈ℝ for k∈ℐ. It can be checked that the target formation is achieved if e_k = 0. The control objective in this paper is formally stated as follows.
	
	
	
	
	
	
	
	
	
	
	
	
	
	Control Objective: By using the local available sensing measurements from the agent itself and its neighbor agents, we aim to design a distributed adaptive formation controller u_i for (<ref>) such that: 
	
		
  𝒪_1: All signals in the closed-loop system are bounded, and the desired formation, described by the relative offsets {e^d_k}_k∈ℐ, is achieved and maintained; and 
		
  𝒪_2: Multiple prescribed performance requirements for the relative position errors {ẽ_k}_k∈ℐ are guaranteed in a unified control framework without control redesign.
		
		
	
	
	To this end, the following assumption is imposed.
	
		The sensing graph should either be a connected undirected graph or a directed graph that is a directed spanning tree or a directed cycle.
	
	
	
	
	

§ SYSTEM AND PERFORMANCE TRANSFORMATION

	In this section, we formulate the formation performance control problem by constructing a unified performance function. Motivated in part by a previous work <cit.>, we propose a more comprehensive unified performance function to enhance the applicability of the algorithm.
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	

 §.§ Unified Performance Function

	
		Let 𝒫 : (-ι, ι) → (-∞,∞) be a unified performance function for ι∈ℝ^+, which satisfies the following conditions.
		
			
  * lim_y→ι𝒫(y) = ∞, lim_y→ -ι𝒫(y) = -∞, and 𝒫(0)=0;
			
  * The derivative of 𝒫 is lower bounded by a positive constant q and tends to infinity as y→±ι; and
			
  * 𝒫 is continuously differentiable and 𝒫(-y) = -𝒫(y).
			
		
		
	
	
	Obviously, immediate examples of such 𝒫(y) include (but are not limited to) the following functions: 𝒫(y)=y/√(1-y^2) with ι=1 and 𝒫(y)=tan(y) with ι=π/2. 
	Utilizing the unified performance function definition, the second objective 𝒪_2 can be mathematically stated as:
	
    𝒫_k(-δ_kβ_k(t))<ẽ_k(t)<𝒫_k(δ_kβ_k(t)) for k∈ I,

	where 0<δ_k, δ_k≤ 1 are user-chosen parameters. Additionally, β_k(t) is a continuously decaying performance positive function with bounded derivative and a strictly positive limit as t→∞, which is formulated as β_k:=(β_k0-β_kf)exp(-λ_k t)+β_kf
	
	
	
	with 0<β_kf<β_k0≤ι and λ_k∈ℝ^+.
	
	It can be readily seen that by choosing different design parameters δ_k, δ_k, and β_k0, such a performance objective as defined in (<ref>) covers different cases of interest, including lower/upper bounded one-sided performance constraints, asymmetric performance constraints, and global performance constraints, for details see below.
	
		
  * If δ_kβ_k0=δ_kβ_k0=ι, then (<ref>) can represent global performance constraints in the form of -∞ <ẽ_k(0)<∞; 
		
  * If δ_kβ_k0=ι and 0<δ_k<1 (resp. δ_kβ_k0=ι and 0<δ_k<1), then (<ref>) can represent lower bounded one-sided performance constraints in the form of -ϵ_k<ẽ_k(0)<∞ (resp. upper bounded one-sided performance constraints in the form of -∞<ẽ_k(0)<ϵ_k), where ϵ_k=𝒫_k(-δ_kβ_k0) and ϵ_k=𝒫(δ_kβ_k0) are some positive constants; and
		
  * If 0<δ_kβ_k0<ι and 0<δ_kβ_k0<ι, then (<ref>) can also represent asymmetric performance constraints in the form of -ϵ_k<ẽ_k(0)<ϵ_k. 
	
	
	Overall, the designer has the flexibility to choose various design parameters (i.e., δ_k, δ_k, and β_k0) based on practical requirements. It is interesting to underscore that in the case of asymmetric performance constraints, the existing relative error PPB-based results <cit.> are relevant. Their algorithms are special cases of the algorithm developed in this paper.
	
	
	
	
	
	
	
	
	
	
	
	
	

 §.§ Transformed Error Dynamic Model

	To deal with the  constraints in (<ref>), we will adopt the following nonlinear mapping:
	
    s_k   :=ζ_k(t)/(δ_k+ζ_k(t))(δ_k-ζ_k(t)), 
    ζ_k   :=η_k(t)/β_k(t),  η_k :=𝒫^-1_k(ẽ_k(t))

	with ζ_k(t) denoting the modulated error and η_k(t) being the normalized error. Furthermore, the following lemma can be easily derived from the aforementioned transformation.
	
	
		If ζ_k(0) satisfies -δ_k<ζ_k(0)<δ_k and s_k(t) is bounded for ∀ t≥0, then there exist some positive constraints δ_1k and δ_1k so that -δ_k<-δ_1k≤ζ_k(t) ≤δ_1k < δ_k. Furthermore, (<ref>) can be guaranteed.
	 
	
	Proof. The proof of this lemma is similar to <cit.>, therefore it is omitted herein. ▪
	
	It is thus obvious that if an adaptive control law is designed such that s_k(t) is bounded over t∈ [0, ∞), the target 𝒪_2 for the tracking performance is achieved according to the Lemma <ref>. 
	To facilitate the descriptions in control design and stability analysis, the unified performance function 𝒫(y) in this work is chosen as 𝒫(y)=y/√(1-y^2). Upon using the expression of s_k as shown in (<ref>), with the definitions of η_k and β_k in (<ref>), it is seen that
	
	
	
	
	
	
    ṡ_k=∂ s_k/∂ζ_kdζ_k/dt=J_k(ξ_kẽ̇_k-β̇_kζ_k),

	where J_k=μ_k/β_k>0, μ_k=δ_kδ_k+ζ^2_k/(δ_k+ζ_k)^2(δ_k-ζ_k)^2>0,  ξ_k=1/(1+ẽ^2_k)√(ẽ^2_k+1)>0
	
	
	
	are available for control design.
	
	Based on the definition of the formation error ẽ_k, the transformed error dynamics for system (<ref>) are rewritten in the following compact form
	
    Ṡ=   J( ξẽ̇+β̇ζ)=WE^⊤ x_2+D,

	where S=[s_1,…,s_m]^⊤∈ℝ^m, J=diag[J_k]∈ℝ^m× m, ξ=diag[ξ_k]∈ℝ^m× m, β̇=diag[-β̇_k], ζ=[ζ_1,…,ζ_m]^⊤, W=J ξ, and D=Jβ̇ζ. Furthermore, given that each argument of  W is positive, it follows that the diagonal matrix W is positive definite.
	
	By replacing the equation of ẋ_i,1 in (<ref>) with ṡ_k in (<ref>), the performance constraint objective is transformed to the stabilization problem of the following strict-feedback-like system:
	
    Ṡ=WE^⊤ x_2+D,   
    ẋ_q=x_q+1, q=2,…,n-1,  
    ẋ_n=u+φ^⊤θ,

	where x_q=[x_1,q, x_2,q,…, x_N,q]^⊤, u=[u_1, u_2,…, u_N]^⊤, φ=diag[φ^⊤_1, φ^⊤_2, …, φ^⊤_N], and θ=[θ_1, θ_2,…, θ_N]^⊤.
	




	
	
	
	

§ CONTROLLER DESIGN AND STABILITY ANALYSIS

	
	

 §.§ Controller Design

	In this subsection, based on the adaptive backstepping technique <cit.> we propose a novel adaptive formation design approach by utilizing the edge Laplacian. Firstly, we introduce the following change of coordinates:
	
    z_1=     S, 
    
    	z_q=     x_q- α_q-1, q=2,…,n,

	where z_1=[s_1, …, s_m]^⊤ and z_q=[z_1,q, …,z_N,q]^⊤ are intermediate variables, and α_q-1=[α_1,q-1, …, α_N,q-1]^⊤ is the virtual controller, respectively. The design procedure contains n steps. 
	
	 Step 1: From (<ref>), (<ref>) and (<ref>), the derivative of z_1 is
	
    ż_1= WE^⊤(z_2+α_1)+D,

	then the derivative of V_1=1/2z^⊤_1z_1 along  (<ref>) yields,
	
    V̇_1=z^⊤_1[WE^⊤(z_2+α_1)+D].

	
	
		
		We pause here to emphasize that, due to the consideration of relative error performance behaviors, the coupled nonlinear interaction term WE^⊤ and the additional coupling term Jβ̇ζ have been incorporated into the derivative of V_1 as presented in (<ref>). This introduces an obstacle in designing the virtual controller α_1 based on local information to ensure the controllability of the subsystem dynamics (<ref>). The main challenges arise from the following three aspects: 
		
			
  * The transformed error dynamics, as described in (<ref>), are significantly different from the conventional PPC control methodologies<cit.>. This distinction stems from the inclusion of a nonlinear term ξ_k, which is coupled with the formation errors, leading to difficulties in ensuring the controllability of subsystems (<ref>);
			
  * The term WE^⊤ represents a non-square matrix that contains global topological information, making the direct use of its inverse impractical for controller design; and
			
  * The third challenge arises from the fact that in the setup for directed graphs, the Laplacian matrix L is non-symmetric and positive semi-definite. Thus, it is challenging to ensure the controllability of the subsystem dynamics (<ref>) using Laplacian matrix-based design methods.	
		
		
	
	
	To handle the above challenges, we perform the controller design of α_1 using the edge Laplacian technique. The virtual controller α_1 is designed as
	
    α_1=-c_1E_⊙ W z_1,   c_1>0.

	
	By substituting (<ref>) into (<ref>), the derivative of V_1 is given by
	
    V̇_1=-c_1z^⊤_1WE^⊤ E_⊙ W z_1+z^⊤_1WE^⊤ z_2+z^⊤_1D.

	
	
		We highlight that each component of α_1 depends only on local information since E_⊙ represents the incoming edges on each node, i.e., the information available to each agent as defined by the directed graph. Moreover, equation (<ref>) is applicable to both directed spanning-tree, directed-cycle and undirected graph topologies. In the following, we proceed by separately examining each of these topologies.
	
	
	
	Case 1 (Directed spanning tree): In this case, we have 𝒢_t=𝒢. Therefore, E_t=E, R=I_N-1, and E_⊙ t=E_⊙, where E_⊙ t is the in-incidence matrix of the spanning tree. It is not difficult to obtain z_1t=z_1 since R=I_N-1. Then we have
	
    V̇_1   =-c_1z^⊤_1tWE^⊤_t E_⊙ t W^⊤ z_1t+z^⊤_1tWE^⊤_t z_2+z^⊤_1tD
       =-c_1z^⊤_1tWL^s_e W^⊤ z_1t+z^⊤_1tWE^⊤_t z_2+z^⊤_1tD
       ≤ -c_1λ_min(L^s_e)‖ W z_1t‖^2+z^⊤_1tWE^⊤_t z_2+z^⊤_1tD,

	where the third inequality follows from the fact that L^s_e is positive definite according to Lemma <ref>. 
	
	Then after using D=Jβ̇ζ in (<ref>) and applying Young’s inequality to last two term in the right-hand side of (<ref>), we obtain
	
    z^⊤_1t WE^⊤_t z_2    ≤ϑ/2λ_max(E^⊤_t E_t)‖ Wz_1t‖^2+1/2ϑz_2^2,  
    
    	z^⊤_1t J β̇ζ   ≤ϵ/2‖ Jz_1t‖^2 +1/2ϵζ^⊤β̇β̇ζ,

	where ϑ and ϵ are positive constants.
	
	Now, given c_1>0, let ϑ>0 be such that c'_1:=[c_1λ_min(L^s_e)-1/2ϑλ_max(E^⊤_t E_t)]λ_min(W^2) and c'_2=c_2-1/2ϑ with c_2>0 are positive, we bound the (<ref>) as follows:
	
    V̇_1   ≤ -c'_1‖ z_1t‖^2+1/2ϑz_2^2+ϵ/2z^⊤_1tJ J z_1t+1/2ϵζ^⊤β̇β̇ζ
       ≤  -c”_1‖ z_1t‖^2+1/2ϑz_2^2+1/2ϵζ^⊤β̇β̇ζ,

	where c”_1:=c'_1-ϵ/2λ_max(J^2) is positive for a sufficiently small ϵ>0.
	
	Case 2 (Directed cycle): In this case, we have E^⊤ E_⊙+E^⊤_⊙E=E^⊤ E. By using this fact, we obtain 
	
    V̇_1   =-1/2c_1z^⊤_1WE^⊤ EW z_1+z^⊤_1WE^⊤ z_2+z^⊤_1D
       =-1/2c_1z^⊤_1tRWR^⊤ E^⊤_t E_t R W R^⊤ z_1t+z^⊤_1W E^⊤ z_2+z^⊤_1D
       ≤ -1/2c_1λ_min(E^⊤_tE_t)‖ RW R^⊤  z_1t‖^2+z^⊤_1tRWR^⊤ E^⊤_t z_2
          +z^⊤_1Jβ̇ζ,

	where the third inequality follows from the fact that E^⊤_t E_t is positive definite according to Lemma <ref>. 
	
	Now, given c_1>0, let ϑ>0 be such that c'_1:=1/2[c_1λ_min(E^⊤_tE_t)-ϑλ_max(E^⊤_t E_t)]λ_min(RWR^⊤ RWR^⊤ ) and c'_2=c_2-1/2ϑ with c_2>0 are positive. Note that RW R^⊤ RW R^⊤ and RJ^2R^⊤ are positive definite due to R is full row-rank and W and J are positive definite. Then, similar to the case 1, by applying Young’s inequality to last two term in the right-hand side of (<ref>), we obtain
	
    V̇_1   ≤ -c'_1‖ z_1t‖^2+1/2ϑz_2^2+ϵ/2z^⊤_1tRJ JR^⊤ z_1t+1/2ϵζ^⊤β̇β̇ζ
       ≤  -c”_1‖ z_1t‖^2+1/2ϑz_2^2+1/2ϵζ^⊤β̇β̇ζ,

	where c”_1:=c'_1-ϵ/2λ_max(RJ^2R^⊤) is positive for a sufficiently small ϵ>0.

	Case 3 (Undirected graph): In this setup, each agent can not only receive information from its neighbors but also broadcast its own information. Therefore, the virtual controller can be designed as follows:
	
    α_1=-c_1E W z_1,

	where E represents the incidence matrix of the oriented undirected graph. 
	
    By substituting (<ref>) into (<ref>), the derivative of V_1 is given by
	
    V̇_1   =-c_1z^⊤_1WE^⊤ EW z_1+z^⊤_1WE^⊤ z_2+z^⊤_1D
       =-c_1z^⊤_1tRWR^⊤ E^⊤_t E_t R W R^⊤ z_1t+z^⊤_1W E^⊤ z_2+z^⊤_1D
       ≤ -c_1λ_min(E^⊤_tE_t)‖ RW R^⊤  z_1t‖^2+z^⊤_1tRWR^⊤ E^⊤_t z_2
          +z^⊤_1Jβ̇ζ,

	where E^⊤_tE_t represents the edge Laplacian of a spanning tree 𝒢_t⊂𝒢 and is symmetric positive definite. 




    
    It is seen from (<ref>) that V̇_1 has a similar expression. Therefore, by employing the same analysis as in Case 2, we can easily obtain:
	
    V̇_1    ≤  -c”_1‖ z_1t‖^2+1/2ϑz_2^2+1/2ϵζ^⊤β̇β̇ζ,

	where c”_1:=c'_1-ϵ/2λ_max(RJ^2R^⊤)>0 and c'_1:=[c_1λ_min(E^⊤_tE_t)-1/2ϑλ_max(E^⊤_t E_t)]λ_min(RWR^⊤ RWR^⊤ )>0.
	
	In summary, we see from (<ref>), (<ref>), and (<ref>) that the expression of V̇_1 is same for both the considered directed graph and undirected graph. With this in mind, we proceed with the controller design.

	 Step 2:  
	
	Using (<ref>), the derivative of z_2 is computed as
	
    ż_2=ẋ_2-α̇_1=z_3+α_2-α̇_1.

	
	From (<ref>) and (<ref>), the derivative of V_2=V_1+1/2z^⊤_2z_2 is
	
    V̇_2 ≤    -c”_1‖ z_1t‖^2+1/2ϑz_2^2+1/2ϵζ^⊤β̇β̇ζ
       +z^⊤_2 (z_3+α_2-α̇_1).

	
	
	Therefore, we design α_2 as
	
    α_2=-c_2z_2+α̇_1,   c_2>0.

	
	By replacing α_2 from (<ref>) in (<ref>), one has
	
	
    V̇_2 ≤ -c”_1‖ z_1t‖^2-c'_2‖ z_2‖^2+z^⊤_2 z_3+1/2ϵζ^⊤β̇β̇ζ.

	 Step q (q=3, …, n-1): Define the quadratic function V_q as V_q=V_q-1+1/2z^⊤_qz_q, then its derivative is
	
    V̇_q ≤     -c”_1‖ z_1t‖^2-c'_2‖ z_2‖^2-∑_j=3^q-1c_jz^⊤_jz_j +z^⊤_q-1 z_q
        +1/2ϵζ^⊤β̇β̇ζ +z^⊤_q (z_q+1+α_q-α̇_q-1).

	Following the ideas in the above steps, we design α_q as
	
    α_q=-c_qz_q-z_q-1+α̇_q-1,  c_q>0.

	Substituting the virtual control α_q into (<ref>), one has
	
	
	
	
	
	
	
    V̇_q  ≤    -c”_1‖ z_1t‖^2-c'_2‖ z_2‖^2-∑_j=3^q-1c_jz^⊤_jz_j 
         +z^⊤_q z_q+1 +1/2ϵζ^⊤β̇β̇ζ.

	
	 Step n: The derivative of z_n is computed as
	
    ż_n=ẋ_n-α̇_n-1=u+φ^⊤θ-α̇_n-1.



	Define the Lyapunov function candidate V as
	
    V=V_n-1+1/2z^⊤_nz_n+1/2θ̃^⊤Γ^-1θ̃,

	where θ̃=θ-θ̂ is the estimate error of θ with θ̂ being the estimate of θ, and Γ=diag[Γ_i]∈ℝ^N× N is a positive definite matrix with Γ_i>0. Then the derivative of V is
	
    V̇≤     -c”_1‖ z_1t‖^2-c'_2‖ z_2‖^2-∑_j=3^n-1c_jz^⊤_jz_j +z^⊤_n-1 z_n   
       +1/2ϵζ^⊤β̇β̇ζ +z^⊤_n (u+φ^⊤θ-α̇_n-1)-θ̃^⊤Γ^-1θ̇̂̇.

	The actual controller and the adaptive law are designed as:
	
    u   =-c_nz_n-z_n-1+α̇_n-1-φ^⊤θ̂,   c_n>0,
    θ̇̂̇   =Γφ z_n.

	By applying the actual control and adaptive law into (<ref>), we have
	
	
	
	
	
	
	
	
	
    V̇≤     -c”_1‖ z_1t‖^2-c'_2‖ z_2‖^2-∑_j=3^n-1c_jz^⊤_jz_j+1/2ϵζ^⊤β̇β̇ζ.





	
	
	
	
	
	
	
	

 §.§ Stability Analysis

	Here we present our main result in the
	following theorem.
	
		Consider the uncertain nonlinear MASs described by (<ref>) under Assumption <ref>. If the developed control (<ref>) and adaptive law (<ref>) are employed, then target formation can be achieved, ensuring the satisfaction of multiple performance requirements, as well as the boundedness of all closed-loop signals at all time.
	
	
	Proof. 
	
	We carry out the proof of Theorem <ref> by three phases. Firstly, we address the existence and uniqueness of a maximal solution ζ_k(t) over the open set Ω_ζ_k:= {ζ_k(t): ζ_k(t)∈ (-δ_k, δ_k) } for a time interval [0, τ_max) with τ_max being a positive constant. Secondly, we prove that all internal signals are bounded and ζ_k(t) remains strictly within a subset of Ω_ζ_k for t∈ [0, τ_max) under the control law (<ref>) and the adaptive control law (<ref>). Finally, we show that τ_max can be extended to ∞ and subsequently ζ_k(t) ∈ (-δ_k, δ_k) is ensured over time, thus completing the proof.
	
	
	
	Phase I. 
	
	
	
	
	Note that the modulated error ζ_k(t) is given by ζ_k=η_k/β_k. Differentiating ζ_k with respect to time yields:
	
    ζ̇=h_0(t,ζ)=β(ξ E^⊤(z_2-c_1E_⊙ W z_1)+β̇ζ),

	where β=diag[1/β_1, …, 1/β_N]. Differentiating z_q and θ̃, employing (<ref>), (<ref>), (<ref>) as well as the fact that z_q= x_q- α_q-1, and invoking (<ref>), (<ref>), we arrive at
	
    ż_1=h_1(t,z_2,ζ)= WE^⊤(z_2-c_1E_⊙ W z_1)+D,  
    ż_q=h_q(t,z_q+1)=z_q+1-c_qz_q, q=2,…,n-1,  
    ż_n=h_n(t,z_n,θ̃)=-c_nz_n-z_n-1+φ^⊤θ̃,  
    θ̇̃̇=h_n+1(t,z_n)=-Γφ z_n.

	
	Thus, the closed-loop dynamic system of Z(t)=[ζ(t), z_1(t), …, 
z_n(t),θ̃(t)]^⊤ may be written in compact form as:
	
    Ż=h(t,Z)=[h^⊤_0(t,ζ), h^⊤_1(t,z_2,ζ), ⋯, h^⊤_n+1(t,z_n)]^⊤.

	
	Let us also define the open set Ω_Z=Ω_ζ×ℝ^n+1, where Ω_ζ=Ω_ζ_1×Ω_ζ_2×…×Ω_ζ_m. Note that ζ_k(0)∈ (δ_k, δ_k), one observes Z(0)=[ζ(0), z_1(0), …,z_n(0),θ̃(0)]^⊤∈Ω_Z. Since h(t,Z) is continuous in t and locally Lipschitz in Z over the set Ω_Z, the hypotheses of Theorem 54 in <cit.> (see p. 476) are satisfied. Consequently, the Ż-system has a unique and maximal solution over the time interval [0, τ_max), ensuing that ζ_k(t)∈Ω_ζ_k for all t ∈ [0, τ_max), i.e., 
	
    ζ_k(t)∈ (-δ_k, δ_k),  k∈ I,  ∀ t ∈ [0, τ_max).

	
	Phase II. 
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	According to (<ref>), one has
	
    ζ^2<∑_k=1^mmax{δ^2_k,δ^2_k}=ρ,

	where ρ is a positive constant. Then it follows from (<ref>) that
	
    V̇≤  -c”_1‖ z_1t‖^2-c'_2‖ z_2‖^2-∑_j=3^n-1c_jz^⊤_jz_j+ρ/2ϵ^2λ_max(β̇ ^2)

	for ∀ t ∈ [0, τ_max). Integrating both sides of (<ref>) in the interval [0, τ_max) yields that
	
    V(t)+    ∫_0^τ_max(c”_1z^⊤_1tz_1t+c'_2z^⊤_2z_	2+∑_j=3^nc_jz^⊤_jz_j)dt 
       ≤ V(0)+ρ/2ϵ^2∫_0^τ_maxλ_max(β̇ ^2)dt.

	
	Given that β̇_k is smooth and lim_t→∞β̇_k=0, it follows that lim_t→∞λ_max(β̇ ^2)=0, implying that ∫_0^τ_maxλ_max(β̇ ^2)dt is bounded. Based on the definition of V in (<ref>), we establish that z_1t and z_q (q=2,…,n) and θ̃ are bounded and integrable for all t ∈ [0, τ_max). Consequently, the parameter estimate θ̂ remains bounded over the interval [0, τ_max). Furthermore, the boundedness of z_q and ẽ ensures that x_i,q is also bounded, leading to the boundedness of  φ_i(x_i) and α̇_q (q=1,…,n-1) within the same interval. As β_k and p^* are continuously differentiable, ż_q (q=1,…,n) is also bounded. Then from (<ref>) and (<ref>), we conclude that u and θ̇̂̇ are bounded for all t ∈ [0, τ_max). Therefore, all signals in the closed-loop systems are bounded over the interval [0, τ_max).
	
	Additionally, due to the boundedness of s_k, according to the Lemma <ref>, we have
	
    -δ_k<-δ_1k≤ζ_k(t) ≤δ_1k < δ_k, k∈ I, t∈ [0, τ_max).

	
	Phase III. Note that from (<ref>), ζ_k(t) ∈Ω'_ζ_k:= {ζ_k(t)∈ [δ_1k, δ_1k]} for ∀ t∈ [0, τ_max) is a nonempty compact subset of Ω_ζ_k. Hence, assuming τ_max≤∞ and since Ω'_ζ_k⊂Ω_ζ_k, Proposition C.3.6 in <cit.> (see p. 481) dictates the existence of a time instant t' ∈ [0, τ_max) such that ζ_k(t') ∉Ω'_ζ_k, which is a clear contradiction. Therefore, τ_max=∞. Thus, all closed loop signals remain bounded and moreover ζ_k∈Ω'_ζ_k⊂Ω_ζ_k, ∀ t>0. Based on Lemma <ref>, (<ref>) is guaranteed for ∀  t≥0. In addition, by applying the Barbalat's lemma, we have lim_t→∞z_1t=0. Note that z_1t=0 implies S=0 since R is full rank, which further implies ζ=0 and η=0 according to the nonlinear mapping as shown in (<ref>) and (<ref>), and hence ẽ_k=0 for k∈ I. The proof is completed. ▪
	
	
	
	
	
	
		It is worth noting that although most existing works (e.g., <cit.>) can guarantee the asymmetric prescribed performance for the virtual consensus error, it remains unclear that whether the relative error satisfies the performance constraint. Moreover, if the users would like to achieve other different performances, one has to redesign the controller and reanalyze the stability of the closed-loop system, making the control implementation more complex and less flexible. Some efforts (e.g., <cit.>) have been developed to achieve the prescribed performance for the relative errors, but it still only guarantees one special constraint form.
		
		To solve the above issue, in this paper, we introduce a unified performance function (as defined in Definition <ref>) and a series of nonlinear mappings (as shown in (<ref>) and (<ref>)), so that the prescribed performance problem of relative errors is converted into the stabilization problem of a new strict-feedback-like system. Under such a framework, multiple kinds of prescribed performance behaviors can be guaranteed only by solely tuning the design parameters a priori, without the need for control redesign and stability reanalysis, making the design more user-friendly and the implementation less demanding. Especially for the global performance requirement case, the tedious and explosive verification process for the initial constraint in the existing methods is completely eliminated.
	
	
	
		In this work, due to the considerations of multiple performance constraints for relative errors and the digraph setup, a coupled nonlinear interaction term WE^⊤ that contains global graphic information among agents and an additional coupling term Jβ̇ζ are inevitably embedded into the error dynamics (<ref>), therefore, a mismatched term appearing in (<ref>) may exist in each step of the backstepping design procedure, which poses an additional challenge for the stability analysis and control design. To overcome these difficulties, inspired by <cit.>, we propose an edge Laplacian-based adaptive control method, which exhibits the following appealing characteristics. Firstly, by skillful designing the first virtual controller α_1, the nonlinear interconnections term WE^⊤ can be treated as a quadratic function W^⊤ L_eW as shown in (<ref>), then with the aid of the reduced-order representation and the eigenvalue analysis, we utilize the adaptive parameter estimate technique to establish the asymptotic stability of the formation manifold. Secondly, here we need to emphasize that multiple performance behaviors can be  achieved without changing the control structure because the performance index has been completely decoupled from the structure of  the dynamic systems and the underlying graphs. Furthermore, to handle the negative effects caused by the mismatched terms z^⊤_1 WE^⊤ z_2 and z^⊤_1 J β̇ζ, the robust technique is used for further analysis as shown in (<ref>) and (<ref>). As such, the stability of the closed-loop system can be ensured via utilizing the Lyapunov stability. 
		
		
		
		
		
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	

§ SIMULATION RESULTS

	
	To illustrate the effectiveness of the proposed formation controller, we consider a group of 5 mobile robots described by <cit.> interconnected over the directed spanning-tree/directed cycle/undirected graph showed in Fig. <ref>, moving in the plane (i.e., d=2) with the following dynamics:
	
    ẋ_i(t)=v_i(t), i=1,…,5, 
    v̇_i(t)=u_i(t)-φ^⊤_i(x_i,v_i)θ_i,

	
	where x_i=[x_i,1,x_i,2]^⊤∈ℝ^2 and v_i=[v_i,1,v_i,2]^⊤∈ℝ^2 are, respectively, the position and velocity of the mobile robots, φ^⊤_i(x_i,v_i)θ_i is an unknown friction satisfying φ_i=diag[tanh(10v_i,1)-tanh(100v_i,1), tanh(10v_i,2)-tanh(100v_i,2)] and θ_i=[0.25,0.25]^⊤.
	
	
	
	
	
	The desired formation corresponds to a pentagon described by p^d_i(t)=[cos(2(i-1)π/5), sin(2(i-1)π/5)]^⊤ for i=1,…,5, and the desired relative position vector is denoted as e^d_k=[e^d_kx, e^d_ky]^⊤ for k=1,…,5. For Fig. <ref>(a) (directed spanning tree case), e^d_k is set to e^d_1=p^d_2-p^d_1, e^d_2=p^d_3-p^d_2, e^d_3=p^d_4-p^d_3, and e^d_4=p^d_5-p^d_4. For Fig. <ref>(b) (directed cycle case),  e^d_k is set to e^d_1=p^d_2-p^d_1, e^d_2=p^d_3-p^d_2, e^d_3=p^d_4-p^d_3, e^d_4=p^d_5-p^d_4, and e^d_5=p^d_5-p^d_1; For Fig. <ref>(c) (undirected graph case),  e^d_k is set to e^d_1=p^d_2-p^d_1, e^d_2=p^d_3-p^d_2, e^d_3=p^d_4-p^d_3, e^d_4=p^d_5-p^d_4, and e^d_5=p^d_1-p^d_5. In the following, to show the validity and the advantages of the proposed algorithm, we will verify the asymmetric and global performance behaviors just by choosing different design parameters under a fixed controller. To simplify notations, we choose the identical performance function for each edge, implying that each agent has the same communication/sensing capabilities, namely, 
	
    𝒫_k(δ_kβ_k(t))=𝒫(δβ(t)) and 𝒫_k(-δ_kβ_k(t))=𝒫(-δβ(t))

	for k=1,…,5, where β(t)=(β_0-β_f)exp(-λ t)+β_f.
	
	
	
	
	

 §.§ Asymmetric Performance Behavior

	To verify that the proposed control is able to achieve the asymmetric prescribed tracking performance. In this part, for the directed spanning tree case, the agents' initial positions are set as x_1,1(0)=[1,1.2]^⊤, x_2,1(0)=[0.6, 2.3]^⊤, x_3,1(0)=[-0.8, 2.1]^⊤, x_4,1(0)=[-1.4, 0.5]^⊤, and x_5,1(0)=[-0.2,0.35]^⊤. For the directed cycle and undirected graph case, the agents' initial positions are set as x_1,1(0)=[1.2,2]^⊤, x_2,1(0)=[0.5, 3]^⊤, x_3,1(0)=[-0.8, 2.5]^⊤, x_4,1(0)=[-0.9, 1.5]^⊤, and x_5,1(0)=[0.4,1]^⊤; the initial velocities for each agent are set as v_i(0)=zeros(2,1). To fulfill the initial constraint, the key design parameters δ, δ and β_0 are selected as δ=0.3 and δ=β_0=0.8 such that each initial relative error falls within the initial prescribed boundary, i.e., -0.25=𝒫(-δβ_0)<ẽ_kq(0)<𝒫(δβ_0)=0.83, q∈{x,y} and k=1,…,5. The other parameters are chosen as: λ=1, c_1=1, c_2=5, Γ_i=I_2, and β_f=0.1. By implementing adaptive control law (<ref>) and (<ref>), the simulation results are shown in Fig. <ref>. From Fig. <ref>(top) it is observed that the five agents eventually achieve and maintain the target formation shape during their motions. The responses of edge errors are depicted in the middle and bottom of Fig. <ref>, respectively, from which it can be seen that in all cases—(a) directed spanning tree, (b) directed cycle, and (c) undirected graph—the edge errors asymptotically converge to zero within the prescribed performance bounds over time.
	
	It should be noted that in order to guarantee the asymmetric performance behavior of the relative errors, it is crucial to carefully choose the key design parameters δ, δ and β_0 to meet the initial constraints (i.e., 𝒫(-δβ_0)<ẽ_kq(0)<𝒫(δβ_0)). 
	This situation also exists in <cit.>. However, this parameter selection process is exceedingly tedious and time-consuming, for example, in an N-agent MAS with m edges, it necessitates the design and execution of a performance boundary selection algorithm at least m times to identify feasible parameters. To circumvent such challenges, this work proposes a solution by considering global performance characteristics, as discussed below.
	
	
	

 §.§ Global Performance Behavior

	
	In this part, the agents' initial positions are set as x_1,1(0)=[1.6,-0.2]^⊤, x_2,1(0)=[1.2, -0.1]^⊤, x_3,1(0)=[1.75, 0]^⊤, x_4,1(0)=[1.4, 0.15]^⊤, and  x_5,1(0)=[1.5,-0.1]^⊤, and the initial velocities are also set as v_i(0)=zeros(2,1). The key design parameters δ, δ, and β_0 are selected as δ=δ=β_0=1, i.e., -∞=𝒫(-δβ_0)<ẽ_kq(0)<𝒫(δβ_0)=∞, q∈{x,y}, which shows that there is no any constraint on the initial states. The other parameters are chosen as: λ=1, c_1=2, c_2=5, Γ_i=I_2, and β_f=0.1. By implementing adaptive control law (<ref>) and (<ref>), the simulation results are shown in Fig. <ref>. From Fig. <ref>(top) it can be seen that the five agents eventually achieve and maintain the target formation shape during their motions. Figs. <ref> middle and bottom show that the edge errors asymptotically converge to zero within the prescribed boundary, which indicates that the performance behaviors are independent of initial conditions, illustrating the effectiveness of the proposed approach.
	
	
	

§ CONCLUSION

	In this paper, a novel adaptive formation control algorithm has been proposed for a group of agents described by uncertain high-order nonlinear dynamic models. The features that distinguish our method from the existing ones include: 1) the relative error constraints between the agents with physical meaning are taken into account, which makes the developed algorithm advantageous for practical applications;  2) it is able to fulfill various performance requirements merely by selecting design parameters a priori, making the control redesign and stability reanalysis not required; and 3) the dynamic model and underlying graph considered in this work are more general such that most of existing results are the special cases of the proposed control. The simulation results have shown the efficacy of our approach. Future research may consider multiple time-varying performance constraints and derive a formation control algorithm in the local reference frame.




IEEEtran






























