Paper Title: Analyzing Customer-Facing Vendor Experiences with Time Series Forecasting and Monte Carlo Techniques.


    


    











    








eBay]Vivek Kaushik
vikaushik@ebay.com
eBay]Jason Tang cor
jastang@ebay.com

[eBay]eBay Inc., 2025 Hamilton Ave., San Jose, CA 95125
[cor]Corresponding author
[t1] This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors






§ ABSTRACT 
eBay partners with external vendors, which allows customers to freely select a vendor to complete their eBay experiences. However, vendor outages can hinder customer experiences. Consequently, eBay can disable a problematic vendor to prevent customer loss. Disabling the vendor too late risks losing customers willing to switch to other vendors, while disabling it too early risks losing those unwilling to switch. In this paper, we propose a data-driven solution to answer whether eBay should disable a problematic vendor and when to disable it. Our solution involves forecasting customer behavior. First, we use a multiplicative seasonality model to represent behavior if all vendors are fully functioning. Next, we use a Monte Carlo simulation to represent behavior if the problematic vendor remains enabled. Finally, we use a linear model to represent behavior if the vendor is disabled. By comparing these forecasts, we determine the optimal time for eBay to disable the problematic vendor.



    Seasonality Exponential smoothing Regression Combining forecasts Business cycles


    










§ INTRODUCTION


In today’s landscape of rapid innovation, applications often use the services of third-party vendors to enable additional functionality quickly and easily. eBay is no exception. As an eCommerce company, eBay partners with many external vendors to serve various business functions. In some cases, multiple vendors support the same business function in order to provide the richest experience for all customers. 

As with any complex system, however, vendors are subject to outages during which they cannot operate normally. These outages could impact customers' ability to properly complete their experiences on eBay. Ideally, if a specific vendor is experiencing minimal impact during an outage, a customer using that vendor could have an initial failed experience with it, but should eventually have a successful experience upon retrying. 

However, if the vendor is experiencing a high degree of impact, then customer retries may not be successful. Customers with unsuccessful retries may choose to switch to a different vendor to complete their experiences. If the vendor's impact is severe enough, then eBay can temporarily disable, or wire off, the vendor, meaning customers would no longer see that vendor as an option. Consequently, more customers will choose to complete their experiences with a different vendor. At the same time, historical data has shown many customers have a very strong preference for using a particular vendor and will choose not to complete their experience if it is not shown as an option. We capture these effects in Fig. <ref>.




eBay's operations team makes the decision to wire off a problematic vendor. During past incidents, however, these decisions have not necessarily been data-driven. The decision to wire off a vendor is a critical one for the business; when it is executed, it typically is when the problematic vendor is experiencing a full outage, which means no customer requests with that vendor can successfully be served. During this type of situation, customers who prefer to only use the problematic vendor will not complete their experiences at all. On the other hand, customers who could otherwise use a different vendor are stuck retrying with the problematic vendor as long as it is being displayed to them. Over time, this results in net fewer successful customer experiences than if eBay had wired off the problematic vendor earlier during the incident.

In this paper, we present a data-driven approach to determine whether a given problematic vendor should be disabled and provide an optimal time for the wire-off. This optimal disabling time maximizes the total number of successfully completed customer experiences across all vendors. 

First, as an intermediary step, we forecast each vendor's baseline customer behavior. More specifically, we forecast the total number of successful customer experiences for each vendor assuming all vendors are fully functioning. Historical data has shown baseline customer experience exhibits weekly seasonal behavior. Moreover, the seasonality varies with a trend. Therefore, for each vendor, we use a multiplicative seasonality model to represent its baseline customer behavior. We demonstrate how to forecast the seasonal and trend components associated with the baseline model, which in turn allows us to forecast each vendor's baseline customer behavior.

Next, we forecast the wired-on customer behavior. More specifically, we forecast the total number of successful customer experiences across all vendors assuming the given problematic vendor remains enabled. In order to accomplish this, we first forecast the problematic vendor's availability, which we interpret as the probability that a customer would have an initial successful experience with the problematic vendor. Historical incidents have shown that the problematic vendor's availability exhibits a decreasing trend over time; therefore, we use a Double Exponential Smoothing model to represent its availability. We describe the procedure to compute the smoothing and trend components associated with this model from historical data, which in turn allows us to forecast the problematic vendor's availability. Then, we consider several probability distributions that are constructed from historical incident data during which the problematic vendor remained enabled. These distributions model the following customer decisions: retrying the experience after an initial unsuccessful attempt with the problematic vendor, switching to another vendor, and the time between retries. We incorporate the baseline customer experience forecasts from the previous paragraph, the problematic vendor's availability forecast, and the aforementioned customer decision distributions into a Monte Carlo simulation. We run the Monte Carlo simulation to project future decisions made by customers pertaining to their experiences. Finally, we aggregate these simulation results together to forecast the wired-on customer behavior.

Lastly, we forecast the wired-off customer behavior. More specifically, we forecast the total number of successful customer experiences across all enabled vendors assuming the given problematic vendor is disabled. Historical incidents have shown that customers who migrate from using the problematic vendor to using any other vendor typically do so at a constant rate over time. Therefore, we use a simple linear model to represent the wired-off customer experience. The independent variable in the model is the disabled vendor's predicted baseline customer experiences as described above, while the dependent variable is the difference between the total number of successful customer experiences during the wire-off and total number of predicted baseline customer experiences across all the enabled vendors. Forecasting the wired-off customer behavior amounts to determining the slope of the regression line.


Finally, in order to arrive at a data-driven decision for wiring off the given problematic vendor, we compare the wired-on and wired-off customer behavior forecasts from the previous two paragraphs. In particular, we check whether there is a future time at which the total number of predicted wired-off customer experiences begins to exceed the total number of predicted wired-on customer experiences. If such a time exists, then we choose to disable the problematic vendor at that particular time. Otherwise, we keep the problematic vendor enabled.









§ CUSTOMER BEHAVIOR MODELS
 

Customer behavior is the underlying factor driving eBay's business metrics. In order to determine the optimal wire-off time for a given problematic vendor, we will need to consider models that represent distinct types of customer behavior. We use these models to forecast each vendor's volume, which we interpret as the total number of customer experiences with that vendor. Throughout this section, we assume there are N total vendors for some fixed positive integer N.



 §.§ Baseline Model
 

Consider the baseline situation in which all N vendors are fully functioning, so that any customer is guaranteed to have an initial successful experience with their choice of vendor. We discuss how to forecast each vendor's baseline volume. For each 1 ≤ n ≤ N and each t ≥ 0, we let C_n,t be the n-th vendor's baseline volume at the time t. Vendor volume is typically observed every minute, so we assume all times have granularity in minutes. In order to clearly distinguish between past, present, and future times, we will index time in the following way: we fix a t_0 to be the current time, and for each m ∈, we let t_m be the time corresponding to m minutes since t_0. For any m<0, we interpret t_m as a historical time and for any m>0, we interpret t_m as a future time. We assume that volume for current and historical times is known, while volume for future times is unknown. We also assume that volume is positive for all times. 

Historical evidence has shown each vendor's baseline volume exhibits weekly seasonality along with trend. Moreover, due to varying customer activity levels throughout time, the seasonality directly varies with the trend (see Fig. <ref>). Therefore, we use a multiplicative seasonality model <cit.> to represent each vendor's baseline volume. Equivalently, we use an additive seasonality model to represent the logarithm of each vendor's baseline volume. Explicitly, for the n-th vendor, we consider the model: 

    
    ln(C_n,t_m) = ln( s_n,t_m ) + ln(g_n,t_m) +ϵ_n,t_m,


where s_n,t_m and g_n,t_m are the seasonal and trend components at time t_m, respectively, and {ϵ_n,t_m}_m ∈ is an independent and identically distributed sequence of Gaussian random variables with mean 0.



  §.§.§ Seasonal Component

We discuss how to forecast the n-th vendor's seasonal component in Eq. (<ref>). To do this, we consider the following model for the logarithmic seasonal component:

    
    ln(s_n,t_m) = ∑_h=1^H_nα_n,hcos(2 π   h t_m/L) + β_n,hsin(2 π    h t_m/L) + ζ_n,t_m


for each m ∈, in which 
    
        
  * H_n is the known number of harmonics;
        
  * L=10080 is the total number of minutes in a week;
        
  * α_n,h and β_n,h are the unknown Fourier coefficients for each 1 ≤ h ≤ H_n;
        
  * {ζ_n,t_m}_m ∈ is an independent sequence of Gaussian random variables with mean 0 and common variance σ_n^2 for some constant σ_n>0.
    
More compactly, for each m ∈, we can rewrite Eq. (<ref>) as 

    
    ln(s_n,t_m) = 𝐱_n,t_m^⊤ β_n  + ζ_n,t_m,


where
    
    𝐱_n,t_m   = ( cos(2 π (1) t_m/L)  ,  sin(2 π (1) t_m/L)  ,  … ,  cos(2 π (H_n) t_m/L)  ,  sin(2 π (H_n) t_m/L)  )

is the vector of seasonality features at time t_m and 
    
    β_n    = (α_n,1 ,  β_n,1 ,  … ,  α_n,H_n ,  β_n,H_n) .

is the vector of unknown Fourier Coefficients. 


Suppose 𝐬_n,M ={ s_n,t_m}_m = -M^0 ∈^M+1 is a vector of seasonal component historical observations. Let ln(𝐬_n,M) = {ln(s_n,t_m) }_m = -M^0 be the vector as a result of taking the logarithm of each historical observation. Forecasting the seasonal component s_n,t_m for m>0 amounts to determining the Fourier coefficient vector β_n. We utilize Maximum a Posteriori (MAP) estimation, the main technique used in <cit.>. The goal of MAP estimation is to seek the parameter β_n ∈^2H_n maximizing f (  β_n  |  ln(𝐬_n,M) ), the conditional probability density function of the parameter β_n given the observed data. Equivalently, via Bayes Rule, we seek the parameter β_n maximizing the function ϕ: ^2H_n→ defined by
    
    ϕ(β_n)    = f (   ln(𝐬_n,M)   |  β_n )  f(β_n),

where f(β_n) is a specified prior probability density function of the parameter β_n. We assume β_n has a Gaussian prior distribution with mean 0 and covariance matrix (σ'_n )^2  𝐈_2 H_n for some fixed seasonality prior scale σ'_n>0, where 𝐈_2H_n is the 2H_n × 2H_n identity matrix. Moreover, by our model assumptions, ln(𝐬_n,M) is Gaussian with mean 𝐗_n,M β_n and covariance matrix σ_n^2  𝐈_M+1, in which 𝐗_n,M is the (M+1) × 2H_n matrix with m-th row equal to 𝐱_n,t_1-m^⊤. Therefore, we can write out the function ϕ in Eq. (<ref>) explicitly:
    
    ϕ(β_n)    = exp(-ln(𝐬_n,M) - 𝐗_n,M β_n ^2/2 σ_n^2)   exp(- β_n^2/2 (σ_n')^2)/(2 π)^M+2/2σ_n σ'_n 
       = exp(-ln(𝐬_n,M) - 𝐗_n,M β_n ^2/2 σ_n^2 - β_n^2/2 (σ_n')^2)/(2 π)^M+2/2σ_n σ'_n .

Computing the gradient of ϕ with respect to β_n, we obtain
    
    (∇ϕ)(β_n)    = -( 𝐗_n,M^⊤ 𝐗_n,M β_n - 𝐗_n,M^⊤ ln(𝐬_n,M) /σ_n^2 + β_n/(σ_n' )^2)  ϕ(β_n)  .

Upon setting the gradient equal to 0 and solving for β_n, we obtain the MAP estimate
    
    β_n    = ( 𝐗_n,M^⊤ 𝐗_n,M + (σ_n/σ_n')^2 𝐈_2H_n)^-1𝐗_n,M^⊤ ln(𝐬_n,M) .


Finally, for each m>0, we compute s_n,t_m, the predicted seasonal component of the n-th vendor's baseline volume at time t_m, by taking the inner product 𝐱_n,t_m^⊤ β_n and exponentiating the result.









  §.§.§ Trend Component


Next, we forecast the n-th vendor's trend component in Eq. (<ref>). Let { u_n,d}_d=1^D_n is a collection of D_n known times during which the vendor's n-th trend growth rate changes. We refer to such times as trend changepoints. Consequently, for each m ∈, define 𝐚_n,t_m∈^D_n to be the vector whose d-th component is 1 if t_m ≥ u_n,d and 0 otherwise. We assume the trend component obeys a piecewise linear model of the form:

    
    ln(g_n,t_m) = (κ_n+𝐚_n,t_m^⊤ δ_n ) t_m + 𝐚_n,t_m^⊤γ_n +θ_n + ζ'_n,t_m


for each m ∈, where 
    
        
  * κ_n ∈ is an unknown base growth rate;
        
  * δ_n ∈^D_n is an unknown vector of growth rate adjustments;
        
  * γ_n ∈^D_n is an unknown vector of continuity correction factors;
        
  * θ_n ∈ is an unknown base offset;
        
  * {ζ'_n,t_m}_m ∈ is an independent and identically distributed sequence of Gaussian random variables with mean 0.
    
In particular, the expression κ_n+𝐚_n,t_m^⊤ δ_n appearing in Eq. (<ref>) is the sum of the base growth rate κ_n and all D_n growth rate adjustments at time t_m. Meanwhile, the vector γ_n ensures the map t ↦ln(g_n,t) is a continuous function and is determined once δ_n is determined. 

Assume 𝐠_n,M = { g_n,t_m}_m=-M^0 ∈^M+1 is a vector of historical trend component observations, where M is a fixed positive integer. Let ln(𝐠_n,M) = {ln(g_n,t_m) }_m=-M^0 be the vector formed by taking the logarithm of each historical trend component observation. In addition, let { u_n,d}_d=1^D_n be a collection of known historical trend changepoints with D_n<M+1. Forecasting the trend component g_n,t_m for each m>0 amounts to determining the parameters κ_n, δ_n, and θ_n. We consider the MAP estimation technique, in which we seek the parameters maximizing the conditional probability density function f(κ_n, δ_n, θ_n  |   ln(𝐠_n,M) ). Following <cit.>, we impose the prior assumptions that the parameters κ_n and θ_n are both Gaussian with mean 0, and changepoint prior scale λ_n>0, we impose the prior assumption δ_n has Laplace distribution with mean 0 and covariance matrix λ_n  𝐈_D_n. We let κ_n, δ_n, θ_n be the resulting MAP estimates, and we set γ_n to have d-th component equal to -u_n,d δ_n,d, for each 1 ≤ d ≤ D_n, where δ_n,d is d-th component of the MAP estimate δ_n.


Now, for each m>0, we compute g_n,t_m, the predicted trend component of the n-th vendor's baseline volume at future time t_m. We randomly declare the time t_m to be a trend changepoint with probability D_n/M+1 and not a trend changepoint with probability 1-D_n/M+1. For each m>0, we randomly choose the future growth rate δ_n,D_n+m at time t_m according to the aforementioned prior Laplace distribution if t_m is labeled a trend changepoint and set δ_n,D_n+m=0 otherwise. Consequently, we set γ_n, D_n +m = - t_m  δ_n,D_n+m and compute

    
    ln(g_n,t_m)    = ( κ_n + (1  ,  … ,  1_D_n+m times)^⊤ (δ_n     ,  δ_n,D_n+1 ,  … ,  δ_n,D_n+m) )  t_m  
            + ( (1  ,  … ,  1_D_n+m times)^⊤ (γ_n   ,  γ_n,D_n+1 ,  … ,  γ_n,D_n+m)  + θ_n
            ),


where
    
    (δ_n   ,  δ_n,D_n+1 ,  … ,  δ_n,D_n+m)

is the (D_n+m)-dimensional vector formed by concatenating the MAP estimate δ_n from the previous paragraph with the vector (δ_n,D_n+1 ,  … ,  δ_n,D_n+m), and (γ_n   ,  γ_n,D_n+1 ,  … ,  γ_n,D_n+m) is defined similarly. We take the exponential of Eq. (<ref>) to obtain the predicted trend component at time t_m.

Finally, for all m>0, we obtain C_n,t_m, the  n-th vendor's predicted baseline volume at time t_m, by multiplying the corresponding seasonal and trend component forecasts at time t_m together:

    
    C_n,t_m=s_n,t_m g_n,t_m.
 

Henceforth, we refer to the computed number C_n,t_m as the expected baseline volume for vendor n at time t_m.




 §.§ Wired-On Model
 

Suppose one particular vendor is experiencing an outage, but it still remains enabled to customers. Throughout this section, we let 1 ≤ n_0 ≤ N be the unique index corresponding to the problematic vendor. We describe our approach to forecasting the total wired-on volume, which is the total volume across all N vendors. First, we forecast the availability of vendor n_0, which we interpret as the probability that a customer would have a successful experience with it on their first attempt. Then, we incorporate the n_0-th vendor's availability forecast and the expected baseline volume forecasts described in the previous section into a Monte Carlo simulation. At each future time, we use the Monte Carlo simulation to project the ultimate decision of a customer initially using the problematic vendor along with the time they make the decision. Finally, we aggregate the Monte Carlo simulation results together to forecast the wired-on volume.



  §.§.§ Vendor Availability Model
 

Let a_n_0,t be the problematic vendor's availability at time t. Vendor availability is not necessarily known at future times, and there is often limited data from the past. Furthermore, when a vendor is experiencing issues, its availability typically exhibits a decreasing trend. Therefore,  we consider a Double Exponential Smoothing model <cit.> to represent the problematic vendor's availability, which is explicitly

    
    S_n_0,t_m   = α  (a_n_0,t_m) + (1-α)   (S_n_0,t_m-1 + b_n_0,t_m-1) 
    
            b_n_0,t_m   = η  (S_n_0,t_m - S_n_0,t_m-1) + (1-η)   b_n_0,t_m-1,


for each m ∈, where
    
        
  * α∈ [0,1] is an unknown smoothing factor;
        
  * η∈ [0,1] is an unknown trend factor;
        
  * S_n_0,t_m∈ is an unknown smoothing component;
        
  * b_n_0,t_m∈ is an unknown trend component;
    

Let {  a_n_0,t_m}_m=-M^0 be a historical sequence of the problematic vendor's availability observations for some M>0. Forecasting the problematic vendor's availability a_n_0,t_m for all m>0 amounts to determining the factors α and η based on the historical observations. First, we initialize the smoothing component in Eq. (<ref>) and the trend component in Eq. (<ref>) by setting
    
    S_n_0,t_-M   :=a_n_0,t_-M
    
            b_n_0,t_-M   :=a_n_0,t_-M+1-a_n_0,t_-M.

With these initialized values, we seek the optimal factors α and η that minimize the quantity

    
    { a_n_0,t_m - S_n_0,t_m -b_n_0,t_m}_m=-M^0  _2/√(M+1)   =  √(1/M+1∑_m=-M^0 (a_n_0,t_m - S_n_0,t_m -b_n_0,t_m)^2),


the root mean-squared error between {  a_n_0,t_m}_m=-M^0 and { S_n_0,t_m +b_n_0,t_m}_m=-M^0. The root mean-squared error is a continuous function of the tuple (α,η) with domain equal to the unit square [0,1]^2. By the extreme value theorem, we are guaranteed optimal factors α and η that minimize the root mean-squared error. Seeking such factors, however, is a nonlinear optimization problem that cannot be solved analytically. Nevertheless, if we can determine these optimal factors, then we can compute a_n_0,t_m, the problematic vendor's predicted availability at time t_m for each m>0, via the formula

    
    a_n_0,t_m = S_n_0,t_0 +  m   b_n_0, t_0.






  §.§.§ Customer Behavior Monte Carlo Simulation
 

We now describe the Monte Carlo simulation to forecast customer behavior when the problematic vendor remains enabled. Recall n_0 is the index corresponding to the problematic vendor. The Monte Carlo simulation combines together each vendor's expected baseline volume forecasts, the problematic vendor's availability forecast, and certain probability distributions modeling specific customer decisions. We compute these distributions over a historical time period during which the problematic vendor had previously experienced issues.

First, for each k ≥ 0, we consider the retry distribution, which is a Bernoulli distribution with parameter

    
    π_n_0,k = (a customer retries with any vendor |  customer has failed k times with n_0),


where  stands for probability. The second distribution is the switch distribution, which is another Bernoulli distribution with parameter 
    ρ_n_0,k = (customer switches to another vendor |  customer is retrying after failing k times with n_0).

Lastly, we consider the interattempt time distribution, which has cumulative distribution function

    
    τ_n_0(s) = (customer takes at least s seconds to retry with any vendor |  customer failed with n_0),


where s is a non-negative integer.




Given a customer who initially attempts to use vendor n_0 at time t_m for some integer m, we describe the Monte Carlo simulation to project their ultimate decision, which we illustrate in Fig. <ref>. We initialize the the number of failures k = 0. In order to capture time delayed effects during our actual time of interest, we initialize m to be some negative integer. We simulate the outcome (success or failure) of the customer's initial attempt with the problematic vendor according to the Bernoulli distribution with parameter a_n_0,t_m as defined in Eq. (<ref>). When m ≤ 0, we use the problematic vendor's actual availability a_n_0,t_m in place of the predicted availability. If the customer succeeds, then we record that outcome at time t_m and terminate the simulation. Otherwise, we increment k and simulate the customer's decision whether to retry the experience with any vendor based on the retry distribution defined in Eq. (<ref>). If the customer decides to abandon the experience, we record their decision and terminate the simulation. Otherwise, we simulate the customer's decision whether to switch to another vendor based on the switch distribution defined in Eq. (<ref>). We also simulate the time ε (in seconds) taken for the customer to execute their decision based on the aforementioned interattempt time distribution defined in Eq. (<ref>). We terminate the simulation if the customer switches to another vendor and record this decision at time t_⌊ m+ 
ε/60⌋. Otherwise, we repeat the entire  procedure, but we replace m with m+
ε/60. This process is described in Algorithm <ref>.



Finally, we forecast the total wired-on volume by aggregating the Monte Carlo simulation results from the previous paragraph. Recall that C_n_0,t_m is the problematic vendor's baseline volume at time t_m, as defined in Eq. (<ref>). Here, we interpret the integer ⌊C_n_0,t_m⌋ as the predicted total number of customers who use the problematic vendor at time t_m. For each 1 ≤ i ≤⌊C_n_0,t_m⌋, we project the i-th customer's ultimate decision and their time of decision based on the Monte Carlo simulation. Now, we let A_n_0,t_m be the total number of customers projected to have successful experiences with the problematic vendor at time t_m. We also let A_other,t_m be the total number of customers projected to have successful experiences with other vendors at time t_m. Lastly, we let C_other,t_m be the total sum of all expected baseline volume forecasts across all vendors excluding the problematic vendor. Using these predicted numbers, we compute W_n_0,t_m,on, the predicted wired-on volume at time t_m, via the formula

    
    W_n_0,t_m,on = A_n_0,t_m + A_other,t_m +C_other,t_m.


 


















 §.§ Wired-Off Model
 
Now, suppose the problematic vendor n_0 is disabled. Let W_n_0,t_m, off be the total wired-off volume at time t_m, which is the total volume across all enabled vendors at time t_m.  In this situation, customers will not be able to retry their experiences with the problematic vendor. Past incident data has shown that the ratio of additional volume for vendors other than the problematic one to the baseline volume of the problematic vendor is stationary over time (Fig. <ref>); we performed an Augmented Dickey-Fuller test <cit.> which resulted in a p-value of 2.82 × 10^-18. Therefore, we do not need to consider any time-dependent effects of the rate at which customers who typically use the problematic vendor would switch to use another vendor. We consider a simple linear model of the form

    
    W_n_0,t_m, off  = Δ_n_0C_n_0,t_m + C_other,t_m + ϵ_n_0, t_m,




to represent the wired-off volume, where for each m ∈, 
    
        
  * C_n_0,t_m is the disabled vendor's expected baseline volume at time t_m, as defined in Eq. (<ref>);

        
  * C_other,t_m is the total expected baseline volume across all enabled vendors at time t_m;

        
  * Δ_n_0∈ is the unknown slope;

        
  * {ϵ_n_0,t_m}_m ∈ is an independent and identical sequence of Gaussian random variables with mean 0;
    

Forecasting the wired-off volume amounts to determining the slope Δ_n_0 based on historical data over a timeframe during which problematic vendor n_0 was disabled. To this end, let T_hist be a finite collection of historical times corresponding to a previous known incident during which the problematic vendor was disabled. Suppose we have a vector 𝐖_n_0, T_hist, off = { W_n_0,t, off}_t ∈ T_hist containing the total wired-off volume observations over T_hist. Via Eq. (<ref>), we compute the predicted vector 𝐂_n_0,T_hist = {C_n_0,t}_t ∈ T_hist containing the expected baseline volume observations corresponding to the disabled problematic vendor. Then, we compute the predicted vector 𝐂_other,T_hist =  {∑_n  n_0C_n,t}_t ∈ T_hist, in which each observation at time t is the sum of forecasted baseline volumes at time t across all enabled vendors. Now, we can obtain an estimate for the slope Δ_n_0 by solving the linear equation 

    
    𝐖_n_0, T_hist, off = Δ_n_0𝐂_n_0,T_hist + 𝐂_other,T_hist,


for Δ_n_0. Upon subtracting  𝐂_other,T_hist from both sides of Eq. (<ref>), left multiplying both sides by 𝐂_n_0,T_hist^⊤, and finally dividing both sides of the equation by (𝐂_n_0,T_hist) ^ ⊤𝐂_n_0,T_hist = 𝐂_n_0,T_hist_2^2, we obtain the estimate

    
    Δ_n_0   = (𝐂_n_0,T_hist)^⊤(𝐖_n_0, off, T_hist - 𝐂_other,T_hist) /𝐂_n_0,T_hist_2^2.


Thus, for all m>0, we can compute W_n_0,t_m, off, the predicted wired-off volume at time t_m, via the formula
    
    W_n_0,t_m, off = Δ_n_0C_n_0,t_m + C_other,t_m.


















§ RESULTS AND VALIDATION


In this section, we discuss sample results and the validation of our forecasting models on vendor volume data. We apply our methodology to real incidents and analyze customer and business benefits.




 §.§ Expected Baseline Volume Forecast Validation


For each 1 ≤ n ≤ N, recall that we represent the n-th vendor's expected baseline volume through the additive seasonal model for its logarithm as defined in 
Eq. (<ref>). In order to forecast the baseline volume, we leverage Prophet, a forecasting library developed by Meta. In Section <ref>, we discussed how to forecast the trend and seasonal components individually via MAP estimations given historical data associated with such components. For computational efficiency, however, Prophet computes a single MAP estimate for the concatenated list of parameters Φ_n=(β_n, κ_n, δ_n, θ_n), in which we recall β_n is the vector of Fourier Coefficients in the seasonal component model, and (κ_n, δ_n, θ_n) is the list of parameters in the trend component model. This single MAP estimate for Φ_n is computed based on the n-th vendor's historical baseline volume data.

Moreover, Prophet exposes several hyperparameters to control model fitting. We list the main ones that affect forecasting in Table <ref>. The default values of such hyperparameters, however, typically do not result in a good fit for vendor data. Therefore, we employ an efficient random hyperparameter search <cit.> to determine an optimal set of hyperparameters that minimizes the root mean-squared error between the actual future baseline volume { C_n,t_m}_m ≥ 0 and the expected baseline volume {C_n,t_m}_m ≥ 0 computed via Eq. (<ref>). We list the hyperparameters that we tune, along with their lower and upper bounds over our search space, in  Table <ref>. Furthermore, in Fig. <ref>, we display an example of forecasting a vendor's baseline volume based on two weeks of past historical volume data as a result of the hyperparameter grid search, computing the MAP estimate for the concatenated list of parameters Φ_n, and using Eq. (<ref>).










 §.§ Availability Forecast Validation
 

Recall n_0 is the index corresponding to a problematic vendor and that we forecast the problematic vendor's availability using a Double Exponential Smoothing model governed by Eq. (<ref>) and Eq. (<ref>). We demonstrate how we validate such forecasts. Because the availability data is limited, we perform the validation on a rolling basis. We fix a positive integer W, the rolling window size in minutes, and we fix a positive integer R, the forecasting horizon size. For each M ≥ 0, we consider the historical availability rolling window time-series {  a_n,t_m}_m=M-W^M, consider a Double Exponential Smoothing model for the time-series, and compute the rolling horizon forecast time-series {a_n,t_m}_m=M+1^M+R via Eq. (<ref>). We choose the optimal smoothing and trend factors α and η in Eqs. (<ref>) and (<ref>), respectively, again employing an efficient random hyperparameter search. Finally, we compute the associated rolling horizon mean-squared error between the rolling true horizon time-series {  a_n,t_m}_m=M+1^M+R and the rolling horizon forecast time-series {a_n,t_m}_m=M+1^M+R. The closer the rolling horizon mean-squared errors are to 0, the better the corresponding rolling horizon availability forecasts are. We show a sample true historical availability time-series, rolling horizon forecast time-series, and associated rolling horizon mean-squared errors in Fig. <ref>. We see that the Double Exponential Smoothing models are acceptable because their associated root mean-squared errors are low.






 §.§ Wired-Off Volume Forecast Validation


Recall when the problematic vendor n_0 is disabled, we use a simple linear model to represent the total wired-off volume across all enabled vendors, as presented in Eq. (<ref>). More specifically, we fit a simple linear model of the difference between the total wired-off volume and the aggregate sum of enabled vendors' expected baseline volumes against the disabled vendor's expected baseline volumes. 

We perform the regression diagnostics to validate the use of the linear model. For each m>0, let W_n_0, t_m,off and W_n_0, t_m,off be the actual wired-off volume and predicted wired-off volume at time t_m, respectively. For each m>0, we define the residual 
    
    r_n,t_m   = W_n, t_m,off- W_n, t_m,off,

at time t_m. We show the results of three different wired-off linear models in Fig. <ref> corresponding to three different situations when a problematic vendor was actually disabled. 

Firstly, in a generic linear model, the plot of the residuals against the predicted values is symmetrically distributed about the horizontal axis. In the second row of Fig. <ref>, we plot the corresponding residuals against predicted values for each wired-off linear model. We see these plots exhibit visible symmetry about the horizontal axis. 

Next, in a generic simple linear model, the recursive residuals, which are the standardized one time-step ahead prediction errors, are independent and identically Gaussian distributed. In order to validate this for our sample wired-off linear models, we formally perform the Harvey-Collier test <cit.> at a fixed percent significance level. In this test, the null hypothesis states the recursive residuals are normally distributed. Therefore, we need to check whether we do not reject the null hypothesis. We compute Harvey-Collier test statistics corresponding to our sample wired-off linear models and tabulate their p-values in the first column of Table <ref>. At the 5 percent significance level, since the p-values are much greater than 5 percent, we do not reject the Harvey-Collier null hypothesis in any of our wired-off linear models.

Next, in a generic simple linear model, the majority of auto-correlations at lag 1 lies within the 95 percent confidence bound about 0 i.e. approximately within [ -√(2/M), √(2/M)], where M is the number of observations. Moreover, we formally perform the Durbin-Watson test <cit.>, whose null hypothesis states there is no auto-correlation within the residuals. In a generic linear model, the residuals are independent, and so we need to check whether we do not reject the null hypothesis. A heuristic condition to not reject the null hypothesis is if the Durbin-Watson test statistic lies in the interval [1.5,3.5]. We tabulate the Durbin-Watson test statistics corresponding to our sampled wired-off linear models in the second column of Table <ref>. Because all our three obtained test statistics lie between 1.5 and 3.5,  we do not reject the Durbin-Watson null hypothesis in any of our sample wired-off linear models.

Lastly, in a generic linear model, the residuals are normally distributed. To validate this, we inspect the quantiles of the standardized residuals against the theoretical quantiles of the standard normal distribution and check whether the observations lie close to the straight line y=x.  For our sample wired-off linear models, we plot the corresponding quantiles of the standardized residuals against the theoretical quantiles of the standard normal distribution in the last row of Fig. <ref>. We see these plots exhibit visible symmetry.










 §.§ Wire-off Recommendation



With the forecasts {W_n,t_m, off}_m = 1^R  and {W_n,t_m, on}_m = 1^R, where R is the fixed future horizon size defined in Section <ref>, we determine the smallest index m^* in which  


    W_n,t_m, off > W_n,t_m, on


for all m ≥ m^* in our simulated window. If m^* exists, we make a decision to wire off vendor n starting at time t_m^*. Otherwise, we keep vendor n wired on.





Figure <ref> shows our simulated trends for a real incident in which a vendor was experiencing issues. The optimal wire off time according to Eq. (<ref>) is considerably earlier than the actual wire off time. We define t_actual to be the actual wire-off time and t_m^* to be the optimal wire off time based on Eq. (<ref>) and our simulation. Table <ref> shows the lead time t_actual - t_m^* of our optimal simulated wire off time over the actual wire off time for several historical incidents. We see that our optimal wire off time is often several minutes before the actual wire off time. Depending on the experience and time of day, this can equate to a considerable number of additional completed customer experiences.




Thus, using our approach, we can see benefits to both our customers and the business over the existing solution. In the short term, we are immediately providing an improved experience which reduces friction for our customers. In the long term, the cumulative improved experiences will grow over several vendor incidents; this not only means more overall customer experiences completed, but also greater customer satisfaction over time, which will drive people to continue using eBay. 








§ FUTURE WORK

In this paper, we predicted customer behavior to arrive at the decision to wire off a problematic vendor. A natural question to ask is when a disabled vendor should be enabled again for customers. We hope to use the methodology in this paper to arrive at a data-driven answer to this question. This problem poses several additional challenges; namely, we would only like to re-enable a vendor when we know its impact has been mitigated. However, after the vendor is wired off, we no longer see real-time metrics and must therefore find another way to measure its availability. A solution for this may depend on the data available for particular customer experience and/or vendor. 

Furthermore, recall the key assumption for our approach to forecasting completed customer experiences is the existence of only one problematic vendor. In many situations, however, more than one vendor could experience issues, which would severely impact eBay's business. We hope to modify the probabilistic model in Section <ref> in order to forecast customer behavior when multiple problematic vendors are either wired on or wired off. 







elsarticle-harv







 Paper Title: An effect analysis of the balancing techniques on the counterfactual explanations of student success prediction models.







Counterfactual explanations of student success prediction models






Cavus and Kuzilek
Eskisehir Technical University, Department of Statistics, Turkiye 

mustafacavus@eskisehir.edu.tr Humboldt University of Berlin, Unter den Linden
6, Berlin, Germany

jakub.kuzilek@hu-berlin.de







An effect analysis of the balancing techniques on the counterfactual explanations of student success prediction models
    https://orcid.org/0000-0002-6172-5449
    < g r a p h i c s >
Mustafa Cavus1corresponding author https://orcid.org/0000-0002-8656-0599
    < g r a p h i c s >
Jakub Kuzilek2
    August 5, 2024
=================================================================================================================================================================================




§ ABSTRACT 
In the past decade, we have experienced a massive boom in the usage of digital solutions in higher education. Due to this boom, large amounts of data have enabled advanced data analysis methods to support learners and examine learning processes. One of the dominant research directions in learning analytics is predictive modeling of learners' success using various machine learning methods. To build learners' and teachers' trust in such methods and systems, exploring the methods and methodologies that enable relevant stakeholders to deeply understand the underlying machine-learning models is necessary. In this context, counterfactual explanations from explainable machine learning tools are promising. Several counterfactual generation methods hold much promise, but the features must be actionable and causal to be effective. Thus, obtaining which counterfactual generation method suits the student success prediction models in terms of desiderata, stability, and robustness is essential. Although a few studies have been published in recent years on the use of counterfactual explanations in educational sciences, they have yet to discuss which counterfactual generation method is more suitable for this problem. This paper analyzed the effectiveness of commonly used counterfactual generation methods, such as WhatIf Counterfactual Explanations, Multi-Objective Counterfactual Explanations, and Nearest Instance Counterfactual Explanations after balancing. This contribution presents a case study using the Open University Learning Analytics dataset to demonstrate the practical usefulness of counterfactual explanations. The results illustrate the method's effectiveness and describe concrete steps that could be taken to alter the model's prediction.







§ INTRODUCTION


For centuries universities have been collecting information about their students. With the rise of Information and Communication Technologies, the information collected is transformed from paper-based collections to digital domains. The introduction of new digital education formats and the information collection shift resulted in storing vast amounts of student and study-related data including student demographics, assessment, learning design, and context. In combination with the advancement in Data Mining and Machine Learning (ML) research, the collected data enabled new research exploring the educational domain. The most prominent research fields are Educational Data Mining (EDM) and Learning Analytics (LA), which explore the educational domain from two different perspectives (Siemens and Baker, 2012). More recently, the concerns about the use of Artificial Intelligence (AI) have become stronger uncovering the limitations and possible problems such as bias and explainability of models developed. As a consequence, new data and AI regulations such as GDPR and the AI Act in the EU have been established (Hoel et al., 2017). As a consequence trust in the analytical tools and AI methods in higher education has been reduced leading to the new approach in LA research called Trusted Learning Analytics (TLA) (Drachsler H., 2018). The TLA approach focuses on using intrinsically explainable `white box` AI models and systems. This significantly reduces the opportunity of using more “user-unfriendly” models such as Random Forests (RF) or Neural Networks. Luckily, the field of Explainable Artificial Intelligence (XAI) provides researchers with the methods with the potential to unlock the `black box` models for use in TLA systems (Drachsler H., 2018). The trend of using XAI methods in the educational domain is highly resonating within the research community resulting in more research in the area in recent years (e.g., Human-Centric eXplainable AI in Education Workshop at 17th Educational Data Mining Conference[https://hexed-workshop.github.io/]). 

There are various tasks within the LA that focus on supporting learners and educators using various tools and methods. However, one of the most common objectives is the predictive modeling of learner success (with varying definitions of success), which focuses on the identification of the learners in need of help with their studies (Papamitsiou and Economides, 2014). Within the task of success prediction, the legacy learner and learning data are utilized for training the prediction model using the ML algorithm (Arnold and Pistilli, 2012; Waheed et al, 2020; Adnan et al., 2021). From the LA point of view, the prediction delivered by the ML model is used as a trigger for educational intervention. Thus the model itself is used as a tool by the lecturer, teaching assistant, or anyone responsible for supporting the students. Yet, there is a constant demand for providing not just the prediction itself, but also the “reasons behind the model decision” (Kuzilek et al., 2015). At this stage, again, the XAI comes into play and fosters the objectives of TLA (Drachsler H., 2018).

In the context of ML, the predictive models pursue the highest predictive accuracy, thus very often the so-called `black-box` models are preferred over the so-called `white-box` models, which in addition to the prediction provide intrinsically interpretable prediction (Guidotti et al., 2018; Biecek et al., 2021; Holzinger et al., 2022). However, to enable the power of XAI for the `black-box` models the post-hoc methods can be used (Pinto and Paquette, 2024). The XAI methods are primarily categorized into global and local. At the global level, they reveal which variables are important in the model. In contrast, at the local level, they answer questions about the contributions of variables in generating individual predictions (Molnar et al., 2020; Cavus et al., 2023). However, commonly used global and local tools, while sufficient for understanding the prediction made for a particular observation, are not sufficient for generating a counterfactual understanding of an undesirable outcome. Commonly used XAI methods (both local and global) are adequate for understanding particular observation predictions and not for generating a counterfactual understanding of an undesirable outcome (e. g. negative class in a binary classification problem).
To improve understanding of the undesirable outcome the method of counterfactual explanations (CEs) has become popular. CEs are defined as the minimal change in the variable values to flip the model's prediction into the intended outcome (Artelt and Hammer, 2019). In the frame of learner success prediction, the models may indicate an unfavorable outcome, but they do not provide recommendations to reverse the learner situation. Counterfactual explanations provide the extension of the baseline model and provide such a recommendation by highlighting necessary changes in the learner profile to reverse the negative outcome. Learners, teachers, and curriculum designers are guided toward actions or measures to be taken through their generated explanations.

The use of counterfactual explanations in LA has been explored in several studies (Tsiakmaki et al., 2021; Zhang et al., 2023; Afrin et al., 2023). All of the research works focused on providing a frame for delivering actionable insights to relevant stakeholders using the CE. Facing numerous counterfactual explanations due to the nature of optimization problems requires selecting those explanations that fulfill specific criteria beneficial for the stakeholder. Because of their background, challenges, and needs differences, each learner requires personalized counterfactual (Smith et al., 2022). 

The research presented in this paper focuses on using CE measures for the evaluation of the effect of balancing techniques used on the raw imbalanced dataset. More specifically we focus on the following research questions:


RQ1: What is the most appropriate method for generating the counterfactual explanations after balancing?


RQ2: How do balancing techniques affect the counterfactual explanations of student success prediction models?


This study compares the qualities of different counterfactual generation methods for students whose success prediction model developed after balancing the training dataset anticipates failing.  For the reproducibility of the developed approach, we used the Open University Learning Analytics Dataset (OULAD) (Kuzilek et al., 2017) as a raw data source. The study is essential in two ways: (1) because the missing evaluation of the counterfactual quality can lead to inefficient explanations, and this may compromise their trustworthiness (Artelt et al., 2021), (2) there is no uniformly better method for each domain (Dandl et al., 2023) and this is the first benchmark in the domain of LA, and (3) there are no many investigations on the effect of balancing methods on the counterfactual explanations (Gunonu et al, 2024).
The rest of the paper is organized using the following analysis approach. It examines the effect of balancing strategies on the quality of counterfactuals generated by the three most commonly used methods. Finally, the results are discussed.





§ METHODS


This section contains details of the dataset, counterfactual explanations, resampling methods, and the experimental design.



 §.§ Data

The OULAD dataset has been released by the research team of the Open University, the largest distance-learning institution in the UK. It is utilized to analyze the impact of the balancing strategies on the counterfactual generation methods. Typical course duration at the Open University is nine months and the course includes multiple assignments and a final exam. The most important assignments are Tutor Marked Assignments (TMAs), which represent critical milestones throughout the course. The dataset comprises learner demographics, assessment results, and click-stream log data from the interactions of learners with a Moodle-like Learning Management System (LMS).

For the analysis, the STEM course DDD and its 2013J and 2014J presentations studied by 3741 students have been selected. The course includes six TMAs. The final student result was used as the target variable for model training. Students with the result “Distinction” have been merged with students with the result “Pass”. Reducing the prediction task to binary classification to classes: “Pass” and “Fail”. We excluded the actively withdrawn students (n = 1328). The resulting dataset includes data from 2296 students. With the focus on actionable insights into the learner activity, we focused on the click-stream data only since it contains the most vital information (Kuzilek et al., 2015). Thus, the dataset consists of 42 predictors, which are numerical variables containing the weekly learner summary of online interactions with the LMS. Table <ref> provides descriptions of the selected variables.





 §.§ Counterfactual explanations

Counterfactual explanations (CE) illustrate "what-if" scenarios that emphasize the necessary alterations to the input data to change a model's output (Watcher et al., 2017). X = [x_1, x_2, ..., x_p] represent a data matrix with n observations and p variables and y be the response vector. The objective is to identify a function f: X → y that minimizes the expected value of the loss function L in predictive modeling. A counterfactual x^'∈𝐑^p of observation x ∈𝐑^p is determined by solving the following optimization problem:


    argmin_x^'∈𝐑^p L[f(x^'), y^'] + d(x, x^')


where 𝐑^p represents p-dimensional real space, L is a loss function that penalized the difference between the prediction f(x^') and the desired outcome y^', and d is a distance function between the observation x and x^'. A CE specifies the necessary adjustments in one or more variables to change the model's prediction. The distance function d regulates the proximity between the original observation and the counterfactual. 

Figure <ref> visualizes an observation and its counterfactuals. Assume that f is a student success prediction model and x is a vector consists the variable values of a student. The prediction of the model f for the student x who has failed. The red zone shows the fail area, and the green one shows the pass area. They are divided by the decision boundary of the model. The CEs x^'_1, x^'_2, x^'_3 represent the ways how the student can pass. 



Counterfactuals strive to minimize the distance between the target observation and the counterfactual; however, additional properties are essential for a counterfactual explanation (Wachter et al., 2017; Karimi et al., 2020). Sparsity suggests altering the minimal number of variables to keep the explanation straightforward. Minimality aims for the most minor feasible changes in the variable values. Validity is ensured by reducing the difference between the counterfactual instance x^' and the original observation x while ensuring the model's output matches the desired label y^'. Proximity emphasizes the necessity for a slight variation between the factual and counterfactual features. Plausibility requires that counterfactual explanations remain realistic and closely follow the underlying data distribution. Over 120 known counterfactual generation methods; see Warren et al. (2023) for further details. However, we focused on three widely used counterfactual methods WhatIf Counterfactual Explanations, Multi-Objective Counterfactual Explanations, and Nearest Instance Counterfactual Explanations to facilitate the comparison of counterfactual quality.



What-if counterfactual explanations. The What-if method (WhatIf) finds the observations closest to the observation x from the other observations in terms of Gower distance, solving the following optimization problem (Wexler et al., 2019):


    x^'∈ argmin_x ∈ X d(x, x^').


Multi-objective counterfactual explanations. The Multiobjective Counterfactual Explanations (MOC) method aims to generate counterfactual explanations by optimizing multiple objectives simultaneously (Dandl et al., 2020). These objectives often include validity, proximity, sparsity, and plausibility. 


    x^'∈ min_x ( o_v(f̂(x), y^'), o_p(x, x^'), o_s(x, x^'), o_pl(x, X) )


where o_v, o_p, o_s, o_pl are the objective functions for the desired properties validity, proximity, sparsity, and plausibility, respectively. Thus, it is expected that the counterfactuals generated by the MOC method are valid, proximity, sparse, and plausible. 

Nearest instance counterfactual explanations. The Nearest Instance Counterfactual Explanations (NICE) method identifies observations that are most similar to a given observation using the heterogeneous Euclidean overlap method (Burghmans et al., 2023). This approach allows for two options in the objective function, depending on the properties of proximity and sparsity, offering flexibility in how it can be applied.


The WhatIf method produces counterfactuals that are valid, proximal, and plausible. It has been demonstrated that the MOC method generates a higher number of counterfactuals that are closer to the training data and require fewer feature changes compared to other counterfactual methods (Dandl et al., 2020). Additionally, NICE specifically generates proximity-focused counterfactuals. However, no single method consistently outperforms others across datasets from various domains (Dandl et al., 2023). Therefore, evaluating the quality of the generated counterfactuals is essential, and we will conduct experiments to evaluate this in the following section.



 §.§ Balancing techniques


The most commonly encountered challenge in designing predictive models with high discriminatory performance is an imbalanced class distribution in the response variable. In the binary case, the imbalance problem occurs when one class is observed less frequently. Models with such response variables tend to be biased toward the majority class in their predictions. Consequently, when dealing with the imbalance problem, models often have a significantly lower performance in correctly predicting the minority class than the majority class. In real-world problems, the class of interest is generally the minority class. For example, in predicting student dropouts, students who drop out are observed less frequently than those who do not. In the classification problem of predicting whether a student will complete a specific educational material or content module, students who do not complete the material are observed less frequently than those who do. In learning analytics, when considering student success prediction models, students who fail are observed less frequently than those who succeed. In these examples, students who drop out, do not complete educational materials, and fail constitute the minority class. Due to the nature of these problems, the focus is on identifying the minority class. The inaccurate models in correctly predicting the minority class is a problem that must be overcome in such scenarios. 

Solutions to this problem are divided into three categories: data-based, model-based, and weighting-based methods. The most commonly used data-based methods involve balancing class distributions through random undersampling or oversampling and synthetic data generation techniques. In undersampling, a subset of the majority class is randomly selected to match the minority class, whereas, in oversampling, the number of observations in the minority class is increased through resampling to match the size of the majority class (Chawla, 2010). In synthetic data generation methods, new observations are artificially generated from the distribution of the minority class to balance the size with the majority class (Elyan et al., 2021; Liu, 2022). Model-based methods are specific models developed to address the imbalance problem (Yin et al., 2020; Gu et al., 2022). Weighting-based methods aim to achieve higher prediction performance by penalizing the model more for errors in predicting the minority class (Zong et al., 2013; Tao et al., 2019). Although there are many methods to solve the classification problem in unbalanced data, in recent years, it has been found that these methods generally need to be revised and have adverse effects on classification models (Junior and Pisani, 2022; Stando et al., 2024; Cavus and Biecek, 2024; Carriero et al., 2024). These criticisms, mainly focusing on oversampling, undersampling, and synthetic data generation methods, brought the cost-sensitive approach to the fore (Gunonu et al., 2024). This study used data-based and weighting-based methods due to the mentioned criticism, their practical applications, and their frequent usage in the literature.




 §.§ Experimental design


This paper focuses on which method provides the highest quality counterfactual explanations for the student success prediction model trained with and without hyperparameter tuning (i.e., vanilla model) regarding the imbalancedness problem using the OULAD dataset. Thus, the approach followed, which is given in Figure <ref>, is (1) balancing the dataset, (2) training the model with and without hyperparameter tuning, (3) generating the counterfactuals, and (3) evaluating the effect of the balancing techniques of the imbalancedness problem producing the evaluation criteria.  

 



Balancing. Two balancing strategies are used. The dataset is balanced using several resampling methods such as undersampling, oversampling, and SMOTE, and the models are trained on the original dataset with the cost-sensitive approach. 



Modeling. The random forest algorithm is used in modeling because tree-based models exhibit lower prediction performance than alternative complex models in classifying tabular datasets (Grinsztajn et al., 2022). It is trained with and without hyperparameter tuning to achieve higher prediction performance. The performance of the random forest models trained on imbalanced (i.e., Original), balanced datasets by the oversampling, undersampling, SMOTE, and also trained with the cost-sensitive approach are compared. The costs are chosen as 2.37931 for the minority class (i.e., failed students) and 1 for the majority class regarding the imbalance ratio. Moreover, to achieve better predictive performance the models are tuned in terms of hyperparameters , , and  using the 10-fold repeated cross-validation in addition to the vanilla versions of the model which is trained with the default values of the hyperparameters.



Counterfactual generation. After the modeling phase, the counterfactuals are generated for failing students which are estimated by the models using MOC, sparsity-based NICE (NICE_sp), proximity-based NICE (NICE_pr), and What-If methods. 



§ RESULTS & DISCUSSION

In this section, the results are summarized. Firstly, the performance of the models is compared, and then the counterfactuals are evaluated to determine the best counterfactual generation method for the case considered in the paper.



Model performance. The performance of the random forest models trained on imbalanced and balanced datasets by the oversampling, undersampling, SMOTE, and cost-sensitive approach are given in Table <ref>. The imbalance ratio of the test set is 2.41 (number of observations in the majority class/number of observations in the minority class), thus the performance evaluations should be using the F1 score as well as accuracy and AUC. 



The vanilla Random Forest models generally outperform tuned models in terms of accuracy and F1 scores across most balancing strategies, particularly on original data and some resampling methods. Vanilla models demonstrate higher accuracy and more balanced F1 scores, especially under oversampling and SMOTE techniques. On the other hand, tuned models achieve slightly higher AUC values with cost-sensitive learning and SMOTE, indicating better classification discrimination. Sampling methods like oversampling and SMOTE improve performance for both vanilla and tuned models, while undersampling tends to decrease accuracy and F1 scores but maintains stable AUC values. Cost-sensitive learning offers balanced improvements, with both model types benefiting from enhanced AUC scores. Overall, while vanilla models excel in accuracy and F1 scores, tuned models show enhanced AUC values in specific conditions, highlighting the trade-offs between different performance metrics and modeling approaches. The tuned values of the hyperparameters for the models are given in Table <ref> in the Appendix.



Counterfactual evaluations. The counterfactual generation methods can generate more than one explanation for an observation, also each method may generate different explanations. The number of counterfactuals generated by the methods is given in Table <ref>. The MOC generates the highest number of counterfactuals independently from the balancing strategy and model while the NICE methods generate the lowest number of counterfactuals. The differences between the number of counterfactuals between the balancing strategies depend on the number of students that were predicted as failed by the models. The number of counterfactuals for the models is slightly different because of the difference between the models caused by the hyperparameter optimization.



It is necessary to evaluate the quality or usefulness of the counterfactuals before deployment. Thus, we conduct a comparison study to analyze the effect of the conditions regarding the balancing and modeling strategies on the counterfactual quality. We aim to determine the best counterfactual generation method to find actionable insights from the student success prediction models trained on the OULAD dataset. The quality of counterfactuals is visualized using error bar plots as in Figure <ref>. An error bar plot shows the variability or uncertainty of data. It features error bars that extend above and below the median of the observations. Error bars can show measures of dispersion such as standard deviation, standard error, or confidence intervals, providing a visual indication of the reliability and precision of the data. Figure <ref> demonstrates that each method exhibits varying performance regarding quality metrics across different balancing and modeling strategies.



NICE_sp and NICE_pr consistently demonstrate superior performance with the models trained on the original dataset. The minimality and plausibility values are particularly low, with medians near 0 and minimal variability, suggesting robust performance. On the other hand, MOC and WI show much higher values, especially in minimality where median values reach around 30, indicating suboptimal outcomes. Similarly, in metrics like proximity and sparsity, NICE_sp and NICE_pr maintain low values, whereas MOC and WI exhibit considerably higher values, suggesting that these methods struggle with the original data distribution. 

When applying the Undersampling method, there is a general improvement in minimality values across all methods, though MOC and WI still trail behind NICE_sp and NICE_pr. While NICE_sp and NICE_pr continue to perform well with relatively low values across all metrics, the error bars suggest a slight increase in variability. MOC and WI, although showing some improvement, still exhibit higher plausibility and proximity values, indicating that undersampling does not fully mitigate their performance issues. 
The Oversampling method highlights the strengths of NICE_sp and NICE_pr even further. These methods maintain low values across all metrics, particularly in minimality and plausibility, where their performance remains nearly flawless with median values close to 0. In contrast, MOC and WI continue to struggle, showing higher values across metrics such as proximity and sparsity, with only marginal improvements compared to the Original and Undersampling strategies. This suggests that while oversampling enhances performance for NICE_sp and NICE_pr, it does not sufficiently benefit MOC and WI.

Moving to SMOTE, NICE_sp, and NICE_pr once again emerge as the top performers, maintaining low values across all metrics. The proximity and sparsity values for these methods remain minimal, reflecting strong and consistent performance. MOC and WI, however, continue to display higher values in metrics like minimality and validity, suggesting that even with synthetic data generation, these methods are less effective. The error bars for MOC and WI also indicate greater variability, reinforcing the idea that SMOTE does not significantly improve their robustness.

Finally, the cost-sensitive approach shows that NICE_sp and NICE_pr maintain their strong performance, with median values remaining low across all metrics. Particularly in minimality and plausibility, these methods exhibit near-perfect performance, with minimal error bars indicating consistent results. MOC and WI show slight reductions in their median values for some metrics, but they still lag significantly, with higher values in proximity and sparsity indicating ongoing performance issues. The consistent superiority of NICE_sp and NICE_pr across different balancing strategies, including Cost-sensitive approaches, underscores their robustness and reliability.

Tuned models consistently show improved performance compared to their vanilla counterparts across various balancing strategies. Tuned models trained on the original dataset exhibit lower minimality and plausibility values, indicating enhanced performance. In the Undersampling strategy, the gap between tuned and vanilla models narrows slightly, but tuned models still outperform vanilla ones. With Oversampling and SMOTE, the advantage of tuning becomes more pronounced, as tuned models maintain lower values across all metrics, while vanilla models show higher variability. The cost-sensitive approach further highlights the superiority of tuned models, particularly in minimality and validity, where they consistently demonstrate lower values and greater consistency. Overall, tuning leads to better and more reliable performance across different data conditions and metrics.

When focusing on RQ1: “What is the most appropriate method for generating counterfactual explanations after balancing?” the analysis highlighted the consistent superiority of NICE_sp and NICE_pr across various balancing strategies and metrics, demonstrating their robustness and reliability. To answer RQ2: “How do balancing techniques affect the counterfactual explanations of student success prediction models?” we find out that the impact of different data balancing strategies, such as SMOTE and the cost-sensitive approaches, further underscores the adaptability of these methods compared to MOC and WI, which generally underperform. Additionally, tuned models outperform their vanilla counterparts across all conditions, emphasizing the importance of model optimization in achieving optimal performance across diverse balancing strategies.



§ CONCLUSION

This study explored the impact of various balancing techniques on the generation of counterfactual explanations within student success prediction models. Our findings indicate that the choice of balancing strategy significantly influences the quality and characteristics of the counterfactuals generated by different methods, such as Multi-Objective Counterfactual Explanations (MOC), Nearest Instance Counterfactual Explanations (NICE), and WhatIf.


Effectiveness of balancing techniques. The results suggest that certain balancing techniques improve the validity and plausibility of counterfactuals, aligning them more closely with realistic scenarios that educators and students can act upon. For example, balancing methods that mitigate class imbalances not only enhanced the performance of the predictive models but also resulted in more actionable and sparse counterfactual explanations. These findings are consistent with previous research, which emphasizes the importance of balancing in training robust models for educational predictions (Artelt et al., 2021).


Effect analysis of counterfactual generation methods. Among the methods tested, MOC consistently produced counterfactuals that were closer to the original data distribution, showing a higher degree of plausibility and sparsity. This is particularly valuable in educational settings where changes to multiple variables might not be feasible. In contrast, the NICE method, which focuses on proximity, often generated explanations that were more straightforward but potentially less realistic. This trade-off highlights the need to select counterfactual generation methods based on the specific requirements of the educational context.


Implications for Educational Interventions. The insights gained from this study have significant implications for how educational institutions might use counterfactual explanations to inform interventions. By understanding how different balancing techniques affect the characteristics of counterfactuals, educators can better choose models and explanations that are not only accurate but also actionable and interpretable for students and staff.


This study contributes to the growing field of explainable artificial intelligence in education by demonstrating the critical role of balancing techniques in generating effective counterfactual explanations. These findings pave the way for more refined and targeted educational interventions, ultimately contributing to more personalized and supportive learning environments.




§ LIMITATIONS AND FUTURE WORK


While this study provides a comprehensive analysis, some limitations warrant further investigation. The focus on a single dataset and specific counterfactual methods may limit the generalizability of the results. Future research should explore these effects across different datasets and additional counterfactual methods. Moreover, the long-term impact of using such explanations on student outcomes should be evaluated to better understand their practical utility in educational settings.



§ ACKNOWLEDGMENTS

The work in this paper is supported by the German Federal Ministry of Education and Research (BMBF), grant no. 16DHBKI045.




§ SUPPLEMENTAL MATERIALS


The materials for reproducing the experiments performed and the dataset are accessible in the following repository: https://github.com/mcavs/JMEEP_paperhttps://github.com/mcavs/JMEEP_paper



§ REFERENCES


Adnan, M., Habib, A., Ashraf, J., Mussadiq, S., Raza, A. A., Abid, M., & Khan, S. U. (2021). Predicting at-risk students at different percentages of course length for early intervention using machine learning models. IEEE Access, 9, 7519-7539.


Afrin, F., Hamilton, M., & Thevathyan, C. (2023, June). Exploring Counterfactual Explanations for Predicting Student Success. In International Conference on Computational Science (pp. 413-420). Cham: Springer Nature Switzerland.


Arnold K. E. and Pistilli M. D. (2012). Course signals at Purdue: using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12). Association for Computing Machinery, New York, NY, USA, 267–270.


Artelt, A., & Hammer, B. (2019). On the computation of counterfactual explanations: A survey. arXiv preprint arXiv:1911.07749.


Artelt, A., Vaquet, V., Velioglu, R., Hinder, F., Brinkrolf, J., Schilling, M., & Hammer, B. (2021). Evaluating the robustness of counterfactual explanations. In 2021 IEEE Symposium Series on Computational Intelligence (pp. 01-09). IEEE.


Biecek, P., & Burzykowski, T. (2021). Explanatory model analysis: explore, explain, and examine predictive models. Chapman and Hall/CRC.


Brughmans, D., Leyman, P., & Martens, D. (2023). Nice: an algorithm for nearest instance counterfactual explanations. Data mining and knowledge discovery, 1-39.


Carriero, A., Luijken, K., de Hond, A., Moons, K. G., van Calster, B., & van Smeden, M. (2024). The harms of class imbalance corrections for machine learning based prediction models: a simulation study. arXiv preprint arXiv:2404.19494.


Cavus, M., & Biecek, P. (2024). An Experimental Study on the Rashomon Effect of Balancing Methods in Imbalanced Classification. arXiv preprint arXiv 2405.01557.


Cavus, M., Stando, A., & Biecek, P. (2023). Glocal Explanations of Expected Goal Models in Soccer. arXiv preprint arXiv:2308.15559.


Chawla, N. V. (2010). Data mining for imbalanced datasets: An overview. Data mining and knowledge discovery handbook, 875-886.


Dandl, S., Hofheinz, A., Binder, M., Bischl, B., & Casalicchio, G. (2023). Counterfactuals: an R package for counterfactual explanation methods. arXiv preprint arXiv:2304.06569.


Dandl, S., Molnar, C., Binder, M., & Bischl, B. (2020). Multi-objective counterfactual explanations. At the international conference on parallel problem solving from nature, Cham: Springer International Publishing, 448–469.


Drachsler, H. (2018). Trusted Learning Analytics. Synergie, 6, 40-43.


Elyan, E., Moreno-Garcia, C. F., & Jayne, C. (2021). CDSMOTE: class decomposition and synthetic minority class oversampling technique for imbalanced-data classification. Neural computing and applications, 33, 2839-2851.


Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35, 507–520.


Guidotti, R. (2022). Counterfactual explanations and how to find them: literature review and benchmarking. Data Mining and Knowledge Discovery, 1-55.


Gunonu, S., Altun, G., & Cavus, M. (2024). Explainable bank failure prediction models: Counterfactual explanations to reduce the failure risk. arXiv preprint arXiv:2407.11089.


Gu, Q., Tian, J., Li, X., & Jiang, S. (2022). A novel Random Forest integrated model for imbalanced data classification problem. Knowledge-Based Systems, 250, 109050.


Hoel, T., Griffiths, D., & Chen, W. (2017). The influence of data protection and privacy frameworks on the design of learning analytics systems. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 243-252).


Holzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022). Explainable AI methods-a brief overview. In International workshop on extending explainable AI beyond deep models and classifiers (pp. 13-38). Springer, Cham.


Junior, J. D. S. F., & Pisani, P. H. (2022). Performance and model complexity on imbalanced datasets using resampling and cost-sensitive algorithms. In Fourth International Workshop on Learning with Imbalanced Domains: Theory and Applications (pp. 83-97). PMLR.


Karimi, A. H., Barthe, G., Balle, B., & Valera, I. (2020). Model-agnostic counterfactual explanations for consequential decisions. At the International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, 895–905.


Kuzilek, J., Hlosta, M., & Zdrahal, Z. (2017). Open university learning analytics dataset. Scientific data, 4(1), 1–8.


Kuzilek, J., Hlosta, M., Herrmannova, D., Zdrahal, Z., Vaclavek, J., & Wolff, A. (2015). OU Analyse: analysing at-risk students at The Open University. Learning Analytics Review, 1-16.


Liu, J. (2022). Importance-SMOTE: a synthetic minority oversampling method for noisy imbalanced data. Soft Computing, 26(3), 1141-1163.


Molnar, C. (2020). Interpretable machine learning. Lulu.com.


Papamitsiou Z., and Economides A. (2014). Learning Analytics and Educational Data Mining in Practice: A Systematic Literature Review of Empirical Evidence. Journal of Educational Technology & Society, 17(4), 49–64.


Pinto, J. D., & Paquette, L. (2024). Towards a Unified Framework for Evaluating Explanations. arXiv preprint arXiv:2405.14016.


Siemens, G. and Baker, R. (2012). Learning analytics and educational data mining: towards communication and collaboration. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12). Association for Computing Machinery, New York, NY, USA, 252–254.


Smith, B. I., Chimedza, C., & Bührmann, J. H. (2022). Individualized help for at-risk students using model-agnostic and counterfactual explanations. Education and Information Technologies, 1-20.


Stando, A., Cavus, M., & Biecek, P. (2024, June). The effect of balancing methods on model behavior in imbalanced classification problems. In Fifth International Workshop on Learning with Imbalanced Domains: Theory and Applications (pp. 16-30). PMLR.


Tao, X., Li, Q., Guo, W., Ren, C., Li, C., Liu, R., & Zou, J. (2019). Self-adaptive cost weights-based support vector machine cost-sensitive ensemble for imbalanced data classification. Information Sciences, 487, 31-56.


Tsiakmaki, M., & Ragos, O. (2021). A Case Study of Interpretable Counterfactual Explanations for the Task of Predicting Student Academic Performance. In 2021 25th International Conference on Circuits, Systems, Communications and Computers (CSCC) (pp. 120-125). IEEE.


Wachter, S., Mittelstadt, B., & Russell, C. (2017). Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harv. JL & Tech., 31, 841–872.


Waheed, H., Hassan, S. U., Aljohani, N. R., Hardman, J., Alelyani, S., & Nawaz, R. (2020). Predicting the academic performance of students from VLE big data using deep learning models. Computers in Human Behavior, 104, 106189.


Warren, G., Keane, M. T., Gueret, C., & Delaney, E. (2023). Explaining groups of instances counterfactually for XAI: a use case, algorithm and user study for group-counterfactuals. arXiv preprint arXiv:2303.09297.


Wexler, J., Pushkarna, M., Bolukbasi, T., Wattenberg, M., Viégas, F., & Wilson, J. (2019). The what-if tool: Interactive probing of machine learning models. IEEE transactions on visualization and computer graphics, 26(1), 56-65.


Yin, J., Gan, C., Zhao, K., Lin, X., Quan, Z., & Wang, Z. J. (2020). A novel model for imbalanced data classification. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 6680-6687).


Zhang, H., Dong, J., Lv, C., Lin, Y., & Bai, J. (2023). Visual analytics of potential dropout behavior patterns in online learning based on counterfactual explanation. Journal of Visualization, 26(3), 723-741.


Zong, W., Huang, G. B., & Chen, Y. (2013). Weighted extreme learning machine for imbalance learning. Neurocomputing, 101, 229-242.




§ APPENDIX








 Paper Title: Alpha-VI DeepONet: A prior-robust variational Bayesian approach for enhancing DeepONets with uncertainty quantification.


Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
    

Hans-Alexander Kruse 0009-0006-5568-7841,
Tim Puhlfürß 0000-0001-8421-8071,
Walid Maalej 0000-0002-6899-4393


Universität Hamburg
  
Hamburg, Germany

hans-alexander.kruse@studium.uni-hamburg.de,
tim.puhlfuerss@uni-hamburg.de,
walid.maalej@uni-hamburg.de

    August 5, 2024
======================================================================================================================================================================================================================================================================



        
§ ABSTRACT 
We introduce a novel deep operator network (DeepONet) framework that incorporates generalised variational inference (GVI) using Rényi's α-divergence to learn complex operators while quantifying uncertainty. By incorporating Bayesian neural networks as the building blocks for the branch and trunk networks, our framework endows DeepONet with uncertainty quantification. The use of Rényi's α-divergence, instead of the Kullback-Leibler divergence (KLD), commonly used in standard variational inference, mitigates issues related to prior misspecification that are prevalent in Variational Bayesian DeepONets. This approach offers enhanced flexibility and robustness.

We demonstrate that modifying the variational objective function yields superior results in terms of minimising the mean squared error and improving the negative log-likelihood on the test set. Our framework's efficacy is validated across various mechanical systems, where it outperforms both deterministic and standard KLD-based VI DeepONets in predictive accuracy and uncertainty quantification. The hyperparameter α, which controls the degree of robustness, can be tuned to optimise performance for specific problems.

We apply this approach to a range of mechanics problems, including gravity pendulum, advection-diffusion, and diffusion-reaction systems. Our findings underscore the potential of α-VI DeepONet to advance the field of data-driven operator learning and its applications in engineering and scientific domains.









§ INTRODUCTION
 

As scientific machine learning methodologies take centre stage across diverse disciplines in science and engineering, there is an increased interest in adopting data-driven methods to analyse, emulate, and optimise complex physical systems. The behaviour of such systems is often described by laws expressed as systems of ordinary differential equations (ODEs) and partial differential equations (PDEs) <cit.>. A classical task then involves the use of analytical or computational tools to solve such equations across a range of scenarios, e.g., different domain geometries, input parameters, and initial and boundary conditions (IBCs). Solving these so-called parametric PDE problems requires learning the solution operator that maps variable input entities to the corresponding solution of the underlying PDE system. Tackling this task using traditional tools (e.g., finite element methods <cit.>) bears a formidable cost, as independent simulations need to be performed for every different domain geometry, input parameters, or IBCs. Driven by this challenge, a growing field of neural operators for solving parametric PDEs has come forth recently. 

Learning to solve PDEs is closely related to operator learning. Instead of learning to solve a specific PDE, it can be advantageous to learn the operator that maps a functional parameter of the PDE (such as initial values, boundary conditions, force fields, or material parameters) to the solution associated with the given parameter. Neural operators exemplify this approach by learning to solve entire classes of PDEs simultaneously. These specially designed deep neural networks can represent the solution map of parametric PDEs in a discretisation-invariant manner, meaning the model can be queried at any arbitrary output location. Unlike neural networks that map between two finite-dimensional vector spaces, neural operators can represent mappings between spaces with infinite dimensions <cit.>. Popular operator learning frameworks include deep operator networks (DeepONets) <cit.> and Fourier neural operator (FNO) <cit.>. This article focuses on DeepONet, which has been applied to a wide range of problems, including solid and fluid mechanics <cit.>, reliability analysis <cit.>, heat transfer <cit.>, and fracture mechanics <cit.>. De et al. <cit.> utilised DeepONets for modelling uncertain and partially unknown systems. Cai et al. <cit.> applied DeepONets to model field variables across multiple scales in multi-physics problems while He et al. <cit.> employed DeepONets to predict full-field, nonlinear elastic–plastic stress responses in complex geometries. Other notable applications of DeepONets include weather forecasting <cit.>, reduced-order modelling <cit.>, finance <cit.>, and others <cit.>.

While DeepONets excel at solving differential equations (DEs), a crucial aspect often overlooked is the uncertainty quantification of the predictions.  Traditional DeepONet architectures have been found to produce over-confident predictions, implying they are poorly calibrated <cit.>. This means they might underestimate the true range of possible outcomes, potentially leading to unreliable results. Additionally, they do not quantify the inherent uncertainties associated with their predictions.
In engineering applications like aerodynamics or structural analysis, uncertainty quantification is paramount. By quantifying uncertainties in predictions related to phenomena such as stress distribution, engineers can make informed decisions about the risk and reliability of their designs. 

Several attempts have been made to incorporate uncertainty quantification within the DeepONet architecture. However, existing approaches face some limitations. Lin et al. <cit.> proposed a Bayesian DeepONet based on replica-exchange stochastic gradient Langevin diffusion <cit.> that considers the standard deviation of the output to be known, which may not always be realistic. Additionally, training the replicas in this approach is computationally expensive. Yang et al. <cit.> used randomised priors and trained an ensemble of deterministic models, which can also be computationally demanding depending upon the size of the ensemble. While the method of ensembles offers some degree of uncertainty estimation through multiple model predictions, it does not adhere to the more principled framework that is provided by Bayes' rule and hence cannot be considered truly Bayesian. 

Variational inference (VI)-based Bayesian methods offer a promising framework for incorporating uncertainty quantification into deep learning models. This approach has given rise to Bayesian neural networks (BNNs) <cit.>. In BNNs, the parameters of the neural network are treated as random variables, which allow the uncertainty in the parameters to propagate through the network, ultimately reflecting the uncertainty in the model predictions. The variational inference approach is particularly advantageous for neural networks with a large number of parameters, as traditional Markov chain Monte Carlo (MCMC)-based methods become computationally expensive to use. Variational inference, on the other hand, provides a scalable alternative for approximating the posterior distribution of the parameters in BNN. 

It seems natural to extend the idea of uncertainty quantification in BNNs to that of a Bayesian DeepONet. Building on this idea, Garg et al. <cit.> proposed VB-DeepONet that uses BNNs as building blocks for the DeepONet architecture. However, VB-DeepONet inherits certain challenges associated with BNNs. Their approach employs a fully factorised standard Gaussian distribution as the prior over the DeepONet model parameters. Such a prior might not accurately reflect the true distribution of these parameters, leading to a sub-optimal approximation of the posterior distribution. As indicated in Li et al. <cit.>, Knoblauch et al. <cit.>, and Wenzel et al. <cit.>, such a misspecified prior can hinder the overall effectiveness of variational inference, where the approximated posterior concentrates around a single point leading to overconfident predictions and underestimated uncertainty <cit.>. This hinders the model's ability to effectively quantify uncertainties. 

We propose to address these limitations in VB-DeepONet by introducing prior-robust variational inference for DeepONets. Standard VI relies on the Kullback-Leibler divergence (KLD), which is sensitive to prior selection, especially when the number of parameters is large. Alternative divergence measures exist that offer robustness to misspecified priors. Knoblauch et al. <cit.> introduced the Generalised Variational Inference (GVI) framework that allows for defining divergence metrics beyond KLD in the context of BNNs. GVI incorporates a hyperparameter that controls the degree of robustness to prior misspecification. This leads to posteriors that are less influenced by priors which deviate significantly from the observed data <cit.>. 

In this paper, we extend the GVI framework to DeepONets to achieve greater robustness to prior misspecification. In particular, we propose the use of Rényi's α-divergence <cit.> as a robust alternative to KLD within the VI framework. This novel operator learning approach using GVI leads to an improved approximated posterior distribution, demonstrably enhancing both prediction accuracy and the quality of uncertainty estimates as evidenced by log-likelihood values on unseen test data.

The rest of the article will provide background on deterministic DeepONets, followed by details of the proposed variational Bayesian framework and four numerical examples highlighting the efficacy of our approach. We conclude by discussing the results of different hyperparameter settings and recovering standard KLD-VI results as a special case.



§ BACKGROUND ON DEEPONETS

Operator networks were first introduced by Chen and Chen <cit.>, where they considered shallow networks with a single hidden layer. Subsequently, this concept was significantly developed with deep neural network architectures, leading to the creation of DeepONets <cit.>. To define operators, it is useful to consider two different classes of functions residing in two separate Banach spaces, 𝒜 and 𝒮, respectively. One class of functions, a(y) ∈𝒜, with domain in Ω_Y ∈ℝ^D, while the other class of functions, s(y) ∈𝒮, also having domain in Ω_Y. An operator 𝒢 can be defined as a mapping between these two function spaces, 𝒢: 𝒜→𝒮 such that s(y) = 𝒢(a(y)).

DeepONet is a specialised deep neural network designed to approximate the operator 𝒢. In the following sections, we review the DeepONet and its variational Bayesian extension (VB-DeepONet), and discuss their limitations in the context of uncertainty quantification.



 §.§ Deterministic DeepONets

Consider a governing equation for a physical phenomenon within a spatial domain Ω over a time period (0, T]:
 
    
    ℱ(s)(x, t)    = u(x,t),  x, t∈Ω× (0,T], 
    
            ℬ(s)(x,t)    = s_b(x,t),  x, t∈∂Ω× (0,T],
    
            ℐ(s)(x, 0)    = s_0(x),   x ∈Ω.


Here, ℱ is a nonlinear differential operator, x is the location in space, t is the time; s denotes the solution of the differential equation, and u denotes the external source function. Furthermore, ℬ(s)(x,t) = s_b(x) and ℐ(s)(x, 0) = s_0(x) define the boundary and initial conditions, respectively. The true solution operator is denoted by 𝒢(u)(x,t). To keep the notations compact, we denote the tuple (x,t) by y. 

DeepONets <cit.> are designed to learn an approximate operator 𝒢_θ parameterised by a deep neural network with parameter vector θ. 

The architecture for the DeepONet consists of two main sub-networks: a branch neural network and a trunk neural network (see <ref>). 

    
  * Branch Network: This is a neural network, with trainable parameters θ_B, that maps from space ℝ^M →ℝ^P. It takes as input M number of sensor measurements of the function a(y) ∈𝒜, represented by the M-dimensional vector a = ay_1, …, ay_M (where y_k x_r, t_v denotes some location x_r and some time instant t_v), and it outputs a vector of weights b_1(a), …, b_P(a).

    
  * Trunk Network: This is a neural network, with trainable parameters, θ_T, that takes as input y∈Ω_Y and outputs a P-dimensional basis vector ψ_1(y), …, ψ_P(y).
  
The final output of the DeepONet 𝒢_θ: ℝ^M ×ℝ^D →ℝ is obtained by taking a dot product of the outputs from the branch and trunk networks, which approximates the value of s(y):

    s(y) ≈ s(y; θ) =𝒢_θ(a)(y) ≈ b_0 + ∑_p=1^P b_pa; θ_B ψ_p y; θ_T.

The trainable parameters of the DeepONet θ consist of the bias b_0 and the combined parameters of the branch and trunk networks, i.e., θ = b_0, θ_B, θ_T.

Training a DeepONet involves supervised learning using pairs of the representative function a(y) and the corresponding solution s(y). A representative training set looks like:

    𝒟 = a^(i), y_k, s^(i)y_k: 1 ≤ i ≤ N_1, 1 ≤ k ≤ N_2

where a^(i) is the i^th training example of the  M-dimensional vector a obtained at M locations of y, and s^(i) is the corresponding solution function obtained at N_2 locations of y from solving the PDE (<ref>). Note that the N_2 locations may differ from M sensor locations where the representative input functions a are measured. The M sensor locations can be either random or uniformly spaced and remain fixed during training. 

The parameter vector θ is estimated by solving the optimisation problem over the training dataset 𝒟:

    θ̂   = _θ1/N_1 N_2∑_i=1^N_1∑_k=1^N_2𝒢_θa^(i)(y_k)  -  s^(i)y_k^2 
        = 1/N_1 N_2∑_i=1^N_1∑_k=1^N_2b_0 + ∑_p=1^P b_p a^(i); θ_B ψ_p y_k; θ_T  -  s^(i)y_k^2.

The optimisation problem can be solved using stochastic gradient descent <cit.> to obtain the optimal parameter values for the DeepONet. This approach is referred to as D-DeepONet in the rest of the article, where “D” stands for deterministic. 

A key limitation of D-DeepONet is its inability to quantify uncertainty in the predictions. As it produces a point estimate, it fails to account for potential measurement errors or uncertainties in the model parameters. 



 §.§ Variational Bayesian DeepONets (VB-DeepONet)

VB-DeepONet <cit.> addresses the limitations of D-DeepONet by incorporating layers of Bayesian neural networks within both the branch and trunk networks of the DeepONet architecture. This integration allows the model to capture prediction uncertainties, enhancing its robustness against overfitting.

However, VB-DeepONet faces challenges related to posterior inference using standard variational inference. A common practice in VB-DeepONet is to employ a fully factorised standard Gaussian distribution as the prior for the DeepONet model parameter vector θ. This prior assumes pairwise independence among model parameters, implying that each parameter distribution is unimodal. Although intended to be non-informative, this choice of prior can still exert significant influence on the posterior distribution given the number of parameters of deep neural networks can be quite large. Hence, this prior might not accurately reflect the true distribution of the parameters, potentially leading to sub-optimal uncertainty quantification. Moreover, standard KLD-VI with mean-field Gaussian variational families often exhibit “mode-seeking” behaviour, where the approximated posterior tends to concentrate around a single point. Additionally, if the model parameters exhibit high correlations, these VI approximations can produce overly confident predictions, failing to capture the true range of uncertainty. These issues can significantly hinder the VB-DeepONet's ability to effectively quantify uncertainties, a crucial aspect for reliable engineering applications.

The next section discusses the proposed α-VI DeepONet approach that uses an alternative divergence measure, the Rényi's-α divergence, to address the limitations of VI in VB-DeepONet and achieve robust uncertainty quantification.



§ PROPOSED Α-VI DEEPONET
 
We propose using Bayesian neural networks in both the trunk and branch networks (shown in <ref>) instead of their deterministic counterparts, similar to VB-DeepONet. However, a fully factorised normal distribution is hardly ideal as a prior for BNNs. Constructing alternative prior beliefs that accurately reflect our judgments could be computationally prohibitive too. Nonetheless, by employing an alternative divergence metric (D), more robust posterior beliefs can be produced with an imperfect prior. Therefore, in the proposed approach, we seek to use a divergence metric, different from KLD, that results in posteriors more robust to poorly specified priors and provides reliable uncertainty quantification. 

In the deterministic DeepONet, the parameter vector θ of the deep neural network (from <ref>) is a point estimate. However, as noted in <cit.>, multiple different parameter settings can perform equally well, implying that the set of values that each parameter can take may be captured by a distribution over those plausible values. A Bayesian paradigm allows us to place distributions on the model parameters, representing uncertainty about their exact values. Using Bayes' rule, we update these parameter distributions with the data (𝒟 from <ref>) as follows:

    p(θ | 𝒟) = p(𝒟 | θ)  p(θ)/p(𝒟),

where p(𝒟 | θ) is the likelihood and p(θ) is the prior distribution over the model parameter vector. The denominator, p(𝒟) = ∫ p(𝒟| θ)  p(θ)  dθ, is often an intractable constant known as marginal likelihood. The posterior distribution,
p(θ | 𝒟 ), allows us to perform inference on unseen data (𝒟^*). For prediction, we use this posterior to compute the predictive distribution:

    p(𝒟^* | 𝒟, θ) = ∫ p(𝒟^* | θ) p(θ | 𝒟) dθ.

We now examine these individual components representing our modelling choices that influence the learned posterior distribution. 



 §.§ Fully Factorised Prior

In BNNs, we often default to a fully factorised standard normal distribution as a prior over the model parameters. This implies that each parameter component is assumed to be independent and drawn from a Gaussian distribution (denoted by 𝒩) with zero mean and unit variance. Mathematically, this can be expressed as:

    p(θ) = ∏_l=1^L p(θ_l) = ∏_l=1^L𝒩(θ_l; 0,1) = 𝒩 (θ; , _L),

where L denotes the dimensionality of the model parameter vector θ. It is important to acknowledge that this fully factorised prior is a “misspecified prior” as it does not capture the interdependencies that exist between the individual components θ_l of the parameter vector θ. 
To alleviate the negative influence of this “misspecified prior,” we use a different divergence metric in <ref>.






 §.§ Likelihood Function

The likelihood function describes the probability of observing the solutions given the model parameters (and deterministic input functions). We write the likelihood function considering that each example i is independent and identically distributed, given the input functions and parameters, as follows:

    p(𝒟 | θ) = ∏_i=1^N_1 ps^(i)(y_1), …, s^(i)(y_N_2) |a^(i), θ,

where N_1 represents the total number of examples of pairs a and s.
We further assume that the outputs s^(i)(y_k) at different locations are also statistically independent of each other k = 1,…, N_2. Consequently, the joint distribution of the solution for the i^th example across all N_2 locations factorises into a product of independent marginal distributions for each location:

    p(𝒟 | θ) = ∏_i=1^N_1∏_k=1^N_2 ps^(i)(y_k) |a^(i), θ.

Furthermore, we assume that the individual output distributions ps^(i)(y_k) |a^(i), θ at each location follow a Gaussian distribution. The mean and standard deviation of these distributions depend on the input function a, location y, and model parameters θ. This dependence reflects the influence of the input function and location on the solution value, mediated by the model parameters. Mathematically, this is expressed as:

    p(𝒟 | θ)    = ∏_i=1^N_1∏_k=1^N_2𝒩s^(i)(y_k);  μ_ka^(i), y_k, θ, σ_ka^(i), y_k, θ
    
           = ∏_i=1^N_1∏_k=1^N_2𝒩s_k^(i);  μ_k(θ) , σ_k (θ),

where μ_k represents the mean and σ_k represents the standard deviation. 
For notational convenience, we use the shorthand notation s_k^(i)≡ s^(i)(y_k) to represent the solution value at location y_k for the i^th solution data. Additionally, we only highlight the dependence of the mean and standard deviation on the model parameters θ for brevity. The other variables, a^(i) and y_k, are treated as deterministic (and known) quantities in this context.

A sample prediction from this network represents a stochastic pass through the random BNN parameters, followed by a sample from the Gaussian likelihood. A number of these samples are taken to quantify the total uncertainty and construct the confidence intervals.



 §.§ Posterior Approximation via Variational Inference

Within a fully Bayesian framework, our goal is to determine the posterior distribution of the model parameters given the observed data. This posterior distribution, denoted as p(θ | 𝒟), represents the probability of various parameter configurations (θ) after observing the data 𝒟.
Unfortunately, due to the intractable normalising constant as explained in <ref>, directly computing the posterior distribution analytically is not feasible. 

Generally, there are two approaches to solve this class of problems - sampling-based approaches, e.g., Markov chain Monte Carlo and approximation-based optimisation approaches, e.g., variational inference, expectation propagation (EP). 
Sampling-based approaches, exemplified by MCMC-techniques, are often regarded as the gold standard for Bayesian inference <cit.>. They involve constructing a Markov chain that gradually explores the parameter space, eventually converging to a distribution proportional to the posterior. However, for problems with high-dimensional parameter spaces (often exceeding 20,000 parameters in our case), these approaches can become computationally impractical <cit.>. We note here that the proposed approach puts no limit on the complexity of the model, that is the number of parameters. Hence, we will focus on the approximation-based approaches, where a simpler variational distribution is introduced to approximate the posterior, and the key lies in optimising the parameters of the variational distribution to minimise the Kullback-Liebler divergence between itself and the intractable posterior. This optimisation process allows us to efficiently train complex models despite the high dimensionality of the parameter space.



  §.§.§ Variational Inference with KLD

VI <cit.> introduces a surrogate posterior family of distributions, denoted by Q, that is tractable to sample from. The goal is to find a member of Q distribution that closely resembles the true posterior in terms of the shape. VI accomplishes this by minimising the KLD between the approximate posterior q(θ;η) ∈ Q and the true posterior p(θ|𝒟). The KLD, denoted as D_KL[ q || p], quantifies the difference between the q distribution and the reference p distribution weighted by the q distribution, and is mathematically defined as:

    D_KLq(θ) ||  p(θ) = ∫ q(θ)logq(θ)/p(θ)dθ = 𝔼_q(θ)[ log(q(θ)) - log(p(θ)) ].
 
Using the KLD, the minimisation problem in VI is formulated as finding the best parameters η^* of the variational distribution that brings it closer to the true posterior:

    q(θ; η^*) = _q(θ;η) ∈ Q D_KL q(θ; η) || p(θ|𝒟) ,

where η are the parameters of the variational distribution, q(θ; η). Here, the KLD measures the information lost by approximating the true posterior with the variational distribution q. However, directly computing this integral is impractical because the true posterior is unknown. VI addresses this by leveraging the connection between the KLD and the evidence lower bound (). 
VI rewrites the KLD using the following identity:

    D_KL q(θ; η) || p(θ | 𝒟)    = 𝔼_q(θ;η)log(q(θ; η)) - logp(𝒟 | θ) p(θ)/p(𝒟)
    
        =log p(𝒟) - 𝔼_q(θ;η)log(p(𝒟 | θ)) - D_KLq(θ; η) || p(θ)_
    
       = log p(𝒟) + ℒ(q) ,

where the expectation in the  (in <ref>) is taken with respect to the approximate posterior q(θ; η). Note that p(𝒟) is the model evidence, which is a constant term for a given dataset and also constant with respect to q(θ;η) and can therefore be ignored during the minimisation process <ref>. 
From <ref>, we see that maximising the  is equivalent to minimising the KLD (as KLD is non-negative). So, VI aims to maximise the  which turns out to be the minimisation of the variational free energy ℒ(q) (see <ref>.
This translates to finding an approximate posterior q(θ;η) that balances two key factors (common in Bayesian inference): 

    
  * Data fit: The first term, 𝔼_q(θ;η) [log(p(𝒟 | θ))], represents the expected log-likelihood of the data under the approximate posterior. A high value for this term indicates a good fit between the model and the data.

    
  * Prior regularisation: The second term, D_KL[q(θ; η) || p(θ)], is the KL divergence between the approximate posterior and the prior distribution, p(θ). It acts as a regulariser, penalising overly complex posterior distributions that deviate significantly from the prior.


The expressions of these terms highlight the impact of our prior modelling choices on the optimisation problem. A poorly specified prior – independent standard Gaussian distribution – over parameters, as is the case with BNNs, can degrade the quality of the posterior approximation. However, it remains the most expedient choice for BNNs from the perspective of computational expense. As suggested by Knoblauch et al. <cit.>, we can opt for a difference divergence metric instead of KLD to alleviate the negative influence of misspecified prior. For example, KLD is known to exhibit a mode-seeking behaviour <cit.>. By selecting a different divergence metric one can improve the quality of the posterior distribution, both in terms of fitting the data and providing better uncertainty estimates <cit.>.

In this context, we examine the effect of using the flexible Rényi's-α divergence on the posterior predictive performance. We modify the variational free energy objective by replacing the KLD with Rényi's-α divergence:

    ℒ(q) = -𝔼_q(θ;η) [log p(𝒟| θ)] + D_AR^(α)[q(θ;η)||p(θ)].

To compute the expected value of the log-likelihood term, we use Monte-Carlo integration by drawing samples from the variational distribution q given the parameters η. The likelihood function is computed using <ref> and we approximate the first term in <ref>, as follows:

    𝔼_q(θ;η)[log(p(𝒟|θ))]    ≈1/N_q∑_c=1^N_qlog p(𝒟|θ^(c)), θ^(c)∼ q(θ;η) 
    
            = 1/N_q∑_c=1^N_qlog∏_i=1^N_1∏_k=1^N_2𝒩 s_k^(i); μ_kθ^(c) , σ_kθ^(c), θ^(c)∼ q(θ;η) 
    
            = 1/N_q∑_c=1^N_q∑_i=1^N_1∑_k=1^N_2log𝒩s^(i)_k ; μ_kθ^(c)σ_kθ^(c), θ^(c)∼ q(θ;η).

where N_q is the number of Monte Carlo samples, and θ^(c) is the c^th sample drawn from q(θ;η). In practice, we found that using 25 Monte Carlo samples (N_q = 25) is sufficient for accurate approximation, aligning with findings in 
<cit.> which suggest that even a single sample can yield satisfactory results.

In the next section, we detail Rényi's α-divergence as an alternative to KLD, which represents the second term of our objective function.




 §.§ Rényi's-α divergence
 
Rényi's α-divergence, introduced by Rényi <cit.>, offers a robust alternative to KLD. It is characterised by a hyperparameter α, which controls the degree of robustness, and it converges to KLD as a limiting case when α approaches 1. We denote this divergence by D_AR^(α) and follow the parameterisation provided in <cit.>:

    D_AR^(α)[q(θ)||p(θ)] = 1/α(α-1)log( 𝔼_q(θ)[ ( p(θ)/q(θ))^1-α] ).

This divergence can be computed in closed form when both q and p are normal distributions. However, in general, we approximate this divergence across various distributions by Monte-Carlo sampling. In the limit as α approaches 1, D_AR^(α) simplifies to D_KL:

    lim_α→ 1 D_AR^(α)[·] = D_KL[·].


In the context of our problem, D_AR^(α) represents the second term of the objective function in <ref>, which is approximated as follows:

    D_AR^(α)[q(θ;η)||p(θ)] ≈1/α(α-1)log(1/N_q∑_c=1^N_q[ ( p(θ^(c)))/q(θ^(c);η))^1-α] ),

where θ^(c) denotes the c^th Monte Carlo sample from q(θ;η). 



In <ref>, we plot divergence values D_AR^(α) between q(θ) and p(θ), assuming both are normal distributions with p(θ) as a standard normal distribution. The divergence value decreases as α increases, with KLD as the case when α = 1. 

Note that the objective function in <ref> contains a data-loss term and a complexity term. The behaviour of the optimisation problem differs significantly depending on whether α is greater than or less than 1. For α > 1, D_AR^(α) exhibits a mode-seeking behaviour, causing the posteriors to become more concentrated while being penalised less for deviating from the prior distribution. In this regime, the objective function places more emphasis on the data-loss term, and the complexity term – which measures the divergence between the prior and the variational posterior – receives less weight as α increases. In contrast, for α < 1, D_AR^(α) encourages a mass-covering behaviour, akin to expectation propagation. As α decreases below 1, the values of D_AR^(α) increase, resulting in more weight being assigned to the complexity term and less to the data-loss term. It is important to note, however, that a larger regularisation term does not necessarily imply that a misspecified prior will dominate the inference, as demonstrated in the experiments by <cit.>.

Given that the true posterior is unknown, it is uncertain whether the mode-seeking or mass-covering behaviour is preferable, making the choice problem-dependent. For scenarios where the true posterior is multimodal, a divergence that produces wider parameter variances (i.e., mass-covering behavior) may be beneficial for achieving more robust predictive estimates. In particular, we aim to set D_AR^(α) such that the posterior is robust to priors that strongly contradict observed data, and provides reliable uncertainty quantification. We rely on validation sets and adjust α as a hyperparameter to minimise metrics such as the normalised mean squared error or negative log-likelihood.



 §.§ GVI Posterior Approximation

We choose the approximate posterior to be the mean-field normal (MFN) variational family given by:

    Q_MFN = 
        ∏_l=1^L𝒩(θ_l; μ_l^q, σ_l^q).


We find the q(θ;η) within the Q_MFN family that minimises the objective function. This class of approximate variational family is commonly used for BNNs. Once the variational family is decided, the optimisation reduces to finding the parameters, η =[μ^q,σ^q], of the variational family that minimises the loss, ℒ(q) (<ref>). Thus the optimisation problem tries to find:

    η^* = _η{ℒ(q) }.




 §.§ Training

To optimise the variational free energy objective function (acting as the loss function), as defined in <ref>, we employ the Bayes-by-Backprop algorithm <cit.>. This algorithm approximates the expectation within the objective function through Monte Carlo sampling. Specifically, we use N_q = 25 samples for this approximation. 

A key technique for efficient training is the reparameterisation trick <cit.>. This allows us to automatic differentiation to compute the unbiased gradients of the loss function with respect to the parameters and hence enables the use of gradient-based optimisers commonly employed in deep learning.  In our implementation, we utilise the Adam optimiser <cit.> with its default settings. The entire training procedure is carried out in TensorFlow Probability <cit.>.

For Gaussian distributions, the reparameterisation trick involves sampling a standard Gaussian random variable ϵ, and then scaling it by the standard deviation parameter, σ^q, and shifting it by the mean, μ^q. To ensure the positivity of the standard deviation, we employ a soft-plus activation function. The resulting parameterisation can be expressed mathematically as follows:

    θ =μ^q + log (1 + exp(σ^q)) ⊙ϵ

where ⊙ denotes element-wise multiplication. 

Combining the expected negative log-likelihood (<ref>) and the Rényi's α divergence (<ref>), we obtain the final variational free energy objective:

    ℒ(q) =
        - 1/N_q∑_c=1^N_q∑_i=1^N_1∑_k=1^N_2log𝒩s^(i)_k ; μ_kθ^(c) , σ_kθ^(c)  + D_AR^(α)qθ^(c);η||pθ^(c),

where θ^(c)∼ qθ^(c);η.

To optimise the variational parameters, η =[μ^q,σ^q]. we compute the gradients using the Bayes-by-Backprop equations:

    Δ_μ^q = ∂ℒ(q)/∂θ + ∂ℒ(q)/∂μ^q


    Δ_σ^q = ∂ℒ(q)/∂θϵ/1 + exp(-σ^q) + ∂ℒ(q)/∂σ^q,

where the ∂ℒ(q)/∂θ term is obtained through standard backpropagation. The overall training procedure is summarised in <ref>.






§ NUMERICAL STUDIES

To assess the performance of our proposed approach, we conduct a numerical study involving four prototypical problems: two ODEs – the antiderivative operator and the gravity pendulum – and two PDEs – diffusion-reaction and advection-diffusion equations. These problems represent a diverse spectrum of complexities, allowing for a rigorous evaluation of our model's capabilities. 

We investigate the influence of the hyperparameter α on posterior predictive performance while maintaining consistent DeepONet architecture and optimisation parameter settings across all experiments. Detailed architectural specifications for each problem are tabulated in <ref>. To optimise variational parameters, we employ a full-batch Adam optimiser with standard hyperparameter settings: maximum 10000 epochs, learning rate = 0.001, β_1 = 0.9, and β_2 = 0.999. We execute ten independent optimisation runs with varying initialisations for each problem, discarding non-convergent cases identified by oscillatory loss behaviour.

The average values and standard deviations of the converged runs are reported for eleven different values of α  in <ref> and <ref>. We evaluate the model test performance using two complementary metrics: the normalised mean squared error (NMSE) and the negative log-likelihood (NLL). NMSE provides a straightforward measure of prediction error by averaging the squared differences between observed and predicted mean values of responses, normalised by the norm of the observed values. NLL, on the other hand, assesses how well the predicted probability distribution aligns with actual outcomes, accounting for both the prediction mean and its uncertainty.





 §.§ Problem 1: Antiderivative Operator

We commence our numerical investigation with the fundamental problem of the antiderivative operator over the domain x ∈ (0,1]:

    ds/dx = u(x),

subject to the initial condition, s(0) = 0. In this case, the independent variable is exclusively x, rendering y = x. Our objective is to approximate the solution s(x), driven by the input function u(x): 

    s(x) = s(0) + ∫_0^x u(τ)dτ,  x ∈[0,1],

Here, the input function a(y) from the general formulation is equivalent to the source term u(x).

To construct the training dataset, we sample input functions u(x) from a zero-mean Gaussian random field (GRF) characterised by:


    
    u(x)    ∼𝒢𝒫(0, κ(x_1,x_2)) 
    κ(x_1,x_2)    = exp( -x_1 - x_2^2/2 ℓ^2),


where κ is the radial basis function (RBF) covariance kernel with length-scale ℓ. As outlined in <ref>, we generate a training set comprising N_1 = 3000 input functions u(x) discretised at M=100 equidistant points within the domain (implying a^(i)∈ℝ^100, i=1,…, N_1). For each input vector a^(i), we compute the corresponding solution at N_2 = 20 randomly selected points, i.e., s^(i)(x_k),  k=1,…, N_2. Notably,  once the operator is learned, solutions can be evaluated at any arbitrary location within the domain. To assess generalisation performance, we employ a test set of 10000 input functions, evaluating predicted solutions s(x) for each of these test input functions at 100 equidistant locations in the domain. 

<ref> showcases the model's predictive capabilities by visualising solutions for two representative test cases chosen to highlight its performance. This ensemble of points, representing the solution queried at the 100 equidistant locations, provides a visual approximation of the mean solution and the associated uncertainty. A comparative analysis with predictions from D-DeepONet and ground truth solutions for these representative examples further underscores the efficacy of using a different value of α than 1. For a comprehensive assessment, we present a comparison of performance using NMSE and NLL metrics averaged over all 10000 test cases for eleven different α values ranging from 0.25 to 3.00. Tables <ref> and <ref> present these results. In both metrics, α = 1.25 achieved the lowest NMSE and NLL. Notably, the NMSE is reduced by more than 50% compared to the standard KLD-VI (with α=1.00), indicating a substantial improvement in mean predictions.





 §.§ Problem 2: Gravity Pendulum under External Forcing

We next consider the nonlinear dynamics of a gravity pendulum subjected to an external force.  The system is governed by the following nonlinear ODE:

    d^2ϕ/dt^2 = -k sinϕ + u(t),

where ϕ represents the angular displacement, k is a constant determined by gravitational acceleration and pendulum length, and u(t) is the time-dependent external forcing function. The time domain for simulation is set to t ∈ (0,1]. Transforming this second-order ODE into a state-space form yields:

    ds/dt =
    [ ds_1/dt; ds_2/dt;         ] =
    [              s_2; -k sins_1 + u(t);                  ],

with initial conditions s(0) = 0    0^T, where s_1 = ϕ and s_2 = dϕ/dt. Given the purely time-dependent nature of the problem. the domain reduces to a scalar temporal variable, i.e., y = t. For this problem, we consider the generalised input function a(y) as equal to the time-dependent forcing function u(t).

To construct the training dataset, we set k=1 and generate N_1 = 3500 training input functions, u(t), sampled from the GRF defined in <ref> at M=100 time points. For each training input function, we compute the corresponding reference solution, s_1, using a fourth-order Runge-Kutta integrator at N_2 = 20 randomly selected time points. This data is used to train the operator. A test set of 10000 input functions is employed to evaluate the predicted solution for each test input function at 100 equidistant points within the time domain t ∈ (0,1].  <ref> presents the model's predictions for two representative test inputs.

A visual comparison of the test prediction cases reveals that the deterministic D-DeepONet struggles to accurately capture the reference solution. The standard KLD-VI solution, obtained at α=1.00, yields an improvement in the mean predictions, while the α-VI DeepONet with α = 2.00 demonstrates superior performance over both in terms of mean prediction accuracy, as evident in <ref>. In terms of distributional fit, as measured by negative log-likelihood, the standard KLD-VI has the lowest NLL values and exhibits a better performance (<ref>). However, the NLL for α = 2.00 is only marginally higher. Considering both the metrics, α = 2.00 offers a favourable balance between accurate mean predictions and reasonable distributional fit.






 §.§ Problem 3: Diffusion-Reaction System

We extend our analysis to the diffusion-reaction PDE, which involves derivatives in both spatial and temporal coordinates. The two coordinates are denoted by the tuple y = {x,t}, where x represents the spatial location and t represents time. Diffusion-reaction equations model the combined effects of diffusion and chemical reactions within a system, finding applications in diverse fields where heat transfer, mass transfer, and chemical kinetics occur simultaneously. 

Given an external source term u(x), we consider the following diffusion-reaction PDE:

    ∂ s/∂ t = D_c ∂^2 s/∂ x^2 + ks^2 + u(x),  x ∈ (0,1),   t ∈ (0,1].

This equation represents a diffusion-reaction system influenced by a source term and characterised by a quadratic dependence on the solution. We set the diffusion coefficient D_c = 0.01 and the reaction rate k=0.01. In this case, the source term u(x) serves as the input function, equivalent to a(x) without the dependence on the temporal variable t. We consider the PDE with zero initial and boundary conditions i.e., s(x,0) = s(0,t) = s(1,t) = 0.

To generate the training data, we numerically solve the PDE using a finite difference method on a (x × t)≡(100×100) grid, following a similar approach as in <cit.>. The training dataset comprises N_1 = 500 distinct source terms, u(x), generated from a Gaussian random field with ℓ = 0.5 (<ref>), evaluated at M=100 equidistant points in space. For each training input function, the solution s(y) is obtained at N_2 = 100 random points y sampled from the 100×100 grid; these sampled solution points are denoted by filled white circles in <ref>. Subsequently, we employ this training dataset to learn the operator mapping from u(x) to the PDE solution s(x,t).


To assess generalisation performance, we employ a separate test set comprising 10000 diverse source terms. <ref> presents a visual comparison of the model's predicted solution against the reference solution for a representative test source term across the entire 100 × 100 spatiotemporal grid. The figure also includes the associated uncertainty (one standard deviation) and absolute error. It is seen that regions of low response values (visualised as bluish areas) exhibit reduced smoothness in the predicted mean solution compared to the reference. As such, these regions demonstrate higher uncertainty, as evidenced by increased standard deviation.

A comprehensive performance assessment of predictive performance across different values of α is conducted using NMSE and NLL metrics, the averages of which are summarised in Tables <ref> and <ref>, respectively. Consistent with previous findings, α values greater than 1 generally outperform the standard KLD-VI case (α=1.00). For this specific problem, α = 3.00 achieved the lowest NMSE, while α = 2.50 yielded the lowest NLL. Notably, the performance metrics at  α = 2.00 are only marginally inferior to the optimal values, suggesting a degree of robustness in the model's performance within this parameter range. 





 §.§ Problem 4: Advection-Diffusion Equation

We now extend our analysis to the advection-diffusion PDE, which combines both advection (transport due to fluid flow) and diffusion (random movement of particles). This equation is fundamental to various physical processes, including solute transport in fluids. We consider the following advection-diffusion equation: 

    ∂ s/∂ x + ∂ s/∂ t -D_c ∂ s^2/∂ x^2 = 0,  x ∈ (0,1),    t ∈ (0,1],

subject to a parametric initial condition s(x,0) = u sin^2(2π x) and periodic boundary conditions i.e., s(0,t) = s(1,t). The diffusion coefficient is set to D_c = 0.1. 
In this case, the operator maps the initial condition s(x,0) to the solution s(x,t) at the final time. As such, the input function is the initial condition itself, defined as a(x) = s(x,0) = u(sin^2(2π x)). 

To construct the training dataset, we generate N_1 = 1000 unique initial conditions by sampling u from a GRF (with input domain defined by sin^2(2 π x)) with length-scale ℓ = 0.5 (<ref>), discretised at M=100 spatial points. The PDE is then solved numerically on a spatiotemporal grid (x × t) ≡ (100×100) using a finite difference method. For each initial condition, the solution s(x,t) is evaluated at N_2= 100 randomly sampled points within the grid, following similarly as the data generation process outlined in <ref>. This dataset serves to train the operator.

For evaluation, a separate test set of 10000 initial conditions is employed. <ref> presents a comparison of the model's predicted solution against the reference solution for a sample test initial condition across the entire spatiotemporal domain. A quantitative evaluation of the model's predictive performance across different values of α was conducted using the NMSE and NLL metrics, summarised in Tables <ref> and <ref>. In contrast to the previous examples where larger α values (i.e., values greater than 1) typically yielded superior performance, this problem exhibits optimal results at  α = 0.5 in terms of both NMSE and NLL. While values of α greater than 1 typically led to subpar performance in this case, the specific value of α= 1.75 can be considered as a reasonable choice for α values greater than 1.







To assess the model's robustness and generalisation capabilities beyond training data distribution, we conducted out-of-distribution (OOD) testing. Two distinct OOD datasets were generated, each comprising 100 examples.

The first OOD dataset was constructed by altering the kernel length-scale of the GRF (<ref>) used to generate initial conditions. While the training data utilised a length-scale of ℓ = 0.5, the OOD dataset employed a reduced length-scale of ℓ = 0.2 within the RBF kernel. This modification introduces increased fluctuations in the generated initial conditions compared to the training distribution.

The second OOD dataset was generated using a fundamentally different kernel, the rational quadratic kernel, which is defined as:

    κ(x_1,x_2) = exp( 1 + ||x_1-x_2||^2/2 ρℓ^2)^-ρ,

This kernel is parameterised by an additional scale mixture parameter, ρ. For our experiments, we set ρ = 1.0 and ℓ = 0.5 to generate 100 OOD test cases. This represents 100 different initial conditions for the advection-diffusion example.

The predictive performance of models trained with three different  α values: 0.5, 1.0 (standard KLD-VI), and 1.75, was evaluated on these OOD test datasets. Similar to the in-distribution analysis, the NMSE and NLL metrics were computed. The average results, summarised in  <ref>, indicate that the models exhibit reasonable generalisation capabilities on OOD data. Consistent with the in-distribution findings, α =0.5 outperformed both the standard KLD-VI (α=1.00) and the α=1.75 model in terms of both NMSE and NLL for both OOD datasets. This improvement over the standard KLD-VI model translated to upto a 7% reduction in NMSE and an 8.7% reduction in NLL for the tested OOD scenarios. Interestingly, the performance on the rational quadratic kernel samples was superior to that on the RBF kernel samples.






§ DISCUSSION

The results from the numerical investigation demonstrate the superior performance of the proposed α-VI DeepONet compared to both the deterministic D-DeepONet and the standard KLD-VI DeepONet, as evidenced by the consistently lower NMSE and NLL values across all four problems (Figures <ref> and <ref>). This underscores the effectiveness of the α-VI DeepONet framework in capturing complex input-output relationships and quantifying associated uncertainties.





A key finding is the influence of the hyperparameter α on model performance, which varies across different problems. While a universal optimal α value is desirable, our results indicate that cross-validation is necessary to determine the most suitable setting for each specific problem. Notably, the optimal α values for all four problems were mostly different from the standard KLD-VI case (α = 1), highlighting the limitations of the KL divergence under prior misspecification.

The choice of α is greatly influenced by the underlying structure of the true posterior distribution. Mode-seeking methods may struggle in scenarios with multi-modal posteriors, as they are susceptible to local optima. Conversely, mass-covering methods may be less effective when the true posterior exhibits widely separated modes. Therefore, the optimal α value is inherently problem-dependent.

For problems like the antiderivative, gravity pendulum, and diffusion-reaction problems, larger α values (greater than 1) yielded superior performance, suggesting a preference for mode-seeking behaviour. Conversely, the advection-diffusion problem favoured smaller α values (less than 1), indicating a need for mass-covering behaviour. The selection of α should consider both NMSE and NLL, as these metrics provide complementary insights – NMSE looks only at the mean prediction, whereas NLL considers distributional fit – into model performance. 

While the α-VI DeepONet offers significant flexibility, it is important to acknowledge the associated computational overhead. Compared to the deterministic DeepONet, the α-VI DeepONet incurs notably higher training times. This increase arises from two primary factors: the approximation of the Rényi α-divergence, which necessitates Monte Carlo sampling, and the doubling of trainable parameters due to the estimation of both mean and variance for each weight. However, compared to a standard KLD-VI DeepONet, the training times are approximately 1.1x longer, which is reasonable for the gain in flexibility.



§ CONCLUSIONS

In this work, we have introduced a novel, prior-robust, uncertainty-aware DeepONet framework grounded in generalised variational inference (GVI). This framework enables the learning of complex nonlinear operators while providing robust uncertainty quantification. Unlike previous approaches, our framework adopts an optimisation-centric perspective on Bayesian modeling by minimising the GVI objective. By replacing the Kullback-Liebler divergence with Rényi's α divergence, we enhance the model's robustness to prior misspecification and achieve superior predictive performance compared to the standard KLD-VI approach. The hyperparameter α offers flexibility in controlling the degree of robustness, allowing adaptation to different problem characteristics.

Our numerical investigation across four benchmark problems consistently demonstrates the advantages of the α-VI DeepONet over both deterministic and standard variational methods in terms of NMSE and NLL. The optimal α value varies across problems, highlighting the importance of tuning the hyperparameter.

While the α-VI DeepONet offers significant improvements, it is not without limitations. The introduction of the α hyperparameter increases model complexity and requires additional computational resources due to the approximation of Rényi's α divergence. Furthermore, the mean-field assumption underlying the variational posterior may not be adequate in all cases.

Future research directions include exploring alternative variational families (e.g., normalising flows) to address the limitations of the mean-field approximation and developing more efficient approximation techniques for Rényi's α-divergence. Additionally, investigating the application of the proposed framework to a wider range of complex systems is a promising avenue for future work.



§ ACKNOWLEDGEMENTS

R. Nayek acknowledges the financial support received from the Science and Engineering Research Board (SERB) via Grant number SRG/2022/001410, and the Ministry of Ports, Shipping and Waterways. 

unsrtnat





 Paper Title: Combining audio control and style transfer using latent diffusion.




[
    Amy Navarathna,1,2 Benjamin J. Carey,1,2 James S. Bennett,3 Soroush Khademi,1,2 and Warwick P. Bowen1,2,4*
    August 5, 2024
==============================================================================================================



§ ABSTRACT 

Deep generative models are now able to synthesize high-quality audio signals, shifting the critical aspect in their development from audio quality to control capabilities. Although text-to-music generation is getting largely adopted by the general public, explicit control and example-based style transfer are more adequate modalities to capture the intents of artists and musicians. 

In this paper, we aim to unify explicit control and style transfer within a single model by separating local and global information to capture musical structure and timbre respectively. To do so, we leverage the capabilities of diffusion autoencoders to extract semantic features, in order to build two representation spaces. We enforce disentanglement between those spaces using an adversarial criterion and a two-stage training strategy. Our resulting model can generate audio matching a timbre target, while specifying structure either with explicit controls or through another audio example. We evaluate our model on one-shot timbre transfer and MIDI-to-audio tasks on instrumental recordings and show that we outperform existing baselines in terms of audio quality and target fidelity. Furthermore, we show that our method can generate cover versions of complete musical pieces by transferring rhythmic and melodic content to the style of a target audio in a different genre. 

 




§ INTRODUCTION





Deep generative models are now particularly successful at synthesising high-quality, realistic audio signals. Hence, the major impediment to their broader use in creative workflows is not their audio quality anymore, but rather how end-users can have complete control over the generation process. Following early works on unconditional generation <cit.>, multiple methods proposed to enable control by conditioning generation on semantic tags or audio-descriptors <cit.>. However, such supervised approaches remain limited to the use of explicit descriptors and are constrained by their reliance on annotated datasets. The recent development of language models and representation learning led to impressive performance in text-conditioned generation, mainly relying on transformers <cit.> or diffusion models <cit.>. However, concepts such as timbre, musical style or genres boundaries are usually elusive and highly subjective. Hence, text descriptions might remain limited to common sounds and insufficient to precisely capture musical intentions. 





A parallel stream of research to alleviate those issues is to guide specific aspects of the generation process by providing audio examples. Most approaches in this audio-to-audio editing are focused on timbre transfer, where the timbre of a given sound is applied on the content of another. While some works can transfer any audio to the timbre of a given training set <cit.>; others achieve many-to-many timbre transfer but only between a small set of predefined instrument classes <cit.>. One-shot timbre transfer between different instrument recordings have been achieved using Variational Autoencoders (VAE) <cit.>, but these models rely on a latent bottleneck to enforce disentanglement between timbre and pitch, which hampers their ability to generate high-quality audio on real-world data. More recently, a text-inversion technique was proposed to perform musical style transfer between arbitrary content and style examples <cit.>, but it relies on a large pretrained text-to-music model and requires optimisation prior to each transfer, resulting in very slow inference. 



In this paper, we aim to unify explicit control through audio descriptors or MIDI sequences and style transfer within a single model. To do so, we separate local, time-varying factors of variations and global information, capturing musical structure and timbre in two separate representation spaces. We slightly abuse the terms structure and timbre: by structure, we designate time-varying features, e.g. melody, loudness; by timbre, we designate global features such as actual timbre, but also style or genre. The principle of our method is depicted in Figure <ref>.
Our approach is based on the recently proposed diffusion autoencoder <cit.>, which trains a semantic encoder to condition a diffusion model, in order to achieve both high-quality generation while being able to extract and control high-level features from the data. We extend this approach by building separate representations for timbre and structure, while enforcing their disentanglement with an adversarial criterion combined with a two-stage training strategy. Our method can generate audio matching a timbre target, while specifying the musical structure either with explicit controls (such as MIDI data input) or an audio example. For computation efficiency, our diffusion model operates in the latent space of pretrained autoencoders, resulting in faster than real-time inference on GPU[Experiments were conducted on a NVIDIA A5000 GPU]. 


First, we benchmark our model on a one-shot timbre transfer tasks and demonstrate that our model improves upon existing baselines in terms of audio quality, timbre similarity as well as note onsets and pitch accuracy. On the same dataset, we show that our model can also generate audio from MIDI input and a target timbre example with performances superior to a state-of-the-art MIDI-to-audio baseline. 
Finally, we show that our method can be applied to complete musical pieces and generate cover versions of a track by transferring its rhythmic and melodic content to the style of a target audio in a different genre. We provide audio examples, additional experiments and source code on a supporting webpage[ <https://nilsdem.github.io/control-transfer-diffusion/>]




§ BACKGROUND










 §.§ Diffusion models



Diffusion models (DMs) are a family of generative models that learn to reverse a stochastic process that gradually adds noise to the input data. These models benefit from high-quality generation, stable training and conditioning abilities, which led to their widespread adoption in image <cit.> and audio generation <cit.>. 

Formally, we define a forward process q(𝐱_1:T|𝐱_0) = ∏_t=1^T q(𝐱_t|𝐱_t-1), which is a Markov chain that increasingly adds noise to the data 𝐱_0 by relying on the conditional distribution 


    q(𝐱_t | 𝐱_t-1) = 𝒩(𝐱_t; √(1 - β_t)𝐱_t-1, β_t 𝐈),


where β_t are hyperparameters defining the noise levels at times t ∈{0,T}, with T ∈ℕ. We are interested in learning the reverse diffusion process p_θ(𝐱_t-1 | 𝐱_t), from which we can iteratively denoise a random sample 𝐱_T ∼𝒩(0, 𝐈) to a data sample 𝐱_0 ∼ p(𝐱_0). In a recenty study, <cit.> made a connection between diffusion and denoising score matching <cit.>, leading to a simplified formulation and improved experimental results. The authors show that the reverse process can be approximated by learning a denoising network ϵ_θ that predicts the noise ϵ∼𝒩(ϵ, 0, 𝐈) used to corrupt the data. This results in a simpler training objective 



    min_θ∈Θ  𝔼_t,𝐱_0,ϵ [ ϵ_θ(√(α_t)𝐱_0 + √(1 - α_t)ϵ, t) - ϵ ] ,


where α_t = ∏_s=1^t (1 - β_s), and ϵ_θ is usually parametrized by a UNet <cit.>. 



 §.§ Diffusion autoencoders


DMs naturally yield a series of latent variables 𝐱_1:T through their forward process. However, these stochastic variables built from increasingly adding noise do not capture much semantic information over the data. Although the more recent proposal of Denoising Diffusion Implicit Models (DDIMs) <cit.> extends the original diffusion formulation to a deterministic process allowing each data input to be mapped to a unique latent code 𝐱_T, it still fails to extract and organise high-level features from of the data. Diffusion autoencoders <cit.> alleviate this issue by employing a learnable encoder that compress the data to a semantic latent code 𝐳_sem = E_ϕ (𝐱_0), which then conditions a diffusion decoder. The semantic encoder and the DDIM decoder are trained jointly, following the objective


    min_θ, ϕ𝔼_t,𝐱_0,ϵ [ ϵ_θ(√(α_t)𝐱_0 + √(1 - α_t)ϵ, E_ϕ (𝐱_0), t) - ϵ ]


On image applications, the authors show that the semantic code captures high-level attributes such as person identity, smile or presence of glasses, and can be used for downstream tasks such as conditional generation and attributes manipulation, while achieving state-of-art reconstruction. This approach was also successfully applied to audio <cit.>, where the authors encode magnitude spectrograms into a semantic latent space, allowing to achieve high quality text-conditioned waveform generation. 




 §.§ Control in audio generation


A straightforward approach to extend unconditional generative models in order to provide instruments befitting artistic control is to introduce conditioning on explicit controls. The DDSP model <cit.> proposes explicit pitch and loudness conditioning, while FaderRave <cit.> extended explicit control to non-differentiable time-varying attributes, but both methods remain limited to explicit descriptors and annotated datasets.  

While recent text-to-music methods like MusicGen <cit.> and Music ControlNet <cit.>  have incorporated melody conditioning capabilities, their expressiveness remains constrained by the need to define subjective timbre properties through text prompts. Li et al. <cit.> proposed to use text-inversion in order to extract pseudo-words that represent timbre directly from audio, but their method is computationally intensive and requires to perform an optimisation for each new timbre target. Timbre conditioning directly from a waveform example was also recently proposed the context of bass accompaniment generation<cit.>.







 §.§ Unsupervised disentanglement in sequential data


Many works proposed to model sequential data as a combination of local (time-variant) and global (time-invariant) factors of variation. Notably, the disentangled sequential autoencoder <cit.> relies on simple architecture biases and parameter tuning to obtain disantangled local and global latent variables. Following this work, multiple methods improved the learned representation by explicitely minimizing the mutual information between the two learned variables <cit.>. It was shown that disentanglement can be further improved with contrastive learning and domain-specific transformations that preserve local or global attributes <cit.>.
























More specifically in audio generation, SS-VAE <cit.> employs a Vector-Quantized VAE to achieve disentanglement through compression on quantized structure latent codes, combined with timbre-preserving data augmentations. Luo et al. <cit.> follow a two-stage training strategy similar to ours, and improve disentanglement by enforcing the consistency of the global and local latent variables in style or content transfer. However, both disentanglement strategies degrades reconstruction accuracy, which on top of spectrogram inversion based synthesis leads in poor audio quality. 



§ METHOD


Our approach is based on the assumption that musical audio samples can be seen as specific instances of a set of latent features that are separated between global features that capture style, and time-varying features that capture the local evolution of the signal. Although diffusion models are capable of high-quality conditional generation, they are computationally expensive when dealing with high-dimensional data. Hence, we employ an invertible audio codec to first compress the audio into a low-dimensional latent space, onto which we can build an efficient generative model. We extend the DiffAE <cit.> architecture to two semantic encoders in order to extract separately timbre and structure features from input samples. To further disentangle the learned features and improve transfer as well as explicit control performances, we employ an adversarial training strategy during training. In this section, we detail our proposed model depicted in Figure <ref>.




 §.§ Audio codec


We build our audio codec as a convolutional autoencoder based on the RAVE model <cit.> architecture, featuring the adversarial discriminator recently proposed in <cit.>. The model compresses audio waveforms 𝐱 into an invertible latent sequence 𝐳∈ℝ^L × D, where D and L are the embedding space and time dimensions respectively. On top of the reconstruction and adversarial training objectives of RAVE, we introduce a penalty on the latent codes f(𝐳) = max(0, |𝐳| - 1) to enforce that most latent codes are distributed between -1 and 1. 



 §.§ Model structure



We extract a timbre representation  𝐯_T ∈ℝ^D_T from an audio target using an encoder E_ϕ^T applied to the latent sequence 𝐳 obtained with our audio codec. For structure, we extract a temporal representation 𝐯_S ∈ℝ^L× D_S from either an audio input or an explicit control signal 𝐜 (such as a MIDI sequence), using an encoder E_ψ^S. In the case of an audio input, it would be natural to also infer it from the latent sequence 𝐳. However, we found experimentally that it is particularly difficult to extract fine structure information from the highly-compressed representation 𝐳. Hence, we instead infer structure form the Constant Q Transform (CQT) <cit.> of the target signal, which has been shown to be a well-suited representation for pitch extraction tasks <cit.>. For explicit control, the sequence 𝐜 is directly used as input for E_ψ^S.

To generate audio, we sample a noise vector 𝐳_T and decode it to a latent code 𝐳_0 through latent diffusion conditioned on representations 𝐯_S and 𝐯_T. As diffusion formulation, we leverage the recent improvements introduced in <cit.> and parameterise our denoiser network D_θ to predict the data 𝐳_0 instead of ϵ. E_ψ^S, E_ϕ^T and D_θ are trained end-to-end to minimise the following loss function


    L_diff = 𝔼_t,𝐳_0,ϵ D_θ(√(α_t)𝐳_0 + √(1 - α_t)ϵ, 𝐯_S, 𝐯_T, t) - 𝐳_0


We parameterise D_θ as a 1D convolutional UNet with residual blocks and self-attention layers. The two encoders share a  similar architecture as the encoding half of the UNet, with the difference that the timbre encoder compresses the input temporally and applies average pooling on the time dimension of the last layer. We condition the UNet architecture on 𝐯_S through concatenation with each block inputs. For the timbre vector 𝐯_T we use Adaptative Group Normalisation (AdaGN) <cit.>. 





 §.§ Style and content disentanglement

Although splitting the semantic content between two vectors that are constrained on their dimensions already encourages disentanglement between timbre and structure information, there is no theoretical guarantee regarding their separation. Furthermore, the appropriate feature dimensions are highly dependent on the task and dataset at hand. Hence, to enforce disentanglement without constraints on the dimensions, we introduce a two-stage training combined with an adversarial strategy. First, we freeze the structure encoder and train the model to build an adequate timbre representation. To avoid 𝐯_T to encode all of the information required to reconstruct the target 𝐳, we extract timbre from a different sample 𝐳̃ coming from the same track, following the assumption that it shares the same timbre as 𝐳 but with a different structure. 

In the next stage, we introduce a discriminator D_ζ that tries to predict 𝐯_T from 𝐯_S and is trained to minimise 


    L_D = 𝔼_𝐯_𝕊, 𝐯_𝕋 [ 𝐯_T - D_ζ(𝐯_S)  ].


We train the discriminator alternatively with the encoders and denoising network, which try to minimize the following objective


    L_total =  L_diff - γ L_D,


where γ is an hyperparameter balancing between the reconstruction objective and disentanglement. Indeed, increasing L_D maximises the confusion of the timbre information in the structure space. This restrains the diffusion model from reconstructing 𝐳 solely from 𝐯_S and enables independant control of structure and timbre at inference. 



§ EXPERIMENTS


We aim to asses the ability of our model to generate high-quality audio samples that match characteristics of structure and timbre targets, with the structure being either taken from an audio example or through an explicit control signal. 

MIDI-to-audio For explicit structure control, we evaluate the capability of our model to generate audio from a MIDI score and a target recording for timbre. We compare it to a state-of-art baseline in MIDI-to-audio generation. 

Timbre transfer We evaluate the efficiency of our disentanglement strategy on a task of one-shot timbre transfer between polyphonic mono-instrument recordings. In this case, we consider that structure designates the notes being played (in terms of onset timing, pitch and loudness), while timbre corresponds to the remaining characteristics of the sound. We evaluate our model by randomly sampling timbre and structure examples and evaluate audio quality, timbre similarity as well as note accuracy. We compare our model with two example-based timbre transfer methods on synthetic and real recordings. 




 §.§ Dataset


Synthetic Data The Synthesized Lakh Dataset (SLAKH) <cit.> was generated from the LAKH MIDI collection using professional-grade sample-based virtual instruments. Synthesis parameters as well as audio effects settings were randomly chosen resulting in a very diverse set of timbres. We retain only the individual stems of non-percussive instruments, resulting in 400 hours of audio. 


Real Data To the best of our knowledge there is no multi-instrumental dataset of real recordings that contain a very large number of hours of audio. Hence, we combined the following three datasets to conduct our experiments :

    
  * MaestroV2 : Maestro <cit.> is a piano dataset recorded on Disklavier pianos, capturing both audio and notes played, resulting in approximately 200hours of annotated piano recordings. 

     
  * GuitarSet : Guitarset <cit.> is a collection of live guitar performances with solos and accompaniment from various genres and play styles, with a total audio duration of 6 hours.

     
  * URMP : The URMP dataset <cit.> is composed of pieces played by a large variety of classical instruments. For each piece we retain the mono-instrumental recordings, resulting in approximately 4 hours of audio.


As the GuitarSet and URMP are low sample-size datasets, we add synthetic data stems from SLAKH to the training set to facilitate learning. Furthermore, as the different datasets are greatly imbalanced in terms of sample size, we apply a sampling strategy to even the model performance on each dataset. Following <cit.>, if n_i is the number of samples in a given dataset, we draw examples from this dataset during training with probability (n_i / Σ_j n_j)^0.3.











 §.§ Evaluation metrics


We aim to evaluate how our method is able to match the timbre and structure targets characteristics, while maintaining high-quality audio. 

Audio quality We rely on the widely used Frechet Audio Distance (FAD) <cit.> to evaluate how the generated audio distribution matches the dataset distribution, both for reconstructed and transferred samples. We use the available reference implementation of FAD[https://github.com/gudgud96/frechet-audio-distance/tree/main] and use VGGish <cit.> embeddings of the samples to compute the distance


Timbre Similarity To evaluate timbre similarity we employ the metric proposed in the SS-VAE baseline <cit.>. It relies on a triplet network trained to predict if samples are played by the same instrument based on Mel-frequency cepstral coefficients 2-13. We use their implementation and train the metric on the mixing-secrets[https://www.cambridge-mt.com/ms/mtk/] dataset.

Structure To evaluate if our model is able to reproduce the notes of the structure target, we employ a transcription model <cit.> and compare its output to the ground-truth MIDI data. As metric, we use Onset F1 score from mir-eval, where two notes are considered identical if they have identical pitch and onsets within ± 50 ms of each other. 


 §.§ Baselines


For the timbre transfer experiments, we compare our method to SS-VAE <cit.> and Music Style Transfer <cit.> presented in Section <ref>. We train both models on the real and synthetic datasets, using the official implementation. For explicit control, we evaluate our method against a MIDI-to-audio model <cit.> that was also trained on the SLAKH dataset. 
We use the small configuration of the publicly available pretrained model, as larger models do not fit on our NVIDIA A5000 GPU. 



 §.§ Training details


We start by training our audio codec for 1M steps before training our diffusion model for 500k steps, with an initial timbre learning stage of 100k steps. The overall training takes one day on NVIDIA A5000 GPU. For all experiments we rely on the AdamW optimizer <cit.> with a constant learning rate of 1e^-4 and a batch size of 48. For inference we use the deterministic sampler proposed in <cit.> with 40 diffusion steps. 



§ RESULTS




 §.§ MIDI-to-audio


First, we evaluate our model performance in MIDI-to-audio generation in terms of audio quality, timbre similarity and Onset F1 score. We detail our results in Table <ref> for reconstruction and transfer setups, where the target timbre corresponds to either the instrument of the MIDI sequence or a different sample. In both cases, we obtain higher similarity, as our dedicated timbre embedding captures timbre much more precisely than simple label conditioning on instrument categories. Interestingly, we also obtain better F1 scores, although we did not design our model specifically for MIDI inputs as opposed to <cit.> where authors employ a dedicated note sequence embedding strategy. 

To assess the benefit of our disentanglement strategy, we experiment with bypassing the structure encoder and directly conditioning the UNet on the MIDI sequence (Ours w/o E_S entry in  Table <ref>). This results in overall better Onset F1 score, but degrades timbre similarity and FAD. This demonstrates that our disentanglement strategy improves the capability of the model to precisely render the timbre of the target recording. As described in Section <ref>, the Onset F1 score characterises the difference between the generated notes and the input MIDI sequence. The lower accuracy obtained with our full model in the transfer setup can be explained by the fact that some MIDI sequence are not plausible scores for some target instruments (such as playing chords with a flute). Through the disentangled structure encoding, our model is capable of adapting the input MIDI sequence to the range and capabilities of the target instrument, which results in more realistic sounding samples. We encourage the reader to listen to the examples on our supporting webpage that support this statement. 



 §.§ Timbre transfer


Synthetic data Here, we first evaluate timbre transfer on synthetic data and display the results in Table <ref>. The two baselines appear to provide low audio quality and timbre similarity, and both methods obtain lower Onset F1 scores indicating that they are not able to adequately control structure and timbre independently. Our method improve upon the baselines on all three evaluated aspects, and is interestingly able to reach a comparable performance as in the explicitly conditioned MIDI-to-audio setup.

We also performed an ablation study, by evaluating the effect of applying an information bottleneck on the structure latent space instead of using our adversarial strategy. As mentioned in Section <ref>, the model is able to transfer timbre when D_S is small but achieves a low F1 score. Increasing the latent dimensions improves structure fidelity at the cost of degrading timbre similarity. Using our disentanglement strategy, we are able to employ a 32-dimensional latent space and achieve higher timbre similarity with a slight decrease in note accuracy. Although multiple definitions of timbre transfer are possible, we argue that the most convincing timbre transfer do not necessarily imply a perfect note structure F1 score. In the case of a transfer between monophonic instruments playing in the same pitch range, we can expect all the notes from one recording to be transferred to the other. However, when performing transfer between recordings with very distinct timbre such as a piano playing in its high range and a bass, an interesting transfer would rather be the bass playing the main melodic line a few octaves lower than the piano, which would result in a low F1 score. The improvement in terms of FAD between the distribution of transferred samples and the original data obtained with our disentanglement strategy supports that our method generates more realistic transfers, as an instrument playing notes outside its usual range would be considered as out-of-distribution. 






Real data. We present our results for transfer between real instrumental recordings in Table <ref>. Our model improves even further on the existing baselines for which real instruments timbre seems particularly challenging. Even without adversarial regularisation, our model obtains better FAD, timbre similarity and note accuracy. Our disentanglement strategy further improves timbre match, although the relative decrease in note accuracy appears to be greater than on synthetic data. We believe this is mainly due to a necessary simplification of note structure when transferring complex piano recordings to the mainly monophonic URMP instruments. The improvement of transfer quality captured by the FAD and timbre similarity supports this interpretation. 

Qualitatively, we found that on top of generating realistic samples with the appropriate structure, the model is also able to add characteristic sound artefacts of the target instrument such as fret or hammer noises, as well as matching precise acoustic features of the original recording such as reverb or background noise. 



§ COMPLETE MUSIC STYLE TRANSFER





Finally, we apply our model to the task of creating cover versions of a song that match the style of an example in a different genre. We rely on an in-house dataset of 200 hours of jazz, dub, lofi hip-hop and rock. We use the same model architecture to extract structure from the original track and timbre from the cover targets, with two minor modifications. First, we introduce temporal compression in the structure encoder to avoid 𝐯_T to capture information that is too precisely located in time. Second, we condition the UNet on a BPM time series to help it generating coherent rhythms. Without those modifications, the rhythmic elements from timbre and structure targets are conflicting with each other, resulting in somewhat chaotic generations. We evaluate our model using the cover detection algorithm proposed in <cit.>, which outputs a cover probability based on melodic and harmonic similarities between tracks. To assess style transfer, we rely on the text and audio joint-embedding model CLAP <cit.> and classify genre based on the cosine similarity between the audio and genre label embeddings. 
We compare our method to MusicGen <cit.>, a text-to-music generation model with audio-based melody conditioning. We derive an input prompt from the target genre and use the structure target for melody. 

Our model without regularisation obtains a better cover identification than MusicGen, and our disentanglement strategy further improves transfer resulting in higher genre accuracy. Qualitatively, MusicGen seems to only extract the main melodic idea from the structure audio, whereas our method is able to capture most of the harmonic and melodic content. Furthermore, as our model extract style directly form audio rather than through a text prompt, it transfers the different structural elements towards the instruments actually present in the timbre target rather than just the typical instruments of the genre.





§ CONCLUSION


We presented a simple method to learn disentangled timbre and structure representations. To the best of our knowledge, this is the first model capable of generating realistic, high-quality audio through transfer and MIDI rendering. We leave for future works improvements on the trade-off between reconstruction and disentanglement and applications to more complex musical datasets. Furthermore, we aim to initiate a reflection on the characterisation of the elusive concept of musical style transfer, which we believe to be an exciting stream of research towards a broader use of deep generative models in artistic work-flows. 





























 Paper Title: Cost-Effective Hallucination Detection for LLMs.









Cost-Effective Hallucination Detection for LLMs
    Simon Valentin^1[1]
        Jinmiao Fu^2[1]
        Gianluca Detommaso^3These authors contributed equally to this research. 
        Shaoyuan Xu^2
        Giovanni Zappella^1
        Bryan Wang^2
    
    ^1Amazon Web Services, Berlin, Germany 

    

    ^2Amazon, Seattle, USA 

    

    ^2Helsing, Berlin, Germany 

    


=======================================================================================================================================================================================================



§ ABSTRACT 
Large language models (LLMs)
can be prone to hallucinations — generating unreliable outputs that are unfaithful to their inputs, external facts or internally inconsistent.
In this work, we address several challenges for post-hoc hallucination detection in production settings.
Our pipeline for hallucination detection entails: first, producing a confidence score representing the likelihood that a generated answer is a hallucination; second, calibrating the score conditional on attributes of the inputs and candidate response; finally, performing detection by thresholding the calibrated score. We benchmark a variety of state-of-the-art scoring methods on different datasets, encompassing question answering, fact checking, and summarization tasks. We employ diverse LLMs to ensure a comprehensive assessment of performance.
We show that calibrating individual scoring methods is critical for ensuring risk-aware downstream decision making. 
Based on findings that no individual score performs best in all situations, we propose a multi-scoring framework, which combines different scores and achieves top performance across all datasets.
We further introduce cost-effective 
multi-scoring,
which can match or even outperform more expensive detection methods, while significantly reducing computational overhead. 





§ INTRODUCTION

Despite their impressive capabilities,  large language models (LLMs) can be prone to generating hallucinations — undesirable outputs that are incorrect, unfaithful, or inconsistent with respect to the inputs (or the output itself) <cit.>. 
These unreliable behaviors pose significant risks for adopting LLMs in real-world applications. 
Challenges in detecting hallucinations lie, among other things, in hallucinations taking different forms, being context-dependent and sometimes being in conflict with other desirable properties of generated text <cit.>. 
Hallucinations may be harmless in some contexts, but can be undesired or potentially dangerous in other applications (e.g., erroneous medical advice).
Detecting and quantifying hallucination risk is thus a critical capability to enable safe applications of LLMs and improve generated outputs.



Prior work has proposed various approaches for detecting and mitigating hallucinations in LLM-generated outputs, including verifying faithfulness to inputs <cit.>, assessing internal coherence <cit.>, consulting external knowledge sources <cit.>, and quantifying model uncertainty <cit.>.
However, deploying these methods in production settings is far from trivial due to several challenges:
First, there is limited comparative evaluation illuminating how different detection methods perform. 
Second, existing approaches for detecting hallucinations differ greatly in their computational demands, and guidelines are lacking on cost-effectiveness trade-offs to inform method selection for real-world applications with constraints.
Third, hallucination detection in the real world often requires careful consideration of risks and false positive/negative trade-offs, requiring methods to provide well-calibrated probability scores.
Fourth, many applications of LLMs take the form of calls to black-box APIs, which sometimes requires methods to employ workarounds to assess the model's confidence in its generated output. 

In this work, we provide a framework for detecting hallucinations in the outputs of any LLM in a model-agnostic manner.
Our approach relies on quantifying the probability that a generated answer is a hallucination. 
After computing initial scores, we employ state-of-the-art calibration techniques to provide calibrated probabilities of the generation containing a hallucination, which can subsequently be used for decision making or other downstream tasks. 
We evaluate a variety of scores proposed in the literature for hallucination detection on several metrics, across multiple datasets encompassing question answering, fact checking, and summarization tasks. We employ a range of different LLMs to ensure a comprehensive assessment of performance. 

Critically, as we show that no single score performs best across all datasets, 
we introduce
multi-scoring, 
a simple way of combining multiple calibrated scores, which achieves superior performance compared to 
any individual score alone.
Furthermore, 
we 
propose cost-effective multi-scoring, which 
finds the subset of best-performing scores for any fixed cost budget, and combines them in a multi-scoring fashion. Our empirical demonstrations reveal that cost-effective multi-scoring not only matches but often surpasses the performance of individual scores that incur significantly higher costs. Consequently, our proposed method achieves superior hallucination detection outcomes while maintaining a substantially lower cost footprint.


We summarize our contributions as follows: 1. We benchmark a variety of hallucination detection methods across the literature on several metrics, over different datasets and LLMs. 2. We introduce multi-scoring, a novel approach that aggregates multiple complementary scores and outperforms individual scores. 3. We further propose cost-effective multi-scoring, which optimally balances detection performance and computational constraints. 





§ DETECTING LLM HALLUCINATIONS




 §.§ Formalizing Hallucination Detection

We study the problem of quantifying the probability that a generated output from a language model contains hallucinations.
More formally, let 
x represent an input token sequence to a language model 𝒢, and let 
z represent a generated output text sequence from the model. 
We define a binary random variable y ∈{0,1} that indicates whether z is a “permissible” output (y=1) or contains a “hallucination” (y=0). 

Our goal is to develop a scoring function to model the probability that a given output text contains a hallucination conditioned on the input.[Note that while we are interested in whether the entire output contains any hallucination, one could easily adapt the methods at different granularities, such as the sentence or phrase level.]
This is critical, as in real-world scenarios, we need to set risk-aware thresholds, balancing false positive/negative rates to accommodate the production requirement.
Having access to scores allows us to flexibly set the threshold.
Conceptually, the key reason to model this probabilistically is that there is inherent uncertainty in determining whether a given text contains a hallucination or not. 
Even human raters may disagree on the assessment, based on their own knowledge and definition.
Some key contributors to this epistemic uncertainty are: First, that no system has complete world knowledge to perfectly assess factual correctness. 
Second, that there is ambiguity in whether something is a hallucination. 
Finally, any automatic scoring model may make occasional errors, so probabilistic scores reflect confidence. 
Therefore, while conditional on x and z the true hallucination label y is fixed, our estimate of y remains uncertain. The probabilistic score thus reflects this epistemic uncertainty — the degree of belief that z contains a hallucination given the available knowledge:
p(y = 0 |x, z; ψ), where ψ denotes parameters of the scoring function. We refer to this conditional probability function as the hallucination score, denoted s_ψ(x, z).
The hallucination score can then be applied to downstream tasks, including making risk-aware binary decisions.

We discuss concrete instances of hallucination scores in the following subsection.
Generally, the form of s_ψ can vary between hallucination detection approaches. 
In particular, s_ψ may depend on multiple candidate texts generated for the same input, as implemented by a number of hallucination detection methods proposed in the literature.
We denote K candidate generated outputs as Z = [z_1,...,z_K]. 
Then s_ψ(x, Z) could quantify inconsistencies within texts in Z using different metrics ψ, as discussed below.



 §.§ Scoring Methods

Many hallucination detection methods proposed in the literature make use of LLMs to “judge” the output of an LLM (either the same or a different one). 
For clarity, we thus distinguish between generator and detector LLMs. 
The generator is defined as the model used to generate the original response. 
The detector LLM is the model used to score a generated text for the presence of hallucinations. 
In general, the generator and detector LLMs may coincide. 
However, it may be more desirable to use different LLMs in some scenarios, e.g., when computational cost is a greater concern, where a smaller LLM may be used to judge outputs of a more expensive LLM, or when using hallucination scoring methods that require white-/grey-box access, while the generator is black-box. 

Today, many interactions with LLMs take the form of API-calls.
Typically, only the output tokens are returned with no access to the logits of the predicted tokens,  treating the LLM effectively as a black box. 
Sometimes, inference parameters may be accessible, allowing for setting different temperature (among others) values, thereby providing some (grey-box) access to the model. 
Generally, hallucination detection methods vary in their required model access, ranging from black-box APIs to full white-box access to the model weights.
That is, while some methods only require token-level outputs, other methods may need access to the logits of the generated tokens, or some control over inference parameters like temperature. 

In our experiments, we evaluate a comprehensive set of hallucination scoring methods.
Generally, we do not make any assumptions about the generator LLM being white-, grey- or black-box. 
We divide methods into single-generation methods, which require only one generated output, and multi-generation methods which are based on multiple alternative generations. 





 §.§ Single-generation

We first provide an overview over a set of hallucination detection methods that are based on scoring the hallucination from a single given generated output. 



  
Inverse Perplexity This method provides a prominent instance of using a model's logits over output tokens to score the confidence in the output. Computed as the inverse of the exponentiated average negative log-likelihood of the LLM's response <cit.>,

    Perplexity^-1(W) = exp(1/N∑_i=1^N log p(w_i|w_i-1,...,w_1)),

inverse perplexity thus provides a sequence-length normalized expression of the model's confidence. 
Here, the model may either be the generator LLM or a different detector LLM.
Using a different LLM from the generator as a plug-in estimator implies that the confidence estimate is detached from the generator. 
If our generator provides access to the model's logits we can directly use them to score hallucinations.
Alternatively, we can use another LLM to generate the logits of output tokens. 



  
P(True) This method works by prompting an LLM whether an answer is correct or not, then making use of the logits of the next token <cit.>. 
Given a prompt like the following: 


We compute P(True) by first taking the softmax over the first generated output token's logits, 
and then normalizing across the “True” and “False” tokens.[Note that alternatively, we can prompt the LLM to bind the True/False tokens to symbols like response options A or B <cit.>. However, this did not change results in our experiments. Further, if the detector LLM provides only black-box access, one can sample the corresponding tokens and use the empirical distribution to approximate, but thereby incurring a higher computational cost.]
That is, let 𝐳^(1) = (z^(1)_1, …, z^(1)_|V|) be the logit vector over the vocabulary V for the first generated token when asking the LLM to evaluate whether the answer is correct. We then take the softmax over these logits, 
p(w |𝐳^(1)) = exp(z^(1)_w)/∑_w' ∈ Vexp(z^(1)_w').
Let w_True, w_False∈ V be the “True” and “False” token ids. Then we can compute P(True) as:

    P(True|𝐳^(1)) = p(w_True|𝐳^(1))/p(w_True|𝐳^(1)) + p(w_False|𝐳^(1)).


In addition to checking whether the response is correct, we provide variations of P(True), where we check whether the output contradicts the input in P(InputContradict), the output contradicts itself in P(SelfContradict), or the output contradicts generally established facts in P(FactContradict), with the followings prompts: 








  
NLI Text Classification Natural language inference (NLI) models provide an alternative to assess the correctness of the model output. As hallucination detection requires checking for contradictions, we compute the score as 1-S_contradict, where S_contradict refers to the softmax probability of the output conflicting with the question. Specifically, we use a DeBERTa model fine-tuned on an MNLI task <cit.> as the underlying NLI model, which we refer to as NLI (DeBERTa).




  
Verbalized Probabilities Instead of analyzing a model's logits, scores are elicited by asking an LLM to provide a confidence verbatim, i.e., by generating tokens that indicate the numerical confidence <cit.> with the following prompt:

We note that moderated LLMs accessed through APIs may in practice decline to answer if the context does not provide sufficient information, rather than assigning a low confidence score.  



 §.§ Multi-generation
 Multi-generation methods are based on quantifying the consistency across multiple generated outputs from the generator LLM. 
This follows the notion of white-/grey-box uncertainty quantification via logits that if the LLM is confident in its response, multiple generated responses will probably be alike and contain compatible facts. 
Conversely, for fabricated information, sampled responses are more likely to differ and contradict one another. Crucially, this rests on the assumption that the model's confidence is calibrated, as we discuss and evaluate below.



  
SelfCheckGPT This method exploits a DeBERTa NLI model to assess whether the answer A_0 is consistent with K alternative generated answers A_1, ..., A_K <cit.>. Each sentence of A_0 is compared against the full set of answers A_1, ..., A_K. A consistency score per sentence is obtained via an NLI model, then the final score is obtained by averaging over the sentence-wise scores. 



  
Similarity Degree This method is based on computing pairwise similarities between the multiple responses using NLI models, and then quantifying uncertainty based on the distribution of similarities <cit.>. Here, we compute the pairwise similarities between responses via the contradict class probability of an NLI model and construct a degree matrix, where each diagonal element corresponds to the total (sum) similarity of the corresponding response to all other responses. We use the degree of the candidate response A_0 as the confidence estimate, following favourable results in prior work <cit.>, but note that other metrics have also been proposed in the original paper.



  
NeMO Guardrails: Hallucination Rail 
Unlike SelfCheckGPT or Similarity Degree, the score here is computed via one LLM call, where we check for agreement between the concatenated K additional generations and the candidate output and not averaged over sentences or computed on pairs of responses. Specifically, we use the softmax probability normalized over yes/no tokens using the following prompt:





 §.§ Calibration

Initial hallucination scores may not be properly calibrated, which can lead to poor downstream decisions. Seminal work by Guo et al. <cit.> has demonstrated that neural models tend to be miscalibrated, particularly in the form of models being overconfident. 
Later work, focusing on language models, has confirmed this across a wide range of tasks, models and datasets <cit.>. 
Notably, there is also research which demonstrates that given particular prompts and in-distribution tasks, LLMs can be well-calibrated <cit.>. However, this has been shown to be brittle and dependent on context <cit.>. 

Formally, our score outputs probabilities p_ψ(ŷ=0|x, z) that z contains a hallucination, parameterized by ψ. 
The score is calibrated if, for any probability level p ∈ [0,1], the average observed frequency of hallucinations matches the predicted probability:

    𝔼[y | p_ψ(ŷ=0|x, z) = p] = p


A naive approach for obtaining scores would be to compute the model's probabilities marginally, ignoring the context/question x and generated response z and directly estimating p(y=1). 
However, this does not allow for conditioning on individual inputs to obtain calibrated probabilities p(y=1 |x, z) for specific x, and z, which is, however, impossible to guarantee <cit.>. 
Common calibration methods include temperature scaling (Platt scaling) of logit outputs <cit.>, isotonic regression <cit.> or histogram binning <cit.>, which operate marginally, and can thereby not account for different confidence levels for different inputs (e.g., with an LLM being more confident on certain domains than others).

An alternative is to partition the inputs into G groups and compute calibration separately for each group g∈ G. However, this assumes the groups are disjoint and does not handle inputs belonging to multiple groups, which is often necessary <cit.>.
More advanced calibration methods, such as multicalibration, which we use in this work, allow defining G potentially overlapping groups  <cit.>. 
Prior work has shown the effectiveness of modern calibration techniques in scoring LLM confidence, albeit in a white-box setting <cit.>.
We describe our calibration approach in the experimental section.



 §.§ Multi-Scoring: Combining Scores

Different scoring methods capture different aspects of hallucinations, e.g., incorrect, non-factual, non-grounded, irrelevant, inconsistent with other answers, etc. As a result, some scores may work better on some data or for some specific models, and worse on others. Therefore, we design a multi-scoring method to combine the complementary information from individual scores into a single, strong predictor.

Denote each available score by s_n, for n=1, … ,N. To obtain an aggregated score, we run a logistic regression using as features the concatenations of the logit of each score, i.e. [logit(s_n(x, z)), …, logit(s_N(x, z)) ], and the labels of the calibration dataset as target variables. [Note that we also conducted experiments with alternative models, such as XGBoost <cit.> and Random Forest <cit.>, but results did not show significant improvements, so we opted for logistic regression for simplicity.]





 §.§ Cost-Effective Multi-Scoring

Scoring methods based on multiple generations incur the cost of additional generations as well as the cost of checking their consistency. 
Similarly, multi-scoring can be a viable choice when there are only few LLM generations to check for hallucinations, but incurs considerable computational cost, especially when using multi-generation methods. 
To avoid prohibitive computational costs in practice, we propose cost-effective multi-scoring, where we set a fixed computational budget and compute the best performing combination of scores that stay within the specified budget. 

Given an input text x and generated text z, we have N scoring functions s_1(x, z), ..., s_N(x, z) with associated costs c_1, ..., c_N (e.g., number of generations required). We are given a total computational budget B. Our goal is to find the optimal subset of scores S^* ⊆{1,...,N} that maximizes detection performance while staying within budget B:


    S^∗ = _S: S ⊆{1, …, N}ℒ(f(s_i(x, z))_i ∈ S)   s.t.  ∑_i ∈ S c_i ≤ B


where ℒ measures loss on a validation set. This is a constrained optimization problem over subsets of scoring functions. When B=min_i c_i, it reduces to selecting the single best score at the lowest cost. When B=∑_i c_i, it recovers full multi-scoring. In between, the optimal subset S^* provides the best trade-off between performance and cost. 
We note that in general, this problem is computationally challenging, given the exponential runtime complexity. 
However, given that there are only generally a small set of potential scores (N ∼ 10) and the logistic regression is very fast to fit, computing all combinations takes only 1.8 seconds on a single Intel Xeon processor (3.1 GHz), iterating over all candidate solutions sequentially. 
When this approach is not feasible, some exemplary alternatives include classic greedy forward-selection methods or regularising the model via an L1 penalty while scaling the regularisation term to accommodate score cost. 
Overall, cost-effective multi-scoring allows for flexibly balancing multiple scores under computational constraints.



  
Quantifying Cost
While the cost c_i of scoring function s_i is presented abstractly above, accurately quantifying computational cost can be challenging in practice. 
The actual runtime per method is a reasonable proxy, however the runtime depends on model architecture, hardware acceleration, batching, etc. 
One approach is to directly benchmark each scoring function's average runtime empirically on the target hardware. 
However, this overlooks nuances like caching effects and ignores runtime variability due to implementational differences.
To simplify our analysis, and as LLM calls are generally much more expensive than calling smaller NLI models based on parameter size, we leverage the number of LLM calls required per method as a proxy. See Table <ref> for an overview. 
If more precision is desired, we suggest empirically running different scores and computing their computational cost in the actual application, as the cost will depend on the precise setup and a range of factors.




§ EXPERIMENTS




 §.§ Experimental Setup



  §.§.§ Datasets




  
TriviaQA TriviaQA <cit.> is a commonly used factual open-respon­se QA dataset. Originally set up as a reading comprehension task, it is today often used without context as a closed-book (free-recall) task <cit.>. 
We use the validation fold, containing 17944 question-answer pairs. We use  <cit.> to generate candidate answers. To decide whether a given response should be labelled as positive or negative, we check whether the correct answer is contained in the generated answer after removing formatting, following the original evaluation script <cit.>.  



  
FEVER The closed-response Fact Extraction and VERification dataset <cit.> provides a comprehensive benchmark of factual hallucination detection. We take the test fold, containing 14027 labelled examples of source documents, claims and whether they are  ,  or contain . For the purpose of hallucination detection, we look at all claims that are either  or .



  
HaluEval We include the hallucination detection dataset HaluEval <cit.> and use the summarization task, which contains 10000 labeled examples of source documents, summaries and labels for whether the summary contains hallucinations. 



  
BIG-bench It provides an evaluation of LLMs on a diverse set of tasks <cit.>. We select 11 tasks suitable for hallucination detection, leading a total of 8664 labelled examples from the validation fold. 



  §.§.§ Metrics

Practical applications of hallucination detection require calibrated scores, but often also binary decisions over whether a given output is hallucinated or not. 
To this end we report the Brier score <cit.>, binary decision metrics F1 score and accuracy. 



  §.§.§ LLMs

We conduct experiments with  <cit.>,  <cit.>,  <cit.> and  <cit.>.
All models are used with default configurations via HuggingFace Transformers <cit.>.



  §.§.§ Calibration

The calibration step is performed via the following multicalibration approach, using the Fortuna library <cit.>:
To obtain groups, we compute embeddings of the input text x and the generated response z, such that our embedding is e[ (x), (z) ]. We obtain embeddings from Universal AnglE Embedding <cit.>, which is the SOTA in the MTEB benchmark <cit.> at the time of writing. 
We subsequently reduce the dimension of x via UMAP <cit.> and perform soft-clustering via Gaussian Mixture Models, as a simple off-the-shelf algorithm (fitting the number of cluster components via BIC <cit.>).
The calibration error is measured for each group g∈ G separately. 
The model is then iteratively patched on the group with the largest error until the calibration error drops below a threshold for all groups.[Note that alternative approaches are possible <cit.>.] 
This provably converges to a calibrated model with theoretical guarantees <cit.>.
We fit the calibrator to a random calibration fold of 80% of the data and report only held-out test results. 
For all binary predictions, we set the threshold to the 50th percentile to not impose preferences over false true/negative rates, but note that in practice any threshold could be applied. 



 §.§ Individual Scoring Methods

Table <ref> presents the results of different hallucination detection methods on all datasets. Multi-generation methods (SelfCheckGPT-NLI, HallucinationRail and SimilarityDegree) are employed only for TriviaQA, since the latter provides the true answer to each question, which can be compared to the alternative generated answers to assess their agreement. In contrast, for HaluEval, BIG-Bench and FEVER, the true answer is not provided. Instead, we compare the given candidate answer from the dataset against the provided binary label of correctness.
If we exclude Multi-Scoring methods, multi-generation methods such as SelfCheckGPT-NLI and SimilarityDegree (with 10 generations) achieve the best performance on TriviaQA, P(True) is the best on HaluEval and BIG-Bench, and P(InputContradict) performs best on FEVER. 
Thus, we find that there is no single best scoring method across all datasets. 
This is likely because different scoring methods capture different aspects of hallucination, supporting the notion that hallucination is a multi-faceted concept. 
Multi-generation methods measure hallucination based on the consistency of the generator LLM's responses and on the ability of the comparison method to identify whether multiple alternative responses are in agreement. 
We find that SelfCheckGPT-NLI performs best in our experiments in comparison to SimilarityScore and HallucinationRail.
While all of these methods follow similar approaches, there are subtle differences in how they assess the consistency between different responses.
More generally, as we discuss below, multi-generation methods are appropriate only if one can assume that there is exactly one correct response, but can fail otherwise. 

Other methods are based on different notions of hallucination, such as NLI (DeBERTa) measuring the entailment of the response given the input. 
Interestingly, variants of P(True) can be appropriate for detecting different kinds of hallucinations. 
P(True) explicitly asks the evaluator LLM to check whether the response is correct. 
In some contexts, the evaluator LLM may have the ability to directly evaluate this. 
However, in other situations, as we see with the FEVER dataset, other methods such as P(InputContradict) can be more appropriate, if we are trying to directly target a specific form of hallucination, which may not be subsumed under a general “correct or not?” prompt. 
Other applied scenarios may target even more different (or more specific) kinds of hallucinations, though the variants we include in this work are designed to cover the space in a reasonable manner. 
Similarly, while the datasets included in these experiments were selected to cover a wide range of different kinds of hallucinations, real-world applications may show even new kinds of hallucinations, for which no public datasets are available.
Overall, these findings highlight the need for our multi-scoring method which can absorb the strength of each individual method and can be easily applied to a concrete hallucination detection setting while only requiring a relatively small amount of labeled data. 

Overall, neither the inverse perplexity score nor the NLI (DeBERTa) scores emerge as the best scores for any of the datasets we consider. While they may add information that can be exploited in (cost-effective) multi-scoring, as reported below, individually they perform worse than some of the other scores. 






 §.§ Multi-Scoring

To evaluate the proposed multi-scoring method, we conduct experiments on all datasets combining the scores via logistic regression. 
As presented in Table <ref>, the multi-scoring ensemble achieves an F1 score of 0.9106 on TriviaQA, outperforming the best individual F1 score of 0.8614 (achieved by SelfCheckGPT-NLI).
Similarly, Brier and Accuracy also show the highest performance for multi-scoring.
For HaluEval, we also find that multi-scoring outperforms the best individual score (P(True)) in all metrics. 
BIG-Bench shows more nuance, as multi-scoring outperforms the best individual score (P(True)) on Brier and Accuracy, but not on F1. 
This reflects the fact that the metrics capture different properties, and practical considerations may require a decision as to which metric should be prioritized in a given application.
For FEVER, multi-score again outperforms the best individual score (P(InputContradict)) in all metrics. 

Thus we see that combining scores performs better than the top performing individual scores, showing that combinations of different scoring signals complement each other and can boost performance. 
This demonstrates that combining complementary signals enables more robust hallucination detection, while balancing the strength of each scoring method.
Crucially, while some settings may benefit from combinations of different signals (such as when trying to detect different kinds of hallucinations in a given generated output), in other settings the data-driven selection of an informative signal may be sufficient (such as when trying to detect a more narrowly defined notion of hallucination). 
These scenarios are covered by our use of multi-scoring, which allows for arriving at optimally combined hallucination scores (or binary decisions) for a given application.
Meanwhile, as computational cost can be a concern with scaling LLM applications, it may not be desirable to always use a full ensemble of scores, and we would rather compute the most performant score at a fixed computational cost.



 §.§ Cost-effective Multi-Scoring

Shown in Table <ref>, cost-effective multi-scoring methods are also amongst the top performers, but at considerably lower cost compared to multi-score. We see that at cost C=1, measured as the number of LLM calls (but see our discussion about measuring cost above), we recover the best individual scores, which varies across each dataset. As we increase the budget, the cost-effective multi-scores converge to the performance of the multi-score itself. Notice that in several instances, cost-effective multi-score with C=2 already performs as good as multi-score itself. 
For TriviaQA, we see that performance increases on all metrics as we increase the computational budget, with cost-effective multi-score at C=5 outperforming SelfCheckGPT-NLI as the most performant individual scores at half the computational cost in all metrics but Brier.
HaluEval shows no improvement beyond a budget of C=2, which is likely due to the kind of hallucination to detect being more narrowly defined and well-capture by a small number of signals, thereby not benefiting from additional scores. 
For BIG-Bench, as discussed for multi-score, we recover the best individual score at C=1, and match multi-score at higher budgets.  
FEVER results indicate that, again, we recover the best individual score at budget C=1, and can already recover the full multi-score at a budget of C=3.

We now take a closer look at cost-effect multi-scoring when including multi-generation scoring methods.
The overall cost budget is varied over the entire range. For each budget, we solve the constrained optimization in Eq. <ref> to find the optimal subset S^* of scores.
We compare the hallucination detection F1 score achieved by the cost-effective ensemble versus individual scoring functions and the full ensemble with all scores. Figure <ref> shows the results for TriviaQA for responses generated via . With a minimal budget of B=1, cost-effective selection recovers the best single method, as expected. As the budget increases, it selectively adds more expensive functions, gradually improving F1, though the gains are marginal at higher budgets per unit cost. At the maximum budget, it recovers the unconstrained full ensemble performance. In between, cost-effective multi-scoring incorporates both less and more and expensive methods to maximize detection within the computational constraints.



These results demonstrate that the proposed cost-effective multi-scoring approach can intelligently balance the trade-off between computational expense and hallucination detection effectiveness. It outperforms individual scoring functions and makes selective use of more costly scores to maximize detection performance under a fixed computational budget.

A key consideration in practice would be whether to include potentially more costly multi-generation methods, such as SelfCheckG­PT-NLI. 
In particular as implied by the results presented in Table <ref>, we may be interested in reducing the number of required multiple generations while matching performance at lower computational costs. 
To this end, we compare SelfCheckGPT (as the single best performing method for TriviaQA) alone with SelfCheckGPT combined with P(True) at different numbers of additional responses generated via . 
Here, we count the evaluation of P(True) as one additional LLM call, just as generating one additional output from the generator LLM. 
As presented in Figure <ref>, at one additional generation, SelfCheckGPT (in the degenerate case of only one generated output) combined with P(True) thus recovers P(True). 
As the number of additional generations increases, the combination quickly outperforms SelfCheck alone. 
In particular, we observe that cost-effective multi-scoring with 3 LLM calls is already as good as SelfCheckGPT with 9 LLM calls in our experiment.
This highlights how we can save costs by combining multi-generation methods with other scores while requiring fewer additional generated responses from the generator LLM than if we wanted to achieve the same performance with multi-generation methods alone. 





 §.§ Exploration of Relationships between Scores


As discussed above, different scoring methods can target different kinds of hallucations. 
To explore this empirically, we here explore their relationships via Spearman rank correlations. 
As presented in Figure <ref>, calibrated scores across all different individual scores are positively correlated. 
Meanwhile, the magnitude of their correlations is smaller than would perhaps be  expected if one were to consider hallucinations as a uniform phenomenon. 
At the same time, we can see certain “clusters” of more strongly inter-correlated scores emerge.
For example, multi-generation methods including SelfCheckGPT, HallucinationRail and SimilarityDegree, which check the consistency across multiple generations, show comparably high correlations. 
P(True) and P(True) Verbalised and P(InputContradict) emerge as a similarly correlated cluster of stronger correlations. 
Overall, this supports the idea that different scores can capture distinct information, and the need to empirically select appropriate (combinations of) scores for a given application, as we propose with cost-effective multi-scoring.






 §.§ Experiments across Different LLMs

Most of the scoring methods considered in this work rely on detector LLMs to compute scores for hallucination detection. 
In Table <ref> of Appendix <ref>, thus present additional data for different LLMs, namely ,  and .
Overall, we see that hallucination detection performance is correlated with performance on general LLM benchmarks[E.g., <https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard>]. 
Thus, more overall capable LLMs are likely to perform better at hallucination detection than less capable ones.
However, concrete applications may need to take the inference cost of LLMs into account, where one may not always be able to use the most expensive model. 
We note that cost-effective multi-scoring could here also make use of scores computed via LLMs with different computational costs to find cost-effective combinations. 



 §.§ The Importance of Calibration

To analyze the impact of calibration, we conduct an ablation study by evaluating model performance with and without calibrating the individual scores.
Table <ref> in Appendix <ref> indicates that model performance clearly drops in most of the datasets and metrics without calibration. This demonstrates the benefit of calibration for more accurate and risk-aware downstream applications.



 §.§ Exploring Multi-Generation Assumptions





Our results indicate that methods based on the uncertainty among multiple generations can provide strong signals when such multiple generations are available. 
However, such methods can also suffer from problems in practice. 
Methods based on multiple generations make use of the uncertainty of the generator, that is the distribution over generations z given the input x, for model 𝒢, i.e., p_𝒢(z|x) and the consistency between actual sampled generations z. 
We are interested in classifying whether the candidate output contains a hallucination or not, 
that is estimating and making use of p(y |x, z). 
The generating model's uncertainty p_𝒢(z|x) can be a useful proxy for p(y |x, z), but only under particular conditions. 

On a technical level, sampling multiple answers requires access to the generator's temperature parameter, as temperatures of zero collapse the generations to a single response, as is mentioned, e.g., in <cit.>. 
Therefore, these methods are technically grey-box models, as they require some access to the model's inference parameters. 

More conceptually, the generator's uncertainty (or confidence <cit.>) over p_𝒢(z|x) reflects uncertainty over generated tokens. 
All recently proposed multi-scoring methods we are aware of are based on the idea of scoring the consistency across multiple generations using model-based metrics. 
However, self-consistency across multiple generations is neither a necessary nor sufficient criterion for a hallucination to be present. 
Sufficiency is not given as a model may consistently provide an incorrect response. 
Self-consistency across multiple generations is also not necessary, as many tasks allow for multiple hallucination-free answers that are contradictory to each other. 

As a relatively harmless example, in tasking a model to generate cooking recipes for lunch, the model may generate, among other things, a recipe for a salad and a recipe for a soup. Clearly, the steps in preparing these dishes contain contradictory information, while otherwise being free of hallucinations in themselves. 
Thus, if there is more than one correct response, the self-consistency assessment in multi-generation methods may falsely score an output as likely to be hallucinated. 
As there are numerous situations where there exist multiple correct responses, assuming that there is only one could lead to worse LLM responses, also via a loss of diversity.

Practically, methods based on multiple generations can be costly due to added computational overhead, as we have seen above.
Finally, these methods are based on the assumption that LLMs are calibrated. 
As such, multi-generation methods do not allow for detecting cases where the generator LLM is confident yet wrong. 









§ CONCLUSION


In this work, we compared a comprehensive set of scoring methods to provide calibrated probability scores for the presence of hallucinations in generated LLM outputs.
Overall, we have observed that no single hallucination detection score performs best across all datasets.
Our experiments showed that combinations of scores, as suggested with multi-scoring, can effectively combine complementary signals to yield higher hallucination detection performance than any individual score.
Further, we demonstrate that cost-effective multi-scoring can find the highest performing scores at a given computational budget.

More generally, our findings support the notion that hallucinations can be rather multi-faceted than present a uniform phenomenon.
Thus, detecting hallucinations may require different methods.
In concrete settings, one may be interested in even more fine-grained detection of particular types of hallucinations, such as in specific domains like code generation <cit.>.

The approach outlined in this work requires only a small amount of labeled data to calibrate hallucination scores and combine scores via (cost-effective) multi-scoring.
However, it is important to acknowledge the limitations of the current work. Future research could explore the effectiveness of this approach on more diverse datasets, investigate alternative scoring methods, or extend the methodology to multi-modal models. 
While improvements in LLM performance can be expected to also lower their rate of occurrence, hallucinations are unlikely to go away completely <cit.>.
Thus, even as models improve further, detecting hallucinations is likely to remain relevant for applying LLMs in a reliable and trustworthy manner in future.

In summary, our work presents a promising approach for detecting hallucinations in LLM outputs by combining multiple scoring methods in a cost-effective way, offering a pathway towards more reliable and trustworthy language models that can be applied in real-world settings with more confidence.


















§ ADDITIONAL RESULTS














 Paper Title: Learning to Embed Distributions via Maximum Kernel Entropy.




Floquet Engineering of Polaritonic Amplification in Dispersive Photonic Time Crystals
    Vladimir M. Shalaev
    August 5, 2024
=====================================================================================




§ ABSTRACT 
Empirical data can often be considered as samples from a set of probability distributions. Kernel methods have emerged as a natural approach for learning to classify these distributions. Although numerous kernels between distributions have been proposed, applying kernel methods to distribution regression tasks remains challenging, primarily because selecting a suitable kernel is not straightforward. Surprisingly, the question of learning a data-dependent distribution kernel has received little attention. In this paper, we propose a novel objective for the unsupervised learning of data-dependent distribution kernel, based on the principle of entropy maximization in the space of probability measure embeddings. We examine the theoretical properties of the latent embedding space induced by our objective, demonstrating that its geometric structure is well-suited for solving downstream discriminative tasks. Finally, we demonstrate the performance of the learned kernel across different modalities.




§ INTRODUCTION

Most discriminative learning methods conventionally assume that each data point is represented as a real-valued vector. In practical scenarios, however, data points often manifest as a 'set' of features or a 'group' of objects. A quintessential example is the task of predicting a health indicator based on multiple blood measurements. In this case, the single data point of a patient has multiple, or a distribution of, measurements. One approach to accommodate such cases involves representing each input point as a probability distribution. Beyond mere convenience, it is more appropriate to model input points as distributions when dealing with missing data or measurement uncertainty, as often encountered when facing the abundance of data, which commonly presents a challenge for data-rich fields such as genetics, neuroscience, meteorology, astrophysics, or economics.


The task of regressing a mapping of probability distributions to a real-valued response is known as distribution regression. Distribution regression has been successfully applied in various fields, such as voting behavior prediction <cit.>, dark matter halo mass learning <cit.>, human cancer cells detection <cit.>, brain-age prediction <cit.>, among others <cit.>. The versatility and effectiveness of this framework underscore its power in solving complex problems <cit.>. Kernel methods have become a widely used approach for solving distribution regression tasks by exploiting a kernel between distributions  referred to as a distribution kernel. Despite the multitude of proposed kernels, the practical application of kernel methods remains challenging due to the nontrivial choice of the appropriate kernel. While some efforts focus on identifying kernels with broad applicability and favorable statistical properties <cit.>, others aim to tailor kernels to the geometric characteristics of specific input spaces <cit.>. Remarkably, the question of learning data-dependent kernels has received limited attention. This study is thus driven by a fundamental question: What are the underlying principles that facilitate the unsupervised learning of an effective kernel, one that optimally encapsulates the data properties and is well suited for discriminative learning on distributions?

In this work, we leverage a key insight: an appropriate selection of the distribution kernel enables the embedding of a set of distributions into the space of covariance operators. Building on this theoretical idea, we claim that quantum entropy maximization of the corresponding covariance operator is a suitable guiding principle to learn data-dependent kernel. This, combined with a careful design of kernel parametrization, let us to devise a differentiable optimization objective for learning data-specific distribution embedding kernels from unlabeled datasets (i.e. unsupervised) (<Ref>). We show that the entropy maximization principle facilitates learning of the latent space with geometrical configuration suitable for solving discriminative tasks <cit.>. We empirically demonstrate the performance of our method by performing classification tasks in multiple modalities.

In summary, our unsupervised data-dependent distribution kernel learning framework introduces a theoretically grounded alternative to the common practice of hand-picking kernels. Such framework could be further leveraged for generalizing existing learning approaches <cit.> catalyzing the use of distribution-based representations within the broader scientific community.



§ PRELIMINARIES

We first introduce the main concepts necessary to formalize our learning framework: kernel mean embeddings and covariance operator embeddings.



 §.§ Kernel Embeddings of Distributions


Consider an input space 𝒳 and a positive-definite (p.d.) kernel k: 𝒳×𝒳→ℝ. Let ℋ be the corresponding reproducible kernel Hilbert space (RKHS) induced by such kernel. Consider a probability distribution P∈𝒫(𝒳). The kernel mean embedding map embeds the distribution P as a function in Hilbert space:


    μ_P≡μ(P) ∫_𝒳 k(x, ·)   dP(x) = ∫_𝒳ϕ(x)    dP(x) ,

where ϕ: 𝒳→ℋ is a feature map such that ϕ(x) = k(x, ·).

Importantly, if the kernel k is characteristic <cit.>, the mapping μ: 𝒫(𝒳) →ℋ is injective, implying that all information about the original distribution is preserved in ℋ. This last property underscores much of power under recent applications of kernel mean embeddings <cit.>. 
The natural empirical estimator for the kernel mean embedding approximates the true distribution with a finite sum of Dirac delta functions:

    μ̂_P1/N∑_i=1^N ϕ(x_i) ∈ℋ

where x_1, …, x_N ∼P are N empirical i.i.d. samples. The estimator has a dimension-free sample complexity error rate of 𝒪(N^-1/2) <cit.>.

Additionally, we denote d_k: 𝒳×𝒳→ℝ_≥ 0 a kernel metric induced by a kernel k,

    d_k(x, x^') = ϕ(x) - ϕ(x^') _ℋ  .

Note that d_k is a metric on 𝒳 if feature map ϕ is injective.



 §.§ Covariance Operators and Entropy

A second way of mapping a probability distribution to a Hilbert space can be defined by means of a covariance operators. For a given feature map ϕ(x) = k(x, ·): 𝒳→ℋ[Assuming k to be a continuous positive definite (p.d.) kernel and 𝒳 to be compact.], and a given probability distribution P∈𝒫(𝒳), the covariance operator embedding is defined as:

    Σ_P∫_𝒳ϕ(x) ⊗ϕ(x)   dP(x)

where ⊗ is a tensor product. Σ_P is a self-adjoint positive semi-definite (p.s.d.) operator acting on ℋ. Such operator can be seen as a mean embedding w.r.t. the feature map x ↦ϕ(x) ⊗ϕ(x) and therefore, for a universal kernel k, the map P↦Σ_P is injective (see <cit.>).

Similarly to <Ref>, the natural empirical estimator is:

    Σ̂_P = 1/N∑_i=1^Nϕ(x_i) ⊗ϕ(x_i)

where x_1, …, x_N ∼P are N i.i.d. samples.

For a translation-invariant kernel k(x, x^') = ψ(x - x^') normalized such that k(x, x) =  1, the covariance operator Σ_P is a density operator <cit.>. Henceforth, entropy measures can be applied to it, and the quantum Rényi entropy of the order α can be defined as:

    𝒮_α(Σ_P) 1/1 - αlogtr[ (Σ_P)^α]  = 1/1 - αlog∑_iλ_i^α

where {λ_i }_i are the eigenvalues of Σ_P.
The Von Neumann entropy can be seen as a special case of Rényi entropy in the limit α→ 1. However, in our work, we focus primarily on the second-order case of Rényi entropy, i.e. α = 2 (see <cit.> for an in-depth overview of the properties and theory of quantum entropies).



§ UNSUPERVISED DISTRIBUTION KERNEL LEARNING



 §.§ Distribution Regression


In this section, we discuss the key topics in distribution regression, including problem setup, the notion of a 2-stage sampling process, and the common solutions to regression employing kernel methods.

Distribution regression extends the common regression framework to the setup where covariates are given as probability distributions available only through samples. Formally, consider the task of finding a regressor f: 𝒫(𝒳) →𝒴 from the dataset of samples Γ_M = {(P_i, y_i)}_i=1^M where P_i ∈𝒫(𝒳) are distributions provided as a set of i.i.d. empirical samples x_1, …, x_N_i∼P_i (see <cit.> for a comprehensive analysis). A viable approach to solving this problem is to define a kernel K: 𝒫(𝒳) ×𝒫(𝒳) →ℝ that is universal in 𝒫(𝒳). By setting up such a kernel K, we can utilize kernel-based regression techniques, with SVM often being the preferred method for classification tasks <cit.> or Kernel Ridge Regression (KRR) when the space 𝒴 is continuous <cit.> [As pointed out in <cit.>, under mild conditions, KRR can also be used for classification problems.]. To this end, several kernels have been proposed over time (see details in <Ref>). 

One possibility, proposed by <cit.>, is to introduce a kernel in the input space, the so called embedding kernel k_emb: 𝒳×𝒳→ℝ and exploit the induced mean embeddings μ_emb: 𝒫(𝒳) →ℋ_emb to map input distributions to points in RKHS. Subsequently, to define a second level kernel, distribution kernel K_distr: ℋ_emb×ℋ_emb→ℝ between points in the RKHS ℋ_emb. The simplest choice for such a distribution kernel is the linear kernel:

    K_l(μ_P, μ_Q) ⟨μ_P, μ_Q⟩_ℋ_emb
    = ∬_𝒳×𝒳 k(x, x^')   d P(x)   d Q(x^') .

A standard alternative to the linear kernel is a Gaussian kernel with bandwidth parameter λ > 0:

    K_RBF(μ_P, μ_Q) exp( - λ/2‖μ_P - μ_Q‖_ℋ_emb^2 ) = exp( - λ/2 d_k_emb(μ_P, μ_Q)^2 )   ,

which was shown to be universal in 𝒫(𝒳) <cit.>. The Gaussian kernel K_RBF can be computed from the linear kernel K_l as d_k_emb(μ_P, μ_Q)^2 = K_l(P, P) + K_l(Q, Q) - 2 K_l(P, Q).

In practice, we only have access to a finite number of samples of each input distribution, thus the distribution kernel is approximated using the natural estimator for the kernel mean embedding. The related excess risk for the regression solution is analyzed in <cit.>.





 §.§ Dataset Embedding


Instead of using standard kernels designed to encapsulate the geometry of the input space, we consider learning a data-dependent kernel, tailored to the specific properties of the dataset. In a similar vein, <cit.> proposed learning an optimal kernel (or equivalently, a feature map) jointly with the classifier to address the text modality. In this work, we focus on an unsupervised problem, aiming to learn a data-dependent kernel between probability distributions without access to classification labels.

We first introduce proper parametrization to ensure both expressivity and robustness followed by the definition of the optimization objective. Leveraging the idea of 2-level kernel setup, we define the embedding kernel as

    k_θ: 𝒳×𝒳→ℝ = k_emb(f_θ(x), f_θ(x^')) .

where f_θ is a trainable encoder function f_θ: 𝒳→𝒵, 𝒵 is a latent encoding space, and k_emb is a kernel defined on the latent space k_emb: 𝒵×𝒵→ℝ. The encoder function f_θ transforms every input probability distribution P∈𝒫(𝒳) into a distribution over the latent space ℙ_θ∈𝒫(𝒵) [By the definition of the encoding process, P_θ is a push-forward measure. For an empirical probability distribution q=∑_iδ(x_i)∈𝒫(𝒳) and a measurable map f: 𝒳→𝒵, the push-forward measure f # q ∈𝒫(𝒵) is defined as ∑_iδ(f(x_i)).] (<Ref>). Furthermore, we denote RKHS corresponding to the kernel k_emb as ℋ_emb and the kernel mean embedding map as μ_emb (see <Ref>).

    P∈𝒫(𝒳)  P_θ∈𝒫(𝒵)  μ_emb(P) ∈ℋ_emb .

These transformations define mean embeddings for each input probability distributions through the first level, embedding kernel k_emb (<Ref>). The second level, distribution kernel K_distr: ℋ_emb×ℋ_emb→ℝ is defined over the mean embeddings μ_emb(P)'s. We can now consider the embedding dataset 𝒟_M = {P_i }_i=1^M as an empirical covariance operator (see <Ref>), i.e.

    𝒟_M Σ̂_𝒟 = 1/M∑_P∈𝒟_MK_distr(μ_emb(P), ·) ⊗ K_distr(μ_emb(P), ·)  .

As Σ̂_𝒟 encapsulates information about the entire dataset, we term it dataset embedding. With the assumption that the dataset 𝒟_M is sampled i.i.d. from the (unknown) true meta-distribution 𝒟, Σ̂_𝒟 is a natural estimator to approximate the true covariance operator Σ_𝒟. To simplify the notation we use Σ_𝒟 in place of Σ_𝒟 unless required by the context.

Both the embedding kernel k_emb and the distribution kernel K_distr remain fixed throughout the training, learning happens by adjusting the parametrization of the latent space encoder f_θ. Such separation ensures expressivity while conforming to all technical requirements for a distribution kernel. Throughout the paper, we make the following assumptions on the latent space 𝒵 and embedding kernel k.

Latent space 𝒵 is a compact subset of ℝ^d. Kernel k_emb: 𝒵×𝒵→ℝ is a p.d. characteristic translation-invariant kernel k_emb(z, z^') = f( z - z^'^2) such that -f^' is completely monotone on (0, ∞) (see Definition 2.2.4 of <cit.>) and ∀ z ∈𝒵:   k_emb(z, z) = 1.



To facilitate our optimization process (which will be explained shortly), we opt for 𝒵=𝕊^d-1 (the d-dimensional hypersphere) and Gaussian kernel both for k_emb and K_distr (see <Ref>). We retain other suitable choices as potential avenues for future research.



 §.§ Unsupervised Optimization Objective


This dataset level representation depends on the choice of first and second level kernels k, K and, in turn, on the trainable function f_θ parameterized by the set of parameters θ. In this work, we propose learning the parameters θ to maximize quantum entropy of the dataset embedding, i.e.,

    θ = {𝒮_2(Σ_𝒟)  - logtr[ (Σ_𝒟)^2 ] } .

As we will describe in brief, optimizing this target has clear benefits inherited from the underlying geometry of the setup. But, first, we show how to empirically compute 𝒮_2(Σ_𝒟). Building upon previous work <cit.>[See the proof of Proposition 6 in <cit.>.], we exploit the following property of the covariance estimator:

    tr[ (Σ_𝒟)^2 ] = tr[ ( 1/M K_𝒟)^2 ]

where K_𝒟∈ℝ^M × M is the distribution kernel matrix, with [K_𝒟]_ij = K_distr(μ_P_i, μ_P_j).  This equation follows directly from the fact that Σ_𝒟 and 1/M K_𝒟 share the same set of eigenvalues. Leveraging this relationship, we can define tractable unsupervised training loss, which we term Maximum Distribution Kernel Entropy (MDKE), with respect to the parameters of the encoder f_θ:

    ℒ_MDKE(θ)   - 𝒮_2(Σ_𝒟) = logtr[ ( 1/M K_𝒟)^2 ]  = log∑_i=1^Mλ_i^2 ( 1/M K_𝒟)  = log1/M K_𝒟_F^2

where the latter relies on the fact that the Frobenius norm A _F^2 = ∑_iλ_i^2(A), where λ_i(A) are eigenvalues A.

The MDKE objective is differentiable w.r.t. θ for commonly used kernels, provided that the encoder f_θ is differentiable as well. While the entropy estimator 𝒮_2(Σ_𝒟) is convex in the kernel matrix K_𝒟, the objective as a whole is generally not convex in θ. However, in practice, as we show in <Ref>, mini-batch Stochastic Gradient Descent (SGD) proves to be an effective method for optimizing this objective. The effectiveness of this optimization process is significantly influenced by the parameters of the Gaussian kernels. We elaborate on the methodologies for kernel bandwidth selection in <Ref>.

The Frobenius norm formulation in the loss <Ref> significantly reduces computational complexity. However, as we have observed in some of our experiments, it can lead to the collapse of small eigenvalues of K_𝒟, particularly near the optimal value of the objective. To address this challenge we introduced a regularized version of the loss ℒ_MDKE-R that incorporates optional regularization, based on the determinant K_𝒟 inspired by the connection with Fekete points (see details in <Ref>).



 §.§ Geometrical Interpretation
 

The optimization objective is specifically designed to minimize the variance within each distribution (inner-distribution variance) while simultaneously maximizing the spread of distributions over the compact latent space 𝒵=𝒮^d-1. This shaping of the distributions embeddings in the latent space facilitates easier separation in downstream tasks. In this section we show that the geometry of the optimal (w.r.t. the MDKE loss) configuration of mean embeddings in the RKHS attains describe properties. For doing so we leverage the notion of distributional variance  𝕍_ℋ (Definition 1 in <cit.>). 


For a set of M probability distributions 𝒟_M, distributional variance 𝕍_ℋ(𝒟_M) of the mean embeddings in the RKHS ℋ is given by

    𝕍_ℋ(𝒟_M) 1/Mtr[ G ] - 1/M^2∑_i=1^M∑_j=1^MG_ij ,

where G is the M × M Gram matrix of mean embeddings in ℋ, i.e. G_ij = ⟨μ_P_i, μ_P_j⟩_ℋ <cit.>.


Here we show that the distributional variance 𝕍_ℋ can be equally reformulated into two separate contributions:

    𝕍_ℋ(𝒟_M) ≡1/M∑_i=1^M‖μ_P_i‖_ℋ^2  - ‖μ_P̅‖_ℋ^2 ,

where P̅ denotes mixture distribution with elements of 𝒟_M being uniformly weighted mixture components (see proof in the <Ref>).

The relevance of distributional variance for MDKE objective is established by the following result. 


For a set of M probability distributions 𝒟_M, the second-order Rényi entropy 𝒮_2 of the empirical covariance operator embedding Σ̂_𝒟 induced by the choice of Gaussian distribution kernel K_RBF over points in the RKHS ℋ_emb, - as defined in <Ref>, - is upper bounded by the distributional variance 𝕍_ℋ_emb(𝒟_M), i.e.,


    1/2 γ𝒮_2(Σ̂_𝒟) ≤𝕍_ℋ_emb(𝒟_M)


where γ is the bandwidth of the distribution kernel K_RBF.

The proof of this proposition is provided in <Ref>. This result formalizes the fact that our objective increases distributional variance, pushing up the average squared norm of mean embedding of input distributions while minimizing squared norm of the mean embedding of the mixture. We further explore the geometrical implications of such optimization by formalizing connection between the variance of the distribution and the squared norm of the its mean embedding in RKHS.


Under <Ref>, the maximum norm of kernel mean embedding is attained by Dirac distributions {δ_z }_z ∈𝒵.

This result is trivial due to the fact that the set of mean embeddings is contained in the convex hull of { k_emb(z, ·) }_z ∈𝒵, and, under <Ref>, ∀ z ∈𝒵:  k_emb(z, ·) _ℋ_emb^2 = k_emb(z, z) = 1.


Under <Ref>, uniform distribution 𝒰(𝒵) is a unique solution of

    _P ∈𝒫(𝒵){μ_emb(P) _ℋ_emb^2 ≡∬_𝒵×𝒵 k_emb(z, z^')   d P(z)   d P(z^') }.


The key intuition here comes from the fact that minimization of the squared norm of mean embedding in RKHS could be seen as minimization of total interaction energy over the given surface where the potential is defined by the kernel k. Thus <Ref> is a special case of Theorem 6.2.1 of <cit.>. Similar setup was used w.r.t. Gaussian potential over the unit hypersphere in Proposition 1 from <cit.>. The reformulation in <Ref> together with <Ref> immediately suggests that the framework could be seen as an extension of the Hyperspherical Uniformity Gap <cit.> to infinite-dimensional spaces of probability distributions This extension maintains the goal of reducing variance among input distributions while maximizing the separability between their means. See <Ref> for a broader explanation of the connection.

More generally, utilizing the fact that under <Ref>, the kernel metric d_k_emb (see <Ref>) is a monotonically increasing function of Euclidean metric on the latent space, we establish a precise connection between the generalized variance <cit.> and the norm of the  mean embedding. Further details can be found in <Ref>.

An attempt to directly optimize 𝕍_ℋ_emb using SGD resulted in significantly weaker outcomes. While a thorough mathematical explanation necessitates further investigation, we contend that this issue aligns with the recurring challenge reported across various studies regarding direct optimization over Maximum Mean Discrepancy (MMD). We hypothesize that optimal solutions exhibit similar geometric configurations, while the entropy of the covariance operator providing a smoother objective. Nonetheless, 𝕍_ℋ_emb retains its value as an insightful and intuitive measure for describing the geometric configuration of the learned system.



 §.§ An Illustrative Example

We use a simple example to illustrate the connection between geometrical configurations of embedded distributions and distribution kernel entropy 𝒮_2(Σ_𝒟) (see <Ref>). We sample a number of points from 6 different Gaussian distributions and project on a sphere 𝕊^2 varying their projected variance γ. As γ decreases, the distributional variance of the overall distribution of Gaussians increases (<Ref>). For very small γ each distribution converges to a point (a Dirac distribution). This results in the entropy interpolating between lower and upper bounds, demonstrating how entropy behaves in response to changes in distribution variance. <Ref> showcases the behavior of two terms comprising distributional variance (<Ref>): the average kernel norm of the distributions alongside the kernel norm of the mixture. The increase in entropy and variance corresponds to a 'flattening' effect on the spectrum of the distribution kernel matrix. This example provides a simplified picture of how input distributions configurations influence kernel entropy.




 §.§ Limitations


Runtime complexity. The applicability of a data-dependent distribution kernel to solving discriminative tasks relies on the structure of the dataset being well-suited for distribution regression modeling. The model performs best when the number of input distributions is relatively small (e.g., thousands rather than millions), while the number of samples per distribution is large. It is crucial to note that the computational complexity of the proposed method, which is a common concern in practical applications, is most favorable for the tasks described. A detailed analysis of runtime complexity can be found in <Ref>.

Broader impact. We wish to emphasize that the distribution regression framework has emerged as a powerful tool for analysis and predictive modeling, especially in domains where traditional methods face challenges, including social science, economics, and medical studies. We urge researchers and practitioners applying distribution regression in these areas to give special consideration to issues such as bias, fairness, data quality, and interpretability, -  aspects that are currently under-researched in the context of distributional regression, largely due to the relative novelty.



§ RELATED WORK





 §.§ Distribution Regression

Distribution regression was introduced in <cit.>, while the seminal work of <cit.> provides a comprehensive theoretical analysis of this regression technique. A natural approach to solving distribution regression problems involves using kernels between measures. Notable examples include the Fisher kernel <cit.>, Bhattacharyya kernel <cit.>, Probability product kernel <cit.>, kernels based on nonparametric divergence estimates <cit.>, and Sliced Wasserstein kernels <cit.>. <cit.> proposed leveraging the mean embedding of measures in RKHS, and <cit.> provided theoretical guarantees for learning a ridge regressor from distribution embeddings in Hilbert space to the outputs. Distribution kernels have been successfully applied in various kernel-based methods, such as SVM <cit.>, Ridge Regression <cit.>, Bayesian Regression <cit.>, and Gaussian Process Regression <cit.>. They have also been adapted for different modalities like distribution to distribution regression <cit.>, sequential data <cit.>, and more. For an in-depth exploration of the diverse methodologies employed in distributional regression settings, we invite readers to consult <ref>.



 §.§ Matrix Information Theory

Quantum entropy, including Rényi entropy, is a powerful metric to describe information in a unique way (see <cit.> for foundational insights). <cit.> designed the measure of entropy using operators in RKHS to mimic Rényi entropy's behavior, offering the advantage of direct estimation from data. <cit.> applied von Neumann entropy of the density matrix to the covariance operator embedding of probability distributions, thereby defining an information-theoretic framework utilizing kernel methods. In machine learning, especially within self-supervised learning (SSL) setups, entropy concepts have recently found novel applications. Our study builds on most recent developments <cit.> by applying quantum Rényi entropy to the covariance operator in RKHS.



§ EXPERIMENTS


We here demonstrate that our proposed method successfully performs unsupervised learning of data-dependent distribution kernel across different modalities. The experimental setup is divided into two phases: unsupervised pre-training and downstream regression classification using the learned kernel.

For each dataset, we select a hold-out validation subset with balanced classes, while the remainder of the dataset is utilized for unsupervised pre-training. We use mini-batch ADAM <cit.> with a static learning rate of 0.0005. We report mini-batch based (instead of epoch based) training dynamics as our tasks do not require cycling over the entire dataset to converge to the optimal loss value. All experiments use Gaussian kernel both as an embedding kernel and distribution kernel, the hyperparameter selection is performed as described in <Ref>.

Once the samples encoder f_θ is learned, we employ it to compute distribution kernel Gram matrix, used as an input to the Support Vector Machine (SVM) for solving downstream classification tasks. A grid search with 5 splits (70/30) is conducted to optimize the strength of the squared l_2 regularization penalty C, exploring 50 values over the log-spaced range {10^-7, …, 10^5}. The best estimator is then applied to evaluate classification accuracy on the validation subset, which we report.

Additional experiments exploring the application of data-dependent distribution kernels in domains where distribution regression models are less common, such as image and text, are presented in <Ref>.



 §.§ Flow Cytometry



Flow Cytometry (FC) is a widely used technique for measuring chemical characteristics of mixed cell population. Because population-level properties are described through (randomized) sampling of cells, FC is used as a canonical setup of distribution regression. For this study we used a dataset <cit.> where more than 100.000 cells are measured per each patient (subject). For each cell a total of ten parameters are reported, hence, we treated each subject as an empirical distribution over ℝ^10. We considered downstream classification tasks on two different sets of labels. The first ('Tissue' classification) contains peripheral blood (pB) and bone marrow (BM) samples from N=44 subjects. The second ('Leukemia' classification) presents healthy and leukemia BM cell samples, N=50. Classes were balanced in both cases.
l8.2cm
 Distribution regression accuracy on Flow Cytometry datasets. Code available at https://github.com/hidden-for-review.






Model     2cTissue     2cLeukemia 
 2-5 
                   Acc.     Var.        Acc.     Var.    
 
GMM-FV             93.07%            ± 0.308            94.80%          ± 0.186                

SW1                87.10%            ± 0.530            95.07%          ± 0.111                

SW2                81.71%            ± 0.341            95.30%          ± 0.224                

MMD Linear         82.42%            ± 0.840            90.57%          ± 0.208                

MMD Gaussian       81.71%            ± 0.574            92.23%          ± 0.216                

MMD Cauchy         81.57%            ± 0.662            93.77%          ± 0.080                

MMD IMQ            82.89%            ± 0.698            91.43%          ± 0.217                

MDKE Rand        77.50%                ± 0.002                    89.50%                ± 0.003                

MDKE No Reg.    95.30%                ± 0.002                    92.46%                ± 0.002                
 
MDKE Reg.        98.89%                ± 0.010                    94.57%                ± 0.005                
 








We sampled 16 subjects for Tissue classification and 20 subjects for Leukemia for training. Unsupervised learning was performed over the entire dataset. The encoder f_θ was parametrized by a 2-layers neural network (NN) with ReLU nonlinearity and l_2 normalized output (on the unit hypersphere 𝕊^9). Per each subject we sampled a small percentage of cells, and we report performance for the sample size of 200. We repeated each training and testing phase for 100 times to track the variance induced by this aggressive subsampling.

To demonstrate the impact of unsupervised pre-training, we compared several methods across multiple configurations (<Ref>):

a) Kernels on distributions. This group includes Fisher kernels applied to parametric Gaussian Mixture Model (GMM) estimates, as suggested in <cit.>, along with Sliced Wasserstein-1 and Sliced Wasserstein-2 kernels <cit.>.

b) MMD kernels. Here, we employ a Gaussian embedding kernel for mean embeddings, marked as 'MMD' (Maximum Mean Discrepancy) in the table. This category includes various options for the distribution kernel, such as linear, Gaussian, Cauchy, and inverse multiquadrics.

c) MDKE. We explore various configurations of the encoder optimized with the MDKE objective. We report performance for randomly initialized encoder, and for unsupervised pre-trained encoder with and without regularization. Random initialization happens only once, and all subsequent accuracy measurements are taken using the same encoder.

The variance reported for each model is measured across multiple runs to demonstrate the effect of the sampling. Importantly, the optimization of the MDKE objective results in embedding kernels with significantly lower variance compared to non-data-specific kernels.




§ CONCLUSION

In this work, we presented an unsupervised way of learning data-dependent distribution kernel. While previous studies in distribution regression predominantly relied on hand-crafted kernels, our work, in contrast, demonstrates that entropy maximization can serve as a powerful guiding principle for learning adaptable, data-dependent kernel in the space of distributions. Our empirical findings show that this technique can not only serve as a pre-training step to enhance the performance of downstream distribution regression tasks, but also facilitate complex analyses of the input space. The interpretation of the learning dynamics induced by the proposed objective relies on a theoretical link between the quantum entropy of the dataset embedding and distributional variance. This theoretical link, which we have proven, enables us to approach the optimization from a geometrical perspective, providing crucial insights into the flexibility of the learned latent space encoding.

We hope that theoretically grounded way of learning data-dependent kernel for distribution regression tasks will become a strong alternative to the common practice of hand-picking kernels. More broadly, our results present a methodology for leveraging the distributional nature of input data along side the novel perspective on the encoding of complex input spaces. This highlights the potential to extend the application of more advanced learning methods, embracing the ever-increasing complexity of data by going beyond more conventional vector-based representations.



  
*Acknowledgments. 
We thank the Allen Institute for Brain Science founder, Paul G. Allen, for his vision, encouragement, and support.

abbrvnat







§ PROOFS


In this section, we present the proofs for the propositions outlined in our study. To ensure clarity, we will first restate the setup and introduce necessary concepts.

We define the input space as 𝒳, and 𝒫(𝒳) represents the space of probability distributions over 𝒳. Consider a dataset of M probability distributions, denoted as 𝒟_M = {P_i ∈𝒫(𝒳) }_i=1^M. With a p.d. characteristic embedding kernel k: 𝒳×𝒳→ℝ, the corresponding RKHS ℋ, and the feature map ϕ: 𝒳→ℋ, we define the mean embedding map μ: 𝒫(𝒳) →ℋ such that μ_P = μ(P) ∫_𝒳ϕ(x)   d P(x). A p.d. translation-invariant characteristic distribution kernel K: 𝒫(𝒳) ×𝒫(𝒳) →ℝ is defined using the mean embeddings of corresponding distributions. For simplicity, μ_i denotes μ(P_i).

Additional concepts essential for our proofs include the Gram matrix of mean embeddings G ∈ℝ^M × M, representing the inner products of mean embeddings in the dataset, i.e., G  [⟨μ_i, μ_j ⟩_ℋ]_ij. The kernel matrix K_𝒟∈ℝ^M × M with respect to the distribution kernel K is denoted as K_𝒟 [K(P_i, P_j)]_ij. We also recall the definition of distributional variance 𝕍_ℋ (see <Ref>):

    𝕍_ℋ(𝒟_M) 1/Mtr[G] - 1/M^2∑_i=1^M∑_j=1^MG_ij

These definitions and notations will be referenced throughout the proofs.



 §.§ Kernel Norms Gap and Distributional Variance


For both cases of input distributions being empirical probability distributions or continuous densities, we define mixture distribution P̅, with a slight abuse of notation:

    P̅(x) 1/M∑_i=1^M P_i(x)


For the mixture distribution P̅ and the Gram matrix G, the following relationship holds:

    μ_P̅_ℋ^2 = 1/M^2∑_i=1^M∑_j=1^MG_ij



Proof. We begin by recalling that the inner product between mean embeddings μ_i and μ_j in ℋ is given by:

    ⟨μ_i, μ_j ⟩_ℋ = ∬_𝒳×𝒳k(x, x^')   d P_i(x)   d P_j(x^')


Substituting this into the expression for the Gram matrix, we have:

    1/M^2∑_i=1^M∑_j=1^MG_ij   = 1/M^2∑_i=1^M∑_j=1^M∬_𝒳×𝒳k(x, x')    d P_i(x)   d P_j(x')
       = ∬_𝒳×𝒳 k(x, x')    d ( 1/M∑_i=1^MP_i(x) )   d ( 1/M∑_j=1^MP_j(x') ) 
       = ∬_𝒳×𝒳 k(x, x')    d P̅(x)   d P̅(x')  = μ_P̅_ℋ^2


This completes the proof of <Ref>.

By incorporating <Ref> into the definition of distributional variance (see <Ref>), and noting that the trace of G is the sum of squared norms of input distributions (i.e.,  tr[G] = ∑_i=1^M μ_P_i_ℋ^2), we obtain:

    𝕍_ℋ(𝒟_M) ≡1/M∑_i=1^Mμ_P_i_ℋ^2 - μ_P̅_ℋ^2

This result offers a new and intuitive perspective on distributional variance, conceptualizing it as the difference between the average squared norm of individual input distributions and the squared norm of the mixture distribution.



 §.§ Pairwise Distance and Distributional Variance




For a dataset 𝒟_M of M probability distributions, we define the average pairwise distance between kernel mean embeddings in ℋ as 𝕁_ℋ(𝒟_M), given by:

    𝕁_ℋ(𝒟_M) 1/M^2∑_i=1^M∑_j=1^M||μ_i - μ_j||_ℋ^2



For the distributional variance 𝕍_ℋ of the dataset 𝒟_M, the following relationship holds:

    𝕍_ℋ(𝒟_M) ≡1/2·𝕁_ℋ(𝒟_M)



Proof. Starting with the definition of 𝕁_ℋ(𝒟_M):

    𝕁_ℋ(𝒟_M)    = 1/M^2∑_i=1^M∑_j=1^M||μ_i - μ_j||_ℋ^2
       = 1/M^2∑_i=1^M∑_j=1^M⟨μ_i - μ_j, μ_i - μ_j ⟩_ℋ
       = 1/M^2∑_i=1^M∑_j=1^M( ⟨μ_i, μ_i ⟩_ℋ + ⟨μ_j, μ_j ⟩_ℋ - 2⟨μ_i, μ_j ⟩_ℋ) 
       = 1/M^2( 2 M ∑_i=1^M⟨μ_i, μ_i ⟩_ℋ - 2∑_i=1^M∑_j=1^M⟨μ_i, μ_j ⟩_ℋ) 
       = 2 ( 1/M∑_i=1^M⟨μ_i, μ_i ⟩_ℋ_diagonal of G - 1/M^2∑_i=1^M∑_j=1^M⟨μ_i, μ_j ⟩_ℋ)

Recognizing that ∑_i=1^M⟨μ_i, μ_i ⟩_ℋ = tr[G], we can equate expression in the brackets to 𝕍_ℋ(𝒟_M), leading to:

    𝕍_ℋ(𝒟_M) ≡1/2·𝕁_ℋ(𝒟_M)

<Ref> will be instrumental in further proofs, establishing a crucial link between distributional variance in ℋ and quantum entropy of the covariance operator embedding Σ_𝒟.



 §.§ Distribution Kernel Entropy Upper-Bound


In this section, we provide the proof for the key theoretical result stated in <Ref>.

Consider a dataset 𝒟 consisting of probability distributions {P∈𝒫(𝒳) }_i sampled i.i.d. from an unknown meta-distribution 𝔻. We assert that the second-order Rényi entropy 𝒮_2 of the empirical covariance operator embedding Σ_𝒟, induced by the choice of Gaussian distribution kernel K_RBF over points in the RKHS ℋ, is upper-bounded by the distributional variance 𝕍_ℋ(𝒟):

    1/2 γ𝒮_2(Σ̂_𝒟) ≤𝕍_ℋ(𝒟)

where γ is the bandwidth of the distribution kernel K_RBF.

Starting from the properties of 𝒮_2(Σ̂_𝒟) stated in  <Ref>:

    𝒮_2(Σ̂_𝒟) = 𝒮_2 ( 1/M K_𝒟) = -log∑_i=1^M∑_j=1^M( 1/M K_ij) ^2  = -log( 1/M^2∑_i=1^M∑_j=1^MK_ij ^2)


Applying Jensen's inequality, considering the concavity of the log, to <Ref>, we have:

    𝒮_2(Σ̂_𝒟)    ≤ - 1/M^2∑_i=1^M∑_j=1^MlogK_ij^2
       = - 1/M^2∑_i=1^M∑_j=1^Mlogexp(-γ/2μ_i - μ_j_ℋ^2)^2
       = - 1/M^2∑_i=1^M∑_j=1^Mlogexp(-γμ_i - μ_j_ℋ^2)
       =  γ( 1/M^2∑_i=1^M∑_j=1^Mμ_i - μ_j_ℋ^2) 
       = γ·𝕁_ℋ(𝒟) 
       = 2 γ·𝕍_ℋ(𝒟)

<Ref> uses the definition of K_RBF from <Ref>, <Ref> is derived from the definition of the average pairwise distance in <Ref>, and <Ref> follows from <Ref>. This completes the proof of <Ref>, establishing the upper bound for the second-order quantum Rényi entropy of the covariance operator embedding Σ_𝒟 in terms of the distributional variance in RKHS ℋ. 

Technical Note: It is important to note that 𝒮_2 is typically measured using the base-2 logarithm (log_2) rather than the natural logarithm. However, the proof remains accurate with a proper re-scaling to account for the change in the logarithmic base.



 §.§ Generalized Variance in RKHS



In this section we prove the connection between generalized variance and norm of the kernel mean embedding.

So far we established the squared norm of mean embedding is maximized by Dirac points (zero variance) and is minimized by uniform distribution (max variance). We now formalize the intuition that larger squared norm in embedding kernel RHKS corresponds to smaller variance in the latent space. We first note that under  <Ref>, the kernel metric d_k_emb (see <Ref>) induced by the choice of k_emb is a monotonically increasing function of Euclidean metric on the latent space, thus is representative of the geometry of encoded input distributions. Which let us use a generalized notion of variance here.



(<cit.>) Let X be a random variable which takes values in a Fréchet space ℱ equipped with seminorm ·_α. And suppose that X is square-integratable, in a sense that 𝔼 X _α^2 < ∞. Let μ∈ℱ be a Pettis integral of X (i.e. generalization of the mean). Generalized variance of X w.r.t. seminorm ·_α is defined as following

    Var_α[X] 𝔼 X - μ_α^2   .



Note that for a random variable in RKHS defined as a push-forward of a probability distribution P∈𝒫(𝒵) over the latent space, i.e. X = z ∼ϕ#P satisfies conditions of <Ref> with ℱ = ℋ, ·_α = ·_ℋ_emb, and a Pettis integral being a kernel mean embedding. Denote the described variance as Var_ℋ_emb[P]. We now show that


Under <Ref>, for every P∈𝒫(ℤ),

    Var_ℋ_emb[P] = 1 - μ_emb(P) _ℋ_emb^2.





  
Proof.

    Var_ℋ_emb[P]    = 𝔼_x ∼ Pϕ(x) - μ_P _ℋ_emb^2 
       = ∫ϕ(x) - μ_P _ℋ_emb^2   d P(x) 
       = ∫⟨ϕ(x) - μ_P, ϕ(x) - μ_P ⟩_ℋ_emb  d P(x) 
       = ∫( ϕ(x) _ℋ_emb^2 + μ_P _ℋ_emb^2 - 2 ⟨ϕ(x), μ_P ⟩_ℋ_emb)   d P(x) 
       = ∫ϕ(x) _ℋ_emb^2   d P(x)  + ∫μ_P _ℋ_emb^2   d P(x)  - 2 ∫⟨ϕ(x), μ_P ⟩_ℋ_emb  d P(x) 
       = ∫ 1   d P(x)  + μ_P _ℋ_emb^2 ∫ 1   d P(x)  - 2  ⟨∫ϕ(x)   d P(x) , μ_P ⟩_ℋ_emb
       = 1 + μ_P _ℋ_emb^2 - 2 μ_P _ℋ_emb^2 
       = 1 - μ_P _ℋ_emb^2

Which concludes the proof.

The proposition demonstrates a motivation behind training the encoder to minimize variance in the latent space by maximizing the (average) squared norm of the mean embeddings.



§ PRACTICAL ASPECTS OF LEARNING





 §.§ Kernel Hyperparameter Selection


In practice, the effectiveness of the optimization process is significantly influenced by the parameters of the Gaussian kernels. Setting the bandwidth parameter γ either too low or too high can hinder the model's ability to learn effectively. This issue is a well-known challenge in working with kernel methods, where determining the optimal kernel bandwidth has been an area of extensive study.

In our work, we have employed an empirical approach, which involves adjusting γ based on the idealized structure of the dataset after its projection onto the latent space. Specifically, we set γ such that 1/γ equals 10 times the average distance to the nearest neighbor in the set of points sampled uniformly on 𝕊^d-1 (inspired by the experimental approach in <cit.>). The number of points is chosen to be the number of distributions in the training set for the distribution kernel. For the embedding kernel, it is set to the number of distributions in a batch multiplied by the number of samples per distribution used for unsupervised training.



 §.§ Optional Regularization




As the theoretical maximum of 𝒮_2 entropy is attained when the spectrum of 1/M K_𝒟 is uniform, the optimal encoder with respect to ℒ_MDKE also tends to maximize the determinant of the kernel Gram matrix K_𝒟. This relationship is intuitive when considering that the determinant of a matrix is the product of its eigenvalues. The points that maximize such determinant K_𝒟 are theoretically known as Fekete points <cit.> and, in our case, are relative to the distribution kernel. Fekete points have been shown by <cit.> to be an optimal configuration for learning kernel interpolants, making them particularly suitable for downstream tasks framed as kernel regression. As such, encoders optimized under the ℒ_MDKE objective facilitate more accurate and robust performance in subsequent regression tasks.

In practice, we found that optimizing the MDKE objective posed certain numerical challenges, particularly due to the tendency of too small eigenvalues in the distribution kernel matrix K_𝒟 to collapse near the optimal value of the objective when using large batch size for training. To mitigate this issue and prevent undesirable optimization behavior, we have introduced a regularization term to the original objective. This term is inspired by the concept of Fekete points configuration, leading to the following loss function:

    ℒ_MDKE-R(θ)     -𝒮_2(Σ_𝒟) + ϵ·Ω(Σ_𝒟) 
       = log1/M K_𝒟_F^2 - ϵ·log| 1/M K_𝒟|


Here, ϵ serves as a hyperparameter to control the strength of regularization. The regularization term Ω(Σ_𝒟) is designed to stabilize the optimization process by counteracting the effects of the collapse of small eigenvalues. The empirical evidence supporting the effectiveness of regularization in stabilizing the training dynamics for the sentence representation learning experiment (as detailed in <Ref>) is showcased in <Ref>.



 §.§ Runtime Complexity


Scalability of the kernel methods is typically a key concern when it comes to practical applications. The runtime complexity of the proposed method could be decomposed into two components:


  * Computation of the distribution kernel Gram matrix. This involves computing O(N^2) inner products between distributions where N is a number of distributions. Each inner product involves computing matrix of pairwise distances between samples from each distribution, in the case of computing the Gaussian kernel over points on the hypersphere, the runtime complexity is the complexity of multiplying matrices ℝ^M × d where M is a number of samples. All computations could be efficiently parallelized.

  * Solving regression using distribution kernel Gram matrix. The complexity comes from matrix inverse and is typically estimated to be O(N^3) with a range of methods proposed to reduce the complexity <cit.>.

With this being said, we want to highlight that the runtime complexity of the proposed method is most favorable exactly for those tasks and datasets where distributional regression is an appropriate modeling approach. A significant  number of samples per  distribution ensures a high accuracy in approximating the kernel between a pair of distributions, and, at the same time, a relatively small number of distributions in the dataset alleviates issues related to storing the distribution kernel Gram matrix in memory and performing computations on it




§ CONNECTION TO OTHER FRAMEWORKS



 §.§ Distributional Regression Landscape



As the distributional regression task differs from common Machine Learning (ML) setups where inputs are given as vectors, the effective practical solution requires unique considerations.

The most obvious approach would be to ignore the fact that inputs are given as distributions and learn classifier on the space of samples, with a proper aggregation of posteriors (e.g. with a simple sum over histograms). This approach, while being simple, has been shown not to yield practically useful results, and was explicitly excluded from reported performance on different tasks by different authors <cit.>.


Viable approaches to solving distributional regression could be, approximately, split up into the following categories.



  
Discriminate generative models. The idea is to fit each input distribution to a parametric family, e.g. Gaussian Mixture Model (GMM), to use available closed-form solutions to compute kernel or similarity or distance between distributions. This group of methods originated from the work on hidden Markov models and it's applications to processing sequence modalities, like text, DNA, proteins, and more. Early work <cit.> on learning discriminative classifier for generative models leveraged the fact that parametric models forms a Riemannian manifold with local metric given by Fisher information, they derived kernel function termed Fisher kernel suitable for running SVM between generative models. Driving motivation was classification between hidden Markov models, with the developed method being applied to DNA and protein sequence analysis. Following the same modeling approach of analysing DNA sequences with discriminative models between Markov models, <cit.> proposed to use Bhattacharyya distance between distributions from exponential families to derived so-called Expected Likelihood Kernel. <cit.> explored the method of computing kernel between distributions as the integral of the product of pairs of distributions, termed Probability Product Kernel. A new family of kernels was applied to the same setup of discriminating task on text modality, hidden Markov models for biological data. The model was also successfully applied for analysis of linear dynamical systems for time series data. Critical advantage of this work was access to computationally effective way of computing kernel for the distributions without having access to analytical closed-form parametrization (only relying on samples). In <ref> we used fitting distributions to GMM with Fisher Kernel applied to learn parameters as a baseline for the performance on the task.



  
Point clouds. This group includes methods that model each input distribution as a set of points (also known as 'point cloud' or 'feature group' in computer vision) and use kernel or similarity function defined on sets. Large portion of the methods in this group arised in computer vision (CV) field when local features extractor were widely employed to pre-process images yielding either per-image histograms or sets of low dimensional vectors. The group includes kernels based on nonparametric divergence estimates, quantized set kernels, and so-called 'nearest-neighbor' set kernels to name a few <cit.>. Such kernel were employed to many CV tasks though the successfully application required not only a good kernel but also high-quality feature extractor. Special attention goes to methods leveraging  Wasserstein distance (including both kernel-based and similarity-based solutions). Wasserstein distance as a metric-aware discrepancy measure for probability distributions is a natural choice of deriving kernels or similarity functions between point clouds <cit.>. While being computationally problematic for large-scale problems, Sliced Wasserstein kernels were succesfully adopted <cit.> as they provided both reliable way for the point set comparison with a reduced cost leveraging sliced formulation. In <ref> we compared performance of Sliced Wasserstein-1 and Sliced Wasserstein-2 kernels, with the latter yielding significantly lower performance. Such a behavior is consistent with theoretical analysis stating that Sliced Wasserstein-1 has favoriable properties when compared to kernel based on Wasserstein-2 distance.



  
Kernel Mean Embeddidngs. <cit.> proposed to leverage  kernel mean embedding of measures in RKHS, so that the distributional regression could now be casted to a regression the corresponding Hilbert space. While the original work leveraged kernel between RKHS embeddings to train SVM, <cit.> provided theoretical guarantees for the  for learning Ridge regressor from distribution embeddings in the Hilbert space to the outputs. <cit.> applied the same approach in Bayesian regression settings, and <cit.> used it for Gaussian Process Regression. The setup gives a great deal of flexibility by choosing the kernel in RKHS, with a few being tested in practice. Gaussian kernel is a common choice, due to it's universality <cit.>. Among others, inverse multiquadric (IMQ) is a popular choice, typically paired with random features to improve runtime complexity of the algorithm. Linear kernel in RKHS, despite not being universal, was reported to produce competitive results in multiple practical setups. In <ref> we compared performance for 4 different kernels in RKHS, namely linear, Gaussian, Cauchy, and IMQ, -  all defined as functions of Maximum Mean Discrepancy (MMD), which gives as a way to compute the value of the kernel from finite number of samples with high accuracy. While all kernels demonstrated different performance in terms of the test accuracy, the difference reported is not that substantial.

Distribution kernels were successfully applied in different kernel-based methods, such as , as well as in different modalities, like distribution to distribution regression <cit.>, distribution regression for sequential data <cit.>, and more. In cases where input data is naturally represented as a probability distribution but not immediately applicable for existing distribution regression solutions, pre-processing or encoding of inputs is required. <cit.> proposed learning a latent space representation for input data to apply distribution regression in the text modality, with the encoder being trained jointly with the classifier. To our best knowledge, methods for learning data-dependent kernel for solving distributional regressions were not previously reported.



 §.§ Hyperspherical Uniformity Gap



The link between entropy maximization in the space of distribution, the gap between average norm and norm of the mixture distribution marginalized over the dataset (i.e. 'average' distribution), and properties of distributions that minimize and maximize kernel mean embedding norm (as described in <Ref>) unveils a subtle yet significant connection to the concept of the Hyperspherical Uniformity Gap (HUG), introduced in <cit.>. HUG has been developed to generalize the phenomenon of neural collapse observed in supervised classification settings. In our approach, working directly with samples from input distributions grants us explicit access to the 'grouping' of points in the input space. Even though the HUG framework setup does not have a notion of 'grouping', we can close the gap by noting that class labels provided with the dataset implicitly create 'groupings' of points, which can be interpreted as empirical samples drawn from latent probability distributions (one for each class).

However, a notable distinction lies in the dimensional aspect: HUG focuses on the distribution of points on the surface of a finite-dimensional hypersphere, whereas our work encompasses an infinite-dimensional hyperball setting, given the use of points in kernel induced RKHSs. Furthermore, the loss function introduced in our study presents a unified optimization objective that weaves together both the inner-group and inter-group dynamics while being derived from first principles. In the HUG framework these two aspects are addressed separately.

Establishing a more formal connection between these frameworks emerges as a promising direction for future research. Such an endeavor could offer a novel perspective on supervised classification, particularly by conceptualizing class prototypes as probability distributions rather than mere vectors. This exploration might bridge the gap between these distinct approaches, enriching our understanding of classification paradigms in high-dimensional spaces.




§ ADDITIONAL EXPERIMENTS



In the following section, we present additional experiments on learning data-dependent distribution kernels in the image and text domains. While representing text as empirical samples from the finite space of tokens (i.e., words from the dictionary) is quite common, the choice to model images as histograms over pixel positions is more subtle. We demonstrate that, in both scenarios, unsupervised pre-training of the encoder yields distribution kernel that achieves strong performance on downstream classification tasks, showcasing the versatility of the proposed learning framework in scenarios where distribution regression formulations are uncommon.



 §.§ Image Classification Tasks


MNIST <cit.> and Fashion-MNIST <cit.> consist of 28 × 28 pixel grayscale images divided into 10 classes. We considered each individual image to be a probability distribution (via rescaling pixel intensities so that l_1(image)=1) over the discrete space of pixel positions (i.e., histogram). Given this support space, the encoder f_θ is a discrete map which we implemented as a table lookup (i.e., embeddings) from pixel indices to the points on the hypersphere 𝒮^31. Embeddings were initialized by sampling points uniformly. Gradients of the MDKE objective with respect to the embeddings parameters were computed via automatic differentiation using projected gradient steps to ensure that the embeddings remain on the hypersphere. The small size of the support space enables the exact computation of the inner product between kernel mean embeddings of input distributions <Ref> and, subsequently, the distribution kernel <Ref> during both training and evaluation. This ensured a lower variance of the accuracy for the downstream classification.

Performing MNIST classification upon pre-training with our unsupervised encoder significantly improves the baseline (random initialization of latent embeddings) accuracy of 85.0% by reaching a plateau at 92.15%.

To understand the data-dependency of our encoding procedure, we analyzed the latent spaces of MNIST and Fashion-MNIST datasets.  The visualization of pixel-level interaction, computed using Gaussian kernel Gram matrix, reveals complex, dataset-specific interactions (<Ref>, <ref>). Spectral clustering using the kernel Gram matrix provides deeper insight into the pixel interaction landscape. The chart shows clusters of pixels with correlated intensities (<Ref>, <ref>), with the number of clusters set to 10 empirically.






 §.§ Text Classification Tasks


To assess our method's performance in a larger discrete support space, we utilized the "20 Newsgroups" <cit.>, a multi-class text classification dataset. We reduced the size of the dataset to 5 classes (resulting in 2,628 sentences and 38,969 unique words) by subsampling both training and test subsets. We treated sentences as empirical distributions over words, assuming word sets to be enough for topic classification, despite no positional info.

The encoder f_θ mirrored the setup used in the MNIST case (<Ref>), with l_2 normalized word embeddings on 𝕊^31.

However, while in the MNIST case embeddings computations were performed exactly, here considering the entire embedding kernel Gram matrix is impractical due to its large size. Instead, we optimized embeddings by randomly sampling 20 words per sentence, making the inner product between embeddings a stochastic approximation. This setup is meant to confirm that the optimization of the proposed MDKE objective yields a solution that is robust w.r.t. the excessive risk induced by first-level subsampling. 

During the unsupervised pre-training phase, we observed a steady decrease in the training loss (<Ref>) despite the small batch size (50 sentences with total of 1,000 words). Importantly, for this experiment we employed a regularized version of the objective (MDKE-R). We discuss the rationale and empirical evidence supporting the use of this regularization scheme in <Ref>. During evaluation, to reduce computational complexity, we sample 2,000 sentences for the train and 1,000 for the test split, keeping classes balanced. The maximum classification accuracy achieved was approximately 89.3%, while the random initialization performance averaged at 37. 5% (<Ref>). The framework’s high accuracy in downstream classification showcases its prowess in learning potent latent representations, even when dealing with large, finite input distributions.





§ IMPLEMENTATION DETAILS



In this section, we provide an example illustrating the implementation of the proposed method using the PyTorch framework. All experiments were performed on a single machine with 1 GPU and 6 CPUs.

Functions to compute distribution kernel Gram matrix:


def pairwise_kernel(x, gamma1):
    B, T = x.size(0), x.size(1)
    X_unroll = x.reshape(B*T, -1)
    dist = (X_unroll[:, None, :] - X_unroll[None, :, :])**2
    dist = torch.sum(dist, dim=2)
    G = F.avg_pool2d(
        dist[None, : , :],
        kernel_size=(T, T),
        stride=(T, T)
    )
    return torch.exp(-(gamma1/2.) * G.squeeze(0))

def distribution_kernel_gram(x, gamma1, gamma2):
    Gxy = pairwise_kernel(x, gamma1=gamma1)
    Gx = Gy = torch.diag(Gxy)
    G = Gx[:, None] + Gy[None, :] - 2*Gxy
    K = torch.exp(-(gamma2/2.) * G)
    return Gxy, K


Distribution kernel entropy estimator and the MKDE loss:


def distribution_kernel_entropy(K):
    B = K.size(0)
    Cov = (1/B) * K
    return -(Cov ** 2).sum().log2()

def mkde_loss(encoder, X, gamma1, gamma2):
    # X.shape is (n_distributions, n_samples, d_input)
    Z = encoder(X)
    # Z.shape is (n_distributions, n_samples, d_latent)
    _, K = distribution_kernel_gram(Z, gamma1, gamma2)
    # K.shape is (n_distributions, n_distributions)
    loss = -distribution_kernel_entropy(K)
    return loss


The code presented here follows the setup presented in <Ref> using Gaussian kernel both as embedding and as a distribution kernel. Other kernels could be used by adjusting implementation of both helper functions accordingly. When training on large datasets, <cit.> might be used to avoid memory overflow in average pooling.




 Paper Title: Transient anisotropic kernel for probabilistic learning on manifolds.










1]Christian Soize cor1
christian.soize@univ-eiffel.fr
2]Roger Ghanem
ghanem@usc.edu
[cor1]Corresponding author: C. Soize, christian.soize@univ-eiffel.fr
[1]Université Gustave Eiffel, MSME UMR 8208, 5 bd Descartes, 77454 Marne-la-Vallée, France
[2]University of Southern California, 210 KAP Hall, Los Angeles, CA 90089, United States




§ ABSTRACT 
PLoM (Probabilistic Learning on Manifolds) is a method introduced in 2016 for handling small training datasets by projecting an Itô equation from a stochastic dissipative Hamiltonian dynamical system, acting as the MCMC generator, for which the KDE-estimated probability measure with the training dataset is the invariant measure. PLoM performs a projection on a reduced-order vector basis related to the training dataset, using the diffusion maps (DMAPS) basis constructed with a time-independent isotropic kernel. In this paper, we propose a new ISDE projection vector basis built from a transient anisotropic kernel, providing an alternative to the DMAPS basis to improve statistical surrogates for stochastic manifolds with heterogeneous data.  The construction ensures that for times near the initial time, the DMAPS basis coincides with the transient basis. For larger times, the differences between the two bases are characterized by the angle of their spanned vector subspaces. The optimal instant yielding the optimal transient basis is determined using an estimation of mutual information from Information Theory, which is normalized by the entropy estimation to account for the effects of the number of realizations used in the estimations.
Consequently, this new vector basis better represents statistical dependencies in the learned probability measure for any dimension. Three applications with varying levels of statistical complexity and data heterogeneity validate the proposed theory, showing that the transient anisotropic kernel improves the learned probability measure.



Transient kernelprobabilistic learning PLoM diffusion maps Fokker-Planck operator spectrum






§ INTRODUCTION






 §.§ Objectives of the paper



PLoM (Probabilistic Learning on Manifolds), introduced in 2016 <cit.>, is a method and algorithm specifically developed for cases where the training dataset consists of a small number of data points. This method is based on projecting an Itô equation associated with a stochastic dissipative Hamiltonian dynamical system, which acts as the MCMC generator from the probability measure estimated using the KDE method applied to the points of the training dataset. The projection basis is the diffusion maps (DMAPS) basis associated with a time-independent isotropic kernel, introduced in <cit.>.

Since 2016, all extensions and applications of PLoM (see Section <ref>) have been carried out using the isotropic kernel. Through these applications, we have seen that the isotropic kernel allows for obtaining quality results, even for heterogeneous data and systems of great statistical complexity in small and large dimensions. However, improving the construction of statistical surrogates for stochastic manifolds involving conditional statistics and very heterogeneous data using PLoM based on a transient anisotropic kernel (time-dependent) was an analysis project. In this paper, we address this problem. We propose a new construction of the ISDE projection vector basis, built from a transient anisotropic kernel, which improves the representation of the statistical dependencies of the learned joint probability measure in any dimension.



 §.§ Framework of the considered problem



Machine learning tools and artificial intelligence <cit.>, such as probabilistic and statistical learning <cit.>, are used in UQ for problems that would require computer resources not available with the most usual approaches. Thus, methods have emerged in the field of engineering sciences, such as learning on manifolds <cit.> and physics-informed probabilistic learning <cit.>.

Probabilistic learning is a very active domain of research for constructing surrogate models (see for instance, <cit.>). Probabilistic Learning on Manifolds (PLoM) is a tool in computational statistics, introduced in 2016 <cit.>, which can be viewed as a tool for scientific machine learning. The PLoM approach has specifically been developed for small dataset cases <cit.>. The method avoids the scattering of learned realizations associated with the probability distribution to preserve its concentration in the neighborhood of the random manifold defined by the parameterized computational model.

his method allows for solving unsupervised and supervised problems under uncertainty when the training datasets are small. This situation is encountered in many problems in physics and engineering science with expensive function evaluations. The exploration of the admissible solution space in these situations is thus hampered by available computational resources.

Several extensions have been proposed to account for implicit constraints induced by physics, computational models, and measurements <cit.>, to reduce the stochastic dimension using a statistical partition approach <cit.>, and to update the prior probability distribution with a target dataset, whose points are, for instance, experimental realizations of the system observations <cit.>. Consequently, PLoM, constrained by a stochastic computational model and statistical moments or samples/realizations, allows for performing probabilistic learning inference and constructing predictive statistical surrogate models for large parameterized stochastic computational models.

This last capability of PLoM can also be viewed as an alternative method to Bayesian inference for high dimensions
<cit.> and is a complementary approach to existing methods in machine learning for sampling distributions on manifolds under constraints. Although a Bayesian inference methodology has also been developed using probabilistic learning on manifolds for high dimensions <cit.>.

PLoM has successfully been adapted to tackle these challenges for several related problems, including nonconvex optimization under uncertainty <cit.>, fracture paths in random composites <cit.>, concurrent multiscale simulations in random media <cit.>, stochastic homogenization in random elastic media <cit.>, ultrasonic transmission techniques in cortical bone microstructures <cit.>, updating digital twins under uncertainties <cit.>, updating under-observed dynamical systems <cit.>, calculation of the Sobol indices <cit.>, dynamic monitoring <cit.>, surrogate modeling of structural seismic response <cit.>, probabilistic-learning-based stochastic surrogate models from small incomplete datasets <cit.>, and polynomial-chaos-based conditional statistics for probabilistic learning of atomic collisions <cit.>, as well as for aeroacoustic liner impedance metamodels from simulation and experimental data <cit.>.



 §.§ Methodology proposed and organization of the paper



Starting with a training dataset of n_d realizations of ν random variables, we consider the probability flow from each of these n_d realizations towards the ν-dimensional sampling probability distribution of the training dataset. We construct the associated coupled Fokker-Planck (FKP) equations with each of the realizations as initial condition. These describe the evolution of the transition probability measures over the graph described by the training dataset, that transport each independent realization, viewed as a concentrated measure at the initial time, into the joint probability measure, consisting of the common stationary probability measure  (also called the steady-state solution) of the FKP equations. This evolution describes a trajectory along which transition probabilities are consistent with both the training dataset and its postulated joint probability density function (that we approximate using a Gaussian Kernel Density Estimate (KDE)). Each of these probabilities provides a distinct geometric characterization of the training dataset, with its own plausible model of statistical dependence, resulting in a transient anisotropic kernel from which we construct a time evolving PLoM. We then set criteria for selecting among these PLoM models, which is tantamount to identifying the most appropriate
statistical dependence structure for the learned dataset, along the flow characterized by the FKP equations. The following
description of the paper’s organization provides a coherent summary of the proposed methodology.

In Section <ref>, we define the training dataset constituted of given realizations of a non-Gaussian normalized vector-valued random variable  (centered and with an identity covariance matrix) and define the associated probability measure P_, whose density is estimated using the Gaussian Kernel Density Estimation (GKDE) method.

Section <ref> deals with a short summary of formal results, introducing an Itô stochastic differential equation (ISDE) and the derived Fokker-Planck (FPK) equation for which P_ is the invariant measure, which is the steady-state solution. We then introduce a formal formulation of the eigenvalue problem of the FPK operator and the nonstationary solution of the Fokker-Planck equation with a deterministic initial condition.

Section <ref> is devoted to the time-dependent kernel k_t derived from the time-dependent solution of the Fokker-Planck equation with a deterministic initial condition. We then define the time-dependent operator K_t associated with the transient kernel k_t, which is a Hilbert-Schmidt operator. We propose a construction of its finite approximation, represented by a matrix [K(t)], using a sampling of P_ of the bilinear form associated with K_t, and introduce the corresponding finite approximation of the eigenvalue problem.

In Section <ref>, we present the direct construction of  matrix [K(t)]. We introduce the transition probability density function as the solution of the ISDE with a deterministic initial condition and study its existence, uniqueness, and properties. We then rewrite the ISDE in matrix form and propose a time-discrete approximation of this matrix-valued ISDE based on an Euler scheme. We introduce convergence criteria to verify convergence. Finally, we construct an explicit representation of the matrix [K(t)] based on the nonstationary solution of the matrix-valued ISDE with a deterministic initial condition.

A numerical illustration of the proposed formulation is given in Section <ref>, for which an explicit solution is known.

Section <ref> deals with the construction and study of the vector basis for PLoM derived from the transient anisotropic kernel, connected to the DMAPS basis constructed with the isotropic kernel. We construct a time-dependent matrix [K̃()] of the transient anisotropic kernel derived from the matrix [K̂()], whose fundamental property is its convergence to the matrix [K_] of the DMAPS isotropic kernel as time approaches zero. The time-dependent reduced-order transient basis is then the eigenvectors of [K̃()] associated with its dominant positive eigenvalues.

In order to qualify and quantify the gain of the constructed reduced-order transient basis, ROTB() at an instant sampled , with respect to the reduced-order DMAPS basis, RODB, we introduce the angle between the subspaces spanned by ROTB() and RODB. We propose a methodology for identifying the optimal instant sampled, which maximizes a selection criterion. This criterion is based on the estimation of mutual information from Information Theory, normalized using entropy estimation to account for the effects of the number of realizations used in the statistical estimators. Such a criterion allows for selecting the best joint learned probability measure with respect to statistical dependencies.

In Section <ref>, we present three applications, each with a specific level of complexity and data heterogeneity in the training dataset. The first application is created such that the probability measure of  in ^9, defined by the points of the training dataset, is concentrated in a multiconnected domain of ^9. The constituent connected parts are manifolds of dimensions much lower than 9, each with different dimensions. These parts may or may not be connected to each other. The training dataset of the second application consists of realizations of the random vector  with values in ^8, generated using a polynomial chaos expansion of degree 6 of a real-valued random variable, whose random germ has a dimension of 2, with each of the two random germs being a uniform random variable with different support. There are therefore 28 terms in this expansion, and the 8 components of  are defined as the random terms of given rank, defining a relatively complex random manifold in ^8. The third application results from a statistical treatment of an experimental database containing photon measurements in the ATLAS detector at CERN. The PCA step of PLoM has been performed, and 45 components have been extracted to obtain the training dataset for the ^45-valued random variable . This application is in higher dimension than the first two but has less statistical complexity.

The paper is completed by two appendices. <ref> presents an overview of the probabilistic learning on manifolds (PLoM) algorithm and its parameterization, using either the DMAPS basis (RODB) or the transient basis (ROTB()) as the projection basis. <ref> provides the formulas for estimating the Kullback-Leibler divergence, mutual information, and entropy from a set of realizations.

It should be noted that there are some very brief repetitions, which have been deliberately made to facilitate reading.



 §.§ Convention for the variables, vectors, and matrices



x,η: lower-case Latin or Greek letters are deterministic real variables.

,: boldface lower-case Latin or Greek letters are deterministic vectors.

X: upper-case Latin letters are real-valued random variables.

: boldface upper-case Latin letters are vector-valued random variables.

[x]: lower-case Latin letters between brackets are deterministic matrices.

[]: boldface upper-case letters between brackets are matrix-valued random variables.



 §.§ Algebraic notations



, ^*: set of natural numbers including 0, excluding 0.

, ^+, ^+*: set of real numbers, subset [0,+∞[, subset ]0, +∞[.

^ν: Euclidean vector space of dimension ν.

_n,m, _n : set of the (n× m), (n× n),  real matrices.

_n^+, _n^+0: set of  the positive-definite, positive,  (n× n) real matrices.

[I_n]: identity matrix in _n.

‖  [x] ‖_F: Frobenius norm of matrix  [x].

∇, : gradient and divergence operators in ^ν.

δ_0: Dirac measure on ^ν at point _ν.




 §.§ Convention used for random variables



In this paper, for any finite integer m≥ 1, the Euclidean space ^m is equipped with the σ-algebra _^m. If  is a ^m-valued random variable defined on the probability space (Θ,,),  is a  mapping θ↦(θ) from Θ into ^m, measurable from (Θ,) into (^m,_^m), and (θ) is a realization (sample) of  for θ∈Θ. The probability distribution of  is the probability measure P_(d) on the measurable set (^m,_^m) (we will simply say on ^m). The Lebesgue measure on ^m is noted d and
when P_(d) is written as p_()  d, p_ is the probability density function (pdf) on ^m of P_(d) with respect to d. Finally, E denotes the mathematical expectation operator that is such that
E{} = ∫_^m  P_(d).




§ DEFINING THE PROBABILITY MEASURE P_ OF RANDOM VECTOR 




Let _() = {^j, j = 1,…, n_d} be the set of n_d > 1 independent realizations ^j∈^ν, with ν≥ 1, of a second-order ^ν-valued random variable defined on a probability space (Θ,,). Let _d∈^ν and [C_d]∈_n_d be the associated empirical estimates of the mean value and the covariance matrix constructed with the points of _(),


    _d = 1/n_d∑_j=1^n_d^j    ,   
    [C_d] =  1/n_d -1∑_j=1^n_d (^j - _d )⊗ (^j - _d )  .


It is assumed that _() is such that


    _d =  _ν   ,   
    [C_d] =  [I_ν]  .


Let ↦ p_() be the probability density function on ^ν, with respect do the Lebesgue measure d, defined by


    p_() = 1/n_d∑_j=1^n_d1/(√(2 π) ŝ)^νexp ( -1/2 ŝ^2‖ - ŝ/s^j‖^2 )    ,   ∀∈^ν  ,


where ŝ and s are defined by


    s =  ( 4/n_d(2+ν) )^1/(ν+4)   ,   ŝ  = s/√(s^2 + (n_d-1)/n_d)  .


Eqs. (<ref>) and (<ref>) correspond to the Gaussian kernel-density estimation (KDE) constructed using the n_d independent realizations of _() involving the modification <cit.> of the usual formulation <cit.>, in which s is the Silverman bandwidth.

Let  be the second-order ^ν-valued random variable, defined on a probability space (Θ,,), whose probability measure P_(d) = p_()  d on ^ν is defined by the probability density function p_ given by Eq. (<ref>). It can be seen that, for any fixed n_d > 1, we have


    E{} = ∫_^ν  P_(d) = 1/2ŝ^2 _d = _ν  ,



    E{⊗} = ∫_^ν⊗  P_(d)
    = ŝ^2  [I_ν] + ŝ^2/s^2 (n_d-1)/n_d [C_d] = [I_ν]   .


Eqs. (<ref>) and (<ref>) show that  is a normalized ^ν-valued random variable. The probability density function p_ defined by Eq. (<ref>) is rewritten, for all  in ^ν, as


    p_() = c_ν ξ()     ,   ξ() = e^-Φ()  ,


in which  c_ν =  (√(2 π) ŝ)^-ν and where Φ() = -log(ξ()) is such that


    Φ() = - log ( 1/n_d∑_j=1^n_dexp ( -1/2 ŝ^2‖ - ŝ/s^j‖^2 )  )    ,   ∀∈^ν  .






§ SHORT SUMMARY OF FORMAL RESULTS




This section is limited to a summary of essential results, which will be used in Section <ref> and which are formally presented.




 §.§ Itô stochastic differential equation related to P_



We introduce an  Itô stochastic differential equation (ISDE) on ^ν, with initial condition, for which P_(d) is the invariant measure. A classical candidate to such an ISDE is written as



    d(t) =     ((t))  dt + d(t)    ,    t > 0   , 
    (0) =     ∈^ν  ,   a.s.   ,


where the drift vector is the function ↦() from ^ν into ^ν defined by


    () = -1/2 ∇Φ()    ,   ∀∈^ν  .


In Eq. (<ref>), (t) = (W_1(t),… ,W_ν(t)) is the normalized Wiener stochastic process <cit.> on ^+, with values in ^ν, which is a stochastic process with independent increments, such that (0) = _ν  a.s, and for  0≤τ < t < +∞, the increment Δ_τ  t = (t) - (τ) is a Gaussian ^ν-valued second-order random variable, centered and with a covariance matrix that is written as


    [C_Δ_τ t] = E{Δ_τ t⊗Δ_τ t} = (t-τ)  [I_ν]   .


It should be noted that Eqs. (<ref>) and (<ref>) is equivalent to


    (t) =  + ∫_0^t ((τ))  dτ + ∫_0^t d(τ)    ,    t ≥  0   .


In Section <ref>, we will see that {(t), t∈^+} is a homogeneous diffusion stochastic process, which is asymptotically stationary for t→ +∞. Assuming that the transition probability measure of (t) given (0) = admits a density with respect to d, such that, for all t > 0, for all  and  in ^ν, and for any Borelian  in ^ν, we have


    { (t) ∈ | (0) =  }    =  ∫_ρ(,t  | ,0)   d  ,  
    lim_t → 0_+ρ(,t  | ,0)   d     = δ_0( - )    ,  
    ∫_^νρ(,t  | ,0)   d    = 1   .






 §.§ FKP equation associated with the ISDE



For all  in ^ν, the transition probability density function (,t)↦ρ(,t  | ,0) from
^ν×^+ into ^+ verifies the following Fokker-Planck (FKP) equation (see for instance <cit.>),


    ∂ρ/∂ t + L_(ρ) = 0    ,    t > 0   ,


with the initial condition for t=0 defined by Eq. (<ref>). The Fokker-Planck operator L_ can be written, after a small algebraic manipulation and for any sufficiently differentiable function ↦() from ^ν into , as


    {L_()}() = -1/2{ p_() ∇ ( ()/p_() ) }  .


The detailed balance (the probability current vanishes) is satisfied and the steady state solution of Eq. (<ref>) is the pdf p_ defined by Eq. (<ref>) <cit.>. We then have, for = p_,


    L_(p_) = 0   .






 §.§ Return to the invariant measure P_



The invariant measure (see Proposition <ref>) P_(d) = p_()  d is such that, for all  in ^ν and for all t≥ 0,


    p_() = ∫_^νρ(,t  | ,0)   p_()   d  .


The ISDE defined by Eqs. (<ref>) and (<ref>) admits an asymptotic (t→ +∞) stationary solution whose marginal probability density function of order one is p_. Consequently, for all  and  in ^ν, we have


    lim_t → +∞ρ(,t  | ,0)  = p_()   .






 §.§ Formal formulation of the eigenvalue problem of the FKP operator



The eigenvalue problem, posed in an adapted functional space, is written as


    L_() =  λ   ,


for which the current must vanish at infinity, yielding the condition,


    lim_‖‖→ +∞  p_()  ‖ ∇  ( p_()^-1()   )  ‖  = 0  .


Continuing the development within  a formal framework, such as that used in <cit.>, we introduce the change of function,


    () = p_()^1/2  q()    ,   ∈^ν   ,    q: ^ν→  .


Let L̂_ be the linear operator defined, for q belonging to an admissible set of functions,


    {L̂_(q)}() =  p_()^-1/2 L_(p_()^1/2  q(y))    ,   ∈^ν  .


Therefore, the eigenvalue problem defined by Eqs. (<ref>) and (<ref>) can  be rewritten in q as


    L̂_(q) =  λ  q   ,


with the condition at infinity,


    lim_‖‖→ +∞ p_()^1/2 ‖ ∇  ( p_()^-1/2 q()   )  ‖  = 0  .






 

Using Eq. (<ref>), which shows that p_()^-1∇ p_() = - ∇Φ, and using
 Eqs. (<ref>) and (<ref>), it can be seen that


    {L̂_(q)}() =  ()  q() - 1/2∇^2 q()    ,   ∈^ν  ,


in which ∇^2 is the Laplacian operator in ^ν and where ↦() is the function from ^ν into , which is defined, for all  in ^ν, as


    () =  1/8‖ ∇Φ()  ‖^2  -  1/4∇^2 Φ()   .










 §.§ Properties of operator L̂_



Let δ q be a function from ^ν into , belonging to the admissible set that allows the evaluation of the bracket


    ⟨L̂_ (q)   , δ q⟩  = ∫_^ν{L̂_(q)}()   δ q()   d  .


Removing  and using Eq. (<ref>) with Eq. (<ref>) yields


    ⟨L̂_ (q)   , δ q⟩  = -1/2∫_^ν
    (p_^-1/2 δ q)  { p_ ∇ ( p_^-1/2  q)}  d  .


Using the condition at infinity, defined by Eq. (<ref>), Eq. (<ref>) can be rewritten as,


    ⟨L̂_ (q)   , δ q⟩  = 1/2∫_^ν
    p_ ⟨∇(p_^-1/2  q)   , ∇( p_^-1/2 δ q) ⟩_^ν  d  .



(a) Eq. (<ref>) shows that L̂_ is a symmetric and positive operator.

(b) Eqs. (<ref>) and (<ref>) show that


    L̂_ (q_0) = 0   for   q_0 = p_^1/2  .


In Proposition <ref>, it will be proven that the ISDE defined by Eq. (<ref>), with the initial condition defined by  (<ref>),  has a unique solution and a unique invariant measure p_()  d. Consequently the dimension of the null space of operator L_ is 1. Since p_()  d is a bounded positive measure (probability measure), the right-hand side of Eq. (<ref>) shows that the null space  of L̂_, which is also of dimension  1, is constituted of the function q_0 = p_^1/2. For q=δ q ≠ q_0, and ‖ q_0‖≠ 0, we have ⟨L̂_ (q)   ,  q⟩  > 0. Therefore, L̂_ is a positive operator (in the quotient space by the null space).



[On the spectrum of operator L̂_] 
It is assumed that p_ defined by Eq. (<ref>), which is constructed with the n_d points {^j, j=1,…, n_d} of the training dataset, is such that the spectrum of L̂_ is countable. Due to (a) and (b), we then deduce that the eigenvalues of L̂_ (defined by Eqs. (<ref>) and (<ref>)) are positive except one that is zero. We will also assume that the multiplicity of each eigenvalue is finite.






 §.§ Eigenvalue problem for operator L̂_



Under Hypothesis <ref>, the eigenvalue problem L̂_(q_α) = λ_α q_α for operator
L̂_, with the condition defined by Eq. (<ref>), is such that


    0 =λ_0 < λ_1 ≤λ_2 ≤…  ,


the multiplicity of each eigenvalue being finite.

We will admit that the family {q_α,α∈} of the eigenfunctions is a Hilbert basis of L^2(^ν). We then have


    ⟨ q_α , q_β⟩_L^2 = ∫_^ν q_α()   q_β()  d = δ_αβ  .


The eigenfunction q_0 associated with λ_0=0, is such that (see Eq. (<ref>)),


    q_0 = p_^1/2   ,  ‖  q_0‖_L^2 = 1   ,


and we have


    ∑_α∈ q_α()   q_α()  d = δ_0(-)   .


From Eqs. (<ref>) and (<ref>), it can be deduced that


    ∀α≥ 1    ,   ∫_^ν p_()^1/2  q_α()   d = 0   .






 §.§ Nonstationary solution of the Fokker-Planck equation with initial condition



The transition probability density function ρ(,t | ,0) introduced in Section <ref> and satisfying Eq. (<ref>) with the initial condition defined by Eq. (<ref>), can  be written, using the Hilbert basis {q_α,α∈} defined in Section <ref>, as


    ρ(,t | ,0) = p_()^1/2  p_()^-1/2 ∑_α∈ e^-λ_α t q_α()   q_α()    ,    t > 0  .


This representation of ρ(,t | ,0), defined by Eq. (<ref>), actually satisfied all the required properties:

Eqs. (<ref>), (<ref>), and  (<ref>) yield ∫_^νρ(,t | ,0)   d = 1.

Eqs. (<ref>) and  (<ref>) yield lim_t→ +∞ρ(,t | ,0)   d = p_().

Eq. (<ref>) yields lim_t→ 0_+ρ(,t | ,0)   d = δ_0(-).

Eqs. (<ref>) and  (<ref>) yield ∫_^νρ(,t | ,0)  p_()   d = p_() that is Eq. (<ref>).




§ TIME-DEPENDENT KERNEL, ITS ASSOCIATED OPERATOR, AND FINITE APPROXIMATION




In this section,  we define the kernel k_t(,) and give its basic properties directly deduced from the properties of ρ(,t | ,0) and p_(), without using the spectral representation defined by Eq. (<ref>). From the spectral representation of k_t(,), we deduce its spectral representation using the spectral representation of  ρ(,t | ,0), defined by Eq. (<ref>). Finally, we define the linear operator K_t associated with kernel k_t and we give the spectral representation of operator K_t.




 §.§ Definition of the kernel k_t and its basic probabilistic properties



The kernel k_t(,) associated with the transition probability density function ρ(,t | ,0) is defined as follows.


 
For every fixed t > 0, the kernel function (,)↦ k_t(,), from ^ν×^ν into ^+, is defined by


    k_t(,) = ρ(,t | ,0)/p_()  .





The following Lemma gives basic properties of kernel k_t.


 

For every fixed t > 0, and for all  and  in ^ν, we have the following properties:


    (a)      ∫_^ν k_t(,)  p_()  d = 1    ,   ∫_^ν k_t(,)  p_()  d = 1                           , 
    
    (b)      ∫_^ν∫_^ν   k_t(,)  p_()  p_() d  d =1   ,  
    
    (c)      lim_t→ 0_+   k_t(,)  p_()  d = δ_0(-)    ,   lim_t→ +∞ k_t(,) =1                                     ,  
    
    (d)       (,)↦ρ(,t | ,0) ∈ C^0(^ν×^ν) ⇒
                          (,)↦  k_t(,) ∈ C^0(^ν×^ν)                   ,  
    
    (e)       k_t(,) = k_t(,)                                                                     .





 (Lemma <ref>). Definition <ref> is used.

(a) The first equation in Eq. (<ref>) is due to Eq. (<ref>) and the second one is due to Eq. (<ref>).

(b)  Eq. (<ref>)  is directly deduced from Eq. (<ref>).

(c) The first equation in Eq. (<ref>) is due to Eq. (<ref>) and the second one is due to Eq. (<ref>).

(d) For all  in ^ν, p_(y) > 0 and p_∈ C^0(^ν). The hypothesis
ρ(· ,t | ·,0)∈ C^0(^ν×^ν) yields Eq. (<ref>).

(e) let p_(t),(,t;,0) be the joint pdf of (t) with  in which (t) is the solution of Eq. (<ref>) for fixed t > 0, with the random initial condition (0) =. We have the classical property related to the definition of the invariant measure,
p_(t)(,t) = ∫_^ν p_(t),(,t;,0)  d =  ∫_^νρ(,t | ,0)  p_()  d = p_(). For all  and  in ^ν, and for t > 0, we have,
ρ(,t | ,0)  p_() = p_(t),(,t;,0) =p_,(t)(,0;,t) =
ρ(,0 | , t)  p_(t)(,t) = ρ(,0 | , t)  p_().
Consequently,  k_t(,)= ρ(,t | ,0) p_() / (p_() p_()) =
ρ(,0 | , t)  p_() / (p_() p_()) = k_t(,).







 §.§ Hypothesis and properties of kernel k_t from its representation



In Section <ref>, we introduced an hypothesis of existence of a discrete (countable) spectrum {λ_α,α∈} of the FKP operator L̂_. In this section, we study the spectral representation of kernel k_t, for every fixed t > 0, deduced from the time-dependent spectral representation of ρ(,t | ,0), defined by Eq. (<ref>).


 

Let = L^2(^ν;p_) be the Hilbert space of the square-integrable real-valued functions on ^ν, with respect to the probability measure p_()  d on ^ν, equipped with the inner product and the associated norm,


    ⟨ u  , ⟩_ = ∫_^ν u() ()   p_()  d   ,   ‖ u‖_ = ⟨ u  , u ⟩_^1/2  .







 

Let {q_α,α∈} be the Hilbert basis of L^2(^ν) introduced in Section <ref>.
For all α∈, we defined the real-valued function ψ_α on ^ν such that


    ψ_α() = q_α()  p_()^-1/2   ,   ∀∈^ν  .


Then, {ψ_α,α∈} is a Hilbert basis of  and we have,


    (a)      ψ_α∈   ,   ‖ψ_α‖_ = ‖ q_α‖_L^2 = 1
                  ,   ∀α∈  , 
    
    (b)      ⟨ψ_α  , ψ_β⟩_ = ∫_^νψ_α() ψ_β()  
                  p_()  d = δ_αβ   ,   ∀ (α,β)∈×  , 
    
    (c)      ψ_0() = 1   ,  ∀∈^ν   ,   ‖ψ_0‖_ = 1                , 
    
    (d)      ∫_^νψ_α()  p_()   d = 0    ,   ∀α∈^*    , 
    
    (e)      ∑_α∈ψ_α() ψ_β()   p_()   d = δ_0(-)
                     ,   ∀ (,) ∈^ν×^ν  .





 (Lemma <ref>).

(a) Since p_^-1/2∈ C^0(^ν), Eqs. (<ref>) yields Eq. (<ref>); combined with
Eq. (<ref>), this yields ‖ψ_α‖_ = ∫_^ν q_α()^2  d = ‖ q_α‖_L^2 = 1.

(b) Using Eq. (<ref>) yields ⟨ϕ_α  , ψ_β⟩_ = ∫_^ν q_α()  q_β()   d = ⟨ q_α  , q_β⟩_L^2= δ_αβ. Thus {ψ_α,α∈} is an orthonormal family in . For all u in L^2(^ν), the linear mapping u↦ = u p_^-1/2 is a continuous injection from L^2(^ν) into   with ‖‖_ = ‖ u ‖_L^2. Therefore, {ψ_α,α∈} is a Hilbert basis of .

(c) The two equations in Eq. (<ref>) are directly deduced from  Eqs. (<ref>) and (<ref>).

(d) Since  ⟨ψ_0  , ψ_α⟩_ = 0 for all α∈^*, we obtain Eq. (<ref>).

(e) Since  {ψ_α,α∈} is a Hilbert basis of , Eq. (<ref>) holds.






 

Let {ψ_α,α∈} be the Hilbert basis of  defined in Lemma <ref>.

(a) For every fixed t > 0, the symmetric kernel k_t can be written, for all  and  in ^ν, as


    k_t(,) =∑_α∈ b_α(t) ψ_α() ψ_α()   ,


in which the family of positive real numbers {b_α(t)= exp(-λ_α  t)  , α∈}, is such that


    1 = b_0(t) > b_1(t) ≥ b_2(t) ≥…  .


(b) If for every fixed t > 0, kernel k_t satisfies


    ∫_^ν∫_^ν k_t(,)^2   p_()  p_()  d  d = c_t^2 < +∞  ,


where c_t > 1  is a positive constant depending on t, then,


    ∑_α∈ b_α(t)^2 = c_t^2 < +∞  .





 (Proposition <ref>).

(a) Substituting  Eq. (<ref>) with q_α() = ψ_α()  p_()^1/2 (see Eq. (<ref>)) into
Eq. (<ref>) yields Eq. (<ref>). From Eq. (<ref>) and since b_α(t)= exp(-λ_α  t), we obtain
Eq. (<ref>).

(b) Assuming Eq. (<ref>), substituting Eq. (<ref>) into Eq. (<ref>), and using Eq. (<ref>) yields
Eq. (<ref>). Since b_0(t) = 1, it can be deduced that c_t^2 > 1 and thus, c_t > 1.






 §.§ Hilbert-Schmidt operator K_t associated with kernel k_t



We now introduce the linear operator in , defined by kernel k_t, and we study its properties and spectrum.


 

For every fixed t > 0, we defined the linear operator K_t from  into  such that, for all u and  in ,


    ⟨ K_t  u  , ⟩_ = ∫_^ν∫_^νk_t(,)  u() ()   p_()  p_()  d  d  ,


where the symmetric kernel k_t verifies the condition defined by Eq. (<ref>).





 

For every fixed t > 0, let K_t be the continuous linear operator defined by Eq. (<ref>), in which kernel k_t is symmetric on ^ν×^ν, and verifies Eq. (<ref>).

(a) For all u and  in , operator K_t is such that


    ⟨ K_t  u  , ⟩_ = ∑_α∈ b_α(t)  ⟨ u  ,ψ_α⟩_ ⟨  ,ψ_α⟩_  ,


and is a positive symmetric operator in . For all u in ,


    K_t  u = ∑_α∈ b_α(t)  ⟨ u  ,ψ_α⟩_ ψ_α  .


(b) For all α in , ψ_α∈ is the eigenfunction independent of t, associated with the positive eigenvalue b_α(t), satisfying Eq. (<ref>), of operator K_t,


    K_t ψ_α = b_α(t)  ψ_α   ,    b_α(t) = exp(-λ_α t)    ,   α∈  ,


which shows that, for all α and β in ,


    ⟨ K_t ψ_α  , ψ_β⟩_ = b_α(t)  δ_αβ  .


(c) For all u in , we have


    ‖ K_t  u ‖_^2 = ∑_α∈ b_α(t)^2  ⟨ u  ,ψ_α⟩_^2   ,


and for Hilbert basis {ψ_α  , α∈} of ,


    ∑_α∈‖ K_t ψ_α‖_^2 =  c_t^2 < + ∞  ,


where c_t^2, defined by Eq. (<ref>), is such that c_t^2 = ∑_α∈ b_α(t)^2, and therefore, is a Hilbert-Schmidt operator in .




 (Proposition <ref>).

Under the condition defined by Eq. (<ref>), it is well known that operator K_t is continuous from  into .

(a) Substituting Eq. (<ref>) into Eq. (<ref>) and using Eq. (<ref>) yield  Eq. (<ref>). This equation shows that K_t is a symmetric and positive operator because ⟨ K_t  u  , u ⟩_ > 0 for all u in  with ‖ u‖_≠ 0. Note that  Eq. (<ref>) is directly deduced from Eq. (<ref>).

(b)  From Eqs. (<ref>) and (<ref>), we have ψ_α∈. Taking u = ψ_β in Eq. (<ref>) and using Eq. (<ref>) yield K_t ψ_β = ∑_α∈ b_α(t)  ⟨ψ_β  ,ψ_α⟩_ ψ_α = b_β(t) ψ_β, Eq. (<ref>) is obtained from Eq. (<ref>) by taking u=ψ_α and = ψ_β.
 The relationship between b_α(t) and λ_α comes from Proposition <ref>.

(c) Using Eqs. (<ref>) and (<ref>) yields Eq. (<ref>). Taking u=ψ_α in Eq. (<ref>) yields
‖ K_t ψ_α‖_^2 = b_α(t)^2. From Eq. (<ref>) and
∑_α∈‖ K_t ψ_α‖_^2 = ∑_α∈ b_α(t)^2, we obtain Eq. (<ref>). It can be deduced (see for instance <cit.>) that K_t is a Hilbert-Schmidt operator in .





 §.§ Finite approximation of operator K_t and of its eigenvalue problem



The Hilbert-Schmidt operator K_t defined by Eq. (<ref>), operates in infinite dimension. The Hilbert basis
{ψ_α  , α∈} (which relates to the Hilbert basis { q_α  , α∈}, see Lemma <ref> and  Eq. (<ref>)) is not explicitly known, thereby preventing the use of the representation defined by Eq. (<ref>).

We must thus construct a finite approximation of K_t.

Since  ^ν is an unbounded set and ν can be very large, classical discretization such as finite-difference or finite-element methods (see <cit.> for Fokker-Planck equation and <cit.> for fractional Fokker-Planck equation) or such methods based on  shape-morphing modes for solving the Fokker-Planck equation as proposed in <cit.>, are not directly adapted for solving the eigenvalue problem of operator L̂_. Another classical method consists in introducing a finite family of functions in , generating a finite dimension subspace of , and in performing the projection of K_t on this finite subspace. Such an approach is not really adapted to operator L̂_ for which a large number of eigenvalues and associated eigenfunctions have to be computed.

It should be noted that a related problem, but distinct from the one addressed, is that of the numerical method for Schrödinger operator and the associated equation (see for instance <cit.> for solving the Schrödinger Equation, <cit.> for the solution of the Schrödinger equation by spectral methods,  <cit.> for numerically solving the time-dependent Schrödinger equation,  and <cit.> for the numerical solution of the Schrödinger equation using finite-difference method).

Nevertheless, such approaches are not well adapted to the objective of the actual developments, which has been detailed in Section <ref>. We then propose to use a statistical sampling of ^ν equipped with the probability measure p_()  d, which will be well adapted to our objective of performing a construction connected to the DMAPS approach.


 

Let us assume that, for every fixed t > 0, we have (,)↦ρ(,t | ,0) ∈ C^0(^ν×^ν). From Eq. (<ref>), it can be deduced that (,)↦  k_t(, ) ∈ C^0(^ν×^ν).
For every fixed t > 0, for all u and  in ∩ C^0(^ν), the restriction to (∩ C^0(^ν))× (∩ C^0(^ν)) of the bilinear form (u,)↦⟨ K_t u  , ⟩_,
defined on × by Eq. (<ref>) with the continuous symmetric function k_t verifying Eq. (<ref>), can be written as


    ⟨ K_t u  , ⟩_  = E{k_t(,)   u()  ()}  ,


in which  is an independent copy of . The joint probability measure P_,(d,d) of  with  is p_()× p_()  d  d.
The real-valued random variables u(), (), and the positive-valued random variable k_t(,), defined on (Θ,,), are second-order random variables,


    E{u()^2 } < +∞   ,    E{()^2 } < +∞   ,    E{k_t(,)^2} < +∞  .


Let Z_t,u, be the real-valued random variable, defined on (Θ,,), such that


    Z_t,u, = k_t(,)   u()  ()  .


Then, Z_t,u, is such that,


    E{Z_t,u,} < +∞  .





 (Proposition <ref>).

(a) Since u and  are continuous functions in , u() and () are real-valued random variables defined on (Θ,,) and are second-order because,


    E{ u()^2} =  ∫_^ν u()^2 p_()  d = ‖ u ‖_^2  < +∞   ,   
    E{()^2}          =  ∫_^ν()^2 p_()  d = ‖‖_^2  < +∞  .


Since k_t∈ C^0(^ν×^ν) (see Eq. (<ref>)) and due to Eq. (<ref>),
k_t(,) is a second-order positive-valued random variable defined on (Θ,,),


    E{k_t(,)^2} =  ∫_^ν∫_^ν k_t(,)^2 p_() p_()    d  d =c_t^2 < +∞  .


(b) For all u in  and for t > 0, from Eq. (<ref>), it can be seen that
(K_t u)() = ∫_^ν k_t(,)  p_()^1/2  u()  p_()^1/2  d.
Using the Cauchy-Schwarz inequality  and Eq. (<ref>) yield


    ‖ K_t u ‖_^2  ≤ ‖ u ‖_^2 ∫_^ν∫_^ν k_t(,)^2 p_() p_()    d  d
    = c_t^2 ‖ u ‖_^2   ,


(which, in passing, shows the continuity of the operator K_t in   as stated at the beginning of the proof of Proposition <ref>).
Consequently, we have ⟨ K_t u  , ⟩_^2 ≤‖ K_t u ‖_^2  ‖‖_^2, which shows that


    ⟨ K_t u  , ⟩_^2  ≤ c_t^2  ‖ u ‖_^2 ‖‖_^2  < +∞  ,


and therefore,  Eq. (<ref>) holds.






 

Let {^i,i=1,…, n_d} and {^j,j=1,…, n_d} be n_d independent copies  of  and , respectively. For every fixed t > 0, and for all u and  in ∩ C^0(^ν), let Z_t,u,^ij be the real valued random variable on (Θ,,), such that, for all i and j in {1,… , n_d},


    Z_t,u,^ij = k_t(^i , ^j)   u(^j) (^i)   .


Let Z_t,u,^ (n_d) be the real-valued random variable on (Θ,,), defined by


    Z_t,u,^ (n_d) = 1/n_d^2∑_i=1^n_d∑_j=1^n_d Z_t,u,^ij  .


Then, Z_t,u,^ (n_d) is an estimator of


    z_  t,u, = E{Z_t,u,} = ⟨ K_t u  , ⟩_  ,


where Z_t,u, is given by Eq. (<ref>).

Let {^j , j=1,… , n_d} be  the n_d independent realizations of  introduced in Section <ref>. Since ^i and ^j are independent copies of  (because  is an independent copy of ), an estimation
of z_  t,u, is a realization z_  t,u,^(n_d) of the estimator  Z_t,u,^ (n_d), which is written as


    z_  t,u,^(n_d) = 1/n_d^2∑_i=1^n_d∑_j=1^n_d k_t(^i,^j)  u(^j) (^i)  .






 

Under the hypotheses and notations of Definition <ref>, the sequence of real-valued random variables
{Z_t,u,^ (n_d)}_n_d on (Θ,,) is convergent in probability to
z_  t,u,,


    ∀ϵ > 0    ,   lim_n_d→ +∞{ | Z_t,u,^ (n_d)
                                   - z_  t,u, |  ≥ ϵ} = 0   .


We have also the almost sure convergence, thanks to the strong law of large numbers,


    {lim_n_d→ +∞Z_t,u,^ (n_d)
                                   = z_  t,u,} = 1   .





 (Lemma <ref>).

The proof uses the usual results from mathematical statistics (see for instance <cit.>).






 

(a) As is well known, the speed of convergence is proportional to 1/√(n_d^2) = 1/n_d and is independent of dimension ν. The quantification of the approximation error could traditionally be estimated using the central limit theorem, which involves the variance of the estimator (see, for instance, <cit.>). In the numerical illustration provided in Section <ref>, we will show the numerical calculation of  the first eigenvalues of the Fokker-Planck operator for which a reference is known.

(b) Lemma <ref> with Definition <ref> and Proposition <ref> allow a finite approximation of ⟨ K_t u  , ⟩_ to be constructed and consequently, to deduce the corresponding finite approximation of the eigenvalue problem defined by Eq. (<ref>).





 

For every fixed t > 0, for all u and  in ∩ C^0(^ν), and for n_d sufficiently large, we have (in the sense of the convergence described in Lemma <ref>),


    ⟨ K_t u  , ⟩_≃⟨   [K(t)]   , ⟩_^n_d  ,


in which ⟨· , ·⟩_^n_d is the usual Euclidean inner product in ^n_d and where [K(t)]∈_n_d^+0 is such that, for all i and j in { 1,…, n_d},


    [K(t)]_ij = 1/n_d  k_t(^i,^j)   ,


and where the vectors  and  in ^ν are such that


    = (u(^1)/√(n_d)  ,  …  ,  u(^n_d)/√(n_d) )   ,   = ((^1)/√(n_d)  ,  …  ,  (^n_d)/√(n_d) )   .


The corresponding finite approximation of the eigenvalue problem defined by Eq. (<ref>) is written as


    [K(t)]  _α(t) = b̂_α(t) _α(t)    ,   _α(t) ∈^n_d  ,


in which


    b̂_0(t) ≥ b̂_1(t)  ≥ … ≥ b̂_n_d-1(t)  ≥ 0  ,


and where the normalization of the eigenvectors is chosen so that for α and β in {0,1,…, n_d-1 },


    ⟨_α(t) , _β(t)  ⟩_^n_d = δ_αβ  .





 (Proposition <ref>).

From Eqs. (<ref>), (<ref>), and (<ref>), it can be deduced that, for n_d sufficiently large,
⟨ K_t u  , ⟩_≃ n_d^-2∑_i=1^n_d∑_j=1^n_d k_t(^i,^j)  u(^j) (^i).
Since k_t(^i,^j) = k_t(^j,^i) by symmetry of k_t (see Eq. (<ref>)), the right-hand side member can be written as ⟨   [K(t)]   , ⟩_^n_d  where [K(t)], , and  are defined by Eqs. (<ref>) and (<ref>). From Proposition <ref>, K_t is a positive operator in , and as Hilbert-Schmidt operator, b_α(t) → 0_+ as α→ +∞. For the finite approximation, matrix [K(t)] is then symmetric  and positive. Using Eq. (<ref>), the finite approximation  of Eq. (<ref>) is then written as Eq. (<ref>). Since
[K(t)] is real, symmetric, and positive, we have Eqs. (<ref>) and we choose the normalization of {_α(t)}_α so that  (<ref>) holds.






 


(a) It should be noted that we have chosen the construction of {_α(t)}_α as an orthonormal basis in ^n_d. Consequently, _α(t) is not related  to ψ_α by a simple sampling, similar to the one described  in  Eq. (<ref>).

(b) In addition, _α(t) depends, a priori, on t, while ψ_α is independent of t. Only for
n_d→ +∞, _α(t) goes to a vector independent of t. Similarly, although b_0(t)= exp(-λ_0 t) = 1 because λ_0 = 0 (see Eq. (<ref>)), we do not have,  a priori, b̂_0(t) = 1, but this equality holds for n_d→ +∞. Nevertheless, for n_d finite, we will exploit this existing dependence on t for the construction of the reduced transient basis at time t, which will be connected to the reduced DMAPS basis for t→ 0.

(c) Note that as n_d →  +∞, b̂_α(t) tends to b_α(t). Since b_α(t) = exp(-λ_α t), that is to say λ_α = -1/t log  b_α(t), we choose to define _α(t) by a similar formula,  such that for t > 0 and α for which b̂_α(t) > 0,


    _α(t)  = -1/t log b̂_α(t)   .


(d) For every fixed t >0, to solve the eigenvalue problem defined by Eq. (<ref>), we have to construct matrix [K(t)] with an adapted methodology. This will be the object of Section <ref>.





§ CONSTRUCTION OF THE MATRIX OF THE FINITE APPROXIMATION




In this section, we present the methodology to construct matrix [K(t)] as defined in Proposition <ref>. This construction requires the numerical evaluation of kernel k_t, because the entry [K(t)]_ij of [K(t)] is
given by k_t(^i,^j)/n_d (see Eq. (<ref>)). According to Definition <ref> (see Eq. (<ref>)), for i and j in {1,… , n_d},


    [K(t)]_ij = 1/n_d ρ(^i,t | ^j,0)/p_(^i)  .


In Eq. (<ref>), p_ is explicitly defined by Eq. (<ref>), and ρ(,t | ,0) is the transient probability density function of the stochastic process {(t), t≥ 0}, starting from (0) = in ^ν. This function is the solution of the ISDE defined by Eq. (<ref>) for t > 0 and with the initial condition (0) = (see Eq. (<ref>)).

We will begin by using a a classical mathematical result concerning the solution of the ISDE, which must be validated for the specific case where the invariant measure is  defined in Section <ref>. Additionally, we will obtain a proof of the properties introduced in Section <ref>.

Then, we will present the numerical method for constructing [K(t)] by numerically solving the ISDE and using nonparametric statistics to estimate ρ(^i,t | ^j,0). Subsequently, we will derive an explicit algebraic formula for [K(t)]_ij.




 §.§ Existence and uniqueness of the solution of ISDE and properties of the transition probability



Let ={(t), t≥ 0} be the ^ν-valued stochastic process satisfying (see Eqs. (<ref>) and (<ref>)),


    d(t) =     ((t))  dt + d(t)    ,    t > 0   , 
    (0) =     ∈^ν  ,   a.s.   ,


with ∈^ν, where the drift ↦() : ^ν↦^ν is defined by Eq. (<ref>) with Eq. (<ref>), and where {(t),t≥ 0} is the normalized Wiener process defined on (Θ,,).



 

Eqs. (<ref>) and (<ref>) define a unique homogeneous diffusion ^ν-valued stochastic process  defined on (Θ,,), whose transition probability measure is homogeneous (that is to say, it depends only on t - 0 = t),


    P_(t)|(0)(,t | ,0) = {(t)∈ | (0)=}   ,    t > 0  ,


where  is any Borel set in ^ν. For all t > 0 and for all  in ^ν, P_(t)|(0)(d , t | ,0) admits a density function ↦ρ(,t | ,0) : ^ν↦ ]0  , +∞[ with respect to the Lebesgue measure d on ^ν, such that


    P_(t) | (0)(d,t | ,0) = ρ(,t | ,0)  d   ,    t > 0   ,
       lim_t→ 0_+ρ(,t | ,0)  d = δ_0(-)   .


Stochastic process  has almost-surely continuous trajectories and for all t > 0, (t) is a second-order random variable,


    ∀ t ≥ 0    ,    E{‖(t)‖^2} < +∞  .


For t→ +∞,  is asymptotic to a stationary stochastic process whose first-order marginal probability measure is the invariant measure p_()  d,


    lim_t→ +∞ρ(,t | ,0)  d = p_()  d  .


The function (,)↦ρ(,t | ,0) is continuous from ^ν×^ν into ^+*,


    ρ(·,t | · ,0) ∈ C^0(^ν×^ν, ^+*)   .





 (Proposition <ref>).
The proof is presented in five steps.

(a) It is easy to prove that drift function b is continuous on ^ν.


(b) Since the diffusion matrix is the identity matrix and b belongs to C^0(^ν,^ν), we can establish the existence of a unique diffusion stochastic process (see <cit.> Ch. VIII, Sec. 2;  <cit.> Ch. IV, Secs. 2, 3, and 5; or <cit.> Ch. V), provided that for all  and ' in ^ν, we have


    ‖()-(')‖ ≤  c ‖ - '‖   ,   ‖()‖ ≤ 
    C  (1 +‖‖)  .


Using Eqs. (<ref>), (<ref>), and (<ref>), () can be rewritten, for all  in ^ν, as


    () =1/2 ξ()^-1∇ξ()    ,  ξ() = 1/n_d∑_j=1^n_dexp{ -1/2ŝ^2‖ŝ/s^j - ‖^2 }  >   0   .


Calculating ∇ξ(), it can be seen that


    2 ‖()‖ ≤ 1/ξ()  n_d  ŝ^2∑_j=1^n_dŝ/s ‖^j‖ exp{ -1/2ŝ^2‖ŝ/s^j - ‖^2 } + 1/ŝ^2 ‖‖  .


Since n_d is finite and since E{‖‖^2} = 1/n_d ∑_j=1^n_d‖^j‖^2 = ν is also finite, we have
sup_j=1,…,n_d‖^j‖ = c_η < +∞ and therefore,


    2 ‖()‖ ≤ 1/2 ( 1/ŝ^2ŝ/s  c_η + 1/ŝ^2 ‖‖ ) ≤  c  (1 + ‖‖)   ,    0 < c < +∞  .


The second inequality in Eq. (<ref>) is then proven. It can now be verified that

    ‖()-(')‖  ≤ 1/2ŝ^2 ( ‖() - (')‖  +  ‖-'‖  )   ,


in which ()= ()/β() where () and β() are written as


    () = ŝ/s∑_j=1^n_d^j exp{ -1/2ŝ^2‖ŝ/s^j - ‖^2 }   ,   β() = ∑_j=1^n_dexp{ -1/2ŝ^2‖ŝ/s^j - ‖^2 }  .


We must have ‖() - (')‖ ≤ c ‖-'‖, that is true if
‖   [∇()] ‖_F  ≤ c̃ < +∞, in which the (ν×ν)-real matrix [∇()] is written as
[∇()] =β()^-1 [∇()] - β()^-2 ()⊗∇β(), and where ‖ · ‖_F is the Frobenius norm. Introducing, temporary, the notation e_j = exp{ -1/2ŝ^2‖ŝ/s^j - ‖^2 }, we have


    [∇()] = 1/ŝ^2 ŝ^2/s^2∑_j=1^n_d∑_j^'=1^n_d e_j e_j^' (^j⊗^j - ^j⊗^j^')/∑_j=1^n_d∑_j^'=1^n_d e_j e_j^'  .


It can then be deduced that,


    ‖  [∇()]  ‖_F  ≤ 1/ŝ^2 ŝ^2/s^2sup_j,j^' ‖^j⊗^j - ^j⊗^j^'‖_F   .


As previously, since sup_j  ‖^j‖ = c_η < +∞, we have ‖  [∇()]  ‖_F  ≤ c̃_F with c̃_F  a finite positive constant independent of , that yields the proof for the Lipchitz continuity of .



(c) We will admit that, for all t >0 and for all  in ^ν, the transition probability measure has a density ↦ρ(,t | ,0) on ^ν with respect to d, and that (,)↦ρ(,t | ,0) is continuous on ^ν×^ν (for all t > 0) (see for instance <cit.> Ch III, Secs. 6 to 9, or <cit.> Ch. 10).


(d)  As a diffusion stochastic process, the trajectories are almost surely continuous functions. Since  is continuous
and ‖()‖ ≤  c  (1+‖‖) implies ‖()‖^2 ≤  2c^2  (1+‖‖^2), and since E{‖(0)‖^2} = ‖‖^2 < +∞ for all fixed  in ^ν, we have Eq. (<ref>) (see <cit.> Ch. IV, Sec. 2).


(e) The existence of the asymptotic stationary solution with Eq. (<ref>) can be found in <cit.> Ch. III, or <cit.> Ch. VI, Secs. 5 and 6.






 §.§ Rewriting the Itô equation in a matrix form



Let {^j(t),t≥ 0} be the solution of Eqs. (<ref>) and (<ref>) with the initial condition ^j(0) =^j∈^ν, in which ^j is defined in Section <ref>. We then have


    ρ(,t | ^j,0)  d = P_^j(t) | ^j(0)(d,t | ^j,0)  .


Let { [ (t) ] , t ≥ 0} be the _ν,n_d-valued stochastic process and [η_d] be the matrix in _ν,n_d such that


    [ (t) ] = [^1(t) …^n_d(t) ]    ,    [η_d] = [^1 …^n_d]   .


Therefore, { [ (t) ] , t ≥ 0} is solution of the matrix-valued ISDE,


    d[(t)] =     1/2 [L([(t)])]  dt + d[(t)]    ,    t > 0   , 
    
    [(0)] =      [η_d]   ,   a.s.   ,


where [y]↦ [L([y])] is the function from _ν,n_d into _ν,n_d, defined, for k∈{1,… , ν} and
j∈{1,… , n_d}, by


    [L([y])]_kj = 1/ξ(^j) ∂ξ(^j) /∂ y^j_k  .


In Eq. (<ref>), ^j is the j-th column of [y] = [^1 …^n_d], where ^j = (y^j_1,… , y^j_ν). Additionally, ξ() is defined by Eq. (<ref>), and { [ (t) ] , t ≥ 0} is the normalized _ν,n_d-valued Wiener stochastic process.




 §.§ Time-discrete approximation of the ISDE and convergence analysis



To estimate [K()]_ij, defined by Eq. (<ref>), using nonparametric statistics, we need to generate realizations of
Eqs. (<ref>) and (<ref>). Consequently, a first stage involves  introducing a time-discrete approximation of
Eq. (<ref>) and analyzing the convergence.


(i) Time sampling Δ t and δ t.

We first define a time sampling , n≥ 1 to be used for estimating [K()]. The time step Δ t
will be defined in Section <ref>. However, Δ t may not be sufficiently small to achieve  a satisfactory
rate of convergence. We then introduce δ t ≤Δ t such that Δ t = n_s ×δ t, with n_s ≥ 1. To discretize Eq. (<ref>), we use the time sampling {μ δ t, μ≥ 1}.


(ii) ISDE discretization.

Assuming that δ t is sufficiently small relative to 1, employing the Euler scheme (as seen, for example, in <cit.>) to discretize the solution { [ (t) ] , t ≥ 0} of Eqs. (<ref>) and (<ref>) yields


    [_μ] =       [_μ -1] + δ t/2  [L([_μ -1])] + √(δ t)  [_μ]
                       ,   μ≥ 1   , 
    
    [_0] =      [η_d]  ,   a.s.          ,


where [_μ] is the approximation of [(t)] at t = μ δ t, and where
{[_μ]_kj  ; k=1,…,ν  ; j = 1,… , n_d  ; μ≥ 1} is an infinite family of independent normalized Gaussian real-valued random variables.



(iii) Convergence of the time-discrete approximation.

For each j in 1,…,n_d, Proposition <ref> can be applied to the stochastic process {^j(t), t≥ 0}, where ^j(t) represents the j-th column of [(t)].
Let { [^δ(t)],t≥ 0} be the time-discrete approximation of { [(t)],t≥ 0}. Take any fixed positive real number T,  let μ_T = int (T/δ t) denotes the nearest integer to T/δ t. It can be observed that as δ t approaches 0, μ_T tends to +∞. The following classical result holds (see, for instance, <cit.>, Page 323).


 

Under Proposition <ref>, the time-discrete approximation { [^δ(t)],t≥ 0} of { [(t)],t≥ 0} converges strongly  at time T if


    lim_δ t → 0 E{ ‖   [_μ_T] - [(T)] ‖^2_F } = 0   ,


where [_μ_T] is determined by Eqs. (<ref>) and (<ref>) up to μ=μ_T.



(iv) Criterion to determine if δ t is small enough.

We can employ a criterion based on weak convergence, which is related to the covariance matrix [C_^j(t)] ∈_ν^+ of the ^ν-valued random variable ^j(t), for t fixed in the interval ]0, T].

Let σ_^2(t) be defined by


    σ_^2(t) = E{ ‖ [(t)] - E{[(t)]} ‖_F^2 } = ∑_j=1^n_d∑_k=1^ν
                   E{ (Y_k^j(t) - E{Y_k^j(t) } )^2}  .


It can easily be seen that


    σ_^2(t) = ∑_j=1^n_d [C_^j(t)]   .


Let σ__μ_T^2  be the corresponding quantity for random matrix [_μ_T],


    σ__μ_T^2 = E{ ‖ [_μ_T] - E{[_μ_T]} ‖_F^2 } = ∑_j=1^n_d [C_^j_μ_T]  .


The criterion can then be based on the following properties,


    lim_δ t → 0  | σ__μ_T - σ_(T) |   =   0   .


We will detailed this criterion in paragraph (vi).



(v) Generation of independent realizations of the time-discrete approximation.

Let n_s≥ 1 and N > 1 be two fixed integers. Let Δ t be fixed and let δ t = Δ t / n_s. Note that μ_T introduced in Section <ref>-(iv) is such that μ_T = n_s× N. We define T, , and _μ by,


    T = N ×Δ t = n_s× N ×δ t    ,    ={1,… , N}   ,   _μ = { 1,… , n_s × N}  .


For all μ in _μ, let { [γ_μ^ℓ]∈_ν,n_d  , ℓ=1,…, n_} be n_ independent
realizations of the random matrix [_μ]. As discussed in  Section <ref>-(ii), it can be seen that the family
{ [γ_μ^ℓ]  , ℓ=1,…, n_  ; μ∈_μ} consists of n_× n_s× N independent realizations. For each ℓ in {1,… , n_}, the realization { [ỹ_μ^ℓ]  , μ∈_μ} of the time-discrete approximation
{ [_μ]  , μ∈_μ} is computed using the following recurrence (refer to Eqs. (<ref>) and (<ref>)),


    [ỹ_μ^ℓ] =      [ỹ_μ-1^ℓ] + δ t/2  [L([ỹ_μ-1^ℓ])] + √(δ t)  [γ_μ^ℓ]
                       ,   μ∈_μ  , 
    
    [ỹ_0^ℓ] =      [η_d]                      ,


In the following, for performing the statistical estimations, we will use the subsequence
{ [y_n^ℓ]  , n∈} of { [ỹ_μ^ℓ]  , μ∈_μ}, such that


    ∀ n ∈   ,    [y_n^ℓ] = [ỹ_μ^ℓ]    ,   μ = n_s× n  .



(vi) Practical criteria for controlling the convergence parameters.

Let Δ t be fixed, as well as n_s, meaning δ t is fixed, to ensure satisfaction of the  convergence criteria introduced in Section <ref>-(iv). An additional practical criterion can be applied to verify the adequacy of all the convergence parameters, as provided in Lemma <ref>.


 

For k∈{1,… , ν}, j∈{1,…, n_d}, and n∈, let [y_ n] and [σ_n] be the matrices in _ν,n_d defined by,


    [y_ n]_kj =  1/n_∑_ℓ=1^n_ [y_n^ℓ]_kj   ,   
    ([σ_n]_kj)^2 =  1/n_∑_ℓ=1^n_ ([y_n^ℓ]_kj - [y_ n]_kj) ^2  .


Let n↦y(n) and n↦σ(n) be the positive-valued functions defined, for n∈, by


    y(n)  = 1/√(ν× n_d)  ‖ [y_ n]  ‖_F
        ,   σ(n) = 1/√(ν× n_d)  ‖ [σ_n]  ‖_F   .


Then, for δ t → 0 and n_→ +∞, we have


    lim_N→ +∞ y(n)  = 0    ,   lim_N→ +∞ σ(n) = 1   .





 (Lemma <ref>).

It can be seen that [y_ n]_kj and ([σ_n]_kj)^2 are the empirical estimates of the mean value and the variance of the time-discrete approximation [_μ]_kj for μ = n_s× n of Y_k^j(). From Proposition <ref>, we know that, for t→ +∞, {(t)  , t≥ 0}, and consequently, {^j(t)  , t≥ 0}, is asymptotically stationary and that Eq. (<ref>) holds. Since E{} = _ν and E{⊗} = [I_ν] (see Eqs. (<ref>) and (<ref>)), if
δ t → 0 and n_→ +∞, we have for N→ +∞,
[y_ n]_kj→ E{H_k} = 0 and  ([σ_n]_kj)^2 → [C_d]_kk = 1.
Since ‖ [y_ n]   ‖_F^2 = ∑_k=1^ν∑_j=1^n_d ( [y_ n]_kj)^2 and
‖ [σ_n]   ‖_F^2 = ∑_k=1^ν∑_j=1^n_d ( [σ_n]_kj)^2, using the same normalization 1/√(ν× n_d) for y(n) and σ(n), we obtain Eq. (<ref>).







 §.§ Estimation of the matrix of the finite approximation



We now can estimate [K()]_ij defined by Eq. (<ref>) using the realizations { [y_n^ℓ],ℓ=1,… , n_} for n∈, defined by Eq. (<ref>).


 
Under Propositions <ref> and <ref>, results and notations of Section <ref>-(v), for
n ∈, an estimate of matrix [K()],  whose entries are defined by Eq. (<ref>), is written as


    [K()] = [B]^-1  [()]  ,


in which [B] is the diagonal matrix in _n_d for which its entries are


    [B]_ij = δ_ij ∑_j'=1^n_dexp{ -1/2 ŝ^ 2‖^i - ŝ/s^j'‖^2 }  .


The entries of matrix [()] in  _n_d are written as,


    [()]_ij =
     1/n_∑_ℓ=1^n_ ( ŝ^ ν/ s^ν_∏_k=1^ν [σ_n]_kj )
      exp{ -1/2∑_k=1^ν ( η_k^i - [y_n^ℓ]_kj/s_ [σ_n]_kj )^2  }  ,


in which s and ŝ are defined by Eq. (<ref>), where s_ is written as


    s_ = {4/n_(2+ν)}^1/(ν +4)  ,


where [y_n^ℓ]∈_ν,n_d is defined by Eq. (<ref>), and where [σ_n]∈_ν,n_d is defined by Eq. (<ref>).




 (Proposition <ref>).

Eq. (<ref>) is deduced from Eq. (<ref>) with t= and with p_(^i) given by Eq. (<ref>) in which =^i, and with the estimate of ρ(^i, | ^j,0) given by the Gaussian kernel-density estimation method (see <cit.>) with the Silverman bandwidth s_ defined by Eq. (<ref>), which is written as


    ρ(^i, | ^j,0) =
     1/n_∑_ℓ=1^n_1/(√(2π) s_)^ν∏_k=1^ν [σ_n]_kjexp{ -1/2∑_k=1^ν ( η_k^i - [y_n^ℓ]_kj/s_ [σ_n]_kj )^2  }  .






  

(a) In Section <ref>, we will provide an illustration by numerically solving the eigenvalue problem defined by Eq. (<ref>) for t= n Δ t, using Eq. (<ref>). Specifically, we consider the Gaussian case with dimension ν=1. In this scenario,  represents a normalized Gaussian real-valued random variable. For this case, a reference solution is available.

(b) Nevertheless, it would be challenging to employ such a formulation with reasonable convergence in very high dimensions, where ν equals several tens or even hundreds. This would necessitate a large value of n_d.

(c) In fact, Proposition <ref> will be employed in Section <ref> to derive an expression that connects to the kernel [_]_ij = exp{ -1/4ε_‖^i-^j ‖^2 }, which is used to calculate the DMAPS basis and proves efficient for large values of ν.






§ NUMERICAL ILLUSTRATION OF THE PROPOSED FORMULATION




As explained in Remark <ref>-(a), this section presents a numerical illustration of the formulation introduced in Section <ref> to solve the approximated eigenvalue problem defined by Eq.(<ref>), which is derived from the eigenvalue problem in Eq.(<ref>). To validate the formulation, we select a reference case where the eigenvalue problem defined by Eq. (<ref>) can be exactly solved. This is feasible when the dimension ν of  is 1 and  is a normalized Gaussian real-valued random variable.




 §.§ Reference case definition and explicit solution



The quantities related to the reference case will be indexed by letter r. The probability density function of H_r on 
is p_H_r(y) = (2π)^-1/2exp(-y^2/2). The potential function Φ_r (see Eq. (<ref>)) is Φ_r(y) = y^2/2 and the drift b_r defined by Eq. (<ref>) is b_r(y)  = -y/2. The ISDE defined by Eqs. (<ref>) and (<ref>) are rewritten as



    dY_r(t)      =  -1/2 Y_r(t)  dt + dW_r(t)    ,    t > 0  , 
    
      Y_r(0)      =  x ∈  .


Consequently, {Y_r(t) , t≥ 0} is a second-order Gaussian stochastic process, which is explicitely defined by


    Y_r(t) = x  e^-t/2 + ∫_0^ t e^-(t-τ)/2  dW_r(τ)   .


A simple calculation shows that the mean value m_r(t) = E{Y_r(t)} and the standard deviation
σ_r(t) = (E{(Y_r(t) -m_r(t))^2})^1/2 of the random variable Y_r(t) for fixed t > 0, are written as


    m_r(t) = x  e^-t/2   ,   σ_r(t) = √(1-e^-t)  .


For all t > 0, as  { Y_r(t) |  Y_r(0) = x} is a Gaussian random variable, the transition probability density function is


    ρ_r(y,t  |  x,0) = 1/√(2π) σ_r(t)exp{ -1/2σ_r(t)^2(y-m_r(t))^2}  .


Note that Eqs. (<ref>) and (<ref>) show that, we effectively have
lim_t→ 0_+ρ_r(y,t  |  x,0)  dy = δ_0(y-x) (see Eq. (<ref>)) and
lim_t→ +∞ρ_r(y,t  |  x,0) = p_H_r(y) (see Eq. (<ref>)).
It can be deduced that the kernel k_r,t(y,x) on ×, defined by Eq. (<ref>) is written, for t > 0, as


    k_r,t(y,x) = 1/√(1 -e^-t)exp{- 1/2e^-t/(1-e^-t) (y^2 + x^2 -2e^t/2 y  x)^2}  .


Let {h_α(y)  , α∈} be the Hermite polynomials and {h_α(y)/√(α!)  , α∈} the Hilbert basis in L^2(;p_H_r),


    ∫_h_α(y)/√(α!) h_β(y)/√(β!)  p_H_r(y)   dy  = δ_αβ   ,    h_0(y)=1    ,     h_1(y) = y    ,     h_α+1(y) = y  h_α(y) - d  /dy h_α(y)  .


Let {_α(y)  , α∈} be the polynomials defined by


    h_α(y) = 1/√(2^α) _α(y/√(2))    ,   _α(y) = √(2^α)  h_α(√(2) y)    .


We have the formula <cit.>,


    1/√(1 - 4a^2)exp{4 a/1-4 a^2 (x̂ ŷ - ax̂^2 - a ŷ^2)} =
    ∑_α∈a^α/α ! _α(x̂)  _α(ŷ)  .


Taking x= √(2) x̂, y= √(2) ŷ, and a =1/2e^-t/2 < 1/2 for t > 0, Eq. (<ref>) can be rewritten as


    k_r,t(y,x) = ∑_α∈ e^-λ_r,α tψ_r,α(y) ψ_r,α(x)   ,




    λ_r,α = α/2   ,   ψ_r,α(y)= 1/√(2^α α !) _α(y/√(2))
    = 1/√(α !)  h_α(y)  ,


and thus, for α and β in ,


    ∫_ψ_r,α(y) ψ_r, β(y)  p_H_r(y)  dy = δ_α, β  .


Comparing Eqs. (<ref>) and (<ref>)  with Eqs. (<ref>) and (<ref>) shows that, for this reference case, the eigenvalues of the Fokker-Planck operator L_ are


    λ_r,α = α/2    ,   α∈  .






 §.§ Estimating the eigenvalues with the proposed numerical formulation



(i) For the convergence analysis, we consider 10 values of n_d constituting the set _d = {100, 300, 400, 800, 1000, 1200,  1500, 1800, 2000, 2200}. For each n_d∈_d, the matrix [η_d] =[η^1 …η^n_d]∈_1,n_d is generated with an adapted generator (instruction randn(1,n_d) for Matlab Gaussian generator).

(ii) For each value of n_d in _d, the generation of n_ = n_d independent realizations { [ỹ_μ^ℓ] , μ∈_μ} with _μ = {1,… , n_s× N} is performed using Eqs. (<ref>) and (<ref>) with n_s=1, δ t = Δ t = 0.061796, N=150. For n_d=1200 and n_ = 1200, Figs. <ref> and <ref> display the graphs of functions n↦y(n) and n↦σ(n) computed  using Eq. (<ref>) with Eq. (<ref>). It can be seen that the criterion defined by Eq. (<ref>) is satisfied.




(iii) For every n_d∈_d, matrix [K(2 Δ t)] ∈_n_d^+0 is calculated using Eq. (<ref>) with
ŝ = 0.2486, ŝ / s = 0.9690, and s_ =0.2565. The first 6 largest eigenvalues, b̂_α(2 Δ t) of
[K(2 Δ t)] and  _α(2 Δ t), simply denoted by _α, are obtained from
Eq. (<ref>).

Fig. <ref> shows the graph of function n_d↦_λ(n_d) that quantifies the relative error between α↦λ_r,α and α↦_α, written as
_λ(n_d) = ∑_α=0^5(λ_r,α- _α)^2 / ∑_α=0^5 λ_r,α^2.
It can be seen that the error decreases as n_d increases. For n_d = n_ = 1200, Fig. <ref> compares the reference
eigenvalues λ_r,α =α/2 (see Eq. (<ref>)) with the computed eigenvalues _α  for α=0,1, …,5. The comparison is good enough.







§ VECTOR BASIS FOR PLOM DERIVED FROM THE TRANSIENT ANISOTROPIC KERNEL, CONNECTED TO THE DMAPS BASIS




We revisit the objective presented in Section <ref>.
The diffusion-maps (DMAPS) basis, used by PLoM <cit.>, is associated with the isotropic kernel detailed in <cit.>. This section introduces the construction of a transient vector basis, based on the transient anisotropic
kernel described in Section <ref> (see Proposition <ref>). This approach incorporates the requirement that as Δ t → 0, the transient kernel at the first time Δ t coincides with the DMAPS isotropic kernel. Consequently, the two vector bases will be linked asymptotically as t→ 0. To develop such a "connected" transient anisotropic kernel to the DMAPS isotropic kernel, we undertake a reparameterization of Eq. (<ref>), which defines the matrix [K()]∈_n_d^+0 for n∈. This reformulation enables the construction of the vector basis for high dimensions (large values of ν) and relatively small n_d, as implemented in PLoM, designed for probabilistic learning with small training datasets.




 §.§ Transient anisotropic kernel connected to the DMAPS isotropic kernel



The following definition provides the construction of a transient anisotropic kernel that is linked to the DMAPS isotropic kernel. This construction draws inspiration from the expression of [K(Δ t)] defined by Eq. (<ref>).



 
For each n in , for Δ t defined in Section <ref>-(i), for n_ and [y_n^ℓ] defined in
Section <ref>-(v), for [σ_n] defined by Eq. (<ref>), and for  a given real ε_ > 0, we define the
matrix K̃()]∈_n_d^+0, such that


    [K̃()] = [B]^-1 [()]  ,


such that, for all i and j in {1,…,n_d}, the matrix [()]∈_n_d has entries,


    [()]_ij =
     1/n_∑_ℓ=1^n_ ( Π_k=1^ν{[σ_n]_kj/Δ t} )^-1exp{ -1/4ε_∑_k=1^ν ( η_k^i - [y_n^ℓ]_kj/[σ_n]_kj/√(Δ t) )^2  }  ,


and [B]∈_n_d is a diagonal matrix whose entries are,


    [B]_ij = δ_ij∑_j'=1^n_dexp{ -1/4ε_‖^i -^j'‖^2 }  .







 

An optimal value, ε_, for the smoothing parameter ε_, is proposed for the PLoM algorithm in <cit.> and is detailed in <ref>-(i). This optimal value enables the analysis of high-dimensional problems (large value of ν).
It should be noted that Proposition <ref> will explain the definition of [K̃()] as defined by Eqs. (<ref>) to (<ref>), showing the connection with DMAPS.





 
We use the notation introduced in Definition <ref>. Let [_]∈_n_d^+0 be the matrix of the DMAPS isotropic kernel, defined for all  i, j ∈{1,… , n_d}, by


    [_]_ij = exp{-1/4 ε_‖^i-^j‖^2}  .


Let [K_]∈_n_d be the matrix defined by


    [K_] = [B]^-1 [_]  ,


where [B] is the diagonal matrix defined by Eq. (<ref>). Let Δ t be defined by


    Δ t = ŝ^2 / κ   ,   κ≥ 1   ,


where ŝ is defined by Eq. (<ref>). Then, as κ→ +∞, and consequently Δ t → 0, we have


    [K̃(Δ t)] → [K_]   ,


which means that, for n=1, the matrix [K̃(nΔ t)] of the transient anisotropic kernel, defined by Eq. (<ref>), converges to the matrix [K_] of the DMAPS isotropic kernel as Δ t → 0.





 (Proposition <ref>).
For n=1, for κ→ +∞, that is to say, for Δ t → 0, Eqs. (<ref>) and (<ref>) show that, for all ℓ∈{1,…,n_}, k∈{1,…,ν}, and j∈{1,… , n_d}, we have [y_1^ℓ]_kj→η_k^j and [σ_n]_kj→√(Δ t), and consequently, [(Δ t)] → [_]. From Eqs. (<ref>) and (<ref>), it can be deduced Eq. (<ref>).





 §.§ Construction of the reduced-order transient basis and its counterpart for the DMAPS basis



In this section, we: (i) review the construction of the reduced-order diffusion-maps (DMAPS) basis (RODB) represented by a matrix
[g_]∈_n_d,m_ with m_ < n_d, and (ii) construct, for each n∈, the reduced-order transient basis (ROTB()) represented by a matrix [g(nΔ t)]∈_n_d,m_.




(i) Reminder of the construction of the RODB.

This construction, due to <cit.>, is the one used by the PLoM algorithm <cit.>.
The matrix [K_], defined by Eq. (<ref>), has positive entries and represents the transition matrix of a Markov chain.
The eigenvalues b_,1,…,b_,n_d and the associated eigenvectors _^1,…,_^n_d are such that


    [K_] _^ β = b_, β _^ β   ,   
    1=b_,1 > b_,2≥…≥ b_,n_d  .


Using Eq. (<ref>), this eigenvalue problem  is rewritten as the symmetric eigenvalue problem,


    ^S_ ^ β = b_, β ^ β   ,   ⟨^ β,^ β'⟩ =δ_ββ'   ,   _^ β = [B]^-1/2^ β  ,


in which ^S_ = [B]^-1/2[_] [B]^-1/2 is a symmetric matrix.
The diffusion-maps basis {_^1,…, _^n_d} forms a vector basis of ^n_d.
As explained in <cit.>, PLoM uses the RODB of order m, which  is defined by
{_^1 …_^ m}.

This basis depends on  ε_ and m.

The optimal value m_ of m is defined (see <cit.>) by


    m_ = ν +1   .


The optimal value ε_ of ε_ is estimated to obtain


    1 = b_,1 > b_,2≃…≃ b_,m_≫ b_,m_ + 1≥…≥ b_,n_d > 0  ,


in which the jump amplitude


    J_ = b_,m_ + 1  /b_,m_  ,


must be equal to J_=0.1 (following <cit.>), but which can also be chosen in the interval [0.1  , 0.5] when ν is large.

Therefore, the RODB is defined for m = m_ and is represented by the matrix,


    [g_] = [_^1 …_^m_] ∈_n_d,m_  .




(ii) Construction of the ROTB().

For each n in , the ROTB() represented by [g(nΔ t)] is constructed as the eigenvectors of matrix
[K̃(nΔ t)] defined by Eq. (<ref>). Taking into account Eq. (<ref>), we want that for n=1, [g(Δ t)]∈_n_d,m_ goes to [g_]∈_n_d,m_ as Δ t → 0. For n > 1, matrix [K̃(nΔ t)] is a priori symmetric, but matrix [(nΔ t)] is not symmetric. For applying a similar approach to the one defined by Eq. (<ref>), we then symmetrize the matrix [()] = [B]^-1/2 [(nΔ t)]   [B]^-1/2, introducing
[^S()] = ([()] + [()]^T)/2.

The transient basis {^1(), … , ^n_d()} associated with the eigenvalues
b̃_1() ≥b̃_2() ≥…≥b̃_n_d()
are then computed by solving the symmetric eigenvalue problem


    [^S()] ^ β() =  b̃_β()  ^ β()    ,   ⟨^ β()   , ^ β'()⟩ = δ_ββ'  .




    ^ β() = [B]^-1/2^ β()   .


The ROTB() is then defined by the matrix



    [g()] = [^1() …^m_()]∈_n_d,m_  ,


in which m_ is defined as explained in Section <ref>-(i).




 §.§ Criteria for comparing the reduced-order transient basis with the reduced-order DMAPS basis



The PLoM algorithm is based on the use of the RODB (see <ref>). With such an RODB based on the isotropic kernel, PLoM has proven generally efficient, even in extremely difficult cases, as demonstrated in numerous publications since 2016. This efficacy will also be evident through the applications presented in Section <ref>. However, for cases involving very heterogeneous data in the training dataset, the learned statistical dependence between the components of the random vector  (through the learned probability measure for ) can a priori be improved using ROTB() for given n. To assess a possible improvement with this reduced-order transient basis compared to the reduced-order DMAPS basis, quantitative criteria are necessary. In this section, we introduce criteria for comparing the two vector bases, and then, in Section <ref>, we will present a methodology for identifying the instance, , that maximizes the selection criteria of the ROTB().


(i) The first criterion will be the angle between the two vector subspaces generated by the two vector bases, ROTB() and RODB. If this angle is close to zero then the two bases coincide. This must be the case when we choose  the instant n=1 for the ROTB()  (See Proposition <ref>).

(ii) The second criterion is the concentration of the learned probability measure in relation to the concentration of the probability measure of the training dataset. It has been proven that the PLoM algorithm, which uses the RODB, was designed to preserve the concentration of the learned probability measure from a small training dataset. For this, in <cit.>, we introduced the indicator d^2, linked to the mean-square convergence, which we will recall and use. We also introduce a second concentration criterion, KL, based on the Kullback-Leibler divergence <cit.> between the learned probability measure and the probability measure of the training dataset.

(iii) The third criterion is mutual information <cit.>, which is defined as the relative entropy introduced by Kullback and Leibler <cit.>. This will be used to quantify the level of statistical dependencies among the components of the centered random vector , whose covariance matrix is the identity matrix. Such mutual information will be estimated for the learned probability measure generated by PLoM, comparing the RODB and the ROTB() as a function of n.

(iv) The last criterion is the Entropy from Information Theory <cit.>, introduced by Shannon <cit.>. The estimation of this entropy is a function of the number of realizations  used for estimating the probability measure. This property will be used in Section <ref> to normalize the estimation of the mutual information.




  §.§.§ Angle between the subspaces spanned by ROTB() and RODB



Let n be fixed in . It is assumed that rank([g_]) = rank([g()])= m_ in which the two matrices are defined by Eqs. (<ref>) and (<ref>). In addition, it is assumed that the null space of the matrix [g()]^T  [g_] is {0}.
Let V() = span{^1(), … , ^m_()} be the m_-dimension subspace of ^n_d (spanned by the ROTB()).
Let V_ = span{^1_, … , _^m_} be the m_-dimension subspace of ^n_d (spanned by the RODB).

For β∈{1,… m_}, let ĝ_^ β = g_^ β / ‖ g_^ β‖ be the normalized vector g_^ β and let ĝ^ β() = g^ β() / ‖ g^ β()‖ be the normalized vector g^ β().
Let [ĝ_] = [ĝ^1_…ĝ_^m_ ] and
[ĝ()] = [ĝ^1()…ĝ()^m_ ] be matrices in _n_d, m_.
The angle γ () between the subspaces V() and V_ is defined by


    γ () = arccos ( σ_min([ĝ()]^T  [ĝ_] ) )   ,


in which σ_min denotes the smallest singular value. If the angle is close to 0, the two subspaces are nearly linearly dependent.



  §.§.§ Indicators related to the learned probability measure



In this section, we detail the indicators used for comparison: concentration of the learned probability measure, mutual information, and entropy. These will be employed to compare the probability measures associated with the training dataset and the learned dataset generated by the PLoM algorithm, as summarized in <ref>. Specifically, we use the reduced-order DMAPS basis (RODB) and, alternatively, the reduced-order transient basis (ROTB()). To facilitate the comparisons presented in the numerical illustrations, we introduce the necessary notations for clarifying the diverse quantities and their numerical calculations.

- Training dataset. In Section <ref>, the independent realizations of the ^ν-valued random variable  are ^1,… ,^n_d, and the matrix [η_d] = [^1…^n_d]∈_ν,n_d represents one realization of the random matrix [], used by PLoM, and defined in <ref>. The probability density function of  is p_ defined by Eq. (<ref>).

- Learned dataset. In <ref>, the learned realizations generated by PLoM are those of the _ν,n_d-valued random variable [_] and are written as [η_^ℓ], ℓ=1,…, n_ (see <ref>). The corresponding realizations _^ℓ', ℓ=1,…, n_, with n_ = n_d × n_, of the ^ν-valued random variable _ are obtained by reshaping. The learned probability density function of _ is denoted by p__. When PLoM is used with RODB, [_], [η_^ℓ], _^ℓ', and p__ will be rewritten as [_], [η_^ℓ], _^ℓ', and p_, respectively. When PLoM is used with ROTB(), [_], [η_^ℓ], _^ℓ', and p__ will be rewritten as [_()], [η_^ℓ()], _^ℓ'(), and
p_( .   ; ), respectively.



(i) Concentration of the learned probability measure for RODB and ROTB().

Based on the mean-square norm, for PLoM with RODB and ROTB(), the learned probability measure concentration is written (see Eq. (<ref>)) as


    d_^  2(m_) = E{ ‖  [_] - [η_d] ‖^2 }   / ‖   [η_d] ‖^2  ,



    d_^  2(m_;) = E{ ‖  [_()] - [η_d] ‖^2 }  /  ‖  [η_d] ‖^2  .


Using the realizations, these quantities are estimated by


    d̂_^  2(m_) = (1/n_)∑_ℓ=1^n_{ ‖   [_^ℓ]  - [η_d] ‖^2 }   / ‖   [η_d] ‖^2   .



    d̂_^  2(m_; ) = (1/n_)∑_ℓ=1^n_{ ‖   [_^ℓ()]  - [η_d] ‖^2 }  / ‖  [η_d] ‖^2   .


The concentration of the learned probability measure can also be estimated using the Kullback-Leiber divergence and its estimation from a set of realizations as presented in <ref>. For the PLoM formulated with RODB and ROTB(), we then have,


    D(p_ ‖  p_ )  = ∫_^ν p_() log ( p_()/p_())   d  ,



    D(p_( .  ;) ‖  p_ )  = ∫_^ν p_(  ; ) log ( p_( ; )/p_())   d  .




(ii) Mutual-information-based components statistical dependencies of the learned probability measure for  RODB and ROTB().

The mutual information and its estimation from a set of realizations are presented in <ref>. For the training dataset and for PLoM formulated with RODB and ROTB(), we have , for ^ν-valued random variables , _, and _(),


    I() = D(p_ ‖ ⊗_k=1^ν p_H_k) =
                  ∫_^ν p_() log ( p_()/Π_k=1^ν p_H_k(η_k))   d  ,




    I(_) = D(p_ ‖ ⊗_k=1^ν p_,k) =
                 ∫_^ν p_() log ( p_()/Π_k=1^ν p_,k(η_k))   d  ,




    I(_;) = D(p_(·  ; )  ‖ ⊗_k=1^ν p_,k(·   ; )) =
       ∫_^ν p_(  ;) log ( p_(  ;)/Π_k=1^ν p_,k(η_k  ;))   d  ,


in which p_,k is the pdf of component H_,k of _ and where p_,k(·   ; ) is the pdf of component
H_,k() of _().



(iii) Entropy for RODB and ROTB().
The entropy and its estimation from a set of realizations are presented in <ref>. For the training dataset and for PLoM formulated with RODB and ROTB(), we have for ^ν-valued random variables , _, and _(),


    S_ = -∫_^ν p_() log p_()   d   ,   
    S_ = -∫_^ν  p_() log p_()   d  ,



    S_() = -∫_^ν  p_(  ; ) log p_(  ; )   d  .





 §.§ Identification methodology for the instant maximizing the selection criterion of the ROTB()



In this section, all quantities denoted with a hat (such as Î()) represent the estimated values of the corresponding quantities without a hat (such as I()), using the realizations as explained in <ref>.
Let n_d be the number of points in the training dataset and n_ = n_d× n_≫ n_d be the number of learned realizations with PLoM (see <ref>), either with the RODB or with the ROTB().



(i) Defining the subset _N of  containing the admissible values of n.

Let N be the largest value of n for which the ROTB(), represented by matrix [g()], is computed. This value N being fixed, the set = {1,…, N} defined by Eq. (<ref>) is fixed.

The subset _N of the admissible values of n is then defined by


    _N = { n∈  ,  1/ν d̂_^ 2(m_;) ≤τ_c ≪ 1}  ,


where d̂_^  2(m_;) is defined by Eq. (<ref>) and where 0 < τ_c ≪ 1 is fixed sufficiently small with respect to 1 in order to preserve the concentration of the learned probability measure (see Section <ref>-(i) and <ref>). It should be noted that the maximum value of d_^  2(n_d) is 2 (corresponding to a measure concentration completely lost, which can be obtained  with classical MCMC algorithms). The PLoM algorithm allows for preserving the concentration of the learned probability measure <cit.> yielding values of the order 0.01 to 0.1 through all the performed applications. For defining the concentration criterion in Eq. (<ref>), we have normalized with respect to dimension ν.


(ii) Characterizing a better ROTB()  compared to RODB.

From Proposition <ref>, in particular from Eq. (<ref>), it can be deduced that, for n=1, we have Î(_  ; 1 ×Δ t) ≃Î(_) for Δ t sufficiently small (which is the considered case).

Let us assume that Î() < Î(_). For n fixed in _N, we will say that the ROTB(), represented by matrix [g()], is better than the RODB, represented by matrix [g_], if


    Î() ≤Î(_  ; ) <  Î(_)   .



(iii) Determining the optimal value n_ of n in _N.

The underlying idea is to select a reduced-order transient basis that gives a learned probability measure whose mutual information is as close as possible to the mutual information of the probability measure p_(), d.

Based on paragraph (ii) above, the optimal transient basis [g(n_)] among the set of possible
transient bases { [g()] , n∈} is obtained for n=n_ such that


    n_ = min_n ∈_NÎ(_  ; )   .



(iv) Defining the normalized estimate of the mutual information.

As explained in <ref>, if n_samp is the number of realizations (either n_d or n_), the entropy estimation asymptotically decreases as -log(n_samp) when n_samp increases.
For comparing the estimate of the mutual information for , which uses n_d realizations, with the one of _ or _(), which uses n_≫ n_d learned realizations, we must normalize the estimated mutual information with respect to the number of realizations. Taking into account the relationship between the mutual information and the entropy estimates (see <ref>), we chose to normalize the estimated mutual information by dividing by the function χ + log(n_samp) (with n_samp=n_d or n_samp=n_), where χ is a real number that must be identified and which must be such that χ + log(n_d) > 0. Since n_ > n_d this condition implies that  χ + log(n_) > 0 is automatically verified.

For the ^ν-valued random variables , _, and _(), the normalized estimated mutual information are then defined by


    Î_() = Î()/χ + log(n_d)   ,   Î_(_) = Î(_)/χ + log(n_)   ,   Î_(_  ; ) = Î(_  ; )/χ + log(n_)  .


in which  Î(), Î(_), and Î(_  ; ) are the estimates (computed with Eq. (<ref>)) of the mutual information I(), I(_), and I(_  ; ) defined by Eqs. (<ref>) to (<ref>).

Based on paragraph (iii) above, constant χ is defined as follows. Let χ_ be the solution  in χ of the equation,


    Î()/χ + log(n_d) = Î(_  ; n_ Δ t)/χ + log(n_)  .


If χ_ + log(n_d) > 0, then χ_ is the desired value of χ.
It should be noted that the optimal value n_ is calculated with the estimation of the non-normalized mutual information (see Eq. (<ref>). Normalization is only introduced for the purpose of comparing the value of this criterion for  and _.




(v) Identification of the instant maximizing the selection criterion of the ROTB().

Let n_ be defined by Eq. (<ref>), and let χ_ be identified by solving Eq. (<ref>). Then, we have


    Î_() = Î_(_  ; n_Δ t) ≤Î_(_  ; )
       ,   ∀ n∈_N  .


We can then conclude that the ROTB(n_Δ t), represented by matrix [g(n_Δ t)], is better than the RODB, represented by matrix [g_]. The proof is straightforward.




§ NUMERICAL APPLICATIONS






 §.§ Preamble




We will present three applications, each with its own specificities. However, as the generation of the training dataset related to the random vector  is relatively complex to describe, we will start with the training dataset related to the normalized random vector  with values in ^ν. The number of realizations in the training dataset is n_d, and the training dataset is represented by the matrix [η_d] ∈_ν,n_d. This means that the PCA step of PLoM, which transforms  into  (see <ref>), is not detailed here. Readers interested in the training dataset for  can request, from the "corresponding author" of the article, the transfer of the matrix [η_d] for each application presented.

We will still briefly give the main specificities of these 3 applications.


Application 1. This application (Appli 1) was created so that the probability measure of  in ^9 (ν=9), which is defined by the points of the training dataset, is concentrated in a multiconnected domain of ^9, with the constituent connected parts being manifolds of dimensions much lower than 9, each having different dimensions. These parts may or may not be connected to each other.

Application 2. For the second application (Appli 2), the n_d realizations of the random vector  with values in ^8 are generated using a polynomial chaos expansion of degree 6 of a real random variable, whose random germ is of dimension 2, with each of the two random germs being a uniform random variable of different support. There are therefore 28 terms in this expansion, and the 8 components of  are defined as the random terms of rank 2, 3, 6, 8, 12, 13, 17, and 19. We thus define a relatively complex random manifold in ^8.

Application 3. The third application (Appli 3) results from a statistical treatment of an experimental database containing photon measurements in the ATLAS detector at CERN. This dataset was obtained by loading the file 'pid22_E262144_eta_20_25_voxalisation.csv' from the free access CERN Open Data Portal. The PCA step of PLoM has been performed, and an extraction of 45 components has been done to obtain the training dataset for the ^45-valued random variable . This application is in higher dimension than the first two, but the statistical complexity is less.



 §.§ Additional convergence analysis conducted for the three applications



For each of the three applications, we will provide detailed analyses of the calculation of the optimal value of the instant n_Δ t, which allows for the selection of the best reduced-order transient basis (ROTB). We will also present the convergence results of the PLoM algorithms under the normalization constraints (see <ref>). However, we cannot show all the convergence results for the other parameters. Below are the different analyses that were carried out with respect to the parameters that control the construction of the reduced-order transient basis:


(i) Convergence with respect to the value of κ. This point is important because Δ t must be sufficiently small to apply Proposition <ref>. For the three applications, we found that κ = 30 is an appropriate value. 


(ii) Convergence with respect to the size of δ t, that is to say, the value of n_s. For the three applications, we found that once Δ t is fixed by the value of κ, the criterion associated with Eq. (<ref>) was satisfied for n_s=1, as well as the evolution of the angle γ(n_Δ t) at the optimal instant was very insensitive to the values of n_s greater than
1. This is due to the fact that Δ t is already small enough to achieve good accuracy with the Euler scheme used to integrate the ISDE. We found that n_s=1, thus δ t = Δ t is a good value.


(iii) Convergence with respect to n_. This analysis was performed by examining the convergence of the angle γ() for n ∈. We observed that a large value of n_ was necessary to achieve convergence. This point is particularly important and must be carefully checked for the applications.




 §.§ Parameters defining the training dataset and controlling  the construction of the RODB



For each application, Table <ref> provides the values of the parameters that define the training dataset and the probability measure P_(d) = p_()  d on ^ν, as well as the parameters defined in Section <ref>-(i), which control the construction of the reduced-order DMAPS basis (RODB).







 §.§ Parameters controlling the construction of the ROTB()



For each application, Table <ref> provides the values of the parameters that control the construction of the reduced-order transient basis ROTB(). In particular, the optimal time n_Δ t is estimated using the indicators defined in Section <ref>. The evolution of these indicators as a function of n will be presented in detail in Sections <ref> to <ref> for the three applications.






 §.§ Parameters of PLoM with RODB and ROTB



For each of the three applications, Table <ref> provides the values of the parameters defined in <ref> and <ref> used by the PLoM algorithm with the reduced-order DMAPS basis (RODM) or the reduced-order transient basis ROTB(). Taking into account the convergence analysis carried out on n_ and the expertise on the statistical convergence of the quantities considered, we choose n_ = n_, a sufficiently large value (see <ref>).
For the PLoM algorithm, the constraints related to normalization, defined by Eq. (<ref>), are always applied in the computation (see <ref>)













 §.§ Results for Application 1




(i) Figure <ref> displays the graphs of the probability density function (pdf)  of components 1, 2, 4, and 5 for  estimated with the n_d realizations of the training dataset, and the pdf estimated with n_ learned realizations, for _ using MCMC without PLoM (a,d,g,j), for _ using  PLoM with RODB (b, e, h, k), and for _ using PLoM with ROTB(n_Δ t) (c, f, i, l).

(ii)  Figure <ref> shows the joint probability density function of components 4 and 5 of  estimated with the n_d realizations of the training dataset (a) and estimated with n_ learned realizations for _ using MCMC without PLoM (b), for _ using PLoM with RODB (c), and for _ using PLoM with ROTB(n_Δ t) (d).

(iii)  In Fig. <ref>, the clouds of n_ points corresponding to n_ learned realizations can be seen for components 1, 2, 3 (a,b,c) and components 3, 4, 5 (d,e,f). These are shown for _ using MCMC without PLoM (a,d), for _ using PLoM with RODB (b,e), and for _ using PLoM with ROTB(n_Δ t) (c,f).

(iv) Figure <ref> plots the functions that characterize the reduced-order transient basis ROTB() as a function of time :




  * The eigenvalues of matrix [K_] and those of the of symmetrized matrix [K̃()] are shown in
Fig. <ref>.


  * The probability-measure concentration using the d^ 2/ν-criterion is shown in Fig. <ref>.
For the learning without PLoM, the d^ 2-concentration is 0.6465, which shows that the concentration is lost, and for the PLoM with the RODM, the concentration is 0.0116, which shows that the concentration is preserved.


  * The other criterion of the probability-measure concentration is given by Kullback measure, shown in Fig. <ref>.
For the learning without PLoM, Kullbach is 0.3486, and for the PLoM with the RODM, Kullback is 3.9435. Comparing Figs. <ref> and <ref> shows that the two criteria are consistent and give the same analysis of the concentration.


  * The angle between the subspaces spanned by RODB and ROTB() is displayed in Fig. <ref>. It can be seen that, for the optimal time 9 Δ t, the angle is 53.03^∘, which is a significant angle showing that the two bases are different while the d^ 2-concentration remains small at 0.0174.


  * The entropy estimation of pdf p_(·  ; ) is given in Fig. <ref>.


  * The normalized mutual information (MI) of the pdfs p_ and p_(·  ; ) is shown in Fig. <ref>.
This figure shows that the optimal value of n is n_ = 9.

For the non-normalized estimation of the mutual information, we have Î() = 4.4668, Î(_  ; n_Δ t) = 6.9996, and Î(_) = 7.0945. For the normalized one, we have
Î_() = Î_(_  ; n_Δ t) = 0.3666 and Î_(_) = 0.3716.



(v) Finally, examination of these figures shows that traditional learning without PLoM gives poor results compared to PLoM, which allows the concentration to be preserved and properly learns the geometry of the probability measure support. We also see that PLoM with the optimal ROTB provides an improvement in learning compared to PLoM with RODM and, therefore, should improve the estimates of conditional statistics thanks to better learning of the joint probability measure.

























 §.§ Results for Application 2




(i) Figure <ref> displays the graphs of the probability density function (pdf)  of components 3, 5, 6, and 8 for  estimated with the n_d realizations of the training dataset, and the pdf estimated with n_ learned realizations, for _ using MCMC without PLoM (a,d,g,j), for _ using  PLoM with RODB (b, e, h, k), and for _ using PLoM with ROTB(n_Δ t) (c, f, i, l).

(ii)  Figure <ref> shows the joint probability density function of components 6 and 8 of  estimated with the n_d realizations of the training dataset (a) and estimated with n_ learned realizations for _ using MCMC without PLoM (b), for _ using PLoM with RODB (c), and for _ using PLoM with ROTB(n_Δ t) (d).

(iii)  In Fig. <ref>, the clouds of n_ points corresponding to n_ learned realizations can be seen for components 2, 5, 6 (a,b,c) and components 3, 6, 8 (d,e,f). These are shown for _ using MCMC without PLoM (a,d), for _ using PLoM with RODB (b,e), and for _ using PLoM with ROTB(n_Δ t) (c,f).

(iv) Figure <ref> plots the functions that characterize the reduced-order transient basis ROTB() as a function of time :




  * The eigenvalues of matrix [K_] and those of the of symmetrized matrix [K̃()] are shown in
Fig. <ref>.


  * The probability-measure concentration using the d^ 2/ν-criterion is shown in Fig. <ref>.
For the learning without PLoM, the d^ 2-concentration is 0.951, which shows that the concentration is lost, and for the PLoM with the RODM, the concentration is 0.0091, which shows that the concentration is preserved.


  * The other criterion of the probability-measure concentration is given by Kullback measure, shown in Fig. <ref>.
For the learning without PLoM, Kullbach is 0.4594, and for the PLoM with the RODM, Kullback is 3.4473. Comparing Figs. <ref> and <ref> shows, similarly to Application 1 that the two criteria are consistent and give the same analysis of the concentration.


  * The angle between the subspaces spanned by RODB and ROTB() is displayed in Fig. <ref>. It can be seen that, for the optimal time 5 Δ t, the angle is 17.1^∘, which is significant, although less than the optimal angle of Application 1. This shows that the two bases are different while the d^ 2-concentration remains small at 0.0153.


  * The entropy of pdf p_(·  ; ) is given in Fig. <ref>.


  * The normalized mutual information (MI) of the pdfs p_ and p_(·  ; ) is shown in Fig. <ref>. This figure shows that the optimal value of n is n_ = 5. Unlike Application 1, the normalized mutual information presents a local minimum, which is also a global minimum over the admissible set _N.
    For the non-normalized estimation of the mutual information, we have Î() = 3.4994, Î(_  ; n_Δ t) = 5.8763, and Î(_) = 5.9382. For the normalized one, we have
    Î_() = Î_(_  ; n_Δ t) = 0.3441, and Î_(_) = 0.3477.



(v) As for Application 1, examination of these figures shows that traditional learning without PLoM gives poor results compared to PLoM, which allows the concentration to be preserved and properly learns the geometry of the probability measure support. We also see that PLoM with the optimal ROTB provides an improvement in learning compared to PLoM with RODM. However, this improvement is less than in the case of Application 1 for which the data are much more heterogeneous (in correlation with the geometric complexity of the probability-measure support). Nevertheless, PLoM with the optimal ROTB is an improvement over PLoM with RODB and consequently, should improve the estimates of conditional statistics thanks to better learning of the joint probability measure.


























 §.§ Results for Application 3




(i) Figure <ref> displays the graphs of the probability density function (pdf)  of components 1, 6, 25, and 40 for  estimated with the n_d realizations of the training dataset, and the pdf estimated with n_ learned realizations, for _ using MCMC without PLoM (a,d,g,j), for _ using  PLoM with RODB (b, e, h, k), and for _ using PLoM with ROTB(n_Δ t) (c, f, i, l).

(ii)  Figure <ref> shows the joint probability density function of components 25 and 40 of  estimated with the n_d realizations of the training dataset (a) and estimated with n_ learned realizations for _ using MCMC without PLoM (b), for _ using PLoM with RODB (c), and for _ using PLoM with ROTB(n_Δ t) (d).

(iii)  In Fig. <ref>, the clouds of n_ points corresponding to n_ learned realizations can be seen for components 1, 6, 12 (a,b,c) and components 12, 25, 40 (d,e,f). These are shown for _ using MCMC without PLoM (a,d), for _ using PLoM with RODB (b,e), and for _ using PLoM with ROTB(n_Δ t) (c,f).

(iv) Figure <ref> plots the functions that characterize the reduced-order transient basis ROTB() as a function of time :




  * The eigenvalues of matrix [K_] and those of the of symmetrized matrix [K̃()] are shown in
Fig. <ref>.


  * The probability-measure concentration using the d^ 2/ν-criterion is shown in Fig. <ref>.
For the learning without PLoM, the d^ 2-concentration is 0.574, which shows that the concentration is lost, and for the PLoM with the RODM, the concentration is 0.067, which shows that the concentration is preserved.


  * The other criterion of the probability-measure concentration is given by Kullback measure, shown in Fig. <ref>.
For the learning without PLoM, Kullbach is 14.35, and for the PLoM with the RODM, Kullback is 5.72. Comparing Figs. <ref> and <ref> shows, similarly to Applications 1 and  2 that the two criteria are consistent and give the same analysis of the concentration.


  * The angle between the subspaces spanned by RODB and ROTB() is displayed in Fig. <ref>. It can be seen that, for the optimal time 9 Δ t, the angle is 9.7^∘, which is significant, although less than the optimal angle of Applications 1 and  2. This shows that the two bases are different while the d^  2-concentration remains small at 0.078.


  * The entropy of pdf p_(·  ; ) is given in Fig. <ref>.


  * The normalized mutual information (MI) of the pdfs p_ and p_(·  ; ) is shown in Fig. <ref>. This figure shows that the optimal value of n is n_ = 9. The behavior of the normalized mutual information is similar to that of Application 1 and does not present a local minimum as in Application 2.
    For the non-normalized estimation of the mutual information, we have Î() = 24.167, Î(_  ; n_Δ t) = 25.115, and Î(_) = 25.673. For the normalized one, we have
    Î_() = Î_(_  ; n_Δ t) = 0.1418 and Î_(_) = 0.1450.



(v) As for Applications 1 and 2, examination of these figures shows that traditional learning without PLoM gives poor results compared to PLoM, which allows the concentration to be preserved and properly learns the geometry of the probability measure support. We also see that PLoM with the optimal ROTB provides an improvement in learning compared to PLoM with RODM. However, this improvement is less than in the case of Applications 1 and 2. For this application, relative  to a relatively high dimension of  ν = 45, the data are more homogeneous than for the other applications (in correlation with the geometric complexity of the probability-measure support). Nevertheless, PLoM with the optimal
ROTB is an improvement over PLoM with RODB and, consequently, should improve the estimates of conditional statistics thanks to better learning of the joint probability measure.





















§ CONCLUSION




In this paper, we have presented the theoretical elements of constructing a time-dependent anisotropic kernel, which allows us to create a data projection basis for PLoM. This basis serves as an alternative to the DMAPS basis built using a time-independent isotropic kernel used by PLoM. We have demonstrated that an optimal time can be determined to obtain an optimal transient basis, best respecting the statistical dependence between the components for the learned joint probability measure.

The proposed theory has been  developed to improve PLoM in cases of highly heterogeneous data. The improvement of the learned joint probability measure is quantified by estimating an objective criterion from information theory, namely the mutual information, which we have normalized relative to the number of realizations using entropy.

This theory is consistent in the sense that, for a time close to the initial time, the DMAPS basis constructed with the time-independent isotropic kernel coincides with the transient basis constructed with the time-dependent anisotropic kernel. Thus, we can characterize the difference between the two bases by the angle of the vector subspaces they generate.

The theory is illustrated through three applications with decreasing levels of data heterogeneity. The three applications confirm that PLoM with the DMAPS basis (time-independent isotropic kernel) always results in learning that preserves the concentration of the measure, unlike the classic MCMC approach. The applications show that it is possible to improve the learned joint probability measure with the transient anisotropic kernel, which a priori allows for better estimates of conditional statistics.



§ ACKNOWLEDGMENTS


The authors acknowledge partial funding from DOE SciDAC FASTMath  Institute, and an ONR MURI on Modeling Turbulence and Chemistry in High Speed Reactive Flows.








§ OVERVIEW OF THE PROBABILISTIC LEARNING ON MANIFOLDS (PLOM) ALGORITHM AND ITS PARAMETERIZATION



The PLoM approach <cit.>, which has specifically been developed for small
data (as opposed to big data) starts from a training dataset _d made up of a relatively small number
n_d of points. It is assumed that _d is generated with an underlying stochastic manifold related
to a ^n_x-valued random variable = (,), defined on a probability space (Θ,,),
in which  is the quantity of interest that is a ^n_q-random variable, where  is the
control parameter that is a ^n_w-random variable, and where n_x = n_q + n_w. Another
^n_u- valued random variable  defined on (Θ,,) is also be considered,
which is an uncontrolled parameter and/or a noise.  Random variable  is assumed to be
written as = (,) in which the measurable mapping  is not explicitly known.
The joint probability distribution P_,(d,d) of   and  is assumed to be given.
The non-Gaussian probability measure P_() = P_,(d,d) of =(,)
is concentrated in a region of ^n_x for which the only available information is the cloud of
the points of training dataset _d. The PLoM method makes it possible to generate the learned dataset
_ for  whose n_≫ n_d points (learned realizations) are generated by the
non-Gaussian probability measure that is estimated using the training dataset. The concentration of
the probability measure is preserved thanks to the use of a diffusion-maps basis that allows to
enrich the available information from the training dataset.

The training dataset _d is made up of the n_d independent realizations
_d^j= (_d^j,_d^j) in ^n_x = ^n_q×^n_w for j∈{1,… , n_d}
of random variable =(,). The PLoM method allows for generating the learned dataset
_ made up of n_≫ n_d  learned realizations {_^ℓ,ℓ = 1, … ,n_}
of random vector . As soon as the learned dataset has been constructed, the learned realizations
for  and  can be extracted as (_^ℓ,_^ℓ) = _^ℓ for ℓ = 1, … ,n_.

Using the learned dataset _, PLoM allows for carrying out any conditional statistics such as ↦ E{()|=} from _w in ^n_ξ, in which  is a given measurable mapping from ^n_q into ^n_ξ, that is to say to construct statistical surrogate models (metamodels) in a probabilistic framework.



 §.§ Reduced representation



The n_d independent realizations {_d^j , j=1,… , n_d} are represented by the matrix [x_d]= [_d^1 …_d^n_d] in _n_x,n_d. Let []= [^1,… ,^n_d] be the random
matrix with values in _n_x,n_d, whose columns are n_d independent copies of random vector . Using the PCA of , random matrix [] is written as,


    [] = [x] + [φ]  [ζ]^1/2  []  ,


in which [] = [^1,… , ^n_d] is a _ν,n_d-valued random matrix, where ν≤ n_x, and where [ζ] is the (ν×ν) diagonal matrix of the ν positive eigenvalues of the empirical estimate of the covariance matrix of . The (n_x×ν) matrix [φ] is made up of the associated eigenvectors such [φ]^T [φ]= [I_ν]. The matrix [x] in _n_x,n_d has identical columns, each one being equal to the empirical estimate ∈^n_x of the mean value of random vector .
The columns of [] are n_d independent copies of a random vector  with values in ^ν.
The realization [η_d] = [^1…^n_d] ∈_ν,n_d of []  is computed by [η_d] =  [ζ]^-1/2 [φ]^T  ([x_d] - [x]).

The value ν is classically calculated in order that the L^2- error function ν↦_(ν) defined by
 

    _(ν) = 1 - ∑_α=1^νζ_α/E{‖‖^2}  ,


be smaller than ε_. If ν < n_x, then there is a statistical reduction.




 §.§ Construction of a reduced-order diffusion-maps basis (RODB) and reduced-order transient basis (ROTB())



 In this section, we begin with the construction of the RODB that is the basis initially used in the PLoM algorithm (see <cit.>).
Concerning the construction of the ROTB(), we refer the read to <ref>-(ii).



(i) Construction of RODB.

This construction corresponds to the one initailly proposed in the PLoM algorithm.
For preserving the concentration of the learned realizations in the region in which the points of the training dataset are concentrated, the PLoM relies on the diffusion-maps method <cit.>. This is an algebraic basis of vector space ^n_d, which  is constructed using the diffusion maps.
Let [_]  and  [B] be the matrices such that, for all i and j in {1,… , n_d},
[_]_ij = exp{-(4 ε_)^-1‖^i-^j‖^2} and
[B]_ij = δ_ij exp{-(4 ε_)^-1‖^i-^j‖^2}, in which ε_ >0 is a smoothing parameter. The eigenvalues b_,1,…,b_,n_d and the associated eigenvectors _^1,…,_^n_d of the right-eigenvalue problem [B]^-1[_] _^β = b_,β _^β are such that 1=b_,1 > b_,2≥…≥ b_,n_d and are computed by solving the eigenvalue problem [B]^-1/2[_] [B]^-1/2 ^β = b_,β ^β with the normalization ⟨^β,^β'⟩ =δ_ββ', and _^β = [B]^-1/2^β. The eigenvector _^1 associated with b_,1=1 is a constant vector.

The diffusion-maps basis {_^1,…,_^α,…, _^n_d} is a vector basis of ^n_d.
For a given integer m < n_d, the reduced-order diffusion-maps basis of order m is defined as the family
{_^1,…,_^m}.

This basis depends on two parameters, ε_ and m, which have to be identified.

As explained in <cit.>, the optimal value m_ of m_ is chosen as m_ = ν +1, and the optimal value ε_ of ε_ is such that


    1 = b_,1 > b_,2≃…≃ b_,m_≫ b_,m_ + 1≥…≥ b_,n_d > 0  ,


with the jump amplitude J_ = b_,m_ + 1  /b_,m_, which is J_=0.1 (following <cit.>), but which can also be chosen in the interval [0.1  , 0.5].
Consequently, the RODB is defined for m=m_ and is represented by the matrix


    [g_] = [_^1 …_^m_]∈_n_d,m_  .




(ii) Construction of ROTB().
Because PLoM will also use the reduced-order transient basis to quantify its efficiency relative to the reduced-order DMAPS basis, we introduce this basis  in this Appendix.
For n fixed in {1,… , N}, the reduced-order transient basis, ROTB(), is represented by the matrix


    [g()] = [^1() …^m_()]∈_n_d,m_  ,


in which m_ is the optimal value identified in <ref>-(i), and where [g()] is constructed in Section <ref>-(ii) (see Eq. (<ref>).



(iii) Reduced-order basis for PLoM.
In this Appendix, the PLoM reduced-order basis will be represented by the matrix [g_]∈_n_d,m_. Depending on the context of its use, this matrix will either be [g_], representing the reduced-order DMAPS basis (RODB) as used in the initial construction of the PLoM, or [g()] for a fixed n, representing the reduced-order transient basis (ROTB()) as proposed in this paper. The latter is introduced with the goal of comparing the efficiency of the two reduced-order vector bases.



 §.§ Reduced-order representation of the random matrices



The reduced-order basis represented by matrix [g_]∈_n_d,m_  spans a subspace of ^n_d that characterizes, for the optimal values m_ and ε_, the local geometry structure of  dataset {^j, j=1,… , n_d}. So the PLoM method introduces the _ν,n_d-valued random matrix [_] = []  [g_]^T with m_ < n_d, corresponding to a data-reduction representation of random matrix [], in which  [] is a _ν,m_-valued random matrix. The MCMC generator of random matrix [] belongs to the class of Hamiltonian Monte Carlo methods, is explicitly described in <cit.>, and is mathematically detailed in Theorem 6.3 of <cit.>. This  generator allows for computing n_ realizations {[_^ℓ],ℓ=1,… , n_} of  [] and therefore, for deducing the n_ realizations {[_^ℓ],ℓ=1,… ,n_} of [_]. The reshaping of matrix [_^ℓ] ∈_ν,n_d allows for obtaining n_ = n_× n_d learned realizations
{_^ℓ',ℓ' =1,… ,n_} of _. These learned realizations allow for estimating converged statistics on _ and then on _ = x + [φ]  [ζ]^1/2 _, such as pdf, moments, or conditional expectation of the type E{()  |  = } for  given in ^n_w and for any given vector-valued function  defined on ^n_q.




 §.§ Criterion for quantifying the concentration of the probability measure of random matrix [_]



The concentration of the probability measure of random matrix [_] is defined (see <cit.>) by


    d^ 2(m_) = E{‖ [_] - [η_d]‖^2} /‖ [η_d]‖^2  .


Let ={m_,m_+1,… , n_d} in which m_ is the optimal value of m.
Theorem 7.8 of <cit.> shows that
min_m∈ d^ 2(m)  ≤ 1 + m_/(n_d-1) < d^2(n_d), which means that the PLoM method, for m=m_ and [g_] is a better method than the usual one corresponding to d^2(n_d)= 1+n_d/(n_d-1)≃ 2.
Using the n_ realizations {[_^ℓ],ℓ=1,… ,n_} of [_], we have the estimate,


    d̂^ 2(m_) = (1/n_)∑_ℓ=1^n_{‖ [_^ℓ]  - [η_d]‖^2} /‖ [η_d]‖^2   .





 §.§ Generation of learned realizations {^ℓ'_, ℓ'=1,…, n_} of random vector _



Let  { ([ (t)], [(t)]), t∈^+ } be  the unique asymptotic (for t→ +∞) stationary diffusion stochastic process with values in _ν,m_×_ν,m_, of the following reduced-order ISDE (stochastic nonlinear second-order dissipative Hamiltonian dynamic system), for t >0,


    d[(t)]      =  [(t)]   dt   , 
    
      d[(t)]     =  [([(t)])]  dt -1/2 f_0 [(t)]  dt + √(f_0)  [d^(t)]   ,


with [(0)] = [η_d]  [a] and [(0)] = [ ]   [a], in which


    [a]  = [g_]  ([g_]^T  [g_])^-1∈_n_d,m_  .


(1) [([(t)])]= [L ( [(t)]   [g_]^T ) ]   [a] is a random matrix  with values in _ν,m_.
For all [u] = [^1 …^n_d] in _ν,n_d with ^j=(u^j_1,… ,u^j_ν) in ^ν, the
matrix [L([u])] in _ν,n_d is defined, for all k = 1,… ,ν and for all j=1,… , n_d, by


    [L([u])]_kj    = 1/p(^j) {∇_^j  p(^j) }_k   , 
                                                p(^j)         = 1/n_d∑_j'=1^n_dexp{ -1/2 ŝ^  2‖ŝ/s^j'-^j‖^2 }  , 
    ∇_^j  p(^j)     = 1/ŝ^ 2  n_d∑_j'=1^n_d (ŝ/s^j' -^j)
                   exp{ -1/2 ŝ^ 2‖ŝ/s^j' -^j‖^2 }  ,


in which ŝ  is the modified Silverman bandwidth  s, which has been introduced in <cit.>,


    ŝ =   s/√(s^2 +n_d -1/n_d)   ,   
    s = {4/n_d(2+ν)}^1/(ν +4)  .



(2) [^(t)] = [^(t)]   [a] where {[^(t)], t∈^+} is the _ν,n_d-valued normalized Wiener process.

(3)  [ ] is the _ν,n_d-valued normalized Gaussian random matrix that is independent of process [^].

(4)  The free parameter f_0, such that 0 < f_0 < 4/ŝ, allows the dissipation term of the nonlinear second-order dynamic system (dissipative Hamiltonian system)  to be controlled in order to kill the transient part induced by the initial conditions. A common value is f_0=4 (note that ŝ< 1).

(5)  We then have [] = lim_t→ +∞[(t)] in probability distribution. The Störmer-Verlet scheme is used <cit.> for solving the reduced-order ISDE, which allows for generating the learned realizations, [z_^1], … , [z_^n_], and then generating the learned realizations [η_^1],… , [η_^n_] such that [η_^ℓ] = [z_^ℓ]  [g_]^T. The implementation of the Störmer-Verlet scheme is detailed, for instance, in the Appendix of <cit.> for parallel computation, introducing the following parameters: the integration time step Δ t_, the initial time t_i =0, and the final integration time t_f = M_0×Δ t_, at which the stationary solution is reached.

(6) The learned realizations {_^ℓ',ℓ' =1,…,n_} of random vector  are then calculated (see Eq. (<ref>)) by _^ℓ' = + [φ]  [μ]^1/2 _^ℓ'.




 §.§ Constraints on the second-order moments of the components of _



In general, the mean value of _  estimated using the n_ learned realizations
{_^ℓ',ℓ'=1,…,n_}, is sufficiently close to zero.
Likewise, the estimate of the covariance matrix of _, which must be the identity matrix, is
sufficiently close to a diagonal matrix. However, sometimes  the
diagonal entries of the estimated covariance matrix can be lower than 1.
Normalization can be recovered by imposing constraints


    E{( H_,k )^2 } = 1    ,    k=1,…, ν  ,


in the algorithm presented in <ref>.
For that, we use the method and the iterative algorithm presented in <cit.> (that is based on Sections 5.5 and 5.6 of <cit.>). The constraints are imposed by using the Kullback-Leibler minimum cross-entropy principle. The resulting optimization problem is formulated using a Lagrange multiplier = (λ_1,…,λ_ν) associated with the constraints. The optimal solution of the Lagrange multiplier is computed using an efficient iterative algorithm. At each iteration, the MCMC generator detailed in <ref> is used.

The  constraints are rewritten as


    E{(_)} =   ,


in which the function = (h_1,… ,h_ν) and the vector = (b_1,… , b_ν) are such that
h_k(_) = (H_,k)^2 and b_k =1 for k in {1,… , ν}.
To take into account the constraints in the algorithm presented in <ref>, Eq. (<ref>) is replaced by the following one,


    [L_([u])]_kj  = 1/p(^j) {∇_^j  p(^j) }_k  -2 λ_k u_k^j  .


It should be noted that Eqs. (<ref>) to (<ref>) can be straightforwardly extended to the case in which the constraint defined by
Eq. (<ref>) is replaced by the full second-order moment constraints  E{_} = _ν and E{_⊗_} = [I_ν], that is to say,


    E{ H_,k}  = 0   ,   E{( H_,k )^2}  = 1    ,   k=1,…, ν   ,   
     E{ H_,k H_,k'}  = 0     ,    1 ≤ k < k' ≤ν  .


The iteration algorithm for computing ^i+1 as a function of ^i is the following,


    ^i+1    = ^i -α_i [Γ”(^i)]^-1 '(^i)    ,    i ≥ 0   ,
    ^0             = _ν  ,


in which '(^i) =  - E{(_^i)} and
[Γ”(^i)] = [{(_^i)}] (the covariance matrix), and where α_i is a relaxation function (less than 1) that is introduced for controlling the convergence as a function of iteration number i.
For given i_2 ≥ 2, for given β_1 and β_2 such that 0 < β_1 < β_2 ≤ 1, α_i can be defined by:

- for  i ≤ i_2,  α_i =  β_1 +(β_2-β_1)(i-1)/(i_2-1);

- for i > i_2, α_i = β_2.

The convergence of the iteration algorithm is controlled by the error function i↦(i)  defined by


    (i) =  ‖ - E{(_^i)}‖ / ‖‖  .


At each iteration i, E{(_^i)} and [{(_^i)}] are estimated by using the n_ learned realizations of _^i obtained by reshaping the learned realizations. If i_last is the last iteration corresponding to convergence, we have _ =  _^i with i = i_last.








§ ESTIMATION OF THE KULLBACK-LEIBLER DIVERGENCE, THE MUTUAL INFORMATION, AND THE ENTROPY FROM A SET OF REALIZATIONS



The definition Kullback-Leibler divergence, the mutual information, and the entropy can be found in
<cit.>. The estimation of these quantities from a set of independent realizations is carried out using the Gaussian kernel density estimation (GKDE) method <cit.>.

For ν > 1, let =(X_1,… ,X_ν) and =(Y_1,… ,Y_ν) be ^ν-valued random variables defined on the probability space (Θ,,), whose probability measures are P_(d) = p_()  d and P_(d) = p_()  d,
in which  the probability density functions p_ and p_ are assumed to be strictly positive. Let {^ℓ,ℓ=1,… , N_x} be N_x independent realizations of  and let {^j,j=1,… , N_y} be N_y independent realizations of .
For k=1,… , ν, let σ_X_k and σ_Y_k be the standard deviation of X_k and Y_k that are estimated (empirical estimator) with the independent realizations.
Finally, we introduce the Silverman bandwidth for the Gaussian KDE estimation of p_ and p_,


    s_x = {4/N_x(2+ν)}^1/(ν +4)   ,    s_y = {4/N_y(2+ν)}^1/(ν +4)  .





 §.§ Estimation of the Kullback-Leibler divergence from a set of realizations



The Kullback-Leibler divergence (or the relative entropy) between p_ and p_ is defined by


    D(p_‖ p_) = ∫_^ν p_() log ( p_()/p_())   d =
    E{log( p_()/p_() ) }  .


The GKDE, D̂(p_‖ p_), of D(p_‖ p_) yields the formula,


    D̂(p_ ‖   p_) =    νlog (s_y/s_x ) + log (N_y/N_x )
                + log (σ_Y_1×…×σ_Y_ν/σ_X_1×…×σ_X_ν ) 
        + 1/N_x∑_ℓ'=1^N_xlog{∑_ℓ=1^N_xexp ( -1/2s_x^2∑_k=1^ν ( x_k^ℓ' - x_k^ℓ/σ_X_k )^2 )/∑_j=1^N_yexp ( -1/2s_y^2∑_k=1^ν ( x_k^ℓ' - y_k^j/σ_Y_k )^2 )}  .





 §.§ Estimation of the mutual information from a set of realizations



The mutual information I() of  allows to quantify the level of statistical dependencies of the components X_1,…, X_ν of =(X_1,… ,X_ν). Let p_X_k be the pdf of real-valued random variable X_k,


    p_X_k(x_k) = ∫_^ν-1 p_ ( x_1,… ,x_k-1,x_k,x_k+1,…,x_ν )  
    dx_1 … dx_k-1  dx_k+1… dx_ν  .


The mutual information I() is defined by


    I() = D(p_ ‖ ⊗_k=1^ν p_X_k) = E{log( p_()/p_X_1(X_1)×…× p_X_ν(X_ν) ) }  .


Eq. (<ref>) shows that, if the components X_1,… ,X_ν are statistically independent, then I()=0.
The GKDE, Î() of I() yields the formula,


    Î()  = 1/N_x∑_ℓ'=1^N_xlog{1/N_x∑_ℓ=1^N_xexp ( -1/2s_x^2∑_k=1^ν ( x_k^ℓ' - x_k^ℓ/σ_X_k )^2 )/∏_k=1^ν ( 1/N_x∑_ℓ”=1^N_xexp ( -1/2s_x^2 ( x_k^ℓ' - x_k^ℓ”/σ_X_k)^2  )  )}  .





 §.§ Estimation of the entropy from a set of realizations



The entropy related to p_ is defined by


    S_ = -∫_^ν p_() log p_()   d =
    - E{log p_()  }  .


The GKDE, Ŝ_, of S_ yields the formula,


    Ŝ_ =    νlog (s_x√(2π)  ) +
                 log (σ_X_1×…×σ_X_ν ) 
        - 1/N_x∑_ℓ'=1^N_xlog{1/N_x∑_ℓ=1^N_xexp ( -1/2s_x^2∑_k=1^ν ( x_k^ℓ' - x_k^ℓ/σ_X_k )^2 )
                                          }  .


Since J_S(N_x) = νlog (s_x√(2π)  ) is asymptotically for N_x→ +∞ in -log(N_x), the entropy decreases when N_x increases. 







§ CONFLICT OF INTEREST


The author declares that he has no conflict of interest.



10
url<#>1urlprefixURL href#1#2#2 #1#1

Soize2016
C. Soize, R. Ghanem, Data-driven probability concentration and sampling on
  manifold, Journal of Computational Physics 321 (2016) 242–258.
https://doi.org/10.1016/j.jcp.2016.05.044
  doi:10.1016/j.jcp.2016.05.044.

Coifman2005
R. Coifman, S. Lafon, A. Lee, M. Maggioni, B. Nadler, F. Warner, S. Zucker,
  Geometric diffusions as a tool for harmonic analysis and structure definition
  of data: Diffusion maps, PNAS 102 (21) (2005) 7426–7431.
https://doi.org/10.1073/pnas.0500334102
  doi:10.1073/pnas.0500334102.

Coifman2006
R. Coifman, S. Lafon, Diffusion maps, Applied and Computational Harmonic
  Analysis 21 (1) (2006) 5–30.
https://doi.org/10.1016/j.acha.2006.04.006
  doi:10.1016/j.acha.2006.04.006.

Korb2010
K. B. Korb, A. E. Nicholson, Bayesian artificial intelligence, CRC press,
  Boca Raton, 2010.

Murphy2012
K. P. Murphy, Machine Learning: A Probabilistic Perspective, MIT press, 2012.

Ghahramani2015
Z. Ghahramani, Probabilistic machine learning and artificial intelligence,
  Nature 521 (7553) (2015) 452–459.
https://doi.org/10.1038/nature14541
  doi:10.1038/nature14541.

Russell2016
S. Russel, P. Norvig, Artifical Intelligence, A Modern Approach, Third Edition,
  Pearson, Harlow, 2016.

Vapnik2000
V. Vapnik, The Nature of Statistical Learning Theory, Springer, New York, 2000.
https://doi.org/10.1007/978-1-4757-3264-1
  doi:10.1007/978-1-4757-3264-1.

Hastie2009
T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning,
  Second Edition, Springer, 2009.
https://doi.org/10.1007/b94608 doi:10.1007/b94608.

James2013
G. James, D. Witten, T. Hastie, R. Tibshirani, An Introduction to Statistical
  Learning, Vol. 112, Springer, 2013.

Taylor2015
J. Taylor, R. J. Tibshirani, Statistical learning and selective inference,
  Proceedings of the National Academy of Sciences 112 (25) (2015) 7629–7634.
https://doi.org/10.1073/pnas.1507583112
  doi:10.1073/pnas.1507583112.

Swischuk2019
R. Swischuk, L. Mainini, B. Peherstorfer, K. Willcox, Projection-based model
  reduction: Formulations for physics-based machine learning, Computers &
  Fluids 179 (2019) 704–717.
https://doi.org/10.1016/j.compfluid.2018.07.021
  doi:10.1016/j.compfluid.2018.07.021.

Oztireli2010
A. C. Öztireli, M. Alexa, M. Gross, Spectral sampling of manifolds, ACM
  Transactions on Graphics (TOG) 29 (6) (2010) 1–8.
https://doi.org/10.1145/1882261.1866190
  doi:10.1145/1882261.1866190.

Perrin2017
G. Perrin, C. Soize, S. Marque-Pucheu, J. Garnier, Nested polynomial trends for
  the improvement of Gaussian process-based predictors, Journal of
  Computational Physics 346 (2017) 389–402.
https://doi.org/10.1016/j.jcp.2017.05.051
  doi:10.1016/j.jcp.2017.05.051.

Kevrekidis2020
Y. Kevrekidis, Manifold learning for parameter reduction, Bulletin of the
  American Physical Society 65 (2020).
https://doi.org/10.1016/j.jcp.2019.04.015
  doi:10.1016/j.jcp.2019.04.015.

Kontolati2022
K. Kontolati, D. Loukrezis, K. R. dos Santos, D. G. Giovanis, M. D. Shields,
  Manifold learning-based polynomial chaos expansions for high-dimensional
  surrogate models, International Journal for Uncertainty Quantification 12 (4)
  (2022).
https://doi.org/10.1615/Int.J.UncertaintyQuantification.2022039936
  doi:10.1615/Int.J.UncertaintyQuantification.2022039936.

Pan2020
S. Pan, K. Duraisamy, Physics-informed probabilistic learning of linear
  embeddings of nonlinear dynamics with guaranteed stability, SIAM Journal on
  Applied Dynamical Systems 19 (1) (2020) 480–509.
https://doi.org/10.1137/19M1267246
  doi:10.1137/19M1267246.

Soize2020a
C. Soize, R. Ghanem, Physics-constrained non-Gaussian probabilistic learning
  on manifolds, International Journal for Numerical Methods in Engineering
  121 (1) (2020) 110–145.
https://doi.org/10.1002/nme.6202
  doi:10.1002/nme.6202.

Soize2021a
C. Soize, R. Ghanem, Probabilistic learning on manifolds constrained by
  nonlinear partial differential equations for small datasets, Computer Methods
  in Applied Mechanics and Engineering 380 (2021) 113777.
https://doi.org/10.1016/j.cma.2021.113777
  doi:10.1016/j.cma.2021.113777.

Talwalkar2008
A. Talwalkar, S. Kumar, H. Rowley, Large-scale manifold learning, in: 2008 IEEE
  Conference on Computer Vision and Pattern Recognition, IEEE, 2008, pp. 1–8.
https://doi.org/10.1109/CVPR.2008.4587670
  doi:10.1109/CVPR.2008.4587670.

Marzouk2016
Y. Marzouk, T. Moselhy, M. Parno, A. Spantini, Sampling via measure transport:
  An introduction, Handbook of uncertainty quantification (2016) 1–41https://doi.org/10.1007/978-3-319-11259-6_23-1
  doi:10.1007/978-3-319-11259-6_23-1.

Parno2018
M. D. Parno, Y. M. Marzouk, Transport map accelerated markov chain Monte
  Carlo, SIAM/ASA Journal on Uncertainty Quantification 6 (2) (2018)
  645–682.
https://doi.org/10.1137/17M1134640
  doi:10.1137/17M1134640.

Perrin2018a
G. Perrin, C. Soize, N. Ouhbi, Data-driven kernel representations for sampling
  with an unknown block dependence structure under correlation constraints,
  Computational Statistics & Data Analysis 119 (2018) 139–154.
https://doi.org/10.1016/j.csda.2017.10.005
  doi:10.1016/j.csda.2017.10.005.

Soize2017a
C. Soize, R. Ghanem, Polynomial chaos representation of databases on manifolds,
  Journal of Computational Physics 335 (2017) 201–221.
https://doi.org/10.1016/j.jcp.2017.01.031
  doi:10.1016/j.jcp.2017.01.031.

Soize2019b
C. Soize, R. Ghanem, C. Safta, X. Huan, Z. P. Vane, J. C. Oefelein, G. Lacaze,
  H. N. Najm, Q. Tang, X. Chen, Entropy-based closure for probabilistic
  learning on manifolds, Journal of Computational Physics 388 (2019) 528–533.
https://doi.org/10.1016/j.jcp.2018.12.029
  doi:10.1016/j.jcp.2018.12.029.

Soize2020b
C. Soize, R. Ghanem, C. Desceliers, Sampling of Bayesian posteriors with a
  non-Gaussian probabilistic learning on manifolds from a small dataset,
  Statistics and Computing 30 (5) (2020) 1433–1457.
https://doi.org/10.1007/s11222-020-09954-6
  doi:10.1007/s11222-020-09954-6.

Soize2020c
C. Soize, R. Ghanem, Probabilistic learning on manifolds, Foundations of Data
  Science 2 (3) (2020) 279–307.
https://doi.org/10.3934/fods.2020013
  doi:10.3934/fods.2020013.

Soize2022b
C. Soize, Probabilistic learning inference of boundary value problem with
  uncertainties based on Kullback-Leibler divergence under implicit
  constraints, Computer Methods in Applied Mechanics and Engineering 395 (2022)
  115078.
https://doi.org/10.1016/j.cma.2022.115078
  doi:10.1016/j.cma.2022.115078.

Soize2022a
C. Soize, R. Ghanem, Probabilistic learning on manifolds (PLoM) with
  partition, International Journal for Numerical Methods in Engineering 123 (1)
  (2022) 268–290.
https://doi.org/10.1002/nme.6856
  doi:10.1002/nme.6856.

Soize2022c
C. Soize, Probabilistic learning constrained by realizations using a weak
  formulation of fourier transform of probability measures, Computational
  Statistics 38 (4) (2023) 1879–1925.
https://doi.org/10.1007/s00180-022-01300-w
  doi:10.1007/s00180-022-01300-w.

Kennedy2001
M. C. Kennedy, A. O'Hagan, Bayesian calibration of computer models, Journal
  of the Royal Statistical Society: Series B (Statistical Methodology) 63 (3)
  (2001) 425–464.
https://doi.org/10.1111/1467-9868.00294
  doi:10.1111/1467-9868.00294.

Marzouk2007
Y. M. Marzouk, H. N. Najm, L. A. Rahn, Stochastic spectral methods for
  efficient Bayesian solution of inverse problems, Journal of Computational
  Physics 224 (2) (2007) 560–586.
https://doi.org/10.1016/j.jcp.2006.10.010
  doi:10.1016/j.jcp.2006.10.010.

Gentle2009
J. E. Gentle, Computational statistics, Springer, New York, 2009.
https://doi.org/10.1007/978-0-387-98144-4
  doi:10.1007/978-0-387-98144-4.

Stuart2010
A. M. Stuart, Inverse problems: a Bayesian perspective, Acta Numerica 19
  (2010) 451–559.
https://doi.org/10.1017/S0962492910000061
  doi:10.1017/S0962492910000061.

Owhadi2015
H. Owhadi, C. Scovel, T. Sullivan, On the brittleness of Bayesian inference,
  SIAM Review 57 (4) (2015) 566–582.
https://doi.org/10.1137/130938633
  doi:10.1137/130938633.

Matthies2016
H. G. Matthies, E. Zander, B. V. Rosić, A. Litvinenko, O. Pajonk, Inverse
  problems in a Bayesian setting, in: Computational Methods for Solids and
  Fluids, Vol. 41, Springer, 2016, pp. 245–286.
https://doi.org/10.1007/978-3-319-27996-1_10
  doi:10.1007/978-3-319-27996-1_10.

Dashti2017
M. Dashti, A. M. Stuart, The Bayesian approach to inverse problems, in:
  R. Ghanem, D. Higdon, O. Houman (Eds.), Handbook of Uncertainty
  Quantification, Springer, Cham, Switzerland, 2017, Ch. 10, pp. 311–428.
https://doi.org/10.1007/978-3-319-12385-1_7
  doi:10.1007/978-3-319-12385-1_7.

Ghanem2017
R. Ghanem, D. Higdon, H. Owhadi, Handbook of Uncertainty Quantification, Vol. 1
  to 3, Springer, Cham, Switzerland, 2017.
https://doi.org/10.1007/978-3-319-12385-1
  doi:10.1007/978-3-319-12385-1.

Spantini2017
A. Spantini, T. Cui, K. Willcox, L. Tenorio, Y. Marzouk, Goal-oriented optimal
  approximations of Bayesian linear inverse problems, SIAM Journal on
  Scientific Computing 39 (5) (2017) S167–S196.
https://doi.org/10.1137/16M1082123
  doi:10.1137/16M1082123.

Perrin2020
G. Perrin, C. Soize, Adaptive method for indirect identification of the
  statistical properties of random fields in a Bayesian framework,
  Computational Statistics 35 (1) (2020) 111–133.
https://doi.org/10.1007/s00180-019-00936-5
  doi:10.1007/s00180-019-00936-5.

Ghanem2018a
R. Ghanem, C. Soize, Probabilistic nonconvex constrained optimization with
  fixed number of function evaluations, International Journal for Numerical
  Methods in Engineering 113 (4) (2018) 719–741.
https://doi.org/10.1002/nme.5632
  doi:10.1002/nme.5632.

Soize2018a
C. Soize, Design optimization under uncertainties of a mesoscale implant in
  biological tissues using a probabilistic learning algorithm, Computational
  Mechanics 62 (3) (2018) 477–497.
https://doi.org/10.1007/s00466-017-1509-x
  doi:10.1007/s00466-017-1509-x.

Ghanem2018b
R. Ghanem, C. Soize, C. Thimmisetty, Optimal well-placement using probabilistic
  learning, Data-Enabled Discovery and Applications 2 (1) (2018) 1–16.
https://doi.org/10.1007/s41688-017-0014-x
  doi:10.1007/s41688-017-0014-x.

Farhat2019
C. Farhat, R. Tezaur, T. Chapman, P. Avery, C. Soize, Feasible probabilistic
  learning method for model-form uncertainty quantification in vibration
  analysis, AIAA Journal 57 (11) (2019) 4978–4991.
https://doi.org/10.2514/1.J057797
  doi:10.2514/1.J057797.

Ghanem2019
R. Ghanem, C. Soize, C. Safta, X. Huan, G. Lacaze, J. C. Oefelein, H. N. Najm,
  Design optimization of a scramjet under uncertainty using probabilistic
  learning on manifolds, Journal of Computational Physics 399 (2019) 108930.
https://doi.org/10.1016/j.jcp.2019.108930
  doi:10.1016/j.jcp.2019.108930.

Almeida2022
J. O. Almeida, F. A. Rochinha, A probabilistic learning approach applied to the
  optimization of wake steering in wind farms, Journal of Computing and
  Information Science in Engineering 23 (1) (2022) 011003.
https://doi.org/10.1115/1.4054501
  doi:10.1115/1.4054501.

Capiez2022
E. Capiez-Lernout, C. Soize, Nonlinear stochastic dynamics of detuned bladed
  disks with uncertain mistuning and detuning optimization using a
  probabilistic machine learning tool, International Journal of Non-Linear
  Mechanics 143 (2022) 104023.
https://doi.org/10.1016/j.ijnonlinmec.2022.104023
  doi:10.1016/j.ijnonlinmec.2022.104023.

Almeida2023
J. O. Almeida, F. A. Rochinha, A probabilistic learning approach applied to the
  optimization of wake steering in wind farms, Journal of Computing and
  Information Science in Engineering 23 (1) (2023) 011003.
https://doi.org/10.1115/1.4054501
  doi:10.1115/1.4054501.

Guilleminot2020
J. Guilleminot, J. E. Dolbow, Data-driven enhancement of fracture paths in
  random composites, Mechanics Research Communications 103 (2020) 103443.
https://doi.org/10.1016/j.mechrescom.2019.103443
  doi:10.1016/j.mechrescom.2019.103443.

Chen2024
P. Chen, J. Guilleminot, C. Soize, Concurrent multiscale simulations of
  nonlinear random materials using probabilistic learning, Computer Methods in
  Applied Mechanics and Engineering 422 (2024) 116837.
https://doi.org/10.1016/j.cma.2024.116837
  doi:10.1016/j.cma.2024.116837.

Soize2023c
C. Soize, An overview on uncertainty quantification and probabilistic learning
  on manifolds in multiscale mechanics of materials, Mathematics and Mechanics
  of Complex Systems 11 (1) (2023) 87–174.
https://doi.org/10.2140/memocs.2023.11.87
  doi:10.2140/memocs.2023.11.87.

Ghanem2022
R. Ghanem, C. Soize, L. Mehrez, V. Aitharaju, Probabilistic learning and
  updating of a digital twin for composite material systems, International
  Journal for Numerical Methods in Engineering 123 (13) (2022) 3004–3020.
https://doi.org/10.1002/nme.6430
  doi:10.1002/nme.6430.

Ezvan2023
O. Ezvan, C. Soize, C. Desceliers, R. Ghanem, Updating an uncertain and
  expensive computational model in structural dynamics based on one single
  target frf using a probabilistic learning tool, Computational Mechanics 71
  (2023) 1161–1177.
https://doi.org/10.1007/s00466-023-02301-2
  doi:10.1007/s00466-023-02301-2.

Capiez2024
E. Capiez-Lernout, C. Ezvan, Olivier Soize, Updating nonlinear stochastic
  dynamics of an uncertain nozzle model using probabilistic learning with
  partial observability and incomplete dataset, ASME Journal of Computing and
  Information Science in Engineering 24 (6) (2024) 061006,1–17.
https://doi.org/10.1115/1.4065312
  doi:10.1115/1.4065312.

Arnst2021
M. Arnst, C. Soize, K. Bulthies, Computation of sobol indices in global
  sensitivity analysis from small data sets by probabilistic learning on
  manifolds, International Journal for Uncertainty Quantification 11 (2) (2021)
  1–23.
https://doi.org/10.1615/Int.J.UncertaintyQuantification.2020032674
  doi:10.1615/Int.J.UncertaintyQuantification.2020032674.

Soize2021e
C. Soize, A. Orcesi, Machine learning for detecting structural changes from
  dynamic monitoring using the probabilistic learning on manifolds, Structure
  and Infrastructure Engineering Journal 17 (10) (2021) 1418–1430.
https://doi.org/10.1080/15732479.2020.1811991
  doi:10.1080/15732479.2020.1811991.

Zhong2023
K. Zhong, J. G. Navarro, S. Govindjee, G. G. Deierlein, Surrogate modeling of
  structural seismic response using Probabilistic Learning on Manifolds,
  Earthquake Engineering and Structural Dynamics 52 (8) (2023) 2407–2428.
https://doi.org/10.1002/eqe.3839
  doi:10.1002/eqe.3839.

Soize2023a
C. Soize, R. Ghanem, Probabilistic-learning-based stochastic surrogate model
  from small incomplete datasets, Computer Methods in Applied Mechanics and
  Engineering (2023).

Soize2024
C. Soize, R. Ghanem, Probabilistic-learning-based stochastic surrogate model
  from small incomplete datasets for nonlinear dynamical systems, Computer
  Methods in Applied Mechanics and Engineering 418 (2023) 116498.
https://doi.org/10.1016/j.cma.2023.116498
  doi:10.1016/j.cma.2023.116498.

Soize2023b
C. Soize, Q.-D. To, Polynomial-chaos-based conditional statistics for
  probabilistic learning with heterogeneous data applied to atomic collisions
  of helium on graphite substrate, Journal of Computational Physics (2023).

Sinha2023
A. Sinha, C. Soize, C. Desceliers, G. Cunha, Aeroacoustic liner impedance
  metamodel from simulation and experimental data using probabilistic learning,
  AIAA Journal 61 (11) (2023) 4926–4934.
https://doi.org/10.2514/1.J062991
  doi:10.2514/1.J062991.

Soize2015
C. Soize, Polynomial chaos expansion of a multimodal random vector, SIAM-ASA
  Journal on Uncertainty Quantification 3 (1) (2015) 34–60.
https://doi.org/10.1137/140968495
  doi:10.1137/140968495.

Bowman1997
A. Bowman, A. Azzalini, Applied Smoothing Techniques for Data Analysis: The
  Kernel Approach With S-Plus Illustrations, Vol. 18, Oxford University Press,
  Oxford: Clarendon Press, New York, 1997.
https://doi.org/10.1007/s001800000033
  doi:10.1007/s001800000033.

Givens2013
G. Givens, J. Hoeting, Computational Statistics, 2nd Edition, John Wiley and
  Sons, Hoboken, New Jersey, 2013.

Doob1953
J. L. Doob, Stochastic processes, John Wiley & Sons, New York, 1953.

Guikhman1980
I. I. Guikhman, A. Skorokhod, Introduction à la Théorie des Processus
  Aléatoires, Edition Mir, 1980.

Friedman2006
A. Friedman, Stochastic Differential Equations and Applications, Dover
  Publications, Inc., Mineola, New York, 2006.

Soize1994
C. Soize, The Fokker-Planck Equation for Stochastic Dynamical Systems and
  its Explicit Steady State Solutions, Vol. Series on Advances in Mathematics
  for Applied Sciences: Vol 17, World Scientific, Singapore, 1994.
https://doi.org/10.1142/2347 doi:10.1142/2347.

Gardiner1985
C. W. Gardiner, Handbook of Stochastic Methods, Second Edition, Springer
  Verlag, Berlin, Heidelberg, 1985.

Risken1989
H. Risken, The Fokker-Planck Equation, Second Edition, Springer Verlag,
  Berlin, Heidelberg, 1989.

Guelfand1967
I. M. Gelfand, N. I. Vilenkin, Les Distributions. Tome 4. Application de
  l'Analyse Harmonique, Dunod, 1967.
https://doi.org/10.1016/0375-9474(67)90547-7
  doi:10.1016/0375-9474(67)90547-7.

Spencer1993
B. Spencer, L. Bergman, On the numerical solution of the Fokker-Planck
  equation for nonlinear stochastic systems, Nonlinear Dynamics 4 (1993)
  357–372.
https://doi.org/10.1007/BF00120671
  doi:10.1007/BF00120671.

Masud2005
A. Masud, L. A. Bergman, Application of multi-scale finite element methods to
  the solution of the Fokker-Planck equation, Computer methods in applied
  mechanics and engineering 194 (12-16) (2005) 1513–1526.
https://doi.org/10.1016/j.cma.2004.06.041
  doi:10.1016/j.cma.2004.06.041.

Kumar2006
P. Kumar, S. Narayanan, Solution of Fokker-Planck equation by finite
  element and finite difference methods for nonlinear systems, Sadhana 31
  (2006) 445–461.
https://doi.org/10.1007/BF02716786
  doi:10.1007/BF02716786.

Pichler2013
L. Pichler, A. Masud, L. A. Bergman, Numerical solution of the
  Fokker-Planck equation by finite difference and finite element methods -
  a comparative study, Computational Methods in Stochastic Dynamics: Volume 2
  (2013) 69–85https://doi.org/10.1007/978-94-007-5134-7_5
  doi:10.1007/978-94-007-5134-7_5.

Deng2009
W. Deng, Finite element method for the space and time fractional
  Fokker-Planck equation, SIAM journal on numerical analysis 47 (1) (2009)
  204–226.
https://doi.org/10.1137/080714130
  doi:10.1137/080714130.

Anderson2024
W. Anderson, M. Farazmand, Fisher information and shape-morphing modes for
  solving the Fokker-Planck equation in higher dimensions, Applied
  Mathematics and Computation 467 (2024) 128489.
https://doi.org/10.1016/j.amc.2023.128489
  doi:10.1016/j.amc.2023.128489.

Popelier2011
P. L. Popelier, Solving the Schrödinger Equation: Has Everything Been
  Tried?, Imperial College Press, 2011.
https://doi.org/10.1142/p780 doi:10.1142/p780.

Feit1982
M. Feit, J. Fleck Jr, A. Steiger, Solution of the Schrödinger equation by
  a spectral method, Journal of Computational Physics 47 (3) (1982) 412–433.
https://doi.org/10.1016/0021-9991(82)90091-2
  doi:10.1016/0021-9991(82)90091-2.

Iitaka1994
T. Iitaka, Solving the time-dependent Schrödinger equation numerically,
  Physical Review E 49 (5) (1994) 4684.
https://doi.org/10.1103/PhysRevE.49.4684
  doi:10.1103/PhysRevE.49.4684.

Simos1997
T. Simos, P. Williams, A finite-difference method for the numerical solution of
  the Schrödinger equation, Journal of Computational and Applied
  Mathematics 79 (2) (1997) 189–205.
https://doi.org/10.1016/S0377-0427(96)00156-2
  doi:10.1016/S0377-0427(96)00156-2.

Serfling1980
R. J. Serfling, Approximation theorems of mathematical statistics, Vol. 162,
  John Wiley & Sons, 1980.

Soize2017b
C. Soize, Uncertainty Quantification. An Accelerated Course with Advanced
  Applications in Computational Engineering, Springer, New York, 2017.
https://doi.org/10.1007/978-3-319-54339-0
  doi:10.1007/978-3-319-54339-0.

Ikeda1981
N. Ikeda, S. Watanabe, Stochastic Differential Equations and Diffusion
  Processes, North-Holland, Amsterdam, 1981.

Khasminskii1980
R. Has'minski, Stochastic Stability of Differential Equations, Sijthoff &
  Noordhoff, Alphen aan den Rijn, The Netherlands, 1980, first English edition
  Khasminskii,2021, Spinger.

Stroock1979
D. W. Stroock, S. S. Varadhan, Multidimensional Diffusion Processes, Vol. 233,
  Springer-Verlag, Berlin,Heidelberg, 1997.

Kloeden1992
P. Kloeden, E. Platen, Numerical Solution of Stochastic Differentials
  Equations, Springer-Verlag, Heidelberg, 1992.

Duong2008
T. Duong, A. Cowling, I. Koch, M. Wand, Feature significance for multivariate
  kernel density estimation, Computational Statistics & Data Analysis 52 (9)
  (2008) 4225–4242.
https://doi.org/10.1016/j.csda.2008.02.035
  doi:10.1016/j.csda.2008.02.035.

Hansen1975
E. R. Hansen, A Table of Series and Products, Prentice-Hall, New York, 1975.

Bhattacharyya1943
A. Bhattacharyya, On the measures of divergence between two statistical
  populations defined by their probability distributions, Bulletin of the
  Calcultta Mathematical Society 35 (1943) 99–109.

Kullback1951
S. Kullback, R. A. Leibler, On information and sufficiency, The Annals of
  Mathematical Statistics 22 (1) (1951) 79–86.
https://doi.org/10.1214/aoms/1177729694
  doi:10.1214/aoms/1177729694.

Kolmogorov1956
A. Kolmogorov, On the shannon theory of information transmission in the case of
  continuous signals, IRE Transactions on Information Theory 2 (4) (1956)
  102–108.

Cover2006
T. M. Cover, J. A. Thomas, Elements of Information Theory, Second Edition, John
  Wiley & Sons, Hoboken, 2006.

Kapur1992
J. N. Kapur, H. K. Kesavan, Entropy Optimization Principles with Applications,
  Academic Press, San Diego, 1992.

Gray2011
R. M. Gray, Entropy and Information Theory, 2nd Edition, Springer, New York,
  2011.
https://doi.org/10.1007/978-1-4419-7970-4
  doi:10.1007/978-1-4419-7970-4.

Shannon1948
C. E. Shannon, A mathematical theory of communication, Bell system technical
  journal 27 (3) (1948) 379–423 & 623–659.
https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
  doi:10.1002/j.1538-7305.1948.tb01338.x.

Lafon2006
S. Lafon, A. B. Lee, Diffusion maps and coarse-graining: A unified framework
  for dimensionality reduction, graph partitioning, and data set
  parameterization, IEEE transactions on pattern analysis and machine
  intelligence 28 (9) (2006) 1393–1403.
https://doi.org/10.1109/TPAMI.2006.184
  doi:10.1109/TPAMI.2006.184.




