Paper Title: A Sampling-based Progressive Hedging Algorithm for Stochastic Programming.



Learning Ordinality in Semantic Segmentation
    Rafael Cristino^1 Ricardo P. M. Cruz^1,2* Jaime S. Cardoso^1,2*
^1 Faculty of Engineering, University of Porto
^2 INESC TEC, Porto
 rpcruz@fe.up.pt, jaime.cardoso@fe.up.pt
    
===============================================================================================================================================================================



§ ABSTRACT 
The progressive hedging algorithm (PHA) is a cornerstone among algorithms for large-scale stochastic programming problems. However, its traditional implementation is hindered by some limitations, including the requirement to solve all scenario subproblems in each iteration, reliance on an explicit probability distribution, and a convergence process that is highly sensitive to the choice of certain penalty parameters. This paper introduces a sampling-based PHA which aims to overcome these limitations. Our approach employs a dynamic selection process for the number of scenario subproblems solved per iteration. It incorporates adaptive sequential sampling for determining sample sizes, a stochastic conjugate subgradient method for direction finding, and a line-search technique to update the dual variables. Experimental results demonstrate that this novel algorithm not only addresses the bottlenecks of the conventional PHA but also potentially surpasses its scalability, representing a substantial improvement in the field of stochastic programming.






§ INTRODUCTION




 §.§ Problem setup

In this paper, we focus on a task of solving a Stochastic Programming (SP) stated as follows.

    min_x ∈ X ⊆ℝ^n f(x) = E_ω̃[h(x,ω̃)],


where ω̃∈ℝ^m denotes a random vector and h(·,ω̃) is a finite valued convex function in x for any particular realization ω̃= ω. Except in very simplistic situations (e.g., when ω̃ has a small number of outcomes), solving the optimization problem  (<ref>) using deterministic algorithms  is infeasible.   To overcome challenges arising from efforts to solve realistic instances, the focus often shifts towards solving a sample average approximation (SAA) problem. In such an  approach, we create an approximation by using |S| i.i.d. realizations of ω̃ called scenarios, and denote them by ω^1,...,ω^|S|. In this setting, (<ref>) can be reformulated as 


    min_x ∈ X f(x) = 1/|S|∑_s=1^|S| h(x, ω^s),







    


In order to establish an optimal solution of (<ref>), many algorithms have been proposed and one of the more popular algorithms among them is the progressive hedging (PH) algorithm <cit.>. To lay the groundwork, we will commence by reviewing the classic PH algorithm.



 §.§ Classic Progressive Hedging Algorithm

The PH Algorithm is a version of alternating direction method of multipliers (ADMM) <cit.> which has attracted significant attention in stochastic programming community, extensively utilized for addressing scenario-based uncertainty in large-scale optimization problems. Originally conceptualized by Rockafellar and Wets in 1991 <cit.>, the algorithm has evolved significantly, finding applications in diverse fields such as finance, energy, and logistics <cit.>. The convergence properties of the PH algorithm have been rigorously analyzed since its inception. Rockafellar and Wets provided the initial proof of convergence for convex problems. Further research, such as that by Ruszczyński and Shapiro <cit.>, explored the conditions under which PH algorithm converges, including scenarios involving non-convexities and other complexities. However, despite its wide-spread popularity, classic PH algorithm remains a deterministic algorithm which immediately hinders its applicability for broader stochastic optimization problems. For instance, in applications where the uncertainty can only be sampled via simulation or the sample space is so vast that it is unrealistic to expect a knowledge of all possible outcomes associated with the data.  More specifically, the limitations include 




















    
  * First, in contrast to several SP algorithms, notably stochastic gradient descent (SGD), classic PH algorithm necessitates the resolution of all subproblems during each iteration. This requirement constitutes a significant computational bottleneck, substantially impeding its scalability and performance.

    
  * Second, the convergence criteria of the PH algorithm are predicated upon the accessibility and knowledge of the probability distribution of the random variables involved. This requirement severely restricts the algorithm's utility in situations where such a distribution remains unknown or is difficult to ascertain, thereby restricting its range of applications.

    
  * Lastly, the algorithm employs a dual update rule which is fundamentally gradient-based and is accurate solely at the current dual point. Also, this approach renders the convergence process highly dependent on the appropriate selection of the penalty parameter which acts as a step size in the dual update. The performance of the algorithm, therefore, hinges on the  specification of the subgradient and the penalty parameter, further complicating its practical performance.


For instances which present such challenges, an adaptive sampling algorithm maybe more suitable because such methods allow a possibility of creating approximations which are able to sequentially adapt to the data which is revealed during the course of the algorithm. In this paper, we will introduce an innovative sampling-based PH algorithm which adaptively samples a subset of scenario subproblems in each iteration, facilitating dynamic scenario updates to enhance computational efficiency. 





 §.§ Adaptive sampling

Classic PH algorithm relies on the knowledge of the probability distribution of the random variables. When the probability distribution is not available, the PH algorithm typically employs sample average approximation (SAA) <cit.> which has a fixed sample size to estimate these uncertainties. However, it is difficult to determine the sample size <cit.> when applying such method to the PH algorithm and convergence properties have only been established for cases
with only a finite number of outcomes. While  adaptive sampling <cit.> has emerged as a powerful approach for instances with a large number of outcomes (especially for multi-dimensional uncertainty), such adaptive sampling has remained elsuive in the context of PH. Adaptive sampling dynamically adjusts the sample size based on ongoing optimization progress, providing a more resource-efficient and precise solution. Some effort has been made to combine adaptive sampling with PH algorithm. For instance, <cit.> provides randomized PH methods which is able to produce new iterates as soon as a single scenario subproblem is solved. <cit.> proposes a SBPHA approach which is a combination of replication and PH algorithm. However, these methods, although provide a "randomized" PH algorithm, still sample a subset of scenarios from a fixed finite collection in each iteration, without dynamically determining the sample size based on the optimization procedure. On the other hand, our sampling-based PH algorithm will strategically sample a subset of scenario subproblems in each iteration, facilitating dynamic scenario updates to enhance computational efficiency. We will also leverage the sample complexity analysis <cit.> to support the choice of the sample size, ensuring a theoretically grounded approach.



 §.§ Stochastic conjugate gradient method


The dual update of classic PH algorithm can be considered as a stochastic first-order (SFO) method <cit.>. First-order gradient-based methods dominate the field for several convincing reasons: low computational cost, simplicity of implementation, and strong empirical results. However SFO methods are not known for their numerical accuracy especially when the functions are non-smooth and the classic convergence rate <cit.> only applies under the assumption that the component functions are smooth.  

In contrast to first-order methods, algorithms based on conjugate gradients (CG) provide finite termination when the objective function is a linear-quadratic deterministic function <cit.>. Furthermore, CG algorithms are also used for large-scale nonsmooth convex optimization problems <cit.>. However, despite its many strengths, (e.g., fast per-iteration
convergence, frequent explicit regularization on step-size, and better parallelization than first-order
methods), the use of conjugate gradient methods for SP is uncommon. Although <cit.> proposed some new stochastic conjugate gradient (SCG) algorithms, their assumptions do not support non-smooth objective functions, and hence not directly applicable to our problems.

In order to solve stochastic optimization problems when functions are non-smooth (as in (<ref>)), it is straightforward to imagine that a stochastic subgradient method could be used. In this paper, we treat the dual optimization problem in PH as a stochastic linear quadratic optimization problem and propose to leverage a stochastic conjugate subgradient (SCS) direction originated from <cit.> which accommodates both the curvature of a quadratic function, as well as the decomposition of subgradients by data points, as is common in stochastic programming <cit.>. This combination enhances the power of stochastic first-order approaches without the additional burden of second-order approximations. In other words, our method shares the spirit of Hessian Free (HF) optimization, which has attracted some attention in the machine learning community <cit.>. As our theoretical analysis and the computational results reveal, the new method provides a “sweet-spot” at the intersection of speed, accuracy, and scalability of algorithms (see Theorems <ref> and <ref>).



 §.§ Penalty parameter and line-search algorithm


In the original Progressive Hedging (PH) algorithm, Rockafellar and Wets briefly discussed the effect of the penalty parameter on the performance of the PH algorithm and the quality of the solution, but they did not provide any guidance on how to choose it. Various choices have since been proposed, often differing with the application under consideration. Readers interested in the selection of the penalty parameter can refer to <cit.>, which offers a comprehensive literature review of related methods.

Our approach differs from these methods by employing a stochastic direction-finding technique <cit.> derived from Wolfe's line-search method <cit.>. Compared with existing approaches, our method is more mathematically motivated and adheres to the principles of optimization <cit.>.




§ SAMPLING-BASED PROGRESSIVE HEDGING ALGORITHM


In this section, we focus on solving (<ref>) using the a sampling-based PHA in which we combine classic PHA<cit.> with adaptive sampling in stochastic programming <cit.>. The main ingredients of the algorithm include six important components:


  * Sequential scenarios sampling. At iteration k, different from classic PH Algorithm which solves all scenarios subproblems, we will sample |S_k| scenarios and only solve the subproblelms corresponding to these scenarios. It is similar as using sample average approximation (SAA) to approximate function f using f_k,  

    f_k(x)= 1/|S_k|∑_s=1^|S_k| h(x, ω^s),

where |S_k| will be determined based on concentration inequalities in Theorem <ref>.  


  * Update the primal variables. In each scenarios s ∈ S_k, similar to the classic PHA, we will first solve the corresponding augmented Lagrangian problem 


    x_k^s = _x^s ∈ X h(x^s, ω^s) + ⟨λ_k^s, x^s - x̅_k-1⟩ + ρ/2 ||x^s - x_k-1||^2.


Then a feasible solution will be 

    x̅_k = 1/|S_k|∑_s=1^|S_k|  x_k^s.


















  * Conjugate subgradient direction finding. The idea here is inspired by Wolfe's non-smooth conjugate subgradient method which uses the smallest norm of the convex combination of the previous search direction d_k-1^s and the current subgradient g_k^s = x_k^s - x̅_k. More specifically, we first solve the following one-dimensional QP


    γ_k^s = _γ^s ∈ [0,1]1/2||γ^s  d_k-1^s + (1-γ^s) g_k^s||^2.



Then we can set the new direction 

    d_k^s=γ_k^s d_k-1^s +(1-γ_k^s)g_k^s := Nr(G_k^s),



where G_k^s={g_k^s,d_k-1^s}.


  * Update the dual variables using line-search. We will update λ̂_k+1^s = λ_k^s + θ_k^s d_k^s. Define 

    L_k^s(λ_k^s + θ^s d_k^s) :=  h(x̂_k^s, ω^s) + ⟨λ_k^s+θ^s d_k^s, x̂_k^s - x̂_k⟩  + ρ/2 ||x̂_k^s - x̂_k||^2,


where δ_k^s is the stochastic searching region, x̂_k^s and x̂_k are obtained by replacing λ_k^s in (<ref>) with λ_k^s + θ d_k^s in each scenario and resolving (<ref>) and (<ref>), respectively. The step size θ_k^s will be obtained by performing an inexact line-search which makes it satisfy the strong-Wolfe condition <cit.>. Let g^s(θ^s) ∈∂_λ^s L_k^s(λ_k^s + θ^s · d_k^s) and define the intervals L and R.

    L={θ^s >0  |  L_k^s(λ_k + θ^s · d_k)-L_k^s(λ_k) ≥ m_1 ||d_k^s||^2 θ^s },
        R ={θ^s >0  |  ∑_s=1^S_k⟨ g^s(θ^s),d_k^s ⟩≤ m_2  ||d_k^s||^2 },

where m_2 < m_1 < 1/2. The output of the step size will satisfy two metrics: (i) Identify a set L which includes points which sufficiently increase the dual objective function approximation L_k^s, and (ii) Identify a set R for which the directional derivative estimate for L_k^s is sufficiently decreased. The algorithm seeks points which belong to L ∩ R. The combination of L and R is called strong-Wolfe condition and it has been shown in <cit.> (Lemma 1) that L ∩ R is not empty.

Intuitively, the above procedure is a line-search along conjugate direction d_k^s and the goal is to (approximately) maximize the dual objective function L_k^s.


  * Stochastic searching region update. Similar to the standard stochastic trust region method <cit.>, let γ>1, η∈ (0,1), λ_k := (λ_k^1,...,λ_k^|S_k|) and L_k(λ_k):= ∑_s=1^|S_k| L_k^s(λ_k^s), if 


    L_k(λ̂_k)-L_k(λ_k-1) > η [L_k-1(λ̂_k)-L_k-1(λ_k-1)],


set δ_k = γδ_k-1 and λ_k = λ̂_k. Otherwise, set δ_k =  δ_k-1/γ and λ_k = λ_k-1.


  * Termination criteria. Define d_k := (d_k^1,...,d_k^|S_k|) and ||d_k|| := 1/|S_k|∑_s=1^|S_k| ||d_k^s||. The algorithm concludes its process when ||d_k||<ε and δ_k < δ_min. As we will show in Theorem <ref>, a dimunitive value of ||d_k|| indicates a small norm of the gradient g_k, fulfilling the optimality condition for an unconstrained convex optimization problem. Additionally, a threshold δ_min is established to prevent premature termination in the initial iterations. Without this threshold, the algorithm may halt prematurely if the initial samples are correctly classified by the initial classifier α_0, resulting in ||d_k|| < ε. However, if δ_min is introduced, based on Theorem <ref>, |S_k| should be chosen such that 

    |S_k| ≥ -8log(ε/2)·M_1^2/κ^2δ_k^4,
   

where κ is the κ-approximation defined in Definition <ref>. This effectively mitigates any early stopping concern with high probability.







§ CONVERGENCE PROPERTIES




 §.§ Scenario Dual Objective Functions


For any ω^s, x̅ and λ^s, define the scenario optimal solution 


    x^s(λ^s, ω^s) = _x ∈ X h(x,ω^s) + ⟨λ^s, x - x̅⟩,


and the corresponding scenario dual objective function 


    L^s(λ^s ; w^s, x̅) = h(x^s,ω^s) + ⟨λ^s, x^s - x̅⟩.



 
    In any iteration of Algorithm <ref>, we have ∑_s=1^|S_k|λ_k^s = 0.



    We will show the lemma by induction. When k = 0, λ_0^s = 0, for all s ∈ S_0. Thus, the statement is correct. Next, when k = 1, we have
    
    λ_1^s = ρ (x_1^s - x̅_k) = ρ (x_1^s - 1/|S_1|∑_s = 1^|S_1| x_1^s),

    then 
    
    ∑_s=1^|S_1|λ_1^s = ρ∑_s=1^|S_1|(x_1^s - 1/|S_1|∑_s = 1^|S_1| x_1^s) = 0.


    Suppose it is true for k = n, i.e., ∑_s=1^|S_n|λ_n^s = 0, then for k = n + 1, 
    
    ∑_s=1^|S_n+1|λ_n+1^s = ∑_s=1^|S_n|λ_n^s + ρ∑_s=1^|S_n+1|(x_n+1^s - x̅_n+1) = 0,


    where the first part is 0 by the induction hypothesis and the second part is 0 by the construction of Algorithm <ref>. We thus conclude the proof.
    


 
    In every iteration of Algorithm <ref>, we have 
    
    1/|S_k|∑_s = 1^|S_k| L_k^s(λ_k^s; ω^s, x̅_k) = 1/|S_k|∑_s = 1^|S_k| L_k^s(λ_k^s; ω^s) := 1/|S_k|∑_s = 1^|S_k| h(x_k^s,ω^s) + ⟨λ_k^s, x_k^s ⟩.




    
    1/|S_k|∑_s = 1^|S_k| L^s(λ^s; ω^s, x̅_k) = 1/|S_k|∑_s = 1^|S_k| [h(x_k^s,ω^s) + ⟨λ_k^s, x_k^s ⟩] +  1/|S_k|∑_s = 1^|S_k|⟨λ_k^s, x̅_k ⟩,

    where the last part is 0 by Lemma <ref>.




 §.§ Stochastic Local Approximation
 

 
    The primal feasible region X is bounded, i.e., there exists a constant C such that for any x ∈ X, we have ||x|| ≤ C.

    
 
    For any ω^s and λ^s, L^s(λ^s; ω^s) is bounded, i.e., there exists a constant M, such that |L^s(λ^s; ω^s)| ≤ M.


 
    For any ω^s, L^s(λ^s; ω^s) possesses a Lipschitz constant 0 < L < ∞ and the sub-differentials are non-empty for every λ^s.


 
    For any ω^s and λ^s, L^s(λ^s; ω^s) has bounded sub-differential set, i.e., for any g^s ∈∂_λ^sL^s(λ^s; ω^s), we have ||g^s|| ≤ G.


   A multi-variable function L_k(λ^1,...,λ^|S|) is a local κ-approximation of L(λ^1,...,λ^|S|) on ℬ(λ_k^s,δ_k), if for all λ^s ∈ℬ(λ_k^s,δ_k),

    |L(λ^1,...,λ^|S|)-L_k(λ^1,...,λ^|S|)| ≤κδ_k^2.



 Let Δ_k, Λ_k^s denote the random counterpart of δ_k,  λ_k^s, respectively, and ℒ_k denote the σ-algebra generated by {ℒ_i}_i=0^k-1.  A sequence of random models {ℒ_k} is said to be μ_1-probabilistically κ-approximation of ℒ with respect to {ℬ(Λ_k^s,Δ_k)} if the event 

    I_k Δ={ℒ_k is κ- approximation of ℒ on ℬ(Λ_k^s,Δ_k)}


satisfies the condition 

    ℙ(I_k|ℒ_k-1) ≥μ_1 > 0.



 
    If one added scenario s_i ∈ S_k/S_k-1 changes from s_i to s_i', then for s ∈ S_k/s_i, we have 
    
    ||g_k^s - (g_k^s)'|| = O(1/|S_k|),   |γ_k^s - (γ_k^s)'| = O(1/|S_k|),  and  ||d_k^s - (d_k^s)'|| = O(1/|S_k|).




    First, note that  for s ∈ S_k/s_i,
    
    g_k^s = x_k^s - x̅_k = x_k^s - x_k^1 + ... + x_k^s + ... + x_k^|S_k|/|S_k|, 
        (g_k^s)' = x_k^s - x̅_k' = x_k^s - x_k^1 + ... + x_k^s' + ... + x_k^|S_k|/|S_k|.

    Thus, by Assumption <ref>, 
    
    ||g_k^s - (g_k^s)'|| = ||x_k^s-x_k^s'||/|S_k|≤2C/|S_k|.


    Also, note that 

    
    γ_k^s = _γ^s ∈ [0,1] ||γ^s  d_k-1^s + (1-γ^s) g_k^s||^2 = ⟨ d_k-1^s, d_k-1^s - g_k^s ⟩/||d_k-1^s - g_k^s||^2, 
        (γ_k^s)' = _γ^s ∈ [0,1] ||γ^s  d_k-1^s + (1-γ^s) (g_k^s)'||^2 = ⟨ d_k-1^s, d_k-1^s - (g_k^s)' ⟩/||d_k-1^s - (g_k^s)'||^2.

    
    Let Δ g_k^s =(g_k^s)' - g_k^s, we have 

    
    γ_k^s - (γ_k^s)'     = ⟨ d_k-1^s, d_k-1^s - g_k^s ⟩· ||d_k-1^s - g_k^s - Δ g_k^s||^2/||d_k-1^s - g_k^s||^2 · ||d_k-1^s - g_k^s - Δ g_k^s||^2
        - ⟨ d_k-1^s, d_k-1^s - g_k^s - Δ g_k^s ⟩· ||d_k-1^s - g_k^s||^2/||d_k-1^s - g_k^s||^2 · ||d_k-1^s - g_k^s - Δ g_k^s||^2
        = ⟨Δ g_k^s, d_k-1^s - g_k^s  ⟩·⟨ d_k-1^s, d_k-1^s - g_k^s ⟩/||d_k-1^s - g_k^s||^2 · ||d_k-1^s - g_k^s - Δ g_k^s||^2
        + ||Δ g_k^s||^2 ·⟨ d_k-1^s, d_k-1^s - g_k^s ⟩ - ⟨Δ g_k^s, d_k-1^s ⟩· ||d_k-1^s - g_k^s||^2/||d_k-1^s - g_k^s||^2 · ||d_k-1^s - g_k^s - Δ g_k^s||^2.


    With Assumption <ref>, Eq. (<ref>) and triangular inequality, we have 

    
    |γ_k^s - (γ_k^s)'| = O(1/|S_k|)


    Finally, to bound ||d_k^s - (d_k^s)'||, note that

    
    d_k^s = γ_k^s g_k^s + (1-γ_k^s) d_k-1^s, 
        (d_k^s)' = (γ_k^s)' (g_k^s)' + (1-(γ_k^s)') d_k-1^s.


    Let Δγ_k^s = (γ_k^s)' - γ_k^s, then

    
    d_k^s - (d_k^s)'     = γ_k^s g_k^s + (1-γ_k^s) d_k-1^s 
        - (γ_k^s + Δγ_k^s)(g_k^s + Δ g_k^s) - (1 - γ_k^s - Δγ_k^s) d_k-1^s 
        = -γ_k^s Δ g_k^s - Δγ_k^s g_k^s - Δγ_k^s Δ g_k^s + Δγ_k^s d_k-1^s.

With Assumption <ref>, Eq. (<ref>) and Eq. (<ref>), we have


    ||d_k^s - (d_k^s)'|| = O(1/|S_k|).



 
    If one added scenario s_i ∈ S_k/S_k-1 changes from s_i to s_i', then for s ∈ S_k/s_i, we have 
    
    ||x_k^s (θ^s) - (x_k^s)'(θ^s)|| = O(1/|S_k|),  and
        ||L_k^s (λ_k^s + θ_k^s d_k^s; ω^s) - L_k^s(λ_k^s + (θ_k^s)' (d_k^s)'; ω^s)|| = O(1/|S_k|).




    For any θ^s, we have

    
    x_k^s(θ^s) = _x^s ∈ X h(x^s, ω^s) + ⟨λ_k^s + θ^s d_k^s, x^s - x̅_k ⟩. 
        (x_k^s)'(θ^s) = _x^s ∈ X h(x^s, ω^s) + ⟨λ_k^s + θ^s (d_k^s+Δ d_k^s), x^s - x̅_k'⟩.


    Note that we have only changed the objective coefficient of x^s and the difference is θ^s ·Δ d_k^s. According to the sensitivity analysis in linear programming, ||Δ d_k^s|| = O(1/|S_k|) and θ^s ∈ [δ_min,δ_max], we have

    
    ||x_k^s(θ^s) - (x_k^s)'(θ^s)|| = O(θ^s · ||Δ d_k^s||) = O(1/|S_k|).


    Also, for any θ^s, consider

    
    g(θ^s; ω^s) :=  h(x_k^s(θ^s), ω^s) + ⟨λ_k^s + θ^s d_k^s, x_k^s(θ^s) - x̅_k ⟩. 
        g'(θ^s; ω^s) :=  h((x_k^s)'(θ^s), ω^s) + ⟨λ_k^s + θ^s (d_k^s)', (x_k^s)'(θ^s) - x̅_k' ⟩.


    Let Δ x_k^s (θ^s) = x_k^s(θ^s) - (x_k^s)'(θ^s) and Δx̅_k = x̅_k'-x̅_k, then we have
    
    g(θ^s; ω^s) - g'(θ^s; ω^s)     = h(x_k^s(θ^s), ω^s) + ⟨λ_k^s + θ^s d_k^s, x_k^s(θ^s) - x̅_k ⟩
        -(h((x_k^s)'(θ^s), ω^s) + ⟨λ_k^s + θ^s (d_k^s)', (x_k^s)'(θ^s) - x̅_k' ⟩)
        = h(x_k^s(θ^s), ω^s) - h(x_k^s(θ^s) + Δ x_k^s (θ^s); ω^s) 
        + ⟨λ_k^s + θ^s d_k^s, x_k^s(θ^s) - x̅_k ⟩
        - ⟨λ_k^s + θ^s (d_k^s + Δ d_k^s), x_k^s(θ^s) + Δ x_k^s(θ^s)  - x̅_k + Δx̅_k ⟩) 
        = h(x_k^s(θ^s), ω^s) - h(x_k^s(θ^s) + Δ x_k^s (θ^s); ω^s) 
        - ⟨λ_k^s, Δ x_k^s(θ^s) ⟩ + ⟨λ_k^s, Δx̅_k ⟩ - ⟨θ^s d_k^s, Δ x_k^s(θ^s) ⟩ + ⟨θ^s d_k^s,  Δx̅_k ⟩
        - ⟨θ^s Δ d_k^s, x_k^s(θ^s) + Δ x_k^s(θ^s)  - x̅_k + Δx̅_k ⟩

    With Assumption <ref>, Lemma <ref> and Eq. (<ref>), we have
    
    || g(θ^s; ω^s) - g'(θ^s; ω^s)|| = O(1/|S_k|).

    Since Eq. (<ref>) is true for any θ^s, we have
    
    || L_k^s (λ_k^s + θ_k^s d_k^s; ω^s) - L_k^s(λ_k^s + (θ_k^s)' (d_k^s)'; ω^s)||
    
                =     ||max_θ^s g(θ^s; ω^s) - max_θ^s g'(θ^s; ω^s)|| 
    ≤    max_θ^s || g(θ^s; ω^s) - g'(θ^s; ω^s)||
    
                =     O(1/|S_k|).



 
    Let L (λ^1,...,λ^|S_k|) := E_{ω^1,...,ω^s}  [1/|S_k|∑_s = 1^|S_k| L_k^s(λ^s; ω^s)]
    and 

    L_k (λ^1,...,λ^|S_k|) := 1/|S_k|∑_s = 1^|S_k| L_k^s(λ^s; ω^s). Let M_1 be a large constant, for any 0<ε<1, κ>4L/δ_min, and
        
    
    |S_k| ≥ -8log(ε/2)·M_1^2/κ^2δ_k^4,

    
    we have
    
    ℙ[|L (λ^1,...,λ^|S_k|) - L_k (λ^1,...,λ^|S_k|)| ≤κδ_k^2,  ∀λ^s ∈ℬ(λ_k^s,δ_k) | ℒ_k-1] ≥ 1-ε.




    For the points λ_k^s, we will first show that given ℒ_k-1, L_k has the bounded difference property and then use McDiarmid's inequality to prove the theorem. Since λ_k^s dependents on ω^1,ω^2,...,ω^|S_k|, the McDiarmid's inequality can not be applied to L_k(λ_k^1,...,λ_k^|S_k|) directly. However, we can express L_k as function of ω^1,ω^2,...,ω^|S_k| and show that L_k(ω^1,ω^2,...,ω^|S_k|) has the bounded difference property. Specifically, the bounded difference property with respect to L_k(ω^1,ω^2,...,ω^|S_k|) is written as


    |L_k(ω^1,ω^2,...,ω^s,...,ω^|S_k|) - L_k(ω^1,ω^2,...,ω^s',...,ω^|S_k|)| = O(1/|S_k|).


We will divide the analysis into 2 cases to bound the changes of L_k^s(λ_k^s;ω^s).


    
  * Case 1: For s ∈ S_k/s_i. Given ℒ_k-1, based on Lemma <ref>, we have
    
    |L_k^s(λ_k^s;ω^s) - L_k^s((λ_k^s)';ω^s)| = O(1/|S_k|).

    
  * Case 2: For s = s_i. Given ℒ_k-1, based on Assumption <ref>, we have
    
    |L_k^s(λ_k^s;ω^s) - L_k^s((λ_k^s)';ω^s')|≤ 2M.



Thus,
    
    |L_k(ω^1,ω^2,...,ω^s,...,ω^|S_k|) - L_k(ω^1,ω^2,...,ω^s',...,ω^|S_k|)| 
    ≤    ∑_s ≠ s_i|L_k^s(λ_k^s;ω^s) - L_k^s((λ_k^s)';ω^s)|/|S_k| + |L_k^s(λ_k^s;ω^s) - L_k^s((λ_k^s)';ω^s')|/|S_k|
    
                =      O(1/|S_k|).

Let the bounded difference be specificed by M_1/|S_k|. Apply McDiarmid's inequality,

    ℙ(|L (λ_k^1,...,λ_k^|S_k|) - L_k (λ_k^1,...,λ_k^|S_k|)| ≤1/2κδ_k^2)
            ≥ 1-2exp(-κ^2δ_k^4 |S_k|^2/8 · |S_k| · M_1^2) ≥ 1-ε,


which indicates that when 

   |S_k| ≥-8log(ε/2)·M_1^2/κ^2δ_k^4, 
 
we have 

    ℙ(|L (λ_k^1,...,λ_k^|S_k|) - L_k (λ_k^1,...,λ_k^|S_k|)| ≤1/2κδ_k^2) ≥ 1-ε.




For any other λ^s ∈ℬ(λ_k^s,δ_k), if |L (λ_k^1,...,λ_k^|S_k|) - L_k (λ_k^1,...,λ_k^|S_k|)| ≤1/2κδ_k^2, then


    |L (λ^1,...,λ^|S_k|) - L_k (λ^1,...,λ^|S_k|)| 
       ≤ |L (λ^1,...,λ^|S_k|) - L(λ_k^1,...,λ_k^|S_k|)| 
        + |L (λ_k^1,...,λ_k^|S_k|) - L_k (λ_k^1,...,λ_k^|S_k|)  | + |L_k (λ_k^1,...,λ_k^|S_k|) - L_k (λ^1,...,λ^|S_k|)| 
       ≤ 2L ·δ_k + 1/2κδ_k^2
       ≤κδ_k^2,


where the second last inequality is due to the Assumption <ref> that L_k and L_k^s are Lipschitz continuous and the last inequality is because κ>4L/δ_min>4L/δ_k. Thus, we conclude that if |S_k| satisfies Equation (<ref>), then 


    ℙ[|L (λ^1,...,λ^|S_k|) - L_k (λ^1,...,λ^|S_k|)| ≤κδ_k^2,  ∀λ^s ∈ℬ(λ_k^s,δ_k)] ≥ 1-ε.







 §.§ Sufficient Increase and Submartingale Property


 The value estimates L_k := L_k(λ_k) and L_k+1 = L_k(λ_k + θ_k ⊙ d_k) are ε_F-accurate estimates of L(λ_k) and L(λ_k + θ_k ⊙ d_k), respectively, for a given δ_k if and only if 

    |L_k - L(λ_k)| ≤ε_F δ_k^2,    |L_k+1-L(λ_k + θ_k ⊙ d_k)| ≤ε_F δ_k^2.



  A sequence of model estimates {L_k, L_k+1} is said to be μ_2-probabilistically ε_F-accurate with respect to {Λ_k,Δ_k,S_k} if the event 

    J_k Δ={F̂_k,F̂_k+1 are ε_F-accurate estimates  for Δ_k }

satisfies the condition 

    ℙ(J_k|ℒ_k-1) ≥μ_2 >0.



 
Suppose that L_k is a κ-approximation of L on ℬ(λ_k^s ,δ_k). If 

    δ_k ≤m_1/4nκ ||d_k||_1   and   C_1=m_1/2n,


then there exist suitable step sizes θ_k^s ∈ L and θ_k^s ||d_k^s|| > δ_k/n such that 

    L(λ_k + θ_k ⊙ d_k)-L(λ_k) ≥ C_1 ||d_k||_1 δ_k.




Since θ_k^s ∈ L and θ_k^s ||d_k^s|| > δ_k/n, for each L_k^s, 

    L_k^s(λ_k^s + θ_k^s d_k^s)-L_k^s(λ_k^s) ≥ m_1||d_k^s||^2 θ_k^s ≥m_1/n||d_k^s|| δ_k.

Thus,

    L_k(λ_k + θ_k ⊙ d_k) -  L_k(λ_k)     = ∑_s=1^|S_k|[L_k^s(λ_k^s + θ_k^s d_k^s)-L_k^s(λ_k^s)] 
       ≥m_1/n||d_k||_1 δ_k

Also, since L_k is κ-approximation of L on ℬ(λ_k^s,δ_k), we have 

    L(λ_k + θ_k ⊙ d_k)-L(λ_k) 
                 = L(λ_k + θ_k ⊙ d_k)-L_k(λ_k + θ_k ⊙ d_k) 
        + L_k(λ_k + θ_k ⊙ d_k) -L_k(λ_k)+L_k(λ_k)-L(λ_k)
       ≥ -2κδ_k^2+m_1/n||d_k||_1 δ_k 
       ≥ C_1||d_k||δ_k,

which concludes the proof.


  Suppose that L_k is κ-approximation of L on the ball ℬ(λ_k^s,δ_k) the estimates (L_k,L_k+1) are ε_F-accurate with ε_F ≤κ. If δ_k ≤ 1 and 

    δ_k ≤min{(1-η_1)C_1 ||d_k||/2κ, ||d_k||/η_2},


then a suitable step sizes θ_k^s ∈ L and θ_k^s ||d_k^s|| > δ_k/n makes the k-th iteration successful.



    Since θ_k^s ∈ L and θ_k^s ||d_k^s|| > δ_k/n, from Lemma <ref>,
    
    L_k(λ_k + θ_k ⊙ d_k)-L_k(λ_k) ≥ 2 C_1 ||d_k||_1 δ_k

    
    Also, L_k is κ-approximation of L on ℬ(λ_k^s,δ_k) and the estimates (L_k,L_k+1) are ε_F-accurate with ε_F ≤κ implies that 
    
    
    ρ_k     = L_k-L_k-1/L_k+1-L_k
        = L_k-L(λ_k + θ_k ⊙ d_k)/L_k+1-L_k+L(λ_k + θ_k ⊙ d_k)-L_k+1/L_k+1-L_k+L_k+1-L_k/L_k+1-L_k
        + L_k-L(λ_k)/L_k+1-L_k+L(λ_k)-L_k-1/L_k+1-L_k,

    
which indicates that

    1-ρ_k ≤ 2κδ_k/C_1||d_k||_1≤ 1-η_1.


Thus, we have ρ_k ≥η_1, ||d_k||_1>η_2δ_k and the iteration is successful. 



  Suppose the function value estimates {(L_k,L_k+1)} are ε_F-accurate and

    ε_F < η_1 η_2 C_1.

If the kth iteration is successful, then the improvement in L is bounded such that 


    L(λ_k + θ_k ⊙ d_k)-L(λ_k) ≥ C_2δ_k^2,


where C_2 Δ= 2η_1 η_2 C_1-2ε_F.



    Since θ_k^s ∈ L and θ_k^s ||d_k^s|| > δ_k/n, from Lemma <ref>,
    
    L_k(λ_k + θ_k ⊙ d_k)-L_k(λ_k) ≥ 2 C_1 ||d_k||_1 δ_k

    Also, since the iteration is successful, we have ||d_k||_1>η_2 δ_k. Thus, we have
    
    
    L_k-L_k+1≥η_1(L_k - L_k-1) ≥ 2η_1 C_1 ||d_k||δ_k ≥ 2η_1 η_2 C_1 δ_k^2.

    Then, since the estimates are ε_F -accurate, we have that the improvement in f can be bounded as  
    
        L(λ_k + θ_k ⊙d_k)-L(λ_k + θ_k ⊙d_k) = L(λ_k + θ_k ⊙d_k)-L_k+1+L_k+1-L_k+L_k-L(λ_k ) ≥C_2 δ_k^2. 
    



  Let the random function V_k Δ=ℒ_k+1-ℒ_k, the corresponding realization be v_k, 


    ζ = max{4nκ,η_2,4κ/C_1(1-η_1)}   and   μ_1μ_2 > 1/2.


Then for any ε>0 such that ||d_k||_1 ≥ε and ζΔ_k ≤ε, we have 

    𝔼[V_k |  ||d_k|| ≥ε, ζΔ_k ≤ε] ≥1/2C_1 ||d_k||_1 Δ_k ≥θεΔ_k,

where θ = 1/2  C_1.
 


First of all, if kth iteration is successful, i.e. λ_k+1=λ_k + θ_k ⊙ d_k, we have 

    v_k = L(λ_k + θ_k ⊙ d_k)-L(λ_k).


If kth iteration is unsuccessful, i.e. λ_k+1=λ_k we have


    v_k = L(λ_k)-L(λ_k)=0.


Then we will divide the analysis into 4 cases according to the states (true/false) observed for the pair (I_k, J_k).

(a) I_k and J_k are both true. Since the f_k is a κ-approximation of f on ℬ(λ_k^s ,δ_k) and condition (<ref>) is satisfied, we have Lemma <ref> holds. Also, since the estimates (f̂_k,f̂_k+1) are ε_F-accurate and condition (<ref>) is satisfied, we have Lemma <ref> holds.  Combining (<ref>) with (<ref>), we have 


    v_k ≥  C_1 ||d_k||δ_k Δ= b_1.




(b) I_k is true but J_k is false. Since f_k is a κ-approximation of f on ℬ(λ_k^s ,δ_k) and condition (<ref>) is satisfied, it follows that Lemma <ref> still holds. If the iteration is successful, we have (<ref>), otherwise we have (<ref>). Thus, we have
v_k ≥ 0.


(c) I_k is false but J_k is true. If the iteration is successful, since the estimates (L_k,L_k+1) are ε_F-accurate and condition (<ref>) is satisfied, Lemma <ref> holds. Hence, 

    v_k ≥ C_2 δ_k^2.



If the iteration is unsuccessful, we have (<ref>). Thus, we have v_k ≥ 0 whether the iteration is successful or not.
     

(d) I_k and J_k are both false. Since L is convex and θ_k^s||d_k^s||<δ_k, for any g(θ_k) ∈∂ L(λ_k + θ_k ⊙ d_k), with Assumption  <ref>, ||g(θ_k)|| ≤ G , we have 


    v_k=f(α̂_k+td_k)-f(α̂_k) ≥ -g(t)^T td_k ≥ - G δ_k.


If the iteration is successful, then

    v_k ≥ -G δ_k Δ= b_2 .


If the iteration is unsuccessful, we have (<ref>). Thus, we have v_k ≥  b_2 whether the iteration is successful or not. 


With the above four cases, we can bound 𝔼[V_k |  ||d_k|| ≥ε, ζΔ_k ≤ε] based on different outcomes of I_k and J_k. Let B_1 and B_2 be the random counterparts of b_1 and b_2.  Then we have 

    𝔼[V_k |  ||d_k|| ≥ε, ζΔ_k ≤ε] 
       ≥μ_1 μ_2 B_1 + (μ_1(1-μ_2)+μ_2(1-μ_1))· 0 + (1-μ_1)(1-μ_2) B_2 
        = μ_1 μ_2  (C_1 ||d_k||Δ_k) - (1-μ_1)(1-μ_2)G Δ_k.


Choose μ_1 ∈ (1/2,1) and μ_2 ∈ (1/2,1) large enough such that 


    μ_1 μ_2 ≥ 1/2   and   μ_1 μ_2/(1-μ_1)(1-μ_2)≥2 G/C_1||d_k||,


we have

    
    𝔼[V_k |  ||d_k|| ≥ε, ζΔ_k ≤ε]     ≥1/2 C_1 ||d_k||Δ_k. 
    




A quick observation of the concluding condition reveals that the requirement Δ_k ≤ε/ζ implies that the supermartingale property holds. This prompts us to impose the condition that restricts δ_max = ε/ζ. This property is formalized in the following corollary. 
  Let 

    T_ε = inf{ k ≥ 0: ||d_k||_1 < ε}.


Then T_ε is a stopping time for the stochastic process Λ^k. Moreover, conditioned on T_ε≥ k, L_k is a submartingale. 





    
    From Theorem <ref>, we have
    
    𝔼[ℒ_k | 𝕃_k-1, T_ε  > k] ≥ L_k-1 + ΘεΔ_k.



Hence, ℒ_k is a submartingale.





 §.§ Convergence Rate
 

Building upon the results established in Theorem <ref>, where ℒ_k is demonstrated to be a submartingale and T_ε is identified as a stopping time, we proceed to construct a renewal reward process for analyzing the bound on the expected value of T_ε.
As highlighted in the abstract, Theorem <ref> confirms the rate of convergence to be O(1/ε^2).
To begin with, let us define the renewal process { A_l } as follows: set A_0 = 0, and for each l > 0, define A_l = inf{ m > A_l-1 : ζΔ_m ≥ε}, with ζ being specified in (<ref>). Additionally, we define the inter-arrival times τ_l = A_l - A_l-1. Lastly, we introduce the counting process N(k) = max{n: A_l ≤ k}, representing the number of renewals occurring up to the k^th iteration.


  Let 1/2 < p=μ_1 μ_2 ≤ℙ(I_k ∩ J_k). Then for all l ≥ 1,


    𝔼[τ_l] ≤p/2p-1.



First,
    
    𝔼[τ_l]     = 𝔼[τ_l | ζΔ_A_l-1 > ε] ·ℙ(ζΔ_A_l-1 > ε) +  𝔼[τ_l | ζΔ_A_l-1 = ε] ·ℙ(ζΔ_A_l-1 = ε)
       ≤max{𝔼[τ_l | ζΔ_A_l-1 > ε], 𝔼[τ_n | ζΔ_A_l-1 = ε] }.

    
If ζΔ_A_l-1 > ε, according to Algorithm <ref>, 


    𝔼[τ_l | ζΔ_A_l-1 > ε] = 1.


If ζΔ_A_l-1 = ε, by Theorem <ref>, we can treat {Δ_A_l-1,...,Δ_A_l} as a random walk, and we have 


    𝔼[τ_l | ζΔ_A_l-1 = ε] ≤p/2p-1.


Combining (<ref>) and (<ref>) completes the proof. 



  Let ζ and θ be the same as in Theorem <ref> and Δ_max = ε/ζ, then

    𝔼[N(T_ε)] ≤2 ζ F_max/θε^2+ζΔ_max/ε = 2 ζ F_max/θε^2+1,


where F_max=max_i ≤ (k ∧ T_ε)|F_i|.



    We will first show that 
    
    R_k ∧ T_ε =ℒ_k ∧ T_ε + Θε∑_j=0^k ∧ T_εΔ_j

is a submartingale. Using (<ref>),


    𝔼[R_k ∧ T_ε|ℱ_k-1]     = 𝔼[F_k ∧ T_ε|ℱ_k-1] + 𝔼[Θε∑_j=0^k ∧ T_εΔ_j | ℱ_k-1 ]
       ≤ F_k-1 - ΘεΔ_k + 𝔼[Θε∑_j=0^k ∧ T_εΔ_j | ℱ_k-1 ] 
        = F_k-1 + 𝔼[Θε∑_j=0^(k-1) ∧ T_εΔ_j | ℱ_k-1 ] 
        = F_k-1 + Θε∑_j=0^(k-1) ∧ T_εΔ_j 
        = R_(k-1) ∧ T_ε,


where the summation in the last expectation in (<ref>) is true by moving ΘεΔ_k inside the summation so that it has one less term if k < T_ε.

If k < T_ε, then 


    |R_k ∧ T_ε| = |R_k| ≤ F_max + Θε k Δ_max.


If k ≥ T_ε, then 


    |R_k ∧ T_ε| = |R_ε| ≤  F_max + Θε T_εΔ_max.


This is also bounded almost surely since T_ε is bounded almost surely. Hence, according to (<ref>) and the optional stopping theorem <cit.>, we have


    𝔼[Θε∑_j=0^ T_εΔ_j] ≤𝔼[R_T_ε] + F_max≤𝔼[R_0] + F_max≤ 2 F_max + ΘεΔ_max.


Furthermore, since the renewal A_n happens when ζΔ_j ≥ε and N(T_ε) is a subset of {0,1,2,...,T_ε}, we have 


    Θε(∑_j=0^ T_εζΔ_j) ≥Θε( N(T_ε) ε).


Combining (<ref>) and (<ref>), we have 
    𝔼[N(T_ε)] ≤2 ζF_max + ζΘεΔ_max/Θε^2 ≤2 ζF_max /Θε^2+ζΔ_max/ε.
      



Under conditions enunciated in <ref>, we have
    
    𝔼[T_ε] ≤p/2p-1(2 ζ F_max/Θε^2+2).




First, note that N(T_ε)+1 is a stopping time for the renewal process { A_n: n ≥ 0}. Thus, using Wald's equation (inequality form) <cit.>, we have 
    

    𝔼[A_N(T_ε)+1] ≤𝔼[τ_1] 𝔼[N(T_ε)+1].

    
Moreover, since A_N(T_ε)+1≥ T_ε, we have


    𝔼[T_ε] ≤𝔼[τ_1] 𝔼[N(T_ε)+1].


Hence, by Lemma <ref> and Lemma <ref>

    𝔼[T_ε] ≤p/2p-1(2 ζF_max /Θε^2+2). 






 §.§ Optimality condition


 
    Let (m_1, m_2) pair satisfies the requirement in Algorithm <ref> (1/4≤ m_1 < 1/2 and 1/4≤ m_2 < m_1). If k is the smallest index for which ||d_k|| < ε, then we have ||g_k|| < 4 ε.



    Suppose the claim is false,  we have ||g_k|| ≥ 4ε. Thus,
    
    ||d_k||^2     = ||λ_k^* g_k-(1-λ_k^*)d_k-1||^2
        = (λ_k^*)^2||g_k||^2+(1-λ_k^*)^2||d_k-1||^2-2λ_k^*(1-λ_k^*)⟨ g_k,d_k-1⟩
       ≥ (λ_k^*)^2||g_k||^2+(1-λ_k^*)^2||d_k-1||^2


where the inequality holds because Algorithm <ref> ensures the R condition in (<ref>). 

We will now divide the analysis into 2 cases: (a) If ||g_k|| > ||d_k-1|| > ε and ||g_k|| > 4 ε, then we have 

    ||d_k||^2 ≥ (17 (λ_k^*)^2 -2 λ_k^* + 1) ε^2


Note that

    λ_k^*=⟨ g_k,d_k-1⟩ + ||g_k||^2/||d_k-1||^2+||g_k||^2+2⟨ g_k,d_k-1⟩≥-m_2||d_k-1||^2+||g_k||^2/||d_k-1||^2+||g_k||^2,


where the inequality holds because R condition in (<ref>) ensures 0 > ⟨ g_k,d_k-1⟩≥ - m_2 ||d_k-1||^2. Thus, we claim that λ_k^* ≥2/17 because it is suffice to show


    - 17 · m_2||d_k-1||^2+17 · ||g_k||^2 ≥ 2 ||d_k-1||^2 + 2 ||g_k||^2.


This can be verified by observing m_2 < m_1 < 1/2 and ||g_k|| > ||d_k-1||. Thus, based on (<ref>) and λ_k^* ≥2/17, we have
||d_k|| ≥ε. This contradicts the assumptions of the lemma. 

(b) If ||d_k-1|| ≥ ||g_k|| > 4 ε, then we have 

    ||d_k||^2 ≥ (32 (λ_k^*)^2 -32 λ_k^* + 16) ε^2 ≥ 8 ε^2.


Thus, ||d_k|| ≥ 2 √(2)ε, which also contradicts the assumptions of the lemma.


 
    If k is the smallest index for which ||d_k|| < ε, then we will have 
    
    |f(x^*) - f(x̅_k)| = O(ε).




    Consider the difference between the primal objective value 

    
    f_k(x_k) = ∑_s=1^|S_k| h(x_k^s, ω^s)


    and the dual objective value

    
    L_k(x_k,λ_k) = ∑_s=1^|S_k| [h(x_k^s, ω^s) + ⟨λ_k^s, x_k^s - x̅_k ⟩ + ρ/2 ||x_k^s - x̅_k||^2].


    Since ||d_k|| < ε, from Lemma <ref>, we have ||g_k|| = ||x_k - x̅_k|| ≤ 4 ε. Thus, we have 

    
    L_k(x_k,λ_k) - f_k(x_k) = ∑_s=1^|S_k| [⟨λ_k^s, x_k^s - x̅_k ⟩ + ρ/2 ||x_k^s - x̅_k||^2] = O(ε).


    Let x_k^* =  f_k(x). From the duality theorem, we have 

    
    L_k(x_k,λ_k) ≤ f_k(x_k^*) ≤ f_k(x_k).


    (<ref>) and (<ref>) implies that 

    
    |f_k(x_k^*) - f_k(x_k)| = O(ε).


    On the other hand, based on Assumption <ref>, we have for every scenario s,

    
    |h(x_k^s,ω^s) - h(x̅_k, ω^s)| ≤ G · |x_k^s - x̅_k|.


    Thus, we have 

    
    |f_k(x_k) - f_k(x̅_k)| = O(ε).


    Eq. (<ref>) and Eq. (<ref>) implies that 

    
    |f_k(x_k^*) - f_k(x̅_k)| = O(ε).


    Also, let x^* =  f(x), we will show that 

    
    |f(x^*) - f(x_k^*)| = O(ε).


    First, note that we can use the same concentration inequality in Theorem <ref> to show that with probability at least (1-α)^2,  

    
    |f_k(x^*) - f(x^*)| ≤ε  and   |f_k(x_k^*) - f(x_k^*)| ≤ε.


    Thus, 

    
    f_k(x^*) ≤ f(x^*) + ε  and   f(x_k^*) ≤ f_k(x_k^*) +ε.


    Based on the definition of x^* and x_k^*, we have

    
    f(x_k^*) ≤ f_k(x_k^*) +ε≤ f_k(x^*) + ε≤ f(x^*) + 2ε,


    which proves Eq. (<ref>). 


    Finally, we can conclude the proof by combining Eq. (<ref>), Eq. (<ref>) and Eq. (<ref>).  
    




§ PRELIMINARY COMPUTATIONAL RESULTS
 

Our computational experiments are based on data sets available at the USC 3D Lab [https://github.com/USC3DLAB/SD]. Specifically, these problems are two-stage stochastic linear programming problems and we have tried lands3, pgp2, 20-term and baa99-20. The study considers two different methods: Classic PHA <cit.> and the sampling-based PHA. We will track the increase of dual objective values with respect to the number of QPs we will solve for both algorithms. Since the algorithms are stochastic, we also include the 95% confidence interval of objective values obtained from stochastic decomposition (SD) <cit.> as a benchmark. The algorithms were implemented on a MacBook Pro 2023 with a 2.6GHz 12-core Apple M3 pro processor and a 32GB of 2400MHz DDR4 onboard memory.



Remark: (1)The objective function values of sampling-based PHA reflect the submartingale property we have shown in Corollary <ref>. On the other hand, the classic PHA does not have such property. (2) For the problems which need more scenarios to convergence, the sampling-based PH algorithm tends to convergence faster than classic PH algorithm. In other word, the sampling-based PH algorithm is more data efficient and more scalable than the classic PH algorithm.  



§ CONCLUSION


We propose a novel sampling-based PH algorithm for large-scale stochastic optimization problems. Compared with classic PH algorithm, the proposed approach (a) does not need to know the explicit form of distribution of the random variables; (b) allows dynamic update of the sample size; (c) uses the conjugate subgradient direction and line-search to update the dual variables (d) implicitly updates the penalty parameter based on the principles of optimization. The illustrative experimental results show that it has good practical performance.

abbrv
 



 Paper Title: Nonasymptotic Analysis of Accelerated Methods With Inexact Oracle Under Absolute Error Bound.




Investigating the role of nuclear parameters on oscillation modes
 in hot Neutron Stars
    Debarati Chatterjee
    August 5, 2024
=======================================================================================




§ ABSTRACT 
Performance analysis of first-order algorithms with inexact oracle has gained recent attention due to various emerging applications in which obtaining exact gradients is impossible or computationally expensive.


Previous research has demonstrated that the performance of accelerated first-order methods is more sensitive to gradient errors compared with non-accelerated ones. This paper investigates the nonasymptotic convergence bound of two accelerated methods with inexact gradients to solve deterministic smooth convex problems. Performance Estimation Problem (PEP) is used as the primary tool to analyze the convergence bounds of the underlying algorithms.  By finding an analytical solution to PEP, we derive novel convergence bounds for Inexact Optimized Gradient Method (OGM) <cit.> and Inexact Fast Gradient Method (FGM) <cit.> with variable inexactness along iterations. Under the absolute error assumption, we derive bounds in which the accumulated errors are independent of the initial conditions and the trajectory of the sequences generated by the algorithms. 

Furthermore, we analyze the tradeoff between the convergence rate and accumulated error that guides finding the optimal stepsize. Finally, we determine the optimal strategy to set the gradient inexactness along iterations (if possible), ensuring that the accumulated error remains subordinate to the convergence rate.








§ INTRODUCTION


We consider the optimization problem

    min_∈^d f(),

where f is convex and has a Lipschitz continuous gradient. We assume the optimal value of (<ref>) to be lower bounded, i.e., f^* >-∞ with f^* ≜min_ f() and _* ≜_ f(). We use _μ,L with μ≥ 0 to denote the class of μ strongly convex functions with Lipschitz continuous gradient with constant L. This paper focuses on the class of merely convex functions with Lipschitz continuous gradient f ∈_0,L. 

For this class of functions, the gradient descent (GD) method has the convergence rate f(_k) - f^* = (k^-1). This rate can be improved to (k^-2) with Nesterovs's fast gradient method (FGM) <cit.>. Recently, using computer-aided analysis, the Optimized Gradient Method (OGM) has been proposed <cit.> that improves the convergence rate of FGM by √(2) constant.

The above rates are based on having access to the exact gradient at any point, which is not the case in many applications. The convergence rate deteriorates when there is an error in the gradient estimate. This paper considers gradient error in the absolute form as

    f() ≜ f() + ,     ≤ b = h(η),

where the error  can be controlled by a parameter η∈^r through a positive function h(·) and the bound holds either deterministically, which is the focus of this work, or with high probability in stochastic settings.

The parameter η represents, e.g., sampling size, computational effort, or accuracy of an approximation to achieve a desired gradient inexactness.

Below, the gradient oracle inexactness is illustrated through a few examples.



 §.§ Examples of inexact gradient oracles

Gradient inexactness is generally the result of the oracle not being evaluated at the desired point. For instance, if some or all coordinates of the desired point are the solution of an optimization problem that is only solved to a suboptimal solution, the resulting gradient will be inexact, e.g., bilevel optimization. In some other scenarios, the desired point results from an expectation that can only be estimated resulting in gradient inexactness see, e.g., nested composition optimization. 


[Bilevel optimization <cit.>]
	Consider the bilevel optimization problem
	
    min_      f(;^*())                    
         ^*() = _ g(,),

	with f being continuously differentiable and g being two times continuously differentiable and strongly convex functions. To solve the problem with a gradient-based method, we first obtain the gradient with respect to  as
	
    f(;^*())     = _ f(;^*()) - ^2_ g(,^*())[^2_ g(,^*())]^-1_ f(;^*()) 
        = _ f(;^*()) - ^*()^⊤_ f(;^*()).

	This requires solving the lower-level problem to optimality, which is unattainable in many scenarios. When the lower-level problem is solved to a suboptimal point () and the gradient is evaluated at this point instead of ^*(), the resulting gradient is inexact:
	
    f(;^*()) ≠f(,()) ≜_ f(,()) - ^2_g(,())[^2_ g(,())]^-1_ f (;()).

	Under some continuity assumptions, the gradient error is shown to be bounded as 
 
	
    f(;()) -  f(;^*())≤ C() - ^*(),

	where C ≜ L_f_ + L_f_C_g_/μ_g + C_f_(L_g_/μ_g + L_g_ C_g_/μ_g^2), with L_f_, L_f_ are the Lipschitz continuity constants of _ f(;) and _ f(;), respectively; L_g_, L_g_ are the Lipschitz constants of ^2_g(,), ^2_g(,) with respect to (w.r.t.) , respectively; C_f_, C_g_ are the upper bounds on _ f(;) and ^2_g(,), respectively; and μ_g is the strong convexity constant of g(,) w.r.t. – see <cit.> for details.

	Now let _k() be the solution of the lower-level problem by the gradient descent method with stepsize 2/μ_g + L_g after k iterations. From the convergence rate of GD for the class _μ,L, the gradient inexactness can be bounded as
	
    f(;_k()) -  f(;^*())≤ C (Q_g - 1/Q_g + 1)^k _0 - ^*(),

	where Q_g ≜ L_g/μ_g. The above bound follows an exponential decay and represents the error bound in (<ref>) with η being the iteration number k.



[Composition optimization]
	Consider the nested composition optimization in the form of
	
    min_ f() ≜ h(())   where  () ≜∑_i=1^N _i(),

    where h:^k→ and :^d→^k are continuously differentiable, and N is a large positive integer. From the chain rule, the exact gradient is f() = ()  h(()). However, assume one can only approximate  and  through sampling as
	
    () ≜∑_i=1^___[i]()   () ≜∑_i=1^___[i] (),

	where _, _ are two subsets of 1,2,⋯,N.

	Assume for any  and , h()≤ C_f and ()≤ C_g. Furthermore, assume h() is Lipschitz smooth with constant L_f, _i[_i() - ()^2] ≤σ^2_, and [_i() - ()^2] ≤σ^2_. Then with probability at least 1 - 1/ϵ, the inexact gradient estimate is bounded as
 
	
    ()  f(()) - ()  f(())^2 ≤ϵ(2C_f^2/_σ_^2 + 2C_g^2 L_f^2/_σ_^2 ).

 The proof of the above bound is presented in <ref>.
    This bound represents the error bound in (<ref>) with η=(_,_).
	Furthermore, if _ = _, for any fixed ϵ, the bound follows a power law decay.


Another common scenario is when f is unattainable, but, they can be approximated by (inexact) function values, i.e., zero-order information, denoted by f̃. In many such scenarios, it can be assumed that f̃() - f()≤ b_f. 

When f() has Lipschitz continuous gradient, the zero-order methods generate inexact gradient estimates with quantifiable error bounds (see <cit.> and references therein).


[Gradient estimation via forward finite difference]
	Let _i ∈^d denote the unit vector with the i-th element equal to 1, and l>0 be the finite difference interval. Define the i-th element of the gradient estimator as
	
    [f()]_i = f̃( + l _i) - f̃()/l.

	The error of the gradient estimate is shown by <cit.> to be bounded as
 
	
    f() -  f()≤√(d)L l/2 + 2√(d)b_f/l,

	which represents the error bound (<ref>) with η=(l,b_f). Note that the forward finite difference requires querying the zero-order oracle d+1 times to approximate the gradient at . 


Another common approach for gradient approximation from (inexact) function values is based on Gaussian smoothing. This method randomly samples directions for gradient estimation; hence, the sample size can be some arbitrary number.
[Gradient estimation via Gaussian smoothing]
	Let _i_i=1^n be i.i.d. random directions following the standard multivariate normal distribution, and define the gradient estimator as
	
    f() = 1/n∑_i=1^n f̃( + l _i) - f̃()/l_i.

	It is shown in <cit.> that
	
    __i_i=1^n[f()] -  f() ≤√(d)Ll + √(d)b_f/l,
 
 which represents the error bound (<ref>) with η=(l,b_f). Furthermore,
	
    f() - __i_i=1^n[f()] ^2 ≤3d/ϵ n(3  f()^2 + L^2 l^2/4 (d+2)(d+4) + 4b_f^2/l^2)

 holds with probability at least 1 - 1/ϵ.
	In the later probabilistic bound, h(η) is determined by l, n and b_f as η=(l,b_f,n). This bound contains a constant, a relative error, and an absolute error terms. 







 §.§ Related works

The study on inexact gradient methods (without acceleration) dates back to 1987 <cit.>, where two notions of inexactness were defined, i.e.,

    f() -  f()    ≤δ,        (absolute error),
     f() -  f()    ≤δ f(),   ∀δ∈[0,1]        (relative error).


Below, we review the literature in stochastic setups for non-accelerated methods and (in more detail) in deterministic setups for accelerated gradient-based methods.




  
Stochastic non-accelerated methods under biased oracles. The above gradient inexactness assumptions have specifically been considered in stochastic setups where f() is a biased stochastic estimate of f().
<cit.> studies the performance of the biased stochastic gradient descent (SGD) method under both absolute and relative bias. <cit.> considers setups where lower bias and variance require higher computational cost and investigates their tradeoff in stochastic optimization. Similarly, <cit.> analyzes bias-variance tradeoff in the convex bandit setting. <cit.> analyzes biased stochastic gradient-based methods under the absolute bias oracles with and without bias-variance tradeoff. <cit.> (and references therein) summarizes different bias assumptions and introduces a hybrid approach integrating absolute and relative biases. <cit.> considers biased gradient estimates with absolute error and proposes adaptive algorithms that adaptively set the bias throughout the iterations of the algorithms.




  
Accelerated gradient methods with inexact gradient oracles.
Nesterov-type acceleration techniques <cit.> with inexact gradient oracles have been analyzed under different inexact assumptions. Below, these assumptions and their connections with the absolute error (which is the focus of this paper) are reviewed; moreover, the convergence performance of deterministic accelerated methods with inexact gradients are analyzed under the corresponding assumptions.

An inexact generalization of Nesterov's Fast Gradient Method (FGM), namely, iFGM, is presented in Algorithm <ref> which obtains accelerated rates of convergence in convex setups. To simplify the exposition, we only consider the merely convex problems and we set the dual update step as gradient descent with projection.




<cit.> analyzes the performance of iFGM to minimize Liptshitz smooth objective function over compact convex sets , with the inexact gradient satisfying

    f() -  f(),  - ≤δ,    ∀, ,∈,

and shows the sequence generated by iFGM satisfies

    f(_K+1) - f_* ≤L_0 - _*/A_K + 3δ .

This inexactness assumption can be derived from the absolute error assumption for the bounded feasible region with diameter D as

    f() -  f(),  - ≤b̅ - ≤b̅max_,∈- = b̅D ≜δ,

which shows the dependence of the error term on the diameter of the feasible region.

The notion of first-order (δ,L)-oracle is proposed in <cit.>, which removes the requirements of boundedness of the feasible set and differentiability of the objective function with a generalized definition of convexity and Lipschitz smoothness. More specifically, <cit.> assumes that the oracle provides an inexact function and subgradient pair (f̃_δ_(), f_δ_()) that satisfies

    0 ≤ f() - (f̃_δ_() +  f_δ_(),-≤L/2 - ^2 + δ_.

With (δ,L)-oracle, <cit.> shows the sequence generated by iFGM satisfies

    f(_K+1) - f_* ≤L _0 - _*^2/A_K + ∑_i=0^K A_i δ__i/A_K.

This result allows different inexactness levels for different iterations. However, to use the above results for an oracle that satisfies the absolute error assumption, the boundedness of the feasible region is needed <cit.>, i.e., an oracles with absolute error assumption f̃() - f()≤Δ_1 and f() -  f()≤Δ_2,  f̃_δ_ = f̃() - Δ_1 - Δ_2 D, f_δ_() =  f(), and δ_ = 2Δ_1 + 2Δ_2 D (with D being the feasible region diameter) is a (δ,L)-oracle.


To overcome the boundedness drawback of the (δ,L)-oracle, <cit.> proposes the inexact first-order oracle of degree q as

    0 ≤ f() - (f() + f_δ(),-≤L/2-^2 + δ-^q,

with q ∈ [0,2). With the appropriate selection of the q and δ parameters, any inexact gradient that satisfies the absolute error condition satisfies the right-hand-side inequality above without the requirement of the bounded feasible region. Under this new oracle assumption, the sequence generated by iFGM satisfies

    f(_K+1) - f_* ≤4(L+q ρ )_0 - _*^2/(K+1)(K+2) + (K+3)(2-q)δ^2/2-q/2ρ ^q/2-q,

with ρ >0 be any fixed number. Minimizing the right-hand-side with respect to ρ, the convergence rate becomes

    f(_K+1) - f_* ≤4L_0 - _*^2/(K+1)(K+2) + 8^q/2_0 - _*^q(K+3)/((K+1)(K+2)(K+3))^q/2δ.

This latter condition allows unbounded regions; however, its left-hand-side inequality requires the inexact gradient to be a subgradient, which can not be guaranteed under the absolute error assumption. Furthermore, in the convergence bound, the error term depends on the initial condition. Finally, the analysis is provided for fixed inexactness levels.


Another accelerated algorithm studied under inexact oracles is the Similar Triangles Method (STM) <cit.>– see the inexact version in Algorithm <ref>. Compared to FGM, it only requires one projection for each iteration and, hence, it is preferable for constrained problems. The convergence properties of iSTM are studied under both absolute and relative errors.


When the feasible region = ^d, i.e., no projection is required, iSTM can be formulated as iFGM. From A_k = A_k-1 + α_k, we can see that the _k updates in iFGM and iSTM are the same with different indices since iFGM updates _k at the end of each iteration while iSTM updates it at the beginning of each iteration. The _k updates for the two algorithms are the same. Hence, we only need to consider the _k updates. From step 3 of iSTM, A_k-1_k-1 = A_k _k - α_k _k-1. Inserting this identity into step 5, and by the update of step 4, we have

    _k = _k + α_k/A_k(_k - _k-1) = _k - α_k^2/A_kLf(_k).

When α_k^2 = A_k, the above update is equivalent to the step 2 in iFGM.


Furthermore, <cit.>
analyzes iSTM under the inexact gradient with absolute error bound.



The sequence generated by iSTM with α_k^2 = A_k, ∀ k, satisfies

    f(_K) - f_* ≤8L_0 - _*^2/K^2 + 3R̃_K δ +K δ^2/2L,

with R̃_K ≜max_0≤ j≤ K_j - _*, _j - _*, _j - _*. 
This convergence bound is only given under fixed inexactness levels and restricts the stepsize as α_k^2 = A_k. Another drawback is that R̃_K is not exactly quantifiable; indeed, <cit.> only shows its boundedness by _0 - _* under the exact gradient oracle.


The boundedness of R̃_K is shown by the same group under the relative error assumption with a relaxed stepsize <cit.>. 





It is shown that the sequence generated by iSTM with α_k = (k+2)^p-1/2α, p ∈ [1,2], and
α = (max1,K^p/4√(δ), K^p/2δ ,K^pδ^2  ) satisfies

    f(_K) - f_* ≤16 α L _0 -_*^2/(K+2)^p.

The paper shows R̃_K is bounded by 2_0 - _* when the stepsize is properly chosen and, hence, there is no separate error term in the convergence rate.


Besides the abovementioned works, several researchers have proposed alternative error assumptions and results. The result of iFGM for the strongly convex setup is provided in <cit.>. <cit.> considers a variation of iFGM with the absolute error setting.

To exploit the tradeoff between the convergence rate and accumulated error, the intermediate gradient method is proposed in <cit.> which is closely related to iFGM. This method is further generalized for other setups in <cit.>.

<cit.> and <cit.> analyze nonaccelerated and accelerated algorithms under relative and absolute errors, respectively, with the latter focusing on the tradeoff between robustness to error and convergence rate–see also <cit.>.   <cit.> assume the error to satisfy the f() -  f()≤ B - _* condition, which can be derived from the relative error bound. <cit.> provides results for stochastic proximal inexact gradient methods with and without acceleration for strongly convex problems, while <cit.> studies generalized monotone variational inequality problems.
Other notable contributions include the analysis of absolute and relative inaccuracy for proximal point methods in <cit.>, the study of proximal gradient methods with absolute error in <cit.>, the inexact proximal gradient method for weakly convex functions with absolute error in <cit.>, and minimization of the accumulated error when the total allocation of error-control resources is fixed <cit.>.



 §.§ Contributions

This paper employs Performance Estimation Problem (PEP) techniques to primarily analyze the convergence rate of the Generalized Optimized Gradient Method with an inexact gradient oracle (iGOGM). Our key contributions are as follows:

    
  * We derive an analytical solution to the relaxed PEP optimization problem for iGOGM under variable oracle inexactness levels. The approach used to find this solution inspires our theoretical proof of the convergence. Notably, this proof can be understood without prior knowledge of PEP techniques.
    
  * The derived convergence bound consists of two components: the exact convergence rate and the accumulated error. Significantly, the error term is independent of the initial condition _0 - _*, and is determined solely by the Lipschitz constant and the step size.
    
  * We provide a comprehensive analysis of the convergence bound from two perspectives: i) We investigate the tradeoff between the convergence rate and the accumulated error. ii) We establish optimal inexactness levels that preserve the method's convergence rate.
    
  * We also provide the PEP-inspired convergence bound for the Generalized Fast Gradient Method with inexact gradient oracle (iGFGM).


We also provide the codes for our numerical experiments as well as the verification of the algebraic steps in our proofs. These codes are available at https://github.com/Yin-LIU/Inexact-Acceleration-PEPhttps://github.com/Yin-LIU/Inexact-Acceleration-PEP.



  
Notations
Vectors and matrices are denoted by bold and capitalized bold letters, respectively. Sets are denoted by calligraphic letters. In the absence of additional instruction, a bold letter coupled with an arrow, e.g., , generally denotes a standard basis vector, which has exactly one element equal to 1 while other elements are 0. _i,j stands for the (i,j)-th element of the matrix . The notation ^\ [i] refers to the submatrix of  obtained by removing its i-th column and row. []_n× m indicates the shape of the matrix and [] is used to emphasize the item inside the brackets is a matrix. · denotes the l_2 norm for vectors and Frobenius norm for matrices.
log(·) represents the natural logarithm.





§ PERFORMANCE ESTIMATION PROBLEMS (PEP)



 §.§ PEP overview

The tool used for analyzing the convergence rate of the inexact accelerated methods is known as Performance Estimation Problem (PEP). This technique is proposed by <cit.> which uses an optimization problem to find the worst-case performance when a given algorithm is applied to a class of functions.

To solve the convex and Lipschitz smooth problems with the general first-order method using an exact gradient oracle, the PEP is defined as

    max_d, f                                                                                                                                       
    _*, _0, ⋯, _k ∈^d                                                                                                                      
     f(_0), ⋯,  f(_K) ∈^d       f(_K) - f(_*)                                                                            
              f ∈_0,L,    _* ∈_ f()                                           
       _0 - _*^2 ≤ R^2                                                                
       _k =_0 - ∑_i=0^k-11/Lθ_k,i f(_i)     k  = 1,⋯,K,

where _0,L denotes the class of merely convex functions with Lipschitz continuous gradients. The general first-order algorithm (GFO) performs the update

    _k =_0 - ∑_i=0^k-11/Lθ_k,i f(_i)     k  = 1,⋯,K,

which covers a wide range of first-order methods, including GD, FGM, Polyak's Heavy Ball Method <cit.>, and Optimized Gradient Method (see <ref>) <cit.>.

The above problem is an infinite dimensional optimization as the variable f is a function. <cit.> proposed a notion of _μ,L-interpolation that only considers function values and gradients at discrete points _k, which are the terms that play a role in the optimization problem.

	Let  be an index set, and consider the set of triples = { (_i,_i,f_i) }_i∈ where _i, _i ∈^d for all i∈. Set  is _μ,L-interpolatable if and only if there exists a function f ∈_μ,L(^d) such that we have both _i =  f(_i) and f(x_i) = f_i for all i ∈.




	Set { (_i, _i, f_i) }_i ∈ I is _μ,L-interpolable (0 ≤μ≤ L ≤∞) if and only if the following set of conditions holds for every pair of indices i ∈ I and j ∈ I:
	
    f_i - f_j - _j^⊤ (_i - _j) ≥1/2(1 - μ/L)( 1/L_i - _j ^2 + μ_i - _j ^2 - 2μ/L (_j - _i)^⊤ (_j - _i) ).



The corresponding discrete PEP model is

    max_d,                                                                                                                   
    ⊂^d×^d×, 
    _0, ⋯, _K ∈^d    f_K - f_* PEP-Exact
          = (_i,_i,f_i)_i ∈*,0,1,⋯,K
       _* = 0,   _0 - _*≤ R^2                                                                            
        f_i - f_j - _j^⊤(_i - _j) ≥1/2L_i - _j^2   ∀ i,j ∈*,0,1,⋯,K
       _k =_0 - ∑_i=0^k-11/Lθ_k,i_k     k = 1,⋯, K.

The theoretical equivalence of the discrete PEP model and the worst-case performance of GFO over the given class of functions is proven in <cit.> with convex interpolation definition that guarantees PEP to generate the tight worst-case performance.

The PEP model can also be used to find algorithms that minimize the worst-case function. In <cit.>, numerical results indicate a better algorithm than the well-known FGM. <cit.> follows this idea and provides an explicit form of the generated algorithm called the Optimized Gradient Method (OGM), which was proven to match the lower bound with a smaller (1) constant <cit.>. <cit.> further reveals the equivalence between OGM and the conjugate gradient method–see also <cit.>.

PEP technique has been successfully applied to various optimization methods, including gradient descent <cit.>, algorithms with line search <cit.>, proximal gradient methods <cit.>, proximal point algorithms <cit.>, algorithms with inexact oracles <cit.>, and many others <cit.>. Besides the deterministic smooth regimes, PEP has also been incorporated for stochastic problems <cit.> and for problems satisfying relative smoothness <cit.>.

When optimizing only over the function class given an algorithm, PEP can be equivalently written as a Semidefinite Programming (SDP) and solved efficiently. However, when optimizing over both algorithm and function, the problem is nonconvex, and finding its global solution is NP-hard. Hence, to find an optimal algorithm a relaxed PEP formulation is generally solved which may not provide an optimal algorithm. <cit.> proposes a method to solve the original nonconvex PEP and provides a numerical guide to design algorithms.


A closely related topic is the analysis based on Integral Quadratic Constraints (IQC), originally studied in control theory <cit.>. Similar to PEP, IQC aims to find worst-case functions through optimization, but it is not an exact formulation of the function class; hence, it can only provide an upper bound. This technique has been used to analyze first-order methods in <cit.> and gradient method with inexact oracles in <cit.>.





 §.§ PEP for optimization with inexact oracle




For inexact gradient oracles, the GFO is generalized to inexact GFO (iGFO) as

    _k = _0 -1/L∑_i=0^k-1θ_k,i(_i + _i),

and the discrete PEP model is adjusted as

    max_d,                                                                                                                   
    ⊂^d×^d×, 
    _0, ⋯, _K ∈^d    f_K - f_* PEP
          = (_i,_i,f_i)_i ∈*,0,1,⋯,K
       _* = 0,   _0 - _*^2≤ R^2                                                                            
        f_i - f_j - _j^⊤(_i - _j) ≥1/2L_i - _j^2   ∀ i,j ∈*,0,1,⋯,K
       _k =_0 - ∑_i=0^k-11/Lθ_k,i(_k + _k)    k = 1,⋯, K                              
       _k^2 ≤ b_k^2   k=0,⋯ K-1,

which is the generalization of the exact model. If b_k ≡ 0 then _k ≡, iGFO reduces to GFO, and (<ref>) reduces to (<ref>).

This formulation can be further simplified by introducing a Gram matrix. Define

    ≜f_0 - f_*     f_1 - f_*    ⋯    f_K - f_*∈^1× (K+1),                                                       
       ≜_0       _0        ⋯   _K-1   _0    ⋯   _K ∈^d × (2K+2),

and the Gram matrix ≜^⊤∈^2K+2 which is a symmetric positive semidefinite (PSD) matrix. Also, we define standard basis vectors _k ∈^K+1 and _k, _k, _k ∈^2K+2 to select corresponding columns in  or  to recover the needed element, which satisfy

    f_k - f_* = _k,   _* = 0,   _k = _k,   _k = _k,                           
       _0 = _0,   _k = _k =  (_0 - 1/L∑_i=0^k-1θ_k,i(_i + _i)).

Note. We note that the subscript k on the standard basis vectors does not mean that the k-th element of that vector is equal to 1. For instance, based on the definitions of  and  in (<ref>), _k is a standard basis vector with k+1-th element equal to 1, _k is a standard basis vector with k+2-th element equal to 1, _k is a standard basis vector with K+k+2-th element equal to 1, and _0 is a standard basis vector with the first element equal to 1. With a slight abuse of the notation, _k (which is not a standard basis vector) is the vector used to represent _k by a linear combination of the columns of  and it is a function of θ_k,i.

Without loss of generality, we assume f_*=0 and _*=0, hence
_* = 0, _* = 0. With this new notation, the PEP problem is written as

    max_d,∈_+^2K+2,∈^1× (K+1)     _K  PEP-Gram
         (_j - _i) + (^ij) ≤ 0,   ∀ i,j ∈*,0,1,⋯,K
       (_0 _0^⊤) - R^2 ≤ 0                                                        
       (_i _i^⊤) - b_i^2 ≤ 0,    i = 0, ⋯, K -1                          
       ()≤ d,

where

    ^ij≜1/2((_i - _j) _j^⊤ + _j(_i - _j)^⊤ ) + 1/2L(_i - _j)(_⃗i⃗ - _j)^⊤.


When dealing with large-scale problems, i.e., when 2K+2 ≤  d, we can drop the rank constraint (G)≤ d without changing the optimal value (see Theorem 5 in <cit.>), and the equivalent problem will be a Semidefinite Programming problem (SDP) written as

    max_∈^2K+2_+,∈^(1× K+1)     _K  SDP-PEP
         (_j - _i) + (^ij) ≤ 0,   ∀ i,j ∈*,0,1,⋯,K
       (_0 _0^⊤) - R^2 ≤ 0                                                       
       (_i _i^⊤) - b_i^2 ≤ 0,    i = 0, ⋯, K -1.


The optimal solution of (<ref>) will provide an exact worst-case performance of a given algorithm for convex problems. It is also possible to find the optimal algorithm with parameter þ if we consider the minimax problem as

    min_θ_k,imax_,∈(SDP-PEP)_K,

where ,∈(SDP-PEP) refers to  and  satisfy the constraints of (<ref>). Since ^i,j is indeed a function of {θ_k,i}, this minimax problem has bilinear terms in the constraints of the inner problem, and solving it for the global solution is NP-hard. OGM's <cit.> idea is to select a subset of the constraints and upper bound the inner maximization by its dual minimization problem. By wisely selecting the constraints, the hard minimax problem can be relaxed as a bilinear minimization problem which can be further relaxed as a linear SDP. Solving this final problem numerically inspires a theoretical solution and results in the first version of OGM (<ref>).


Recall that for FGM, there are two sequences, the primary sequence {_k } and the secondary one {_k }. The OGM is optimized for minimizing f(_k)-f_* while the commonly used measure is f(_K) - f_*. Due to this fact, <cit.> propose a new measure f(_K) -f_*- 1/2L f(_K)^2 since for smooth function and with OGM update rule, f(_K+1) = f(_K - 1/L f(_K)) ≤ f(_K) - 1/2L f(_K)^2. The solution of this new measure results in OGM (<ref>). These two algorithms only differ in the last iteration's α_K update. For simplicity, we only focus on this measure and its corresponding relaxed PEP.

[t]0.48
	

[t]0.48
	








The relaxed optimization problem is

    max_∈_+^2K+2, ∈^(1× K+1)     _K - 1/2L_K_K^⊤P
         (_i+1 - _i) + (^i,i+1) ≤ 0,    i= 0,⋯ K-1,          (v_i,i+1) 
       (_i-_*) + (^*,i) ≤ 0,    i= 0,⋯ K,         (v_*,i)           
       (_0 _0^⊤) - R^2 ≤ 0,                                                 (τ)      
       (_i _i^⊤) - b_i^2 ≤ 0,    i = 0, ⋯, K -1.              (u_i)

One can solve this problem by solving its dual problem. To write the dual of (P), we define the dual variables as , _*, τ, and  as indicated in the parenthesis following each constraint. The Lagrangian function is

    (,,,_*,,τ)                                                                                                                                                                                                                        
    
    	=    _K - 1/2L_K_K^⊤ - ∑_i=0^K-1v_i,i+1((_i+1- _i)+ (^i,i+1)) - ∑_i=0^Kv_*,i((_i - _*)+(^*,i)) 
        - τ((_0_0^⊤ ) - R^2 )  -  ∑_i=0^K-1 u_i ((_i_i^⊤ ) - b_i^2 )                                                                                                    
    
    	=    τ R^2 + ∑_k=0^K-1 u_i b_i^2 + (_K + ∑_i=0^K-1v_i,i+1(_i - _i+1) + ∑_i=0^K (_* - _i))                                                                          
        - ((∑_i=0^K-1 v_i,i+1^i,i+1 + ∑_i=0^K v_*,i^*,i + τ_0_0^⊤ + ∑_i=0^K-1 u_i _i_i^⊤ + 1/2L_K_K^⊤)).

For the problem max_∈_+,(,,,_*,,τ) to have a bounded solution, we can write the corresponding dual problem as

    min_τ,,_*,≥ 0     τ R^2 + ∑_i=0^K-1u_i b_i^2 D
         _K + ∑_i=0^K-1v_i,i+1(_⃗i⃗ - _i+1) + ∑_i=0^K v_*,i (_* - _i) = 0                                                                          
       ∑_i=0^K-1v_i,i+1^i,i+1 + ∑_i=0^K v_*,i^*,i + τ_0 _0^⊤ + ∑_i=0^K u_i _i _i^⊤  + 1/2L_K_K^⊤≽ 0.



Here we omit the process of relaxing the minimax problem and use the solution in <cit.> directly to explain the property of this feasible solution with the inexact oracle. Define A_k = ∑_i=0^k α_i and A_0 = 1, A_-1 = 0, and assume α_k^2 ≤ A_k, the feasible solution of the minimax problem is

    τ = L/4A_K,    v_i,i+1=A_i/A_K,      v_*,0=v_0,1,    v_*,i =v_i,i+1- v_i-1,i,    v_*,K = 1 - v_K-1,K,

and

    θ_k,i= 2α_iα_k + A_k-1/A_k      i=k-1            
    2α_k α_i/A_k + A_k-1/A_kθ_k-1,i      0 ≤ i≤ k-2.



	The above feasible solution works for both OGM and Generalized OGM (discussed in the next section), which means for any stepsize α_k and A_k that satisfy α_k^2 ≤  A_k, (<ref>) and (<ref>) satisfies the constraints of (<ref>).



It is easy to verify that the equality constraint holds with this solution. We still need to find the feasible solution for  to satisfy the positive semidefinite (PSD) constraint. After substitution by (<ref>) and (<ref>), the left-hand-side matrix of the PSD constraint is

    = [l]
    		      L/4A_K    [0]_1× K    []_1 × (K+1)
    
    		      [0]_K × 1    []_K × K    []_K × (K+1)
    
    		      [^⊤]_(K+1) × 1    [^⊤]_(K+1) × K    []_(K+1) × (K+1),

where =( u_0 ⋯ u_K-1 ), _i= - α_i-1/2A_K,

    _i,j=2α_i-1α_i +A_i-1/2LA_K      j=i+1               
    α_i-1α_j-1/LA_K      i+2≤ j ≤ K +1
    	          ,
    	  and  _i,j= A_i-1/LA_K      i=j      
    α_i-1α_j-1/LA_K      i ≠ j 
    .




For K=3, one example of  is listed below

    = ([                L/4 A_3                      0                      0                      0              - 1/2 A_3            - α_1/2 A_3            - α_2/2 A_3            - α_3/2 A_3;                      0                    u_0                      0                      0                      0      1 + 2 α_1/2 L A_3              α_2/L A_3              α_3/L A_3;                      0                      0                    u_1                      0                      0                      0 2 α_1α_2 + A_1/2 L A_3           α_1α_3/L A_3;                      0                      0                      0                    u_2                      0                      0                      0 2 α_2α_3 + A_2/2 L A_3;              - 1/2 A_3                      0                      0                      0                1/L A_3              α_1/L A_3              α_2/L A_3              α_3/L A_3;            - α_1/2 A_3      1 + 2 α_1/2 L A_3                      0                      0              α_1/L A_3              A_1/L A_3           α_1α_2/L A_3           α_1α_3/L A_3;            - α_2/2 A_3              α_2/L A_3 2 α_1α_2 + A_1/2 L A_3                      0              α_2/L A_3           α_1α_2/L A_3              A_2/L A_3           α_2α_3/L A_3;            - α_3/2 A_3              α_3/L A_3           α_1α_3/L A_3 2 α_2α_3 + A_2/2 L A_3              α_3/L A_3           α_1α_3/L A_3           α_2α_3/L A_3                    1/L ])





From the structure of , we can observe that all the rows of  have at least one positive element, hence to make ≽ 0 we must have >. Following the decomposition of <cit.>, i.e., = 1/τ_τ_τ^⊤ + with _τ being the first column of , with the condition A_k = ∑_i=0^kα_i, the residual matrix  has the form

    = [l]
    		      0                               [0]_1× K    [0]_1 × K+1
    
    		      [0]_K × 1    []_K × K    []_K × K+1
    
    		      [0]_K+1 × 1    [^⊤]_K+1 × K    [ - 1/τ^⊤ ]_K+1× K+1,

where - 1/τ^⊤ = (0,(A_i-1-α_i-1^2)/(LA_K)) for any A_k, and equals to  with α_k defined in OGM.  in the example of K=3 is equal to 


    = ([                      0                      0                      0                      0                      0                      0                      0                      0;                      0                    u_0                      0                      0                      0      1 + 2 α_1/2 L A_3              α_2/L A_3              α_3/L A_3;                      0                      0                    u_1                      0                      0                      0 2 α_1α_2 + A_1/2 L A_3           α_1α_3/L A_3;                      0                      0                      0                    u_2                      0                      0                      0 2 α_2α_3 + A_2/2 L A_3;                      0                      0                      0                      0                      0                      0                      0                      0;                      0      1 + 2 α_1/2 L A_3                      0                      0                      0      A_1 - α_1^2/L A_3                      0                      0;                      0              α_2/L A_3 2 α_1α_2 + A_1/2 L A_3                      0                      0                      0        A_2-α_2^2/L A_3                      0;                      0              α_3/L A_3           α_1α_3/L A_3 2 α_2α_3 + A_2/2 L A_3                      0                      0                      0       A_3 - α_3^2/LA_3 ]).


Recalling that ^\[1] denotes the matrix resulting from dropping the first column and row of , this submatrix is exactly the Shur complement of the block [τ] of the matrix . Since τ >0, we know that ≽ 0 if and only if ^\[1]≽ 0. In the following lemma, we will show that = ∞ is the unique feasible solution when A_i - α_i^2 = 0.



	With the given solution in (<ref>) and (<ref>), and A_i - α_i^2 = 0, = ∞ is the unique solution that makes ^\[1]≽ 0. When A_i - α_i^2>0, = ∞ is still feasible but not unique.


	First, we know that all u_i are positive numbers. Define _k be the unit vector that only the k-th element is 1. If ^\[1]≽ 0, then for any c∈ and defining ≜_i+c_j, the condition
	
    0 ≤   ^⊤^\[1] =  c^2 ^\[1]_j,j +  ^\[1]_i,i + 2c^\[1]_i,j

	holds. To show that, there are four different scenarios:
	
		
  * ^\[1]_i,i = 0, ^\[1]_j,j = 0,
		
  * ^\[1]_i,i = u_k, ^\[1]_j,j = u_l,
		
  * ^\[1]_i,i = u_k, ^\[1]_j,j = 0,
		
  * ^\[1]_i,i = 0, ^\[1]_j,j = u_l.
	
For the first and second scenarios, it is obvious that ^\[1]_i,j=0, with diagonal elements being non-negative, the condition holds.
For the third scenario, ^\[1]_i,j≥ 0. Consider the case ^\[1]_i,j > 0, since c can be any negative number, the condition  	^⊤^\[1] = u_k + 2c ^\[1]_i,j≥ 0  holds only when u_k = ∞.
	For the fourth scenario, the situation is similar. ^⊤^\[1] =c^2 u_l + 2c ^\[1]_i,j≥ 0 holds only with u_l = ∞.
	We get the final result by combining all these scenarios.


The exact OGM with feasible solution (<ref>) and = ∞ provides the convergence rate of

    f(_K+1) - f_* ≤LR^2/4A_K + ∑_i=0^K-1 u_i b_i^2 = LR^2/4A_K.


The unique feasible solution = ∞ has no effect on the final convergence rate since b_i ≡ 0. However, with the inexact gradient oracle, such a solution will ruin the convergence rate as the accumulated error approaches ∞. Hence, we need to change either the feasible solution given in (<ref>) or the algorithm's structure defined in (<ref>). The details will be discussed in the next section.



§ GENERALIZED OGM WITH INEXACT GRADIENT ORACLE

The properties of OGM and its generalization have been thoroughly studied. However, the convergence analysis of OGM with inexact gradient oracle has not been performed yet. Furthermore, the existing analysis of FGM with inexact oracles with the absolute error assumption depends on the trajectory of sequences. This section aims at closing these gaps.

It is possible to optimize the algorithm with the same procedure as OGM, but the solution depends on iteration number K, initial condition R, and inexactness levels b_i_i=1^K. It is difficult to find a recursive structure of the algorithm that captures the numerical solution (see the experiment results in <ref>). Hence, we focus on Generalized OGM (GOGM) with inexact oracle and try to find its convergence rate with b_i >0. Furthermore, the convergence bound of Generalized FGM (GFGM) is also analyzed in Section <ref>.

The GOGM is a generalization of OGM (<ref>) as it only requires the stepsize to satisfy α_k^2 ≤ A_k. The inexact GOGM (iGOGM) is provided in <ref>, in which the gradients in steps 2 and 3 are replaced by their inexact alternatives.




The update rule in step 4 guarantees α_k^2 ≤ A_k when λ_k ∈ [0,1] as

    α_k+1^2 = (A_k+1 - A_k)^2 =λ_k+1 A_k+1≤ A_k+1.

Furthermore, when λ = 1, the _k+1 update (step 5) reduce to

    _k+1 = (1 - 1/α_k+1) _k+1 + 1/α_k+1_k+1,

the common update step in both FGM and OGM.



The optimality measure f(_K) - f_* - 1/2L f(_K)^2 used in the analysis of exact GOGM is also a meaningful one for the inexact problem. For iGOGM, we have

    f(_K+1) =    	f (_K-1/L( f(_k) + _K)) 
    ≤    f(_K) -1/L f(_K), f(_K) + _K + 1/2L f(_K)+_K^2 
    
    		=                                                                 f(_K) - 1/2L f(_K)^2 + 1/2L_K^2.

 Comparing to the measure f(_K) - f_* - 1/2L f(_K)^2, the extra term 1/2L_K^2 can be discarded since the constraint _K^2 ≤ b_K^2 is the only one that involves _K and the PEP solution sets _K^2=b_K^2. Since the algorithm only generates _k and _k up to k=K, _K+1 is a dummy variable used for the theoretical guarantee, and setting b_K=0 does not affect the complexity of the algorithm.
 

	






 §.§ PEP approach for the convergence analysis of iGOGM

The goal of this part is to find an analytical feasible solution of (<ref>) with iGOGM stepsize. The feasible solution provides an upper bound of the worst-case convergence rate. To do so, first notice that the values of v_*,i are uniquely determined by v_i,i+1 through the equality constraint in (<ref>) and, hence, can be removed from the decision variables. Next, consider the decision variable τ.
Recall the objective of the problem (<ref>), τ R^2 + ∑_i=0^K-1u_ib_i^2, contains two components: the first one is the convergence rate and the second one is the accumulated error. Given the convergence rate of the exact GOGM, i.e., LR^2/4A_K, we adjust the objective of (<ref>) as

    L/4A_KR^2 + ∑_i=0^K-1u_ib_i^2,

which is the summation of the convergence rate of exact GOGM and the accumulated error ∑_i=0^K-1u_ib_i^2. With these replacements, the decision variables we need to consider are v_i,i+1 and u_i. To simplify the notation, in the following text, we will use  and  to represent the sequences v_i,i+1  and u_i, respectively.

Solving (<ref>) with this new objective, we observe from the numerical solution that  is almost equal to its analytical counterpart for problems with exact oracles given in (<ref>). In this experiment, we set K = 10,20,⋯,100, R = 1,10,1000, and L = 1,10,1000 which result in 90 different scenarios in total. For each scenario, we generate 10 instances with random stepsize parameter λ_k and inexactness levels b_i. In <ref>, three examples of the relative difference between the numerical solution of inexact GOGM v^num._k and the analytical solution of exact GOGM v^anly._k are illustrated. The solid line is an average of 90 instances. Furthermore, <ref> illustrates the histogram of relative error for all 900 instances which also supports this conjecture. 












As we claimed in <ref>, OGM with (<ref>) has a unique feasible solution with = ∞. To make  to be bounded, we have to guarantee that the stepsize condition holds with strict inequality, i.e. A_k - α_k^2 >0. Recall the matrix ^\[1]

    ^\[1] = [l]
    		                      []_K × K    []_K × K+1
    
    		                      [^⊤]_K+1 × K    [ - 1/τ^⊤ ]_K+1× K+1,

where the elements of the first column of  and - 1/τ^⊤ are zero. So we can discard this zero row and column and write the ^\[1,K+2] as

    ^\[1,K+2]= [l]
    		[]_K × K    [^\[1]]_K × K
    
    		[^\[1]⊤]_K × K    []_K× K,

with ≜ [ - 1/τ^⊤]^\[1].

Example of ^\[1,K+2] with K=3 is


    ^\[1,K+2] = ([                    u_0                      0                      0      1 + 2 α_1/2 L A_3              α_2/L A_3              α_3/L A_3;                      0                    u_1                      0                      0 2 α_1α_2 + A_1/2 L A_3           α_1α_3/L A_3;                      0                      0                    u_2                      0                      0 2 α_2α_3 + A_2/2 L A_3;      1 + 2 α_1/2 L A_3                      0                      0      A_1 - α_1^2/L A_3                      0                      0;              α_2/L A_3 2 α_1α_2 + A_1/2 L A_3                      0                      0        A_2-α_2^2/L A_3                      0;              α_3/L A_3           α_1α_3/L A_3 2 α_2α_3 + A_2/2 L A_3                      0                      0       A_3 - α_3^2/LA_3 ]).


We need to find  such that ^\[1,K+2]≽ 0. Here we notice that = ( (A_i-1-α_i-1^2)/(LA_K)) ≻ 0, hence, it is invertible with all elements strictly bigger than 0. We take the Shur complement of the block  and define the result matrix as  with

    _i,j = 
    		            u_i-1 - (A_i-1+2α_i-1α_i)^2/4 L A_K (A_i - α_i^2) - ∑_k=i^K-1α_i-1^2 α_k+1^2/LA_K (A_k+1-α_k+1^2)   i=j             
    
    		            -α_iα_j-1(A_i-1+2α_i-1α_i)/2LA_K(A_i - α_i^2) - ∑_k=i^K-1α_i-1α_j-1α_k+1^2/LA_K(A_k+1-α_k+1^2)   i>j 
    _ji   j>i
    	            .


Example with K = 3 is

0.8!
1

    = [ u_0 - (1+2 α_1 )^2/4L A_3 (A_1 - α_1^2)  - α_2^2/L A_3 (A_2 - α_2^2) - α_3^2/L A_3 (A_3 - α_3^2)                        - (A_1 +2 α_1α_2) α_2/4L A_3 (A_2 - α_2^2) - α_1α_3^2/L A_3 (A_3 - α_3^2)                                                       - (A_2 + 2α_2α_3) α_3/4L A_3 (A_3 - α_3^2);                        - (A_1 + 2α_1α_2) α_2/4L A_3 (A_2 - α_2^2) - α_1α_3^2/L A_3 (A_3 - α_3^2)                    u_1 - (A_1 + 2α_1α_2)^2/4L A_3 (A_2 - α_2^2) - α_1^2α_3^2/L A_3 (A_3 - α_3^2)                                                    - (A_2 + 2α_2α_3) α_1α_3/4L A_3 (A_3 - α_3^2);                                                       - (A_2 + 2α_2α_3) α_3/4L A_3 (A_3 - α_3^2)                                                     - (A_2 + 2α_2α_3) α_1α_3/L A_3 (A_3 - α_3^2)                                                     u_2 - (A_2 + 2α_2α_3)^2/4L A_3 (A_3 - α_3^2) ].




Then, ≽ 0 iff ^\[1,K+2]≽ 0. A nice property of  is that u_i only shows in the diagonal elements and all the off-diagonal elements are negative constants. It is known that a symmetric diagonally dominated matrix with non-negative diagonal entries is PSD, with the definition of the diagonally dominated matrix as

    _i,i≥∑_i≠ j_i,j.

Based on the special structure of , we can simplify the above inequality as

    _i,i≥ -∑_i≠ j_i,j.

Inserting the definition of _i,j, we can calculate the theoretical feasible solution for  as

    u_i ≥û_i ≜A_i(1+2α_i+1)(A_i+2α_iα_i+1)/4LA_K(A_i+1-α_i+1^2)+ ∑_k=i+1^K-1A_k(1+2α_k+1)α_iα_k+1/2L A_K(A_k+1- α_k+1^2).

The solution above also shows that when A_k - α_k^2 = 0, u = ∞, which recovers the unique feasible solution we proved in <ref>.



Surprisingly, this solution matches the numerical solution when b_i ≡ b. In <ref>, we follow the same settings as for <ref> with   equals to the values defined in <ref>. The error between the optimal solution and the analytical  given in (<ref>) is under 10^-3, validating our theoretical result.




Finally, in <ref>, we provide an analytical solution for  that together with the analytical solutions for  and _* (from the problem with exact oracle) provide a feasible solution to  (<ref>).


From the above discussion, we can summarize the results in the following lemma. The lemma provides a feasible solution to problem (<ref>) with iGOGM algorithm.

For <ref>, the following solution is a feasible solution to the problem (<ref>). ∀ i = 0,⋯,K-1
    
    τ   = L/4A_K,    v_i,i+1=A_i/A_K,      v_*,0=v_0,1,    v_*,i =v_i,i+1- v_i-1,i,    v_*,K = 1 - v_K-1,K,
    
         u_i    =  A_i(1+2α_i+1)(A_i+2α_iα_i+1)/4LA_K(A_i+1-α_i+1^2)+ ∑_k=i+1^K-1A_k(1+2α_k+1)α_iα_k+1/2L A_K(A_k+1- α_k+1^2).





 §.§ PEP-inspired convergence analysis of iGOGM

As explained by <cit.>, the feasible solution of PEP provides the proof of the convergence bound for the first-order algorithms. In the following theorem and the corresponding proof, we will show a PEP-inspired approach that can be verified without any knowledge of PEP or even SDP.

	The sequence generated by <ref> satisfies
	
    f(_K) - f_* - 1/2L f(_K)^2 	≤L_0 - _*^2/4A_K + ∑_k=0^K-1 u_k _k^2,

	with u_k = A_k(1+2α_k+1)(A_k+2α_kα_k+1)/4LA_K(A_k+1-α_k+1^2)+ ∑_i=k+1^K-1A_i(1+2α_i+1)α_kα_i+1/2L A_K(A_i+1- α_i+1^2).



	Below are the inequalities used in PEP primal constraints and their corresponding dual variables:

    f(_k+1) - f(_k) +  f(_k+1),_k - _k+1 + 1/2L f(_k)-  f(_k+1)^2 ≤ 0    ⋯     v_k,k+1 = A_k/A_K
        f(_k) - f_* +  f(_k),_* - _k + 1/2L f(_k)^2 ≤ 0                                             ⋯     v_*,k = A_k - A_k-1/A_K

	Multiply inequalities with their corresponding dual variable and sum up, we have the inequality
	
    A_K(f(_K) - f_* - 1/2L f(_K)^2 )                                                                                                                                                                  
    ≤    - ∑_k=0^K-1A_k f(_k+1),_k - _k+1  - ∑_k=0^K(A_k -A_k-1) f(_k),_* - _k
        - 1/2L∑_k=0^K-1A_k f(_k)-  f(_k+1)^2  - 1/2L∑_k=0^K(A_k-A_k-1)  f(_k)^2 - A_K/2L f(_K)^2                                     
    
    		=                    ∑_k=0^Kα_k  f(_k),_0 - _*  - 1/L∑_k=0^K-1∑_i=0^k-1(A_k+1θ_k+1,i - A_k θ_k,i) f(_k+1), f(_i) + _i
        - 1/L∑_k=0^K-1 A_k+1θ_k+1,k f(_k+1), f(_k) + _k  - 1/L∑_k=0^KA_k  f(_k)^2                                                              
        + 1/L∑_k=0^K-1A_k f(_k), f(_k+1)
    =a   ∑_k=0^Kα_k  f(_k),_0 - _* - 1/L∑_k=0^KA_k  f(_k)^2 + 1/L∑_k=0^K-1A_k f(_k), f(_k+1)
        - 2/L∑_k=0^K-1∑_i=0^kα_k+1α_i  f(_k+1), f(_i) + _i  - 1/L∑_k=0^K-1 A_k f(_k+1),  f(_k) + _k
    =i    - 1/ττ(_0-_*) - 1/2∑_k=0^Kα_k f(_k)^2 + τ_0 - _*^2 + ∑_k=0^K( α_k^2/4τ - A_k/L) f(_k)^2    
        + 1/2τ∑_k=0^K∑_i = 0^k-1α_kα_i f(_k),  f(_i) - 2/L∑_k=0^K-1∑_i=0^kα_k+1α_i  f(_k+1), f(_i) + _i
        - 1/L∑_k=0^K-1 A_k f(_k+1),  _k
    =b    -4/LL/4(_0-_*) - 1/2∑_k=0^Kα_k f(_k)^2 + L/4_0 - _*^2 - ∑_k=1^K A_k - α_k^2/L f(_k)^2                      
        - 2/L∑_k=0^K-1∑_i=0^kα_k+1α_i  f(_k+1), _i - 1/L∑_k=0^K-1 A_k f(_k+1),  _k,

	where (a) replaces θ_k,i with (<ref>), and (b) uses τ = L/4 and the fact that A_0 = α_0 = 1.	If _i =, then the above inequality gives the convergence rate of iGOGM with exact oracle.
	
    f(_K) - f_* - 1/2L f(_K)^2≤L_0 - ^*/4A_K


	Next, we will bound the error terms when _i ≠. For any k=0,⋯,K-1,
	
    -  A_k+1 - α_k+1^2/L f(_k+1)^2  - 2/L∑_i=0^kα_k+1α_i  f(_k+1), _i - 1/L A_k f(_k+1),  _k
    =ii   -1/L(A_k+1 - α_k+1^2)(A_k+1 - α_k+1^2) f(_k+1) + ∑_i=0^kα_k+1α_i _i + 1/2A_k _k^2                                                                
        + 1/L(A_k+1 - α_k+1^2)∑_i=0^k-1α_k+1^2α_i^2 _i^2 + 1/L(A_k+1 - α_k+1^2)∑_i=0^k∑_j=0^i-12 α_k+1^2α_iα_j _i,_j
        + 1/L(A_k+1 - α_k+1^2)∑_i=0^k-1α_k+1α_i A_k _i,_k + (2α_k+1α_k + A_k)^2/4L(A_k+1 - α_k+1^2)_k^2.

	Summing over k, and changing the summation indices, we have
	
    ∑_k=0^K-1(-  A_k+1 - α_k+1^2/L f(_k+1)^2  - 2/L∑_i=0^kα_k+1α_i  f(_k+1), _i - 1/L A_k f(_k+1),  _k)                                                        
    ≤   ∑_k=0^K-1(2α_k+1α_k + A_k)^2/4L(A_k+1 - α_k+1^2)_k^2 + ∑_k=0^K-1∑_j=k+1^K-1α_k^2 α_j+1^2/L(A_j+1-α_j+1^2)_k^2                                                                          
        + ∑_k=0^K-1∑_i=0^k-1α_k+1α_i (A_k+ 2α_k+1α_k)/L(A_k+1 - α_k+1^2)_k,_i + ∑_k=0^K-1∑_i=0^k-1∑_j=k+1^K-12α_j+1^2 α_k α_i/L(A_j+1 - α_j+1^2)_k,_i .


	For notation simplicity, we use P_k,i to replace the coefficients of _k,_i as
	
    P_k,k≜   (2α_k+1α_k + A_k)^2/4L(A_k+1 - α_k+1^2) +∑_j=k+1^K-1α_k^2 α_j+1^2/L(A_j+1-α_j+1^2),                                
    
    		P_k,i≜   α_k+1α_i (A_k + 2 α_k+1α_k )/L(A_k+1 - α_k+1^2) + ∑_j=k+1^K-12α_j+1^2 α_k α_i/L(A_j+1 - α_j+1^2).

	Hence, the above equation can be rewritten as
	
    ∑_k=0^K-1 P_k,k_k^2  + ∑_k=0^K-1∑_i=0^k-1 P_k,i_k,_i
    
    		=                          - ∑_k=0^K-1∑_i=0^k-11/2P_k,i_k - _i^2 + ∑_k=0^K-1∑_i=0^k-11/2 P_k,i( _k^2 +_i^2) + ∑_k=0^K-1P_k,k_k^2                                                     
    
    		=                          - ∑_k=0^K-1∑_i=0^k-11/2P_k,i_k - _i^2 + ∑_k=0^K-1∑_i=0^k-11/2 P_k,i_k^2  + ∑_k=0^K-1∑_i=k+1^K-11/2P_i,k_k^2  + ∑_k=0^K-1P_k,k_k^2 
    (iii)≤   ∑_k=0^K-1( P_k,k +1/2∑_i=0^k-1P_k,i + 1/2∑_i=k+1^K-1P_i,k) _k^2.

	With simple calculations, we can show
	
    P_k,k +1/2∑_i=0^k-1P_k,i + 1/2∑_i=k+1^K-1P_i,k
    
    		=                 A_k(1+2α_k+1)(A_k+2α_kα_k+1)/4L(A_k+1-α_k+1^2)+ ∑_i=k+1^K-1A_i(1+2α_i+1)α_kα_i+1/2L (A_i+1- α_i+1^2).

	Inserting these results into the original inequality, we get the final bound.


The key steps of the proof are the equalities (i), (ii), and inequality (iii). The equality (i) can be seen as the step of = 1/τ_τ_τ^⊤ + if one uses the Gram matrix  and standard basis vectors to rewrite this equality. The equality (ii) can be seen as the reformulation of the Shur complement with the block [A_k+1 - α_k+1^2/LA_K] of ^\[1,K+2]. The inequality (iii) is indeed the step of finding u_k such that the matrix  is diagonally dominant. This observation shows that the steps we use to find and verify the feasible solution (≽ 0) can be used to generate the theoretical proof. 

When _k≡ 0, the error term will be zero and the upper bound recovers the result of the exact GOGM. When _k≠ 0, this term is only determined by the total iteration number K and the stepsize α_k. It is independent of the initial condition _0 - _* and can be calculated when α_k is fixed.


	The inexactness assumption and corresponding analysis in <cit.> can also handle unbounded feasible regions. However, their inexactness assumption for convex problems requires the gradient estimate to be a subgradient, which is not a requirement in our work. Furthermore, their inexactness level along iterations has to be the same. Finally, in their FGM analysis, the accumulated error depends on the initial condition R. In contrast, our approach provides a pure separation of the convergence rate with the exact gradient oracle and the accumulated error. Our error term is independent of R and holds for any arbitrary error along iterations.




 §.§ PEP-inspired convergence analysis of iGFGM


Inspired by the approach for the analysis of iGOGM, we also find a convergence rate of inexact Generalized FGM (iGFGM) with the algorithm shown in <ref> in <ref>. Similar to the result of iGOGM, the bound of iGFGM is a summation of the exact convergence rate and accumulated error. 

    The sequence generated by <ref> satisfies
    
    f(_K) - f_* - 1/2L f(_K)^2 ≤L_0 - _*^2/2A_K + ∑_k=0^K-1u_k _k^2,

    with u_k = A_k^2 (1+α_k+1)/2L A_K(2 A_k+1 - α_k+1^2) + ∑_i=k+1^Kα_k A_i-1α_i (1+α_i)/2LA_K(2A_i - α_i^2).




§ EXPLOITING THE RATE-ERROR TRADEOFF AND FINDING THE OPTIMAL INEXACTNESS SCHEDULE

In <ref>, we provided the convergence bound of iGOGM with the explicit expression for the accumulated error. This section first looks into the tradeoff between the convergence rate and accumulated error through the stepsize with a fixed gradient inexactness level along iterations. Second, we aim to optimize the inexactness levels across iterations, for a given stepsize, so that the total effort to obtain the gradient oracles is minimized.



 §.§ Convergence rate and the accumulated error tradeoff

Considering the convergence bound L_0 - _*^2/4A_K + ∑_k=0^K-1u_k b_k^2, with A_k,u_k being functions of α_k, one can propose to minimize the bound with respect to α_k. However, we note that such a problem is a high-order polynomial optimization and its solution is intractable in general. 

Even if one is satisfied with the numerical solution to find the optimized algorithm, then it is preferable to solve the minimax problem (<ref>) following the relaxation approach in <cit.> which we also used in <ref>. 
This procedure generates the optimized stepsizes and the resulting convergence bound is better than the solution of the polynomial optimization problem as the derived theoretical u_i in <ref> is not necessarily the optimal solution.

Here we narrow down the iGOGM to iOGM-a, with the stepsize α_k = i+a/a and A_k = ∑_i=0^ki+a/a= (k+2a)(k+1)/2a. The condition A_k > α_k^2  holds for any k when a> 2. Note that when K ≫ a, the convergence rate is still (K^-2) in the exact case. Furthermore, <cit.> shows that the exact algorithm OGM-a has the asymptotic worst-case bound for function value as a/2K^-2 and the smallest gradient norm as a √(6)/2√(a-2)K^-1.5; it achieves the best performance of these two measures simultaneously when a=4. Both the simplicity of the stepsize and the good performances over function value and gradient norm indicate that OGM-a is a good selection under exact oracles and is worth further study for the inexact oracles too.


Replacing the stepsizes of OGM-a in <ref>, we get the coefficients of the error as

    u_k = 1/4La (K+1)(K+2a) ((a-2)k+2a^2-2a-2)((2a+8)k^4+(11a^2+48a+16)k^3 . 
    .+(20a^3+107a^2+70a+8)k^2  +(12a^4+104a^3+100a^2+24a)k+36a^4+48a^3+16a^2))       
    
    	+ ∑_i=k+1^K-1(k+a)(2i^3+(9a+4)i^2+(13a^2+13a+2)i+6a^3+10a^2+4a/La(K+1)(K+2a)((a-2)i +2a^2-2a-2),

which can be further simplified when a is assigned with a value.

Take a=4 as an example, which has both best convergence rates for function value and the gradient norm under exact oracle, the u_k is

    u_k =    16k^4+384k^3+3280k^2+11424k + 12544/16L(K+1)(K+8)(2k+22)  + ∑_i=k+1^K-1(k+4)(2i^3+40i^2+262i+560)/4L(K+8)(K+1)(2i+22)
    
    	=        (k+7)^2(k+8)(k+2)/2L(K+8)(K+1)(k+11) + ∑_i=k+1^K-1(k+4)(i+8)(i+7)(i+5)/4L(K+1)(K+8)(i+11).


Replacing k+11 with k+8 and i+11 with i+8 in the two denominators, we have a simpler expression with slightly bigger u_k as

    u_k = (k+7)^2(k+2)/2L(K+8)(K+1) + (k+4)(K-k-1)(2K^2+2Kk+35K+2k^2+37k+210)/24L (K+8)(K+1).

Assume b_k ≡ b, then the accumulated error is calculated as

    b^2 ∑_k=0^K-1 u_k =  b^2 K(12K^3+303K^2+2687K + 8758)/480L(K+8).

The above results indicate that the accumulated error is in (b^2K^3/L).


For any arbitrary a values, it is difficult to derive a closed form of u_k since the denominator can only be simplified as k+c structure when a is given as a specific number. Hence, we provide a numerical result of u_k for given K with different a in <ref>. For the same a and k, bigger K results in bigger u_k. For all a values, u_k first increases and then decreases. For a fixed k and K, bigger a results in smaller u_k values.





Based on this result, to reduce the accumulated error, one can consider increasing a. However, it will adversely slow down the convergence rate as A_K = (K+2a)(K+1)/2a and decreases when a is increased. Under this circumstance, we provide the comparison of convergence rate and accumulated error with different iteration numbers K, and a in <ref>, where τ = L/4A_K. The convergence rate τ is smaller when a is smaller while the accumulated error is bigger. When a = 4, ∑_k=0^K-1 u_k is between K^2 and K^3, better than our relaxed analysis result. Another interesting fact is that when K ≪ a (a=10^5 and a = 10^6), the convergence rate τ = (K^-1) and the accumulate error is also (K^1). If K ≪ a, we have α_k = 1, and the proposition below provides the convergence bound of this extreme scenario.


	With the stepsize  defined as α_k ≡ 1, the sequence generated by iGOGM (<ref>) satisfies
	
    f(_K) - f_* - 1/2L f(_K)^2 ≤L_0 - _*/4(K+1) + ∑_k=0^K-13(2K-k+1)/4L(K+1)_k^2.

	Furthermore, when _k≤b̅, ∀ k, the above inequality becomes
	
    f(_K) - f_* - 1/2L f(_K)^2 ≤L_0 - _*/4(K+1) + 9K/8Lb̅^2.












 §.§ Optimal inexactness schedule

Since the derived u_k is not constant, it will be beneficial to set b_k based on the value of u_k. Recall the definition of η from (<ref>). When r=1, we can aim to optimize over the total η-complexity, i.e., ∑_k=0^K-1η_k, so that the accumulated error will not deteriorate the convergence rate.
The problem is given as

    OPT-bmin_≥0     ∑_k=0^K-1η_k =  ∑_k=0^K-1 h^-1(b_k)                   
         ∑_k=0^K-1u_k b_k^2 ≤LR^2/4A_K.

This problem is a convex optimization when h is convex and decreasing over linear constraint and at its optimal solution, the constraint is binding. Hence, the inequality constraint can be replaced with the equality one. Let λ be the dual multiplier of the constraint, the Lagrangian function is

    (,λ) = ∑_k=0^K-1h^-1(b_k) + λ( ∑_k=0^K-1 u_k b_k^2 - LR^2/4A_k).

At the optimal solution ^* and λ^*, from the optimality conditions, we have

    _b_k h^-1(b_k^*) + 2λ^* b_k^* u_k        = 0 ,    k = 0,1⋯,K-1, 
    ∑_k=0^K-1u_k (b_k^*)^2 - LR^2/4A_K    = 0.

Solving the above equations, we have a closed-form solution of ^*.

In the following, we give the closed-form of ^* for two examples of h(η) <cit.>, which are two common structures of the inexact gradient oracles.

	Let h(η) = c_1η^-c_2 with c_1,c_2>0 to be two constants. The inverse function and its gradient are
	
    h^-1(b) = 	(c_1/b)^1/c_2  and   _b h^-1(b) = -c_1^1/c_2 b^-(1+c_2)/c_2/c_2.

	Furthermore, the optimal primal and dual solutions of (<ref>) are
	
    b_k^* = √(L)R /2√(A_K ∑_k=0^K-1 u_k^1/(1+2c_2))u_k^d/(1+2c_2) ,   λ^* = c^1/c_2/2c_2(LR^2/4A_K)^-(1+2c_2)/(2c_2)(∑_k=0^K-1 u_k^1/(1+2c_2))^(1+2c_2)/(2c_2).



With power law decay, (<ref>) is written as 

    - c^1/c_2 (b_k^*)^-(1+c_2)/c_2/c_2 + 2λ^* b_k^* u_k = 0,    k=0,1,⋯, K-1.

From this equation, b_k^* can be calculated as 

    b_k^* = (2λ^* u_k c_2/c_1^1/c_2)^- c_2/1+2c_2.

Inserting this equation into (<ref>), and solve for λ^* we get

    λ^* = c_1^1/c_2/2c_2(LR^2/4A_K)^-(1+2c_2)/(2c_2)(∑_k=0^K-1 u_k^1/(1+2c_2))^(1+2c_2)/(2c_2).

Replacing this solution into (<ref>), the final result can be derived.  




	Let h(η) = q_1 q_2^-η with q_1>0 and q_2>1 be two constants. Then
	
    h^-1(b) = log q_1 - log b/log q_2,   _b h^-1(b) = -1/b log q_2,

	and, the optimal primal and dual solutions are
	
    b_k^* =  √(LR^2/4KA_K u_k),   λ^* = 2KA_K/LR^2 log q_2.




    Similar to the proof of <ref>, from (<ref>) we have 
    
    u_k (b_k^*)^2 = 1/2λ^* log q_2.

    Inserting this into (<ref>),
    
    λ^* = 2K A_K/LR^2 log q_2.

    By the previous two equations, we can derive the final solution.


To quantify the improvement after the optimized inexactness level, we calculate the total η-complexity for iOGM-4 algorithm for the two structures of h(η) functions. We fix L=1 and R=1, then solve the constant b through 

    b̅^2 ∑_k=0^K-1≤LR^2/4A_K,
 and calculate the optimized b by the solutions we derived in <ref>.
In <ref>, we report the result for power law decay with four different values of c_2. Note that the η-complexity decreases with increased c_2. If we focus on the improvement of η-complexity for the optimized b_k, the improvement is in the same order as the complexity of constant b, which shows optimizing over b can decrease the η-complexity significantly. 

As for the exponential decay in <ref>, the improvement is not as large as that of the power law decay. Although the improvement is not as large as that of the power law decay, considering its absolute value, it still represents a considerable improvement.















§ CONCLUSIONS

In this paper, we analyze the Generalized Optimal Gradient Method (GOGM) with the inexact gradient oracle under the absolute error assumption. By utilizing the Performance Estimation Problem (PEP) analysis tool, a new upper bound for the convergence rate of GOGM is derived. This bound is composed of two independent components, the convergence rate obtained from the exact gradient oracle and the accumulated error resulting from the gradient inexactness. Such a bound demonstrates the effect of the inexactness and does not require boundedness of the feasible region. Furthermore, this bound is independent of the initial condition.

From this convergence bound, the optimal schedule to set oracle inexactness along iterations is proposed. Such a study aims to minimize the sampling or computational effort for gradient estimation while maintaining the convergence rate.




plainnat









§ SUPPLEMENTARY LEMMAS AND PROOFS 



 §.§ Proof of composition optimization

	
		Assume for any  and , f()≤ C_f and ()≤ C_g. Also, f() is Lipschitz smooth with constant L_f, _i[_i() - ()^2] ≤σ^2_ and [_i() - ()^2] ≤σ^2_. Assume _ and _ are properly chosen so 2C_f^2/b^2_σ_^2 + 2C_g^2 L_f^2/b^2_σ_^2 ≤ 1, then the error of the estimated gradient is bounded as 
    ()  f(()) - ()  f(())^2 ≤ b^2,

		with probability at least 1 - 2C_f^2/b^2_σ_^2 + 2C_g^2 L_f^2/b^2_σ_^2.
	
	
		By Young's inequality,
		
    ()  f(()) - ()  f(())^2                                                                                                  
    
    			=       ()  f(()) - ()  f(()) + ()  f(()) - ()  f(())^2            
    ≤    2()  f(()) - ()  f(())^2 + 2()  f(()) - ()  f(())^2 
    ≤    2  f(())^2 () - ()^2 + 2 ()^2  f(()) -  f (())^2                      
    ≤    2 C_f^2 () - ()^2 + 2 C_g^2 L_f^2 () - ()^2.

		Taking the expectation on both sides and with the bounded variance assumptions, we have
		
    [()  f(()) - ()  f(())^2] 
    ≤   2C_f^2/_σ_^2 + 2C_g^2 L_f^2/_σ_^2.

		By Markov's inequality, we have
		
    (()  f(()) - ()  f(())^2 ≥ϵ(2C_f^2/_σ_^2 + 2C_g^2 L_f^2/_σ_^2 ))        
    ≤   1/ϵ[()  f(()) - ()  f(())^2] / (2C_f^2/_σ_^2 + 2C_g^2 L_f^2/_σ_^2 ) 
    ≤   1/ϵ.

		Assuming ϵ > 1, i.e., the bound holds non-triviality, we can conclude  that
		
    ()  f(()) - ()  f(())^2 ≤ϵ(2C_f^2/_σ_^2 + 2C_g^2 L_f^2/_σ_^2 )

		holds with probability at least 1 - 1/ϵ.

	

 


 §.§ Convergence rate for inexact generalized FGM





    The sequence _k generated by <ref> is the same sequence as the one generated by iGFO (_k = _0 - 1/L∑_i=0^k-1θ_k,i f(_i)) with
    
    θ_k,i = α_i (A_k - A_i)+A_i/A_k.



    The equivalence of these two sequences is shown through induction. For k=0 and k=1, the equivalence holds trivially.
    Assume _i = _i for all i≤ k with k>1, then 
    
    _k+1 =    (1 - α_k+1/A_k+1) (_k - 1/Lf(_k) ) + α_k+1/A_k+1(_0 - 1/L∑_i=0^kα_i f(_i)) 
    
            =    (1 - α_k+1/A_k+1) (_0 - 1/L∑_i=0^k-1θ_k,if(_i) - 1/Lf(_k) ) + α_k+1/A_k+1(_0 - 1/L∑_i=0^kα_i f(_i)) 
    
            =    _0 - 1/LA_k+1 - α_k+1/A_k+1∑_i=0^k-1α_i(A_k - A_i) +A_i/A_kf(_i) - 1/Lα_k+1/A_k+1∑_i=0^k-1α_i f(_i) 
        - 1/L(1 - α_k+1/A_k+1  + α_k+1α_k/A_k+1)f(_k) 
    
            =    _0 - 1/L∑_i=0^k-1α_i(A_k - A_i) + A_i + (A_k+1 - A_k) α_i/A_k+1f(_i) - 1/Lα_k α_k+1 + A_k/A_k+1f(_k) 
    
            =    _0 - 1/L∑_i=0^k-1α_i(A_k+1 - A_i) + A_i/A_k+1 - 1/Lα_k (A_k+1 - A_k) + A_k/A_k+1f(_k)
    
            =    _0 - 1/L∑_i=0^kα_i(A_k+1 - A_i) + A_i/A_k+1f(_i) 
    
            =   _0 - 1/L∑_i=0^k θ_k+1,if(_i)
    
            =   _k+1.

    Hence the sequences _k and _k are equivalent to each other.





  
Proof of <ref>

Similar to the first step of the proof of <ref>,

    A_K(f(_K) - f_* - 1/2L f(_K)^2 )                                                                                                                                                                  
    ≤    - ∑_k=0^K-1A_k f(_k+1),_k - _k+1  - ∑_k=0^K(A_k -A_k-1) f(_k),_* - _k
        - 1/2L∑_k=0^K-1A_k f(_k)-  f(_k+1)^2  - 1/2L∑_k=0^K(A_k-A_k-1)  f(_k)^2 - A_K/2L f(_K)^2                                     
    
    		=                    ∑_k=0^Kα_k  f(_k),_0 - _*  - 1/L∑_k=0^K-1∑_i=0^k-1(A_k+1θ_k+1,i - A_k θ_k,i) f(_k+1), f(_i) + _i
        - 1/L∑_k=0^K-1 A_k+1θ_k+1,k f(_k+1), f(_k) + _k  - 1/L∑_k=0^KA_k  f(_k)^2                                                              
        + 1/L∑_k=0^K-1A_k f(_k), f(_k+1)
    
                        =    ∑_k=0^Kα_k  f(_k),_0 - _* - 1/L∑_k=0^KA_k  f(_k)^2 + 1/L∑_k=0^K-1A_k f(_k), f(_k+1)
        -  1/L∑_k=0^K-1∑_i=0^k-1α_k+1α_i  f(_k+1), f(_i) + _i  - 1/L∑_k=0^K-1 (α_k α_k+1 + A_k) f(_k+1),  f(_k) + _k
     
    		=     - 1/ττ(_0-_*) - 1/2∑_k=0^Kα_k f(_k)^2 + τ_0 - _*^2 + ∑_k=0^K( α_k^2/4τ - A_k/L) f(_k)^2    
        + 1/2τ∑_k=0^K∑_i = 0^k-1α_kα_i f(_k),  f(_i) - 1/L∑_k=0^K-1∑_i=0^k-1α_k+1α_i  f(_k+1), f(_i) + _i
        - 1/L∑_k=0^K-1 A_k f(_k+1),  _k - 1/L∑_k=0^K-1α_k α_k+1 f(_k+1), f(_k) + _k
    
    		=     -2/LL/2(_0-_*) - 1/2∑_k=0^Kα_k f(_k)^2 + L/2_0 - _*^2 - ∑_k=0^K 2 A_k - α_k^2/2L f(_k)^2                      
       + 1/L∑_k=1^K α_k α_k-1 f(_k),  f(_k-1) - 1/L∑_k=0^K-1∑_i=0^k-1α_k+1α_i  f(_k+1), _i
        - 1/L∑_k=0^K-1 A_k f(_k+1),  _k - 1/L∑_k=0^K-1α_k α_k+1 f(_k+1),  f(_k) + _k
    
                        =     -2/LL/2(_0-_*) - 1/2∑_k=0^Kα_k f(_k)^2 + L/2_0 - _*^2 - ∑_k=0^K 2 A_k - α_k^2/2L f(_k)^2                      
        - 1/L∑_k=0^K-1∑_i=0^k-1α_k+1α_i  f(_k+1), _i  - 1/L∑_k=0^K-1(α_k α_k+1+ A_k) f(_k+1),  _k
    
                        =    -2/LL/2(_0-_*) - 1/2∑_k=0^Kα_k f(_k)^2 + L/2_0 - _*^2 - 2A_0 - α_0^2/2L f(_0)^2 
        - ∑_k=0^K-11/2L(2A_k+1 - α_k+1^2)(2A_k+1 - α_k+1^2)  f(_k+1) + ∑_i=0^k-1α_k+1α_i _i + (α_k α_k+1 + A_k) _k^2 
        + ∑_k=0^K-1∑_i=0^k-1α_k+1^2 α_i^2/2L (2A_k+1 - α_k+1^2)_i^2 + ∑_k=0^K-1(α_k+1α_k + A_k)^2/2L (2A_k+1 - α_k+1^2)_k^2 
        + ∑_k=0^K-1∑_i=0^k-1∑_j=0^i-1α_k+1^2 α_i α_j/L (2A_k+1 - α_k+1^2)_i,_j + ∑_k=0^K-1∑_i=0^k-1α_k+1α_i (α_k α_k+1 + A_k)/L (2A_k+1 - α_k+1^2)_i,_k
    ≤   L/2_0 - _*^2  +  ∑_k=0^K-1(α_k+1α_k + A_k)^2/2L (2A_k+1 - α_k+1^2)_k^2  + ∑_k=0^K-1∑_j=k+1^K-1α_k^2 α_j+1^2/2L (2A_j+1 - α_j+1^2)_k^2 
        + ∑_k=0^K-1∑_i=0^k-1α_k+1α_i (α_k α_k+1 + A_k)/L (2A_k+1 - α_k+1^2)_k,_i +  ∑_k=0^K-1∑_i=0^k-1∑_j=k+1^K-1α_j+1^2 α_k α_i/L (2A_j+1 - α_j+1^2)_k,_i.
    
                Following the same steps in the proof of iGOGM, define 
                
    P̂_k,k≜   (α_k+1α_k + A_k)^2/2L(2 A_k+1 - α_k+1^2) +∑_j=k+1^K-1α_k^2 α_j+1^2/2L(2 A_j+1-α_j+1^2)
    P̂_k,i≜   α_k+1α_i ( α_k+1α_k + A_k )/L(2 A_k+1 - α_k+1^2) + ∑_j=k+1^K-1α_j+1^2 α_k α_i/L(2 A_j+1 - α_j+1^2),

                we can bound the error terms as
                
    ∑_k=0^K-1P̂_k,k_k^2 + ∑_k=0^K-1∑_i=0^k-1P̂_k,i_k, _i
    ≤   ∑_k=0^K-1(P̂_k,k + 1/2∑_i=0^k-1P̂_k,i + 1/2∑_i=k+1^K-1P̂_i,k)_k^2 
    
                       =    ∑_k=0^K-1( A_k^2 (1+α_k+1)/2L(2 A_k+1 - α_k+1^2) + ∑_i=k+1^Kα_k A_i-1α_i (1+α_i)/2L(2A_i - α_i^2)) _k^2.

                Dividing both sides of the inequality by A_K, the final result is obtained.





	

§ NUMERICAL SOLUTIONS FOR OPTIMIZED ALGORITHM WITH INEXACT ORACLE

	We provide the optimized stepsize for different K with L = 1 and R=1. We report three results, the first one is for =, the second one is for b_k^2 ≡b̅^2 = 0.01 and the third one takes b_k to be a decreasing sequence. Note that b̅^2 in the second scenario is set equal to the average value of b_k^2 in the third scenario.






	
	
	
	
	
	
	
	
	
	
	
	
	
	






	
	
	
	
	
	
	
	
	
	


	
	
	
	
	
	
	
	
	
	
	


	
	
	
	
	
	
	
	
	
	
	


	
	
	
	
	
	
	
	
	
	
	

	We can observe that the stepsize at each iteration with the exact gradient oracle remains the same while this is not the case with the inexact oracle. Even for the fixed inexactness level (second scenario), the optimized algorithm changes with the total iteration number. Another fact is that the optimized algorithm can identify a bad gradient estimate (see the first iteration of θ^b_k↓). It almost discards the first iteration since its inexactness is beyond some unknown threshold. Apart from the stepsize result, the optimized objective function value indicates that the initial condition R plays a role since τ for the three scenarios has different values. This is different from the exact case <cit.> as optimized stepsize is independent of R and L.
	






 Paper Title: Solving cluster moment relaxation with hierarchical matrix.


	[
    *
    August 5, 2024
==================

§ ABSTRACT 
		Convex relaxation methods are powerful tools for studying the lowest energy of many-body problems. By relaxing the representability conditions for marginals to a set of local constraints, along with a global semidefinite constraint, a polynomial-time solvable semidefinite program (SDP) that provides a lower bound for the energy can be derived. In this paper, we propose accelerating the solution of such an SDP relaxation by imposing a hierarchical structure on the positive semidefinite (PSD) primal and dual variables. Furthermore, these matrices can be updated efficiently using the algebra of the compressed representations within an augmented Lagrangian method. We achieve quadratic and even near-linear time per-iteration complexity. Through experimentation on the quantum transverse field Ising model, we showcase the capability of our approach to provide a sufficiently accurate lower bound for the exact ground-state energy.
	
	

§ INTRODUCTION

	
	Determining the lowest energy state of a many-body system is one of the most fundamental problems in science and engineering. This type of problem arises in the study of the Ising model <cit.>, graphical modeling <cit.>, sensor network localization <cit.>, and the structure from motion problem <cit.>, to name a few. In these problems, one is usually concerned with minimizing an energy function  ℰ. With the exception of some simple cases, the energy landscape of ℰ is plagued with spurious local minima. Without loss of generality, one can recast the problem of minimizing ℰ as another equivalent minimization problem <cit.>
	
    E_0 = inf_μ∈𝒫(𝒮)∫ℰ(x)μ(dx)

	on the space of measure over a set 𝒮, denoted as 𝒫(𝒮). By moving to the space of measure, one effectively obtains a linear optimization problem that circumvents the non-convexity issue in minimizing ℰ, at the expense of dealing with the high-dimensional measure μ. Generically, the solution of (<ref>) is an extreme point of 𝒫(𝒮) which is a Dirac measure, and the support of such measure gives a minimizer of ℰ. Such a view is adopted in <cit.> when devising a moment-based convex program for the case when ℰ is a low-degree polynomial. 
	
	
	There is an analogous problem in quantum many-body physics, where the ground-state energy minimization problem
	
    E_0 = min_ρ (Ĥρ),   s.t. ρ is positive semidefinite,  (ρ) = 1,

	is commonly solved (for example the quantum Ising and Hubbard model <cit.>). Here, ρ is a density operator subject to certain constraints, and Ĥ is a high-dimensional Hamiltonian operator capturing the interactions between d-sites <cit.>. The difficulty of solving (<ref>) is that the matrix ρ scales exponentially as the number of bodies grows.
	
	

 §.§ Prior works

	The issue with measure or density operator minimization is that minimizing these high-dimensional objects is prohibitively expensive due to the curse of dimensionality. Therefore, instead of working with the high-dimensional measure or density operator, approaches based on moments have been proposed to solve (<ref>) and (<ref>) without the curse of dimensionality. <cit.> proposes the use of moments to solve (<ref>):
	
    m_α := 𝔼_μ(x^α),   x = (x_1,…,x_d),  α = (α_1,…,α_d),

	where α∈{0∪ℕ}^d is a multi-index, and x^α := x_1^α_1… x_d^α_d. A convex relaxation is applied to  the space of {m_α}_α, where an outer approximation to the set of valid moments is given by a convex semidefinite program (SDP). Suppose we place the limit ∑_i α_i ≤ q on the degree of the moments, we have a convex problem in terms of d +q  q moments. Although by increasing q, the truncation threshold for the moments, one can improve the solution quality or even exactly recover the minimizer of the polynomial p <cit.>, for most practical situations, one can only use q=2 due to a large d. 
	
	
	An analogous fermionic quantum mechanical version of the moment-based relaxation is detailed in <cit.>. There, one deals with quantum moments of the form
	
    m_α := Tr(O^αρ),   O=(O_1,…,O_d)

	where O^α := O_1^α_1… O_d^α_d. Again, such a method can scale badly with d, constraining its application to small systems. 
	
	
	To improve the scaling of these methods, recently, cluster moments/marginals semidefinite programming relaxations have been proposed, both for minimizing classical <cit.> and quantum energies  <cit.>. The general idea is that one first clusters the variables/operators, and only forms higher-order moments for intra-cluster variables/operators. This significantly lowers the number of decision variables involved, resulting in O(d)× O(d) type scaling. 
	
	In this paper, we adopt a strategy similar to the variational embedding method <cit.>, where we try to determine local cluster moments and combine them through a global PSD constraint. The difference is that the local cluster density matrices are represented through their moments. In this case, the decision variable is a PSD moment matrix. The main point of this paper is to propose a method to accelerate the PSD optimization problem. Typically, the most computationally expensive step in such an optimization problem is the projection onto the PSD cone, which scales cubically. In <cit.>, translation invariance of the Hamiltonian is exploited to diagonalize the PSD matrix in the Fourier basis with linear time complexity. However, it is unclear how such computational scaling can be achieved for general systems. 
	
	
	

 §.§ Our contributions

	We propose a convex relaxation, the cluster moment relaxation, to solve the energy minimization problem in both classical and quantum settings. Furthermore, we introduce a specific form of hierarchical matrices, which differs from the conventional definition, to represent the primal and dual variables of the proposed semidefinite relaxation. 
	
	The key point is that the constraints of the proposed relaxation can be enforced efficiently using the algebra of hierarchical matrices. Within an augmented Lagrangian method, with a hierarchical dual PSD variable, the optimization can be carried out with quadratic per-iteration complexity. Additionally, if one assumes the primal moment matrix also takes the form of a hierarchical PSD matrix, near-linear per-iteration complexity can be achieved.
	
	
	

 §.§ Organizations

	In Section <ref>, we detail a  convex relaxation framework to solve energy minimization problems. In Section <ref>, we review the augmented Lagrangian method (ALM) for solving the proposed convex program. In Section <ref> and <ref>, we propose the use of hierarchical matrices to accelerate the ALM. In Section <ref>, we demonstrate the efficacy of the method for a quantum spin model.  
	
	

 §.§ Notations

	We use I_n to denote the identity matrix of size n × n. Additionally, we use 0_m× n to denote a zero matrix of size m × n, and when the context is clear, we will omit m and n. Furthermore, let ^n be the space of real symmetric matrices of size n × n, and let ^n_+ be the positive semidefinite matrices in ^n. Similarly, let ^̋n be the space of Hermitian matrices of size n × n, and let ^̋n_+ be the positive semidefinite matrices in ^̋n. For any matrix X in ^n or ^̋n, we may also use X ≽ 0 to denote that X is positive semidefinite.
	
	When discussing a matrix A, the notation A(p,q) refers to its (p,q)-th entry. In a block matrix A, A_ij represents its (i,j)-th block. We may occasionally use A_ij(p,q) to denote the (p,q)-th entry of the (i,j)-th block. For a complex-valued matrix A, (A) and (A) denote its real and imaginary parts, respectively. For a linear operator  on matrices or vectors, its adjoint is denoted by ^*. Lastly, for any positive integer N, we use [N] to denote the set {1,2,⋯,N-1,N}.
	
	
	
	

§ PROPOSED CONVEX RELAXATIONS

	
	While there are many versions of the cluster moments/marginals approach to obtain a convex relaxation of the energy minimization problems, in this paper, we examine a specific kind that only has equality constraints besides a global positive semidefinite constraint. As we shall see, this formulation can be optimized efficiently by our proposed method. The convex relaxation is constructed out of the following ingredients: 
	
		
  * Cluster basis: We first form monomials of variables/operators and cluster them into K different groups 𝒞_i⊂{0∪ℕ}^d, i∈ [K]. Each 𝒞_i is assumed to have C elements, i.e. | C_i | = C. These clusters of monomials are called the cluster basis:
		
			
  * Classical: 𝐯= [𝐯_i]_i∈[K], 𝐯_i = [x^α]_α∈𝒞_i.
			
  * Quantum: 𝐯 = [𝐯_i]_i∈[K], 𝐯_i = [O^α]_α∈𝒞_i.
		
		
  * Product cluster basis: We then take the cluster basis and form their products as follows:
		
			
  * Classical: 𝐯𝐯^*, and 𝐯_i𝐯_j^* = [x^αx^β^*]_α∈𝒞_i,β∈𝒞_j.
			
  * Quantum: 𝐯𝐯^*, and 𝐯_i𝐯_j^* = [O^αO^β^*]_α∈𝒞_i,β∈𝒞_j.
		
		
  * Intra-cluster relationship: The cluster basis and the product of the basis elements within the same cluster satisfy the following linear constraint for j ∈ [K]:
		
			
  * 𝒟_U(𝐯_j)+𝒟_L(𝐯_j^*)+𝒟(𝐯_j𝐯_j^*) = 𝐳. 𝐳 is a vector of scalars (operators) in the classical (quantum) case.
		
		
  * Inter-cluster relationship: The products of the basis elements, 𝐯_i 𝐯_j^* and 𝐯_j 𝐯_i^*, for i< j, i,j∈[K], satisfy the following relationship:
		
			
  * 𝒜_U(𝐯_i𝐯^*_j)+𝒜_L(𝐯_j𝐯^*_i) = 𝐰. 𝐰 is a vector of scalars (operators) in the classical (quantum) case.
		
	
	
	
	
	
	We now use these four ingredients to provide a convex relaxation for the         energy minimization problem in terms of the moment matrix M ∈ℂ^(CK+1) × (CK +1)
	
    Classical: M = 𝔼_μ([ 𝐯; 1 ][ 𝐯^*   1 ]),  Quantum: M = 𝔼_ρ([ 𝐯; I ][ 𝐯^*   I ]).

	Here, the expectations are taken entry-wise and defined as  𝔼_μ(·) := ∫ (·) μ(dx) and  𝔼_ρ(·) := ⟨·, ρ⟩. In what follows, for a matrix A∈ℂ^(CK+1) × (CK +1), we partition A into
	
    A := [   A^(2)   A^(1); A^(1)^*   A^(0) ],    A^(2)∈ℂ^CK × CK, A^(1)∈ℂ^CK × 1, M^(0)∈ℂ.

	We often also write 
	
    A^(2) = [A^(2)_ij]_i=1,j=1^K

	where each A^(2)_ij is a C× C block. 
	
	These ingredients give a set of necessary conditions on the moment matrix:
	
		
  * The inter-cluster relationship gives 𝒜_U(M^(2)_ij)+𝒜_L(M^(2)_ji) = w for i< j, i,j∈ [K], where w:=𝔼_μ,ρ(𝐰). 
		
  * The intra-cluster relationship gives 𝒟_U(M_j^(1))+𝒟_L(M_j^(1)^*)+ 𝒟(M_jj^(2)) = z for j∈[K], where z:=𝔼_μ,ρ(𝐳). Furthermore, M^(0) = 1, since 𝔼_μ,ρ(1) = 1. 
		
  * M≽ 0.
	
	For convenience, we assume w and z are real-valued vectors of sizes P_1 and P_2, respectively. With these necessary conditions, an energy minimization problem can be relaxed into a convex problem as follows:
	
    (P)       min_M Tr(JM)s.t.       Λ_ij∈ℝ^P_1 × 1:𝒜_U(M^(2)_ij)+𝒜_L(M^(2)_ji) = w,  i< j,  i,j∈ [K],       λ_j∈ℝ^P_2 × 1: 𝒟_U(M_j^(1))+𝒟_L(M_j^(1)^*)+ 𝒟(M_jj^(2)) = z,  j∈[K],       γ∈ℝ : M^(0) = 1,       S∈ℍ_+^CK+1: M∈ℍ_+^CK+1.

	
	
	
	

 §.§ Example of quantum energy minimization

	
	
	In physics, it is often the case that we have Hamiltonians with only pairwise interactions, i.e., the loss function in (<ref>) is equal to 
	
    ∑_1 ≤ i ≤ N(Ĥ_i ρ) + ∑_1 ≤ i<j ≤ N(Ĥ_ijρ).

	where Ĥ_i's and Ĥ_ij's are effectively some one-variable and two-variable operators.
	
	One example that we study in this paper is the quantum spin-1/2 system. The basic building blocks of these Hamiltonians are the Pauli matrices and the 2-dimensional identity matrix:
	
    σ^x = [ 0 1; 1 0 ],   σ^y = [  0 -i;  i  0 ],   σ^z = [  1  0;  0 -1 ],   
    	I_2 = [ 1 0; 0 1 ].

	These matrices form a basis for the real vector space of Hermitian operators on ℂ^2. Furthermore, let σ_i^x/y/z denote the operator obtained by tensoring σ^x/y/z on the i-th site with identities I_2 on all other sites, i.e.,
	
    σ_i^α := I_2^⊗ (i-1)⊗σ^α⊗ I_2^⊗ (N-i),  α∈{x,y,z},

	and let I:= I_2^⊗ N. In this case, all Ĥ_i and Ĥ_ij can be written as:
	
    Ĥ_i = ∑_α∈{x,y,z} a^α_i σ_i^α,     1≤ i ≤ N, 
       Ĥ_ij = ∑_α, β∈{x,y,z} a^αβ_ijσ_i^ασ_j^β,    1 ≤ i < j ≤ N,

	for some real constants {a_i^α}_i, α and {a_ij^αβ}_ij, αβ. 
	
	
	
	
	
	
	For example, the celebrated 1-D transverse field Ising (TFI) model <cit.>, a spin-1/2 quantum model, is defined by its Hamiltonian:
	
    Ĥ_TFI = -h ∑_i=1^N σ_i^x -∑_i=1^N σ_i^z σ_i+1^z,

	where the TFI model is assumed to have periodic boundary conditions, i.e., σ_N+1^x/y/z should be identified with σ_1^x/y/z, and h ∈ is a scalar parameter controlling the strength of the external magnetic field along the x axis. 
	
	
	

  §.§.§ Cluster moment relaxation for the TFI

	
	We go through the four ingredients needed to construct a cluster moment relaxation. For an N-spin system, we define 
	
    𝐯 := [𝐯_i]_i=1^N,   𝐯_i = [ σ_i^x^* σ_i^y^* σ_i^z^* ]^*,

	  and 𝐯 is a vector of operators of length 3N. In this case, K=N and C=3. Then we let M of size (3N+1) × (3N+1) be defined by
	
    M^(2)(i,j) =  (𝐯(i) 𝐯(j)^* ρ),    M^(1)(i) =  (𝐯(i)ρ).

	The properties of Pauli operators give rise to inter-cluster and intra-cluster constraints, which are summarized in the second column of Table <ref>.
	
	These relationships between operators then give constraints on the moment matrix
	
		
  * i< j,   i,j∈ [K],
            
    𝒜_U(M^(2)_ij)+𝒜_L(M^(2)_ji) = w ⇒(M^(2)_ij) = 0.
 
		
  * j∈[K],   𝒟_U(M_j^(1))+𝒟_L(M_j^(1)^*)+ 𝒟(M_jj^(2)) = z  ⇒  
		
    (M_j^(1))    =    0  
    		M^(2)_jj - diag({M^(2)_jj(k,k)}^3_k=1)    =   [            0  iM^(1)_j(3) -iM^(1)_j(2); -iM^(1)_j(3)            0  iM^(1)_j(1);  iM^(1)_j(2) -iM^(1)_j(1)            0 ] 
    		M^(2)_jj(k,k)    =    1, k=1,2,3.

	
	
	
	
	
	
	
	
	
	

§ STANDARD AUGMENTED LAGRANGIAN METHOD

	
	In this section, we first derive a standard ALM <cit.> to solve (<ref>), which serves as a foundation to discuss algorithms with reduced complexities in subsequent sections. The dual problem of (P) can be written as
	
    (D)      {Λ_ij}_i<j,{λ_j}_j,
    γ,S∈ℍ_+^CK+1max ∑_i<j≤ K⟨Λ_ij,  w ⟩+∑_j∈[K]⟨λ_j, z⟩ + γs.t.        J- ℱ^*({Λ_ij}_i<j,{λ_j}_j,γ)  = S

	where 
	
    ℱ^*({Λ_ij}_i<j,{λ_j}_j,γ)  := [        𝒟^*(λ_1)     𝒜^*_U(Λ_12)               ⋯     𝒜^*_U(Λ_1K)                      𝒟_U^*(λ_1);     𝒜^*_L(Λ_12)        𝒟^*(λ_2)               ⋱                                               ⋮;               ⋮               ⋱               ⋱ 𝒜^*_U(Λ_(K-1)K)                                ;     𝒜^*_L(Λ_1K)               ⋯ 𝒜^*_L(Λ_(K-1)K)        𝒟^*(λ_K)                      𝒟_U^*(λ_K);      𝒟_L^*(λ_1)               ⋯                      𝒟_L^*(λ_K)                               γ ].

	The dual problem (D) admits an augmented Lagrangian of the form
	
    L_σ (S,{Λ_ij}_i<j, {λ_j}_j,γ; M) := -∑_i<j≤ K⟨Λ_ij, w ⟩ -∑_j∈[K]⟨λ_j, z⟩ - γ +  σ/2J - 
    	ℱ^*({Λ_ij}_i<j,{λ_j}_j,γ)  - S-σ^-1 M_F^2 - M_F^2/2σ

	with a penalty parameter σ > 0. Then, the ALM algorithm for solving (D) is summarized in Algorithm <ref>. 
	
	In Algorithm <ref>, it is worth noting that the primal variable M always satisfies the linear constraint in (P) during the updates. This arises from the property that ALM always provides dual-feasible variables <cit.> (where (P) is the dual problem of (D)). 
	
	In the next subsection, we detail how to perform the minimization sub-problem in Algorithm <ref>. Indeed, since {Λ_ij}_i<j, {λ_j}_j, γ are unconstrained, one can eliminate them from the minimization subproblem, leaving us with a minimization problem only in terms of S. 
	
	
	
	

 §.§ Minimization subproblem in Algorithm <ref>
 
	
	The subproblem in Step <ref> of Algorithm <ref> is a joint minimization problem involving both S and the rest of the dual variables. However, since the minimization subproblem is unconstrained with respect to {Λ_ij}_i<j, {λ_j}_j, γ, one can first eliminate them using the first-order optimality condition and express them in terms of S (and M). We are then left with a minimization subproblem that involves only S. More precisely, 
	
    Λ_ij(S^(2)_ij,S^(2)_ji) = (𝒜_U 𝒜_U^*+ 𝒜_L 𝒜_L^*)^-1( 𝒜_U (J^(2)_ij -S^(2)_ij - σ^-1M^(2)_ij )+ 𝒜_L(J^(2)_ji -S^(2)_ji - σ^-1 M^(2)_ji ) + w/σ),   i<j,  i,j∈ [K],

	
    λ_j(S^(2)_jj,S^(1)_j,S^(1)_j^*) = ( 𝒟𝒟^*+𝒟_U 𝒟_U^* + 𝒟_L 𝒟_L^*)^-1(𝒟(J^(2)_jj-S^(2)_jj -σ^-1M^(2)_jj)+
    𝒟_U(J^(1)_j-S^(1)_j-σ^-1M^(1)_j)+𝒟_L(J^(1)_j^*-S^(1)_j^*-σ^-1M^(1)_j^*)+ z/σ),  i∈ [K],

	and
	
    γ(S^(0)) = J^(0)-S^(0) -σ^-1M^(0) +σ^-1.

	
	
	We now substitute the above dual variables into L_σ:
	
    L^S_σ(S; M) := L_σ(S,{Λ_ij(S^(2)_ij,S^(2)_ji)}_i<j, {λ_j(S^(2)_jj,S^(1)_j,S^(1)_j^*)}_j,γ(S^(0)); M)

	and instead of minimizing L_σ (S,{Λ_ij}_i<j, {λ_j}_j,γ; M), we can minimize L^S_σ(S; M) as we have just eliminated all other dual variables from L_σ (S,{Λ_ij}_i<j, {λ_j}_j,γ; M) except S using first-order optimality conditions. The ALM solely in S is summarized in Algorithm <ref>.
	
	
	
	
	

§ ALM WITH HIERARCHICAL DUAL PSD VARIABLE

	As shown previously, the ALM method (Algorithm <ref>) requires minimizing (<ref>), which is an optimization problem over the PSD cone. Typically, this requires  computing the projection onto the PSD cone, and the computational complexity of this projection is cubic, making it impractical for large-scale problems. 
	
	In <cit.>, the authors propose a solution by introducing a change of variables for the PSD variable in the form of S = RR^*. This strategy effectively circumvents the difficult PSD constraint in the optimization problem, converting it into an unconstrained optimization problem. Moreover, when low-rank solutions of the SDP problem exist, the number of columns of R is chosen minimally, enabling the development of an efficient algorithm using the limited-memory BFGS algorithm. However, experiments conducted using the package <cit.> to directly solve either the primal problem (P) or the dual problem (D) for the TFI model (Section <ref>) indicate a linear increase in the rank in N of both the primal PSD variable M and the dual PSD variable S. Hence, employing a vanilla low-rank decomposition of M or S to solve either the primal or dual problem via the limited-memory BFGS algorithm is unlikely to yield substantial reductions in computation time. 
	
	In this section, we propose a structure for the dual variable S in Algorithm <ref> that allows us to perform the ALM updates with reduced time complexity. For a 1-D TFI model (<ref>) with a small system size N=128 (CK = 384) and an external magnetic field strength parameter h=1, we solve (D) using an ADMM-type method with direct projection onto the PSD cone. We show a heatmap of the PSD variable S in Figure <ref>. We can see from the plot that even though S is not low-rank, it is nearly zero except on a few diagonals near the main diagonal. This observation inspired us to represent the dual PSD variable S using a hierarchical low-rank matrix <cit.>, resulting in an algorithm with quadratic scaling of the per-iteration time complexity. We emphasize that the matrix structure we propose is different from the typical hierarchical structure in the literature in order to encode the PSDness of the variable S. For simplicity, we assume that K is a power of 2 in this and the following sections.
	
	
	
	
	

 §.§ Approximating S with a hierarchical matrix
 
	
	Let S denote the solution of the dual problem (D) with size (CK+1)× (CK+1). We first examine the S^(2) block of the matrix S (as defined in (<ref>)). While our objective is to use a hierarchical matrix to represent S^(2), we must also ensure that S^(2) remains a positive semidefinite matrix. To this end, we use m levels of hierarchy to characterize S^(2). For the l-th level, we define a block diagonal matrix with n_l diagonal blocks, where each block is of size c_l× c_l. Furthermore, we want the block diagonal matrix to be positive semidefinite. Therefore, for the l-th level, we form a matrix:
	
    ^(2)(y^(l)) :=
    	[     y^(l)_1 (y^(l)_1)^*;                             y^(l)_2 (y^(l)_2)^*;                                                                       ⋱;                                                                         y^(l)_n_l (y^(l)_n_l)^* ]∈_̋+^CK,    y^(l)_j ∈^c_l × r_l,    1 ≤ j ≤ n_l.

	This naturally requires n_l c_l = CK, since the size of the matrix S^(2) is CK× CK. Then, as an approximation to S^(2), we define
	
    ^(2)(y) := ^(2)(y^(1))+^(2)(y^(2))+⋯+^(2)(y^(m))
    
    	:=y^(1) (y^(1))^* + 
    	[ y^(2)_1 (y^(2)_1)^*                   0;                   0 y^(2)_2 (y^(2)_2)^* ]
    
    	+⋯+[     y^(m)_1 (y^(m)_1)^*                       0                       ⋯                       0;                       0     y^(m)_2 (y^(m)_2)^*                                               0;                       ⋮                                               ⋱                       ⋮;                       0                       ⋯                       0 y^(m)_n_m (y^(m)_n_m)^*;                         ],

	with 
	
    y := {y^(1), y^(2), ⋯, y^(m)},   and   y^(l) := 
    	[   y_1^(l);   y_2^(l);         ⋮; y_n_l^(l) ]∈^CK × r_l.

	Here, we assume n_l = 2^l-1 for 1 ≤ l ≤ m and n_m = 2^m-1 < K. Our proposal involves representing S^(2) with ^(2)(y), where the number of levels m and the number of columns for each level r_l for 1 ≤ l ≤ m are determined based on the desired accuracy of the algorithm. 
	
	
	
	To approximate the full matrix S, which has one extra row and one extra column compared to S^(2), we just pad ^(2)(y) with an extra row and column of zeros, plus a low-rank matrix:
	
    (y,t) := 
    	[  ^(2)(y) 0_CK × 1; 0_1 × CK  0_1 × 1 ]+tt^* ≈ S.

	Here, y = {y^(1), ⋯, y^(m)} is a set of matrices with y^(l)∈^CK × r_l, and t ∈^CK+1.
	
	
	
	
	

  §.§.§ Validity of the hierarchical matrix representation for S
 
	We now investigate the validity of representing S by a hierarchical matrix (y,t). For this purpose, we use the TFI model as a test problem and investigate the relationship between the system size N, the number of levels m needed, and the number of columns needed for each y^(l). We first solved the dual problem (D) with the Hamiltonian specified in (<ref>), with system sizes N=64, 128, 256, and an external magnetic field strength parameter h=1. For these problem sizes, we solved for S in the full PSD cone rather easily with an accuracy of 10^-4. Then, to see how well these PSD S can be approximated by a hierarchical structure, we fitted the resulting PSD dual variable S with the structure outlined in (<ref>). Let S^* denote the approximate solution to the dual problem (D). Additionally, let y,t be complex-valued parameters that parameterize a hierarchical matrix of the corresponding size, with number of levels m = 3, 4, 5 respectively for N= 64, 128, 256. The number of columns for each level was fixed at 20 for all system sizes. We solved the following optimization problem with system sizes N=64,128,256 for the variables y and t:
	
    min_y, z  Err_S^2 := (y,t)-S^*_F^2/S^*_F^2,

	by running the limited-memory BFGS algorithm provided in the toolbox <cit.> for 1000 iterations. The approximation errors are presented in Table <ref>. From the table, we can see that even with fixed r_l = 20 for all 1 ≤ l ≤ m, we obtained similar accuracy for different system sizes. Therefore, we assume one can use a fixed rank approximation in the hierarchical matrix even for large system sizes.
	
	
	
	
	

 §.§ Update rule with a hierarchically structured variable S
 
	
	
	By substituting the PSD variable S in (<ref>) with a data-sparse hierarchical PSD representation, we can eliminate the challenging PSD constraint on S, hence significantly reducing the per-iteration computational costs. With this hierarchical representation of S, when performing Algorithm <ref> where one needs to minimize the variable-reduced augmented Lagrangian function L^S_σ, we replace S with the hierarchical matrix (y,t) defined in (<ref>). We remind the reader again y is a collection of matrices y := {y^(1), y^(2), ⋯, y^(m)} for y^(l)∈^CK × r_l and t ∈^(CK+1) × 1, with pre-specified number of levels m and number of columns r_1, ⋯, r_m. The resulting algorithm is outlined in Algorithm <ref>.
	
	
	
	We now conduct a complexity analysis of Algorithm <ref>, examining its computational scaling step by step.  To remind the readers, the Lagrangian  L_σ can be split into three different terms:
	
    -∑_i<j⟨Λ_ij, w ⟩ + ∑_i< jσ/2( J^(2)_ij - 𝒜^*_U(Λ_ij) - S^(2)_ij - σ^-1 M^(2)_ij_F^2 + J^(2)_ji - 𝒜^*_L (Λ_ij) - S^(2)_ji - σ^-1 M^(2)_ji_F^2  ),

	
    -∑_j⟨λ_j,z⟩  + σ/2∑_j J^(2)_jj - 𝒟^*(λ_j)- S^(2)_jj - σ^-1 M^(2)_jj_F^2  
    
    	+ σ/2∑_j  J^(1)_j - 𝒟^*_U(λ_j)- S^(1)_j - σ^-1 M^(1)_j_F^2 +σ/2∑_j  J^(1)_j^* - 𝒟^*_L(λ_j)- S^(1)_j^* - σ^-1M^(1)_j^* _F^2,

	and
	
    -γ  + σ/2| J^(0) -γ- S^(0) - σ^-1 M^(0)|^2,

	and L^S_σ is obtained by substituting (<ref>), (<ref>) and (<ref>) into (<ref>), (<ref>) and (<ref>) respectively.
	
	
	
	
	Suppose we use gradient-based methods such as the limited-memory BFGS algorithm for Step 2 in Algorithm <ref>. Since the complexity of computing the gradient is asymptotically the same as the complexity of evaluating the loss function <cit.>, we simply analyze the computational cost of evaluating L^S_σ. The key operations in evaluating the loss L^S_σ consist of evaluating the terms (<ref>), (<ref>) and (<ref>). Since (<ref>) and (<ref>) have O(CK) terms and O(1) terms respectively, the computational complexity for these terms is negligible compared to (<ref>), which has O(C^2 K^2) terms. When substituting (<ref>) into (<ref>), we have 
	
    -∑_i<j⟨Λ_ij(S^(2)_ij,S^(2)_ji), w ⟩ + ∑_i< jσ/2( J^(2)_ij - 𝒜^*_U(Λ_ij(S^(2)_ij,S^(2)_ji)) - S^(2)_ij - σ^-1 M^(2)_ij_F^2 
     + J^(2)_ji - 𝒜^*_L(Λ_ij(S^(2)_ij,S^(2)_ji) ) - S^(2)_ji - σ^-1 M^(2)_ji_F^2  ).

	The question now is, with S = H(y,t), meaning S^(2) is the sum of H^(2)(y) plus a rank-one correction term, what is the complexity of evaluating (<ref>). As the rank-one correction term can also be assimilated into the hierarchical structure, by adding one more column to the first hierarchy y^(1), we can, without loss of generality, assume S^(2) is equal to a hierarchical matrix. From the construction of w in the linear constraints in (<ref>), we know w is a matrix of all zeros, so the term ∑_i<j⟨Λ_ij(S^(2)_ij,S^(2)_ji), w ⟩ disappears. Additionally, the remaining two terms J^(2)_ij - 𝒜^*_U(Λ_ij(S^(2)_ij,S^(2)_ji)) - S^(2)_ij - σ^-1 M^(2)_ij_F^2 and J^(2)_ji - 𝒜^*_L (Λ_ji(S^(2)_ij,S^(2)_ji) ) - S^(2)_ji -σ^-1 M^(2)_ji_F^2 take similar forms, so it suffices to just analyze the complexity of the first term.
 
    As w is the zero matrix, from the definitions of the operator 𝒜^*_U and the optimal Λ_ij(S^(2)_ij,S^(2)_ji) in (<ref>) and (<ref>), it is apparent that ( J^(2)_ij - 𝒜^*_U(Λ_ij(S^(2)_ij,S^(2)_ji)) - S^(2)_ij - σ^-1 M^(2)_ij) is a linear function of J_ij^(2), J_ji^(2), S_ij^(2), S_ji^(2), M_ij^(2) and M_ji^(2). Therefore, we need to evaluate a term of the form 
    ∑_i < j^<_S(S_ij^(2))+^>_S(S_ji^(2))+^<_J(J_ij^(2))+^>_J(J_ji^(2))+^<_M(M_ij^(2))+^>_M(M_ji^(2))_F^2,
 
    where A^<_S, A^<_J, A^<_M, A^>_S, A^>_J, A^>_M: ^C × C→^C × C are all linear operators. The complexity of such an evaluation, up to a constant independent of S^(2), is dominated by the inner products between these terms, and Lemma <ref> gives the complexity of these operations.
    
    It is important to note that although w is assumed to be the zero matrix in our proposed relaxation, the complexity analysis can be conducted in a similar way as long as w is a constant matrix. To illustrate this, let W ∈^CK × CK be a matrix in which every C × C block matrix is equal to w. Consequently, W is a matrix with rank less than or equal to C, and thus is also a hierarchical matrix. Therefore, we can apply Lemma <ref> to analyze the complexity of the inner product between S^(2) and W.
	
	
		Assuming C is a constant, the complexity of
		
    ∑_i<j≤ K(B_ij𝒜(B'_ij))

		for some matrices B,B'∈ℂ^CK× CK and some linear operator 𝒜:ℂ^C× C→ℂ^C× C is:
		
			
  * O(K m^2 r^2) if both B and B' are hierarchical matrices in the form of (<ref>) with m levels and each level having rank r.
			
  * O(K^2 r) if B is an arbitrary matrix and B' is a hierarchical matrix.  
			
  * O(Kmr) if B is sparse with O(K) non-zero entries and B' is a hierarchical matrix.
		
	
	
		For any linear operator 𝒜:ℂ^C× C→ℂ^C× C, ∑_i<j≤ K(B_ij(B'_ij)) can be written as 
        
    ∑_i<j
    i,j∈ [K]∑_k,κ,k',κ'∈ [C] D_(k,k',κ,κ')B_ij(k,κ)B'_ij(k',κ')
 
        for a 4-tensor D_ whose values depend on . If both B and B' take the form in (<ref>), the sum 
 
        ∑_i<j
i,j∈ [K]B_ij(k,κ)B'_ij(k',κ') can be computed with O(Km^2 r^2) complexity (see Proposition <ref>). Then the sum ∑_k,κ,k',κ'∈ [C] contributes a factor of C^4, giving a total complexity of  O(C^4 K m^2 r^2). We ignore the factor C^4 as C is assumed to be a constant. 
		
		The complexity of the second statement can be shown in a similar way. If B' is a hierarchical matrix, using Proposition <ref>, one can show that the sum ∑_i<j
i,j∈ [K]B_ij(k,κ)B'_ij(k',κ') can be computed with O(K^2 r) complexity. The summation ∑_k,κ,k',κ'∈ [C] further contributes a factor of C^4, which is again ignored as C is assumed to be a constant.
		
		The last statement is a direct consequence of B being a sparse matrix. 
	
	
	Based on this lemma, the complexity of ∑_i < j⟨^<_S(S_ij^(2)), ^<_J(J_ij^(2))+^>_J(J_ji^(2)) ⟩ and 
 ∑_i < j⟨^>_S(S_ji^(2)), ^<_J(J_ij^(2))+^>_J(J_ji^(2)) ⟩ is O(Kmr), due to the fact that J is sparse with O(K) non-zero entries. The complexity of ∑_i<j^<_S(S_ij^(2))+^>_S(S_ji^(2))_F^2 is O(Km^2r^2), and the complexity of ∑_i < j⟨^<_S(S_ij^(2)), ^<_M(M_ij^(2))+^>_M(M_ji^(2)) ⟩ and ∑_i < j⟨^>_S(S_ji^(2)), ^<_M(M_ij^(2))+^>_M(M_ji^(2)) ⟩ is O(K^2 r). Assuming r is a constant and m = log_2(CK), the computational cost is dominated by the inner products between blocks of S and M as it has a quadratic growth with respect to K. This stems from the fact that M^(2) is an unstructured matrix. At this point, we have successfully reduced the per-iteration cost of vanilla ALM (Algorithm <ref>) from O(K^3) to O(K^2) by assuming S^(2) takes the form of a hierarchical positive semidefinite matrix in Algorithm <ref>. 
	
	
	
	
	
	
	
	
	

§ ALM WITH HIERARCHICAL PRIMAL AND DUAL PSD VARIABLES

	
	As analyzed in Section <ref>, Algorithm <ref> has an O(K^2) per-iteration complexity due to the lack of structure in the primal variable M. While this is already a speed-up compared to a vanilla ALM with cubic complexity, we propose replacing the direct update rule in  Step <ref> of Algorithm <ref> with a projection step that compresses M in order to obtain a nearly linear per-iteration cost. 
	
	Before discussing how to form a compressed representation for the primal variable, we rewrite the primal variable update in Algorithm <ref> as the solution to the following problem:
	
    _M̃  M̃ - (M + σ(S-J+ ℱ^*({Λ_ij}_i<j,{λ_j}_j,γ))))_F^2, 
       s.t.(M̃) = b.

	Here, S is represented hierarchically as S = (y,t), with  {Λ_ij}_i<j, {λ_j}_j, γ being defined in (<ref>), (<ref>) and (<ref>). The linear constraint (M̃)=b summarizes all linear constraints on the matrix M. Although this constraint in (<ref>) may seem redundant since the primal variable always satisfies the linear constraint in (P) during the updates, it becomes crucial when replacing M̃ with a compressed representation, as the constraint will not be naturally satisfied.
	
	To this end, we aim to find an affine function g: ^̋CK+1→^̋CK+1 such that 𝒜(g(A)) = b for any A ∈^̋CK+1. A choice of g is such that g(A) = 𝒫_null(𝒜)(A)+𝒜^†(b). Here, 𝒜^† denotes the pseudo-inverse of the operator 𝒜. Moreover, the linear constraints in cluster moment relaxation are local in nature and only constrain local blocks. This means one can choose g with
	
    g(A) := 
    	[    g^(1)(A^(2)_11,A^(1)_1)                                                                                                               g̃^(1)(A^(2)_11,A^(1)_1);                               g^(1)(A^(2)_22,A^(1)_2)                                                                                                           ⋮;                                                                                ⋱                                                                                 ;                                                                                     g^(1)(A^(2)_KK,A^(1)_K)                              g̃^(1)(A^(2)_KK,A^(1)_K); g̃^(1)(A^(2)_11,A^(1)_1)^*                          ⋯                            g̃^(1)(A^(2)_KK,A^(1)_K)^*                                          g^(0)(A^(0)) ] +
    	
    [                             0_C× C           g^(2)(A^(2)_12,A^(2)_21)                                  ⋯           g^(2)(A^(2)_1K,A^(2)_K1)                                                                0_C× 1;         g^(2)(A^(2)_12,A^(2)_21)^*                             0_C× C                                  ⋱                                                                                                        ⋮;                                  ⋮                                  ⋱                                  ⋱   g^(2)(A^(2)_(K-1)K,A^(2)_K(K-1))                                                                      ;         g^(2)(A^(2)_1K,A^(2)_K1)^*                                  ⋯ g^(2)(A^(2)_(K-1)K,A^(2)_K(K-1))^*                             0_C× C                                                                0_C× 1;                             0_1× C                                  ⋯                                                                0_1× C                                                                     0 ].

	where g^(2):ℂ^C× C×ℂ^C× C→ℂ^C× C, g^(1):ℂ^C× C×ℂ^C× 1→ℂ^C× C, g̃^(1):ℂ^C× C×ℂ^C× 1→ℂ^C× 1 and g^(0):ℂ→ℂ are all affine transformations that depend on the constant vector b. For example, in order to satisfy the linear constraints of the TFI model of N sites (Section <ref>), we let g: ^̋3N+1→^̋3N+1 be:
	
    g(A) = I_3N+1 + 
    	([         0  A^(2)_12         ⋯  A^(2)_1N   A^(1)_1;  A^(2)_21         0         ⋯  A^(2)_2N   A^(1)_2;         ⋮         ⋮         ⋱         ⋮         ⋮;  A^(2)_N1  A^(2)_N2         ⋯         0   A^(1)_N; A^(1)_1^* A^(1)_2^*         ⋯ A^(1)_N^*         0 ]) 
    
    	+2[ ^*((A^(1)_1))                                          ;                           ⋱                            ;                             ^*((A^(1)_N))              ;                                                       0 ],

	with 
    (D): = 
    	[ D(2,3); D(3,1); D(1,2) ],   : ^̋3 →^3.

	This reparameterization of primal-feasible matrices allows us to rewrite (<ref>) in an equivalent form:
	
    _A∈^̋CK+1  g(A)- (M + σ(S-J+ ℱ^*({Λ_ij}_i<j,{λ_j}_j,γ)) )_F^2.

	Our final step is to approximate A ∈^̋CK+1 with a hierarchical matrix to reduce the cost of the loss function evaluation:
	
    _x,v,A  g(A)- (M + σ(S-J+ ℱ^*({Λ_ij}_i<j,{λ_j}_j,γ)))_F^2 
       s.t. A = (x,v),

	where x={x^(i)∈^CK × r_i}_1 ≤ i ≤ m and v ∈^(CK+1) × 1 for some pre-specified number of levels m, and number of columns r_1, ⋯, r_m. In other words, we propose a hierarchical representation for the parameterization of any primal-feasible variable M. This allows efficient evaluation of the loss function in (<ref>). If the current guess M is represented as M = g((x, v)) for some x and v, and S = (y,t), then the main computational task in evaluating the loss function (<ref>) can be reduced.
	
	
	
	
	We now introduce Algorithm <ref>, which utilizes a hierarchical representation for both the primal and the dual PSD variables.
	
	We highlight that M and S are never explicitly formed as (CK+1)× (CK+1) matrices to ensure efficient computations. The per-iteration computational complexity of Algorithm  <ref> can be analyzed following the approach in Section  <ref> for Algorithm <ref>. We assume the hierarchical representation of the primal and dual PSD variables has the same number of levels m, with a constant number of columns r for each level. As before, in Step <ref> and Step <ref> of the algorithm, the evaluation of the loss consists of computations in Lemma <ref>. When we replace both primal and dual variables with hierarchical matrices, the second scenario in Lemma <ref> is eliminated, and we achieve a near-linear complexity of O(Km^2r^2) with an r that does not grow with K and m = O(log_2(CK)).
	
	

 §.§ Validity of the hierarchical matrix representation for M

        In this section, we investigate the validity of representing M by a hierarchical matrix (x,v), using the same method as in section <ref> for the dual PSD variable S. Instead of fitting the dual PSD variable S, we fitted the primal PSD variable M resulting from solving (D) with the Hamiltonian specified in (<ref>), for system sizes N=64, 128, 256, an external magnetic field strength parameter h=1 and an accuracy of 10^-4. Let M^* be the approximate solution. Let x, v be complex-valued parameters that parameterize a hierarchical matrix of the corresponding
        size, with the number of levels m = 3, 4, 5 respectively for N = 64, 128, 256. The number of columns for each level was fixed at 20 for all system sizes. We solved the following optimization problem
        with system sizes N = 64, 128, 256 for the variables x and t:

        
    min_x,v  Err_M^2 := (x,v)-M^*_F^2/M^*_F^2,

	by running the limited-memory BFGS algorithm provided in the toolbox <cit.> for 1000 iterations. The approximation errors are presented in Table <ref>. The experiment results indicate that as the system size increases, the relative error of the fitted primal PSD variable also tends to increase. This likely contributes to the failure of Algorithm <ref> for  large system sizes, such as N=4096 (see Section <ref> for further details on the experiments). 
	
	
	
        
	
	

§ NUMERICAL EXPERIMENTS

	In this section, we present numerical experiments for the 1-D TFI model using Algorithms <ref> and <ref>, with system sizes N ∈{64,128,256,512,1024,2048,4096}. The penalty parameter is initialized at σ = 0.1 and is adjusted dynamically <cit.> based on primal and dual feasibility to speed up the convergence of the ALM algorithm. For both algorithms, the number of levels m in the hierarchy is set to be m = 3,4,⋯,9 for N=64,128,⋯,4096, and the number of columns for all levels is set to be r=20. In Algorithm <ref>, y, t and M are initialized from the standard normal distribution, with M being primal-feasible. This is achieved by having the initial primal variable M=g(A) for some A∈^̋3N+1, where g is defined in (<ref>). In Algorithm <ref>, x, v, y and t are randomly initialized from the standard normal distribution. Throughout the updates of Algorithms <ref> and <ref>, we evaluate the accuracy of approximate solutions by monitoring the relative primal feasibility, the relative dual feasibility, and the relative duality gap, as detailed in the following.
	
	The primal feasibility of the variable M is governed by how well it satisfies the PSD and linear constraints in (P). In Algorithm <ref>, M is directly updated as a dense matrix, while in Algorithm <ref>, M is maintained using a compressed representation that satisfies the linear constraints. Since M is guaranteed to satisfy the linear constraints in (P), we only monitor its PSDness using the following measure:
	
    η_P := max(0, -λ_min)/1+max(0, λ_max),
 where λ_min and λ_max are the smallest and largest eigenvalues of M.
	
	For the dual problem (D) and a candidate dual variable S, the PSD constraint for S is automatically satisfied because S is maintained as a positive semidefinite hierarchical matrix in both algorithms. We thus only monitor the dual feasibility by how well the dual equality constraint is satisfied using the following measure:
	
    η_D:= S - J
    		+ℱ^*({Λ_ij}_i<j,{λ_j}_j,γ) _F/1+J_F.

	
	Finally, we monitor the relative duality gap by:
	
    η_g := |primal objective-dual objective|/1+|primal objective|+|dual objective|.
 
	
	We terminate the algorithm when η := max(η_P, η_D, η_g) ≤ 10^-3, or when the ALM algorithm has run for 1500 iterations. It is important to highlight that we exploit the hierarchical structure present in the PSD primal and dual variables to efficiently evaluate these convergence metrics.
	
	We examine the ground-state energy recovery for the TFI model on an N × 1 lattice, for system sizes N ∈{64, 128, 256, 512, 1024, 2048, 4096} and an external magnetic field strength parameter h ∈{0.1, 1, 1.5}. Let E_0 denote the true ground-state energy and Ẽ_0 the lower bound of the ground-state energy obtained from (P). The relative error is defined as:
	
    Err_rel := E_0 - Ẽ_0/|E_0|.
 The relative errors for Algorithms <ref> and <ref> are given in Tables <ref> and <ref>. Additionally, we present the evolution of our convergence metrics as a function of the ALM iteration number in Figures <ref> and <ref>, focusing on the 1-D TFI model with a fixed external magnetic field strength parameter h=1 and various system sizes. Alongside the relative primal and dual feasibility measures and the relative duality gap, we also track the per-site primal objective change between subsequent iterations. All convergence metrics are transformed using a base-10 logarithm function. Within 1500 ALM iterations, all metrics drop below 10^-3 for experiments with an external magnetic field strength parameter h=1. We note that it is harder to make Algorithm <ref> converge compared to Algorithm <ref>. For example, we are not able to get the case for N=4096 to converge to the prescribed accuracy 10^-3. We suspect this is due to the fact that it is harder to fit the primal PSD variable by a low-rank hierarchical matrix compared to the dual PSD variable.
	
	
	
	
	
	
	
	
	
	
	
	
	

§ CONCLUSION

	In this paper, we explored the computation of a specific semidefinite relaxation for determining the ground-state energy of many-body problems, which can be solved in polynomial time and provides a reasonable lower bound for the ground-state energy. Additionally, we identified a hierarchical structure in both the positive semidefinite (PSD) primal and dual variables, allowing us to circumvent the expensive projection onto the PSD cone, thereby reducing the per-iteration complexity of the ALM-type algorithm from cubic to quadratic or almost linear. 
	
	The relaxed problem provides only a lower bound for the lowest energy. To evaluate the effectiveness of our approach, we compare the recovered lower bound with the true ground-state energy for the 1-D transverse field Ising model. Notably, for the most challenging case of h=1, where the system undergoes a quantum phase transition, our algorithm still produces a reasonable lower bound. Furthermore, our algorithm can handle systems consisting of up to 4096 spins, whereas previous work on the variational embedding method, such as <cit.>, which employs more accurate yet more expensive constraints, can only manage systems of a few dozen spins without leveraging the periodicity of the model.
	
	Currently, unlike <cit.>, we do not impose conic constraints on each of the off-diagonal blocks of the moment matrix. Including them while leveraging the computational benefits of a hierarchical matrix will be the subject of future investigations. 

    

§ ACKNOWLEDGEMENTS

        Y. W. acknowledges partial support by ASCR Award DE-SC0022232 from the Department of Energy. Y.K. acknowledges partial support by NSF-2111563, NSF-2339439 from the National Science Foundation. Y.K. also acknowledges various interesting discussions with Michael Lindsey on speeding up semidefinite programming. 
	
	
	

§ COMPLEXITY OF OPERATIONS WITH HIERARCHICAL MATRICES
 
	
	In this section, we present several propositions that describe the complexity of manipulating hierarchical matrices.
	
	 
		
			
  * Let A=a_1 a_2^* and B=b_1 b_2^* be two low-rank matrices with a_1, a_2, b_1, b_2∈ℂ^p× r. Then ∑_i<j≤ pA(i,j)B(i,j) has a complexity of O(pr^2).
			
  * Let A=a_1 a_2^* be a low-rank matrix with a_1, a_2∈ℂ^p× r and B∈ℂ^p× p. Then ∑_i<j≤ pA(i,j)B(i,j) has a complexity of O(p^2 r).
		
	
	Based on this, we have the following propositions:
	 
		For two hierarchical matrices A =ℋ^(2)(y) and A' =ℋ^(2)(y') with y = {y^(1), ⋯, y^(m)} and y' = {y'^(1), ⋯, y'^(m)}, where y^(l), y'^(l')∈^CK × r for 1 ≤ l,l' ≤ m, the formula 
		
    ∑_i<j≤ K A_ij(k,κ) A_ij'(k',κ'),  ∀ k,k',κ,κ'

		can be computed with O(Km^2 r^2) complexity. 
	
	
		
    ∑_i<j≤ K A_ij(k,κ) A_ij'(k',κ')   =   ∑^m_l=1∑^m_l'=1∑_i<j≤ K(_ij^(2)(y^(l))(k,κ)  _ij^(2)(y'^(l'))(k',κ')).

		For fixed k, k', κ and κ', [_ij^(2)(y^(l))(k,κ)]_i,j∈ [K] and [_ij^(2)(y'^(l'))(k',κ')]_i,j∈ [K] take the form of 
		
    B = [   B_1^(l);             B_2^(l);                             ⋱;                               B_n_l^(l) ],   B' = [    B'_1^(l');                 B'_2^(l');                                      ⋱;                                        B'_n_l'^(l') ]

		respectively, where the diagonal blocks B_i^(l)∈ℂ^K/n_l× K/n_l and B'_j^(l')∈ℂ^K/n_l'× K/n_l' are rank-r matrices. The complexity of ∑_i<j(_ij^(2)(y^(l))(k,κ)  _ij^(2)(y'^(l'))(k',κ')) can thus be obtained by applying Proposition <ref> (Part 1) for max{n_l,n_l'} times, each time having O(K/max{n_l,n_l'}r^2) complexity, which gives a total complexity of O(K r^2). Incorporating the double sum ∑^m_l=1∑^m_l'=1 gives a final complexity of O(Km^2r^2)
	
	
	
	
	
	
	 
		For a hierarchical matrix A =ℋ^(2)(y) with y = {y^(1), ⋯, y^(m)} where y^(l)∈^CK × r for 1 ≤ l ≤ m, and a dense matrix A'∈ℂ^CK× CK, the formula 
		
    ∑_i<j≤ K A_ij(k,κ) A_ij'(k',κ'),  ∀ k,k',κ,κ'

		can be computed with O(K^2 r) complexity. 
	
	
		
    ∑_i<j≤ K A_ij(k,κ) A_ij'(k',κ')    =   ∑^m_l=1∑_i<j≤ K(_ij^(2)(y^(l))(k,κ)   A_ij'(k',κ')).

		For fixed k and κ, [_ij^(2)(y^(l))(k,κ)]_i,j∈ [K] takes the form of 
		
    B = [   B_1^(l);             B_2^(l);                             ⋱;                               B_n_l^(l) ],

		with some matrices B^(l)_i ∈ℂ^K/n_l × K/n_l each having rank r. With this form, the complexity of 
 ∑_i<j≤ K_ij^(2)(y^(l))(k,κ)   A_ij'(k',κ') can be determined by applying Proposition <ref> (Part 2) n_l times, each time having complexity O(rK^2/n_l^2 ), which gives a total complexity of O(rK^2/n_l). Summing this complexity ∑^m_l=1O(rK^2/n_l), we arrive at a final complexity of O(K^2r).
	
	
	
	plain
	


 Paper Title: HOT: An Efficient Halpern Accelerating Algorithm for Optimal Transport Problems.








































































HOT: An Efficient Halpern Accelerating Algorithm for Optimal Transport Problems
    Guojun Zhang,  Zhexuan Gu, Yancheng Yuan, Defeng Sun
Guojun Zhang and Zhexuan Gu contribute equally to this manuscript.
Corresponding author: Yancheng Yuan.
The authors are with The Hong Kong Polytechnic University. E-mail: guojun.zhang@connect.polyu.hk, zhexuan.gu@connect.polyu.hk, yancheng.yuan@polyu.edu.hk, defeng.sun@polyu.edu.hk.
    
====================================================================================================================================================================================================================================================================================================================================================







§ ABSTRACT 
This paper proposes an efficient HOT algorithm for solving the optimal transport (OT) problems with finite supports. We particularly focus on an efficient implementation of the HOT algorithm for the case where the supports are in ℝ^2 with ground distances calculated by L_2^2-norm. Specifically, we design a Halpern accelerating algorithm to solve the equivalent reduced model of the discrete OT problem. Moreover, we derive a novel procedure to solve the involved linear systems in the HOT algorithm in linear time complexity. Consequently, we can obtain an ε-approximate solution to the optimal transport problem with M supports in O(M^1.5/ε) flops, which significantly improves the best-known computational complexity. We further propose an efficient procedure to recover an optimal transport plan for the original OT problem based on a solution to the reduced model, thereby overcoming the limitations of the reduced OT model in applications that require the transport map. We implement the HOT algorithm in PyTorch and extensive numerical results show the superior performance of the HOT algorithm compared to existing state-of-the-art algorithms for solving the OT problems.   




Optimal transport, Kantorovich-Wasserstein distance, Halpern iteration, Acceleration, Computational complexity. 



































§ INTRODUCTION




ThE Kantorovich-Wasserstein (KW) distance has become a primer choice for measuring the similarity between two probability distributions. It has demonstrated remarkable success in various applications, including color transfer <cit.>, texture synthesis and mixing <cit.>, registration and warping <cit.>, transport-based morphometry <cit.>, and hypothesis testing <cit.>, among others. Despite its powerful geometric framework for comparing probabilities, the KW distance is computationally expensive in general <cit.>. Specifically, it requires solving an optimal transport (OT) problem, which is a (large-scale) linear programming (LP) in a discrete setting. Standard methods, such as the simplex method and the interior point method, suffer from high computational complexity relative to the problem size. Furthermore, these methods are difficult to parallelize, which can hardly benefit from the modern powerful graphics processing units (GPUs). Consequently, solving the LP problem of the discrete OT problem remains daunting in modern data-driven applications due to high computational and memory costs. This paper addresses these two challenges by proposing an efficient and easily parallelizable algorithm for solving an equivalent reduced model of the OT problem.


 §.§ Related work and existing challenges

When it comes to computing the KW distance between two discrete probability distributions with M supports, there are mainly two popular approaches: (i) Computing an approximated KW distance by solving the optimal transport problem with an additional entropy regularization <cit.>; (ii) Computing the KW distance via solving the corresponding LP problem <cit.>. The readers can refer to <cit.> and the references therein for a more detailed discussion of the algorithms for solving OT problems. Before introducing our new algorithm, we briefly discuss the challenges of the aforementioned approaches.


Challenges with the Entropy-regularized approach: Due to its scalability, the Sinkhorn algorithm, and its improved versions have been widely adopted to compute an approximation of the KW distance in applications <cit.>. In particular, the Sinkhorn algorithm can efficiently solve the regularized OT problem when the regularization parameter is moderate (i.e., no less than 10^-2). However, a high-quality approximation of the KW distance is important for better performance in many applications, which requires solving the regularized OT problem with a small regularization parameter. Unfortunately, a small regularization parameter will usually cause numerical issues and
a slower convergence for the Sinkhorn algorithm.  Some stabilized and rescaling techniques <cit.> have been proposed to improve the robustness of solving the regularized OT problems, but the efficiency of the stabilized algorithms is unsatisfactory compared to the Sinkhorn algorithm. 



Challenges with the LP approach: 
Along this line, the interior point method <cit.> and the network simplex method <cit.> are popular choices for obtaining solutions with high accuracy to the moderate scale LP problem. Recently, a semismooth Newton based inexact proximal augmented Lagrangian method <cit.> has been proposed for solving linear programming problems which can solve the OT problem as a special case. The semismooth Newton based algorithm can exploit the sparsity of the solution by the generalized Jacobian and show superior numerical performance compared to Gurobi in some examples.  However, these solvers are not applicable for solving the problem on a very large scale due to the high computational complexity. The urgent need to solve large-scale OT problems in applications inspires extensive research in designing first-order methods. Among them, the Douglas-Rachford splitting algorithm <cit.> and primal-dual hybrid gradient method (PDHG) <cit.>. Jambulapati et al. <cit.> proposed an algorithm based on a dual-extrapolation algorithm to achieve an O(M^2/ε) computational complexity bound for obtaining an ε-approximate solution (in terms of objective function value) [Despite its better complexity bound, the empirical performance of this algorithm is not as efficient as other algorithms, such as accelerated gradient method <cit.>.]. Recently, Zhang et al. <cit.> proposed an efficient Halpern-Peaceman-Rachford (HPR) algorithm for solving the OT model and the Wasserstein barycenter problem, which can obtain an ε-approximate solution (in terms of the Karush-Kuhn-Tucker (KKT) residual) of the OT model in O(M^2/ε) flops. We summarize some known complexity results for solving the OT problem in Table <ref>. Readers can refer to <cit.> and the references therein for a more detailed discussion. 











Beyond the challenges above in computational efficiency, all these algorithms for the original OT problem with M supports require at least a memory cost of O(M^2). This memory cost makes it forbidden to compute the KW distance of two distributions with massive supports (i.e., the OT problem for computing the KW distance of two 512×512 grey images has more than 6.8× 10^10 variables). This motivates researchers to explore possible reduced models for the OT problem. When the ground distances between supports in ℝ^d are calculated by L_1-norm, Ling and Okada proposed an equivalent reduced model with O(dM) memory cost to calculate the earth mover distance (equivalent to the KW distance) <cit.>. Recently, Auricchio et al. <cit.> extended the idea to the case where the ground distances between supports are calculated by L_2^2-norm  and proposed an equivalent reduced model with O(dM^d+1/d) memory cost. The authors in <cit.> adopted the Network Simplex method to solve the reduced model and demonstrated superior performance in terms of computational and memory efficiency compared to the Sinkhorn and the improved Sinkhon method <cit.> on examples of moderate scale. Unfortunately, the efficiency becomes unsatisfactory for very large-scale problems (see Section <ref> for details). Moreover, the transport plan is not available if we solve the reduced model, which is critical for a wide class of applications, such as color transfer<cit.> and texture synthesis <cit.>.




 §.§ Contributions

Motivated by the recent advancements in the accelerated algorithms based on Halpern iteration <cit.>, we propose an efficient Halpern accelerating method for solving the reduced OT problem, which is abbreviated as “HOT” for convenience, to address the challenges in computing the KW distance with a finite number of supports. We particularly focus on an efficient implementation of the HOT algorithm for the case where the supports are in ℝ^2 with ground distances calculated by L_2^2-norm, which includes a wide class of applications as aforementioned. Specifically, HOT adopts a first-order algorithm with Halpern acceleration to solve the equivalent reduced OT model, which can obtain an ε-approximate solution in O(1/ε) iterations <cit.>. More importantly, we design a fast procedure for solving the subproblems with linear time complexity by fully exploiting the problem structure. This also makes the popular alternating direction method of multiplier (ADMM) <cit.> scalable for solving the reduced OT model. Overall, our proposed HOT algorithm can compute an ε-approximation of the KW distance between two histograms supported on M=m × n bins within O((m^2n + n^2m)/ε) flops. This is the best-known computational complexity for computing an approximate KW distance to our knowledge. Moreover, we propose an efficient algorithm to recover a transport plan based on the obtained solution of the reduced model, which releases the power of the reduced model in applications. We implement HOT in PyTorch and extensive numerical results will be shown in Section <ref> to demonstrate the superior and robust performance of HOT for computing the KW distance, compared to state-of-the-art algorithms, including Sinkhorn <cit.>, Improved Sinkhorn <cit.>, Network Simplex method <cit.>, ADMM <cit.>, interior point method (in Gurobi). 

We summarize the main contributions of this paper as follows:

    
  * We propose an efficient HOT algorithm for solving the reduced model of the OT problem with an attractive O(1/ε) iteration complexity guarantee with respect to the KKT residual.
    
  * We designed a highly efficient algorithm for solving the subproblems of the HOT algorithm with linear time complexity.
    
  * We propose an efficient algorithm to recover a transport plan based on the obtained solution of the reduced model, which removes the restriction of the reduced model in applications requiring a transport plan.
    
  * We implement the HOT algorithm in PyTorch, which supports both CPU and GPU computation and is user-friendly for researchers in the machine learning community.
    
  * Extensive numerical testings are conducted and presented to demonstrate the efficiency of the HOT algorithm.


The rest of the paper is organized as follows. We introduce the equivalent reduced OT model in Section <ref>. This section also includes an efficient procedure for recovering the transport plan from a solution to the reduced OT problem. The HOT algorithm and its computational complexity guarantees will be presented in Section <ref>. We present extensive numerical results in Section <ref> and conclude the paper in Section <ref>.


Notation. We denote the n-dimensional real Euclidean space as ℝ^n and the nonnegative orthant of ℝ^n as ℝ^n_+. For any x ∈ℝ^n and y ∈ℝ^n, we define ⟨ x, y ⟩ := ∑_i=1^n x_i y_i and x := √(∑_i=1^n x_i^2), respectively. Additionally, let 1_m (resp. 0_m) denote the m-dimensional vector with all entries being 1 (resp. 0). For a given matrix A ∈ℝ^m × n, we denote A^⊤∈ℝ^n × m as its transpose. For a collection of matrices {A_1, …, A_m}, we denote the block diagonal matrix with diagonal blocks A_i as diag(A_1, …, A_m). A_1 ⊗ A_2 stands for the Kronecker product of matrices A_1 and A_2. Moreover, for a closed convex set C, we denote the indicator function of C and the Euclidean projector over C as δ_C and Π_C(x) := min_z ∈ Cx-z, respectively.







§ KANTOROVICH-WASSERSTEIN DISTANCES


In this section, we first introduce an equivalent reduced model of the OT problem for computing the KW distance between two-dimensional histograms. Subsequently, we present a fast and easily implementable algorithm to reconstruct the optimal transport plan of the original OT model from a solution of this reduced model.


 §.§ An equivalent reduced model of the OT problem

In the following discussion, we assume two-dimensional histograms for simplicity. As previously mentioned, these histograms are widely used in applications as shape and image descriptors. Without loss of generality, we adopt the following assumptions and notations:

    
  * Histograms have supports in M = m × n bins with m rows and n columns;
    
  * The index set for bins is defined as ℐ = {(i, j) | 1 ≤ i ≤ m, 1 ≤ j ≤ n}. We use (i, j) to denote a bin or a node corresponding to it;
    
  * μ^1 and μ^2 are the two histograms to be compared, where each histogram μ^k is defined as {μ^k_i, j|μ^k_i,j≥ 0, (i, j) ∈ℐ,∑_(i,j)∈ℐμ^k_i,j = 1 } for k = 1, 2.
 

With these notations and assumptions, the discrete OT problem for computing KW distance between histograms μ^1 and μ^2 can be defined as follows:

    [                                                                                                                              min_π                                                                                              ∑_(i,j)∈ℐ∑_(k,l)∈ℐ c_i,j;k,lπ_i,j;k,l;                                                                                                                               s.t. ∑_(k,l)∈ℐπ_i,j;k,l = μ^1_i,j,  ∀ (i,j)∈ℐ, 
       ∑_(i,j)∈ℐπ_i,j;k,l = μ^2_k,l,  ∀ (k,l)∈ℐ, 
       π_i,j;k,l≥ 0,   ∀ (i,j)∈ℐ,  ∀ (k,l)∈ℐ, ]

where π is the transport plan between histograms μ^1 and μ^2. The ground distance c_i,j;k,l is commonly defined by the L_p^p distance:

    c_i,j;k,l = (i,j)^⊤ - (k,l)^⊤_p^p = (|i-k|^p + |j-l|^p).

In this paper, we focus on the case where p=2. By exploiting the separable structure of the ground distance, Auricchio et al. <cit.> proposed the following equivalent model in terms of the optimal objective function value:

    [                                                                                                                                                                                                         min_f^(1), f^(2)                                                                                                                                                    ∑_(i,j)∈ℐ[ ∑_k=1^m (k-i)^2 f_i,k,j^(1) + ∑_l=1^n (j-l)^2 f_k,j,l^(2)];                                                                                                                                                                                                                     s.t. ∑_i=1^m f_i,k,j^(1) = ∑_l=1^n f_k,j,l^(2),     ∀ (k,j) ∈ℐ,
       ∑_k=1^m f_i,k,j^(1) = μ^1_i,j,     ∀ (i,j) ∈ℐ, 
       ∑_j=1^n f_k,j,l^(2) = μ^2_k,l,    ∀ (k,l) ∈ℐ, 
       f_i,k,j^(1)≥ 0, f_k,j,l^(2)≥ 0,     ∀ (i,j),(k,l) ∈ℐ, ]

where f^(1)_i,k,j denotes the input flow from bin (i,j) to (k,j) and f^(2)_k,j,l denotes the output flow from bin (k,j) to (k,l). Compared to formulation (<ref>), the formulation (<ref>) offers substantial computational benefits. Specifically, the reduced problem (<ref>) only has mn^2 + m^2n variables, whereas the original model has m^2n^2 variables. Moreover, the reduced model remains an LP problem. Consequently, popular algorithms for LP problems can be applied to solve this reduced model, such as the network-simplex method and the interior point method. Although the computation and memory costs of these mentioned algorithms are lower for the reduced model, it remains a challenge for solving large-scale problems. In this paper, we focus on addressing the challenges by designing a fast algorithm to solve the reduced model (<ref>).


To facilitate the design of the algorithm, we reformulate the model (<ref>) into the following standard form of linear programming:

    [            min_x∈ℝ^N ⟨c, x⟩  + δ_ℝ_+^N(x);                 s.t.               Âx=b̂, ]

where  

    
  * M_3=3M-1, N=m^2n+mn^2;
    
  * x=[ f^(1); f^(2)]∈ℝ^m^2n×ℝ^mn^2 with 
    
    f^(1)={f_i,k,j^(1), ∀(i, j) ∈ℐ, k=1,…,m}, 
    
             f^(2)={f_k,j,l^(2), ∀(k, l) ∈ℐ, j=1,…,n};
  
    
  * c=[c^1;c^2]∈ℝ^m^2n×ℝ^mn^2 with 
    
    c^1={c_i,k,j^(1)=(k-i)^2, ∀(i, j) ∈ℐ, k=1,…,m}, 
    
            c^2={c_k,j,l^(2)=(j-l)^2, ∀(k, l) ∈ℐ, j=1,…,n};

     
  * b̂=[0_M;μ^1;
μ^2]∈ℝ^M×ℝ^M×ℝ^M;
    
  * Â=[[ A_1 A_2; A_3   0;   0 Â_4 ]]  ∈ℝ^(M_3+1)× N with 
    
    A_1=I_M⊗1_m^⊤∈ℝ^M× m^2n,
    A_2=-1_n^⊤⊗ I_M∈ℝ^M× mn^2,
    
        A_3=I_n⊗ (1_m^⊤⊗ I_m) ∈ℝ^M× m^2n,
    Â_4=I_n⊗ (1_n^⊤⊗ I_m) ∈ℝ^M× mn^2.
  

For notational convenience, let I̅_m=[I_m-1,0_m-1]∈ℝ^(m-1)× m. We define 

    A:=[[ A_1 A_2; A_3   0;   0 A_4 ]] ∈ℝ^M_3× N,  b:=[0_M;μ^1; I̅_Mμ^2]∈ℝ^M_3.

with 

    A_4=diag(  1_n^⊤⊗ I_m,…,1_n^⊤⊗ I_m,1_n^⊤⊗I̅_m)∈ℝ^(M-1)× mn^2.

Similar to <cit.>, we can obtain that A  defined in (<ref>) has full row rank, and 

    {x ∈ℝ^N  |  Ax = b} = {x ∈ℝ^N  | Âx = b̂}.

As a result, the linear programming problem (<ref>) is equivalent to 
 
    [           min_x ∈ℝ^N ⟨c, x⟩  + δ_ℝ_+^N(x);                 s.t.                Ax=b. ]

Furthermore, the dual problem of (<ref>) takes the form: 

    y ∈ℝ^M_3, z ∈ℝ^Nmin{-⟨b, y⟩+δ_ℝ_+^N(z)  | A^⊤ y+z=c}.

The KKT conditions associated with (<ref>) and (<ref>) can be  given by

    A^* y+z=c,    A x=b,   ℝ_+^N∋ x ⊥ z ∈ℝ_+^N,

where x ⊥ z means x is perpendicular to z, i.e., ⟨ x, z⟩=0.

  


 §.§ Reconstruct the transport plan from the reduced model

The absence of the transport plan π makes the reduced OT model less favorable in  applications where the transport map is necessary (i.e., color transfer <cit.>). We address this issue by proposing a fast algorithm (shown in Algorithm <ref>) to reconstruct an optimal transport plan of the original model from an optimal solution of the reduced model (<ref>).
 

The following proposition shows that the output of Algorithm <ref> is an optimal transport plan for of the original OT model.

Given an optimal solution (f^(1),f^(2)) to problem (<ref>), the output π of Algorithm <ref> is an optimal solution to the optimal transport problem (<ref>).


Since (f^(1),f^(2)) is an optimal solution to problem (<ref>), we have

    f_i,k,j^(1)≥ 0,    f_k,j,l^(2)≥ 0,   ∀ (i,j) ∈ℐ,   ∀ (k,l) ∈ℐ,

which implies 

    π_i,j;k,l≥ 0,   ∀ (i,j),   ∀ (k,l) ∈ℐ.

Additionally, since 

    f^(1)_i,k,j≤∑_i=1^m f_i,k,j^(1) = ∑_l=1^n f_k,j,l^(2),

by fixing i,j,k, we obtain from Algorithm <ref> that

    ∑_l=1^nπ_i,j;k,l = f^(1)_i,k,j.

Similarly, by fixing j,k,l, we also obtain  

    ∑_i=1^mπ_i,j;k,l = f^(2)_k,j,l.

Hence, according to the constraints in problem (<ref>), we have 

    ∑_(k,l)∈ℐπ_i,j;k,l = ∑_k=1^m f_i,k,j^(1) = μ^1_i,j,
    ∑_(i,j)∈ℐπ_i,j;k,l = ∑_j=1^n f_k,j,l^(2) = μ^2_k,l,

which, together with (<ref>), shows that π is a feasible solution to problem (<ref>). Furthermore, from (<ref>), (<ref>), and the definition of c in (<ref>), we can obtain 

    [                                                                                                                ∑_((i,j),(k,l)) c_i,j;k,lπ_i,j;k,l;                                                                        = ∑_(i,j)∈ℐ [ ∑_k=1^m (k-i)^2 f_i,k,j^(1) + ∑_l=1^n (j-l)^2 f_k,j,l^(2) ]. ]

According to <cit.>, we know that the optimal objective function values of problems (<ref>) and (<ref>) are equivalent. Therefore, π is an optimal solution to problem (<ref>).




§ A HALPERN ACCELERATING ALGORITHM FOR SOLVING OT PROBLEM


In this section, we first introduce an efficient Halpern accelerating method for solving problem (<ref>), which includes the equivalent reduced OT problem (<ref>) as a special case. Subsequently, we present an efficient implementation of the proposed algorithm by designing a novel procedure to solve the involved linear system in linear time complexity.


 §.§ HOT: A Halpern accelerating method for solving OT problem

Given σ>0, the augmented Lagrange function corresponding to the dual problem (<ref>) is defined by, for any (y,z,x)∈ℝ^M_3×ℝ^N×ℝ^N,

    L_σ(y, z ; x):=-⟨b, y⟩+δ_ℝ_+^N(z) +σ/2A^⊤ y+z-c+1/σx^2 -1/σx^2.

For ease of notation, denote w:= (y, z, x). A fast Halpern accelerating method <cit.> for solving OT problems is presented in Algorithm <ref>. A detailed derivation of the algorithm in its current form and more discussions can be found in <cit.> and the references therein.

 
Note that Step 4 in Algorithm <ref> is from the Halpern iteration with a stepsize of 1/k+2. Without Step 4, the HOT algorithm reduces to the ADMM with a unit step size. The following proposition shows the global convergence of the HOT algorithm. The proof of this proposition follows from <cit.>. 
The sequence {w̅^k}= {(y̅^k, z̅^k, x̅^k)} generated by the HOT algorithm in Algorithm <ref> converges to the point w^*=(y^*, z^*, x^*), where (y^*, z^*) is a solution to problem (<ref>) and x^* is a solution to problem (<ref>).


 Next, we analyze the iteration complexity of the HOT algorithm for obtaining an ε-approximate solution, where an appropriate measure for the quality of the solution is crucial. In this paper, we consider the residual mapping associated with the KKT system (<ref>):

    ℛ(w)=([           b-Ax; z-Π_ℝ_+^N(z-x);      c-A^⊤y- z ] )

for any w=(y,z,x)∈ℝ^M_3×ℝ^N×ℝ^N. Note that ℛ(w^*) = 0 is equivalent to the facts that x^* ∈ℝ^N and (y^*, z^*) ∈ℝ^M_3×ℝ^N are the solution to problems (<ref>) and (<ref>), respectively. The KKT residual ℛ(·) is a commonly used and practical measure for the quality of the approximation solution to (<ref>). It follows from <cit.> that the HOT algorithm enjoys an appealing O(1/k) nonergodic convergence rate in terms of the KKT residual for solving (<ref>), which is summarized in the following proposition. 

   Let {(y̅^k,z̅^k,x̅^k)} be the sequence generated by Algorithm <ref>, and let w^*=(y^*,z^*,x^*) be the limit point of the sequence {(y̅^k,z̅^k,x̅^k)} and R_0=x^0-x^* + σ (z^0-z^*). For all k ≥ 0, we have the following bounds:

    ℛ(w̅^k)≤( σ +1/σ) R_0/(k+1).
			 



Note that, without acceleration, the ADMM has an O(1/√(k)) non-ergodic rate in terms of both the objective function value gap and feasibility violations <cit.>. In contrast, the HOT algorithm in Algorithm <ref> achieves an O(1/k) non-ergodic convergence rate, offering significant advantages for solving large-scale OT problems.  









 §.§ A fast implementation of the HOT algorithm

In this section, we present a fast implementation of the HOT algorithm. Through direct calculations, we obtain the following updates of z̅^k and y̅^k for any k ≥ 0:

    
  * Update of z̅^k:

    z̅^k = Π_ℝ_+^N(c - A^⊤y̅^k - x̅^k / σ);


  * Update of y̅^k:

    AA^⊤y̅^k = b/σ - A(x^k/σ + z^k - c).




Therefore, the main computational bottleneck of the HOT algorithm for solving (<ref>) is solving the linear system (<ref>). Note that the dimension of the matrix AA^⊤ is M_3 × M_3 with M_3 defined in (<ref>). In applications, M_3 is usually a huge number. As an illustrative example, for an image with 256 × 256 pixels, M_3 = 196,607. As a result, it is not computationally affordable for computing a (sparse) Cholesky decomposition for the matrix AA^⊤ or solving the linear system (<ref>) with standard direct solver. Indeed, it is computationally expensive even for computing the matrix AA^⊤. Instead, in the remaining part of this subsection, we will derive a linear time complexity procedure for solving the linear equation AA^⊤y = R with a given vector R ∈ℝ^M_3. It is worthwhile mentioning that our procedure does not require calculating nor storing the matrix AA^⊤. Note that A A^⊤ can be written in the following form:
    
    A A^⊤=[[   E_1   E_2   E_3; E_2^⊤   E_4     0; E_3^⊤     0   E_5 ]],
	
	where	
	
	    
  * E_1=(m+n)I_M∈ℝ^M× M;
	    
  * E_2= diag(1_m1_m^⊤,…, 1_m1_m ^⊤,1_m1_m ^⊤)  ∈ℝ^M × M;       
            
  * E_3= -1_n⊗(I_m,…,I_m,I̅_m^⊤ )∈ℝ^M× (M-1); 
	     
  * E_4=  m I_M∈ℝ^M× M;
	     
  * E_5=A_4A_4^⊤=nI_M-1∈ℝ^(M-1)×(M-1).
	
 To better explore the structure of the linear system AA^⊤y=R, we rewrite it equivalently as

    AA^⊤y=[[   E_1   E_2   E_3; E_2^⊤   E_4     0; E_3^⊤     0   E_5 ]][[ y_1; y_2; y_3 ]]=[[ R_1; R_2; R_3 ]],

where y:=(y_1; y_2; y_3)∈ℝ^M×ℝ^M×ℝ^M-1 and R:= (R_1; R_2; R_3)∈ℝ^M×ℝ^M×ℝ^M-1. To further explore the block structure of the linear system, we can denote y_i:= (y_i^1; …; y_i^n) ∈ℝ^m×⋯×ℝ^m for i=1,2, and y_3 = (y_3^1; …; y_3^n) ∈ℝ^m×⋯×ℝ^m×ℝ^m-1. Correspondingly, we write R_i = (R_i^1; …; R_i^n) for i=1,2,3. The next proposition gives an explicit formula of the solution to the linear equation in (<ref>). 

Consider A ∈ℝ^M_3 × N defined in (<ref>). Given R ∈ℝ^M_3, the solution y to AA^⊤y = R in the form (<ref>) is given by:

    y_2^j = 1/m( R_2^j - 1_m^⊤y^j_1 ),    j = 1, …, n, 
           y_3^j = 1/n( R_3^j + ∑_j=1^ny_1^j ),    j = 1, …, n-1, 
           y_3^n = 1/n( R_3^n + I̅_m∑_j=1^ny_1^j ), 
           y_1^j = ŷ_1^j - ŷ_1^a,    j = 1, …, n,

where

    
  * ŷ_1^j = 1/m+n( R̃_1^j + R̃_2^j + R̃_3),    j = 1, …, n, with R̃_1^j = R^j_1 + 1/n1_m^⊤ R^j_1, R̃_2^j = -( 1/m + 1/n) 1_m^⊤ R^j_2, and R̃_3 = 1/n( ∑_j=1^n-1 R^j_3 + I̅_m^⊤ R^n_3) + 1/n^21_M-1^⊤ R_3;
    
  * ŷ_1^a = ( I_m + 1/n1_m1_m^⊤) Ŵ∑_j=1^nŷ_1^j;
    
  * Ŵ = ( -diag( 1/m I_m-1, 1/m+1( 1 - 1/n) ) - 1/w d d^⊤), with d = [ 1/m1_m-1; 1/m+1( 1 - 1/n) ] ∈ℝ^m and w = 1/m - 1/(m+1)( 1 - 1/n).




By some direct calculations, we can solve (<ref>) equivalently as:

    y_2 = R_2 - E_2^⊤ y_1/m, 
           y_3 = R_3 - E_3^⊤ y_1/n, 
          Ẽ_1 y_1 = R_1 - 1/m E_2 R_2 - 1/n E_3 R_3,

where Ẽ_1 = ( E_1 - 1/m E_2 E_2^⊤ - 1/n E_3 E_3^⊤). As a result, the key is to obtain y_1 by solving (<ref>). Let Ê_1 := E_1 - 1/m E_2 E_2^⊤. By direct calculations, we have:

    Ê_1 = diag( Ê_1^1, …, Ê_1^n)

with Ê_1^j = (m+n) I_m - 1_m1_m^⊤,    j = 1, …, n. By the Sherman–Morrison-Woodbury formula, we directly get:

    Ê_1^-1 = diag( (Ê_1^1)^-1, …, (Ê_1^n)^-1)

with (Ê_1^j)^-1 = 1/m+n( I_m + 1/n1_m1_m^⊤), j = 1, …, n.
On the other hand, let:

    Q = n-1/n I_m + 1/nI̅_m^⊤I̅_m∈ℝ^m × m.

Denote Q̂ := Q^1/2 such that Q̂Q̂ = Q, and Q̅ := 1_n⊗Q̂. We can obtain:

    Ẽ_1 = ( Ê_1 - 1/n E_3 E_3^⊤) = Ê_1 - Q̅Q̅^⊤.

Hence, by the Sherman–Morrison-Woodbury formula, we can derive:

    [                             Ẽ_1^-1        = ( Ê_1 - 1/n E_3 E_3^⊤)^-1;                                    = Ê_1^-1 - Ê_1^-1Q̅ W^-1Q̅^⊤Ê_1^-1 ]

with W = -I_m + Q̅^⊤Ê_1^-1Q̅. Note that:

    [                                               W                 = -I_m + ∑_j=1^nQ̂ (E_1^j)^-1Q̂;                                                 = 1/m+n( diag( -m I_m-1, -(m+1) ) + d_0 d_0^⊤), ]

where d_0 = [1_m-1; √(1 - 1/n)] ∈ℝ^m. Applying the Sherman–Morrison-Woodbury formula to W, we can obtain:

    [                                                 W^-1 = (m+n) ( diag( -1/m I_m-1, -1/m+1) - 1/w d_1 d_1^⊤) ]

with d_1 = [1/m1_m-1; 1/m+1√(1 - 1/n)]. It follows that

    [             Q̅ W^-1Q̅^⊤ = 1_n1_n^⊤⊗ (Q̂ W^-1Q̂);                            = 1_n1_n^⊤⊗ (m+n) Ŵ. ]

To explore the block structure of R, we denote:

    R̂_1 = R_1 - 1/m E_2 R_2 - 1/n E_3 R_3 = (R̂_1^1; …; R̂_1^n) ∈ℝ^m×⋯×ℝ^m,

which implies

    R̂_1^j = R^j_1 - 1/m1_m^⊤ R^j_2 + 1/n( ∑_j=1^n-1 R^j_3 + I̅_m^⊤ R^n_3),    j = 1, …, n.

Then, for j = 1, …, n,

    [                             ŷ_1^j                  = (Ê_1^-1R̂_1)^j;                                   = 1/m+n( R̃_1^j + R̃_2^j + R̃_3). ]

Define

    ŷ_1^a := ( I_m + 1/n1_m1_m^⊤) Ŵ∑_j=1^nŷ_1^j.

From (<ref>), (<ref>), and (<ref>), we have:

    y_1^j = ŷ_1^j - ŷ_1^a,    j = 1, …, n.

Substituting y_1 into (<ref>) and (<ref>), we can obtain the results for y_2 and y_3.  This completes the proof.

According to the explicit formula in Proposition <ref>, we can immediately derive the complexity result for solving the linear equation in (<ref>).

Consider A ∈ℝ^M_3 × N defined in (<ref>). The linear system AA^⊤ y = R in the form (<ref>) can be solved in O(M_3) flops.

Based on this corollary, we can determine the per-iteration computational cost of the HOT algorithm in Algorithm <ref> for each iteration.

The per-iteration computational complexity of the HOT algorithm in Algorithm <ref> in terms of flops is O(N).


Since A has at most 2N nonzero elements, the computational cost for calculating Ax and A^⊤y is only O(N). Except for solving linear systems, which can be done in O(M_3) from Corollary <ref>, Algorithm <ref> primarily involves matrix-vector multiplications and vector additions. Hence, the computational cost of Algorithm <ref> for each iteration is O(N).


Combining the iteration complexity in Proposition <ref> and the computational cost for each iteration in Corollary <ref>, we can derive the following overall computational complexity result of the HOT algorithm for solving problem (<ref>).


Let {y̅^k, z̅^k, x̅^k} be the sequence generated by the HOT algorithm in Algorithm <ref>. For any given tolerance ε > 0, the HOT algorithm needs at most

    1/ε(1+σ/σ(x^0-x^* + σz^0-z^*))-1

iterations to return a solution to  the equivalent OT problem (<ref>) such that the KKT residual ℛ(w̅^k)≤ε, where (x^*, z^*) is the limit point of the sequence {x̅^k, z̅^k}. In particular, the overall computational complexity of the HOT algorithm in Algorithm <ref> to achieve this accuracy in terms of flops is

    O((1+σ/σ(x^0-x^* + σs^0-s^*)) m^2n+mn^2/ε).



Note that the complexity of the Sinkhorn method for solving the original OT problem (<ref>) is O(m^2n^2/ε^2) to achieve an ε-accuracy solution in terms of objective function value <cit.>. Even by leveraging the separable  structure of the cost function c defined in (<ref>), the improved Sinkhorn method, as mentioned in <cit.> and <cit.>, still requires O(m^2n + mn^2/ε^2 + m^2n^2) to compute the KW distance. In contrast, the HOT algorithm exhibits lower overall computational complexity, offering a significant advantage in calculating the KW distance between large-scale histograms.









§ EXPERIMENTS


In this section, we comprehensively compare the HOT algorithm with five other state-of-the-art methods on the popular DOTmark dataset <cit.>. Additionally, employing the transport plan derived from Algorithm <ref>, we test the performance of the reduced OT model on the color transfer task. The findings reveal that HOT provides significant advantages over state-of-the-art methods in both memory and computational efficiency, particularly for large-scale problems.



 §.§ Numerical comparison on the DOTmark dataset


The DOTmark dataset <cit.> is a comprehensive collection of benchmark instances for evaluating and comparing algorithms in the field of optimal transport. It consists of a variety of instances categorized into different classes, such as Classic Images, Shapes, and Gaussian Distributions. In this experiment, we selected eight images each from the Classic Images and Shapes categories, as illustrated in Fig.<ref>. These selected images were resized to five different resolutions: 
64 × 64, 128 × 128, 256 × 256, and 512 × 512. Finally, we randomly selected 10 pairs from each category and computed the KW distance for each pair.




To exhibit the superiority of HOT, we compare it with the following five state-of-the-art methods:


    
  * Sinkhorn <cit.> is a widely used algorithm for computing an approximate KW distance by solving an entropy-regularized OT problem. As the Introduction Section discusses, its performance is sensitive to the regularization parameter, denoted as λ. To achieve varying levels of solution accuracy, we selected the λ to be 1%, 0.1%, and 0.01% of the median transport cost, following the setup from <cit.>. Due to the potential numerical instability caused by small λ values, we employed the Log-domain Sinkhorn algorithm implemented by POT <cit.> as the baseline method.  
    
   
    
  * Improved Sinkhorn <cit.> is an improved version of the Sinkhorn algorithm, optimized for computing distances over regular two-dimensional grids. Specifically, since the kernel matrix used by the Sinkhorn algorithm can be constructed using a Kronecker product of two smaller matrices, a matrix-vector product using a matrix of dimension M × M can be replaced by two matrix-matrix products over matrices of dimension √(M)×√(M), resulting in a significant improvement in computational efficiency. In our experiment, based on the MATLAB implementation of the improved Sinkhorn in <cit.>, we further developed the Log-domain improved Sinkhorn using PyTorch. The regularization parameter λ is kept the same as that of the original Sinkhorn method. 
    
 
   
    
  * Gurobi (11.0.1) is a popular optimization solver designed to address a wide range of mathematical programming problems, including linear and quadratic programming. In our experiment, we employ the interior point method (IPM) implemented in Gurobi to solve the reduced model (<ref>). Since it is unnecessary to obtain a basic solution, we disable the cross-over strategy.
  
    
  * Network Simplex implemented in the Lemon C++ graph library [https://lemon.cs.elte.hu/], is a highly efficient algorithm for solving uncapacitated minimum cost flow problems <cit.>. It has demonstrated the computational advantages over the Sinkhorn type methods in <cit.>  for 
    small size images by solving the reduced model (<ref>).
   
    
  * ADMM <cit.> is a popular first-order primal-dual method for solving large-scale optimization problems. It has shown great potential in solving large-scale optimal transport problems (<ref>) in a GPU setting <cit.>. In this numerical experiment, we use a generalized ADMM <cit.> to solve the equivalent model (<ref>) (replacing Step 4 in Algorithm <ref> with w^k+1= (1-ρ)w^k+ ρw̅^k and setting ρ=1.7) as the baseline to evaluate the acceleration effect of the Halpern iteration.


All experiments are conducted on an Ubuntu 22.04 server equipped with an Intel(R) Xeon(R) Platinum 8480C processor and an Nvidia GeForce RTX 4090 GPU with 24 GB of RAM. Due to hardware specifications, we limit the maximum memory usage of each algorithm to 24 GB. Additionally, we set the maximum one-call running time of each algorithm to 3600 seconds. For the HOT and ADMM methods, we adopt a stopping criterion based on the relative KKT residual:

    KKT_res = max{A^⊤ y + z - c/1 + c, min(x, z)/1 + x + z, Ax - b /1 +  b }.

Other methods use their default stopping criteria. We terminate all tested algorithms, except for Network Simplex which is an exact algorithm with a stopping tolerance of 1E-6. Finally, since different methods have varying stopping criteria, we use the following metrics to fairly evaluate the quality of the solutions: the `relative objective gap' (gap) and the `relative primal feasibility error' (feaserr). These metrics are defined as follows:

    [    gap = | ⟨ c, x ⟩ - ⟨ c, x_b⟩ |/| ⟨ c, x_b⟩ | + 1,; feaserr = max{min(x, 0) /1 +  x ,  Ax - b /1 +  b }, ]

where x_b is the solution obtained using Gurobi with the tolerance set to 1E-8.


We present the average results of 10 pairs within each category for all tested algorithms in Table <ref>. Since Gurobi runs out of memory for images sized from 256 × 256 to 512 × 512, we only report the feaserr for the HOT, Network Simplex, and ADMM. Additionally, Table <ref> only shows the results of Sinkhorn-type methods with λ = 0.01% of the median transport cost, as this parameter returns a solution of comparable quality to other methods. For more results of the improved Sinkhorn with different regularization parameters, refer to Table <ref>. Due to space constraints, we omit the results of the Sinkhorn with varying parameters, which exhibit similar performance to the improved Sinkhorn in terms of the gap.



We summarized some key findings in Table <ref> from the perspective of computational efficiency and memory cost:

    
  * Computational efficiency: HOT can return a comparable solution in terms of the gap and feaserr in the shortest time. Although the Network Simplex and Gurobi have computational advantages for computing the KW distance for small-scale images, these methods cannot handle large-scale problems effectively. IPM implemented in Gurobi requires solving a linear equation that relies heavily on Cholesky decomposition, causing the computational cost of each iteration to increase rapidly with image size. Additionally, the inherent sequential nature of the Network Simplex algorithm makes it challenging to parallelize effectively. In contrast, HOT and improved Sinkhorn benefit from low per-iteration costs and are easily parallelizable. However, the improved Sinkhorn needs to recover the solution to the original problem (<ref>) to compute the KW distance, making it unsuitable for large-scale problems. As a result, for images sized 128 × 128, HOT achieves a 17.44x speedup over Network Simplex, a 15.83x speedup over Gurobi, and a 19.54x speedup over improved Sinkhorn. Additionally, compared to ADMM, HOT benefits from a superior O(1/k) iteration complexity and saves 40% of iterations for images sized 512 × 512.    
    
    
  * Memory cost: HOT demonstrates a significant advantage in memory efficiency by exploiting the sparse structure of A in (<ref>) and utilizing the explicit solution of the linear equation presented in Proposition <ref> to avoid sparse Cholesky decomposition. In contrast, Sinkhorn-type methods need to recover the solution to the original OT problem (<ref>) and maintain the transport cost to calculate the KW distance, which involves M^2 variables. Given that these variables are stored using 64-bit floating-point representation, the Sinkhorn-based methods require at least 32GB of memory for images sized 256 × 256, which far exceeds the available 24GB. While the IPM in Gurobi solves the reduced model (<ref>), each iteration requires performing a sparse Cholesky decomposition, making it unsuitable for images sized 256 × 256 or larger.










To further illustrate the benefit of the explicit solution of the linear system presented in Proposition <ref>, we conduct a comparison between solving the linear equation using Proposition <ref> and sparse Cholesky decomposition. Given that the average iteration number of HOT is around 1500, we solve the linear system (<ref>) 1500 times for different image sizes. The results are shown in Fig.<ref>. It is clear that the computational costs of Cholesky decomposition increase rapidly as image size increases. Additionally, the forward-backward substitution method used to solve the linear system is also time-consuming because it cannot be efficiently parallelized. In contrast, solving linear systems using Proposition <ref> can be easily parallelized. It consumes significantly less time and remains constant as the size varies because it only requires O(M_3) flops, as shown in Corollary <ref>.









 §.§ An Application in Color Transfer

Color transfer <cit.> based on the OT model has found important applications in various fields, including digital image processing, computer graphics, and visual arts. It involves transferring the color characteristics from a target image to a source image to achieve a desired visual effect. Inspired by its convincing performance in color grading and color histogram manipulation <cit.>, we conduct the color transfer over the CIE-Lab domain by applying the optimal transport model to the 1D luminance channel and the 2D chrominance channel independently. Note that the 1D optimal transport plan can be found efficiently <cit.>. To efficiently conduct the optimal transport over the 2D chrominance channel, we first solve the equivalent reduced OT model using the HOT Algorithm <ref> and then recover an optimal transport map using Algorithm <ref>.  The performance of color transfer for selected image pairs can be found in Fig.<ref> and Fig.<ref>.






§ CONCLUSION


In this paper, we proposed an efficient and scalable HOT algorithm for computing the KW distance with finite supports. In particular, the HOT algorithm solves an equivalent reduced OT model where the involved linear systems are solved by a novel procedure in linear time complexity. Consequently, we can obtain an ε-approximate solution to the OT problem with M supports in ℝ^2 in O(M^1.5/ε) flops, significantly enhancing the best-known computational complexity. Additionally, we have designed an efficient algorithm to recover an optimal transport map from a solution to the reduced OT model, thereby overcoming the limitations of the reduced OT model in applications that require the transport map. Extensive numerical results presented in this paper demonstrated the superior performance of the HOT algorithm. For future research directions, we aim to design an efficient implementation of the HOT algorithms for solving the OT problems with discrete supports in the more general ℝ^d space. We also consider designing an efficient algorithm for solving the Wasserstein barycenter problem with discrete supports.







§ ACKNOWLEDGMENT



The research of Yancheng Yuan was supported by The Hong Kong Polytechnic University under the grant P0045485.  The research of Defeng Sun was supported by grants from the Research Grants Council of
the Hong Kong Special Administrative Region, China (GRF Project No. 15303720). 





































IEEEtran










































 Paper Title: The Stochastic Conjugate Subgradient Algorithm For Kernel Support Vector Machines.



Thermalization propagation front and robustness against avalanches in localized systems
    Angelo Russomanno
    
=======================================================================================




§ ABSTRACT 
Stochastic First-Order (SFO) methods have been a cornerstone in addressing a broad spectrum of modern machine learning (ML) challenges. However, their efficacy is increasingly questioned, especially in large-scale applications where empirical evidence indicates potential performance limitations. In response, this paper proposes an innovative method specifically designed for kernel support vector machines (SVMs). This method not only achieves faster convergence per iteration but also exhibits enhanced scalability when compared to conventional SFO techniques. Diverging from traditional sample average approximation strategies that typically frame kernel SVM as an “all-in-one” Quadratic Program (QP), our approach adopts adaptive sampling. This strategy incrementally refines approximation accuracy on an “as-needed” basis. Crucially, this approach also inspires a decomposition-based algorithm, effectively decomposing parameter selection from error estimation, with the latter being independently determined for each data point. To exploit the quadratic nature of the kernel matrix, we introduce a stochastic conjugate subgradient method. This method preserves many benefits of first-order approaches while adeptly handling both nonlinearity and non-smooth aspects of the SVM problem. Thus, it extends beyond the capabilities of standard SFO algorithms for non-smooth convex optimization. The convergence rate of this novel method is thoroughly analyzed within this paper. Our experimental results demonstrate that the proposed algorithm not only maintains but potentially exceeds the scalability of SFO methods. Moreover, it significantly enhances both speed and accuracy of the optimization process.




Kernel SVM, Stochastic Programming, Non-Smooth Convex Optimization




65K05, 90C15, 90C20, 90C25




§ INTRODUCTION




Kernel support vector machines (SVM), a subset of supervised learning models, are instrumental in data-driven classification tasks <cit.>. Consider a dataset of covariates represented as S={ (z_i,w_i) }_i=1^m, where the pairs correspond to features and their associated labels respectively. The primary objective is to predict the label W given a random feature Z utilizing the classifier β. In scenarios where the feature vector Z is exceedingly large or the data deviates from linear separability, it becomes necessary to apply a non-linear transformation to the feature vector Z, commonly referred to as kernel mapping ϕ. Consequently, the kernel SVM problem is formulated as:


    min_β g(β) := 1/2||β||^2+E [max{0,1-W ⟨β, ϕ(Z) ⟩}],


where the expectation E is taken with respect to the joint distribution of (Z,W). Directly addressing the problem as defined in Equation (<ref>) is infeasible due to two significant challenges: (a) the joint distribution of the pair (Z,W) is typically unknown, and (b) the kernel mapping ϕ is not directly accessible. To overcome the first challenge (a), the focus generally shifts towards resolving an empirical risk minimization (ERM) problem, specifically through a sample average approximation (SAA) approach. With regard to the second challenge (b), while direct access to ϕ may not be available, the inner product within the kernel mapping, ⟨ϕ(z_i), ϕ(z_j) ⟩, may be available.  This provides a valuable workaround <cit.>. Essentially, one uses the Representer Theorem<cit.> to express β in the form of β=∑_i=1^mα_i ϕ(z_i). This transformation enables a finite sample kernel SVM approximation as follows:


    min_αf_S(α): = 1/2⟨α, Q α⟩ + 1/m∑_i=1^mmax{0,1-w_i ⟨α, Q_i ⟩},


where α = [α_1, α_2, ..., α_m] is the decision varaible, Q ∈ℝ^m × m is the kernel matrix with Q_ij = ⟨ϕ(z_i), ϕ(z_j) ⟩ and Q_i refers to the i-th row of Q. It is significant to observe that so long as we have access to Q_ij=⟨ϕ(z_i), ϕ(z_j) ⟩, we can solve (<ref>) without having the direct knowledge of the kenel mapping ϕ.  Notably, a variety of kernel mappings, including but not limited to polynomial, Gaussian, and Laplacian kernels, often result in a positive definite matrix Q <cit.>. This characteristic implies that equation (<ref>) constitutes a convex optimization problem.

Equation (<ref>) can be interpreted as an “all-in-one” Quadratic Program (aioQP). However, the computational feasibility of solving the aioQP becomes challenging with extremely large datasets, such as those containing millions of data points. To address this problem, our paper introduces an adaptive sampling method aimed at improving the quality of the approximation locally, while enhancing the accuracy of the global approximation on an “as-needed” basis. The approximation at iteration k, utilizing a significantly smaller sample size |S_k| << m, is expressed as:


    min_αf_k(α) := 1/2⟨α, Q^k α⟩ + 1/|S_k|∑_i=1^|S_k|max{0,1-w_i ⟨α, Q_i^k ⟩},


where α = [α_1, α_2, …, α_|S_k|] is the decision variable, Q^k ∈ℝ^m × m is the kernel matrix with Q^k_ij = ⟨ϕ(z_i), ϕ(z_j) ⟩ and Q^k_i refers to the i-th row of Q^k. We will demonstrate that our proposed approach not only offers enhanced accuracy but also has greater scalability compared to Stochastic First Order (SFO) methods in tackling large-scale kernel SVM problems. To lay the groundwork, we will commence by reviewing a range of classic stochastic methods, many of which are applicable to kernel SVM.















   



 §.§ Stochastic first-order methods

SFO methods such as “stochastic gradient descent" (SGD) <cit.> or dual coordinate ascent <cit.> are very popular for solving machine learning problems. For black-box non-smooth optimization, SGD-type algorithms using secant approximations have been proposed by <cit.>. It is also worth noting that a recent survey <cit.> summarizes this type of optimization method comprehensively. Simple first-order gradient-based methods dominate the field for several convincing reasons: low computational cost, simplicity of implementation, and strong empirical results.   
For example, in order to solve (<ref>), it is common to adopt a specialized version of SGD such as Pegasos <cit.> where the subgradient calculations can be carried out very rapidly and in parallel (for a given α∈ℝ^n). <cit.> shows that the Pegasos algorithm obtains an ε_p-sub-optimal solution in time O(1/(λε_p)), where λ is the condition number of Q. This run-time does not depend on the number of variables n, and therefore is favorable when n is very large.  Finally, asymptotitic analysis via the work of Polyak and Juditsky <cit.> has shown that SFO exhibits asymptotic normality for smooth stochastic optimization and a similar result has been recently obtained in <cit.> for the case of non-smooth stochastic convex optimization problems. 

While the aforementioned methods have low computational requirements in any one iteration, there are some disadvantages: a) the convergence rate of such algorithms depend heavily on the condition number of the Hessian of the quadratic function, b) SFO methods are not known for their numerical accuracy especially when the functions are non-smooth and, c) the convergence rate mentioned in the previous paragraph only applies under the assumption that the component functions are
smooth. Since (<ref>) has non-smooth component functions, such a convergence rate is not guaranteed.   





 §.§ Stochastic methods for general non-linear  optimization

For general non-linear stochastic optimization, there are two broad classes of algorithms, most of which are stochastic  generalizations of smooth deterministic optimization algorithms. For gradient based direction finding,  stochastic analogs were studied by <cit.>, whereas stochastic analogs of trust region methods (STORM) appear in a series of papers <cit.>. From a computational standpoint, direction-finding algorithms may have faster iterations, whereas trust region methods may provide stronger convergence properties <cit.>. However, the aforementioned stochastic methods rely on the existence of gradients which may not be appropriate for non-smooth stochastic problems such as kernel SVM in (<ref>). Readers interested in the non-smooth, non-convex case may refer to <cit.>.



 §.§ Stochastic conjugate gradient method

In contrast to first-order methods, algorithms based on Conjugate Gradients (CG) provide finite termination when the objective function is a linear-quadratic deterministic function <cit.>. Furthermore, CG algorithms are also used for large-scale nonsmooth convex optimization problems <cit.>. However, despite its many strengths, (e.g., fast per-iteration
convergence, frequent explicit regularization on step-size, and better parallelization than first-order
methods), the use of conjugate gradient methods for Stochastic Optimization is uncommon. Although <cit.> proposed some new stochastic conjugate gradient (SCG) algorithms, their assumptions do not support non-smooth objective functions, and hence not directly applicable to kernel SVM problems.
The following assumptions, which are common for kernel SVM models, will be imposed throughout this paper.

 
    
  * The optimization problems (<ref>)-(<ref>) are convex and finite valued. Thus, extended real-valued functions are not permissible in our framework. As a consequence, we assume the function f and f_k possess the Lipschitz constant 0 < L_f < ∞ and their sub-differentials are non-empty for every α. Note that when the sample size is large enough, L_f will represent an upper bound of the Lipschitz constant on both f and f_k. Efficient and accurate estimation of the Lipschitz constant holds significant practical importance. Readers interested in this topic may find valuable insights in the work of <cit.>.
    
    
  * The inner product of the kernel function and the classifier is bounded, i.e., for any kernel mapping ϕ(z) and classifier β,  |⟨ϕ(z), β⟩| < M. Intuitively, this assumption indicates that the classification error can not be unbounded, which is reasonable for kernel SVM. 









In this paper, we adopt a direction-finding approach in which the direction is obtained via a specialized problem of finding a vector with the smallest norm over a line segment. This approach was originally designed for deterministic non-smooth convex optimization problems <cit.>. 

In order to solve stochastic optimization problems when functions are non-smooth (as in  (<ref>)), it is straightforward to imagine that a stochastic subgradient method could be used. In this paper, we treat (<ref>) as a stochastic piecewise linear quadratic optimization problem and propose a new stochastic conjugate subgradient (SCS) algorithm which accommodates both the curvature of a quadratic function, as well as the decomposition of subgradients by data points, as is common in stochastic programming <cit.>. This combination enhances the power of stochastic first-order approaches without the additional burden of second-order approximations. In other words, our method shares the spirit of Hessian Free (HF) optimization, which has attracted some attention in the machine learning community <cit.>. As our theoretical analysis and the computational results reveal, the new method provides a “sweet-spot” at the intersection of speed, accuracy, and scalability of algorithms (see Theorems <ref> and <ref>).

 

  





 §.§ Contributions


(1) Our methodology extends Wolfe's deterministic conjugate subgradient method <cit.> into the stochastic domain. While a straightforward application of <cit.> to stochastic programming (SP) problems using Sample Average Approximation (SAA) <cit.> might be considered, our focus is on solving Kernel SVM problems with extremely large datasets such as Hepmass with 3.5 million samples. Consequently, we employ an adaptive sampling strategy over a deterministic finite sum approximation. This approach bears some resemblance to Stochastic Decomposition (SD) <cit.>, although SD employs a "trust-region" strategy for iterative progress. More notably though, deterministic algorithms for finite-sum SAA can impose significant computational and memory demands, especially with large datasets.  This observation is corroborated by our experimental results. To address the challenges arising from non-smoothness as well as extremely large datasets, our novel algorithm integrates several disparate notions such as adaptive sampling, decomposition, conjugate subgradients, and line search techniques. We demonstrate that this amalgamation effectively approximates solutions for kernel SVM problems with extremely large datasets.

(2) The ensuing analysis of our algorithm establishes a convergence rate of O(1/ε^2) for kernel SVM, where ε denotes the desired accuracy. This contrasts with the convergence rate of O(1/√(k)) reported in <cit.>, applicable to smooth, albeit far more demanding black-box and non-strongly convex problems. 

(3) In contrast to trust-region methods, which necessitate solving a constrained convex optimization problem in each iteration, our algorithm derives a descent direction by solving a one-dimensional convex optimization problem.

(4) Our computational experiments demonstrate that, from an optimization standpoint, the solutions produced by SCS yield lower objective function values compared with SFO methods, maintaining competitive computational times for small to medium-sized datasets. More crucially, as the dataset size increases, the efficiency and accuracy of our Stochastic Conjugate Subgradient (SCS) algorithm surpasses those of algorithms such as Pegasos, a specialized first-order methods for kernel SVM. 

The structure of this paper is as follows.  In <ref>, the discussion evolves from mathematical concepts to computational algorithms. We will discuss the SCS algorithm for solving large scale kernel SVM via a combination of Wolfe's non-smooth conjugate subgradient method <cit.> and adaptive sampling which helps leverage a successive sampling methodology as in Stochastic Decomposition <cit.>. This combination not only introduces some elements of recursive estimation into kernel SVM, but also bypasses the need for constrained QP algorithms, or even the need for solving a large linear system (of equations) rapidly. In <ref>, we will present the convergence analysis of the SCS algorithm by showing that the algorithm generates a super-martingale sequence which converges in expectation. In <ref>, we will provide some technical details about its implementation and compare the computational results with its two progenitors: the Pegasos algorithm (a SFO method) and Wolfe's algorithm <cit.>, which is a deterministic non-smooth optimization method. In the context of this comparison, we will discuss some advantages and disadvantages of adopting the SCS algorithm for solving the kernel SVM problem. Finally, our conclusions are provided in <ref>. The notations used in this paper are summarized in Appendix <ref>.

















  






 
 








 
 
  



§ STOCHASTIC CONJUGATE SUBGRADIENT (SCS) ALGORITHM


In this section, we focus on solving (<ref>) using the SCS  algorithm in which we combine the non-smooth CG method <cit.> with adaptive sampling and decomposition as in stochastic programming <cit.>. In this setting, decomposition refers to a treatment of the piecewise linear (max) function and the quadratic part separately in (<ref>) to find the direction of change for the decision variables. We will first state the SCS algorithm (Algorithm <ref>) and then explain the key concepts underlying this approach.  In the following, we let Nr(G) denote the projection of “0" onto any finite collection of vectors G.  During the course of the algorithm, we will adaptively enlarge the data set which is represented within the SVM problem.  Before the start of the k^th iteration, the set of data available to the algorithm will be denoted S_k-1.  In iteration k we will include additional data points denoted by the set s_k, so that |S_k| = |S_k-1| + |s_k|. The SCS algorithm consists of four important components:


  * Sequential function approximation. In many cases we have a fixed number of data points. However, our focus is on situations where the data can be queried from an oracle sequentially. Thus, we use adaptive sampling <cit.>, 


    min_αf_k(α) = 1/2⟨α, Q^k α⟩ + 1/|S_k|∑_i=1^|S_k|max{0,1-w_i ⟨α, Q_i^k ⟩}.



It is important to note that w_i and Q_i^k are based on realization of random variables (Z,W). Consequently, f_k is a realization of the expectation functional in  (<ref>).
In the remainder of this paper, we will let the kernel matrix Q^k agree with the size of data set (|S_k|) in any iteration.  


  * Direction finding. The idea here is inspired by Wolfe's non-smooth conjugate subgradient method which uses the smallest norm of the convex combination of the previous search direction (-d_k-1) and the current subgradient (g_k). More specifically, we first solve the following one-dimensional QP


    λ_k^* = _λ∈ [0,1]1/2||λ· (-d_k-1) + (1-λ) · g_k||^2.



Then we can set the new search direction as

    d_k=-[λ_k^* · (-d_k-1)+(1-λ_k^*) · g_k] := -Nr(G_k),



where G_k={-d_k-1,g_k}, and λ_k^* denotes the optimal weight for the convex combination.  Clearly if one fixes λ = 0, then the search direction reduces to that of the subgradient method.


  * Choice of step size. This is achieved by combining concepts of deterministic Wolfe's algorithm, stochastic trust region and line search methods <cit.>. The step actually applies Wolfe's line search criteria to the function which is a sampled realization of the expectation functional. Let g(t) ∈∂ f_k-1(α̂_k + t · d_k) and define the intervals L and R.

    L={t>0  |  f_k-1(α̂_k + t · d_k)-f_k-1(α̂_k) ≤ -m_1||d_k||^2 t },
        R ={t>0  |  0 > ⟨ g(t),d_k ⟩≥ -m_2||d_k||^2 }.

The above conditions are consistent with the line-search rules in Wolfe's deterministic algorithm <cit.>. The output of the step size will satisfy two metrics: (i) Identify a set L which includes points which reduce the objective function approximation f_k-1, and (ii) Identify a set R for which the directional derivative estimate for f_k-1 is improved. The algorithm seeks points which belong to L ∩ R. The combination of L and R is called strong Wolfe condition and it has been shown in <cit.> Lemma 1 that L ∩ R is not empty.


  * Termination criteria. The algorithm concludes its process when ||d_k||<ε and δ_k < δ_min. As we will show in Theorem <ref>, a dimunitive value of ||d_k|| indicates a small norm of the subgradient g ∈ f(α̂_k), fulfilling the optimality condition for an unconstrained convex optimization problem. Additionally, a threshold δ_min is established to prevent premature termination in the initial iterations. Without this threshold, the algorithm may halt prematurely if the initial samples are correctly classified by the initial classifier α_0, resulting in ||d_k|| < ε. However, if δ_min is introduced, based on Theorem <ref>, |S_k| should be chosen such that 

    |S_k| ≥ -8log(ε/2)·(M+1)^2/κ^2δ_min^4,
   

where κ is the κ-approximation defined in Definition <ref>. This effectively mitigates any early stopping concern with high probability.



Remark 1: (i) The initial choice of step size δ_0 may require some tuning depending on the data set but it does not affect the convergence property of this algorithm.
 
(ii) According to Theorem <ref>, δ_max := ε/ζ and 

    ζ = max{4nκ,η_2,4κ/C_1(1-η_1)},

 where C_1=m_1/2n, 1< n ∈ℤ^+.

 (iii) Note that replacing d_k with an arbitrary subgradient may not ensure descent unless the function f_k-1 is smooth. Thus, simply substituting d_k with g_k and using SGD-based method may not guarantee the convergence rate shown in Theorem 
 <ref>.

 (iv) In general, optimization problems are stated with the decision variables in a fixed dimension. However, since we have designed an adaptive sampling approach, the dimension of our decision variables grows iteratively according to the increase in sample size.  
 

 



Remark 2: Algorithm <ref> will have two possible outcomes. One is it returns a suitable step size t such that t ∈ L ∩ R and t ||d_k|| ≥δ_k/n. The other is it keeps shrinking the step size until t ||d_k|| < δ_k/n. This case is captured in step 11 of Algorithm 2 and the algorithm will return t = 0. Since α_k = α̂_k-1 + t_k d_k, t=0 means the decision variable will not be updated in this iteration. In the next iteration, Algorithm <ref> will generate more samples and these new samples may improve the search direction. This whole procedure creates a subroutine to first terminate the line-search algorithm and then improve the search direction by further sampling. This phenomenon is similar to retaining the previous incumbent solution in the stochastic decomposition algorithm <cit.>.

 


§ CONVERGENCE PROPERTIES


While the SCS algorithm and the stochastic trust region method share some similarities in their proofs of convergence, the specifics of these arguments are different because the former is based on direction-finding and the latter is based on trust regions. For example, the fully-linear property in stochastic trust region method plays no role in the convergence proof of this paper. Similarly, the sufficient decrease condition (Lemma <ref>) in our algorithm relies on Wolfe's line search algorithm while the condition used for a stochastic trust region method requires the Cauchy decrease property.   

In the following subsections, Theorem <ref> shows the sample complexity of the algorithm, Theorem <ref> and the subsequent corollary show that sufficient descent can be obtained and as a result, the stochastic sequence is a supermartingale. 
In <ref>, we apply the renewal reward theorem and optional stopping time theorem <cit.> to establish the  convergence rate of the SCS algorithm. The specific results are provided in Theorem <ref>. 

Notational Guide. We will adopt the convention that random variables will be denoted by capital letter (e.g., B) while a realization will be denoted by the corresponding lower case letter (e.g., b). Also, to avoid any notational confusion, please note that ℬ will denote a ball and C_i's are ordinary constants.



 §.§ Stochastic Local Approximations
 
In this subsection, we aim to demonstrate that with an adequate number of samples, the function f_k will serve as an “effective” stochastic local approximation of f. The criteria for what constitute “effective" in this context is provided in Definition <ref>, which provides a precise metric. Additionally, Definition <ref> extends this concept to a stochastic framework. 

   A function f_k is a local κ-approximation of f on ℬ(α̂_k,δ_k), if for all α∈ℬ(α̂_k,δ_k),

    |f(α)-f_k(α)| ≤κδ_k^2.











 Let Δ_k, Â_k denote the random counterpart of δ_k,  α̂_k, respectively, and ℱ_k denote the σ-algebra generated by {ℱ_i,F̂_i,
F̂_i+1}_i=0^k-1. Notice that this definition allows us to include the value of the so-called incumbent sequences (of Stochastic Decomposition <cit.>) as part of the σ-algebra. A sequence of random models {ℱ_k} is said to be μ_1-probabilistically κ-approximation of f with respect to {ℬ(Â_k,Δ_k)} if the event 

    I_k Δ={ℱ_k is κ- approximation of f on ℬ(Â_k,Δ_k)}


satisfies the condition 

    ℙ(I_k|ℱ_k-1) ≥μ_1 > 0.














Remark 2: (i) Unlike <cit.> which imposes a generic set of guidelines for stochastic line-search methods including requirements on the gradient approximation, our setup accommodates non-smoothness of the kernel SVM problems. For this reason, our requirements simply focus on sufficiently accurate function estimation without invoking gradient approximations.
(ii) In the interest of clarity, we emphasize that the incumbent decision α̂_k has its random counterpart denoted as Â_k.



Given the relation β = ∑_i=i^|S_k|α_iϕ(z_i), we can define the function f(α) as a transformation of g(β)

    g(β)=f(α):=1/2⟨α, Q^k α⟩ +  E[max{0,1-W ⟨α, Q_i^k(Z) ⟩}],


where Q_i^k(Z) = [⟨ϕ(z_1), ϕ(Z) ⟩, ⟨ϕ(z_2), ϕ(Z) ⟩,...,⟨ϕ(z_|S_k|), ϕ(Z) ⟩]. Similarly, we can also express f_k(α) in terms of g_k(β) as follows.


    f_k(α)=g_k(β):=1/2||β||^2 + 1/|S_k|∑_i=1^|S_k|max{0,1-w_i ⟨β, ϕ(z_i) ⟩}.


  
Assume that f and f_k are Lipschitz continuous with constant L_f, for any 0 < ε <1, κ > 4L_f/δ_min. If for all i, we have |⟨β, ϕ(z_i)⟩| ≤ M and 

    |S_k| ≥ -8log(ε/2)·(M+1)^2/κ^2δ_k^4,

then

    ℙ(|f_k(α)-f(α)| ≤κδ_k^2,  ∀α∈ℬ(α̂_k,δ_k) ) ≥ 1-ε.




First, for the point α̂_k, given that β̂_k = ∑_i=i^|S_k|α̂_k,iϕ(z_i), we have 

    f(α̂_k)=g(β̂_k)=1/2||β̂_k||^2+E [max{0,1-W ⟨β̂_k, ϕ(Z) ⟩}],
        f_k(α̂_k)=g_k(β̂_k)=
            1/2||β̂_k||^2 + 1/|S_k|∑_i=1^|S_k|max{0,1-w_i ⟨β̂_k, ϕ(z_i) ⟩}.


Thus, 


    f_k(α̂_k)-f(α̂_k)     =g_k(β̂_k)-g(β̂_k)
        =1/|S_k|∑_i=1^|S_k|max{0,1-w_i ⟨β̂_k, ϕ(z_i) ⟩}-E [max{0,1-W ⟨β̂_k, ϕ(Z) ⟩}].

Using the assumption that |⟨β, ϕ(z_i)⟩| ≤ M for all i, Hoeffding's inequality implies that

    ℙ(|f_k(α̂_k)-f(α̂_k)| ≤1/2κδ_k^2) ≥ 1-2exp (-κ^2δ_k^4 |S_k|^2/8 · |S_k| · (M+1)^2 ) ≥ 1-ε,

which indicates that when 

   |S_k| ≥-8log(ε/2)·(M+1)^2/κ^2δ_k^4, 
 
we have 

    ℙ(|f_k(α̂_k)-f(α̂_k)| ≤1/2κδ_k^2 ) ≥ 1-ε.


For any other α∈ℬ(α̂_k,δ_k), if |f_k(α̂_k)-f(α̂_k)| ≤1/2κδ_k^2, then


    |f_k(α)-f(α)|    ≤ |f_k(α)-f_k(α_k)|+|f_k(α_k)-f(α_k)|+ |f(α_k)-f(α)| 
       ≤ 2L_f ·δ_k + 1/2κδ_k^2
       ≤κδ_k^2,


where the second last inequality is due to the assumption that f_k and f are Lipschitz continuous and the last inequality is because κ>4L_f/δ_min>4L_f/δ_k. Thus, we conclude that if |S_k| satisfies Equation (<ref>), then 


    ℙ(|f_k(α)-f(α)| ≤κδ_k^2,  ∀α∈ℬ(α̂_k,δ_k) ) ≥ 1-ε.





 §.§ Sufficient Decreases and Supermartingale Property
 
This section establishes the stochastic variant of the sufficient decrease condition, known as the supermartingale property, in Theorem <ref>. The proof is segmented into four distinct cases, each based on whether the conditions of κ-approximation (as defined in Definition <ref>) and ε_F-accuracy (as outlined in Definition <ref>) are met or not. Lemmas <ref> - <ref> detail the outcomes for these various situations. Following these lemmas, the supermartingale property and its comprehensive proof are presented.

 The value estimates f̂_k Δ=f̂_k(α̂_k) and f̂_k+1Δ=f̂_k(α̂_k+td_k) are ε_F-accurate estimates of f(α̂_k) and f(α̂_k+td_k), respectively, for a given δ_k if and only if 

    |f̂_k-f(α̂_k)| ≤ε_F δ_k^2,    |f̂_k+1-f(α̂_k+td_k)| ≤ε_F δ_k^2.



  A sequence of model estimates {F̂_k,F̂_k+1} is said to be μ_2-probabilistically ε_F-accurate with respect to {Â_k,Δ_k,S_k} if the event 

    J_k Δ={F̂_k,F̂_k+1 are ε_F-accurate estimates  for Δ_k }

satisfies the condition 

    ℙ(J_k|ℱ_k-1) ≥μ_2 >0.



 
Suppose that f_k is a κ-approximation of f on ℬ(α̂_k ,δ_k) and C_1 = 1/8n, where n is specified in Algorithm <ref>. If 
    
    δ_k ≤1/16nκ ||d_k||


    and Algorithm <ref> returns a suitable step size t ∈ L and t ||d_k|| > δ_k/n, then we have 
    
    f(α̂_k+t d_k)-f(α̂_k) ≤ -C_1 ||d_k|| δ_k.



Since t ∈ L and t ||d_k|| > δ_k/n, we have 

    f_k(α̂_k+t d_k)-f_k(α̂_k) ≤ -m_1||d_k||^2 t ≤ -m_1/n||d_k||δ_k.

Also, since f̂_k is κ-approximation of f on ℬ(α̂_k,δ_k), we have 

    f(α̂_k+t d_k)-f(α̂_k) 
                 = f(α̂_k+t d_k)-f_k(α̂_k+t d_k)+f_k(α̂_k+t d_k)
       -f_k(α̂_k)+f_k(α̂_k)-f(α̂_k)
       ≤ 2κδ_k^2-m_1/n||d_k||δ_k 
       ≤ -C_1||d_k||δ_k,

which concludes the proof.


  Suppose that f_k is κ-approximation of f on the ball ℬ(α̂_k,δ_k) and the estimates (f̂_k,f̂_k+1) are ε_F-accurate with ε_F ≤κ. Moreover, if 

    δ_k ≤min{(1-η_1)C_1 ||d_k||/2κ, ||d_k||/η_2},


and Algorithm <ref> returns a suitable step size t ∈ L and t ||d_k|| > δ_k/n, then the k-th iteration will be successful (Update α̂_k ←β_k, and δ_k ←min{γδ_k-1,δ_max}).



    Since t ∈ L and t ||d_k|| > δ_k/n, we have
    
    f_k(α̂_k+t d_k)-f_k(α̂_k)  ≤ -m_1/n ||d_k|| δ_k ≤ -2C_1 ||d_k|| δ_k,

    
    where the last inequality holds because m_1 ≥1/4. Also, f_k is κ-approximation of f on ℬ(α̂_k,δ_k) and the estimates (f̂_k,f̂_k+1) are ε_F-accurate with ε_F ≤κ implies that 
    
    
    ρ_k     = f̂_k+1-f̂_k/f_k(α̂_k-t d_k)-f_k(α̂_k)
        = f̂_k+1-f(α̂_k+t d_k)/f_k(α̂_k+t d_k)-f_k(α̂_k)+f(α̂_k+t d_k)-f_k(α̂_k+t d_k)/f_k(α̂_k+t d_k)-f_k(α̂_k)+f_k(α̂_k+t d_k)-f_k(α̂_k)/f_k(α̂_k+t d_k)-f_k(α̂_k)
        + f_k(α̂_k)-f(α̂_k)/f_k(α̂_k+t d_k)-f_k(α̂_k)+f(α̂_k)-f̂_k/f_k(α̂_k+t d_k)-f_k(α̂_k),

    
which indicates that

    1-ρ_k ≤ 2κδ_k/C_1||d_k||≤ 1-η_1.


Thus, we have ρ_k ≥η_1, ||d_k||>η_2δ_k and the iteration is successful. 


  Suppose the function value estimates {(f̂_k,f̂_k+1)} are ε_F-accurate and

    ε_F < η_1 η_2 C_1.

If the kth iteration is successful, then the improvement in f is bounded such that 


    f(α̂^k+1)-f(α̂_k) ≤ -C_2δ_k^2,


where C_2 Δ= 2η_1 η_2 C_1-2ε_F.



    Since t ∈ L and t ||d_k|| > δ_k/n, we have
    
    f_k(α̂_k+t d_k)-f_k(α̂_k) ≤ -2C_1 ||d_k|| δ_k,

    Also, since the iteration is successful, we have ||d_k||>η_2 δ_k. Thus, we have
    
    
    f̂_k-f̂_k+1≥η_1(f(α̂_k)-f(α̂_k+td_k)) ≥ 2η_1 C_1 ||d_k||δ_k ≥ 2η_1 η_2 C_1 δ_k^2.

    Then, since the estimates are ε_F -accurate, we have that the improvement in f can be bounded as  
    
        f(α̂_k+t d_k)-f(α̂_k) ≤f(α̂_k+t d_k)-f̂_k+1+f̂_k+1-f̂_k+f̂_k-f(α̂_k) ≤-C_2 δ_k^2. 
    


  Let the random function V_k Δ= F_k+1-F_k, the corresponding realization be v_k, 


    ζ = max{4nκ,η_2,4κ/C_1(1-η_1)}   and   μ_1μ_2 > 1/2.


Then for any ε>0 such that ||d_k|| ≥ε and ζΔ_k ≤ε, we have 

    𝔼[V_k |  ||d_k|| ≥ε, ζΔ_k ≤ε] ≤ -1/2C_1 ||d_k|| Δ_k ≤ -θεΔ_k,

where θ = 1/2  C_1.
 


First of all, if kth iteration is successful, i.e. α̂_k+1=α̂_k-td_k we have 

    v_k = f(α̂_k+td_k)-f(α̂_k).


If kth iteration is unsuccessful, i.e. α̂_k+1=α̂_k we have


    v_k = f(α̂_k)-f(α̂_k)=0.


Then we will divide the analysis into 4 cases according to the states (true/false) observed for the pair (I_k, J_k).

(a) I_k and J_k are both true. Since the f_k is a κ-approximation of f on ℬ(α̂_k ,δ_k) and condition (<ref>) is satisfied, we have Lemma <ref> holds. Also, since the estimates (f̂_k,f̂_k+1) are ε_F-accurate and condition (<ref>) is satisfied, we have Lemma <ref> holds.  Combining (<ref>) with (<ref>), we have 


    v_k ≤  -C_1 ||d_k||δ_k Δ= b_1.




(b) I_k is true but J_k is false. Since f_k is a κ-approximation of f on ℬ(α̂_k ,δ_k) ∩ X and condition (<ref>) is satisfied, it follows that Lemma <ref> still holds. If the iteration is successful, we have (<ref>), otherwise we have (<ref>). Thus, we have
v_k ≤ 0.


(c) I_k is false but J_k is true. If the iteration is successful, since the estimates (f̂_k,f̂_k+1) are ε_F-accurate and condition (<ref>) is satisfied, Lemma <ref> holds. Hence, 

    v_k ≤ -C_2 δ_k^2.



If the iteration is unsuccessful, we have (<ref>). Thus, we have v_k ≤ 0 whether the iteration is successful or not.
     

(d) I_k and J_k are both false. Since f is convex and t||d_k||<δ_k, for any g(t) ∈∂ f(α̂_k+td_k), with  the assumption in <ref> which implies that ||g(t)|| is bounded, we have 


    v_k=f(α̂_k+td_k)-f(α̂_k) ≤ g(t)^T td_k ≤ C_3δ_k.


If the iteration is successful, then

    v_k ≤ C_3 δ_k Δ= b_2 .


If the iteration is unsuccessful, we have (<ref>). Thus, we have v_k ≤  b_2 whether the iteration is successful or not. 


With the above four cases, we can bound 𝔼[V_k |  ||d_k|| ≥ε, ζΔ_k ≤ε] based on different outcomes of I_k and J_k. Let B_1 and B_2 be the random counterparts of b_1 and b_2.  Then we have 

    𝔼[V_k |  ||d_k|| ≥ε, ζΔ_k ≤ε] 
       ≤μ_1 μ_2 B_1 + (μ_1(1-μ_2)+μ_2(1-μ_1))· 0 + (1-μ_1)(1-μ_2) B_2 
        = μ_1 μ_2  (- C_1 ||d_k||Δ_k) + (1-μ_1)(1-μ_2)C_3 Δ_k.


Choose μ_1 ∈ (1/2,1) and μ_2 ∈ (1/2,1) large enough such that 


    μ_1 μ_2 ≥ 1/2   and   μ_1 μ_2/(1-μ_1)(1-μ_2)≥2 C_3/c_1||d_k||,


we have

    
    𝔼[V_k |  ||d_k|| ≥ε, ζΔ_k ≤ε]     ≤- 1/2 C_1 ||d_k||Δ_k. 
    



A quick observation of the concluding condition reveals that the requirement Δ_k ≤ε/ζ implies that the supermartingale property holds. This prompts us to impose the condition that restricts δ_max = ε/ζ. This property is formalized in the following corollary. 
 Let 

    T_ε = inf{ k ≥ 0: ||d_k|| < ε}.


Then T_ε is a stopping time for the stochastic process Â^k. Moreover, conditioned on T_ε≥ k, F_k is a supermartingale. 





    
    From Theorem <ref>, we have
    
    𝔼[F_k | ℱ_k-1, T_ε  > k] ≤ F_k-1 - ΘεΔ_k.



Hence, F_k is a supermartingale. 




 §.§ Convergence Rate
 

Building upon the results established in Theorem <ref>, where f_k is demonstrated to be a supermartingale and T_ε is identified as a stopping time, we proceed to construct a renewal reward process for analyzing the bound on the expected value of T_ε.
As highlighted in the abstract, Theorem <ref> confirms the rate of convergence to be O(1/ε^2).
To begin with, let us define the renewal process { A_l } as follows: set A_0 = 0, and for each l > 0, define A_l = inf{ m > A_l-1 : ζΔ_m ≥ε}, with ζ being specified in (<ref>). Additionally, we define the inter-arrival times τ_l = A_l - A_l-1. Lastly, we introduce the counting process N(k) = max{n: A_l ≤ k}, representing the number of renewals occurring up to the k^th iteration.


  Let 1/2 < p=μ_1 μ_2 ≤ℙ(I_k ∩ J_k). Then for all l ≥ 1,


    𝔼[τ_l] ≤p/2p-1.



First,
    
    𝔼[τ_l]     = 𝔼[τ_l | ζΔ_A_l-1 > ε] ·ℙ(ζΔ_A_l-1 > ε) +  𝔼[τ_l | ζΔ_A_l-1 = ε] ·ℙ(ζΔ_A_l-1 = ε)
       ≤max{𝔼[τ_l | ζΔ_A_l-1 > ε], 𝔼[τ_n | ζΔ_A_l-1 = ε] }.

    
If ζΔ_A_l-1 > ε, according to algorithm <ref>, 


    𝔼[τ_l | ζΔ_A_l-1 > ε] = 1.


If ζΔ_A_l-1 = ε, by Theorem <ref>, we can treat {Δ_A_l-1,...,Δ_A_l} as a random walk, and we have 


    𝔼[τ_l | ζΔ_A_l-1 = ε] ≤p/2p-1.


Combining (<ref>) and (<ref>) completes the proof. 



  Let ζ and θ be the same as in Theorem <ref> and Δ_max = ε/ζ, then

    𝔼[N(T_ε)] ≤2 ζ F_max/θε^2+ζΔ_max/ε = 2 ζ F_max/θε^2+1,


where F_max=max_i ≤ (k ∧ T_ε)|F_i|.



    We will first show that 
    
    R_k ∧ T_ε =F_k ∧ T_ε + Θε∑_j=0^k ∧ T_εΔ_j

is a supermartingale. Using (<ref>),


    𝔼[R_k ∧ T_ε|ℱ_k-1]     = 𝔼[F_k ∧ T_ε|ℱ_k-1] + 𝔼[Θε∑_j=0^k ∧ T_εΔ_j | ℱ_k-1 ]
       ≤ F_k-1 - ΘεΔ_k + 𝔼[Θε∑_j=0^k ∧ T_εΔ_j | ℱ_k-1 ] 
        = F_k-1 + 𝔼[Θε∑_j=0^(k-1) ∧ T_εΔ_j | ℱ_k-1 ] 
        = F_k-1 + Θε∑_j=0^(k-1) ∧ T_εΔ_j 
        = R_(k-1) ∧ T_ε,


where the summation in the last expectation in (<ref>) is true by moving ΘεΔ_k inside the summation so that it has one less term if k < T_ε.

If k < T_ε, then 


    |R_k ∧ T_ε| = |R_k| ≤ F_max + Θε k Δ_max.


If k ≥ T_ε, then 


    |R_k ∧ T_ε| = |R_ε| ≤  F_max + Θε T_εΔ_max.


This is also bounded almost surely since T_ε is bounded almost surely. Hence, according to (<ref>) and the optional stopping theorem <cit.>, we have


    𝔼[Θε∑_j=0^ T_εΔ_j] ≤𝔼[R_T_ε] + F_max≤𝔼[R_0] + F_max≤ 2 F_max + ΘεΔ_max.


Furthermore, since the renewal A_n happens when ζΔ_j ≥ε and N(T_ε) is a subset of {0,1,2,...,T_ε}, we have 


    Θε(∑_j=0^ T_εζΔ_j) ≥Θε( N(T_ε) ε).


Combining (<ref>) and (<ref>), we have 
    𝔼[N(T_ε)] ≤2 ζF_max + ζΘεΔ_max/Θε^2 ≤2 ζF_max /Θε^2+ζΔ_max/ε.
      



Under conditions enunciated in <ref>, we have
    
    𝔼[T_ε] ≤p/2p-1(2 ζ F_max/Θε^2+2).




First, note that N(T_ε)+1 is a stopping time for the renewal process { A_n: n ≥ 0}. Thus, using Wald's equation (inequality form) <cit.>, we have 
    

    𝔼[A_N(T_ε)+1] ≤𝔼[τ_1] 𝔼[N(T_ε)+1].

    
Moreover, since A_N(T_ε)+1≥ T_ε, we have


    𝔼[T_ε] ≤𝔼[τ_1] 𝔼[N(T_ε)+1].


Hence, by Lemma <ref> and Lemma <ref>

    𝔼[T_ε] ≤p/2p-1(2 ζF_max /Θε^2+2). 





 §.§ Optimality Condition

Theorem <ref> demonstrates that if ||d_k||<ε, the ε-optimality condition for f(x) will be satisfied. The error bound on this optimality is linked to both the sample size and the subgradient of f_k(x). Readers seeking the in-depth proof of Theorem <ref> can refer to Appendix <ref>.

 
    Let d^* = - _g ∈∂ f(α̂_k) ||g||. If k is the smallest index for which ||d_k|| < ε and |S_k| ≥-2(ε')^2/L^2logδ, then 
    
    P(||d^*|| < 4 ε + ε') ≥ 1-δ.





§ PRELIMINARY COMPUTATIONAL RESULTS
 




























    
    
    










 
  










Our computational experiments are based on data sets available at the UCI Machine Learning Repository <cit.>. Our study considers three different methods: Wolfe's algorithm <cit.>, kernel Pegasos algorithm[The computations below are based on our implementation of the Pegasos Algorithm.] and the SCS algorithm. Wolfe's algorithm uses a deterministic objective function (i.e., the SAA) and is treated as a benchmark, while Pegasos and SCS use (<ref>) as the objective function.
However, in order to compare the progress of the objective function  values for different algorithms, we will track the values of (<ref>) for all three algorithms. The values of {f(α̂_k)}_k=1^k=50 and {f(α̂_k)}_k=-50^k=-1 are shown in Figures <ref> and <ref>, where the notation k = 1, …, 50, and k=-50, …, -1 represent the solution value in the first fifty, and last fifty (respectively) iterations of each algorithms. The algorithms were implemented on a MacBook Pro 2019 with a 2.6GHz 6-core Intel Core i7 processor and a 16GB of 2400MHz DDR4 onboard memory.







Remark 3: (i) The objective function values validated for  the SCS algorithm are lower than those for the Pegasos algorithm. (ii) The SCS algorithm also converges faster than the Pegasos Algorithm. (iii) The beauty of the Pegasos algorithm is that it provides a decision rule without providing a solution to (<ref>). This decision rule provides a classifier without any reference to optimality. On the other hand, the SCS algorithm is truly an optimization algorithm  with a stopping rule in which the steps (and stopping) are guided by d_k (||d_k||<ε).

The performance of all three algorithms for different data sets are given in Table <ref>. To make the experiment more persuasive, we divide the data into training and testing parts. We obtain a classifier using the training data and apply such classifiers in the testing data set to estimate the accuracy. Here, accuracy reflects the fraction of total number of data points that have been correctly classified.  
Since Wolfe's algorithm requires the entire data set, instances with very large data sets are beyond its reach and are reported by N/A in Table <ref>. The accuracy and training time reported in the Table <ref> are based on the average of 20 independent runs for each pair (data, algorithm) using different random seeds.  









Remark 4: Conventional wisdom in ML suggests that SFO methods provide the best balance between scalability and accuracy of stochastic optimization problems. However from Table <ref>, we can see that this general impression may not only be questionable but also encourage an over-emphasis on SFO methods. In fact, the SCS algorithm performs better than the Pegasos algorithm both in accuracy and in computational speed, especially for the larger data sets in our experiment. Specifically, for the four largest data sets in our experiment, the computational times and accuracy reported for the SCS algorithm are superior to those obtained using Pegasos. Thus, the SCS algorithm gives an alternative approach which is quite competitive for Kernel SVM problems.    







§ CONCLUSION

Our approach leads to a class of online algorithms that go beyond first-order approximations and incorporates stochastic  analysis of stopping times in a direction-finding method. Specifically, it includes several features of Wolfe's method (e.g., stability and good convergence rate) as well as online aspects of Pegasos. This combination promotes improvements on both deterministic and stochastic methods that have been previously used for SVM. In contrast to the Pegasos algorithm, we find that the optimization performance of the SCS algorithm is more reliable and provides consistently lower objective values (see Figures <ref> and <ref>) in out-of-sample validation. It appears that such reliability may be difficult to achieve using first-order methods without additional fine-tuning. Although our experimental results are based on some fixed data sets, the SCS approach can be effective for classification of streaming data as well. Finally, we should emphasize that although the SCS algorithm in this paper is specifically designed for kernel SVM, it can be generalized to other non-smooth stochastic convex optimization problems, as well as constraimed problems using primal-dual methods similar to <cit.>. We also conjecture that many of the asymptotic properties (e.g., asymptotic normality) of SGD methods may also be studied for the SCS algorithm <cit.>.  


Acknowledgment. This paper is supported, in part, by a grant from AFOSR FA9550-20-1-0006.  The final phase of this paper was funded via ONR grant N00014-20-1-2077.






siamplain






§ PROOF OF OPTIMALITY CONDITION
 
In this section, we want to prove that ||d_k|| < ε implies that the ε-optimality condition of f(α̂_k), as stated in Theorem <ref>. The error bound on optimality is related to the sample size as well as the the subgradient of f_k(α̂_k). First of all, we will define a few well-known notations.


    
  * d^* = - _g ∈∂ f(α̂_k) ||g||: negative of the smallest norm subgradient of f(α̂_k).
    
  * d_k^* = - _g ∈∂ f_k(α̂_k) ||g||: negative of the smallest norm subgradient of f_k(α̂_k).
    
  * d_k = -Nr({g_k,-d_k-1}), where g_k ∈∂ f_k(α̂_k).
    
  * Subdifferential of f at z': ∂ f(z') = {q: f(z) ≥ f(z') + q^T (z-z')}.
    
  * ε-subdifferential of f at z': ∂ f_ε(z') = {q: f(z) ≥ f(z') + q^T (z-z')-ε}.
    
  * Directional derivative of f at z in direction d: f'(z;d)=max{d^Tq: q ∈∂ f(z)}.
    
  * ε-directional derivative of f at z in direction d: f_ε'(z;d)=max{d^Tq: q ∈∂_ε f(z)}.
    
  * 𝒳(z) := -min_||d|| = 1 f'(z;d) and 𝒳_k(z) := -min_||d|| = 1 f_k'(z;d).
    
  * 𝒳_ε(z) := -min_||d|| = 1 f_ε'(z;d) and d̅_k(z) = - _||d|| = 1 f_ε'(z;d).


 
    Let the tuple (m_1, m_2) satisfy the requirement in Algorithm <ref> (1/4≤ m_1 < 1/2 and 1/4≤ m_2 < m_1). If k is the smallest index for which ||d_k|| < ε, then we have ||d_k^*|| < 4 ε.



    Suppose the claim is false, then by the definition of d_k^*, we have ||g_k|| ≥ ||d_k^*|| ≥ 4ε. Thus,
    
    ||d_k||^2     = ||λ_k^* g_k-(1-λ_k^*)d_k-1||^2
        = (λ_k^*)^2||g_k||^2+(1-λ_k^*)^2||d_k-1||^2-2λ_k^*(1-λ_k^*)⟨ g_k,d_k-1⟩
       ≥ (λ_k^*)^2||g_k||^2+(1-λ_k^*)^2||d_k-1||^2


where the inequality holds because Algorithm <ref> ensures the condition R in (<ref>). 

We will now divide the analysis into 2 cases which are examined below: a) ||g_k|| > ||d_k-1|| ≥ε and ||g_k|| ≥ 4 ε  and b) ||d_k-1|| ≥ ||g_k|| ≥ 4 ε. 

(a) In this case from (<ref>) we have 

    ||d_k||^2 ≥(17 (λ_k^*)^2 -2 λ_k^* + 1 ) ε^2


Note that

    λ_k^*=⟨ g_k,d_k-1⟩ + ||g_k||^2/||d_k-1||^2+||g_k||^2+2⟨ g_k,d_k-1⟩≥-m_2||d_k-1||^2+||g_k||^2/||d_k-1||^2+||g_k||^2,


where the inequality holds because the R condition in (<ref>) ensures 0 > ⟨ g_k,d_k-1⟩≥ - m_2 ||d_k-1||^2. Thus, we claim that λ_k^* ≥2/17 because it is suffice to show


    - 17 · m_2||d_k-1||^2+17 · ||g_k||^2 ≥ 2 ||d_k-1||^2 + 2 ||g_k||^2.


This can be verified by observing m_2 < m_1 < 1/2 and ||g_k|| > ||d_k-1||. Thus, based on (<ref>) and λ_k^* ≥2/17, we have
||d_k|| ≥ε. This contradicts the assumptions of the lemma. 

(b) If ||d_k-1|| ≥ ||g_k|| ≥ 4 ε, then we have 

    ||d_k||^2 ≥(32 (λ_k^*)^2 -32 λ_k^* + 16 ) ε^2 ≥ 8 ε^2.


Thus, ||d_k|| ≥ 2 √(2)ε, which also contradicts the assumptions of the lemma.


 
    ||d^*|| = 𝒳(α̂_k) and ||d_k^*|| = 𝒳_k(α̂_k).



    According to the definition of 𝒳(α̂_k),
    
    𝒳(α̂_k)=-min_||d|| = 1 f'(α̂_k;d) = -min_||d|| = 1max_q ∈∂ f(α̂_k)d^Tq


Note that the min-max  problem (<ref>) has a stationary point


    q̅=_q ∈∂ f(α̂_k) ||q||,   d̅ = - q̅/||q̅||.


Thus, 𝒳(α̂_k) = ||q̅|| = ||d^*||. Similarly, we can also prove that 𝒳_k(α̂_k) = ||d_k^*||.



 
    If |S_k| ≥-2(ε')^2/L^2logδ, then P(𝒳_ε(α̂_k)-𝒳_k(α̂_k) ≤ε') ≥ 1-δ.



By definition, d̅_k(α̂_k) = - _||d|| = 1 f_ε'(α̂_k;d). Hence,
    
    𝒳_ε(α̂_k)-𝒳_k(α̂_k)
               ≤1/|S_k|∑_i=1^|S_k| h'(α̂_k;d̅_k(α̂_k),ω_i)-E[h_ε'(α̂_k;d̅_k(α̂_k),ω)]
        = 1/|S_k|∑_i=1^|S_k| [h'(α̂_k;d̅_k(α̂_k),ω_i)-h_ε'(α̂_k;d̅_k(α̂_k),ω_i)]
        + 1/|S_k|∑_i=1^|S_k| h_ε'(α̂_k;d̅_k(α̂_k),ω_i) - E[h_ε'(α̂_k;d̅_k(α̂_k),ω)]

Note that for each ω_i, we have h'(α̂_k;d̅_k(α̂_k),ω_i) ≤ h_ε'(α̂_k;d̅_k(α̂_k),ω_i), since ∂ h(α̂_k,ω_i) ⊆∂ h_ε(α̂_k,ω_i). Also, by Hoeffding's inequality, if |S_k| ≥-2(ε')^2/L^2logδ, then 


    P(1/|S_k|∑_i=1^|S_k| h_ε'(α̂_k;d̅_k(α̂_k),ω_i) - E[h_ε'(α̂_k;d̅_k(α̂_k),ω)] ≤ε') ≥ 1-δ,


which concludes the proof.


 (Theorem <ref>)
    First, by Theorem <ref> and Lemma <ref>, if |S_k| ≥-2(ε')^2/L^2logδ, then with probability at least 1-δ,
    
    ||d^*||-||d_k^*||=lim_ε→ 0𝒳_ε (α̂_k) -𝒳_k(α̂_k) ≤ε'.

    Combining with Lemma <ref>, we have
    
    P (||d^*|| < 4 ε + ε' ) ≥ 1 - δ.






§ NOTATION TABLES











 Paper Title: Fisher-Rao distance between truncated distributions and robustness analysis in uncertainty quantification.


Article title]Fisher-Rao distance between truncated distributions and robustness analysis in uncertainty quantification

[1,2]Baalu Belay Ketemabaalu-belay.ketema@edf.fr
2,3,4]Nicolas Bousquet 
2]Francesco Costantino
2]Fabrice Gamboa
1,2,4]Bertrand Iooss
1]Roman Sueur

[1]EDF R&D, 6 quai Watier, 78401, Chatou, France

[2]IMT, Université Paul Sabatier, 118 Route de Narbonne, Toulouse, 31062, France

[3]LPSM, Sorbonne Université, 4 place Jussieu, Paris, 75005, France

[4]SINCLAIR AI Laboratory, EDF R&D, Sorbonne Université, Palaiseau, 91120, France

Input variables in numerical models are often subject to several levels of uncertainty, usually modeled by probability distributions. In the context of uncertainty quantification applied to these models, studying the robustness of output quantities with respect to the input distributions requires: (a) defining variational classes for these distributions; (b) calculating boundary values for the output quantities of interest with respect to these variational classes. The latter should be defined in a way that is consistent with the information structure defined by the “baseline” choice of input distributions. Considering parametric families, the variational classes are defined using the geodesic distance in their Riemannian manifold, a generic approach to such problems. Theoretical results and application tools are provided to justify and facilitate the implementation of such robustness studies, in concrete situations where these distributions are truncated – a setting frequently encountered in applications. The feasibility of our approach is illustrated in a simplified industrial case study.



[
    [
    August 5, 2024
==================






§ INTRODUCTION


In applied mathematics, the importance of uncertainty quantification (UQ) stems from the considerable development of engineering and decision support methods. Generally, these methods are based on numerical models or models learned from data (not to mention hybrid ones) and provide predictive diagnoses. Robustness analyses (RA) form an important part of such studies <cit.>.
Indeed, the goal of RA is to quantify the impact of the modeling assumptions in the model input variables.
This step is crucial in many applications, for example when a certification from authorities is required <cit.>.

Very generically, RA  can be  formalized as follows. Denoting G(X)=Y a deterministic model of interest with inputs X:=(X_1,...,X_d) and output Y, X is modeled as a random variable with distribution P_X on ℝ^d (either theoretical or empirical).
Let us define  some quantity of interest (QoI) on Y. For instance, many authors in structural reliability consider a unidimensional output Y and define QoI(Y) as a function of a probability, quantile or superquantile of Y induced by P_X given G <cit.>.  Often P_X suffers from  misspecification.  
Indeed, the assessment of a statistical distribution for X from a mixture of observations and external knowledge, a current approach in many engineering fields <cit.>, is subject to 
modeling errors. 
The stacking of these errors is  generally interpreted as  so-called epistemic uncertainties <cit.>. 
An empirical vision of this problem, in a machine learning framework, is known as the domain shift problem <cit.>. 
The aim of RA  studies is therefore to check, through some robustness indices, whether QoI(Y) remains weakly dependent on this misspecification of P_X, by making P_X vary in a given variational class C, a subspace of the space P of feasible distributions on X.
Therefore, following the semantics proposed by <cit.>, RA studies are based on  distributional perturbation methods.

Defining C was the subject of many works in recent years <cit.>.
A major contribution to the field was made by <cit.>, providing considerable motivation for our work. In this paper, the inputs X_1,...,X_d are assumed independent and P_X varies only through its d marginals P_X_i in a certain family 𝒫_i of distributions on ℝ. The authors then argue that a generic RA with respect to (w.r.t.) input domain misspecification requires to handle the information geometry of each 𝒫_i 
through its Fisher-Rao (FR) distance d_i <cit.>. This requires that each 𝒫_i is 
a parametric family 𝒫_i={P_X_i,θ}_θ∈Θ_i of distributions on ℝ containing P_X_i and with well-defined Fisher information. The authors also explain that a good candidate for the variational class 𝒞 is given by a collection of concentric FR spheres Λ_iδ with radii δ∈(0,δ_max] centered at the marginal distribution P_X_i, for each i. Then, they propose to compute robustness indices based on the maximum and minimum values of a QoI over these FR spheres Λ_iδ.  

This approach promotes a more objective RA framework than the perturbations of probability distributions based on entropy criteria,  that involve inherently subjective form constraints, as generalized moment constraints proposed in <cit.>. Besides, perturbations based on the intrinsic geometry of distributions are by essence multivariate, which allows to go beyond RA based on optimal transport techniques modifying marginal percentiles  <cit.>. 

The proposal made by <cit.> is part of a slow but progressive trend in UQ to manipulate Riemannian and information geometry.
Information theory was used in <cit.> as early as 2008 
to define sensitivity indices based on input-output mutual information. One of the first use of information geometry in UQ may be found in <cit.> for building reduced order models from snapshots. Other works in this field followed since, for instance in Bayesian UQ <cit.>. Very recently, a symplectic decomposition of the Fisher information matrix is used for parameter sensitivity analysis in <cit.> while information geometric techniques are put in action for studying sloppy models in <cit.>. But up to our knowledge, the only work proposing an information geometry-based method for RA applied to numerical models, handling two-parameter distributions (but suffering from poor accuracy performance linked to Monte Carlo approximations of Fisher information) remains <cit.>. 

This relatively slow adoption by the UQ community is probably due to: (a) a lack of knowledge about the required geometrical tools; and (b) a lack of known results (except those produced in <cit.>) concerning the theoretical features and practical calculations, for concrete problems, of parametric distributions defining C.


This twofold need is the main motivation of our work. Let us discuss the results below. First, we provide an illustration of the  method and highlight the explicit correspondence between the FR spheres Λ_iδ  and distributional perturbations on selected families of 
probability distributions (normal, log-normal, Gumbel, Gamma, triangular, exponential, etc.).  Second, we obtain new explicit computations on Fisher information and Christoffel symbols for a truncated version of the normal distributions. This is of particular interest as they are often used in applications <cit.>. This allows us to derive the corresponding FR distances. Lastly, we  present more general results for handling two common classes of families of distributions, namely location-scale and pushforward families. 



More precisely, we obtain these results by taking the following path. We first propose in Section <ref> an initiation or reminder on information geometry, in a classical theoretical setting. 
This description is adapted to the special case of truncated distributions in Section <ref>, where we also detail the RA framework, explain the distributional perturbation method and illustrate the latter on the normal family in usual and truncated cases. Section <ref> gives the main algorithm for computing the FR spheres Λ_δ and illustrates the perturbation method on the families mentioned above. A numerical experiment is proposed in Section <ref>, describing the 
optimization procedure and illustrating the feasibility of robustness analyses, through the study of a simplified flood model. Besides, a discussion section closes our paper, opening towards several research perspectives aimed at extending the use of this type of robustness analysis in UQ problems.

We postpone to Appendix <ref> some technical details on the statistical procedure for estimating (with bootstrap-based confidence intervals) the robustness indices proposed in <cit.>. Proofs and additional geometric explanations are given in Appendix <ref>. Some tedious technical computations and explanations on the Fisher information and Christoffel symbols are provided in a Supplementary Material. 
Finally, Table <ref> provides the main acronyms used in this paper.





§ RIEMANNIAN GEOMETRY OF PROBABILITY DISTRIBUTIONS


Since the pioneering works by Hotelling <cit.>, Rao <cit.> and Jeffreys <cit.>, exploring the geometry of parametric families of distributions has become a very active reseach area, 
which has notably produced new interpretations of the mechanisms of statistical inference <cit.>. However, this framework often remains unfamiliar to UQ practitioners. 
Therefore basic concepts and tools of Riemannian and information geometry – which can be skipped on first reading – are briefly presented in the following paragraphs. Readers interested in more details are referred to  <cit.>, <cit.>, <cit.> and <cit.>.



 §.§ Basic notions


Riemannian manifolds are smooth manifolds M equipped with a metric tensor usually denoted g (page 38 in <cit.>, Definition 2.1). At a given base point p∈ M, this metric tensor denoted g_p is a scalar product on each tangent space T_p M of M and varies smoothly w.r.t. the base point p. This metric allows to generalize on manifolds many  geometric notions like angles, norms and distances already known on the 2D plane or 3D space. For instance, the distance between two points  and  on M is defined as the length of the shortest paths connecting  and 
    d(,):= inf_α(0) = ,  α(1) = ℓ(α)

where ℓ(α) is the length of the curve α:[0,1] → M defined as

    ℓ(α) = ∫_0^1 |α̇(τ)|_α(τ)  dτ

and |v|_ is the norm of v∈ T_ M defined as |v|_= √(g_(v,v)). This distance is called the Riemannian geodesic distance and it is intrinsic to the metric g. 

Affine connections, an important class of objects in geometry, are differential operators that allow to compute directional derivatives of vector fields. On Riemannian manifolds there exists a unique one, called the Levi-Civita connection and denoted ∇̅, that is compatible to the metric (and torsion-free). Roughly speaking, the connection ∇̅ provides a natural extension of the classical formula for differentiating a scalar product (see Definition 3.1 page 53 in <cit.>). Denoting m the dimension of M, at each point p∈ M this connection is encoded by m^2(m+1)/2 coefficients Γ̅_ij^k(p) for 1≤ i,j,k≤ m called Christoffel symbols, which depend on a local chart around p, say x^1,...,x^m. In this chart, these symbols are given by

    Γ̅_ij^k  = ∑_l=1^mg^kl/2( ∂ g_il/∂ x^j + ∂ g_jl/∂ x^i - ∂ g_ij/∂ x^l),
  
where g_ij and g^ij correspond resp. to the ij coefficient of the matrix of g and its inverse and ∂ g_ij/∂ x^l are the partial derivatives of g_ij  in the chart x^1,...,x^m. All these quantities are evaluated at point p∈ M.

Affine connections allow us to define the second derivative, or “acceleration", of a curve γ: [0,1] → M 
by seeing t ↦γ̇_t as a vector field along γ and computing its directional derivative at point γ_t and in the direction γ̇_t as

    ∇̅_γ̇_tγ̇_t ∈ T_γ_t M,

which can be thought as the derivative of γ̇ i.e. the second derivative of γ. Curves that have no “acceleration" are called geodesics and they solve a second order differential equation given by ∇̅_γ̇γ̇= 0, or in coordinates (with Einstein summation convention)

    γ̈^k + γ̇^i Γ̅_ij^k γ̇^j = 0,   for all  k=1,...,m.
  

Geodesics γ locally minimize the Riemannian distance (<ref>) since for small t>0 

    d(γ_0,γ_t) = ∫_0^t |γ̇_τ|_γ_τ  dτ = t|γ̇_0|_γ_0.

This shows that  
geodesics γ are constant speed curves,  
since the distance between γ_0 and γ_t is given by the initial speed |γ̇_0|_γ_0 multiplied by the travel time t.



 §.§ Geometry on a family of parametric probability distributions

Given a parametric family 𝒫 = {P_θ}_θ∈Θ 
of probability measures, with Θ an open set in ℝ^d, the seminal idea proposed by Hotelling and Rao <cit.> is to describe 𝒫 as a Riemannian manifold endowed with the Riemannian metric given by the Fisher Information matrix (FIM) I_θ, whose components are 

    (I_θ)_ij = 𝔼_X∼ P_θ[∂_i log p_θ(X)∂_j log p_θ(X)]

where p_θ = dP_θ/dμ is a probability density function (pdf) of P_θ w.r.t. a reference measure ν and ∂_i log p_θ(x) = ∂/∂θ^ilog p_θ(x).

A few regularity conditions are usually assumed to ensure the FIM exists and is positive semi-definite. For instance (P_θ) should be independent from θ∈Θ;  
the map θ↦ p_θ(x) should be smooth;  and 
the integration 𝔼_X∼ P_θ(·) and differentiation ∂/∂θ^i operations should be interchangeable. 
In addition, linear independence of the functions {x ↦∂_i p_θ(x)}_1≤ i≤ d for all θ guarantees the positive-definiteness of I_θ. 
More details on the required regularity assumptions are provided in <cit.>. 

Under these assumptions, the FIM I_θ is the matrix representation of a scalar product g_θ on the tangent space of 𝒫 at P_θ, usually called Fisher metric. 
Assuming the smoothness of (<ref>) w.r.t. θ, the couple (𝒫,g) becomes a Riemannian manifold. This allows to conduct geometrical analyses on 𝒫 using the intrinsic Riemannian distance (<ref>), geodesic curves (<ref>) and many more tools.
With our previous notations, the manifold is M:=𝒫 and the metric tensor is g_θ.

For a parametric family of distributions 𝒫, the Riemannian geodesic distance d given by (<ref>) is called the FR distance. This distance is intrinsic to the family 𝒫 and is invariant under reparameterization 
<cit.>. In addition, for P_θ in the vicinity of P_θ_0, the FR distance behaves like the Kullback-Leibler divergence 𝒟: 

    𝒟(P_θ | P_θ_0) = 1/2 d(P_θ,P_θ_0)^2 + o(d(P_θ,P_θ_0)^2),

where 𝒟(P_θ| P_θ_0):= 𝔼_X∼ P_θ[log(P_θ/P_θ_0(X))]. However, a global statistical interpretation for d is less obvious. 
Computing an approximation of d is still possible if the Christoffel symbols (<ref>) are available. This will be further discussed in Section <ref>.





§ ROBUSTNESS ANALYSIS AND TRUNCATED PROBABILITY DISTRIBUTIONS




 §.§ Robustness analysis framework 

Echoing the concepts and notations introduced in the first section, let us consider the RA framework <cit.> defined by:

    
  * A QoI on the stochastic variable Y:=G() where G is a deterministic forward model and
    = (X_1,…,X_d) is a random vector with distribution P_;
    
  * A family 𝒫_i, for each input X_i, of probability distributions each containing the baseline marginal P_X_i of P_, with pdf f_i;
    
  * A distance d_i on each family 𝒫_i, that will be precised in the sequel. This allows to define the δ-perturbations of f_i as any other element  f_iδ∈𝒫_i at d_i-distance δ from f_i. The collection of such elements is denoted by Λ_iδ and forms a metric sphere in 𝒫_i; 
    
    
  * A set {S_i}_1≤ i ≤ d  of robustness indices, as for example the perturbed law-based indices (PLI ; <cit.>) defined for nonzero QoI:
    
    S_i(f_iδ)  =  QoI(Y^iδ)/QoI(Y) - 1,

    where Y^iδ:=G(X^iδ) for X^iδ∼ P_X^iδ and P_X^iδ is the joint distribution of  for which only the i-th marginal pdf f_i has been replaced by some f_iδ∈Λ_iδ. In other words, P_X^iδ has density f_1⊗…⊗ f_iδ⊗…⊗ f_d.


Having set the RA framework, we now explain how the RA study of G is converted into a mathematical problem. We then precise the distance d_i used on 𝒫_i before defining truncated distributions and explaining our interest in such families for RA. The δ-perturbations for distance d_i are illustrated in the normal family for both the truncated and non-truncated case. 

The indices S_i can be seen as functions f_iδ∈𝒫_i ↦ S_i(f_iδ) ∈ℝ quantifying the robustness of QoI(Y) w.r.t. perturbations on f_i. A value close to zero for S_i(f_iδ) means that QoI(Y) is not significantly affected when perturbing f_i to f_iδ, reflecting its robustness w.r.t. the distribution of X_i. The following bounds or synthetic indices, introduced in <cit.>, quantify the worst impact of such perturbations given a δ ranging in (0,δ_max]:

    S_iδ^-:=min_f_iδ∈Λ_iδ S_i(f_iδ)     and   S_iδ^+:=max_f_iδ∈Λ_iδ S_i(f_iδ) ⋆.

Ultimately, a maximum level δ_max must be elicited to get a practically usable methodology, which remains an open topic (see Section <ref>).

In practice, the robustness indices {S_i(f_iδ)}_i cannot be computed explicitly
and must be estimated using evaluations from the model G. Details on the estimation procedure used in <cit.> are provided in Appendix <ref>. Similarly, the resolution of (⋆) cannot
be achieved through usual gradient-based optimization techniques and thus requires a dedicated
algorithm.



 §.§ Defining the perturbation sets 

 


Defining how the baseline pdf (or equivalently distribution) f_i should be  perturbed has been a research subject considered by numerous authors <cit.>. 
In this article, for the sake of genericity and following the recommendations given in <cit.>, we assume 𝒫_i is a parametric family 𝒫_i={f_i,θ}_θ∈Θ_i containing f_i.
Each 𝒫_i is endowed with the Fisher metric as explained in Section <ref> and d_i is taken as the corresponding FR distance. Accordingly, the subsets {Λ_iδ}_δ≤δ_max are defined as concentric FR spheres centered at f_i with radii δ∈ (0,δ_max]:

    Λ_iδ:= {f_i,θ∈𝒫_i |  d_i(f_i,f_i,θ)=δ}.


Let us illustrate this distributional perturbation method on 𝒫:={𝒩(μ,σ^2)}_(μ,σ)∈Θ the normal family by computing the spheres Λ_δ, the exact method for such computations is detailed in Section <ref>. Let us first point out that the FIM in the (μ,σ)-parameterization is easily obtained (see <cit.>). Indeed, we have

    I_θ = 1/σ^2[ 1 0; 0 2 ],    θ=(μ,σ) ∈Θ,

which defines the same geometry as the so-called Poincaré metric on the Poincaré half-plane ℍ:={x+iy  |  y>0} (see <cit.>, as well as Section <ref> below on location-scale families). Most geometric quantities can thus be computed explicitly. This is not the case in general. For instance the Christoffel symbols Γ̅_ij^k (<ref>) are given by

    Γ̅_12^1=Γ̅_21^1 =Γ̅_22^2= -1/σ,    Γ̅_11^2=1/2σ,    Γ̅_11^1 = Γ̅_22^1 = Γ̅_21^2 = Γ̅_12^2 = 0.


Figure <ref> gives an illustration of the concentric FR spheres and δ - perturbations of a normal density function.





 §.§ Robustness analysis for truncated distributions


Specifying the RA framework for truncated marginals of P_X appears particularly 
relevant in many engineering problems (e.g. in risk, reliability and safety issues <cit.>). 

In UQ problems, uncertain (stochastic) inputs of computer models G used for such analyses are often considered on a restriction of their support, focusing on the most critical situations (i.e., leading to critical output values Y). For instance, in <cit.> the authors consider two case studies in flood and nuclear risks, each involving computer models where the input distributions (e.g., Gumbel, normal, log-normal) are truncated on sub-domains that are most likely to lead to undesired output situations. Such studies, that typically aim to test designs, are obviously concerned with robustness analyses. Other truncated distributions can originate from the physical limitations of sensors <cit.> or to respect the physical meaning of input variables (e.g., in hydrology <cit.>, wind energy production <cit.> or wind lidar calibration <cit.>).    

Therefore, we focus on a 1D marginal component X of , and consider interval truncation. 
Assuming that the possible distributions of X are modeled by a parametric family 𝒫:={
P_θ}_θ∈Θ with densities f_θ, the truncated distributions of X∼ P_θ on an interval [a,b] are defined as the conditional probability distributions Q_θ of X∼ P_θ over [a,b] with corresponding pdf q_θ resp. given by

    Q_θ(A):=P_θ(A∩[a,b])/P_θ([a,b])    and    q_θ(x) := f_θ(x) /P_θ([a,b])1_x∈ [a,b].

We can therefore define the FIM, the FR distance and other geometric quantities on the truncated family 𝒬:={Q_θ}_θ∈Θ.

Note that beyond the interval truncation considered here, truncated marginal distributions can be defined on any measurable subset B⊂ℝ as well as for densities on ℝ^d.

Before moving on to the next section, let us illustrate this by considering a truncated version of the normal family on [a,b], denoted 𝒩_[a,b], whose pdfs are given by


    q_θ(x) = 1/N_θ1/√(2π)σexp(-(x-μ)^2/2σ^2)1_x∈[a,b],    for θ = (μ,σ),

where N_θ=(√(2π)σ)^-1∫_a^b  exp(-(x-μ)^2/(2σ^2)) dx is the normalizing constant. We proved  the following result. 

 
    The FIM for the 𝒩_[a,b] family is given by

    I_θ^[a,b] = 1/σ^2[                                      ∂_μμ_[a,b]                                      ∂_σμ_[a,b];                                      ∂_σμ_[a,b] σ^-1( ∂_σσ_[a,b]^2 + 2(μ_[a,b] - μ) ∂_σμ_[a,b]) ],

where μ_[a,b] and σ_[a,b]^2 are functions of θ=(μ,σ) given resp. by the expectation and variance of q_θ.

The exact computation of the FIM coefficients and Christoffel symbols 
is detailed in the online supplementary materials. 
FR Spheres Λ_δ and δ-perturbations in the truncated family 𝒩_[-2,2] are illustrated in Figure <ref>.



We observed that the spheres in the truncated family on the left of Figure <ref> are more distorted, compared to the spheres of the non-truncated normal family in Figure <ref>. We also saw that, numerically for the truncated pdf 𝒩_[-2,2](0,1), a radius δ larger than 0.5 seems to give a non-compact FR sphere. This surprising phenomenon is discussed in Section <ref> of the Appendix in a more general context. 

Furthermore, on the right of both figures, we show the δ-perturbations densities from the nominal density, given a value for δ. They represent distributions that are in the vicinity of the latter for the FR distance. For the truncated pdfs, the δ - perturbations of 𝒩_[-2,2](0,1) are quite distinct: some have been affected by the truncation procedure, while not much for others. In addition, some have flattened and are close to the uniform distribution on [-2,2].





§ ILLUSTRATION OF THE DENSITY PERTURBATION METHOD AND FISHER-RAO SPHERES
 

For the sake of clarity, we illustrate here the nature of δ-perturbations for  some well known parametric families, in usual and truncated situations. These δ-perturbations are obtained by computing the FR spheres Λ_δ in 𝒫_i={f_i,θ}_θ∈Θ, that constraints the optimization problem (⋆). Let us first state the following classical result from Riemannian geometry.
 
Let (M,g) be a Riemannian manifold. The sphere 𝒮(p,δ) for the geodesic distance centered at p ∈ M with small enough radius δ>0 can be identified as
    
    𝒮(p,δ)={γ(1)  |  γ geodesic s.t. γ(0)=p, γ̇(0)=v  and  |v|_p=δ}.

     

This result is not always true for large radii, see Appendix <ref> for more details and Figure <ref> for an illustration of a possible outcome. However, Proposition <ref> allows us to discretize small geodesic spheres through the following algorithm, valid in any Riemannian manifold, where K is the number of discretization points 

Hence for a parametric family 𝒫_i={f_i,θ}_θ∈Θ, we can apply this algorithm in the coordinate chart Θ to obtain K points θ_ℓ∈Θ such that {f_i,θ_ℓ}_ℓ=1,...,K discretizes the sphere Λ_δ in 𝒫_i.
In practice, since we mainly consider two-parameter families, the K tangent vectors v_ℓ with norm δ are taken at uniform angle i.e. proportional to the vectors (cos(2πℓ/K),sin(2πℓ/K))^⊺, ℓ=1,…,K. 

For each two-parameter family considered below, 
we computed the FIM and the Christoffel symbols Γ̅_ij^k (<ref>), and applied Euler method to solve the geodesic equation (<ref>). This  slightly differs from the Hamiltonian method used in <cit.>. When the Christoffel symbols are not known explicitly (Gumbel and Gamma), we approximated them using (basic) numerical integration and differentiation methods.



 §.§ Triangular distributions 

The family 𝒯_[a,b] of triangular distributions supported on [a,b] contains the densities q_m given by 

    q_m(x)= 2(b-a)^-1[(x-a)(m-a)^-11_x∈ [a,m] + (b-x)(b-m)^-11_x∈ (m,b]],   m∈ [a,b].

The Fisher information is given by
i_m = [(b-m)(m-a)]^-1  for m ∈ (a,b).
Since 𝒯_[a,b] is a one parameter family, the FR distance d can be easily computed:

    d(q_m_0,q_m_1) = |∫_m_0^m_1√(i_m) dm| = | arcsin(m_1-α/√(β)) - arcsin(m_0-α/√(β))|,

where α:=a+b/2 and β:=(a-b/2)^2. This allows to explicitly compute the FR spheres (see Appendix <ref>).
Note that the radius cannot be too large as 𝒯_[a,b] has finite diameter

    diam(𝒯_[a,b]) := sup_m_0,m_1 ∈ [a,b] d(q_m_0,q_m_1) = |arcsin(-1)-arcsin(1)| = π.

Hence, the radius δ satisfies δ≤δ_max=π/2. 
Figure <ref> gives an illustration. 




The family 𝒯_[a,b], being a (connected[A connected manifold is a set that is not formed by two or more disjoint sets i.e. it is in one piece.]) 1-dimensional Riemannian manifold, is isometric[Two spaces are isometric if there exists a smooth invertible map between them that preserves the Riemannian structure.] to an Euclidean (straight) interval, of which there exists three types: a half line [0,∞), a finite interval [0,ℓ] (for some positive ℓ∈ℝ) or the whole line ℝ. The family 𝒯_[a,b] corresponds to [0,π] since it has a diameter of π.




 §.§ Location-scale families

Location-scale families are two parameter families obtained by translating and scaling an initial smooth probability density p: ℝ→ (0,∞). Let m∈ℝ and s>0, then

    p_θ(x) = 1/s p(x-m/s),     θ=(m,s).

Here, m and s are resp. the location and scale parameters. 
Many well known families fall in this model. For instance, this is the case for the normal, Gumbel and Cauchy distributions. The following proposition shows that all location-scale families share a common geometry. 


    The FIM of a location-scale family is given by
    I_θ = [ α/s^2 γ/s^2; γ/s^2 β/s^2 ],
    where α,β,γ do not depend on θ=(m,s) and are given by integral expressions involving p and p'. 
    In addition, there exists a linear reparameterization with Jacobian matrix P^⊺ such that
    
    I_θ = P·(1/s^2[ 1 0; 0 1 ]) · P^⊺.

    Hence, we find that the Fisher metric of location-scale families is nothing more than the Poincaré metric.


The proof of Proposition <ref> is based on the spectral theorem. 
We also refer to <cit.> for a proof dealing with an even density p:ℝ→ (0,∞). Up to our knowledge, concluding the same for truncated location-scale families is an open problem.

Numerical computations of FR spheres for the Gumbel family in the truncated (Gumb_[a,b]) and non-truncated (Gumb) situations, as well as the δ-perturbations for both, are illustrated in Figure <ref>. 







 §.§ Bijective push-forward of a parametric family


Here, we work with a family of probability distributions obtained by pushing forward a reference family 𝒫 :={P_θ}_θ∈Θ on some space 𝒳 to 𝒴 by a smooth invertible map 
    h:𝒳→𝒴.
 
For instance, if we take h(x) = e^x and 𝒫 to be the family of normal distributions, then h_#𝒫 := {h_# P_θ}_θ∈Θ becomes the log-normal family. Here h_# P_θ denotes
the push-forward distribution. Set 

    R_θ := h_# P_θ

and let ℛ:= {R_θ}_θ∈Θ be the push-forward family. Our goal is to determine the FIM of a truncated version of ℛ on some subset of 𝒴. Let us first remind the following theorem
 If 𝒳,𝒴 are open subsets of ℝ^d and h:𝒳→𝒴 is smooth and invertible, then for any regular parametric family 𝒫 on 𝒳 the pushforward family ℛ:=h_#𝒫 has the same FIM as 𝒫.


This result directly implies that geodesics and FR spheres in 𝒫 are the same in ℛ (i.e., they are isometric). As an example, this holds for the normal and the log-normal families.

Now, we would like to compute the FIM of a truncated version of ℛ. Let us first fix some notations. From now on, we denote B⊂𝒴 the truncation domain (a measurable subset) for ℛ and  A:= h^-1(B)⊂𝒳 its preimage. 
We denote the truncated family of 𝒫 on A (resp. of ℛ on B) as 𝒫_A :={P_θ^A}_θ∈Θ
(resp. ℛ_B :={R_θ^B}_θ∈Θ). 
In addition, let I_θ^A (resp. K_θ^B) denotes the FIM of 𝒫_A (resp. of ℛ_B).

Since we want to compute the FIM of ℛ_B={(h_#P_θ)^B}_θ∈Θ, if we apply the previous theorem on 𝒫_A={P_θ^A}_θ∈Θ with the restriction of h to A, we get that 𝒫_A and 


    h_#𝒫_A := {h_#(P_θ^A)}_θ∈Θ

have the same FIM. 

The following result states that h_#𝒫_A and ℛ_B are the same, 
the proof being obvious and intuitive. 

Under the previous notations, it comes  
    (h_#P_θ)^B = h_#(P_θ^A),     for all θ in Θ.

Therefore ℛ_B and h_#𝒫_A are the same family.
Moreover, ℛ_B and 𝒫_A have the same Riemannian structure since their FIM coincide:

    I_θ^A = K_θ^B,     for all θ in Θ.



This result allows us to deduce K_θ^B, the FIM of the truncated push-forward family ℛ_B on 𝒴, if we know I_θ^A, the FIM of the initial truncated family 𝒫_A on 𝒳 where A =h^-1(B), see Example <ref>. The following diagram summarizes our considerations:



(𝒳, P_θ) [r, "tr. on A"] [d,  " h_#"]        (𝒳, P_θ^A) [d, " h_#"] 

(𝒴, R_θ) [r, "tr. on B" ]         (𝒴,R_θ^B)



The previous lemma tells that the push-forward operation and the truncation operation commute. In addition, the standard push-forward theorem shows that the left-down arrow preserves the Fisher information, and the lemma guarantees that the Fisher information is preserved by the right-down arrow.

 
    For the family log𝒩 of log-normal distributions
    
    r_θ(x) =1/xσ√(2π)exp(-(log x -μ)^2/2σ^2)1_x>0,     for θ = (μ,σ),

    Lemma <ref> implies that the FIM of log𝒩_[a,b], the truncated version of this family on [a,b], is given by
    I_θ ^[log a, log b],
    where I^[α,β] is the FIM of 𝒩_[α,β] provided in Proposition <ref>. In particular, the Levi-Civita Christoffel symbols of 𝒩_[log a,log b] and log𝒩_[a,b] are identical 
    as well as their geodesics and FR spheres on the upper halfplane. Figure <ref> gives an illustration of this result.




A natural question would be to know whether the previous lemma still applies when h:𝒳→𝒴 is a sufficient statistic for the family 𝒫. Indeed, in this case, the Fisher metric of 𝒫_A and ℛ_B are the same. The proof, which is obvious, is provided in Appendix <ref>.



 §.§ Exponential families


Exponential families are wide families of probability distributions, with normal, log-normal, gamma, binomial, Poisson and geometric distributions amongst their most notable members. The densities f_θ of an exponential family with reference distribution ν on ℝ^m are given, for θ∈Θ, by

    f_θ(x)=e^x·θ - ψ(θ),    x∈ℝ^m, θ∈Θ.

Here, we denote x·θ the Euclidean scalar product on ℝ^m, the function ψ is the log-Laplace transform of ν and Θ is the domain of ψ. The FIM for exponential families is given by the Hessian of ψ <cit.>, hence exponential families can be seen as Hessian manifolds <cit.>.

A truncated exponential family is built from an initial exponential family truncated on an interval [a,b], here we take m=1. These truncation bounds are usually considered as parameters <cit.>, and therefore these families are not regular and the FIM is not necessarily well defined. In this subsection, we consider truncated exponential families on an interval [a,b] where a,b are fixed  – a special case of truncated distributions studied by <cit.>. 

The truncated densities with normalizing constant N_θ also form an exponential family since

    N_θ^-1f_θ(x)1_x∈ [a,b]dν(x) = e^x·θ - ψ(θ) - log N_θdν_[a,b](x),

where ν_[a,b](·):=∫_·1_[a,b] dν is the reference distribution and ψ̃(θ):= ψ(θ) + log N_θ is the log-Laplace transform of ν_[a,b]. Hence the FIM of this family is given by the Hessian of ψ̃. 
See Figure <ref> for illustrations in the non-truncated case, and refer to Appendix <ref> and the Supplementary Material for details on computations.






§ NUMERICAL EXPERIMENTS: ROBUSTNESS STUDY OF A FLOOD MODEL



In this section, the RA methodology discussed in Section <ref> to is applied for studying the robustness of an analytic flood model. This model has been studied in previous articles such as <cit.>. The code we developed is available at https://github.com/baalub/Truncated-distributions.githttps://github.com/baalub/Truncated-distributions.git





 §.§ Context

The flood model computes a river's maximal annual water level H assuming constant and uniform flow rate. This is useful information for predicting if flooding will occur. The model is defined as 

    H := G(K,Q,Z_m,Z_v) = Q^0.6(300K √(Z_m - Z_v/5000))^-0.6,

where Q∈ [0,3000] is the maximal annual flow rate, K ∈ [15,90] is a roughness (Strickler-Manning) coefficient and Z_m ∈ [54,56] and Z_v ∈ [49,51] are resp. the upstream and downstream heights of the river. Figure <ref> gives an illustration of the river and Table <ref> summarizes the baseline distributions, selected from legacy data and  expert knowledge.






The output H corresponds to the river depth and flooding will occur if H + Z_v≥ Z_d, where Z_d is the height of the dyke. The QoI we look at is the 90% quantile of the output H.
We perform the RA method presented above to determine which inputs are robust w.r.t. input density perturbations. As indicated in Section <ref>, we compute 

    min_f_iδ∈Λ_iδS_i(f_iδ)     and   max_f_iδ∈Λ_iδS_i(f_iδ) ⋆⋆

where S_i is an estimator of S_i (see Appendix <ref> for details), i∈{Q,K,Z_m,Z_v} and δ∈{0.1,0.2,...,1}. Here, we picked δ_max=1 for practical reasons. Three-dimensional plots of the quantile estimator on the concentric FR spheres are provided on Figure <ref>. Besides, Figure <ref> highlights the maximum and minimum values of S_i on concentric FR spheres, as a function of δ. The confidence intervals accompanying this result are built using bootstrap techniques, detailed in  Appendix <ref>. 







 §.§ Optimization procedure

Computing the bounds in (⋆⋆) for the triangular inputs Z_m and Z_v was not challenging since in these cases the FR spheres Λ_iδ were made up of two points. For the truncated normal K and truncated Gumbel Q inputs, our approach for computing (⋆⋆) is to discretize each sphere Λ_iδ, for δ∈{0.1,0.2,...,1}, into 100 points. We then optimize the estimator S_i of the robustness index. We use this approach because the estimator S_i is piecewise constant (see Appendix <ref>) and thus a gradient-based method cannot be used. Improving this optimization procedure is a subject of current work, evoked further in the Discussion section.  

Discretizing the concentric FR spheres takes around 8 minutes for each input K and Q. Evaluating S_i on each point of the discretization (1000 points) for a sample of size N=10^4 took around 3 hours in total for all the inputs K,Q,Z_m,Z_v on an 11^th gen Intel(R) Core i5-1135G7 @2.40Hz computer. The time required for the estimation of the robustness index highly depends on the choice of the QoI. Usually, estimating the expectation or a probability threshold of the output takes significantly less time than a quantile or a superquantile.



 §.§ Analysis of the numerical results


From Figure <ref>, we observe that the output of the model G is sensitive to density perturbations on K and Q of level of at most δ_max = 1. For the inputs Z_m and Z_v, it seems that perturbing their initial densities does not affect much the output.

Figure <ref> highlights that as the perturbation level δ increases a minimizer of S_Q has a tendency to give more weight to points in [0,3000] that are close to 0. 
This is not surprising since G is increasing as a function of Q. 







This RA result for the flood model is comparable to previous RA studies in papers such as <cit.>. Indeed, in this paper, the inputs Z_m and Z_v were also the most robust to input density perturbations. However, in contrast to our case, the input Q was the least robust in <cit.>. This may be due to the difference in the baseline distributions for Q and K.



§ DISCUSSION
 

In this paper, we have explained the RA methodology based on the FR distance on families of probability distributions. This distance allowed us to define δ - perturbations of densities in some parametric family. We then illustrated the feasibility of this δ - perturbation method in some classical families of distributions both in the truncated and non-truncated case. This required either an explicit formula for the FR distance or an approximation of the Christoffel symbols of the corresponding Fisher metric. This showed that the FR distance, which might seem hard to compute at first glance, can be handled rather easily and that the FR spheres can be approximated with good precision (at least for small radii). We also proved a result that allows us to deduce the Fisher metric for some truncated families obtained by a bijective push-forward operation of another truncated family (Lemma <ref>). Finally, we applied the RA method to an analytic flood model to illustrate its applicability.

Compared to the previous RA methods, the advantage offered by this approach is that it gives a mathematically rigorous and objective way for defining density perturbations. Nevertheless, the method leads to obstacles that have to be overcome in practice.

First, the optimization method used for problem (⋆⋆) requires to discretize the FR spheres into finite points and then to optimize S_i on these points. 
The choice for such an optimization procedure is related to the estimation method for S_i, but obviously it does not scale well to higher dimensions. One solution would be to regularize S_i and perform Riemannian gradient descent on the FR spheres. Moreover, one can also consider the same optimization problem constrained to the FR ball with radius δ_max instead of on each concentric spheres, which reduces to solving a single optimization problem and allows to obtain the maximum and minimum value of S_i for all δ≤δ_max.

Second, we only considered the case where the QoI on the output Y is a quantile for the flood model. However, QoIs for which (⋆) becomes an expectation optimization problem can be directly tackled with stochastic optimization algorithms, which allows to avoid optimizing an estimator of the objective function. This will be the topic of a future work.

Another central question is obviously to set a maximum value δ_max in practice. To do so, let us first note that this maximum value can be imposed by the intrinsic geometry of the family of distributions (see the example of the family of triangular distributions). In contrast, other families with infinite diameter (location-scale families for instance) do not impose such constraints and spheres of arbitrarily large radii are well defined. Furthermore, some families with infinite diameter can contain spheres that are not compact. This may indicate the existence of distributions outside the family that are at finite FR distance. Thus, this can be useful for defining a possible δ_max at a density f_θ_0

    δ_max:= sup{δ>0  |  𝒮(f_θ_0,δ)  is compact},

where 𝒮(f_θ_0,δ) is the FR sphere at f_θ_0 with radius δ. 

This non-compactness property of spheres is linked to geodesic completeness of the family, through the Hopf-Rinow theorem <cit.>. 
Up to our knowledge, completeness of families of distributions is not well studied in the literature (even less for truncated ones). Hence determining an intrinsic δ_max in this way for a specific family is a challenging open problem. Actually δ_max is only defined for a point in a specific family. Note further that compactness of FR spheres Λ, coupled with a continuity assumption on f_iδ∈Λ↦ S_i(f_iδ), is useful for guarantying existence of solutions to (⋆). 

Moreover, choosing a δ_max should also ensure that distributions within the FR sphere defined in this way have “practical" plausibility with regard to the UQ problem. For example, we could try to impose that f_iδ_max should correspond to the least informative plausible distribution (e.g., a uniform distribution) or the most penalizing possible distribution (in the sense of least favorable prior distributions in the Bayesian framework <cit.>).

Another approach, extrinsic to the geometric problem and which would enable us to compare different perturbations approaches for RA, could be to interpret δ_max as the result of an inverse problem. With a view to testing robustness against a limit risk defined, for example, as a limit output α-quantile q_lim(α) for Y, with 0<α≪ 1, as in <cit.>, consider

    δ_i,max    =    min_δ_i|q_lim(α) - q(α ; δ_i) |

where q(α ; δ_i) is the α-quantile of Y_i=G(X_1,…,X_i,δ_i,…,X_d) where the X_j∼ f_j for j≠ i and X_i,δ_i∼ f_iδ_i. Such a definition  would give a practical, model-agnostic and more easily interpretable meaning to the different values of δ_max. It would, however, require the resolution of d inverse problems. This could be computationally time-consuming. 

While the methodology has been illustrated in situations where the components of  are independent, it is easily adaptable to cases of stochastic dependence modeled by copulas. These are amongst usual implementation choices in UQ studies (e.g., <cit.>). Nevertheless, adaptations are needed to handle other common types of joint modeling, such as hierarchical or graph-based approaches.







§ ESTIMATION PROCEDURE FOR A GIVEN QOI
 
 


 §.§ Quantile estimation procedure


For all forward models G, a QoI of the output Y often cannot be computed explicitly. Nevertheless, an estimate can be provided by using a statistical estimation procedure. We describe here how we construct this estimator, following  the importance sampling approach suggested by <cit.>. For better reading and to fit our experiments, we consider that all inputs X=(X_1,...,X_d)
are independent, but generalizing the approach to non-independent inputs is straightforward. 

By the law of large numbers, the estimator F_iδ (t) converges almost surely to the cumulative distribution function of Y^θ for all t∈ℝ. The quantile estimator q̂_iδ of q^α(Y^iδ) is built by plug-in:

    q̂_iδ := inf{ t ∈ℝ |  F_iδ (t) ≥α}.

Hence q^α(Y^iδ) can be estimated for all i and for all δ given only a sample of P_X. The estimator of the PLI robustness index S_i (Equation (<ref>)) is then given by

    S_i(f_iδ) := q̂_iδ/q̂_i - 1,

where q̂_i is the empirical quantile estimator of q^α(Y) where Y:=G(X) and X∼ f_1⊗···⊗ f_d. The asymptotic properties of S_i are studied in <cit.>.

The estimators F_iδ(t) and q̂_iδ can be seen as functions of the parameter θ∈Θ_i. The space Θ_i is a parameter space of the family 𝒫_i of possible perturbation densities of f_i. Since the estimator F_iδ(t) is piecewise constant as a function of t, jumping on the sample points Y_j, and as a function of θ, its generalized inverse q̂_iδ is also piecewise constant taking values in {Y_1,...,Y_N} 

    θ∈Θ_i →q̂_iδ∈{Y_1,...,Y_N}.

Similarly since q̂_i also takes finite values, this shows that the PLI estimator S_i also takes finite values and 

    min_f_iδ∈Λ_iδS_i(f_iδ)     and   max_f_iδ∈Λ_iδS_i(f_iδ)

is a discrete optimization problem. Therefore, a gradient-based optimization method cannot be used directly.



 §.§ Bootstrap confidence intervals

We used a bootstrap procedure to compute confidence intervals for

    min_f_iδ∈Λ_iδ S_i(f_iδ)     and   max_f_iδ∈Λ_iδ S_i(f_iδ).

Focusing only on the maximization problem, for each δ, we approximated the maximum value of S_i on Λ_iδ by
max_f_iδ∈Λ̅_iδS_i(f_iδ),
where Λ̅_iδ is a discretization of the FR sphere Λ_iδ (100 points in our simulation). Next we compute the maximum points f_iδ^* on the discrete FR sphere Λ_iδ directly. This means that 
S_i (f_iδ^*) = max_f_iδ∈Λ̅_iδS_iδ,
and since S_i (f_iδ^*) is an estimator built on the sample {Y_1,...,Y_N}, we can use a bootstrap procedure to obtain a confidence interval for max_f_iδ∈Λ̅_iδS_i (f_iδ). This is expected to give a confidence interval for 
max_f_iδ∈Λ_iδS_i (f_iδ),
provided Λ̅_iδ is refined enough.



§ MORE DETAILS ON GEOMETRY AND PROOFS



 §.§ Computation of Fisher-Rao spheres using geodesics
 

As explained in Section <ref>, the sphere approximation algorithm is based on Proposition <ref>. But this method may fail for large radii. The reasons of this failure may be either the incompleteness of the manifold or finiteness of the injectivity radius, which will be defined in the following.  



  §.§.§ Completeness

A connected Riemannian manifold (M,g) is geodesically complete if all geodesics γ are defined for all time t∈ℝ. This is equivalent to the completeness of M as a metric space endowed with the geodesic distance d, by the Hopf-Rinow theorem <cit.>. This theorem additionally states that a connected manifold M is geodesically complete if and only if every closed and bounded set in M is compact. This implies that if the manifold is not complete, then there may exist some spheres (which are always closed and bounded) that are not compact. Note that, around a given point, spheres with small enough radii are always compact. Therefore, the non-compactness of spheres may only arise for large enough radii.

For the sphere approximation algorithm from Section <ref>, approximating non-compact spheres can lead to numerical issues. This is because for a given chart on which the geodesic equation is numerically solved, depending on the initial conditions, the solution (the geodesic) γ(t) may blow-up in finite time t_*. Hence, it will be harder for the approximated solution γ̅(t) to be accurate as t gets closer to t_*.
 


  §.§.§ Injectivity radius

Let (M,g) be a connected Riemannian manifold. If the injectivity radius ι_M=∞, then Proposition <ref> can be used for all δ>0. On the contrary, if ι_M<∞, the radius δ has to be small enough or the manifold needs to verify some additional properties in order for Proposition <ref> to hold for all δ>0. More precisely, let us first define the exponential map at p∈ M

    exp_p  :     Ω_p ⊂ T_p M ⟶ M 
        v ⟼γ_p,v(1).

Here, γ_p,v is the unique geodesic verifying the conditions γ_p,v(0)=p and γ̇_p,v(0)=v. The set Ω_p is defined as the set of vectors v ∈ T_p M such that γ_p,v is defined until time t=1. It is a neighborhood of the null vector 0_p in T_p M. In addition, the exponential map exp_p is a local diffeomorphism in a small neighborhood around 0_p. 
The question whether exp_p is a global diffeomorphism is adressed using the notion of injectivity radius. 

Let us first denote B(p,r) and 𝒮(p,r) resp. the open ball and sphere in M centered at p with radius r for the geodesic distance. Further, let us denote b(0_p,r):= {v ∈ T_p M  |  |v|_p< r} and s(0_p,r):= {v ∈ T_p M  |  |v|_p=r} resp. the open ball and sphere centered at 0_p with radius r in T_p M. Since exp_p is a local diffeomorphism around 0_p for all p in M, there exists r_p>0 such that for r≤ r_p we have exp_p : b(0_p,r_p) → B(p,r_p) is a diffeomorphism. We also have that exp_p(s(0_p,r_p)) = 𝒮(p,r_p) and exp_p(b(0_p,r_p)) = B(p,r_p). The injectivity radius ι_p at point p is defined as the largest radius r for which exp_p is a diffeomorphism onto the geodesic ball B(p,r)⊂ M. That is,

    ι_p := sup{ r > 0  |  exp_p  is a diffeomorphism on  b(0_p,r) }.

Further, the injectivity radius of the manifold is defined as ι_M:= inf_p∈ Mι_p. 

If ι_M is infinite, then exp_p : T_p M → M is a global diffeomorphism (for any p in M). This implies that for all r>0 the spheres 𝒮(p,r) in M can be identified with

    𝒮(p,r)=exp_p(s(0_p,r))={γ(1)  |  γ geodesic s.t. γ(0)=p, γ̇(0)=v  and  |v|_p=δ}.

Hence, this justifies the use of the initial algorithm from Section <ref> for all radius r>0. This also implies that the manifold is complete.

Now, if the injectivity radius ι_M<∞, then there exists p ∈ M such that for r≥ι_p the exponential map exp_p fails to be a diffeomorphism from b(0_p,r) to B(p,r). 
The good news is that if the manifold is complete, the exponential map is always onto 𝒮(p,r) 

    𝒮(p,r) ⊂exp_p( s(0_p,r)).

This guaranties that the sphere approximation algorithm from Section <ref> gives at least some points that approximate 𝒮(p,r) (but also some that do not necessarily fall on 𝒮(p,r)). This is true since M is assumed to be connected.
In general, it is not obvious to determine whether ι_M is infinite or not. Some curvature conditions (Hadamard theorem, see <cit.>) lead to lower bounds for ι_p. Nevertheless, in practice these conditions are hard to verify. 



 §.§ Proofs
 


The fact that ℛ_B and 𝒫_A have the same FIM is a direct consequence of the first assertion and the standard pushforward theorem. Indeed, if we assume ℛ_B=h_#𝒫_A, then it implies that they have the same FIM. Moreover, since 𝒫_A and h_#𝒫_A have the same FIM (standard pushforward theorem), the second assertion is obvious.

Now let us prove the first assertion. Let C be a measurable subset on 𝒴. We have on the one hand

    h_#(P_θ^A)(C) = P_θ^A(h^-1(C)) = P_θ(h^-1(C)∩ A)/P_θ(A).
 
On the other hand, 
we have

    (h_#P_θ)^B(C) = h_#P_θ(C∩ B)/h_#P_θ(B)=P_θ(h^-1(C∩ B))/P_θ(h^-1(B))   =P_θ(h^-1(C)∩ h^-1(B))/P_θ(A)
       =P_θ(h^-1(C)∩ A)/P_θ(A).

This allows to conclude the proof. Note that h does not need be bijective in the proof.



Under the previous notations, if we now assume that h:𝒳→𝒴 is a sufficient statistic for 𝒫, then the first assertion of Lemma <ref> still holds. Note that in this case, h is also a sufficient statistic of a truncated version of 𝒫 i.e. 𝒫_A (see <cit.>). In addition, the Fisher-Neyman factorisation theorem (see <cit.>) implies that

    p_θ(x) = u(x) r_θ(h(x)),

where p_θ and r_θ are resp. the pdf of P_θ and R_θ:=h_# P_θ and u is a non-negative function on 𝒳. Let us denote resp. p_θ^A and r_θ^B the truncated pdfs of p_θ on A and r_θ on B. We can compute the FIM of K_θ^B of the truncated family ℛ_B={R_θ^B}_θ∈Θ,

    (K_θ^B)_ij   = ∫_𝒴(∂_i log r_θ^B(y))(∂_j log r_θ^B(y)) dR_θ^B(y) 
       = ∫_𝒳(∂_i log r_θ^B(h(x)))(∂_j log r_θ^B(h(x))) dP_θ^A(x) 
       = ∫_𝒳(∂_i logp_θ(x)/u(h(x))R_θ(B))(∂_j logp_θ(x)/u(h(x))R_θ(B)) dP_θ^A(x)  
       = ∫_𝒳(∂_i logp_θ(x)/P_θ(A))(∂_j logp_θ(x)/P_θ(A)) dP_θ^A(x) 
       = ∫_𝒳(∂_i log p_θ^A(x))(∂_j log p_θ^A(x)) dP_θ^A(x) 
       = (I_θ^A)_ij,

where (B1) is obtained from the transfer theorem, (B2) and (B4) are resp. obtained by the definition of r_θ^B and p_θ^A as well as the Fisher-Neyman theorem and (B3) holds because u(h(x)) does not depend on θ and R_θ(B)=P_θ(A).    



    Let p_θ:ℝ→ (0,∞) be defined by 

    p_θ(x) = 1/s p(x-m/s),

where θ=(m,s) ∈ℝ× (0,∞) is the parameter. Here, p is the initial density. Let us compute the FIM for this family. First, taking the partial derivatives of log p_θ we get

    ∂_m log p_θ(x) = -1/s p'( x-m/s)/p( x-m/s)      and      ∂_s log p_θ(x) =( -x-m/s^2) p'( x-m/s)/p( x-m/s)  -1/s.


Now, we can compute 

    (I_θ)_11 = ∫_ℝ(-1/s p'( x-m/s)/p( x-m/s))^2 1/sp( x-m/s) dx
        = 1/s^2α
 

    (I_θ)_12   = ∫_ℝ(-1/s p'( x-m/s)/p( x-m/s)) ( ( -x-m/s^2) p'( x-m/s)/p( x-m/s)  -1/s) 1/sp( x-m/s) dx= 1/s^2γ


    (I_θ)_22 = ∫_ℝ(( -y/s) p'(y)/p(y)  -1/s)^2 1/sp(y) s dy
        =1/s^2β.

Here α,β,γ are given by

    α=∫_ℝ( p'(y)/p(y))^2 p(y) dy,    β=∫_ℝ(yp'(y)/p(y)  + 1)^2 p(y) dy

and 

    γ = ∫_ℝ(-p'(y)/p(y)) (  -yp'(y)/p(y) - 1 ) p(y) dy.

We refer to <cit.> for a similar computation in higher dimension. We assume that these quantities are well defined and finite. In this case, the FIM for the location-scale family of p is given by 

    I_θ = 1/s^2[ α γ; γ β ].

The matrix [ α γ; γ β ] can be diagonalized on an orthonormal basis of eigenvectors

    I_θ =  Q_αβγ(1/s^2[ λ_1   0;   0 λ_2 ]) Q_αβγ^⊺ = Q_αβγ[ √(λ_1)      0;      0 √(λ_2) ](1/s^2[ 1 0; 0 1 ])
        [ √(λ_1)      0;      0 √(λ_2) ]Q_αβγ^⊺.
 
Here, Q_αβγ is an orthogonal change-of-basis matrix and λ_1,λ_2≥0 are the eigenvalues and they depend on α,β,γ. By denoting P_αβγ := [ √(λ_1)      0;      0 √(λ_2) ]Q_αβγ we thus get 

    I_θ = P_αβγ(1/s^2[ 1 0; 0 1 ]) P_αβγ^⊺.

This proves that the FIM is, up to reparameterization, the Poincaré metric on ℍ := {(x,y) ∈ℝ^2  |  y >0}.



For any location-scale family, the change-of-basis matrix P_αβγ depends on α,β,γ therefore having a good approximation of these integral quantities will allow us to approximate P_αβγ. This will in turn allow us to compute geometric quantities (distance, curvature, angle,...) for the location-scale family using only the Poincaré metric. Indeed, these quantities are already known for the latter.
    


 §.§ Explicit Fisher-Rao distance computation in the triangular and exponential distributions
 
The triangular and exponential distribution families only depend on a single parameter. We can thus easily compute the Fisher information and the Fisher-Rao distance exlicitly.


  §.§.§ Triangular family

The triangular family 𝒯_[a,b] of densities q_m on the interval [a,b] is given by

    q_m(x)= 2(x-a)/(b-a)(m-a)1_x∈ [a,m] + 2(b-x)/(b-a)(b-m)1_x∈ (m,b],

where m∈(a,b) is the parameter. The Fisher information i_m is computed as follows: 
    
  * for x∈ [a,m], 
    ∂_m log q_m(x) =  - ∂_m log(m-a) = -1/m-a,
    
  * for x ∈ (m,b],
    ∂_m log q_m(x) = -∂_m log(b-m) = 1/b-m.

Hence, we have

    i_m    = ∫_a^b (∂_m log q_m(x))^2 q_m(x) dx 
       = ∫_a^m 1/(m-a)^22(x-a)/(b-a)(m-a) dx + ∫_m^b 1/(b-m)^22(b-x)/(b-a)(b-m) dx 
       = 1/(m-a)^3(b-a)(m-a)^2 + 1/(b-m)^3(b-a)(b-m)^2
       = 1/b-a(1/m-a + 1/b-m) = 1/(m-a)(b-m).

As {q_m : m ∈ (a,b)} is a one parameter family, every geodesic correspond to a segment γ(t) = m_0 + tv, up to reparameterization. But, since the integral in the definition of the Fisher-Rao distance 
is invariant under reparameterization, we can explicitly compute the distance between two points using the segment γ(t) = m_0 + tv. To begin with, let α:= (a+b)/2 and β := (a-b/2)^2. We have (b-m)(m-a) = -(m-α)^2 + β. Then, for m_0,m_1 ∈ (a,b), we compute 

    d(m_0,m_1)    =| ∫_m_0^m_1dm/√(-(m-α)^2 + β)| = |∫_m_0^m_1dm/√(β)√(-(m-α/√(β))^2 + 1)|  
       u:=m-α/√(β)=|∫_m_0-α/√(β)^m_1-α/√(β)du/√(1-u^2)| 
       =| arcsin(m_1-α/√(β))-arcsin(m_0-α/√(β))|.

This means that the Fisher-Rao sphere centered at m_0 ∈ (a,b) with small radius δ>0, is explicitly determined by the following two points

    m_+ = √(β)sin(δ + arcsin(m_0-α/√(β)) ) + α,    m≥ m_0, 
    
    m_- = √(β)sin(arcsin(m_0-α/√(β)) - δ) + α,    m < m_0.





  §.§.§ Exponential distributions family


The exponential family ℰ has densities given by

    p_λ(x) = λ e^-λ x1_x>0,    λ∈ (0,∞).

The Fisher information is given by i_λ = λ^-2 (see <cit.>, page 25). Similar to the triangular family, the Fisher-Rao geodesic distance in this case is

    d(λ_0,λ_1) = | ∫_λ_0^λ_1√(1/λ^2) dλ| =  |log(λ_1/λ_0) |,    λ_0,λ_1>0.

So that, the Fisher-Rao sphere centered at λ_0 with radius δ is {λ_-,λ_+} given by

    λ_+ = λ_0 e^δ,    λ_+≥λ_0,
    λ_- = λ_0 e^-δ,    λ_-≤λ_0.

Note that this family has infinite diameter since 

    diam(ℰ) := sup_λ_0,λ_1 ∈ (0,∞) d(λ_0,λ_1)=sup_λ_0,λ_1 ∈ (0,∞)|log(λ_1/λ_0) |=∞.







§ DUALISTIC STRUCTURE IN INFORMATION GEOMETRY
 
As explained in the main article, the Riemannian structure of 𝒫 allows to obtain the Levi-Civita affine connection ∇̅. In addition, Eguchi's formulas <cit.> endows the space 𝒫 with an additional dualistic structure:

    Γ_ij,k (θ_0) =  - ∂_θ^i∂_θ^j∂_θ_0^k[ 𝒟(P_θ | P_θ_0) ]_θ=θ_0,
    
        ^*Γ_ij,k (θ_0) =  - ∂_θ^k∂_θ_0^i∂_θ_0^j[ 𝒟(P_θ | P_θ_0) ]_θ=θ_0.

The coefficients Γ_ij,k and ^*Γ_ij,k are the so-called Christoffel symbols (of the first kind) of two affine connections on 𝒫 denoted by ∇ and ∇^* that verify a dualistic (conjugacy) property <cit.>. This property implies that the unique Levi-Civita connection ∇̅ (resp. Christoffel symbols Γ̅) of the Fisher metric g is given by the average of the two dual connections (resp. Christoffel symbols)

    ∇̅= ∇ +  ^*∇/2   and   Γ̅_ij,k = Γ_ij,k +  ^*Γ_ij,k/2.

The Chistoffel symbols of the first kind Γ̅_ij,k and the Christoffel symbols (of the second kind) Γ̅_ij^k defined in the main article are related by the following formula

    Γ̅_ij^k = ∑_l=1^m Γ̅_ij,l g^lk.

Here, g^lk is the inverse matrix coefficients of g. These two previous formulas provide another way for computing the Levi-Civita Christoffel symbols. This will be useful in Section <ref> for the truncated normal family.


 


§ COMPUTATIONS OF FIM AND CHRISTOFFEL SYMBOL COEFFICIENTS
 

We will now compute the FIM coefficients and the Christoffel symbols for some families of probability distributions. This will prove Proposition 3.1 from the main article. These computations are motivated by the fact that we need to compute the Christoffel symbols in order to approximate the Fisher-Rao spheres.



 §.§ Truncated normal family
 
We will explicitly compute the FIM I_θ^[a,b] and Christoffel symbols Γ̅_ij,k of the truncated normal family 𝒩_[a,b]. We use the formula <cit.>

    (I_θ^[a,b])_ij = -∂_θ^i∂_θ_0^j[𝒟(q_θ | q_θ_0)]_θ=θ_0.

Here, I_θ^[a,b] is the FIM and q_θ is the truncated density and 𝒟 is the Kullback-Leibler divergence defined as 𝒟(q_θ| q_θ_0):= 𝔼_X∼ P_θ[log(q_θ/q_θ_0(X))]. For the Christoffel symbols Γ̅_ij,k, we first compute Γ_ij,k and Γ_ij,k^* (see Section <ref> for explanation)

    Γ_ij,k (θ_0) =  - ∂_θ^i∂_θ^j∂_θ_0^k[ 𝒟(q_θ | q_θ_0)]_θ=θ_0   and   ^* Γ_ij,k  (θ_0) =  - ∂_θ^k∂_θ_0^i∂_θ_0^j[ 𝒟(q_θ | q_θ_0) ]_θ=θ_0.

Then, we take the average to get the Levi-Civita Christoffel symbols Γ̅_ij,k.



  §.§.§ Computing the conditional expectation and variance of Gaussian variables

Let B=[a,b], it denotes the truncation domain. Denoting P_θ =𝒩(μ,σ^2), for θ=(μ,σ), with density p_θ, the conditional expectation and variance over B of a Gaussian random variable is given by 

    μ_B := 𝔼_X∼ q_θ(X) =  1/P_θ(B)∫_B x p_θ(x) dx = -σ^2/P_θ(B)∫_B -(x -μ)/σ^2 p_θ(x) dx + μ = -σ^2 [q_θ]_a ^b + μ,

 and

    σ_B^2     := Var_X∼ q_θ(X) 
       =1/P_θ(B)∫_B (x-μ_B)^2p_θ(x) dx 
       = 1/P_θ(B)( ∫_B (x-μ)^2p_θ(x) dx  + 2(μ - μ_B)∫_B (x-μ) p_θ(x) dx + ∫_B (μ-μ_B)^2p_θ(x) dx  ) 
       = -σ^2/P_θ(B)([ (x-μ)p_θ]_a^b -∫_B p_θ(x)dx   ) -2(μ - μ_B)^2 + (μ-μ_B)^2
       = -σ^2 ( [ (x-μ)q_θ]_a^b -1 ) - (μ - μ_B)^2.





  §.§.§ Computing the Kullback-Leibler divergence of truncated Gaussian densities

Let us compute the Kullback-Leibler divergence on the truncated normal family 

    D(θ,θ_0) := 𝒟( q_θ | q_θ_0) = ∫ q_θlog(q_θ/q_θ_0) = ∫ q_θlog q_θ - ∫ q_θlog q_θ_0.

Setting A_θ := ∫ q_θlog q_θ, we have

    D(θ,θ_0)    = A_θ - ∫ q_θlog q_θ_0
       = A_θ - ∫_B p_θ/P_θ(B)log p_θ_0 + ∫_B p_θ/P_θ(B)log P_θ_0(B) 
        = A_θ - ∫_B p_θ/P_θ(B)log p_θ_0 + log P_θ_0(B)_=:C_θ_0
        = A_θ + C_θ_0 - 1/P_θ(B)∫_B p_θlog p_θ_0 ,

A_θ and C_θ_0 depend on θ and θ_0 respectively. Let us now compute the last term explicitly:

    ∫_B p_θlog p_θ_0   = ∫_B p_θ( -log (√(2π)σ_0) - (x-μ_0)^2/2σ_0^2) 
        = -log (√(2π)σ_0) P_θ(B) -1/2σ_0^2∫_B (x-μ_0)^2 p_θ.

We finally obtain

    1/2σ_0^2∫_B (x-μ_0)^2 p_θ   = 1/2σ_0^2(∫_B (x-μ_B)^2 p_θ + 2(μ_B - μ_0) ∫_B (x-μ_B)p_θ_=0 +(μ_B - μ_0)^2 P_θ(B) )
        = P_θ(B) σ_B^2/2σ_0^2 +(μ_B - μ_0)^2/2σ_0^2 P_θ(B).


Now we can express (<ref>) using μ_B and σ_B^2 (which depend on θ)

    D(θ,θ_0) = A_θ + C_θ_0  + log(√(2π)) + logσ_0 + σ_B^2 + (μ_B - μ_0)^2/2σ_0^2.




  §.§.§ Computing the Fisher Information matrix
 

In the computation of the FIM using (<ref>), we only end up with terms that depend both on θ_0 and θ. This means that A_θ and C_θ_0 disappear. Therefore, we omit them from the start. For ∂_μ_0 D and ∂_σ_0 D we obtain 

    ∂_μ_0 D = 2(-1)(μ_B - μ_0)/2σ_0^2= -μ_B - μ_0/σ_0^2    and    ∂_σ_0 D = 1/σ_0 - σ_B^2 + (μ_B -μ_0)^2/σ_0^3.

Now for ∂_μ∂_μ_0 D, ∂_σ∂_μ_0 D and ∂_σ∂_σ_0 D, we get

    ∂_μ∂_μ_0 D     = - ∂_μμ_B(θ) /σ_0^2,
    ∂_σ∂_μ_0 D    = -∂_σμ_B(θ)/σ_0^2,
    ∂_σ∂_σ_0 D    = -1/σ_0^3( ∂_σσ_B^2(θ) + 2(μ_B - μ_0) ∂_σμ_B(θ) ).
 

Let us now compute the terms ∂_μμ_B(θ), ∂_σμ_B (θ) and ∂_σσ_B^2 (θ):

    ∂_μμ_B(θ)    = 1 - σ^2 [∂_μ q_θ(x) ]_a^b = 1 - σ^2 [ q_θ(x)( x-μ/σ^2 + [q_θ]_a^b) ]_a^b,
    ∂_σμ_B(θ)    = -2σ[ q_θ(x) ]_a^b -σ^2 [ ∂_σ q_θ(x) ]_a^b
       = - 2 σ[q_θ]_a^b  -σ[ q_θ(x)([(x- μ)q_θ]_a^b - 1  +(x-μ)^2/σ^2) ]_a^b
       = -σ[ q_θ(x)([(x- μ)q_θ]_a^b + 1  +(x-μ)^2/σ^2) ]_a^b
    ∂_σ
    σ_B^2(θ)    = 2σ - 2σ[ (x-μ)q_θ]_a^b - σ^2[ (x-μ)∂_σ q_θ]_a^b - 2(μ-μ_B)∂_σμ_B(θ) 
       = 2σ( 1 - [ (x-μ)q_θ]_a^b ) - σ^2[ (x-μ)q_θ(x)( x-μ/σ^2 + [q_θ]_a^b) ]_a^b
        - 2(μ-μ_B)∂_σμ_B(θ).


Lastly, coming back to (<ref>), (<ref>) and (<ref>) we get

    ∂_μ∂_μ_0 D     = - ∂_μμ_B(θ) /σ_0^2 = - 1/σ_0^2 + σ^2 [ q_θ(x)( x-μ/σ^2 + [q_θ]_a^b) ]_a^b/σ_0^2,
    ∂_σ∂_μ_0 D    = -∂_σμ_B(θ)/σ_0^2 = σ[ q_θ(x)([(x- μ)q_θ]_a^b + 1  +(x-μ)^2/σ^2) ]_a^b/σ_0^2,
    ∂_σ∂_σ_0 D    = -1/σ_0^3( ∂_σσ_B^2(θ) + 2(μ_B - μ_0) ∂_σμ_B(θ) ) 
       = -1/σ_0^3(2σ( 1 - [ (x-μ)q_θ]_a^b ) - σ^2[ (x-μ)q_θ(x)( x-μ/σ^2 + [q_θ]_a^b) ]_a^b  ).
 
Further, taking θ=θ_0 and multiplying by -1, we obtain

    (I_θ_0)_11=-∂_μ∂_μ_0 D_|θ=θ_0   =  1/σ_0^2 - [ q_θ_0(x)( x-μ_0/σ_0^2 + [q_θ_0]_a^b) ]_a^b,
    
        (I_θ_0)_12 =-∂_σ∂_μ_0 D_|θ=θ_0   = -σ_0 [ q_θ_0(x)([(x- μ_0)q_θ_0]_a^b + 1  +(x-μ_0)^2/σ_0^2) ]_a^b/σ_0^2,
    
    (I_θ_0)_22=-∂_σ∂_σ_0 D_|θ=θ_0   = 2/σ_0^2 - 2 [ (x-μ_0)q_θ_0]_a^b /σ_0^2 -  [ (x-μ_0)/σ_0 q_θ_0(x)( x-μ_0/σ_0^2 + [q_θ_0]_a^b) ]_a^b  .
 


  §.§.§ A few remarks on the truncated normal family

Here, we observe that when a→ -∞ and b→∞, we get

    (I_θ_0)_11→1/σ_0^2,     (I_θ_0)_12→ 0    and     (I_θ_0)_22→2/σ_0^2.


In other words, I_θ^[a,b] converges to the FIM of the non-truncated normal family I_θ. 

Now, if we fix a and b, for θ=(μ,σ) such that μ∈ (a,b) and σ is close to 0, then the normalization constant N_θ:= ∫_a^b (√(2π)σ)^-1exp(-(x-μ)^2/(2σ^2) dx
is close to 1. Indeed, most of the mass of the normal density p_θ is already inside [a,b]. As a consequence, q_θ and 𝒩(μ,σ^2) are close. So that, for such θ, the matrices I_θ and I_θ^[a,b] are also close. This means that in areas where μ∈ (a,b) and σ is close to 0, both of these metrics induce a very similar geometry.













  §.§.§ Computing the Levi-Civita Christoffel symbols

As explained previously, the Levi-Civita Christoffel symbols Γ̅_ij,k is computed by averaging the Christoffel symbols of Γ_ij,k and ^*Γ_ij,k

    Γ̅_ij,k = Γ_ij,k +  ^*Γ_ij,k/2.

It remains to express it as a function of μ_B and σ_B^2 and of their partial derivatives w.r.t. μ and σ.


  
Computing ^* Γ_ij,k 

First, for ∂_μ_0 D and ∂_σ_0 D we have

    ∂_μ_0 D    = -μ_B - μ_0/σ_0^2,
    ∂_σ_0 D    = 1/σ_0 - σ_B^2 + (μ_B -μ_0)^2/σ_0^3.
 

We differentiate these two expressions with respect to ∂_μ_0 and ∂_σ_0 to obtain ∂_μ_0∂_μ_0 D, ∂_μ_0∂_σ_0 D and ∂_σ_0∂_σ_0 D :

    ∂_μ_0∂_μ_0 D    = 1/σ_0^2,
    ∂_μ_0∂_σ_0 D    = 2(μ_B - μ_0)/σ_0^3,
    ∂_σ_0∂_σ_0 D    = -1/σ_0^2 + 3σ_B^2 + (μ_B - μ_0)^2/σ_0^4.


Now, it remains to differentiate these three expressions w.r.t. to μ and σ to obtain the following six functions ∂_μ∂_μ_0∂_μ_0 D, ∂_σ∂_μ_0∂_μ_0 D,  ∂_μ∂_μ_0∂_σ_0 D, ∂_σ∂_μ_0∂_σ_0 D, ∂_μ∂_σ_0∂_σ_0 D and ∂_σ∂_σ_0∂_σ_0 D. We obtain 

    ∂_μ∂_μ_0∂_μ_0 D    = 0,
    ∂_σ∂_μ_0∂_μ_0 D    =  0,
    ∂_σ∂_μ_0∂_σ_0 D    =  2 ∂_σ [μ_B(θ) ]/σ_0^3,
    ∂_μ∂_σ_0∂_σ_0 D    = 3/σ_0^4( ∂_μ [σ_B^2(θ) ] + 2 ∂_μ [μ_B(θ)] (μ_B-μ_0) ),
    ∂_σ∂_σ_0∂_σ_0 D    = 3/σ_0^4( ∂_σ [σ_B^2(θ) ] + 2 ∂_σ [μ_B(θ)] (μ_B-μ_0) ).


Let us now compute the partial derivatives of μ_B and σ_B^2 in terms of the truncated Gaussian density q_θ
 
    ∂_μμ_B(θ)    = 1 - σ^2 [ ∂_μ q_θ]_a^b,
    ∂_σμ_B(θ)    = -2σ[ q_θ]_a^b - σ^2 [ ∂_σ q_θ]_a^b,
    ∂_μσ_B^2 (θ)    = - σ^2  [- q_θ +(x-μ) ∂_μ q_θ]_a^b - 2 (μ - μ_B) ( 1 - ∂_μ[μ_B(θ)] ),
    ∂_σσ_B^2(θ)    = 2σ( 1 - [ (x-μ)q_θ]_a^b ) - σ^2[ (x-μ)∂_σ q_θ]_a^b  + 2(μ-μ_B)∂_σμ_B(θ).

    
    
    

Coming back to the symbols ^* Γ_ij,k, we have

    ^* Γ_11,1   =  -∂_μ∂_μ_0∂_μ_0 D_|θ = θ_0 = 0, 
    
    ^* Γ_11,2   = -∂_σ∂_μ_0∂_μ_0 D_|θ = θ_0 =  0,
    
    ^* Γ_12,1   =  ^*Γ_21,1= -∂_μ∂_μ_0∂_σ_0 D_|θ = θ_0 = - 2 ∂_μ [μ_B(θ_0) ]/σ_0^3,
    
    ^* Γ_12,2   = ^* Γ_21,2= -∂_σ∂_μ_0∂_σ_0 D_|θ = θ_0 =  -2 ∂_σ [μ_B(θ_0) ]/σ_0^3,
    
    ^* Γ_22,1   = -∂_μ∂_σ_0∂_σ_0 D_|θ = θ_0 = -3/σ_0^4( ∂_μ [σ_B^2(θ_0) ] + 2 ∂_μ [μ_B(θ_0)] (μ_B(θ_0)-μ_0) ),
    
    ^* Γ_22,2   = -∂_σ∂_σ_0∂_σ_0 D_|θ = θ_0 = -3/σ_0^4( ∂_σ [σ_B^2(θ_0) ] + 2 ∂_σ [μ_B(θ_0)] (μ_B(θ_0)-μ_0) ).

   







  
Computing Γ_ij,k

We have already computed  ∂_μ_0 D and ∂_σ_0 D 

    ∂_μ_0 D = -μ_B - μ_0/σ_0^2   and   ∂_σ_0 D = 1/σ_0 - σ_B^2 + (μ_B -μ_0)^2/σ_0^3.

We also computed ∂_μ∂_μ_0 D, ∂_σ∂_μ_0 D and ∂_σ∂_σ_0 D

    ∂_μ∂_μ_0 D     = - ∂_μμ_B(θ) /σ_0^2,
    ∂_σ∂_μ_0 D    = -∂_σμ_B(θ)/σ_0^2,
    ∂_σ∂_σ_0 D    = -1/σ_0^3( ∂_σσ_B^2(θ) + 2(μ_B - μ_0) ∂_σμ_B(θ) ).
 
It remains to compute ∂_μ∂_σ_0 D

    ∂_μ∂_σ_0  D = -1/σ_0^3( ∂_μ [σ_B^2]  + 2 ∂_μ[μ_B](μ_B - μ_0) ).

Now differentiating the last four expressions with respect to ∂_μ and ∂_σ, we obtain

    ∂_μ∂_μ∂_μ_0 D     = - ∂_μ^2 μ_B(θ) /σ_0^2,
    ∂_σ∂_μ∂_μ_0 D     = - ∂_σμ ^2 μ_B(θ) /σ_0^2,
    ∂_σ∂_σ∂_μ_0 D    = -∂_σ^2 μ_B(θ)/σ_0^2,
    ∂_μ∂_μ∂_σ_0 D    = -1/σ_0^3( ∂_μ^2 σ_B^2(θ) + 2(μ_B - μ_0) ∂_μ^2 μ_B(θ) + 2(∂_μμ_B(θ) )^2 ),
    ∂_μ∂_σ∂_σ_0 D    = -1/σ_0^3( ∂_μ∂_σσ_B^2(θ) + 2(μ_B - μ_0) ∂_μσ^2 μ_B(θ) + 2(∂_σμ_B(θ) )(∂_μμ_B(θ) ) ),
    ∂_σ∂_σ∂_σ_0 D    = -1/σ_0^3( ∂_σ^2 σ_B^2(θ) + 2(μ_B - μ_0) ∂_σ^2 μ_B(θ) + 2(∂_σμ_B(θ) )^2 ).
 
Thus, the Christoffel symbols Γ_ij,k are given by

    Γ_11,1   = -∂_μ∂_μ∂_μ_0 D_|θ=θ_0  = ∂_μ^2 μ_B(θ_0) /σ_0^2,
    Γ_21,1   =Γ_12,1 = -∂_σ∂_μ∂_μ_0 D_|θ=θ_0  =  ∂_σμ ^2 μ_B(θ_0) /σ_0^2,
    Γ_22,1   = -∂_σ∂_σ∂_μ_0 D_|θ=θ_0 = ∂_σ^2 μ_B(θ_0)/σ_0^2,
    Γ_11,2   = -∂_μ∂_μ∂_σ_0 D_|θ=θ_0 = 1/σ_0^3( ∂_μ^2 σ_B^2(θ_0) + 2(μ_B(θ_0) - μ_0) ∂_μ^2 μ_B(θ_0) + 2(∂_μμ_B(θ_0) )^2 ),
    Γ_12,2   =Γ_21,2 = -∂_μ∂_σ∂_σ_0 D_|θ=θ_0
       = 1/σ_0^3( ∂_μ∂_σσ_B^2(θ_0) + 2(μ_B(θ_0) - μ_0) ∂_μσ^2 μ_B(θ_0) +2(∂_σμ_B(θ_0) )(∂_μμ_B(θ_0) ) ),
    Γ_22,2   = -∂_σ∂_σ∂_σ_0 D_|θ=θ_0 = 1/σ_0^3( ∂_σ^2 σ_B^2(θ_0) + 2(μ_B(θ_0) - μ_0) ∂_σ^2 μ_B(θ_0) + 2(∂_σμ_B(θ_0) )^2 ).
 



  
First and second order partial derivatives of μ_B and σ_B^2 with respect to (μ,σ)
The two conjugate Christoffel symbols Γ_ij,k and Γ_ij,k^* are functions of the first and second order partial derivatives of μ_B and σ_B^2. Now we explicitly compute these derivatives. To begin with, let us compute the first order partial derivatives of μ_B:

    ∂_μμ_B(θ)    = 1 - σ^2 [ ∂_μ q_θ]_a^b,
    ∂_σμ_B(θ)    = -2σ[ q_θ]_a^b - σ^2 [ ∂_σ q_θ]_a^b,
    ∂_μσ_B^2 (θ)    = - σ^2  [- q_θ +(x-μ) ∂_μ q_θ]_a^b - 2 (μ - μ_B) ( 1 - ∂_μ[μ_B(θ)] ),
    ∂_σσ_B^2(θ)    = 2σ( 1 - [ (x-μ)q_θ]_a^b ) - σ^2[ (x-μ)∂_σ q_θ]_a^b + 2(μ-μ_B)∂_σμ_B(θ).

For the second order partial derivatives of μ_B and σ_B ^2, we obtain 

    ∂_μ ^2 μ_B(θ)    = -σ^2 [∂_μ ^2 q_θ]_a ^b,
    ∂_σμ ^2 μ_B(θ)    = -2 σ[ ∂_μ q_θ]_a ^b - σ^2 [ ∂_σμ ^2 q_θ]_a ^b,
    ∂_σ ^2 μ_B(θ)    = -2[q_θ]_a ^b -4σ[∂_σ q_θ]_a ^b - σ^2 [∂_σμ ^2 q_θ]_a ^b,


    ∂_μ^2 σ_B^2    =-σ^2 [-2∂_μ q_θ + (x-μ) ∂_μ ^2 q_θ]_a^b -2(1-∂_μμ_B(θ) )^2 + 2(μ - μ_B) ∂_μ ^2 μ_B, 
    ∂_σμ^2 σ_B^2    =  -2σ[-q_θ + (x-μ) ∂_μ q_θ]_a^b - σ^2 [ -∂_σ q_θ + (x-μ)∂_σμ ^2 q_θ]_a^b
       + 2 ∂_σμ_B (1- ∂_μμ_B) + 2∂_σμ ^2  μ_B (μ - μ_B),
    ∂_σ^2 σ_B^2    =  2(1-[(x-μ) q_θ]_a^b )- 4σ[(x-μ) ∂_σ q_θ]_a^b - σ^2 [(x-μ) ∂_σ ^2 q_θ]_a^b 
       + 2 (∂_σμ_B )^2  - 2 (μ - μ_B) ∂_σ ^2 μ_B.




  
First and second order partial derivatives of q_θ with respect to (μ,σ)
We also need to compute the partial derivatives of q_θ. For the first order, we get

    ∂_μ q_θ(x)    = q_θ(x) ( x-μ/σ^2 + [q_θ(x) ]_a ^b  ),
    ∂_σ q_θ(x)    = q_θ(x) (  1/σ[ (x-μ) q_θ(x) ]_a ^b -1/σ + (x-μ)^2/σ^3).

The computation of the second order derivatives follows

    ∂_μ ^2 q_θ(x)    = ∂_μ q_θ(x) ( x-μ/σ^2 + [q_θ(x) ]_a ^b  ) + q_θ(x) ( -1/σ^2 + [∂_μ q_θ(x) ]_a ^b  ),
    ∂_σμ ^2 q_θ(x)    = ∂_σ q_θ(x) ( x-μ/σ^2 + [q_θ(x) ]_a ^b  ) + q_θ(x) ( -2(x-μ)/σ^3 + [∂_σ q_θ(x) ]_a ^b  ),
    ∂_σ ^2 q_θ(x)    =  ∂_σ q_θ(x) ( 1/σ[ (x-μ) q_θ(x) ]_a ^b -1/σ + (x-μ)^2/σ^3) 
       + q_θ(x) ( -1/σ[ (x-μ) q_θ(x) ]_a ^b 
    +1/σ[ (x-μ) ∂_σ q_θ(x) ]_a ^b +1/σ^2 + -3(x-μ)^2/σ^4).
 

Let us summarize the previous computations:

    
  * we computed the FIM of the truncated normal family on [a,b];
    
  * we expressed the Christoffel symbols as a function of the conditional mean μ_[a,b] and conditional variance σ_[a,b]^2 (that are functions of θ) as well as their partial derivatives;
    
  * we then gave the explicit expression of these partial derivatives as a function of the truncated density q_θ on [a,b] and the partial derivatives of the latter;
    
  * lastly, we explicitly computed the partial derivatives of q_θ.

This concludes the computation of the FIM and Christoffel symbols of the truncated normal family on [a,b].





 §.§ Gumbel family

The Gumbel family is a location-scale family where the initial density p is given by p(x) = exp(-x - e^-x).


  §.§.§ Non-truncated case

The FIM for the non-truncated version is given by

    I_θ = 1/s^2[     1  γ -1;  γ -1 β + 1 ],

where γ≈ 0.5772... is the Euler-Mascheroni constant and β is given by the integral

    β = ∫_0^∞log(x)^2 x e^-xdx.

In this case, the Christoffel symbols can be easily computed. Indeed, they only depend on s. For the non-truncated Gumbel family, we numerically compute β using the Gaussian quadrature method of SciPy[For documentation, see https://docs.scipy.org/doc/scipy/tutorial/integrate.htmlhttps://docs.scipy.org/doc/scipy/tutorial/integrate.html].


  §.§.§ Truncated case

The density of the truncated Gumbel distribution on [a,b] writes q_θ(x) = 1/N_θ p_θ(x) 1_x∈ [a,b], where N_θ is the normalizing constant

    N_θ = ∫_a^b 1/s^2exp(-x-m/s - e^-x-m/s) dx = F(b-m/s) - F(a-m/s),

and F the cumulative distribution function of p given by
F(x) = exp(-e^-x).
In this case, we computed the FIM for the truncated family using the formula

    (I_θ^[a,b])_ij = - 𝔼_X ∼ q_θ[ ∂_θ_iθ_jlog q_θ(X) ],

where (θ_1,θ_2) = (m,s). For X ∼ q_θ, we have

    log q_θ(X) = -2log s - X-m/s - exp( -X-m/s) - log N_θ.

Taking the Hessian w.r.t. θ=(m,s) and integrating against q_θ leads to

    (I_θ^[a,b])_ij = -∂_θ_iθ_j   2log s - ∫_a^b ∂_θ_iθ_j[x-m/s] q_θ(x) dx 
       - ∫_a^b ∂_θ_iθ_j[exp( -x-m/s)] q_θ(x) dx - (Hess_θlog N_θ)_ij.
 
Notice that the first and forth term in the last equality can be explicitly computed. For the second and third ones, we need to resp. compute integrals of the form (after a variable substitution)

    ∫_c^d y exp( -y -e^-y)dy     and    ∫_c^d y^2 exp( -2y -e^-y)dy.

Here, c and d depend on a,b,m,s. We computed these integrals using the Gaussian quadrature method of SciPy. 
For the Christoffel symbols, we used the standard finite difference method for differentiating the coefficients of I_θ^[a,b] w.r.t. θ:

    ∂_m I_θ^[a,b]≈I_θ + (h,0)^[a,b] - I_θ^[a,b]/h   and   ∂_s I_θ^[a,b]≈I_θ + (0,h)^[a,b] - I_θ^[a,b]/h,

where we took h = 10^-7. Of course, more advanced numerical differentiation methods can be used instead.



 §.§ Gamma family

The pdf p_θ of the Gamma family writes

    p_θ(x) = β^α/Γ(α)x^α-1e^-β x,    θ = (α,β) ∈ (0,∞)^2.

Here, Γ is the Euler integral of the second kind. In this framework we do not consider a truncated version of this family. The FIM I_θ for this family can be easily computed. Indeed, we have

    I_θ = [ ψ'(α)  -1/β;  -1/β α/β^2 ].

Here, the function ψ is the digamma function. To numerically compute its first and second order derivatives, we used the previously discussed finite difference scheme with h=10^-7. 



§ A FEW ADDITIONAL ILLUSTRATIVE FIGURES FOR THE DENSITY PERTURBATION METHOD












plain





 Paper Title: A Fast Algorithm for Convex Composite Bi-Level Optimization.



[
    [
    30 July 2024
================



§ ABSTRACT 
	In this paper, we study convex bi-level optimization problems where both the inner and outer levels are given as a composite convex minimization. We propose the Fast Bi-level Proximal Gradient (FBi-PG) algorithm, which can be interpreted as applying FISTA to a dynamic regularized composite objective function. The dynamic nature of the regularization parameters allows to achieve an optimal fast convergence rate of O(1/k^2) in terms of the inner objective function. This is the fastest known convergence rate under no additional restrictive assumptions. We also show that FBi-PG achieves sub-linear simultaneous rates in terms of both the inner and outer objective functions. Moreover, we show that under an Hölderian type error bound assumption on the inner objective function, the FBi-PG algorithm converges to an optimal solution of the bi-level optimization problem. Finally, we present numerical experiments demonstrating the performances of the proposed scheme. 


Keywords. Convex minimization, bi-level optimization, first order methods, dynamic proximal gradient, fast convergence rate, global convergence.


AMS subject classifications. 90C25, 65K05.



§ INTRODUCTION

	Bi-level optimization emerges as a dynamic and evolving research field within the broad optimization area. In this paper, we consider the bi-level convex minimization problem which aims at minimizing a convex function, called an outer objective function over the optimal solution set of another convex minimization of a function called the inner objective. We are interested in studying a convex bi-level model where both objectives in the inner and outer levels are instances of composite convex minimization problems. Our focus is on achieving faster convergence rates and corresponding pointwise global convergence for a first-order algorithm to this important class of bi-level composite convex optimization problems.


	Following the terminology of inner and outer problems, we are interested in considering an outer problem given by the classical convex composite model
	
    OPmin_∈ X^∗{ω():= σ() + ψ() },

	where X^∗ is the, assumed non-empty, set of minimizers of the inner level problem which is also described via another convex composite model
	
    Pmin_∈^n{φ() := f() + g() }.

 	The functions f , σ : ^n→ are smooth and convex, while the functions g , ψ : ^n→ are non-smooth and convex. We will provide the exact details of the problem's data in the forthcoming section. Moreover, we call φ and ω the inner and outer objective functions, respectively. Throughout the paper, we denote by X' ⊂ X^∗ the set of all optimal solutions of the outer optimization problem (<ref>).


	This type of bi-level minimization problems, is also known sometimes under the name simple bi-level programming as being a particular case of the more general bi-level programming model which involves, in both the inner/outer problems, parametric representation featuring two variables. This simpler model is however very flexible and as such relevant in modelling various modern applications; see, for instance, <cit.> for a thorough overview of bi-level optimization problems as well as their applications, and the literature therein.


	The implicit constraints imposed by the inner minimization problem make such problems difficult for developing simple and efficient algorithms. In recent years, there has been a growing interest in deriving theoretical guarantees beyond mere convergence results for convex bi-level optimization problems under various type of assumptions on the problem's data. For some of the most recent advances in this area, we refer the readers to <cit.> and the very recent paper <cit.>, which both also provide ample details on the past and current developments. Despite many of these interesting results, a grand challenge in this field remains open: developing a first-order algorithm capable of achieving fast convergence rates for convex bi-level optimization problems akin to the existing faster rates of first-order schemes for the traditional (single level) convex composite minimization <cit.>.
Motivated by the remarkable success of fast first-order algorithms for traditional convex composite optimization problems just alluded above, our objective is to derive similar accelerated rates for the more challenging setting of bi-level convex composite optimization problems. Toward this goal, we propose employing the Tikhonov regularization technique, which involves approaching bi-level optimization as the minimization of either a single fixed regularized (penalized) or a dynamic regularized composite convex objective. The convex composite structure of the regularized objective thus naturally call for applying any fast first-order scheme on it. However, at this point we faced a hurdle. As we shall see below, we cannot apply directly existing convergence rate results due to either the lack of knowledge of the fixed regularization parameter or the dynamic change of the regularized composite objective. Thus, to tackle this inherent difficulty, a fresh analysis/approach is required.


	The paper is organized as follows. The subsequent section will comprehensively discuss the Tikhonov technique and its strong connections to bi-level optimization problems. This will lead us to propose the Fast Bi-level Proximal Gradient (FBi-PG) algorithm that can be simply interpreted as applying FISTA <cit.> to the dynamic regularized objective. The algorithm FBi-PG is equipped with a dynamic update of the regularization parameter, which is itself characterized by the choice of another fixed parameter allowing us to devise accelerated schemes. This is developed in Section <ref>, where we will prove our main rate of convergence results. These analyses highlight the significance and crucial roles played by the dynamic regularization parameter, which is user-determined in regulating the performance of the FBi-PG scheme, a method which (for the first time) is proven to be capable of achieving the fast convergence rate of O(1/k^2) in terms of the inner optimization problem, see Theorem <ref>. Additionally, we establish convergence rates concerning the outer objective function, and simultaneous rates concerning both functions. In the classical context of convex composite minimization, it is well-known that the sequence generated by FISTA does not necessarily converge to an optimal solution. This issue was however successfully addressed in <cit.> through a smart analysis of a variant of FISTA. The natural question is then if such a result can be extended to the more involved convex composite bi-level setting. In Section <ref>, we positively address this question by adapting and extending the analysis of <cit.> to this case, and under an Hölder type error bound assumption <cit.>, we prove the convergence of the generated sequences by FBi-PG to an optimal solution of the bi-level problem (<ref>). Finally, in Section <ref> we present numerical experiments  demonstrating the significant rate/improvement achieved by FBi-PG in comparison to other current state of the art schemes.
 	


§ THE FBI PROXIMAL GRADIENT ALGORITHM
 
	


 §.§ Motivation and Difficulties
 
	Several algorithmic approaches exist for addressing bi-level optimization problems. One of the earliest ideas dates back to the book of Tikhonov and Arsenin <cit.>, who introduced what is now known as the Tikhonov regularization technique. In their original work, the goal was to find a minimal norm solution (meaning the outer objective function is ω≡σ = ·^2) to a linear optimization problem. Its primary advantage lies in simplifying the bi-level problem into a single regularized formulation, which avoids handling two optimization problems. In the context of our paper, it means to tackle the following penalized problem
	
    Pen_αmin_∈^n{ F_α() ≡φ() + αω() },

	where α > 0 is the regularization parameter. In this context, we mention the work <cit.>, which studies among other questions, the connections between bi-level problems and their regularized counterpart in a general convex setting. They proved that there is a small enough α > 0 such that an optimal solution of the regularized problem (<ref>) is an optimal solution of the bi-level problem (<ref>). Thus, a seemingly natural approach to obtain rates of convergence for the convex composite bi-level problem, based on the later idea of a-priori fixing the regularization parameter α, would be to simply apply any first-order scheme on the regularized objective function F_α, for some predefined parameter α, to obtain rates on the regularized objective function F_α. Indeed, for instance, a straightforward application of FISTA <cit.> on minimizing F_α with say an L_α-Lipschitz gradient part of the objective will generate a sequence k which satisfies
	
    F_α(^k) - F_α(') ≤2L_α^0 - '^2/(k + 1)^2,     ∀ ' ∈ X'.

 	Using the definition of F_α, and since φ(^k) - φ(') ≥ 0 for all k ∈, one immediately deduce the following fast rate of convergence for the outer problem
 	
    ω(^k) - ω(') ≤2L_α^0 - '^2/α(k + 1)^2,     ∀ ' ∈ X'.

 	Unfortunately, finding/knowing the value of a fixed regularization parameter α, for which a solution of the regularized problem recovers an optimal solution of the bi-level problem, remains itself a challenging task. Furthermore, finding a corresponding rate for the inner optimization problem remains a difficult question, and even if successfully addressed will also depend on the unknown value of α. At this juncture, it is important to note that the same arguments used above for FISTA are actually independent of neither the algorithm that is used or the structure of the optimization problem at hand. Therefore, obtaining rates of convergence for bi-level optimization problems can't rely on existing results for single level optimization problems, and requires a fresh specially designed approach.
The first work to provide an efficient use of the Tikhonov regularization technique tailored made for bi-level optimization problems, is <cit.>. It was confined to the convex bi-level problem with a smooth and convex outer objective ω and with the inner objective function φ being the sum of a smooth and convex function and an indicator of a closed and convex set. The resulting projected gradient algorithm applied on F_α proposed in <cit.> suggests that at each iteration k we regularize the outer objective function ω with a dynamic parameter α_k, and it is proved that if α_k→ 0 as k →∞ as long as ∑_k = 1^∞α_k = ∞, then any limit point of the generated sequence is an optimal solution of the bi-level optimization problem. Therefore, following <cit.>, using a dynamic regularization parameter at each iteration seems to be the right approach to overcome the lack of knowledge of the “right" regularization parameter α. Yet, one of the interesting open questions about this approach and which we address in this work pertains to establish convergence rates, and in particular fast convergent rates, as well as the corresponding pointwise global converge results for the more general convex composite bi-level problem (<ref>).


	A first attempt in this direction was taken in the thesis <cit.>. In that thesis, the fast scheme  based on the accelerated scheme of <cit.> (see also <cit.> and references therein), which is different from FISTA was used. However, that study did not cover the general convex composite bi-level problem (<ref>), the derived fast rate of convergence result was limited only to the inner problem, and no simultaneous rate was established. Moreover, pointwise global convergence was not derived. Inspired by <cit.>, and using FISTA as our basis, here we are able to go much beyond <cit.> by significantly extending both the rate and the global convergence results under various scenarios; see below for a description of our main results.



 §.§ Problem's Statement and the FBi Algorithm
 
	Motivated by the above discussion, in this paper, we extend the framework of FISTA proposed in <cit.> to introduce the Fast Bi-level Proximal Gradient (FBi-PG) algorithm designed specifically for solving convex bi-level optimization problems. Our approach incorporates the Tikhonov regularization technique, utilizing a dynamically updated regularization parameter to enhance the algorithm's performance in achieving a fast scheme. More precisely, given γ > 0, we will use the following sequence of regularizing parameters αk, which is defined for an a ∈ such that a ≥ 2, by
	
    α_k = (k + a)^-γ,    k ∈.
		
	For the simplicity of the developments below we define α_-1 := 0. Note that we could have set a ≡ 1. The additional parameter a is already introduced here in order to simplify/unify our presentation, which also allows us to handle the pointwise global convergence, see Section <ref>.
Before presenting the algorithm, considering the inner and outer optimization problems (<ref>) and (<ref>), throughout the paper we will use the following standard assumption on the problem's data.
	 
		
    		
  (i) The functions f , σ : ^n→ are convex, continuously differentiable having Lipschitz continuous gradients with the Lipschitz parameters β_f and β_σ, respectively. We denote β≡β_f + β_σ∈_++.
    		
  (ii) The functions g , ψ : ^n→ (-∞ , ∞] are proper, lower semicontinuous and convex.
    		
  (iii) The set X^∗ of all optimal solutions of problem (<ref>) is non-empty.
    		
  (iv) inf_∈^nω() ≡ω^∗ > -∞.
		
		
	Moreover, throughout the paper we systematically define, for all k ∈, the smooth regularized function f_k≡ f + α_kσ, the non-smooth regularized function g_k≡ g + α_kψ, and the regularized function F_k≡φ + α_kω.
	 
		The functions f and σ having Lipschitz continuous gradients with the Lipschitz parameters β_f and β_σ, respectively. Hence, for any k ∈, the function f_k also has a Lipschitz continuous gradient with the parameter β_f + α_kβ_σ. Since α_k≤ 1 then β_f + α_kβ_σ≤β_f + β_σ = β.
	
	Based on this remark we see that the regularized objective function F_k also has a composite structure of the sum of a smooth function f_k with a Lipschitz continuous gradient and a non-smooth function g_k. We can thus apply FISTA <cit.> on the dynamic objective F_k, k ∈, to produce the FBi-PG algorithm, which is described below.

	16cmFast Bi-level Proximal Gradient
			
				
    Input: γ > 0 and 2 ≤ a ∈.
				
    Initialization: t_-1 = 0, t_0 = 1 and ^0 = ^-1∈^n.
				
    General Step (k = 0, 1 , 2 , …):
					
    α_k    = (k + a)^-γ
    ^k    = ^k + t^-1_k(t_k - 1 - 1)(^k - ^k - 1) 
    ^k + 1    = _∈^n{ g_k() + ^T∇ f_k(^k) + β/2 - ^k^2}
    
    						t_k + 1    = (1 + √(1 + 4t_k^2))/2

			




	The FBi-PG algorithm is associated with a parameter γ > 0, via the regularization parameter α_k = (k + a)^-γ, that enables us to capture several instances in a unified fashion. The user can choose this parameter which regulates the algorithm's convergence rate. Before concluding this section, we summarize the main results of our paper.
	
    	
  (i) For γ > 2, FBi-PG achieves a fast rate of O(1/k^2) in terms of the inner optimization problem.
    	
  (ii) For 0 < γ < 2, FBi-PG achieves a rate of O(1/k^2 - γ) in terms of the outer optimization problem.
    	
  (iii) For γ = 1, FBi-PG achieves a simultaneous rate of O(ln(k + 1)/k) on both the inner and outer optimization problems.
    	
  (iv) For 1 < γ < 2, under Hölderian error bound for the inner objective function, FBi-PG achieves a simultaneous rate of O(1/k^2) and O(1/k^2 - γ) on the inner and outer optimization problems, respectively.
    	
  (v)  For 1 < γ < 2 and a > 2, under an Hölderian error bound for the inner objective function, FBi-PG generates a globally convergent sequence to an optimal solution of the bi-level problem (<ref>).
	
	


§ RATE OF CONVERGENCE ANALYSIS OF FBI-PG
 
	The section is divided into three parts, where we start with a few key results, which will serve us in deriving the main results. In Section <ref>, we will prove the fast rate of FBi-PG including rates in terms of the outer objective function and simultaneous rates concerning both inner and outer objective functions. In Section <ref>, we will prove the improved simultaneous rates under the Hölderian error bound condition for the inner objective function.
	


 §.§ Main Pillars of the Analysis
 
	In order to obtain our basic inequality for the development of all the announced results, we will need the following classical proximal inequality (see <cit.>).
	 
		Let s : ^n→ be a convex and continuously differentiable function having a Lipschitz continuous gradient with the Lipschitz parameter L > 0, and let q : ^n→ (-∞ , ∞] be a proper, lower semicontinuous and convex function. For a given vector ∈^n, we define
		
    ^+ := _∈^n{ q() + ^T∇ s() + L/2 - ^2}.

		Then, for all ∈q, we have
		
    q(^+) + s(^+) - (q() + s()) ≤L/2( - ^2 -  - ^+^2).
		
	
  	At each iteration k ∈, the steps of FBi-PG can be seen as a direct application of the well-known FISTA algorithm <cit.> on the regularized function F_k. The next technical result can be obtained from <cit.>. However, for the sake of completeness, we include its simple proof.     
	 
		Let k be a sequence generated by FBi-PG. For all k ∈ and ∈^n, we have
		
    t_k^2(F_k(^k + 1) - F_k()) + β/2^k + 1 - ^2≤(t_k^2 - t_k)(F_k(^k) - F_k()) + β/2^k - ^2,

		where we define ^k := (1 - t_k - 1)^k - 1 + t_k - 1^k.
	
	
		Let k ∈. From the ^k + 1 step (see (<ref>)), using Proposition <ref> with s(·) = f_k(·), q(·) = g_k(·), = ^k, ^+ = ^k + 1, = (1 - t_k^-1)^k + t_k^-1 and L = β we get for any ∈^n that (recall F_k≡ f_k + g_k)
        
    F_k(^k + 1)    ≤ F_k((1 - t_k^-1)^k + t_k^-1) +β/2((1 - t_k^-1)^k + t_k^-1 - ^k^2 - (1 - t_k^-1)^k + t_k^-1 - ^k + 1^2) 
        = F_k((1 - t_k^-1)^k + t_k^-1) + β/2t_k^2(^k - ^2 - ^k + 1 - ^2),

        where the last equality follows from the definitions of ^k and ^k (see (<ref>)). Since F_k is convex, the inequality above yields
        
    F_k(^k+1) ≤(1 - t_k^-1)F_k(^k) + t_k^-1F_k() + β/2t_k^2(^k - ^2 - ^k + 1 - ^2).

        Subtracting F_k() from both sides and multiplying both sides with t_k^2, we obtain after rearranging that
         
    t_k^2(F_k(^k + 1) - F_k()) + β/2^k + 1 - ^2≤(1 - t_k^-1)t_k^2(F_k(^k) - F_k()) + β/2^k - ^2,

        which proves the desired result.
	
	From now on, we cannot any more apply the rate of convergence results of FISTA (<cit.> or <cit.>) to obtain relevant faster rates for the algorithm FBi-PG, and we need a new line of analysis. Indeed, the dynamic nature of the regularized function F_k, k ∈, precludes the possibility of using directly the rates established for FISTA on F_k, unless one either pick a fixed parameter α_k≡α > 0, for all k ∈, or one fixes in advance the iteration counter at some predefined value k ≡ K. However, both strategies are not  practically and theoretically viable. Indeed, as discussed earlier in Section <ref>, in the former case (among other troubles!) the value of the “right" regularizing parameter α is in general not available, and in the latter case it is immediate to see that from (<ref>) with α_K := (K + 1)^-γ (for some 0< γ < 2) one obtains
	
    ω(^K) - ω(') ≤2β^0 - '^2/(K + 1)^2 - γ,     ∀ ' ∈ X',

 	which shows that the fast rate is lost! Furthermore, in general, it makes no sense to fix in advance the number of iterations to be performed by the algorithm FBi-PG since, eventually in order to warrant convergence of the sequence generated by FBi-PG to an optimal solution of the bi-level optimization problem (<ref>), we need to have the sequence of regularized parameters αk to converge to 0 as k  →∞. The numerical results presented in Section <ref> will further illustrate and corroborate this situation. The next results will be useful to overcome the alluded difficulties.


	For convenience, we define for all k ∈
	
    d_k := t_k - 1^2 - (t_k^2 - t_k) and η_k := t_k - 1^2α_k - 1 - (t_k^2 - t_k)α_k.

	We note that since t_k^2 - t_k≤ t_k - 1^2, then d_k = t_k - 1^2 - (t_k^2 - t_k) ≥ 0 for all k ∈. Likewise, using the definition of α_k (see (<ref>)) we have that α_k < α_k - 1 and thus η_k = t_k - 1^2α_k - 1 - (t_k^2 - t_k)α_k≥ t_k - 1^2(α_k - 1 - α_k) > 0 for all k ∈.    
     
    	Let k be a sequence generated by FBi-PG. For all k ∈ and ' ∈ X', we have
    	
    t_k - 1^2(F_k - 1(^k) - F_k - 1(')) ≤β/2^0 - '^2 - ∑_s = 0^k - 1η_s(ω(^s) - ω(')).

    
    
    	From Proposition <ref> (see (<ref>)) we have for all k ∈ and ∈^n that
		
    t_k^2(F_k(^k + 1) - F_k()) + β/2^k + 1 - ^2≤(t_k^2 - t_k)(F_k(^k) - F_k()) + β/2^k - ^2.
    	    	
    	 Now, using the definition of F_k≡φ + α_kω, we get that
    	
    (t_k^2 - t_k)(F_k(^k) - F_k())     = (t_k^2 - t_k)(φ(^k) - φ()) + (t_k^2 - t_k)α_k(ω(^k) - ω()) 
        = t_k - 1^2(φ(^k) - φ()) + (t_k^2 - t_k - t_k-1^2) (φ(^k) - φ()) 
        + (α_k - 1t_k - 1^2 - η_k)(ω(^k) - ω()) 
        = t_k - 1^2(F_k - 1(^k) - F_k - 1()) - d_k(φ(^k) - φ())  
        - η_k(ω(^k) - ω()),
    	
    	where the second equality follows from the fact that (t_k^2 - t_k)α_k = t_k - 1^2α_k - 1 - η_k while the last equality follows from the definition of F_k - 1≡φ + α_k - 1ω and the fact that t_k^2 - t_k - t_k-1^2 = -d_k. Combining it with (<ref>) for k = s yields that
		
    t_s^2(F_s(^s + 1) - F_s()) + β/2^s + 1 - ^2   ≤ t_s - 1^2(F_s - 1(^s) - F_s - 1()) + β/2^s - ^2
        - d_s(φ(^s) - φ()) - η_s(ω(^s) - ω()).
    	    	
        Summing (<ref>) over s = 0 , 1 , … , k - 1 results in
        
    t_k - 1^2(F_k - 1(^k) - F_k - 1()) + β/2^k - ^2   ≤ t_-1^2(F_-1(^0) - F_-1()) + β/2^0 - ^2
        - ∑_s = 0^k - 1 d_s(φ(^s) - φ()) - ∑_s = 0^k - 1η_s(ω(^s) - ω()) 
        = β/2^0 - ^2 - ∑_s = 0^k - 1 d_s(φ(^s) - φ()) 
        - ∑_s = 0^k - 1η_s(ω(^s) - ω()),

		where the equality follows from the fact that since φ is proper (see Assumption <ref>) and α_-1 = 0, it follows that F_-1≡φ + α_-1ω = φ < ∞, along with the facts that t_-1 = 0 and ^0 = ^0. Eliminating the non-negative term β^k - ^2/2 on the left-hand side, and plugging = ' ∈ X' in (<ref>) it follows that 
		
    t_k - 1^2(F_k - 1(^k) - F_k - 1(')) ≤β/2^0 - '^2 - ∑_s = 0^k - 1 d_s(φ(^s) - φ(')) - ∑_s = 0^k - 1η_s(ω(^s) - ω(')).

		Since ' ∈ X' we have that φ(^s) - φ(') ≥ 0 for all s ∈, and since d_s≥ 0 for all s ∈, the desired result follows.
	
	We conclude this part with the following result.
	 
		Let k be a sequence generated by FBi-PG. For all k ∈ and ' ∈ X', we have
		
    t_k - 1^2(φ(^k) - φ(')) ≤β/2^0 - '^2 + (ω(') - ω^∗)∑_s = 0^k - 1α_st_s.

	
	
		Let k ∈. For the simplicity of the proof we define the following non-negative quantities (recall that ω^∗ = inf_∈^nω()) Ω_max := ω(') - ω^∗ and Ω_k := ω(^k) - ω^∗, k ∈. Using the definition of F_k - 1≡φ + α_k - 1ω in Proposition <ref> (see (<ref>)) we obtain
 		
    t_k -1^2(φ(^k) - φ(')) + α_k - 1t_k - 1^2(Ω_k-Ω_max) ≤β/2^0 - '^2- ∑_s = 0^k - 1η_s( Ω_k-Ω_max).

    	Now, since η_k > 0 and Ω_k≥ 0 for all k ∈, it follows from the above inequality that
    	
    t_k -1^2(φ(^k) - φ(')) - α_k - 1t_k - 1^2Ω_max≤β/2^0 - '^2+ Ω_max∑_s = 0^k - 1η_s.

 	To complete the proof, note that from the definition of η_k = α_k - 1t_k - 1^2 - (t_k^2 - t_k)α_k, we have that
 		
    ∑_s = 0^k - 1η_s = ∑_s = 0^k - 1(α_s - 1t_s - 1^2 - α_st_s^2)
        + ∑_s = 0^k - 1α_st_s = α_-1t_-1^2 - α_k - 1t_k - 1^2 + ∑_s = 0^k - 1α_st_s = -α_k - 1t_k - 1^2 + ∑_s = 0^k - 1α_st_s,

    	where the last equality follows from the fact that α_-1 = t_-1 = 0. Plugging it in (<ref>), the desired result follows.
    	 	



 §.§ Rates of Convergence of FBi-PG
 
	In this section, we will prove three rate of convergence results. In order to simplify our developments below we will take (recall that a ∈ such that a ≥ 2)
	
    t_k = k + a/a,    k ∈.

	Moreover, this choice is not accidental and is also crucial for the pointwise convergence analysis developed in Section <ref>. However, it should be noted that all our results of Section <ref> are valid for any sequence tk, which satisfies t_k^2 - t_k≤ t_k - 1^2 for all k ∈. It is easy to check that our choice of t_k in (<ref>) satisfies this requirement. Indeed, t_k - 1^2 - t_k^2 + t_k = ((a - 1)^2 + (a - 2)k)/a^2 > 0, with a≥ 2.
We begin with two technical lemmas to simplify the proofs afterwards. The first one was proved in <cit.>.
	 
		Let n_1∈ and n_2∈∪{∞} be such that n_1≤ n_2. Then, for any 0<r<1, we have
		
    ∑_n = n_1^n_2 n^-r≤n^1 - r_2 - (n_1 - 1)^1 - r/1 - r.
		
 		The result remains true for r > 1 when n_1≥ 2.
		
	 
    	For all k ≥ 1, the following inequality holds true
    	
    ∑_s = 0^k - 1α_st_s≤((k + a - 1)^2 - γ - 1)(2 - γ)^-1,     0 < γ < 2, 
    ln(k + a - 1 ),    γ = 2, 
    (1 - (k + a - 1)^2 - γ)(γ - 2)^-1,    γ > 2.

	
    
        Let k ≥ 1. Using the definitions of α_k and t_k (see (<ref>) and (<ref>), respectively), we obtain that
        
    ∑_s = 0^k - 1α_st_s = 1/a∑_s = 0^k - 1(s + a)^1 - γ = 1/a∑_s = a^k + a - 1 s^1 - γ≤1/a∑_s = 2^k + a - 1 s^1 - γ,

		where the last inequality follows from the fact that a ≥ 2 since we sum (potentially) more elements. Now, we split to three cases. First, if γ = 2, we get from (<ref>) that
        
    ∑_s = 0^k - 1α_st_s≤1/a∑_s = 2^k + a - 1 s^-1≤1/aln(k + a - 1).

		For all 0 < γ≠ 2, we first obtain from (<ref>) and Lemma <ref> with n_1 = 2, n_2 = k + a - 1 and r = γ - 1, that
        
    ∑_s = 2^k + a - 1 s^1 - γ≤(k + a - 1)^2 - γ - 1^2 - γ/2 - γ = (k + a - 1)^2 - γ - 1/2 - γ.

        If 0 < γ < 2, the desired result follows immediately from (<ref>). If γ > 2, we obtain from (<ref>) that
        
    ∑_s = 0^k - 1α_st_s≤1/a·(k + a - 1)^2 - γ - 1/2 - γ = 1/a·1 - (k + a - 1)^2 - γ/γ - 2.

        This concludes the proof of the desired result.
    
	Now, we are ready to prove the first fast rate of convergence result for convex bi-level optimization problems. Here we show that our FBi-PG algorithm achieves the desired fast rate in terms of the inner problem, thus providing a significant improvement in comparison to existing results (<cit.>). 
	 
    	Let k be a sequence generated by FBi-PG with γ > 2. For all k ∈ and ' ∈ X', the following inequality holds true
    	
    φ(^k) - φ(') ≤a^2/2(k + 1)^2(β^0 - '^2 + 2/γ - 2(ω(') - ω^∗)).

	
    
        From Proposition <ref> (see (<ref>)) we have that
		
    t_k - 1^2(φ(^k) - φ(')) ≤β/2^0 - '^2 + (ω(') - ω^∗)∑_s = 0^k - 1α_st_s.

		Multiplying both sides by t_k - 1^-2 > 0 yields that
    	
    φ(^k) - φ(') ≤ t_k - 1^-2β/2^0 - '^2 + (ω(') - ω^∗)t_k - 1^-2∑_s = 0^k - 1α_st_s.

    	Now, from Lemma <ref> with γ > 2, we have that
   	 	
    t_k - 1^-2∑_s = 0^k - 1α_st_s≤t_k - 1^-2/γ - 2(1 - (k + a - 1)^2 - γ) ≤t_k - 1^-2/γ - 2,

   	 	which combined with (<ref>) yields that (recall that ω(') - ω^∗≥ 0 since ω^∗ = inf_∈^nω())
    	
    φ(^k) - φ(') ≤ t_k - 1^-2β/2^0 - '^2 + t_k - 1^-2/γ - 2(ω(') - ω^∗) = t_k - 1^-2/2(β^0 - '^2 + 2/γ - 2(ω(') - ω^∗)).

   	 	The desired result now follows from the fact that t_k - 1^-1 = a/(k + a - 1) ≤ a/(k + 1), where the inequality follows from the fact that a ≥ 2.
    
    While our primary goal was to achieve a fast rate of convergence, the results obtained above also enable us to derive the following rates of convergence concerning the inner problem. These rates are instrumental for deriving below additional results, particularly simultaneous rates pertaining to both the inner and outer optimization problems.
	 
    	Let k be a sequence generated by FBi-PG. For all k ∈ and ' ∈ X', we have
    	
    φ(^k) - φ(') ≤a^2β/2(k + 1)^2^0 - '^2 + a^2c_k/(k + 1)^2(ω(') - ω^∗),

    	where
    	
    c_k =
        		(2 - γ)^-1(k + 1)^2 - γ,     0 < γ < 2, 
    ln(k + a - 1),    γ = 2.

	
    
        From (<ref>), we obtain that
    	
    φ(^k) - φ(')    ≤ t_k - 1^-2β/2^0 - '^2 + (ω(') - ω^∗)t_k - 1^-2∑_s = 0^k - 1α_st_s
       ≤a^2β/2(k + 1)^2^0 - '^2 + (ω(') - ω^∗)t_k - 1^-2∑_s = 0^k - 1α_st_s,

    	where the last inequality follows from the fact that t_k - 1^-1 = a/(k + a - 1) ≤ a/(k + 1) since we used the fact that a ≥ 2. Now, to obtain the desired result, we split the proof to two cases. If γ = 2, we immediately get from Lemma <ref> that
    	 
    t_k - 1^-2∑_s = 0^k - 1α_st_s≤ t_k - 1^-2ln(k + a - 1) = a^2ln(k + a - 1)/(k + a - 1)^2≤a^2ln(k + a - 1)/(k + 1)^2,

   	 	where the last inequality follows from the fact that a ≥ 2. Moreover, when 0 < γ < 2, we have from Lemma <ref> that
   	 	
    t_k - 1^-2∑_s = 0^k - 1α_st_s≤t_k - 1^-2(k + a - 1)^2 - γ - t_k - 1^-2/2 - γ≤a^2/(2 - γ)(k + a - 1)^γ≤a^2/(2 - γ)(k + 1)^γ,

   	 	where the second inequality follows from the definition of t_k (see (<ref>)) after eliminating the term t_k - 1^2≥ 0 and the last inequality follows from the fact that  a ≥ 2. This concludes the proof.
    
    Now, we can prove a rate of convergence in terms of the outer optimization problem.
	 
    	Let k be a sequence generated by FBi-PG with γ∈(0 , 2). For all k ∈ and ' ∈ X', the following inequality holds true
        
    min_1 ≤ s ≤ kω(^s) - ω(') ≤a^2β/2(k + 1)^2 - γ^0 - '^2.

	
    
        For the simplicity of the proof, we denote η̅_k = 1/∑_s = 1^kη_s > 0 (recall that η_k > 0 for all k ∈, see (<ref>)). We first prove the result under the assumption that η̅_k - 1∑_s = 1^k - 1η_sω(^s) - ω(') ≥ 0. Indeed, in this case, we have that
   		
    0 ≤η̅_k - 1∑_s = 1^k - 1η_sω(^s) - ω(') = η̅_k - 1∑_s = 1^k - 1η_s(ω(^s) - ω(')),

   		and therefore
   		
    0 ≤∑_s = 1^k - 1η_s(ω(^s) - ω(')) = ∑_s = 0^k - 1η_s(ω(^s) - ω(')),

   		where the equality follows from the fact that η_0 = 0. Using Proposition <ref> (see (<ref>)) combined with (<ref>), we get that
   		
    t_k - 1^2(F_k - 1(^k) - F_k - 1(')) ≤β/2^0 - '^2 - ∑_s = 0^k - 1η_s(ω(^s) - ω(')) ≤β/2^0 - '^2.

   		From the definition of F_k - 1≡φ + α_k - 1ω we get that
    	
    t_k - 1^2(φ(^k) - φ(')) + α_k - 1t_k - 1^2(ω(^k) - ω(')) = t_k - 1^2(F_k - 1(^k) - F_k - 1(')) ≤β/2^0 - '^2,

   		and since φ(^k) - φ(') ≥ 0 for all k ∈, after dividing both sides by t_k - 1^2α_k - 1 we then obtain that
   		
    ω(^k) - ω(') ≤α_k -1^-1t_k - 1^-2β/2^0 - '^2 = a^2β/2(k + a - 1)^2 - γ^0 - '^2,

		where the last equality follows from the definitions of α_k and t_k (see (<ref>) and (<ref>), respectively). Recalling that a ≥ 2 immediately proves (<ref>) since obviously min_1 ≤ s ≤ kω(^s) ≤ω(^k). Since the proof was done under the assumption that η̅_k - 1∑_s = 1^k - 1η_sω(^s) - ω(') ≥ 0, we note that if
   		
    η̅_k - 1∑_s = 1^k - 1η_sω(^s) - ω(') ≤ 0,

		then we have that
   		
    min_1 ≤ s ≤ kω(^s) - ω(') ≤min_1 ≤ s ≤ k - 1ω(^s) - ω(') ≤η̅_k - 1∑_s = 1^k - 1η_sω(^s) - ω(') ≤ 0,

   		and therefore (<ref>) obviously true.
			
	Now, we prove a simultaneous ergodic rate of convergence in terms of both the inner and outer optimization problems.
	 
    	Let k be a sequence generated by FBi-PG with γ = 1. For all k ∈ and ' ∈ X', the following inequalities hold true
    	
    φ(^k) - φ(') ≤π^2a^2β/12k^0 - '^2 + a^2ln(k + 1)/k(ω(') - ω^∗),

		and
        
    ω(^k) - ω(') ≤a^2β/2(k + 1)^0 - '^2,

        where ^k = {ω() :   = ^k , ^k} and ^k = ∑_s = 1^k^s/k.
		
	
		In order to prove (<ref>), for all 1 ≤ s ≤ k, we get from Proposition <ref> (see (<ref>)) with k = s that (recall that γ = 1, and hence c_s≡ s + 1)
		
    φ(^s) - φ(') ≤a^2β/2(s + 1)^2^0 - '^2 + a^2/s + 1(ω(') - ω^∗).

		Therefore, from the convexity of φ we obtain that
		
    φ(^k) - φ(')    ≤1/k∑_s = 1^k(φ(^s) - φ(')) 
       ≤1/k∑_s = 1^k(a^2β/2(s + 1)^2^0 - '^2 + a^2/s + 1(ω(') - ω^∗)) 
        = 1/k(a^2β/2^0 - '^2∑_s = 2^k + 11/s^2 + a^2(ω(') - ω^∗)∑_s = 2^k + 11/s).
	
   		Now, using classical results we have that ∑_s = 2^k + 1 s^-2 < π^2/6 and ∑_s = 2^k + 1 s^-1≤ln(k + 1), which implies that
		
    φ(^k) - φ(') ≤π^2a^2β/12k^0 - '^2 + a^2ln(k + 1)/k(ω(') - ω^∗).
	
		Again from Proposition <ref> (see (<ref>)) and recall that γ = 1 (hence c_k ≡ k + 1) we obtain that
		
    φ(^k) - φ(')    ≤a^2β/2(k + 1)^2^0 - '^2 + a^2/k + 1(ω(') - ω^∗) 
       ≤π^2a^2β/12k^0 - '^2 + a^2ln(k + 1)/k(ω(') - ω^∗),

		where the last inequality follows from the fact that 1/2 < π^2/12 and thanks to the convexity of the function tln(t) for t ≥ 0 we have that k < (k + 1)ln(k + 1). This concludes the proof of (<ref>) due to the definition of ^k by combining (<ref>) and (<ref>). 
Now, in order to prove (<ref>), since γ = 1 and with t_k = (k + a)/a (see (<ref>)) we get from the definition of α_k (see (<ref>)) that
		
    η_k = α_k - 1t_k - 1^2 - α_kt_k^2 + α_kt_k = k + a - 1/a^2 - k + a/a^2 + 1/a = a - 1/a^2.
		
		Hence, η̅_k := 1/∑_s = 1^kη_k = a^2/(k(a - 1)) for all k ∈ and therefore ^k = ∑_s = 1^k^s/k = η̅_k∑_s = 1^kη_s^s.
Following the proof of Theorem <ref> (see (<ref>) and (<ref>)) we easily obtain that 
        
    min{ω(^k) , η̅_k∑_s = 1^kη_sω(^s) }  - ω(') ≤a^2β/2(k + 1)^0 - '^2.
		
        Moreover, from the convexity of ω, we get that
		
    ω(^k) ≤1/k∑_s = 1^kω(^s) = a^2/k(a - 1)∑_s = 1^kη_sω(^s) = η̅_k∑_s = 1^kη_sω(^s) ,
		
   		where the first and last equalities follow from the facts that η_k = (a - 1)/a^2 and η̅_k = a^2/(k(a - 1)), respectively. Integrating it into (<ref>) yields that
		
    min{ω(^k) , ω(^k) } - ω(') ≤min{ω(^k) , η̅_k∑_s = 1^kη_sω(^s) } - ω(') ≤a^2β/2(k + 1)^0 - '^2.
	
   		The desired result now follows immediately from the definition of ^k.
	



 §.§ Improved Simultaneous Rates
 
	In order to obtain improved simultaneous rates of convergence, we will impose an Hölderian type error bound assumption on the inner objective function φ. For more details and examples of functions satisfying such property, we refer the interested readers to <cit.>. We first recall the definition.
	
		A proper, lower semicontinuous and convex function h : ^n→ satisfies an Hölderian error bound with a parameter τ > 0 if for all ∈^n
		
    τ·( ,  h)^2≤ h() - min_∈^n h().
			
	
	Throughout this section, we make the following additional assumption.
	 
    	The following items hold.
    	
        	
  (i) φ satisfies the Hölderian error bound with a parameter τ > 0.
        	
  (ii) ω∩ X' ≠∅.
    	
	
	It should be noted that Assumption B(ii) is very mild, as any function with a full domain, such as ·_1, would satisfy it. Moreover, note that the subdifferential ∂ω(') is not empty for any ' ∈ω∩ X'.
Before establishing the promised improved simultaneous rates of convergence, we first need the following simple bound for η_k, k ∈, which will be used throughout the section. 
	  
		1 < γ < 2. For all k ∈, one has 
		
    η_k  < 1/2(k + 1)^1 - γ.

   	
	 
		Using the definitions of α_k = (k + a)^-γ and t_k = (k + a)/a for all k ∈, it follows that
		
    η_k = α_k - 1t_k - 1^2 - α_k(t_k^2 - t_k) = 1/a^2(k + a - 1)^2 - γ - 1/a^2(k + a)^2 - γ + 1/a(k + a)^1 - γ < 1/2(k + 1)^1 - γ,

		where the last inequality follows from the facts that (k + a - 1)^2 - γ < (k + a)^2 - γ and (k + a)^1 - γ < (k + 1)^1 - γ since 1 < γ < 2 and a ≥ 2 > 1. 
	
	In the next theorem, we establish the improved simultaneous convergence rates.
	 
    	Let k be a sequence generated by FBi-PG with 1 < γ < 2. For all k ∈ and ' ∈ω∩ X', the following inequalities hold true
    	
        	
  (i) φ(^k) - φ(') ≤aC/(k + 1)^2,
            
  (ii) ω(') - ω(^k) ≤C/a^2(k + 1)
        	
  (iii) ω(^k) - ω(') ≤C/(k + 1)^2 - γ,
    	
    	where C = a^2max{β^0 - '^2 + (ω(') - ω^∗) , a^3ρ^2τ^-1(γ - 1)^-2} and ρ = ξ for some ξ∈∂ω(').
	
    
        For the simplicity of the proof, we denote Φ_k = φ(^k) - φ(') for all k ∈. We will prove the first item using an induction argument on k ∈. For k = 1, we obtain from Proposition <ref> that (recall that t_0 = 1)
        
    Φ_1 = t_0^-1(φ(^1) - φ(')) ≤β/2^0 - '^2 + (ω(') - ω^∗)α_0t_0≤C/2a^2≤aC/4,

		where the second inequality follows from the definition of C and the fact that a_0 = a^-γ < 1/2 since γ > 1 and a ≥ 2, and the last inequality also follows from the fact that a ≥ 2. This proves the induction base with k = 1.
Let k ∈. We assume now that item (i) is true for all  1 ≤ s ≤ k - 1 and we will prove it for s = k. For all 1 ≤ s ≤ k - 1, we have that
        
    ξ^T(' - ^s) = ξ^T(' - P_X^∗(^s)) + ξ^T(P_X^∗(^s) - ^s) ≤ξ^T(P_X^∗(^s) - ^s),

        where P_X^∗ denotes the orthogonal projection onto the nonempty set X^∗. It should be noted that the last inequality follows from the first-order optimality condition of the constrained minimization of ω over X^∗, which reads (recall that X' = {ω() :  ∈ X^∗}):
		
    ∈ξ + ∂δ_X^∗(') = ξ + N_X^∗('),
		
        where ξ∈∂ω('), and hence with the definition of the normal cone N_X^∗(·) it follows that ξ^T(' - ) ≤ 0 for all ∈ X^∗. Therefore, since P_X^∗(^s) ∈ X^∗, one has ξ^T(' - P_X^∗(^s)) ≤ 0. Now, using the Hölderian error bound of the function φ (see Assumption <ref>(i)) yields
        
    ξ^T(P_X^∗(^s) - ^s) ≤ξ·P_X^∗(^s) - ^s = ρ·(^s , X^∗) ≤ρ/√(τ)√(Φ_s),

        where the equality follows from the facts that ξ = ρ and the definition of the distance function. Hence, using the gradient inequality on the convex function ω, we obtain that (recall that ξ∈∂ω('))
        
    ω(') - ω(^s) ≤ξ^T(' - ^s) ≤ξ^T(P_X^∗(^s) - ^s) ≤ρ/√(τ)√(Φ_s),

        where the second inequality follows from (<ref>) and the last inequality follows from (<ref>). Using the induction assumption on Φ_s for all 1 ≤ s ≤ k - 1, we have thus obtained
        
    ω(') - ω(^s) ≤ρ/√(τ)√(Φ_s)≤ρ√(a)√(C)/√(τ)(s + 1)^-1.

       Therefore, using (<ref>) and (<ref>), we obtain that (recall that η_0 = 0)
        
    ∑_s = 0^k - 1η_s(ω(') - ω(^s)) = ∑_s = 1^k - 1η_s(ω(') - ω(^s)) ≤ρ√(a)√(C)/2√(τ)∑_s = 1^k - 1(s + 1)^-γ = ρ√(a)√(C)/2√(τ)∑_s = 2^k s^-γ.

		Using Lemma <ref> with n_1 = 2, n_2 = k and r = γ > 1, we have
        
    ∑_s = 0^k - 1η_s(ω(') - ω(^s)) ≤ρ√(a)√(C)/2√(τ)∑_s = 2^k s^-γ = ρ√(a)√(C)/2√(τ)·1^1 - γ - (k - 1)^1 - γ/γ - 1≤ρ√(a)√(C)/2√(τ)(γ - 1).

        From Proposition <ref> and (<ref>), we get that
    	
    t_k - 1^2(F_k - 1(^k) - F_k - 1(')) ≤β/2^0 - '^2 - ∑_s = 0^k - 1η_s(ω(^s) - ω(')) ≤β/2^0 - '^2 + ρ√(a)√(C)/2√(τ)(γ - 1).

		Hence, from the definition of C, we get
    	
    t_k - 1^2(F_k - 1(^k) - F_k - 1(')) ≤C/2a^2 + √(C)/2·√(C)/a^2 = C/a^2.

    	Moreover, using (<ref>) with s = k, we have that
    	
    ω(^k) - ω(') ≥ -ρ/√(τ)√(Φ_k).

    	Using this, the definition of F_k - 1≡φ + α_k - 1ω and (<ref>) yields that
    	
    Φ_k - ρα_k - 1/√(τ)√(Φ_k)   ≤Φ_k + α_k - 1(ω(^k) - ω(')) = F_k - 1(^k) - F_k - 1(') ≤Ct_k - 1^-2/a^2≤C/(k + 1)^2,

    	where the last inequality follows from the fact that t_k - 1^-1 = a/(k + a - 1) ≤ a/(k + 1) since a ≥ 2. Now, we will prove the induction step by splitting the proof into two cases. First, we assume that
    	
    Φ_k/a≤Φ_k - ρα_k - 1/√(τ)√(Φ_k).

    	Using (<ref>) yields that
    	
    Φ_k/a≤Φ_k - ρα_k - 1/√(τ)√(Φ_k)≤C/(k + 1)^2,

    	and therefore we obviously have that Φ_k≤ aC/(k +  1)^2, as required. Second, we assume the converse, that is
    	
    Φ_k - ρα_k - 1/√(τ)√(Φ_k) < Φ_k/a.

    	Therefore, by rearranging the inequality we get that Φ_k (a-1)/a < (ρα_k - 1/√(τ))√(Φ_k). Dividing both sides by (a - 1)√(Φ_k)/a and then squaring both sides yields
    	
    Φ_k < a^2ρ^2/τ(a-1)^2α_k - 1^2≤a^2ρ^2/τ(a-1)^2(k + 1)^-2≤a^2ρ^2/τ(γ - 1)^2(k + 1)^-2≤C/a^3(k + 1)^2,

    	where the second inequality follows from the definition of α_k - 1 = (k + a - 1)^-γ < (k + 1)^-1 since γ > 1 and a ≥ 2, the third inequality follows from the fact that 0 < γ - 1 < a - 1 since a ≥ 2 and 1 < γ < 2 while the last inequality follows again from the definition of C. Thus, we obtain that Φ_k≤ aC/(k + 1)^2 since a ≥ 2, which proves the desired result.
Now, we will prove the second item. Using (<ref>) with s = k, yields
        
    ω(') - ω(^k) ≤ρ√(a)√(C)/√(τ)(k + 1)^-1≤ρ√(a)√(C)/√(τ)(γ - 1)(k + 1)^-1≤C/a^2(k + 1)^-1,

		where the second inequality follows from the fact that 1 < (γ - 1)^-1 since 1 < γ < 2 and the last inequality follows from the definition of C. For proving the last item, using (<ref>), we have that
    	
    α_k - 1(ω(^k) - ω(')) ≤Φ_k + α_k - 1(ω(^k) - ω(')) = F_k - 1(^k) - F_k - 1(') ≤Ct_k - 1^-2/a^2,

    	where the first inequality follows from the fact that Φ_k≥ 0 and the equality follows from the definition of F_k - 1≡φ + α_k - 1ω. Hence, immediately from the definitions of α_k and t_k (see (<ref>) and (<ref>), respectively) we obtain that
    	
    ω(^k) - ω(') ≤Ct_k - 1^-2/a^2α_k - 1 = C/(k + a - 1)^2 - γ≤C/(k + 1)^2 - γ,

        where the last inequality follows from the fact that a - 1 ≥ 1 since a ≥ 2. This proves the desired result.
	
	


§ POINTWISE CONVERGENCE ANALYSIS
 
	This section is devoted for proving that the sequence k generated by FBi-PG converges to an optimal solution of the bi-level optimization problem (<ref>). This result is achieved under the Assumption <ref>, meaning in the setting where the inner objective function φ(·) satisfies the Hölderian error bound property. Moreover, throughout this section we assume that the sequences αk and tk are defined with 1 < γ < 2 and a > 2. 


	As already mentioned, each iteration of FBi-PG can be seen as an iteration of FISTA applied to the regularized function F_k. Unfortunately, due to the dynamic nature of the regularizing sequence αk, much like we could not apply the convergence rate results of FISTA to get corresponding fast rate for FBi-PG, we also cannot directly apply the existing pointwise convergence results of FISTA established in <cit.> to derive the corresponding convergence of the sequence k generated by FBi-PG to an optimal solution of the bi-level optimization problem (<ref>). Nevertheless, inspired by the proof techniques developed in <cit.>, below we show that within a similar line of proof,  we can adapt and extend  their result for handling the bi-level setting.


	Throughout the rest of this section, we will define for all k ∈ and ' ∈ X' the following two quantities
	
    δ_k = 1/2^k - ^k - 1^2  and  μ_k = 1/2^k - '^2.

	The main tasks of the forthcoming analysis will be to establish that both sequences δk and μk converge, from which the culminating main convergence result will follow by standard arguments.


	We begin with a technical lemma (see appendix for the proof) that will be useful in proving the main result.
	 
    	Let k be a sequence generated by FBi-PG with 1 < γ < 2 and a > 2. Then, there exists M > 0 such that ∑_s = 0^k - 1 t_s - 1δ_s < M for all k ∈.
	
	For all s ∈, we define π_s,k = 1 for k < s and otherwise we define
	
    π_s,k := ∏_j = s^kλ_j,

	where λ_k := t_k^-1(t_k - 1 - 1) for all k ∈. 


	The following bound is based on a result obtained in <cit.> with a minor modification to our setting.
	
		For all s ≥ 1 we have that
		
    ∑_k = 0^∞π_s,k≤5a/2t_s - 1.

	
	
		Let s ≥ 1. Since π_s,k = 1 for all k < s we obtain that
		
    ∑_k = 0^∞π_s,k = ∑_k = 0^s - 1π_s,k + ∑_k = s^∞π_s,k = ∑_k = 0^s - 1 1 + ∑_k = s^∞π_s,k = s + ∑_k = s^∞π_s,k≤ s + s + 5/2 = 3s + 5/2,

		where the inequality follows from <cit.>. Now, since a - 1 > 1 we have that 3s + 5 < 5(s + a - 1) = 5at_s - 1 where the last equality follows from the definition of t_s (see (<ref>)). This completes the proof.
	
 	The next theorem proves that the FBi-PG's sequence, k, is convergent to an optimal solution of the bi-level problem (<ref>).
	 
		Let k be a sequence generated by FBi-PG with 1 < γ < 2 and a > 2. Then, the sequence k converges to some ' ∈ X'.
	
    
		Let ' ∈ X'. Since X' ⊂ X^∗ it follows that ∈∂φ(') = ∇ f(') + ∂ g('). Therefore, -∇ f(') ∈∂ g('). Taking ' ∈∂ψ('), it follows that
		
    -∇ f(') + α_k' ∈∂ g(') + α_k∂ψ(') ⊂∂(g + α_kψ)(') = ∂ g_k('),

		where the inclusion follows from standard convex subdifferential calculus <cit.> and the last equality follows from the definition of g_k≡ g + α_kψ. Writing the optimality condition of the proximal step (<ref>) yields that
		
    β(^k - ^k + 1) - ∇ f_k(^k) ∈∂ g_k(^k + 1).

		Now, since the function g_k is convex it follows that its subdifferential ∂ g_k is a monotone mapping <cit.>. Therefore, we have that
		
    0    ≤β(^k - ^k + 1) - ∇ f_k(^k) - (-∇ f(') + α_k') , ^k + 1 - '
        = β(^k - ^k + 1) - ∇ f_k(^k) + ∇ f_k(') - α_k(∇σ(') + ') , ^k + 1 - '
        = β^k - ^k + 1 , ^k + 1 - ' + ∇ f_k(') - ∇ f_k(^k) , ^k + 1 - ' + α_k∇σ(') + ' , ' - ^k + 1,

		where the first equality follows from the fact that f_k≡ f + α_kσ. Since f_k is convex and has a Lipschitz continuous gradient with a parameter β (see Remark <ref>), the later is also (1/β)-co-coercive mapping and hence we have
		
    ∇ f_k(') - ∇ f_k(^k) , ^k + 1 - '    = ∇ f_k(') - ∇ f_k(^k) , ^k + 1 - ^k + ^k - '
       ≤∇ f_k(') - ∇ f_k(^k) , ^k + 1 - ^k - 1/β∇ f_k(') - ∇ f_k(^k)^2
       ≤1/β∇ f_k(') - ∇ f_k(^k)^2 + β/4^k + 1 - ^k^2
        - 1/β∇ f_k(') - ∇ f_k(^k)^2
        = β/4^k + 1 - ^k^2,

		where the last inequality follows from the fact that ^T≤^2/β + β^2/4 for all , ∈^n and β > 0. Combining (<ref>) and (<ref>) yields after dividing both sides by β that
		
    0 ≤^k - ^k + 1 , ^k + 1 - ' + 1/4^k + 1 - ^k^2 + ξ^k + 1,

		where ξ^k≡α_k - 1∇σ(') + ' , ' -  ^k/β for all k ∈. Moreover, using the fundamental Pythagoras identity we obtain that
		
    ^k - ^k + 1 , ^k + 1 - '    = ^k - ^k , ^k + 1 - ' + ^k - ^k + 1 , ^k + 1 - '
        = ^k - ^k , ^k + 1 - ' + 1/2(^k - '^2 - ^k + 1 - '^2 - ^k + 1 - ^k^2) 
        = ^k - ^k , ^k + 1 - ' + μ_k - μ_k + 1 - δ_k + 1,

		where the third equality follows from the definitions of δ_k and μ_k (see (<ref>)). Since λ_k = t_k^-1(t_k - 1 - 1) we can write from (<ref>) that ^k - ^k = λ_k(^k  - ^k - 1). Using again the fundamental Pythagoras identity we obtain that
		
    ^k - ^k , ^k + 1 - '    = ^k - ^k , ^k + 1 - ^k + ^k - ^k , ^k - '
        = ^k - ^k , ^k + 1 - ^k + λ_k^k - ^k - 1 , ^k - '
        = ^k - ^k , ^k + 1 - ^k + λ_k(μ_k - μ_k - 1 + δ_k).

		Another application of the Pythagoras identity yields that
		
    ^k - ^k , ^k + 1 - ^k    = 1/2(^k - ^k^2 - ^k + 1 - ^k^2 + ^k + 1 - ^k^2) 
        = λ_k^2δ_k - 1/2^k + 1 - ^k^2 + δ_k + 1,

		where the last equality uses the fact that ^k - ^k = λ_k(^k - ^k - 1) and the definition of δ_k. Combining (<ref>), (<ref>), (<ref>) and (<ref>) yields after eliminating -^k + 1 - ^k^2/4 ≤ 0 that
		
    μ_k + 1 - μ_k≤λ_k(μ_k - μ_k - 1 + δ_k) + λ_k^2δ_k + ξ^k + 1 = λ_k(μ_k - μ_k - 1) + (λ_k^2 + λ_k)δ_k + ξ^k + 1.

		Using the fact that λ_k + λ_k^2≤ 2λ_k since λ_k^2≤λ_k (recall that λ_k≤ 1 for all k ∈), we obtain
		
    μ_k + 1 - μ_k≤λ_k(μ_k - μ_k - 1) + 2λ_kδ_k + ξ^k + 1.

		By defining θ_k = [μ_k - μ_k - 1]_+ := max{ , μ_k - μ_k - 1} for all k ≥ 1, it follows from (<ref>) that
		
    θ_k + 1    = [μ_k + 1 - μ_k]_+
       ≤[λ_k(μ_k - μ_k - 1) + 2λ_kδ_k + ξ^k + 1]_+
       ≤λ_kθ_k + 2λ_k[δ_k]_+ + [ξ^k + 1]_+
        = λ_k(θ_k + 2δ_k) + [ξ^k + 1]_+,

		where the first inequality follows from the fact that [x]_+≤ [y]_+ when x ≤ y, the second inequality follows from the fact that [x + y]_+≤ [x]_+ + [y]_+, and the equality follows from the fact that λ_k , δ_k≥ 0. Thus, applying (<ref>) successively yields, for all k ≥ 2, that (we use the convention that ∏_j = k + 1^kλ_j = 1)
		
    θ_k + 1   ≤λ_kθ_k + 2λ_kδ_k + [ξ^k + 1]_+
       ≤λ_k(λ_k - 1(θ_k - 1 + 2δ_k - 1) + [ξ^k]_+) + 2λ_kδ_k + [ξ^k + 1]_+
        = λ_kλ_k - 1θ_k - 1 + 2λ_kλ_k - 1δ_k - 1 + 2λ_kδ_k + λ_k[ξ^k]_+ + [ξ^k + 1]_+
        = ∏_j = 2^kλ_jθ_2 + 2∑_s = 2^k∏_j = s^kλ_jδ_s + ∑_s = 3^k + 1∏_j = s^kλ_j[ξ^s]_+
       ≤ 2∑_s = 2^k∏_j = s^kλ_jδ_s + ∑_s = 2^k + 1∏_j = s^kλ_j[ξ^s]_+
        = 2∑_s = 2^kπ_s,kδ_s + ∑_s = 2^k + 1π_s,k[ξ^s]_+,

		where the last inequality follows from (<ref>) with k = 1 which implies that θ_2≤[ξ^2]_+ since λ_1 = 0 (recall λ_1 = t_1^-1(t_0 - 1) = 0) and the last equality follows from the definition of π_s,k (see (<ref>)). Note that the inequality (<ref>) also holds true for k = 1 under the convention that ∑_s = 2^1π_s,kδ_s = 0 (recall that π_2,1 = 1). Therefore, summing (<ref>) from k = 0 to ∞, we have
		
    ∑_k = 0^∞θ_k + 1   ≤ 2∑_k = 0^∞∑_s = 2^kπ_s,kδ_s + ∑_k = 0^∞∑_s = 2^k + 1π_s,k[ξ^s]_+
       ≤ 2∑_k = 0^∞∑_s = 2^∞π_s,kδ_s + ∑_k = 0^∞∑_s = 2^∞π_s,k[ξ^s]_+
        = 2∑_s = 2^∞(∑_k = 0^∞π_s,k)δ_s + ∑_s = 2^∞(∑_k = 0^∞π_s,k)[ξ^s]_+
       ≤ 5a∑_s = 2^∞ t_s - 1δ_s + 5a/2∑_s = 2^∞ t_s - 1[ξ^s]_+,

		where the last inequality follows from Lemma <ref>. Next, we show that θk is summable. This goal will be achieved by showing that each of two terms on the right-hand side of (<ref>) is finite. From Lemma <ref> there exists M_1 > 0 such that
		
    ∑_s = 2^∞ t_s -1δ_s≤∑_s = 0^∞ t_s -1δ_s≤ M_1,

		proving the finiteness of the first term. Now, from the definition of ξ^k and since ∇σ(') + ' ∈∂ω(') (recall that ω≡σ + ψ and ' ∈∂ψ(')), we have that
		
    ξ^k = α_k - 1/β∇σ(') + ' , ' -  ^k = α_k - 1/βξ , ' -  ^k,

		with ξ∈∂ω('), and hence we obtain that
		
    ξ^k = α_k - 1/βξ , ' -  ^k≤ρα_k - 1/β√(τ)√(φ(^k) - φ('))≤ρα_k - 1√(aC)/β√(τ)(k + 1) = ρ√(aC)/β√(τ)(k + 1)^1 + γ,

		where the first inequality follows from (<ref>) combined with (<ref>), the second inequality follows from Theorem <ref>(i) and the last equality follows from the definition of α_k := (k + a)^-γ (see (<ref>)) and the fact that (k + a - 1)^-γ < (k + 1)^-γ since a > 2. Therefore, from (<ref>), we deduce that
		
    ∑_s = 1^∞ t_s - 1[ξ^s]_+≤ρ√(aC)/β√(τ)∑_s = 1^∞t_s - 1/(s + 1)^1 + γ≤ρ√(aC)/2β√(τ)∑_s = 1^∞1/(s + 1)^γ = ρ√(aC)/2β√(τ)∑_s = 2^∞1/s^γ = ρ√(aC)/2β√(τ)·1/γ - 1,

		where the second inequality follows from the fact that t_s - 1 = (s - 1)/a + 1 ≤(s - 1)/2 + 1 = (s + 1)/2 (recall that a > 2) and the last inequality follows from Lemma <ref> with n_1 = 2, n_2 = ∞ and r = γ > 1. This proves the finiteness of the second term in (<ref>) and therefore completes the proof that θk is summable.
		
		Now, we define a_k = μ_k - ∑_s = 1^kθ_s and since μ_k≥ 0 for all k ∈ and θk is summable, we see that ak is bounded from below. On the other hand, since θ_k = [μ_k - μ_k - 1]_+, we get that
		
    a_k + 1 = μ_k + 1 - θ_k + 1 - ∑_s = 1^kθ_s≤μ_k + 1 - μ_k + 1 + μ_k - ∑_s = 1^kθ_s = a_k,

		and hence ak is also a non-increasing sequence and thus is convergent. Since μ_k = a_k + ∑_s = 1^kθ_s and both sequences converge, it follows that μk converges. To complete the proof we will prove that the limit is zero. Since μk converges it is also bounded, and therefore from the definition of μ_k, k ∈, we also obtain that k is bounded. Hence, it has at least one converging sub-sequence, which we denote by {^j_k}_k ∈ and its limit point with . From the fact that φ is lower semi-continuous (see Assumption <ref>)) and using Theorem <ref>(i), it follows that
		
    φ() ≤lim inf_k →∞φ(^j_k) = φ(').

		Similarly, we obtain that ω() ≤ω('). Since ' ∈ X', these two facts yield that ∈ X'. On the other hand, as we proved above, the sequence {^k - }_k ∈ converges for all ∈ X' and in particular for =. Thus,
		
    lim_k →∞^k -  = lim_k →∞^j_k -  = 0,

		which concludes the proof.
	
	


§ NUMERICAL RESULTS
 
	This section is devoted to evaluate FBi-PG numerically and compare its performance against existing algorithms that are proposed to solve convex bi-level optimization problems. For this purpose, we will follow the numerical experimental set-up of <cit.>. Here is the description of the numerical experiments. We conduct two experiments on real-world problems
	
		
  * Songs release year prediction using linear regression
		
  * Fake news classification using logistic regression.
	
	In each experiment, we run FBi-PG with three parameters γ = 1.3, γ = 1.5 and γ = 3 (all with a = 2), which corresponds to three possible inner rates of convergence (see Theorem <ref> and Proposition <ref>), and compare it to the following algorithms
	
    	
  (i) Bi-SG of <cit.> with α = 0.95 and c = 1.
    	
  (ii) FISTA <cit.> applied on the regularized problem (<ref>) with a fixed regularization parameter α_K = 1/K when the number of iterations is K = 10^5.
	
	It should be noted that due to the numerical results of <cit.>, which provide a numerical comparison between several existing algorithm for convex bi-level optimization problems, we present here only the Bi-SG <cit.>, which was shown to be superior in the experiments conducted there.
All experiments were run on 11th Gen Intel(R) Core(TM) i7-1195G7 @ 2.90GHz 2.92 GHz with a total RAM memory of 16GB and 4 physical cores (using a free account of Google colab notebook). We run each algorithm from the same starting point ^0 = and provide details about both the inner and outer optimization problems. In terms of the inner optimization problem, for each method 1 ≤ l ≤ 5, we record the following measure over iterations
	
    Δ^lφ(k) := φ(_(l)^k) - φ^∗,
	
	where _(l)^k is the point computed by the method l after k iterations, and φ^∗ is the optimal value of φ, which we compute using an off-the-shelf solver[https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression].
As for the outer optimization problem, we present a plot that record the progress of the values of ω versus -log(Δ^lφ(k)), namely the number of accuracy digits, for an algorithm l at iteration k. This plot aims at understanding the behaviour of the algorithm in solving both the inner and outer optimization problems. On the x-axis we plot the values of -log(Δ^lφ(k)), which is an indicator of how accurate the inner problem has been solved. On the y-axis we plot the values of the outer objective function ω as we get more accurate in solving the inner problem. Since the x-axis represents the negative of (the log of) the optimality gap of the inner-level problem, algorithms that were not able to achieve a good accuracy in solving the inner problem appear as a “short" line in this plot. In between the algorithms that achieve a good accuracy in the inner optimality gap, we seek the algorithm that obtains the minimal value of ω.



 §.§ Songs Release Year Prediction Using Linear Regression
	
	In this experiment, we used the YearPredictionMSD dataset[https://archive.ics.uci.edu/ml/datasets/yearpredictionmsd] to train a linear regression model to predict songs release years. We considered a sample of the data-set with 2000 uniformly i.i.d randomly selected songs (without replacements). For each sample, we use A to denote the feature matrix induced from it. Therefore, the (i , j) entry of A is the value of the j-th attribute of the i-th song in the sample. In addition, we use b to denote the release year of the songs in the sample. That is, the i-th entry in  is the release year of the i-th song in the sample. Our goal is to train a predictor by finding a weights vector  that minimizes the following objective function
	
    φ() = 1/2NA - ^2,

	where N is the number of songs in the sample. In this case, we consider the outer objective function ω≡·_1. It should be noted that this inner objective function indeed satisfies the Hölderian error bound property <cit.>. 

	







	
	In this experiment, we observe that the fixed regularization parameter we utilized performs comparably to the FBi-PG algorithm with all values of γ. However, as discussed in the paper, the major drawback of this approach is the inability of determining the “right" regularization parameter in advance when solving a problem. This will be illustrated numerically in the forthcoming additional example.



 §.§ Fake News Classification Using Logistic Regression

	Following the problem description in <cit.>, we immediately present the optimization model. We are given a matrix A ∈^N × m and a vector ∈^N, which contains the labels of the articles. The corresponding logistic loss function is defined by				
	
    φ() ≡1/N∑_i = 1^N(log(w(^T_i))_i + log(1 - w(^T_i))(1 - _i) ),

	where w : →_++ is the sigmoid function defined by w(t) = exp^t/(1 + exp^t) and = (_1 , _2 , … , _N)^T for which _i∈{0 , 1 }. We note that the logistic loss function satisfies the Hölderian error bound property <cit.> and therefore our improved simultaneous rate and convergence guarantees proven in Theorems <ref> and <ref> are valid, respectively. 
In this example, the bi-level approach can be used here to reduce the number of features in the vector  obtained from minimizing the logistic loss φ(·). In other words, we wish to find the sparsest vector in between all the minimizers of the logistic regression function φ(·). Therefore, we take the following classical outer objective function
	
    ω() = _1.
	
	In our experiment, we seek to differentiate between fake and genuine newspaper articles using a dataset from a Kaggle competition[Refer to https://www.kaggle.com/competitions/fakenews/overeview] comprising of 10,413 real and 10,387 fake news articles. Each article is accompanied by textual content and its authenticity status. We randomly uniformly select 500 fake and real newspapers, employing the pre-processing steps from <cit.>, and represent each article with 250 features as detailed in <cit.>.
	
	
	
	In this case, we see that the fixed regularization parameter that we choose doesn't solve the bi-level optimization problem since it “gets stuck" when solving the inner problem. This means that for our choice of regularizing parameter, optimal solutions of the regularized problem don't coincide with optimal solutions of the bi-level optimization problem. However, we see in the left-hand side plot that all versions of FBi-PG solve the inner problem. Moreover, as our theory shows, the inner rate of FBi-PG is getting better as γ increases.
	

		


§ APPENDIX: PROOF OF LEMMA <REF>
 
	We begin with two technical lemmas that will be useful in proving the required result. Before doing so, we recall the definitions of the sequences dk and ηk, which will be used below, and are respectively defined by
	
    d_k := t_k - 1^2 - (t_k^2 - t_k)   and  η_k := t_k - 1^2α_k - 1 - (t_k^2 - t_k)α_k.

	We begin with a result that will be used in the sequel.
     
    	Let k be a sequence generated by FBi-PG with 1 < γ < 2 and a > 2. Suppose that bk⊆_+ is a sequence satisfying b_k≤ bη_k for all k ∈ and some b > 0. Then, there exists M > 0 such that ∑_s = 0^k - 1 b_s(ω(') - ω(^s)) ≤ M for all k ∈ and ' ∈ X'.
    
    
        Let k ∈. From Theorem <ref>(ii), we have for all k ∈ that
        
    ∑_s = 0^k - 1 b_s(ω(') - ω(^s)) ≤∑_s = 0^k - 1 b_s[ω(') - ω(^s)]_+≤C/a^2∑_s = 0^k - 1 b_s(s + 1)^-1,

        where [x]_+≡max{0 , x}. Using (<ref>), we have that b_s≤ bη_s < b(s + 1)^1 - γ/2 for all s ∈. Thus, we obtain that b_s(s + 1)^-1≤ b(s + 1)^-γ/2. Now, from Lemma <ref> with n_1 = 2, n_2 = k and r = γ > 1, we get that
        
    ∑_s = 0^k - 1(s + 1)^-γ = ∑_s = 1^k s^-γ = 1 + ∑_s = 2^k s^-γ≤ 1 + 1^1 - γ - k^1 - γ/γ - 1≤ 1 + 1/γ - 1 = γ/γ - 1.

        Combining (<ref>) with (<ref>) yields the desired result.
    
    Before proving Lemma <ref> about the summability of the sequence { t_k - 1δ_k - 1}_k ∈, we prove the summability of the sequence { d_k(φ(^k) - φ(')) }_k ∈.
	 
    	Let k be a sequence generated by FBi-PG with 1 < γ < 2 and a > 2. Then, there exists M > 0 such that ∑_s = 0^k - 1 d_s(φ(^s) - φ(')) < M for all k ∈ and ' ∈ X'.
	
	
    	Let k ∈ and ' ∈ X'. From the proof of Proposition <ref> ((<ref>)), we have that
   		
    t_k - 1^2(F_k - 1(^k) - F_k - 1(')) ≤β/2^0 - '^2 - ∑_s = 0^k - 1 d_s(φ(^s) - φ(')) - ∑_s = 0^k - 1η_s(ω(^s) - ω(')).

     	Rearranging this inequality yields that
     	
    t_k - 1^2(F_k - 1(^k) - F_k - 1(')) + ∑_s = 0^k - 1 d_s(φ(^s) - φ(')) ≤β/2^0 - '^2 + ∑_s = 0^k - 1η_s(ω(') - ω(^s)).

    	Since ' ∈ X', it follows that φ(^k) - φ(') ≥ 0. Thus, combining with the fact that F_k - 1≡φ + α_k - 1ω, it follows from Theorem <ref>(ii) that
    	
    t_k - 1^2(F_k - 1(^k) - F_k - 1(')) ≥α_k - 1t_k - 1^2(ω(^k) - ω(')) ≥ -α_k - 1t_k - 1^2C/a^2(k + 1).

        Using now the definitions α_k and t_k (see (<ref>) and (<ref>), respectively) yields that
    	
    t_k - 1^2(F_k - 1(^k) - F_k - 1(')) ≥ -C(k + a - 1)^2 - γ/a^4(k + 1)≥ -C(k + a - 1)/a^4(k + 1)≥ -C(ak + a)/a^4(k + 1) = -C/a^2,

    	where the second inequality from the fact that (k + a - 1)^2 - γ≤ k + a - 1 since 2 - γ < 1 (recall that γ > 1) and  a - 1 > 0 (recall that a > 2), and the last inequality follows from the fact that k + a - 1 < k + a < ak + a since a > 1. By plugging it into (<ref>) and adding C/a^3 to both sides, we get that
    	
    ∑_s = 0^k - 1 d_s(φ(^s) - φ(')) ≤C/a^3 + β/2^0 - '^2 + ∑_s = 0^k - 1η_s(ω(') - ω(^s)).

    	Applying Lemma <ref> with b_k≡η_k and b = 1 it follows that there exists M̃ > 0 such that ∑_s = 0^k - 1η_s(ω(') - ω(^s)) ≤M̃. Hence, using this bound in  (<ref>) yields the desired result.
    
	Now, we are ready to prove Lemma <ref>.


    Proof of Lemma <ref>.
   		Let k ∈. From Proposition <ref> with = ^k, we obtain that
        
    t_k^2(F_k(^k + 1) - F_k(^k)) + β/2^k + 1 - ^k^2≤β/2^k - ^k^2.

        Rearranging the inequality yields that
        
    β/2^k + 1 - ^k^2 - β/2^k - ^k^2≤ t_k^2(F_k(^k) - F_k(^k + 1)).

        From (<ref>) and the definition of the sequence ^k we have that ^k + 1 - ^k = t_k(^k + 1 - ^k) and ^k - ^k = (t_k - 1 - 1)(^k - ^k - 1). Therefore, using the definition of δ_k (see (<ref>)), we obtain that
        
    β/2^k + 1 - ^k^2 - β/2^k - ^k^2    = t_k^2β/2^k+1 - ^k^2 - (t_k-1 - 1)^2β/2^k - ^k-1^2
        = β(t_k^2δ_k - (t_k - 1 - 1)^2δ_k - 1) 
        = β(t_k^2δ_k - t_k - 1^2δ_k - 1) + β(2t_k - 1 - 1)δ_k - 1
       ≥β(t_k^2δ_k - t_k - 1^2δ_k - 1) + β t_k - 1δ_k - 1,

        where the last inequality follows from the facts that t_k≥ 1 and δ_k≥ 0 for all k ∈. Plugging this into (<ref>) with k = s yields that
        
    β(t_s^2δ_s - t_s - 1^2δ_s - 1) + β t_s - 1δ_s - 1≤ t_s^2(F_s(^s) - F_k(^s + 1)).

        Summing (<ref>) for s = 0 , 1 , … , k - 1, we get that
        
    β∑_s = 0^k - 1 t_s - 1δ_s - 1 + β(t_k - 1^2δ_k - 1 - t_-1^2δ_-1) = β∑_s = 0^k - 1(t_s^2δ_s - t_s - 1^2δ_s - 1 + t_s - 1δ_s - 1) ≤∑_s = 0^k - 1 t_s^2(F_s(^s) - F_s(^s + 1)).

	   	Since t_-1 = 0 and t_k - 1^2δ_k - 1≥ 0 we obtain that
        
    β∑_s = 0^k - 1 t_s - 1δ_s - 1≤∑_s = 0^k - 1 t_s^2(F_s(^s) - F_s(^s + 1)).

	   	To complete the proof we will prove that the sum on the right-hand side is bounded by some M > 0 for any k ∈. By defining Φ_k = φ(^k) - φ(') and Ω_k = ω(') - ω(^k) for all k ∈, we obtain from the definition of F_k≡φ + α_kω that
        
    F_k(^k) - F_k(^k + 1) = F_k(^k) - F_k(') + F_k(') - F_k(^k + 1) = Φ_k - Φ_k + 1 - α_k(Ω_k - Ω_k + 1),

		and hence
		
    ∑_s = 0^k - 1 t_s^2(F_s(^s) - F_s(^s + 1))     = ∑_s = 0^k - 1 t_s^2(Φ_s - Φ_s + 1 - α_s(Ω_s - Ω_s + 1)) 
        = t_0^2Φ_0 - t_k - 1^2Φ_k + ∑_s = 1^k - 1(t_s^2 - t_s - 1^2)Φ_s - α_0t_0^2Ω_0 + α_k - 1t_k - 1^2Ω_k
        - ∑_s = 1^k - 1(α_st_s^2 - α_s - 1t_s - 1^2)Ω_s
       ≤ t_0^2Φ_0 - α_0t_0^2Ω_0 + a/a - 2∑_s = 1^k - 1 d_sΦ_s + α_k - 1t_k - 1^2Ω_k + ∑_s = 1^k - 1(α_s - 1t_s - 1^2 - α_st_s^2)Ω_s,

        where the last inequality follows from the fact that -t_k - 1^2Φ_k≤ 0 (since Φ_k≥ 0) and that
        
    t_s^2 - t_s - 1^2≤ t_s = s + a/a = a^2d_s - 1/a(a - 2)≤ad_s/a - 2,

        where the second equality follows from the definition of d_k since
         
    d_s = t_s - 1^2 - (t_s^2 - t_s) = a^-2((s + a - 1)^2 - (s + a)s) = a^-2((s + a)(a - 2) + 1).

        Now, we will show that there exists M > 0 such that for all k ∈ we have that
        
    a/a - 2∑_s = 1^k - 1 d_sΦ_s + α_k - 1t_k - 1^2Ω_k + ∑_s = 1^k - 1(α_s - 1t_s - 1^2 - α_st_s^2)Ω_s≤ M.

        We will prove that each of three terms above is bounded from above. First, from Lemma <ref> it follows that there exists M_1 > 0 such that ∑_s = 0^k - 1 d_sΦ_s≤ M_1. Second, using Theorem <ref>(ii) and the definition of Ω_k we obtain that
        
    α_k - 1t_k - 1^2Ω_k = α_k - 1t_k - 1^2(ω(') - ω(^k)) ≤α_k - 1t_k - 1^2C/a^2(k + 1) = (k + a - 1)^2 - γC/a^4(k + 1)≤(k + a)C/a^4(k + 1)≤C/a^3,

        where the second equality from the definitions of α_k and t_k (see (<ref>) and (<ref>), respectively), the second inequality follows from the fact that 2 - γ < 1 since γ > 1, and the last inequality from the fact that k + a < ak + a = a(k + 1) since a > 2. For the third term in (<ref>), we first split it as follows
        
    ∑_s = 1^k - 1(α_s - 1t_s - 1^2 - α_st_s^2 + a/a - 2η_s)Ω_s - a/a - 2∑_s = 1^k - 1η_sΩ_s.

        Moreover, from (<ref>) and the definitions of d_s and η_s, we obtain that
        
    α_st_s≤a/a - 2·α_sd_s = a/a - 2(α_st_s - 1^2 - α_s(t_s^2 - t_s)) ≤a/a - 2(α_s - 1t_s - 1^2 - α_s(t_s^2 - t_s)) = a/a - 2η_s,

        where the last inequality follows from the fact that α_s < α_s - 1. Using it and the definition of η_s we obtain that
        
    α_s - 1t_s - 1^2 - α_st_s^2 + a/a - 2η_s≥α_s - 1t_s - 1^2 - α_st_s^2 + α_st_s = η_s > 0,

        and
        
    α_s - 1t_s - 1^2 - α_st_s^2 + a/a - 2η_s = η_s - α_st_s + a/a - 2η_s≤η_s + a/a - 2η_s = 2a - 2/a - 2η_s,
        
        where the inequality follows from the fact that α_st_s≥ 0. Therefore, by applying Lemma <ref> with b_k≡α_k - 1t_k - 1^2 - α_kt_k^2 + aη_k/(a - 2) < cη_k with b = (2a - 2)/(a - 2) we obtain that there exists M_3 > 0 such that (recall that t_-1 = η_0 = 0 and t_0 = 1)
        
    ∑_s = 1^k - 1(α_s - 1t_s - 1^2 - α_st_s^2 + a/a - 2η_s)Ω_s = ∑_s = 0^k - 1(α_s - 1t_s - 1^2 - α_st_s^2 + a/a - 2η_s)Ω_s + α_0≤ M_3 + α_0.

        To complete the proof, we need to also bound the second term in (<ref>). Indeed, from Proposition <ref> after rearranging and using the definitions of Ω_k and F_k - 1 we have (recall that η_0 = 0)
        
    -∑_s = 1^k - 1η_sΩ_s    = -∑_s = 0^k - 1η_sΩ_s≤ t_k - 1^2(F_k - 1(') - F_k - 1(^k)) + β/2^0 - '^2
        = t_k - 1^2(φ(') - φ(^k) + α_k -1(ω(') - ω(^k))) + β/2^0 - '^2
       ≤ t_k - 1^2α_k -1Ω_k + β/2^0 - '^2
       ≤C/a^3 + β/2^0 - '^2,

        where the second inequality follows from the fact that φ(') ≤φ(^k) and last inequality follows from (<ref>). This completes the proof of the boundedness of (<ref>).


plain

	


