Paper Title: Whitney extension theorems on symmetric spaces, an example.



§ ABSTRACT  H. Whitney introduced in 1934 the problem of extending a function f on a set of points in ℝ^n to an analytic function on the ambient space. In this article we prove Whitney type extension theorems for data on some homogeneous spaces.  We use harmonic analysis on the homogeneous spaces and representation theory of compact as well as noncompact reductive groups. 


[
     R. Zamini[Corresponding author: rahelehzamini1361@yahoo.com] ^1,  S. Ghafouri^2  and F. Goodarzi^3

	^1  Department  of Mathematics, Faculty of Mathematical Sciences and  Computer, Kharazmi University, Thehran, Iran. 

	^2   Department of Mathematics, Faculty of Science, Arak University, Arak, 384817758, Iran.

^3   Department of Statistics, Faculty of Mathematical Sciences, University of Kashan, Kashan, Iran.

    
===================================================================================================================================================================================================================================================================================================================================================================================================================================






 
  

 









§ INTRODUCTION

In 1934, H. Whitney introduced the problem of extending a function f on a set of points in ^n to an analytic function on the ambient space.  In the 2000s, C. Fefferman took up Whitney's work and proved a sharp form of the Whitney extension theorem <cit.>. In particular, if the set of points in Euclidean space is finite of cardinality n, there is an extension which is continuously differentiable of some degree depending on n.

In this paper, we study a version of Whitney's extension problem where the points have an inherent symmetry.  
More precisely, we consider points x^(i) in a symmetric homogeneous  space X=G/H for reductive Lie groups G, H and a nonconstant function f on these points with values  
f(x^(i)) = y^(i) in the complex numbers ℂ.
We are especially interested not just in proving existence of an analytic function extending f, but to obtain a formula for this function which can be effectively computed by an algorithm. 
Initially, 
we weaken the assumption 

    f(x^(i))= y^(i),    1⩽ i ⩽ n.

Instead, we find for all ε > 0 a complex valued function f_n: X→ℂ so that 

    |f_n (x^(i)) - y^(i)| < ε,     1⩽ i ⩽ n.

Observe that the function f_n  depends on ε.

Furthermore, we show how to modify the proof to obtain an exact Whitney extension theorem. 
The function f_n generates a representation Π_n of G, which we call the Whitney representation of the data. 
By acting with the maximal compact subgroup K in G, 
we modify the Whitney function without changing the K-types occuring in its Whitney representation
to obtain an analytic, square integrable function, still denoted f_n,
such that
f_n(x^(i))=y^(i) for all i⩽ n.


We consider  symmetric spaces  of the form G/H for certain orthogonal  Lie groups G and H. The analysis on symmetric  spaces  was intensively studied starting with the work of M. Flensted-Jensen in the late 1970s and culminating in  the Plancherel theorem around 2005 by H. Schlichtkrull and E. van den Ban. In this article, we use  information about the representation of the symmetry group G on the square integrable functions on G/H and in particular about the irreducible subrepresentations. We also use results about the restriction of these representations to orthogonal subgroups.

The group of symmetries of the quadrics 


X(p,q)_± = { (x_i)∈^p+q | x_1^2 + ... +x_p^2 - x_p+1^2 … - x_p+q^2 = ± 1 }


is the noncompact Lie group
G = SO(p,q)^∘,  where p⩾ 2 and the superscript  denotes the connected component.  

This class of homogeneous spaces contains the spaces known in physics as 

 
de Sitter spaces (dS)


respectively as 


anti de Sitter spaces (AdS)  


<cit.>. The anti de Sitter space is considered in physics  as a model of space time with negative curvature.  



 §.§ Main results
 We prove
 
 Let p⩾ 2.
Let x^(i), 1⩽ i ⩽ n, be a set of points in  X(p,2)_- and let f(x^(i)) = y^(i) be a nonconstant, complex-valued function. 
For all ε > 0, there exists an analytic, square integrable  function f_n: X(p,2)_-→ℂ such that 

    |f_n (x^(i)) - y^(i)| < ε.

We call f_n the Whitney function of the data points. 


We approach the extension problem by realizing  space X(p,2)_- as the homogeneous space

    X(p,2)_- = SO(p,2)^∘/SO(p,1)^∘.

This allows us to obtain an explicit formula for the Whitney function f_n using the representation theory of noncompact Lie groups. 
In particular, we build on the work of M. Flensted-Jensen <cit.>, who published  in 1980 an article in which he constructs a family of square integrable functions on symmetric spaces  which are eigenfunctions of all invariant differential operators.  We refer to these functions as Flensted-Jensen functions.











    
 Let x^(i), 1⩽ i ⩽ n, be a set of points in X(p,2)_- and let f(x^(i)) = y^(i) be a nonconstant complex-valued function. 
In the notation of Theorem <ref>, the Whitney function f_n 
can be chosen as a ℂ-linear combination of Flensted-Jensen functions 

    φ_ℓ (ξ', t)= (ξ'_1 - √(-1)ξ'_2 )^ℓcosh (t)^-ℓ,

for integers ℓ⩾ℓ_0 for some lower bound ℓ_0 depending on p. 
Here, (ξ,ξ') = (ξ_1, …, ξ_p,ξ'_1,ξ'_2) is an angular variable and t a radial variable such that 

    (ξ_1sinh(t), ξ_2 sinh(t), …, ξ_psinh(t), ξ'_1cosh(t), ξ'_2cosh(t)) ∈ X(p,2)_-.



In applications of the theorem to a list of observable data, we may assume
that the results of all measurements  are rational. In the case of the three-dimensional symmetric space X(2,2)_+ = SO(2,2)^∘/SO(1,2)^∘, we  prove therefore also  a rational version of our Whitney extension theorem starting from the assumption that all the data points are rational. 


    Let x^(i)∈ X(2,2)_+(), 1⩽ i ⩽ n, be a list of -valued points in the three-dimensional symmetric space X(2,2)_+ realized as the quadric surface 
    
    X(2,2)_+ = {x∈^4: x_1^2 + x_2^2 -x_3^2 - x_4^2 = 1 }
 
    and suppose f is a nonconstant function with f(x^(i)) = y^(i)∈. Let ε > 0. 
    We can choose coefficients a_ℓ in the number field (√(-1)) such that 
    
    f_n = ∑_2 ⩽ℓ⩽ n+1 a_ℓφ_ℓ

is a Whitney function of the points: f_n is analytic, square integrable and K-finite and |f_n(x^(i)) - y^(i)| < ε for all i. Here, the Flensted-Jensen functions φ_ℓ are given by the formula

    φ_ℓ(x) = ( x_1 + √(-1) x_2 )^-ℓ,    x ∈ X(2,2)_+.





 §.§ Related work

In the 2000s, C. Fefferman proved a sharp form of the Whitney extension theorem <cit.>. In particular, if the set of points in Euclidean space is finite of cardinality n, he showed that there is an extension which is continously differentiable of some degree depending on n.  
In this paper we start with a set of points of finite cardinality n, and using  additional assumptions we prove the existence of a Whitney function which is infinitely often differentiable. 
The reason we are able to get an extension of such high regularity is that we take advantage of the fact that the points live in a subset of Euclidean space obeying a high amount of symmetries.

C. Fefferman et al. <cit.> have also studied Whitney extension problems in situations where the points live on a smooth manifold X. The Whitney extension theorems proved in that context, however, are  existence theorems. In contrast, the results given in this article  are effective.  We give an algorithm computing the extension with an explicit formula. This is possible because we build the extension from functions in the discrete spectrum of  the symmetric space G/H. Such Flensted-Jensen functions are solutions to a system of  differential equations  and hence satisfy a high amount symmetries. 

Whitney extension theorems have found a range of applications in data science <cit.>. There is a growing literature on equivariant machine learning, where the data are invariant with respect to a group of symmetries <cit.>. 
Most results in this direction, however, are only available when the symmetry group of the points is either finite or a compact Lie group <cit.>, <cit.>, <cit.>.  
However, many groups of importance in physics are noncompact, such as the Lorentz group SO(3,1)^∘, the symmetry group  SO(p,1)^∘ of X(p,1)_±, and the symmetry group SO(p,2)^∘ of X(p,2)_±.  One motivation for this work is to generalize the results in harmonic analysis applicable to Lie group equivariant machine learning to include noncompact Lie groups. 



 §.§ Contents

The paper has four parts. In this section, we summarize the contents of each part.



  §.§.§ Data on a sphere

In the first part, we prove Whitney extension theorems for points on a sphere S^2 in ^3, whose group of symmetries is the compact group SO(3).
The proof is instructive for the noncompact case, so we give it in details. 
To do so, we use  spherical harmonics, which  will be used in the later parts of the article as well. Indeed, spherical harmonics also play a role in the representation theory of non compact roups and semisimple symmetric spaces, since Flensted-Jensen functions can be written as products of  a Jacobi polynomial, which is the restriction of a harmonic polynomial, with a hyperbolic function.



  §.§.§ Data in the plane

In this part, we prove a Whitney extension theorem for functions on a set of points in the Euclidean 2-space ^2. In contrast to the results by H. Whitney and C. Fefferman, we do not have any limitations on how differentiable the extension can be: We can compute a Whitney extension which is analytic right away. Notice that in this case, we do not need the points to lie in a symmetric space; all we need is that they are contained in the complement of the unit disk, in order to make certain functions square integrable. We include these results here to illustrate our techniques.

The result we prove is as follows. 


    Let  x^(i), 1 ⩽ i ⩽ n, be a set of points in the plane ^2. Assume x^(i) is not contained in the unit disk, and consider a nonconstant complex-valued function f(x^(i)) = y^(i).
    There exists an  analytic,  square integrable function on ^2 with the closed unit disk removed which extends this function. 


We prove this theorem by observing that it falls out, automatically, of the proof of the Whitney extension theorem for the three-dimensional symmetric space X(2,2)_+, given in the next part. 

   


  §.§.§ Data on the symmetric space X(2,2)_+=SO(2,2)^∘ / SO(1,2)^∘

This part forms the core of the paper. 
We discuss the Whitney extension problem for the three-dimensional manifold X(2,2)_+=SO(2,2)^∘/SO(1,2)^∘ . We are especially interested not just in proving the existence, but also to obtain a formula for this function which can be easily computed. More precisely, we consider the question:


    Does there exist
 a smooth, square integrable function 
f ∈ C^∞(X(2,2)_+) such that
f(x^(i)) = y^(i)
for all i ⩽ n? Can we we obtain a closed formula for f?

Note that the manifold  is a fibration of Minkowski space over the circle, SO(2). In particular, we have no well-defined global time direction.
We make the following assumptions: 


  * We consider only finitely many points x^(i), 1⩽ i ⩽ n, although the number may be large.

  * We assume that x^(i)∈ℚ^4 for all i⩽ n and consider a nonconstant function f on these points with values f(x^(i))=y^(i) in . 

  * We assume that the points are in general position, in the sense that they are not contained in the orbit of a proper subgroup of the symmetry group of the ambient symmetric space (cf. <ref>).




Not only do we find a smooth, square integrable function extending the function on the points, 
but we find a constructive way to obtain a formula for this function using the representation theory of noncompact Lie groups and symmetric spaces.
We show that 

  Under these  assumptions, for all  ε >0, there exists an analytic, square integrable function f ∈ C^∞(X(2,2)_+) such that
    
    |f(x^(i)) - y^(i)| < ε

    for all i ⩽ n.


Under the above assumptions, we consider the representation of G generated by f_n in the representation space L^2(X(2,2)_+).
We show that to each Whitney function, we can associate a reducible, unitary representation, Π_n, which we call the Whitney representation of the data. 

We then consider the problem: 
If we enlarge the set of points from cardinality n to n+1, can we  choose a smooth, square integrable, K-finite function
f_n+1∈ L^2(X(2,2)_+) 
such that f_n+1(x^(i)) = y^(i) for all i 
and such that Π_n is a subrepresentation of Π_n+1?
We give an analytic proof that we can always act on the points by the maximal compact subgroup SO(2) ×SO(2)  to arrange that all irreducible subrepresentations below some fixed degree will occur in the Whitney representation. 



  §.§.§ Data on symmetric spaces X(p,2)_- =SO(p,2)^∘/SO(p,1)^∘

We show that our results extend to the symmetric spaces

    X(p,2)_-=SO(p,2)^∘/SO(p,1)^∘

for all p⩾ 2.
The essential feature is that the homogeneous space X(p,2)_- has a holomorphic structure and holomorphic discrete series represesentations.  



 §.§ Acknowledgements

These results were first presented in a lecture at the AIM virtual seminar “Representation Theory and Noncommutative Geometry”. We are grateful to Bent Ørsted for helpful conversations. We would like to thank  David Vogan and Robert Yuncken for insightful questions and comments related to this work.  

The first author was supported by  grant 89086 from the Simons Foundation.









































    


















 

























































§ LOW DIMENSIONAL EXAMPLES




 §.§ The sphere S^2
 
In this subsection we consider a set of points 

    x^(i)∈ S^2 = { x ∈^3: x_1^2 + x_1^2 + x_3^2= 1},     1⩽ i ⩽ n,

and a function f(x^(i)) = y^(i) (1⩽ i⩽ n) which we assume is not constant. 
We think of this as an observable set of data with an inherent spherical symmetry. From this perspective, it is natural to assume 

    x^(i)∈^3,    y^(i)∈
 
for all i, and we will impose this assumption throughout this section. 
Our goal is to find an explicit function interpolating the points. Since the symmetry group of a sphere is SO(3), 
it is natural to solve this problem using the representation theory of SO(3).




 §.§ Preliminaries

There is a natural action of SO(3) on S^2.  
We endow it with the surface area measure, μ, which is invariant under the action of SO(3), and normalized such that μ(S^2)=1. We consider the Hilbert space 
L^2(S^2, μ) of square integrable functions with respect to μ.
Then SO(3) acts on L^2(S^2) as 

    (g ·ψ)(x) = ψ (g^-1x),    g ∈SO(3), ψ∈ L^2(S^2),    x ∈ S^2.

Let Δ =∑_i=1^3 ∂^2/∂ x_i^2 be the Laplacian on ^3. Recall that a harmonic polynomial on ^3 
is a polynomial p = p(x_1,x_2,x_3) ∈ℂ[x_1, x_2, x_3] in three variables
with complex coefficients satisfying 
    
    Δ p = 0.

Example: Let ℓ∈_⩾ 0. The polynomial  
    
    p(x_1,x_2,x_3 ) = (x_1 + √(-1)x_2)^ℓ

is a  harmonic of degree ℓ. 
Note that p is independent of x_3 and is holomorphic as a function of the complex variable
z=x_1 + √(-1)x_2. By <cit.>, it follows that p is a harmonic function of (x_1,x_2) in the sense that 

    ( ∂^2/∂ x_1^2 + ∂^2/∂ x_2^2) p = 0.

In particular, 
Δ p = 0.
The space of spherical harmonics of degree ℓ (ℓ∈_⩾ 0) is the subspace
V_ℓ of L^2(S^2) consisting of restrictions to S^2 of harmonic polynomials on ^3 that are homogeneous of degree ℓ. 
Recall that the space V_ℓ  of spherical harmonics of degree ℓ is an irreducible representation of SO(3) of degree 2ℓ +1.
If V is any irreducible finite-dimensional representation of SO(3), then there is a unique integer ℓ such that V is isomorphic to V_ℓ. The space L^2(S^2) decomposes as a Hilbert space direct sum 
    
    L^2(S^2) ≃⊕̂_ℓ∈_⩾ 0 V_ℓ.

In the physics literature <cit.>, ℓ is called the spin of the representation V. In Lie theoretic language, the irreducible representations of SO(3) are classified by their highest weight <cit.>, and ℓ is the highest weight of V. 



 §.§ The first interpolation theorem


   




 



Consider a set of points x^(i)∈ S^2 ∩^3, 1⩽ i ⩽ n, 
and a function f on the points with values f(x^(i)) = y^(i)∈.
Suppose x^(i)≠ (0,0,1) and (x_1^(j),x_2^(j)) ≠ (x_1^(i),x_2^(i)) whenever i ≠ j.
There are a unique coefficients a_1,… a_ℓ∈(√(-1))
such that the harmonic polynomial 

    f(x_1,x_2,x_3) = ∑_ℓ = 1^n a_ℓ (x_1 + √(-1) x_2)^ℓ

satisfies
f(x^(i))=y^(i) for all i.


Let p_ℓ (x) = (x_1 + √(-1) x_2)^ℓ be the spherical harmonic of degree ℓ.
The n× n matrix (p_ℓ(x^(i)))_i, ℓ is the product of the diagonal matrix

    [ ( x_1^(1) + √(-1) x_2^(1));                                                     ⋱;                            ( x_1^(n) + √(-1) x_2^(n)) ]

with the Vandermonde matrix 

    [                              1     ( x_1^(1) + √(-1) x_2^(1)) ( x_1^(1) + √(-1) x_2^(1))^2 ⋯ ( x_1^(1) + √(-1) x_2^(1))^n-1;                              1     ( x_1^(2) + √(-1) x_2^(2)) ( x_1^(2) + √(-1) x_2^(2))^2 ⋯ ( x_1^(2) + √(-1) x_2^(2))^n-1;                              ⋮;                              1     ( x_1^(n) + √(-1) x_2^(n)) ( x_1^(n) + √(-1) x_2^(n))^2 ⋯ ( x_1^(n) + √(-1) x_2^(n))^n-1 ]

By the formula for a Vandermonde determinant (which holds over any commutative ring, cf. <cit.>),   

    det (p_ℓ(x^(i)))_i, ℓ 
    = ∏_i=1^n  ( x_1^(i) + √(-1) x_2^(i)) ∏_1 ⩽ i < j ⩽ n( ( x_1^(j) + √(-1) x_2^(j)) -  ( x_1^(i) + √(-1) x_2^(i))
    )

Since x^(i)≠ (0,0,1) and (x_1^(j),x_2^(j)) ≠ (x_1^(i),x_2^(i)) for all i ≠ j, this determinant is nonzero.
In particular, the system 

    [ y^(1);     ⋮; y^(n) ]
    =
    [ p_1(x^(1)) p_2(x^(1))          ⋯ p_n(x^(1));          ⋮; p_1(x^(n)) p_2(x^(n))          ⋯ p_n(x^(n)) ][ a_1;   ⋮; a_n ]

has a unique solution in a_ℓ∈(√(-1)).





 §.§ Harmonic polynomials on ^3


Let 𝒫_ℓ be the complex vector space of homogeneous polynomials on ^3 with complex coefficients.    
Recall that the restriction map  
    
    𝒫_ℓ⟶ L^2(S^2)

is injective <cit.>, so we may identify V_ℓ with a space of functions on ^3.
We endow ^3 with the product measure 

    μ⊗ 4π r^2 dr

of the surface area measure μ on S^2 with the measure 4π r^2 dr on (0,∞). The measure μ⊗ 4π r^2 dr is invariant with respect to the natural action of SO(3) on ^3,
and we define L^2(^3) with respect to this measure.
Then L^2(^3) is a unitary representation of SO(3).

  







 
There is a canonical isomorphism of 
    L^2(^3) with the Hilbert tensor product 
    
    L^2(^3) ≃ L^2(S^2)  ⊗̂ L^2((0,∞), 4π r^2 dr).

    Under this isomorphism, a vector f⊗ g in L^2(S^2)  ⊗̂ L^2(0,∞) corresponds to the function 
    f(θ, ϕ )g(r) in L^2(^3) written in polar coordinates. 



    Let ℓ∈_⩾ 0, and let f be a measurable function on 
(0,∞) such that f(r)r^ℓ∈ L^2(0,∞), i.e. 

    ∫_(0,∞) |f(r)|^2 r^2l 4π r^2 dr < ∞.

If p∈ V_ℓ, then p(x)f(|x|) ∈ L^2(^3).
Define 
V_ℓ, f to be the subspace of L^2(^3) of functions of the form

    φ(x) = p(x) f(||x||),    p∈ V_ℓ.

Then V_ℓ,f is invariant and irreducible under the action of SO(3).  


Note that if φ∈ V_ℓ,f, then under the isomorphism in Lemma <ref>, 

    φ(x) = p(x/||x||)||x||^ℓ f(||x||) ≃ p ⊗ g,

where g(r) = r^ℓ f(r) and
p ∈ V_ℓ.



  Let V be any irreducible representation of SO(3) in L^2(^3). Then V is of the form V_ℓ,f for some ℓ and f 
  satisfying the condition in Lemma <ref>. 


See <cit.>. 



 §.§ An open subset  ℝ^2∖B^1 of the plane

Let B^1 = {x ∈^2: x_1^2 + x_2^2 < 1 } be an open ball of radius 1, and let  B^1 = {x ∈^2: x_1^2 + x_2^2 ⩽ 1 } be its closure. 
In this subsection we consider the domain ^2 ∖B^1 and  a nonconstant function on the set of points
x^(i)∈^2 ∖B^1 for i⩽ n. Assume x^(i)∈^2 and that the function maps any
x^(i) to a rational point y^(i)∈.
We equip ^2 ∖B^1 with the measure 

    μ = dθ⊗ r dr,

which is Lebesgue measure in polar coordinates. 

For any integer ℓ⩾ 2, the functions

    φ_ℓ^±(x) = ( x_1 ±√(-1) x_2 )^-ℓ,    x∈^2∖B^1

are in L^2( ^2∖B^1 , dθ⊗ r dr).


  rCl
∫_^2∖B^1 |φ_ℓ^±(x)|^2  dθ⊗r dr
   =    ∫_-π^π∫_1^∞|r cosθ±√(-1) rsinθ|^-2ℓ dθ⊗r dr 

   =   
2π∫_1^∞r^-2 ℓ r dr

which is finite if and only if 2ℓ-1 > 1 ℓ⩾ 2.


 
For any ℓ⩾ 2,
let

    ψ_ℓ^+ (x)= (x_1 + √(-1)x_2 )^-ℓ,  ψ_ℓ^- (x)= (x_1 - √(-1)x_2 )^-ℓ,
       x = (x_1, x_2) ∈ℝ^2∖{(0,0) },

and let x^(i) (1⩽ i ⩽ n) be a sequence of points with x^(i) = (x^(i)_1, x^(i)_2) ∈^2∖{(0,0) } for all i ⩽ n.
Up to multiplication by a diagonal matrix, 
(ψ_ℓ^+(x^(i)))_i, ℓ
and (ψ_ℓ^-(x^(i)))_i, ℓ
where 1⩽ i ⩽ n and 2 ⩽ℓ⩽ n+1,
are Vandermonde matrices. 
In particular, (ψ_ℓ^+(x^(i)))_i, ℓ is invertible if and only if 
the points x^(i) = (x_1^(i), x_2^(i)) have the properties   

    (x_1^(i), x_2^(i)) ≠ (x_1^(j), x_2^(j)) ,   whenever  i < j,    i,j ⩽ n.



Define
    
    α_i = ( x_1^(i) + √(-1) x_2^(i))^-1∈(√(-1)),    1⩽ i ⩽ n,

    then 
    
    (ψ_ℓ^+(x^(i)))_i, ℓ
        =
        [ α_1^2;           ⋱;       α_n^2 ][       1     α_1 α_1^2 ⋯ α_1^n-1;       1     α_2 α_2^2 ⋯ α_2^n-1;       ⋮;       1     α_n α_n^2 ⋯ α_n^n-1 ].

By the formula for a Vandermonde determinant <cit.>, 

    det (ψ_ℓ^+(x^(i)))_i, ℓ 
    = ∏_i=1^n α_i^2 ∏_1 ⩽ i < j ⩽ n (α_j - α_i)




For any ℓ⩾ 2,
let

    ψ_ℓ^± (x)= (x_1 ±√(-1)x_2 )^-ℓ,    x = (x_1, x_2) ∈ℝ^2 ∖B^1 .

Let x^(i) = (x^(i)_1, x^(i)_2) ∈^2 ∖{ (0,0)} be a sequence of points on which a function f is defined such that 
f(x^(i))=y^(i)∈ for all i ⩽ n.
Suppose that 

    x^(i)≠ x^(j)

whenever i<j.
There exist unique coefficients a_ℓ∈(√(-1)) 
such that if we define the function 

    f_n = ∑_l=2^n+1 a_ℓψ_ℓ^+ ∈ L^2(^2∖B^1 ,
    dθ⊗ r dr
    )

then 
f_n( x^(i) ) = y^(i)
for all i ⩽  n.



By Lemma <ref>, 
the column vector (a_ℓ)_ℓ can be computed in terms of the points (x^(i),y^(i)) as  

    (a_ℓ)_ℓ = ( ψ_ℓ^+(x^(i)) )_i,ℓ^-1 (y^(i))_i.





§ THE SYMMETRIC SPACE X(2,2)_+
 




 §.§ Preliminaries


Let p, q ⩾ 1 be integers.
Let (·, ·)_p,q be the symmetric bilinear form on ^p+q 
of signature (p,q)
defined by

    (x,y)_p,q = ∑_k=1^p x_k y_k - ∑_k=p+1^p+q x_k y_k,    x,y ∈^p+q.

The generalized orthogonal group of signature (p,q) is the group

    O(p,q) = { g ∈GL(p+q,) : (gx,gy)_p,q = (x,y)_p,q ∀ x,y∈^p+q},

and SO(p,q)= { g∈O(p,q):  g = 1 }.
Note that if g∈GL(p+q, ), then g∈O(p,q) if and only if  
rCl
(ge_j, ge_k)_p,q    =    0,   j≠k, 
    
(ge_j, ge_j)_p,q    =    1,   j⩽p, 
    
(ge_j, ge_j)_p,q    =    -1,   p+1 ⩽j ⩽p+q. 
    



 Let 
    
    Q(x)=x_1^2 + x_2^2 - x_3^2 -x_4^2,    x ∈^4.

    Then X ={ x∈^4: (x) =1 } is a manifold endowed with a transitive Lie group action by G=SO(2,2), and there is a
    G-equivariant diffeomorphism
    
    X≃SO(2,2)/ SO(1,2).




There is a smooth, injective group homomorphism 
φ: SO(1,2)⟶SO(2,2) of the form 

    φ: g ↦[ 1 0; 0 g ] = 
    [    1    0    0    0;    0 g_11 g_12 g_13;    0 g_21 g_22 g_23;    0 g_31 g_32 g_33 ],    g∈SO(1,2)

since for x,y∈^4 and g∈SO(1,2),
rCl
( 
[ 1 0; 0 g ] x,
[ 1 0; 0 g ] y
)_2,2
   =    x_1 y_1 + 
( 
g 
[ x_2; x_3; x_4 ],
g 
[ y_2; y_3; y_4 ]
)_1,2 

   =   
x_1 y_1 +
( 
[ x_2; x_3; x_4 ],
[ y_2; y_3; y_4 ]
)_1,2
= (x,y)_2,2.

Let G = SO(2,2). Let e_1 = (1,0,0,0)∈^4.
The stabilizer in G at e_1 is 

    G_e_1  = {g ∈ G: g e_1 = e_1  } ={ g∈ G: 
    g = 
    [  1  0;  0 g̃ ]}  = SO(1,2).

By definition, Q(x) = (x,x)_2,2, and the G-orbit of e_1 is

    Ge_1 = { x∈^4: (x,x)_2,2 = 1} = { x∈^4: Q(x) =1 }.

 In particular, the orbit Ge_1 is a manifold, and 
there is a diffeomorphism  

    SO(2,2)/SO(1,2) ⟶ Ge_1 = { x∈^4: Q(x) =1 }.


We follow the conventions in the literature and work with the connected components, so we consider

    X(2,2)_+ = SO(2,2)^∘/SO(1,2)^∘

respectively

    X(2,2)_- =
    SO(2,2)^∘/SO(2,1)^∘.


    Let 𝔰𝔬(2,2) be the Lie algebra of SO(2,2), and let 𝔰𝔬(2,2;ℂ) be its complexification. 
    There is an isomorphism of Lie algebras
    
    φ: 𝔰𝔬(4;ℂ) ⟶𝔰𝔬(2,2;ℂ)

    In particular, if 𝔤_1 ≃𝔰𝔩(2,ℂ),
    then 
    
    𝔰𝔬(2,2;ℂ) ≃𝔤_1 ⊕𝔤_1

    and 
    
    𝔰𝔬(2,2) ≃𝔰𝔩(2;ℂ)_.

  Hence 𝔰𝔬(2,2) is simple as a real Lie algebra, while
    𝔰𝔬(2,2;ℂ) is not simple (but semisimple.)


    Let 
    
    g = [  1         ;     1      ;       -1   ;    -1 ].

Let M_4() be the space of 4× 4 real matrices, and for any matrix X, let tX be its transpose. 
We have g = g^-1 and 

    𝔰𝔬(2,2) ≃{ X∈M_2() :  g ( tX) g^-1 = -X }.

Note that g has a complex square root:

    g^1/2 = 
    [ 1      ;   1    ;     i  ;       i ]∈GL(4,ℂ).

In particular, 

    tX = -X  g ( t ( g^1/2 X g^-1/2 ))  g^-1 = -g^1/2 X g^-1/2.

It follows that there is a Lie algebra isomorphism 

    Ad_g^1/2: 𝔰𝔬(4;ℂ) ⟶𝔰𝔬(2,2;ℂ)

By <cit.>, 

    𝔰𝔬(4;ℂ) ≃𝔰𝔩(2;ℂ)
    ⊕𝔰𝔩(2;ℂ).

As in the proof of <cit.>, these two copies of 𝔰𝔩(2;ℂ) are complex conjugate. 
If 𝔤_1 denotes the first copy in the decomposition, then 
𝔰𝔬(2,2,ℂ) = 𝔤_1 ⊕𝔤_1.
The map φ: 𝔤_1 →𝔰𝔬(2,2;ℂ)

    φ: X ↦ X + X

is an ℝ-linear isomorphism and respects Lie brackets. 
We conclude that 

    𝔰𝔬(2,2) ≃𝔰𝔩(2;ℂ)_.





 §.§ From rational data to Whitney functions. The first case
 











The functions constructed by M. Flensted-Jensen <cit.>  will in our considerations play the role for the noncompact group SO(2,2)^∘ analogous to the role played by spherical harmonics for the compact group SO(3).


Let X = { x ∈ℝ^4: x_1^2 + x_2^2 -x_3^2 -x_4^2 = 1}.
For any integer ℓ≥ 2, define
ψ_ℓ^+: X →ℂ and ψ_ℓ^-: X →ℂ by

    ψ_ℓ^+(x) = ( x_1 + √(-1) x_2 )^-ℓ,   ψ_ℓ^-(x) = ( x_1 - √(-1) x_2 )^-ℓ.

Then
ψ_ℓ^± are Flensted-Jensen functions for the parameter λ=2ℓ-2, and the (𝔤, K)-modules generated by ψ_ℓ^+ and ψ_ℓ^- form a complete set of discrete series representations for X.




Cf. <cit.> Part 4, section 9, p. 215.    




We say  that the  points x^(i) (1⩽ i ⩽ n) are in general position if not all points in the sequence have the same coordinate equal to zero,  i.e.  not all points x^(i)∈ G/H have representatives in G=SO(2,2)^∘ contained in the same  subgroup G' isomorphic to one of the groups SO(1,2)^∘, SO(2,1)^∘,  SO(1,1)^∘, SO(2,0), or SO(0,2). In this section we assume that the points x^(i) are in general position.





Let x^(i) (1⩽ i ⩽ n) be a sequence of points of X=G/H in general position, and assume x^(i)∈^4 for all i.
Let f be a nonconstant function on the points with f(x^(i))=y^(i)∈. We would like to find coefficients a_ℓ such that 

    y^(i) = ∑_ℓ≥ 2 a_ℓψ_ℓ^± (x^(i))

for all i ⩽  n, and where a_ℓ≠ 0 for finitely many ℓ≥ 2.
Then 

    f = ∑_ℓ≥ 2 a_ℓψ_ℓ^±∈ L^2(X)

is a K-finite vector in the representation space L^2(X) with the property that 
f(x^(i)) = y^(i) for all i⩽ n.
We find (a_ℓ) by solving a linear system of equations 

    [ y^(1);     ⋮; y^(n) ]
    =
    [ ψ_2^±(x^(1)) ψ_3^±(x^(1))            ⋯ ψ_m^±(x^(1));            ⋮; ψ_2^±(x^(n)) ψ_3^±(x^(n))            ⋯ ψ_m^±(x^(n)) ][ a_1;   ⋮; a_m ]

for a choice of m ≥ n.

 
    The matrix  (ψ_ℓ^+(x^(i)))_i, ℓ
    where 1⩽ i ⩽ n, 2 ⩽ℓ⩽ n+1
    is Vandermonde (up to multiplication by a diagonal matrix). Furthermore,   
(ψ_ℓ^+(x^(i)))_i, ℓ is invertible if and only if 
the points x^(i) = (x_1^(i), x_2^(i), x_3^(i), x_4^(i)) in X have the property 

    (x_1^(i), x_2^(i)) ≠ (x_1^(j), x_2^(j)) ,   whenever  i < j,    i,j ⩽ n.

In this case, a_ℓ∈(√(-1)) for all 2 ⩽ℓ⩽ n+1, and
the column vector (a_ℓ)_ℓ can be computed in terms of the points (x^(i),y^(i)) as  

    (a_ℓ)_ℓ = ( ψ_ℓ^+(x^(i)) )_i,ℓ^-1 (y^
    (i))_i.




    Follows from Lemma <ref>.




 §.§ Cayley's rational parametrization of SO(n)

A matrix A is called non-exceptional if (I+A) ≠ 0, where I is the identity matrix, and otherwise A is called exceptional.     

 
    Let A be a non-exceptional matrix. Then the matrix S defined by the equation
    
    I + S = 2(I+A)^-1

    is also non-exceptional,  
    (I+A) = 2(I+S)^-1, and A and S are mutually related by the equations 
    
    S = (I-A)(I+A)^-1  = (I+A)^-1(I-A),    A = (I-S)(I+S)^-1  = (I+S)^-1(I-S).

Since the factors commute, we will follow Weyl <cit.> and use the rational notation 

    A = I-S/I+S.


For a matrix R=(r_ij), the  bilinear form ∑_ij r_ijx_i y_j, is A-invariant if and only if 

    tA R A = R,

where tA is the transpose of A.


    Let A and S be a pair of non-exceptional matrices related as in Lemma <ref>, and let R be any matrix. Then tARA = R if and only if 
    
    tS R + RS = 0.

In particular, if R=I and A belongs to the orthogonal group O(n),
then A∈SO(n) and S is skew-symmetric: tS + S = 0. 
Conversely, if S is skew-symmetric matrix, then A ∈SO(n).



    Let A ∈SO(n).
    Then A = A_1A_2 for two non-exceptional matrices
    
    A_1 = I-S_1/I+S_1,    A_2 = I-S_2/I+S_2∈SO(n).

If (I+S_1S_2) ≠ 0, then A is itself non-exceptional and     

    A= I-S/I+S

where 

    S = S_1 +S_2/I+S_1 S_1.



The following theorem implies that there are plenty of rational points in the space X. 


    Let 
    f = a_1 x_1^2 + a_2 x_2^2 + a_3 x_3^2 + a_4 x_4^2 + a_5 x_5^2,    a_i ∈
 
    be a quadratic form over  of rank 5. Then f represents zero in  if and only if it represents zero in ℝ. In particular, there are coordinates x_i ∈ such that
    
    0 =  x_1^2 + x_2^2 - x_3^2 - x_4^2  - x_5^2,

    x_5 ≠ 0, and 
 the point x' = (x_1 / x_5, x_2 /x_5, x_3/ x_5, x_4/x_5) ∈ X(2,2)_+∩^4,
 where 
 
    X(2,2)_+  = {x ∈ℝ^4 :  x_1^2 + x_2^2 - x_3^2 - x_4^2 = 1}.

 For any proper orthogonal matrix A∈SO(2,), the point

    [ A  ;   I ] 
    x' ∈ X(2,2)_+∩^4.
 
In particular, there are infinitely many rational points on X(2,2)_+.


See <cit.> p. 43.



    Let x = (x_1,x_2,x_3,x_4) and x'=(x'_1,x'_2,x'_3,x'_4) be rational points in X(2,2)_+.
    Suppose 
    
    x_1 = x_1',   x_2 = x_2'.

    For any matrix A ∈SO(2, ),
    let 
    
    [ x_1^∘; x_2^∘ ]
    =
    [ a b; c d ][ x'_1; x'_2 ]
    =
    [ ax'_1 + bx'_2; cx'_1 + dx'_2 ]

    and let x^∘_3 =x'_3 and x^∘_4 = x'_4.
    Then the perturbed point x^∘ = (x^∘_1,x^∘_2,x'_3,x'_4) is a rational point on X(2,2)_+.



    By assumption, 
    
    (x'_1)^2 + (x'_2)^2 = 1 + (x'_3)^2 + (x'_4)^2,

    so (x'_1,x'_2) lies on a circle of radius 1 + (x'_3)^2 + (x'_4)^2, and SO(2)
    acts on this circle. 


 
Let ε∈, ε > 0, and define the skew-symmetric matrix
    
    S = [   0   ε; - ε   0;     ].

    Then  
    (I + S) = 1 + ε^2 > 0, so S is non-exceptional, 
and 
    
    A = I-S/I+S = 
    [  1  ε; -ε  1;    ]^-1[  1 -ε;  ε  1;    ]
    =1/1+ε^2[ 1 - ε^2     -2ε;     2 ε 1 - ε^2;         ]

    is a non-exceptional matrix in SO(2,). 
Also, the distance between A and the identity transformation I is

    ||A-I||^2 
    = 8ε^2/1+ε^2,

which tends to 0 as ε tends to zero. 


 
    Let x and x̃ be rational points in X(2,2)_+ whose first and second coordinates coincide: x_1 = x̃_1 and x_2 = x̃_2.
    Then the point 
    
    x̃^(ε) =
    [ 1-ε^2/1+ε^2 x_1 - 2ε/1+ε^2x_2; 2ε/1+ε^2x_1 + 1-ε^2/1+ε^2 x_2;                          x̃_3;                          x̃_4 ]

    is a rational point on X(2,2)_+. Clearly, the points x̃
    and x̃^(ε) are equidistant from x, i.e.
    
    ||x - x̃ ||_ℝ^4 = ||x - x̃^(ε) ||_ℝ^4,
 while  the distance between x̃ and the perturbed point x̃^(ε)
is given by   

    ||x̃ - x̃^(ε) ||^2_ℝ^4 =
    4ε^2/(1+ε^2)^2( (x_2-ε x_1)^2 +(x_1-ε x_2)^2 ),

which tends to zero as ε tends to zero: Indeed, if ε⩽ 1, and 
if C = max_i,j |x_j^(i)|
then 
||x̃ - x̃^(ε) ||_ℝ^4⩽ 2^5/2 C ε.
Finally, 

    x_1 ≠x̃_1^(ε),    
    x_2 ≠x̃_1^(ε).





 §.§ From rational data sets  to Whitney functions: the general case
 

Consider a set of points x^(i) (1⩽ i ⩽ n) and  a nonconstant function 
f(x^(i)) = y^(i)
where x^(i)∈ X(2,2)_+, 
x^(i)_j ∈ for all j ⩽ 4, and y^(i)∈ for all i⩽ n.


  * Define a small error tolerance ε∈, ε > 0. 
    
  * Loop through indices i with 2⩽ i ⩽ n. 
    Let such an i be fixed. By induction, it holds for all j,k < i
    (j≠ k) that 
    
    (x_1^(j),x_2^(j)) ≠ (x_1^(k),x_2^(k)).

    Suppose there is some k < i such that 
      
    (x_1^(k),x_2^(k)) = (x_1^(i),x_2^(i)).

    In particular, k< i is the only index with this property.  
    Apply a non-exceptional proper orthogonal transformation to the point x^(i) 
    to obtain the point x^(i, ε) for ε∈_>0 as in Lemma <ref> with the property  that
     (x_1^(i,ε),x_2^(i,ε)) is distinct from all the points in  the finite list
     
    (x_1^(j),x_2^(j)),   j < i.

    Replace the point x^(i) with the point x^(i, ε).
    
  * After having checked all points and replaced some with small perturbations if necessary, all the assumptions of Proposition <ref> are in place. Compute the column vector 
    
    (a_ℓ)_ℓ=2^n+1
        =
      ( (x_1^(i, ε) + √(-1)x_2^(i,ε))^-ℓ)_1⩽ i ⩽ n, 2 ≤ℓ⩽ n+1^-1 (y^(i))_i=1^n,

    where a_ℓ∈(√(-1)) for all ℓ.
   Then the function 
    f_n = ∑_ℓ=2^n+1 a_ℓψ_ℓ^+
 is a linear combination of n linearly independent Flensted-Jensen functions with the property that 
   
    y_i = f_n(x^(i,ε))
 for all i ⩽ n.


Let (x^(i)) be a data set of n points in G/H in general position and let f(x^(i))=y^(i) (1⩽ i ⩽ n) be a nonconstant function. 
After fixing some ε > 0, 
we associate an analytic, square integrable, K-finite function f_n to the data according to the algorithm in this section <ref>. 
We will refer to f_n as a Whitney function of the data. 




 §.§ From Whitney functions to Whitney representations

  


Let G = SO(2,2)^∘ and H = SO(1,2)^∘
and let f_n be a Whitney function of a data set in G/H.
Let V_f_n be the (𝔤, K)-module in L^2(G/H) generated by f_n. This is the vector space of K-finite functions in L^2(G/H)
obtained by acting on f_n with the universal enveloping algebra U(𝔤_ℂ).
Let Π_n be the representation of G on the completion of V_f_n in L^2(G/H) and let (π_ℓ, V_ℓ) (ℓ⩾ 2) be the Flensted-Jensen representation of G in L^2(G/H) generated by φ_ℓ^+.
The results of M. Flensted-Jensen imply in the special case of X(2,2)_+ that if  ℓ, ℓ' ⩾ 2 are distinct indices, then 
the  representations π_ℓ and π_ℓ'  are inequivalent representations of G. 
Evidently, for a Whitney function f_n = ∑_ℓ⩽ n+1 a_ℓφ^+_ℓ,  

    Π_n = ⊕_ℓ⩽ n+1: a_ℓ≠ 0π_ℓ.

Note that if a_ℓ =0 then the representation π_ℓ is not a direct summand.


Define a hyperplane H_k in ℂ^n as

    H_k ={ z∈ℂ^n: z_k = 0},    1⩽ k⩽ n.

The union ∪_k ⩽ nH_k is a subset of ℂ^n of Lebesgue measure zero. 
Let x^(i) (1⩽ i ⩽ n) be a data set in G/H in general position, and let 
f(x^(i))=y^(i) be a nonconstant function. 
Fix ε >0 and choose modified points x̃^(i) =x^(i,ε)=k_ε. x^(i) such that the matrix 

    T = ( φ_ℓ^+(x̃^(i)))_i,ℓ∈GL(n,ℂ).

Let T^-1 be the inverse of T in GL(n,ℂ).
Suppose 

    (y^(i))_i ⩽ n∈ℂ^n ∖ T(∪_k⩽ n H_k).

Then the Whitney function f_n= ∑ a_ℓφ_ℓ^+ of the data has the property that a_ℓ≠ 0 for all ℓ.
In particular, the  representation Π_nwhich it generates  contains a maximal number of summands:  

    Π_n = ⊕_2 ⩽ℓ⩽ n+1 π_ℓ.



If the vector (y^(i))_i ∈ℂ^n lives in the set 
ℂ^n ∖ T ∪_k ⩽ n H_k
(which has full Lebesgue measure in ℂ^n) 
then
    
    (a_ℓ)_ℓ = T^-1 (y^(i))_i  ∉∪_k ⩽ n H_k

    which means that  a_ℓ≠ 0 for all ℓ.



    Let x^(i) (1⩽ i ⩽ n) be a set of points in G/H in general position, and let 
    f(x^(i)) =y^(i) be a nonconstant function. 
    Fix ε >0 and suppose there is a Whitney function 
    
    f_n = ∑_2⩽ℓ⩽  n+1 a_ℓφ_ℓ^+
 such that |f_n(x^(i))-y^(i)| < ε for all i. 
    There exists δ∈ (-1/m,1/m) for some m∈_⩾ 1 and a Whitney function  
    
    f_n^(δ) = ∑_ℓ a_ℓ^(δ)φ_ℓ^+

    such that 
    
    |f_n^(δ)(x^(i))-y^(i)| < ε  ∀ i ⩽ n,

    and such that 
    a_ℓ^(δ)≠ 0 for all ℓ⩽ n+1. In particular
     the Whitney representation Π_n^(δ) generated by f_n^(δ), is of the form 
    
    Π_n^(δ) = ⊕_2⩽ℓ⩽ n+1π_ℓ.

 

The proof relies on the observation that set of regular points

    (ℂ^n)^reg := { z ∈ℂ^n: z_i ≠ 0  ∀ i ⩽ n}

is dense in ℂ^n.
First, we verify that the function 

    W: z = (z^(i))_i ↦ (a_ℓ(z)_ℓ) ∈ℂ^n,    (z^(i))_i ∈ (G/H)^n

is an open mapping. The map φ: (G/H)^n →ℂ^n × n

    z= (z^(i))_i ↦φ(z) = ( (z_1^(i) + √(-1) z_2^(i))^-ℓ )_i,ℓ

is an analytic, open mapping, and so is the restriction  φ|_φ^-1(GL(n,ℂ)) to the open set φ^-1(GL(n,ℂ)).
Likewise, the evaluation map e_y: GL(n,ℂ) →ℂ^n defined by 
e_y: T ↦ T (y^(i))_i for T∈GL(n,ℂ)
and the inversion map ι: GL(n,ℂ) →GL(n,ℂ) are both open mappings.
By definition, 

    W = e_y ∘ι∘φ|_φ^-1(GL(n,ℂ))

so W is also an open mapping from an open subset of ℂ^n into ℂ^n.

Let 

    S_δ = [  0  δ; -δ  0;    ]

be the non-exceptional skew symmetric matrix whose Cayley transform 
A(S_δ) = (I-S_δ)/(I+S_δ) ∈SO(2).
Define 

    k_δ =
    [ I-S_δ/I+S_δ            ;                       I ]∈ K = SO(2)×SO(2).

Note that x=(x^(i))_i ∈φ^-1(GL(n,ℂ)), so there is some m ∈_⩾ 1
such that k_δ.x ∈φ^-1(GL(n,ℂ)) for all δ∈ (-1/m,1/m). 
The open set 

    U_m,ε = { W( k_δ. x ): δ∈ (-1/m,1/m), | ∑_2 ⩽ℓ⩽ n+1 a_ℓ(k_δ . x)φ_ℓ^+ (x^(i))-y^(i)| < ε}
 
is an open neighborhood of 
(a_ℓ(x))_ℓ in ℂ^n, and therefore it meets the dense set (ℂ^n)^reg.  Choose some δ such that (a_ℓ(k_δ.x))_ℓ∈  U_m,ε∩ (ℂ^n)^reg.
Then the function 

    f_n^(δ) = ∑_2 ⩽ℓ⩽ n+1 a_ℓ(k_δ . x)φ_ℓ^+

has the desired properties. 

We call the representation Π_n^(δ) the  Whitney representation of the data set.   
This answers the question in the introduction and shows


We fix ε > 0.
If we enlarge the set of points from cardinality n to n+1, we can  choose a small real number δ and a
smooth, square integrable, K-finite function
f_n+1^(δ)∈ L^2(X(2,2)_+) such that 
| f_n+1^(δ)(x^(i)) - y^(i) |< ε for all i ⩽ n+1 
and such that the representation  Π_n generated by f_n is a subrepresentation of Π_n+1 ^(δ).





 §.§ An exact Whitney theorem

In  this subsection, we obtain an exact Whitney theorem. 
Let 
    κ_t = [  cos(t)  sin(t); -sin(t)  cos(t) ].

By direct calculation, 
rCl
φ_ℓ^+(κ_t.x)    =    
( x_1cos(t)+x_2sin(t) + √(-1)(x_1(-sin(t)) +x_2cos(t)) )^-ℓ 

   =    ( x_1 (cos(t)-√(-1)sin(t)) + x_2
(sin(t) + √(-1)cos(t)) )^-ℓ 

   =    ( x_1 (cos(t)-√(-1)sin(t)) + √(-1)x_2
( cos(t) - √(-1)sin(t)) )^-ℓ 

   =    e^√(-1)(-t)(-ℓ) ( x_1+√(-1)x_2 )^-ℓ 

   =    e^√(-1)tℓ φ_ℓ^+(x).


As a consequence of the algorithm <ref>, there exists
a real number t_n and a Whitney function
f_n such that 

    f_n(κ_t_n. x^(i))= y^(i)

for all i⩽ n. Then
rCl
y^(i)    =    f_n(κ_t_n.x^(i)) 

   =    ∑_ℓ⩽n+1 a_ℓφ_ℓ^+ ( k_t_n.x^(i)) 

   =    ∑_ℓ⩽n+1 a_ℓe^√(-1)t_nℓ φ_ℓ^+(x^(i)).


In particular, 
if 
    a'_ℓ = e^√(-1)t_nℓa_ℓ,   ℓ⩽ n+1,

then the linear combination 
f_n' = ∑ a'_ℓφ_ℓ^+
of Flensted-Jensen functions is an exact Whitney function for the data. 




§ THE HIGHER DIMENSIONAL SYMMETRIC SPACES  X_±(P,2)



 
  
   

 

 

 

 

 


In this section, we sketch the generalization of  the Whitney extension theorems to the higher dimensional symmetric spaces 

    X(p,2)_-=
    SO(p,2)^∘/SO(p,1)^∘ X(p,2)_+=SO(p,2)^∘/SO(p-1,2)^∘ .
 


 §.§ Generalities

First, we clarify the definitions of certain semisimple symmetric spaces that we will study in the sequel. We refer to <cit.> for further details.
Define 

    X(p,q)_+ = { (x,y) ∈^p+q: |x|^2-|y|^2 = 1  }
 and 

    X(p,q)_- = { (x,y) ∈^p+q: |x|^2-|y|^2 = -1  }.

There is a diffeomorphism 

    X(p,q)_-≃ X(q,p)_+

given by (x,y)↦ (y,x).
The Lie group G=SO(p,q)^∘ acts transitively on X(p,q)_+ and on X(p,q)_-, and there are G-diffeomorphisms

    X(p,q)_+ ≃SO(p,q)^∘/SO(p-1,q)^∘,

and 

    X(p,q)_-≃SO(p,q)^∘/SO(p,q-1)^∘.

In the remaining part of the  paper, we will generalize our results from the space
X(2,2)_- to the spaces X(p,2)_- for all p≥ 2.
We keep q=2 because we want the  maximal compact subgroup K of the symmetry group SO(p,2)^∘ to have an abelian factor which is diffeomorphic to the circle group SO(2), so that our space has a complex structure and we can speak about holomorphic and antiholomorphic  discrete series representations. 

Let p⩾ 2.
Next, we parametrize the symmetric space 

    X(p,2)_-
    = { (x,y) ∈^p+2: |x|^2 - |y|^2 = -1 }

in terms of hyperbolic coordinates. 
The lemma below, which gives the KJH-decomposition of the symmetric space
X(p,q)_-, motivates this reparametrization. 

 
Let p,q⩾ 2 and let  
G=SO(p,q)^∘.
There is a decomposition 

    G = KJH,

where 

    H = [ g  ;   1 ],   g∈SO(p,q-1)^∘,    
    K = [ k_1    ;     k_2 ],    k_1∈SO(p), k_2∈SO(q)

and 

    J = exp[   t;  ; t   ]
    =
    [  cosh(t)           sinh(t);         ; -sinh(t)           cosh(t) ],    t∈.

In particular, G/H ≃ X(p,q)_- is a symmetric space of rank one. 


Let G=SO(p,q)^∘
and let H be the image in G of 
SO(p,q-1)^∘ under the embedding 

    g ↦[ g  ;   1 ],   g∈SO(p,q-1)^∘.

Let

    J/H ⊆ G/H

be a maximal, split, abelian subspace of G/H.
We may decompose G as  

    G = KJH,

where 

    K = [ k_1    ;     k_2 ],    k_1∈SO(p), k_2∈SO(q)

is the maximal compact subgroup of G 









 
 
 


In particular, 

    rank G/H = _ J = 1.



We introduce the coordinates 

    Φ(ξ,ξ',t) = ( ξ_1 sinh(t), ⋯, ξ_psinh(t), ξ_1'cosh(t),
    ξ_2'cosh(t) )

where ∑_j=1^pξ_j^2 = 1 and (ξ'_1)^2 + (ξ'_2)^2 =1.  



  §.§.§ Data points
 
Let G = SO(p,2)^∘ and H = SO(p,1)^∘.
Recall that we say that the points x^(i)∈ G/H  are in general position  if not all points are contained 
in an orbit G'/G'∩ H in G/H of a proper reductive subgroup G' of G isomorphic to one of the orthogonal groups  
SO(p',2)^∘,
SO(p',1)^∘, or 
SO(p',0) for some p'⩽ p.



  §.§.§ Flensted-Jensen functions


If p=2, the Flensted-Jensen functions φ_ℓ^+ on the space
rCl
X    =    X(2,2)_+ = { x∈^4: x_1^2 + x_2^2 -x_3^2 - x_4^2 = 1 } 

   ≃    X(2,2)_- = { (x_3,x_4,x_1,x_2) ∈^4: x_3^2 + x_4^2  - x_1^2 - x_2^2  = -1} 

can be expressed in the new coordinates as follows:
rCl
φ_ℓ^+ (x) = (x_1 + √(-1) x_2)^-ℓ    =    (x_1-√(-1)x_2)^ℓ 1/(x_1^2 + x_2^2)^ℓ 

   =    (ξ'_1-√(-1)ξ'_2)^ℓcosh(t)^ℓ 1/ ( (ξ'_1)^2+(ξ'_2)^2)^ℓcosh(t)^2ℓ 

   =    (ξ'_1-√(-1)ξ'_2)^ℓ cosh(t)^-ℓ,   ℓ⩾2.

We see that φ_ℓ^+ is a product of a spherical harmonic polynomial 

    p_ℓ(ξ'_1,ξ'_2) = (ξ'_1-√(-1)ξ'_2)^ℓ
 of degree ℓ,
and a negative power of a hyperbolic cosine. 
This form of the Flensted-Jensen functions generalizes to symmetric spaces X(p,q)_- = SO(p,q)^∘/SO(p,q-1)^∘ of arbitrary signature (p,q) <cit.>.



Consider the symmetric space X(p,2)_- = SO(p,2)^∘/SO(p,1)^∘.

The functions

    φ_ℓ ( ξ',t) 
    = (ξ'_1-√(-1)ξ'_2)^ℓcosh(t)^-ℓ,   ℓ⩾p+2/2, ℓ∈

are Flensted-Jensen functions for X(p,2)_-.
Any Flensted-Jensen function takes the form 

    p_ℓ(ξ'_1,ξ'_2) cosh(t)^-ℓ,    (ξ'_1)^2 + (ξ'_2)^2 = 1,

for a harmonic polynomial p_ℓ of degree ℓ.
 


 §.§ Whitney functions on X(p,2)_- = SO(p,2)^∘/SO(p,1)^∘




Let p ⩾ 2, and
let K = SO(p) ×SO(2) be the maximal compact subgroup of G = SO(p,2)^∘.
Let H = SO(p,1)^∘ and consider the symmetric space 
X(p,2)_-≃ G/H.
Let x^(i)∈ X(p,2)_-, 1 ⩽ i  ⩽ n, be points in general position
and consider a nonconstant function on these points with values y^(i)∈ℂ, i ⩽ n. 
Let ε > 0.
There exists an analytic, K-finite, square-integrable function f_n on the symmetric space X(p,2)_- such that 

    |f_n(x^(i)) - y^(i)| < ε,   ∀ i ⩽ n.

The function f_n may be chosen as a ℂ-linear combination 

    f_n (ξ', t) = ∑_ℓ a_ℓφ_ℓ (ξ', t)

of Flensted-Jensen functions φ_ℓ(ξ', t) =(ξ'_1-√(-1)ξ'_2)^ℓcosh(t)^-ℓ.



    For each x^(i)∈ G/H, let (ξ^(i),ξ'^(i),t^(i)) be such that 
    
    x^(i)= (ξ_1sinh(t^(i)), …,ξ_psinh(t^(i)) , ξ'^(i)_1cosh(t^(i)), ξ'^(i)_2cosh(t^(i)) ).

    Let ℓ_0 be the smallest integer occurring as the weight 
    of a Flensted-Jensen function φ_ℓ(ξ',t) = (ξ'_1-√(-1)ξ'_2)^ℓcosh(t)^-ℓ (note that ℓ_0 depends on p). 
    Let T be the complex n× n matrix
    
    T =  ( ( (ξ'^(i)_1-√(-1)ξ'^(i)_2 )cosh(t^(i))^-1)^ℓ)_1⩽ i ⩽ n, ℓ_0 ⩽ℓ⩽ℓ_0+n-1

and consider the linear system (a_ℓ)_ℓ_0 ⩽ℓ⩽ℓ_0+n-1 = T(y^(i))_i.    
Since T is the product of the diagonal matrix

    T = 
    [ ( (ξ'^(1)_1-√(-1)ξ'^(1)_2 )cosh(t^(i))^-1)^ℓ_0                                                                                              ;                                                                                             ⋱                                               ;                                                                                               ( (ξ'^(n)_1-√(-1)ξ'^(n)_2 )cosh(t^(i))^-1)^ℓ_0 ]

with the Vandermonde matrix, 

    ( ( (ξ'^(i)_1-√(-1)ξ'^(i)_2 )cosh(t^(i))^-1)^ℓ)_1⩽ i ⩽ n, 0 ⩽ℓ⩽ n-1

the statement follows once the points are sufficiently distinct in X(p,2)_-.
This can be arranged as in the algorithm <ref>: After specifying ε > 0, define the skew symmetric matrix 
S_ε = ( [     ε; -ε    ])
with Cayley transform k_ε = (I-S_ε)/(I+S_ε)
∈SO(2) and move the points x^(i) slightly within their 
K-orbit in G/H by acting with 

    [ I_p × p        ;             k_ε ]∈ K

where I_p × p is the identity matrix in SO(p).

Remark: Holomorphic discrete series representations 
on symmetric spaces of Hermitian type were studied in 
<cit.>, <cit.>.  
The Flensted-Jensen functions on these Hermitian spaces admit closed formulas involving consecutive powers of hyperbolic cosine, indexed by weights <cit.>, consistent with the formulas presented here.  
Consequently, our Whitney type extension theorems can be adapted (at least) to Hermitian spaces whose symmetry group is SU(1,1).




 §.§ Whitney representations
 
As in the classical cases of the sphere in section <ref> and of anti de Sitter space in section <ref> we associate a unitary representation  of the symmetry group SO(p,2)^∘ to a data set in the symmetric space SO(p,2)^∘/SO(p,1)^∘ in general position on which a nonconstant function f is defined. 

Let f_n be the Whitney function associated to this data, and 
let V_f_n be the (𝔤, K)-module  generated by f_n. It defines a unitary admissible representation which we call again the 
Whitney representation of the data set.  It is a direct sum of holomorphic discrete series representations of  SO(p,2)^∘. 

























 §.§ Data in orbits of orthogonal subgroups

Assume first that the orbits have  a holomorphic structure compatible with the holomorphic structure on X(p,,2)_-.
There is a nested sequence of orthogonal symmetric spaces 

    X(2,2)_-⊂ X(3,2)_-⊂⋯  X(p-1,2)_-⊂ X(p,2)_-.

Let p_0 be the smallest integer less than or equal to p such that all the points belong to X(p_0,2)_- and such that 
the data is in general position for the space X(p_0,2)_-.
We may then 
observe that the functions 
φ_ℓ^+
for 
    ℓ=(p+2)/2 + δ, (p+2)/2 + δ + 1,…, (p+2)/2+δ+(n-1),

where δ=1/2 if p is odd and δ =0 if p is even,
form a list of n independent Flensted-Jensen functions for all the spaces X(p',2)_- where p_0 ⩽ p'⩽ p.
In particular, there exists a Whitney function 

    f_n = ∑_(p+1)/2 +δ⩽ℓ⩽ (p+1)/2 +δ+ n-1 a_ℓφ_ℓ^+

for the data which is a Flensted-Jensen function for the ambient space X(p,2)_-: f_n is analytic and square integrable on X(p,2)_- and K-finite for K=SO(p)×SO(2) and f_n(x^(i))=y^(i) for all i.
Furthermore, f_n has the property that when restricted to X(p_0,2), 
f_n can be regarded as a Whitney function for the space X(p_0,2).

Remark: Suppose that the data are in an orbit of an orthogonal subgroup isomorphic to SO(p',1)^∘ for p' < p.
We conjecture that it is  best to proceed in two steps: 
 
     
  * In the first step, we use our algorithm to obtain a Whitney function for the proper orthogonal subspace X(p',1)_- of X(p,2)_-. 
     
  * In the second step, we use the results on holography by Kobayashi and Pevzner <cit.> to obtain an extension to X(p,2)_- and the  group SO(p,2)^∘. The restriction of the Whitney representations to orthogonal subgroups are known by the work of T. Kobayashi <cit.> and we may use this to understand the extension of a spherical Whitney function from a subgroup to the larger group. 
   


 

§ WHITNEY REPRESENTATIONS IN ELEMENTARY PARTICLE PHYSICS

 The Whitney representations of the generalized orthogonal groups introduced in <ref> may be of interest in physics <cit.>. For the sphere in the compact case, the representations are used to describe the hydrogen atom 
<cit.>. In the work of I. E. Segal our Whitney representations are used to describe electrons <cit.>; see also S. Sternberg <cit.>. The proceedings of the biannual conferences in Varna (Bulgaria) organized by V. Dobrev discuss more recent developments in applications of symmetries to physics <cit.>. 

amsplain 


 
*



 Paper Title: A Dirichlet stochastic block model for composition-weighted networks.



Leveraging PRS and PDSCH for Integrated Sensing and Communication Systems
 

This work was funded by the European Commission through the H2020 MSCA 5GSmartFact project under grant agreement number 956670. Also, this work was supported by the project 6G-DISAC under the European Union's Horizon research and innovation program with grant agreement number 101139130.

    Keivan Khosroshahi12, Philippe Sehier2, and Sami Mekki2,

1
Université Paris-Saclay, CNRS, CentraleSupélec, Laboratoire des Signaux et Systèmes, Gif-sur-Yvette, France

2 Nokia Standards, Massy, France

 keivan.khosroshahi@centralesupelec.fr, {philippe.sehier, sami.mekki}@nokia.com

    August 5, 2024
=============================================================================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 
Network data are observed in various applications where the individual entities of the system interact with or are connected to each other, and very often these interactions are defined by their associated strength or importance. Clustering is a common task in network analysis that involves finding groups of nodes which display similarities in the way they interact with the rest of the network. However, most clustering methods use the strengths of connections between entities in their original form, ignoring the possible differences in the capacities of individual nodes to send or receive edges. This often leads to clustering solutions that are heavily influenced by the nodes' capacities. One way to overcome this is to analyse the strengths of connections in relative rather than absolute terms, expressing each edge weight as a proportion of the sending (or receiving) capacity of the respective node. This, however, induces additional modelling constraints that most existing clustering methods are not designed to handle. In this work we propose a stochastic block model for composition-weighted networks based on direct modelling of compositional weight vectors using a Dirichlet mixture, with the parameters determined by the cluster labels of the sender and the receiver nodes. Inference is implemented via an extension of the classification expectation-maximisation algorithm that uses a working independence assumption, expressing the complete data likelihood of each node of the network as a function of fixed cluster labels of the remaining nodes. A model selection criterion is derived to aid the choice of the number of clusters. An alternative approach to clustering in composition-weighted networks based on a mapping to the Euclidean space is also provided. The model is validated using a number of simulation studies, assessing the effect of various initialisation strategies on the model's performance, latent structure recovery, parameter estimation quality and model selection. Furthermore, it is showcased on network data from the Erasmus exchange program and a bike sharing network for the city of London.



Keywords  Compositional data, hybrid likelihood, statistical network analysis, stochastic block model, weighted networks



Acknowledgements This work was supported by the Science Foundation Ireland funded Insight Centre for Data Analytics (Grant number 12/RC/2289_P2).



§ INTRODUCTION
 

Networks are often used to represent relationships between entities, and in the recent past there has been a rapid increase in the collection and availability of such relational data. Network data appear in a diverse range of applications, such as the social sciences <cit.>, biology <cit.>, transportation <cit.>, and finance <cit.>. Given the wide availability of network data, the statistical analysis and modelling of networks is an active area of research, driven by the unique challenges that these data present, largely due to the dependence between entities. For a concise overview of the main models and methods for network data see <cit.>.

One common task in network analysis is clustering, which involves finding groups of nodes that share similar connectivity patterns or have similar characteristics. The two main families of approaches to clustering in networks are algorithm-based and model-based <cit.>. Algorithm-based methods, as the name suggests, involve a set of operations that partition the network into clusters. These methods are often based on some notion of distance or similarity between nodes <cit.>, spectral properties of the Laplacian matrix <cit.>, or an optimisation procedure <cit.>. Model-based methods assume the presence of sub-populations in the data, corresponding to the clusters, which are then modelled by means of probability distributions. As opposed to algorithm-based methods, model-based clustering approaches have the advantage of providing a principled framework where the clustering task is framed as an inferential task for a generative probability model <cit.>.

Stochastic block modelling is a popular model-based clustering framework for network data, which models the connectivity patterns between pairs of nodes, assuming their stochastic equivalence: the presence and the strength of a connection between two nodes is determined by their cluster membership rather than by the nodes themselves. The definition of a stochastic block model as a way to describe the a priori community structure in a network was introduced in <cit.> and <cit.>, and the first attempt to use this model as a clustering tool was made in <cit.>.

The stochastic block model (SBM) was originally developed for the analysis and clustering of binary network data. For this type of data, binary interactions are often defined on a pre-specified threshold or on a subjective evaluation of what constitutes an interaction between two nodes, resulting in a potential information loss. With network data becoming more widespread and complex in the recent years, a number of extensions of the SBM to networks with different edge types and characteristics have been proposed. Particular efforts have been focused on clustering models for weighted networks. For instance, <cit.> extended the SBM to allow for finite set valued edges, such as those taking values in {-1,0,1}. Count edge attributes or multiple edges between pairs of nodes were considered in <cit.>, <cit.>, <cit.>, and <cit.>. To cluster nodes in networks with continuous edge weights, <cit.> propose a weighted SBM based on Gamma distributions. More general frameworks encompassing a wider range of edge types and attribute distributions have been proposed by <cit.>, and <cit.>. We point the reader to <cit.> for a comprehensive review of extensions of the SBM for different data types as well as inferential approaches. An approach related to weighted SBMs is the one introduced by <cit.>. In <cit.>, the authors propose a general framework for performing model-based clustering on multi-layer networks. The approach uses a multivariate Gaussian distribution to model the weights, allowing the correlation between weights connecting pairs of nodes as well as between edge weights across different layers to be accounted for. This work can be seen as a generalisation of a weighted SBM <cit.> that takes into account the correlation structure between the weights.

All of the aforementioned approaches propose models for the analysis of the weights in their original form. However, in some applications, in particular those modelling flows in the network, such as transportation or trade, the observed edge weights are influenced by the nodes' capacities to send and receive edges, which can differ significantly across the network. This often results in clustering solutions that mimic the patterns related to the capacities of the sending and receiving nodes, rather than the weights connecting them. Taking as an example the Erasmus programme network presented in Section <ref>, suppose we are interested in grouping European countries based on similarities in students' preferences for exchange destinations. With the participating countries having very different populations, we expect larger countries, such as Germany, France and the UK, to have larger student populations and so larger volumes of students participating in the programme. These countries are also able to accommodate more incoming students. By contrast, the Nordic countries, like Denmark and Finland, are much smaller in population, and hence they send and receive significantly fewer students. If we were to perform clustering using the raw counts of students going on Erasmus exchange between these countries without taking the differences in population into account, the countries would be roughly divided into groups based on their population sizes, i.e. Germany, France and the UK as one cluster, and Denmark and Finland as another, due to differences in the scale of the weights. Even if the preference profile of the UK students is closer to those of the smaller Nordic countries, the volume of students the country sends is far too large for it to be assigned to the same cluster as Finland and Denmark. Similar issue can arise in flight networks, with small airports only being able to service a limited number of flights per day as opposed to international hubs with multiple runways, or international trade networks, where the value of goods traded is linked to country's GDP <cit.>. The differences in the magnitudes of edge weights can significantly affect the results, making it more challenging to draw conclusions. 

One way to address this is by using relative rather than absolute weights in the network. In the aforementioned cases, this can be done by calculating proportions of flow alongside each edge with respect to a sending (or receiving) node, leading to edge weights of relative size. In other applications, the absolute weights can simply be unavailable for the analysis as the interactions between nodes are measured in relative terms to begin with. For example, in epidemiological networks the edge weights could represent probabilities of interaction between agents, or in infrastructure networks they could correspond to proportions of liquid flowing between units. Networks with relative edge weights can also arise as a result of privacy protection policies, such as in <cit.>, where the absolute information could potentially be used to identify individual actors, banks in this case. 

The main challenge associated with the relative nature of edge weights in the network is their interdependence. Proportional data impose a constant-sum constraint, making the standard tools used for networks with independent weights invalid. To appropriately address this constraint, tools from compositional data analysis can be employed. Compositional data describe parts of some whole, such as percentages, probabilities, and proportions, and they can be naturally observed or constructed from other types of data, such as continuous or count data, by dividing individual components by their sum. There are two main families of approaches to compositional data analysis. The first involves mapping the compositions onto the real line, allowing one to use the standard machinery on the transformed data. Log-ratio transformations are a popular choice in the literature due to their sound theoretical properties, and other alternatives, such as standardisation, are also sometimes used in practice <cit.>. The second approach is based on direct modelling of compositional observations, which often uses the Dirichlet distribution to model the compositions in their original form <cit.>. Both approaches have their relative benefits, such as the scale-invariance of the log-ratio transformations, or desirable distributional properties that come with direct modelling, and so the choice of the most suitable approach is context dependent. For the discussion of the main methods and challenges in compositional data analysis, we suggest the review paper by <cit.>. 

The literature on models and methods for networks with weights of relative nature is very scarce, with most works addressing unique application-specific challenges, often related to biological networks. Examples of such works are <cit.> and <cit.>, where the centered log-ratio transformation is applied to the compositional data to eliminate the constant-sum constraint in an attempt to recover the microbiome interaction network. Another notable work by <cit.>, which is driven by a financial market application, uses direct modelling of proportions via the Dirichlet distribution and proposes a dynamic latent variable modelling framework for the analysis of interbank networks.

To the best of our knowledge, there have been no attempts to develop a general model-based clustering framework for composition-weighted networks. In this work, we propose an extension of the stochastic block model that utilises the relative rather than absolute strength of connections, by leveraging ideas from compositional data analysis. We model the set of edge weights from each node using a Dirichlet distribution, the parameters of which are determined by the cluster labels of the sender and each of the receivers. For inference, we consider the hybrid maximum likelihood approach developed by <cit.>, based on <cit.>, using a variant of the classification expectation-maximisation (EM) algorithm <cit.>. To address model selection, an integrated completed likelihood (ICL) criterion is derived <cit.>. We also briefly introduce an alternative approach to cluster nodes in composition-weighted networks by making use of the log-ratio transformation that maps compositional data onto the real line <cit.>. The code implementing the modelling framework in R <cit.> is available on https://github.com/iuliiapromskaia/dirSBMGitHub.

The structure of this paper is as follows: we start by describing the SBM with compositional Dirichlet-distributed edge weights in Section <ref>, and outline the inferential procedure based on a classification EM algorithm in Section <ref>; Section <ref> presents an alternative method to cluster nodes in the network with compositional edge weights by using the log-ratio transformation of the compositions and then utilising a standard weighted stochastic block model; we test the model performance on synthetic data in Section <ref>, assessing different initialisation strategies, parameter estimation quality, and clustering and model selection performance, also in comparison to alternative approaches; we then illustrate the use of the proposed model in application to the Erasmus programme data from <cit.> and the London bike sharing data from <cit.> in Section <ref>; Section <ref> concludes the paper with a discussion, highlighting the work's impact and the potential for further developments.



§ DIRICHLET STOCHASTIC BLOCK MODEL
 
This section introduces the Dirichlet stochastic block model (DirSBM) for networks with compositional edge weights.

Let 𝐘 be the weighted adjacency matrix of a directed network on n nodes with no self-loops with strictly positive entries, i.e. y_ij>0 for all i ≠ j and y_ii=0 for all i. Define the compositional counterpart of 𝐘, denoted 𝐗, whose row entries 𝐱_i are given by


    𝐱_i = ( y_i1/∑_j≠ i^ny_ij,  y_i2/∑_j≠ i^ny_ij,…,y_i(i-1)/∑_j≠ i^ny_ij,0,y_i(i+1)/∑_j≠ i^ny_ij,…,y_in/∑_j≠ i^ny_ij),


i.e. x_ij>0 for i≠ j and ∑_l x_il=1  ∀ i.  Note that x_ii=0 is due to the absence of self-loops in the original weighted network. Denote with 𝐱_i^* a subvector of 𝐱_i that excludes the zero entry x_ii, and with 𝐗^* the matrix formed by row vectors 𝐱_i^*, i=1,…,n. In practice, in some applications, the weighted adjacency matrix 𝐘 may be unavailable, with the compositional matrix 𝐗 being the only piece of data provided. The Dirichlet stochastic block model with K blocks assumes the following generative process for a network with compositional edge weights 𝐗^*:


    
  * Given a K-dimensional vector of cluster membership proportions θ, generate binary cluster allocations 𝐳_i ∼ Multinom(1,θ), for i=1,…,n. The entries of these vectors, i.e. z_ik=1 when node i is assigned to cluster C_k and 0 otherwise. Denote with 𝐙 an n× K matrix of binary cluster allocations of all nodes in the network, and with 𝐙_-i the (n-1)× K-dimensional matrix of cluster allocations of all nodes in the network that are not i. 
    
  * Let 𝐀={α_kh}_k,h=1^K be a K × K matrix with strictly positive values, corresponding to the Dirichlet distribution concentration parameters. Given the cluster allocations, generate (n-1)-dimensional compositional observations as 
    
    𝐱_i^*|𝐳_i, 𝐙_-i∼ Dir(𝐳_i 𝐀𝐙^⊤_-i),

    for i=1,…,n.


To aid understanding, consider the vector 𝐜 denoting the cluster labels of the nodes in the network, such that c_i = k if i is a member of cluster C_k. The above distribution can also be written as 


    𝐱_i^* |𝐜∼ Dir(α_c_i c_1,…,α_c_i c_i-1,α_c_i c_i+1,…,α_c_i c_n).


The intuition behind this formulation is as follows: for any sender i, the distribution of weights of the edges connecting it to other vertices depends on the cluster memberships of i itself and of all of the receiver nodes j ≠ i. If the sender i and the receiver j are members of the same cluster C_k, the corresponding parameter of the Dirichlet distribution is α_kk, and this parameter value is shared for all such receivers j ∈ C_k. If instead the receiver node is in a different cluster, j ∈ C_h, the corresponding parameter is α_kh.



 §.§ Properties of DirSBM


The model statement in Section <ref> implies that the distribution of weight vectors conditional on cluster labels in the whole network is shared among the nodes assigned to the same cluster, up to a permutation of the ordering of the nodes, which is assumed to be invariant for simplicity.

To illustrate this, consider, for example, the distribution of compositional weights of node i∈ C_1 and rearrange the order of the dimensions so that they are sorted by cluster assignment index in ascending order for ease of notation. Every vector of compositional weights of node i belonging to cluster C_1, denoted 𝐱^*_i ∈ C_1, conditional on the cluster assignments of all the remaining (n-1) nodes, is distributed as


    𝐱^*_i ∈ C_1|𝐙∼ Dir(α_11,…,α_11_(n_1-1) times, α_12,…,α_12_n_2 times, …,α_1h,…,α_1h_n_h times, …,α_1K,…,α_1K_n_K times),


where n_h denotes the membership count of cluster C_h. This is because any member of cluster C_1 is connected to the remaining (n_1-1) members of its own cluster, all n_2 members of cluster C_2, and so on. 

Generalising for a generic node in cluster C_k, k = 1, …, K, we obtain:

    𝐱^*_i ∈ C_k|𝐙∼ Dir(α_k1,…,α_k1_n_k times, …,α_kk,…,α_kk_(n_k - 1) times, …,α_kh,…,α_kh_n_h times, …,α_kK,…,α_kK_n_K times).

Therefore, all the nodes belonging to the same cluster are identically distributed and share the same Dirichlet distribution, whose parameter vector is determined by the cluster allocations of the remaining nodes. This property of the DirSBM can be seen as a counterpart of stochastic equivalence in other stochastic block models <cit.>. The nodes in the network are called stochastically equivalent if they have the same probability of being connected by an edge to any other node in the network, conditional on their cluster labels. In weighted SBMs, such as the ones proposed by <cit.> and <cit.>, this leads to identical distributions of edge weights for any pair of nodes i∈ C_k and j∈ C_h as the parameters of such distributions are only determined by the cluster labels of nodes. The DirSBM cannot be said to strictly exhibit stochastic equivalence as it does not model the probability of edge existence between nodes, but it does share the benefits that arise as a consequence of stochastic equivalence as the identical distribution property of a set of edge weights conditional on cluster labels still holds.



  §.§.§ Interpretation of parameters
 

The parameters of the Dirichlet distribution in their original form are usually difficult to interpret as they are not measured on the same scale as the compositional data. To relate the parameter values to the original compositions, we can compute the expected proportion sent from any node i∈ C_k to any node j∈ C_h, denoted w_kh. In our case, as the parameter values are shared among the pairs of nodes that belong to the same cluster pair, this expected value is given by:


    w_kh = 𝔼[X^*_ij] = α_kh/(n_k-1)α_kk+∑_g ≠ kn_g α_kg     i∈ C_k,j ∈ C_h.


We can use these values to construct a K× K matrix of expected exchange proportions, denoted 𝐖, that is more intuitive to interpret than the original Dirichlet parameter matrix 𝐀. Note that, although the entries of this matrix are proportions, the rows do not sum to 1, since these proportions are shared among all pairs of nodes with the same cluster labels.

It may also be insightful to learn the expected total shares of exchanges between clusters. To calculate these, we make use of the aggregation property of the Dirichlet distribution, which states that if the parts of the Dirichlet observation are added together, the distribution remains Dirichlet, with the updated parameter values being equal to the sum of the corresponding original values <cit.>. Using Equation (<ref>), we aggregate the parts from the same cluster pair, resulting in a compositional vector 

    𝐪_i|𝐙=(∑_j≠ i 
     j∈ C_1 x_ij, …, ∑_j≠ i 
     j∈ C_k x_ij, …∑_j≠ i 
     j∈ C_K x_ij) ∼ Dir(n_1α_k1,…,(n_k-1)α_kk,…,n_Kα_kK),

for i∈ C_k. Hence, the expected cluster-to-cluster exchange shares, denoted with v_kh, are given by:


    v_kh = 𝔼[q_il]  =  
       (n_k-1)α_kk(n_k-1)α_kk+∑_g ≠ kn_g α_kg if  h=k,
    n_hα_kh(n_k-1)α_kk+∑_g ≠ kn_g α_kg if  h≠ k.



The resulting matrix 𝐕={v_kh}_k,h=1^K exhibits a unit-row-sum property.



§ INFERENCE
 

The distribution of a set of edge weights 𝐗, conditional on the cluster indicator variable is


    p(𝐗|𝐙) = ∏_i=1^n p(𝐱_i| 𝐳_i,𝐙_-i) = ∏_i=1^n∏_k=1^K [Γ (∑_j≠ i^nα_j)/∏_j≠ i^nΓ (α_j)∏_j≠ i^n x_ij^α_j-1 ] ^z_ik,  where α_j=∑_h=1^K z_jhα_kh.

This is a product of probability densities of (n-1)-dimensional Dirichlet distributions with the respective parameter vectors determined by the cluster label of the sender node i and of the remaining nodes in the network. 

As usual in model-based clustering, inference proceeds by maximizing the marginal log-likelihood of the data 𝐗 with respect to the model parameters. However, this quantity involves summing over the set of all possible cluster allocations and is computationally intractable <cit.>. A popular algorithm used in model-based clustering is the expectation-maximisation (EM) algorithm <cit.>. It is employed to find the maximum of the marginal log-likelihood of the data by introducing latent variables corresponding to cluster labels and defining the joint log-likelihood of the data and the cluster labels, the complete data log-likelihood. Then, inference for the model consists of estimation of parameters and of the latent cluster assignments. The EM algorithm works by iteratively finding the expectation of the complete data log-likelihood with respect to the posterior distribution of the latent variables (E-step) and maximising this expectation with respect to the model parameters (M-step). 

In the case of the DirSBM, the complete data log-likelihood is:


    l_c(𝐀,θ)    = log p(𝐗,𝐙)
        = ∑_i=1^n∑_k=1^K z_ik [ logΓ (∑_j≠ i^nα_j) - ∑_j≠ i^nlogΓ (α_j) + ∑_j≠ i^n (α_j-1) log x_ij ] + ∑_i=1^n∑_k=1^K z_iklogθ_k.
 

Further details are provided in Appendix <ref>.

In stochastic block modelling, the E-step is often computationally intractable due to the fact that the latent variables are not independent a posteriori. To overcome this problem, the variational EM is often employed for inference in SBMs, which is based on an approximation of the posterior distribution of the latent variables <cit.>. 



 §.§ Hybrid log-likelihood
 

In the DirSBM, a new challenge arises when trying to employ the variational EM for inference, as it is standard in SBMs. This is due to the fact that, in order to derive the evidence lower bound <cit.> one is required to compute the expectation:


    𝔼_𝐙∼ q(𝐙)[ logΓ (∑_j≠ i^n∑_h=1^K z_jhα_kh) - ∑_j≠ i^nlogΓ (∑_h=1^K z_jhα_kh) ],


with respect to q(𝐙), the mean field approximation of the posterior distribution of the latent cluster allocations. This expectation is intractable, because, unlike in standard SBM, even conditional on their cluster labels, the distribution of node i is not independent of that of node j. In fact, due to the model formulation in (<ref>), one needs to know the cluster assignments of all the receiving nodes contained in 𝐙_-i to define the distribution of 𝐱_i given 𝐳_i.

For this reason, we employ the inferential approach based on a hybrid log-likelihood proposed by <cit.>. The approach uses a working independence assumption, whereby each of the latent variables corresponding to the cluster label c_i is seen as a function of fixed values of the remaining cluster labels 𝐜_-i, allowing factorization of the likelihood over the nodes. The observed hybrid log-likelihood, using the notation from (<ref>), is given by


    l^hyb(𝐀,θ) = ∑_i=1^nlog( ∑_k=1^Kθ_k p(𝐱_i | c_i=k,𝐜_-i=𝐜_-i) ),

where the notation 𝐜_-i is employed to indicate the fixed nature of the cluster labels of the remaining (n-1) nodes in the network when considering the contribution of node i to the log-likelihood.


Introducing the set of latent variables 𝐙, we arrive at the complete data hybrid log-likelihood:


    l^hyb_c(𝐀,θ)     = ∑_i=1^n∑_k=1^K z_iklog p(𝐱_i|z_ik=1,𝐙_-i=𝐙_-i) + ∑_i=1^n∑_k=1^K z_iklogθ_k 
       
            = ∑_i=1^n∑_k=1^K z_ik [ logΓ (∑_j≠ i^nα_j) - ∑_j≠ i^nlogΓ (α_j) + ∑_j≠ i^n (α_j-1) log x_ij ] + ∑_i=1^n∑_k=1^K z_iklogθ_k,

where 𝐙 is a binary matrix with entries z_ik=1{c_i=k} and α_j=∑_h=1^Kz_jhα_kh.

With the complete data hybrid log-likelihood, a working independence assumption allows one to implement a variant of the classification EM algorithm <cit.>, where the hard partition of the nodes that are not i is used to compute the cluster assignment probability for i. The steps of the algorithm are outlined below.



E-step: compute the probability of assignment of node i to cluster k at iteration t using


    ẑ_ik^(t) = (z_ik=1|𝐱_i,𝐙_-i^(t-1)) ∝θ̂_k^(t-1)Γ(∑_j≠ i^nα_j^(t-1))∏_j≠ i^nΓ (α_j^(t-1))∏_j≠ i^n x_ij^α_j^(t-1),

with α_j^(t-1)=∑_h=1^Kz_jh^(t-1)α̂_kh^(t-1). The quantity on the right hand side is normalised to fulfill the constraint ∑_k=1^Kẑ_ik^(t)=1.



C-step: following <cit.>, a greedy approach to the C-step is adopted. That is, for each node i=1,…,n, every label swap is considered and i is assigned to the cluster label producing the highest value of the observed hybrid log-likelihood, i.e. 


    c_i^(t) = k=1,…,Kmax∑_l=1^nlog( ∑_h=1^Kθ̂_h^(t-1) p(𝐱_l | c_l=k,𝐜_-l=𝐜_-l) ),


then 𝐜^(t) and 𝐙^(t) are updated. The update of cluster labels happens sequentially for the nodes in the network rather than simultaneously, so any changes in the cluster assignments of nodes before i affect 𝐜_-i and hence the cluster label for i. 



M-step: maximise the expectation of the complete data hybrid log-likelihood


    𝔼[l_c^hyb(𝐀,θ)] =    ∑_i=1^n∑_k=1^Kẑ_ik^(t) (logΓ (∑_j≠ i^nα_j^(t-1)) - ∑_j≠ i^nlogΓ (α_j^(t-1)) + ∑_j≠ i^n (α_j^(t-1) - 1) log x_ij ) 
        + ∑_i=1^n∑_k=1^Kẑ_ik^(t)logθ̂_k^(t-1).


Closed form solution is available for the mixing proportions, so these are updated using


    θ̂_k^(t) = ∑_i=1^nẑ_ik^(t)/n.

The updates for Dirichlet parameters are not available in closed form, so the set of estimates {α̂_kh^(t)}_k,h=1^K are found numerically using the R function optim <cit.>. Since the parameters of the Dirichlet distribution are strictly positive, we use the L-BFGS-B optimisation routine, which allows for optimization with parameters having a bounded range <cit.>. 

The steps above are iterated until the following convergence criterion is met:


    | l^hyb(Â^(t),θ̂^(t))-l^hyb(Â^(t-1),θ̂^(t-1))/l^hyb(Â^(t),θ̂^(t))| < ϵ.

In this work, ϵ=10^-5 was used. More details on the algorithm can be found in Appendix <ref>.



 §.§ Identifiability of parameter ordering

The DirSBM exhibits a minor non-idenfitiability issue related to the ordering of clusters and the ordering of parameters. In particular, once we obtain the final node partition and the parameter estimates using the algorithm described in Section <ref>, the ordering of rows of the estimated Dirichlet concentration matrix 𝐀̂ and of the entries of the mixing proportions vector θ̂ is not guaranteed to match the ordering of clusters in the partition.

To aid understanding, consider an example with 


    𝐜=(1,1,2,1,2), 𝐀 = [ α_11 α_12; α_21 α_22 ] and θ=(θ_1,θ_2).


For observation 1, 𝐱_1^* ∼θ_1 Dir(α_11,α_12,α_11,α_12) + θ_2 Dir(α_21,α_22,α_21,α_22). Now, consider the same partition of nodes with the same cluster order and reorder the rows of the parameter matrix 𝐀 and the entries of θ, i.e.


    𝐜=(1,1,2,1,2), 𝐀' = [ α_21 α_22; α_11 α_12 ] = [ α_11' α_12'; α_21' α_22' ] and θ'=(θ_2,θ_1)=(θ_1',θ_2').


Then, 


    𝐱_1^*    ∼θ_1' Dir(α_11',α_12',α_11',α_12') + θ_2' Dir(α_21',α_22',α_21',α_22') 
        = θ_2 Dir(α_21,α_22,α_21,α_22) + θ_1 Dir(α_11,α_12,α_11,α_12),

which will clearly give rise to the same likelihood value as the originally ordered parameters.

To address this, we create a hard partition based on probabilities 𝐙̂ from the E-step and use the order of clusters in this hard partition as reference since column entries of 𝐙̂ are tied to the individual terms of the Dirichlet mixture. Then we use the matchClasses() function in R <cit.> to match the order of clusters in the E-step to that in the C-step, and use this match to reorder the rows of 𝐀̂ and entries of θ̂ at the convergence of the algorithm.



 §.§ Algorithm initialisation
 
As in most mixture models, extra attention should be paid to the choice of the initialisation strategy since the objective function in question is not unimodal, and the algorithm is only guaranteed to converge to a local optimum <cit.>.

In this paper, we consider the following initialisation strategies, which are compared in the simulation study in Section <ref>:


    
  * Random: each node is assigned to one of K groups randomly with equal probability. Starting with a random partition of nodes gives us an idea of the performance of the algorithm when no information on the data is used when the initial partition is created. To reduce the risk of convergence to a poor local maximum, we run the algorithm with a number of random starting partitions and select the best based on the value of the observed hybrid log-likelihood. In the simulation study in Section <ref>, we set the number of random starting partitions equal to 5 as we want to strike a balance between performance and computational costs. A larger number of random starting partitions have been tested, providing only a marginal improvement on average, but at a higher computational cost. 
    
  * K-means: the k-means algorithm <cit.> is repeatedly used to perform clustering on the data matrix 𝐗^* to reduce the risk of convergence to a local maximum, and the clustering solution of the best run is used as an initial partition. This is a widely used initialisation strategy as it is computationally inexpensive. Since the k-means algorithm itself is initialised at random, we run it 50 times in order to find the initial cluster labels as part of the simulation study.
    
  * K-means on CLR-transformed data (CLR+K-means): as our data are compositional in nature, the centered log-ratio transformation is first applied to the data matrix 𝐗^* <cit.>, which maps the compositions to the real line, then the k-means algorithm is used in the same fashion as above to find an initial partition <cit.>.
    
  * Spectral clustering: eigendecomposition of 𝐗^* is performed and used for dimensionality reduction, then the k-means algorithm is used on the reduced data to get a set of initial cluster labels <cit.>. Spectral clustering is widely used for initialisation as it can handle more complex cluster shapes than k-means itself whilst maintaining low computational costs.
    
  * Binary SBM (BinSBM): a binary version of the network is created from 𝐗 by assigning binary edges where the weighted edge exceeds a pre-specified threshold. After empirical investigation, we selected as a threshold the mean weight, as it tends to provide a reasonably well-connected binary network that retains clustering structure. Once such a binary network is created, a Bernoulli stochastic block model is fitted using the R package blockmodels <cit.> and the clustering partition is used to initialise the algorithm.
    
  * Gaussian SBM (GausSBM): a Gaussian stochastic block model is fitted directly to 𝐗 using the R package blockmodels <cit.> and the obtained clustering solution is used for initialisation.
    



 §.§ Model selection
 
The modelling approach detailed in Section <ref> is only applicable when the number of clusters is set in advance. However, in reality, in many situations the number of clusters is unknown and needs to be inferred from the data. Therefore, a model selection criterion indicating how well models with varying numbers of clusters fit the data is required in order to choose the number of groups appropriately. 

One popular choice in stochastic block modelling is the integrated completed likelihood criterion <cit.>, typically employed for selecting the number of groups in this context; see for example <cit.>, <cit.>, and <cit.>. In the case of DirSBM, the ICL is given by


    ICL(K) = l^hyb_c(Â,θ̂|𝐗,Ẑ,K) - 1/2 K^2log n(n-1) -1/2(K-1)log n,

where l^hyb_c(Â,θ̂|𝐗,Ẑ,K) is the complete data hybrid log-likelihood from Equation (<ref>), evaluated using parameter estimates at convergence of the algorithm, used in place of the complete data log-likelihood following <cit.>. We fit the DirSBM with different numbers of clusters, and compute the respective values of the criterion. The highest ICL value corresponds to the optimal number of groups.

Appendix <ref> contains derivations of the ICL that follow closely the derivations of the original ICL for stochastic block models proposed in <cit.>. The simulation study concerning model selection performance using the proposed framework is contained in Section <ref>.



§ AN ALTERNATIVE APPROACH
 

As discussed in Section <ref>, there are two main approaches to working with compositions in compositional data analysis. One involves applying a transformation to the data that maps them to the Euclidean space, followed by the use of standard tools and models for independent data. The second is based on direct modelling of compositions <cit.>. The methodology proposed in Section <ref> aims to model the compositions directly, but since both approaches have their relative benefits, we propose an alternative that falls under the transformation-based approach, assessing its performance in Section <ref> for comparative purposes.

We first map the compositions onto the real line using the centered log-ratio (CLR) transformation proposed in <cit.>:

    𝐮_i=CLR(𝐱_i)=  ( logx_i1/x̅,…,0,…,logx_in/x̅),

where x̅ =  (∏_j≠ i^n x_ij) ^1/n-1 is the geometric mean of 𝐱, then apply SBM with Gaussian weights <cit.> to the transformed data, assuming that each individual edge weight follows


    u_ij|(z_ik=1,z_jh=1) ∼ N(μ_kh,σ^2).


The centered log-ratio transformation has been chosen for its property of preserving the dimensionality of the observations. That is, it maps the Aitchison simplex sample space 𝕊^n to the standard Euclidean space ℝ^n, unlike many other transformations that map to ℝ^n-1 <cit.>. In this case, each dimension of 𝐮_i corresponds to a transformed weight of an edge connecting node i to each of the other nodes in the network, so the reduction of dimensionality would require a choice of an edge to be dropped, leading to information loss.

The approach of fitting the Gaussain SBM on CLR-transformed data is implemented using the blockmodels and compositions packages in R <cit.>, and its clustering performance is assessed alongside the DirSBM in Section <ref>. This approach is a rather strong competitor for the DirSBM proposed in Section <ref>, in particular in simpler and smaller cases, such as in networks with relatively few nodes with a small number of clusters.

There are two major drawbacks of the alternative approach in comparison to the DirSBM. First of all, the normality assumption in Equation (<ref>) may not be appropriate as the mapping of compositional vectors to the real line (whether by using CLR or any other transformation) only eliminates a unit-sum constraint and does not guarantee convenient distributional behaviour of the resulting transformed data. This means that, in order to utilise this approach, one needs to check the appropriateness of the transformation and the type of weighted SBM to fit to the transformed data. The DirSBM, on the other hand, is a ready-to-use model that takes into account the unique behaviour of the compositional data as it models them directly. The second disadvantage of the transformation-based approach is interpretability. The parameters of the distribution in Equation (<ref>) (or of any other distribution selected to model the transformed data) refer to the auxiliary Euclidean space, and it is unclear how one could use these to understand the patterns present in the original simplex space. As presented in Section <ref>, the parameters of the DirSBM can be used to compute the expected exchange proportions between nodes as well as clusters, making the interpretation of results intuitive.



§ SIMULATION STUDIES
 

In this section, we consider various simulation studies to assess the performance of our model with respect to clustering, parameter estimation, and model selection. We also investigate the effect of the different initialisation strategies listed in Section <ref> on the quality of the final inferred clustering of the nodes.

Artificial data are generated according to the DirSBM presented in Section <ref>. Different numbers of clusters, K={2,3,5}, and network sizes, n={30,50,70,100} are considered. The parameter matrices 𝐀 are defined so that the entries have varying levels of homogeneity, resulting in indirect modifications to the degree of overlap between clusters in the compositions. The term “level of homogeneity" is used to describe the extent to which the between-cluster weight parameters, i.e. {α_kh}_h ≠ k, are close to the respective within-cluster weight parameters, α_kk. When the values of the between- and the within-cluster parameters are set to be more similar, the resulting compositional weights in the network become more homogeneous, thus making separating the clusters more challenging. On the other hand, the lower the homogeneity, the more separated the clusters. Further details on the parameter values used in the simulation studies can be found in Appendix <ref>.

Throughout this section, where relevant, clustering performance is evaluated using the adjusted Rand Index <cit.>, which measures the agreement between two partitions, adjusted for random chance, by comparing the estimated clustering against the true classification of the nodes.



 §.§ Choice of initialisation strategy
 
We consider the different initialisation strategies of Section <ref> and evaluate the model performance with respect to cluster label recovery on simulated data. We generate 50 networks with n=50 nodes with 3 similarly-sized clusters and the parameter matrix 𝐀 is chosen so that the resulting within- and between-cluster edge weights are reasonably, but not excessively, distinct:

    𝐀= [ 1.0 0.7 0.5; 0.9 1.5 0.6; 0.4 0.5 1.2 ].




We initialise the algorithm using the strategies outlined in Section <ref> and compute the ARI. The results of this simulation study are shown in Figure <ref>. From the figure, random initialisation seems to work best on average, with a mean ARI of 0.805, despite being the only strategy that is not informed by the data. It also exhibits relatively low variability in comparison to all but one other strategy, which is k-means on log-ratio transformed data that is clearly inferior, with a mean ARI of only 0.513. Further investigations also reveal that, as well as having the best average ARI and low variability, random initialisation has led to the highest observed hybrid log-likelihood value in 33 out of 50 instances. Given that the number of random partitions considered was only 5, this result also shows that the proposed algorithm is capable of recovering the clustering patterns fairly successfully even when the initial cluster labels are assigned randomly and are not based on some educated guess. 

It may come as a surprise to the reader that out of all the strategies considered, random initialisation performed best, as it gives a starting cluster partition that is completely uninformed. However, there are two aspects of the data modelled by the DirSBM that are worth remembering. Firstly, the data matrix 𝐗^* introduced in Section <ref> is quite unusual in its structure in the sense that the ordering of the dimensions in the compositional edge weight vectors is not fixed and is tied to the index of the sender node. For example, the first entry of observation 𝐱^*_1, i.e. x^*_11, refers to the weight of an edge from node 1 to node 2 (as there are no self-loops and so there is no weight corresponding to an edge from node 1 to itself), whereas, for any other node i ≠ 1, x^*_i1 is the edge weight from node i to node 1. The same shift along the dimensions happens across the whole set of compositional weight vectors as we take each node in turn to be the sender. This means that distance-based algorithms, such as k-means, would not necessarily compute the distances between the matching pairs of nodes' edge weights, e.g. the first set of distances for node 1 would involve the edge weight from node 1 to node 2, and for any other node i, it would involve the edge weight from node i to node 1. This can potentially lead to finding spurious clustering patterns, producing initial partitions that result in convergence to local maxima.

The second aspect is the fact that the edges in the networks are not independent, because the weights are defined in terms of compositions. Using initialisation strategies that assume independence between the edge weights, we risk identifying false patterns in the network and getting stuck in a local maximum as a result. For instance, the Gaussian SBM initialisation could give rise to a misleading initial partition as the unit-sum constraint is ignored. With repeated random initialisation, we are less likely to get stuck in one of the local maxima as we can control the exploration of the partition space by changing the number of random starts.

Throughout the simulation studies, we use random initialisation with 5 starts, and to increase the reliability of results for the real world data in sections <ref> and <ref> we increase the number of random partitions to 20.



 §.§ Parameter estimation performance
 
In this section we examine parameter estimation quality when the number of clusters is known in advance. We consider the cases with 2, 3 and 5 clusters as well as low, medium and high level of parameter homogeneity, as described at the start of Section <ref>. The size of the networks is also varied to inspect its effect on parameter estimation quality. 

The Frobenius distance <cit.> between the true parameter matrix 𝐀 and its estimate Â is calculated for 50 synthetic data sets in each scenario, and the results are presented in Figure <ref>. As expected, larger networks lead to better parameter estimates, showing that the proposed hybrid log-likelihood classification EM leads to consistent parameter estimates as the size of the network increases. Notably, the biggest improvement occurs when we move from n=30 to n=50. This comes as no surprise since the performance of the model with regards to the recovery of latent clustering improves significantly when the number of nodes increases from 30 to 50 and above, as seen in Section <ref>. We can also note that higher homogeneity of Dirichlet parameters makes the estimation more challenging as a result of higher difficulty in separating the clusters, and the parameter estimates are closer to their true counterparts when the number of clusters is lower. This is again an expected result, as, when the number of clusters increases, recovering the underlying partition of the nodes is more challenging and each individual parameter is estimated from a smaller share of the data.






 §.§ Clustering structure recovery
 


To assess the quality of the clustering results, we compare the clustering performance of the following models using the ARI:

    
  * Binary SBM fitted on the binary version of the network (BinSBM): 

    We test the approach that is often taken in practice where a weighted network is transformed into a binary one. Following closely the description of using binary SBM as an initialisation strategy from Section <ref>, we construct a binary network and fit the Bernoulli SBM using the blockmodels package in R <cit.>. 
    
  * Naive Gaussian SBM (GausSBM):
    

    We fit SBM with Gaussian edge weights <cit.> directly to the data. This model is used to test the robustness to whether accounting for the constant-sum constraint on the sender edge weights or not in different scenarios, as similar naive approaches are sometimes implemented in applications.
    
  * Gaussian SBM on log-ratio transformed data (CLR+GausSBM): 

    The data are mapped onto the real line using a centered log-ratio transformation, then SBM with Gaussian weights is fitted to the transformed data, as described in Section <ref>.
    
  * DirSBM (DirSBM):
    

    We fit the DirSBM described in Section <ref> directly to the data.
    

Figure <ref> illustrates the performance of all 4 models on artificial data sets with 2, 3 and 5 clusters across multiple network sizes and varying levels of Dirichlet parameter homogeneity. Each boxplot is based on 50 networks. The DirSBM was initialised with 5 random cluster partitions based on the results of the simulation study in Section <ref>.

It is evident that the performance of all models improves with sample size, often approaching an ARI of 1 as we reach 100 nodes. In the case of BinSBM and GausSBM, this can be due to the fact that, in larger networks, the compositional samples are rather high-dimensional, and so the constant-sum constraint does not induce as much dependence between the individual dimensions as in smaller networks, resulting in a good fit of the models that assume independence between the weights. In the case of CLR+GausSBM and DirSBM, the improvement of performance with sample size can be associated with more accurate parameter estimation. Smaller networks present a greater challenge, due to both stronger dependence between the dimensions of compositions and the limited amount of data available for parameter estimation.

The CLR+GausSBM and DirSBM tend to outperform the competitors in terms of clustering structure recovery, and the gap in performance is significantly wider for smaller networks and higher homogeneity of Dirichlet parameters. It is also notable that the performance of BinSBM and the GausSBM on the raw compositional data often exhibits much higher variability indicated by wider boxplots.

With regards to the comparison between the DirSBM and the CLR+GausSBM, the two methods tend to perform similarly well. Looking at Figures <ref> and <ref>, we observe that in many scenarios the DirSBM performs better and with lower overall variability, making it a more a suitable model in the cases with larger number of clusters in the data. Considering Figure <ref>, where the number of underlying clusters is 2, we note that CLR+GausSBM tends to perform better than DirSBM. Albeit a simplistic approach, CLR+GausSBM seems to present a valid alternative when modelling compositional networks if it is expected that the number of clusters is small.



 §.§ Model selection performance
 
We assess the model selection performance using the integrated completed likelihood (ICL) detailed in Section <ref>. Table <ref> reports the confusion matrices of the true number of clusters K and the optimal number of clusters K̂ as selected by ICL.






We observe that ICL is successful in selecting the correct number of clusters in larger networks with any level of parameter homogeneity, which is to be expected provided that both the latent structure recovery and the parameter estimation improve with network size. It is also not surprising that, in smaller networks with higher levels of homogeneity and/or higher numbers of clusters, the model selection becomes a much more complex task, and the number of clusters tends to be underestimated by the ICL. This is due to the fact that in cases with medium and high levels of homogeneity the clusters are much harder to separate, coupled with the difficulty of accurately estimating the parameters in smaller networks, as seen in Sections <ref> and <ref> respectively.



§ APPLICATION TO REAL WORLD NETWORK DATA




 §.§ Erasmus programme data
 
The Erasmus programme is the most popular university student exchange programme in Europe, spanning thousands of Higher Education institutions in more than 30 countries. <cit.> have produced an extensive data set of student, staff and teacher mobility in the Erasmus network by combining data from a number of different sources.

The aim of our analysis is to understand global mobility patterns in the Erasmus network, focusing on student exchanges. Due to significant differences in the student populations of the European countries, analysis of raw counts of students per pair of countries may not provide particular insights regarding the popularity of destinations, as the biggest countries dominate simply due to their overall hosting capacity. 

After aggregating the student numbers to country level for the 2012-2013 academic year, a directed network with countries as nodes and proportions of students as edge weights is constructed, with the proportions summing to 1 for the sending countries. The Erasmus network is very dense (≈ 87% of all possible edges are present), and there is no reason to believe that the remaining edges are not allowed to exist since all countries in the Erasmus network are allowed to exchange students. Therefore, instead of treating the edges as truly missing, we assign a value of 0.001 to the interactions with zero count and treat the network as fully connected. In compositional data analysis, zero values present a great challenge and are an active area of research since standard tools are not designed to handle zero-valued compositional parts <cit.>. Replacing zeros by some small value is a standard approach in the case of zero counts that works generally well, especially considering its simplicity. Since in our case 0.001 is added to the original count data rather than to the compositions themselves, the final results should be robust relative to the magnitude of the added constant.

We implement DirSBM with K ranging from 1 to 6 and use the ICL to select the number of clusters. 
According to the ICL, the optimal number of clusters is 3 (ICL=3764), followed by a solution with 5 clusters (ICL=3718). As the Erasmus network is fairly small (33 nodes), the ICL might underestimate the number of clusters due to the penalty being too strong in the context of small n, as we saw in Section <ref>, therefore we examine the results of both K=3 and K=5 cases.

Figure <ref> illustrates the solution with 3 clusters on a map of Europe, noting the membership of each cluster on the side as smaller countries are difficult to see on the plot. Figure <ref> represents the total percentage flows between the clusters, based on the estimated matrix 𝐕̂. The estimate of the parameter matrix 𝐀 as well as the expected node-to-node and cluster-to-cluster exchange matrices Ŵ and 𝐕̂ from Equations (<ref>) and (<ref>) respectively are provided in Appendix <ref>. 



The clustered compositions are reasonably interpretable in a real-world context. The red group seems to dominate the student preference list across all clusters as indicated by high country-wise exchange percentages (12.6, 8.7 and 10 respectively) from the first column of Ŵ, and it consists of Germany, Spain, France, Italy and the UK. The red cluster collectively also notably receives the highest total percentages. The blue cluster can be described as containing countries that are still very popular exchange destinations, but not as popular as the likes of Germany and France, possibly due to their relatively small size or more challenging national language. Although roughly equal percentages of students are expected to be retained by the blue cluster and to be sent to the countries in the red cluster, 41.4% and 43.7% respectively, these shares are split between 11 countries instead of 5, hence each individual country in the blue cluster is expected to receive smaller shares in comparison to countries in the red cluster; for instance, any red cluster country is expected to receive 10% of students from any green cluster country, whereas for blue cluster countries this share is only 3%. The green cluster contains the less popular exchange destinations, such as Bulgaria and Slovakia, as indicated by significantly lower expected country-wise and cluster-wise exchange shares. Collectively, the green cluster also retains the least of students, as only 16.8% go on exchange within the cluster itself, while the remaining 83.2% go on exchange to the countries in the red and blue clusters. It can also be noted that the countries in the red cluster are generally closely co-located in geographical terms, as are most countries in the blue cluster, except for Portugal and Turkey, and the countries in the green cluster are more spread out.




The second best solution according to ICL with K=5 is presented in Figure <ref>, and the parameter estimates as well as matrices 𝐕̂ and Ŵ are available in Appendix <ref>.

The red cluster, now consisting of only Germany, Spain and France, is the cluster of countries receiving the highest percentages of students from the countries of all clusters, ranging between the expected 8.1% to 19.6% per country pair, and the second largest total shares, dominated by the blue cluster by a small margin. It should be noted that the total shares received by the red cluster are only split between 3 countries instead of 11 like in the case of the blue cluster. This cluster is also characterised by retaining 39.1% of its students, and sending 43.3% to the blue cluster, implying a lack of diversity of exchange destinations among its students. The blue cluster can be described as containing reasonably popular exchange destinations, but not quite as popular as the red countries, and, similarly to the red cluster, the majority of students go on exchange within the cluster itself (42.2%) or to the red cluster (38.1%). The green cluster, despite consisting of 9 countries (8 Eastern European countries and Cyprus), does not attract large shares of students (total cluster exchange shares are between 2.4% and 10.6%), which may be due to the fact that such countries are less well known exchange destinations. Green countries have a somewhat more diverse preference profile in comparison to the first two clusters, as the parameter estimates are much closer to each other, but there is still a strong preference for the red and blue cluster countries. Purple countries, which are Switzerland, Hungary, Ireland, Norway and Turkey, can be described as medium-preference exchange destinations. They have a strong preference for the red and blue cluster countries, and very limited interest in countries within its own cluster, as well as in green and orange clusters. The orange cluster is the one having the most homogeneous preferences, as indicated by the smallest differences between the parameter estimates, and its countries tend to be among the least popular destinations for exchange. This can possibly be explained by the fact that these countries are either very small, are located very far from the centre of Europe or are not members of the European Union (as of the 2012-2013 academic year). The green, purple and orange clusters retain very small shares or transfers, 14.2%, 2.4% and 5.6% respectively, while the majority of students from these clusters tend to go on exchange to countries in the red and blue clusters. A final observation is that there seems to be a connection between the exchange preferences of Erasmus students and the level of economic development of the hosting country or the prestige of going on exchange to specific destinations. The countries with stronger economies tend to be grouped together and to exhibit higher expected country-to-country exchange proportions, whereas countries with weaker economies or newer members of the programme, such as Turkey and Croatia, tend to be separated from the more established members.



 §.§ London bike sharing data
 

We illustrate DirSBM in application to the London bike sharing data from 2014 provided by the <cit.>. We start by counting the number of trips taken between pairs of stations and identifying the busiest stations for our analysis. To do so, we look at the top 100 start and end stations in terms of volume of bikes exchanged and consider their intersection, i.e. the stations that are both among the most popular start and end points of the journey, resulting in an overall network of 85 stations. The trips only between these 85 stations account for over 10% of the total year's trips. The network is very dense, with just under 2% of edges having zero weights, hence, similarly to the Erasmus programme network from Section <ref>, we set the weights of zero-weighted edges equal to 0.001. We then compute the compositional edge weights by dividing the count weights by the total number of trips taken from the start station. 

We fit the DirSBMs with number of clusters between 1 and 8 clusters and select the optimal number of clusters based on the ICL. According to the criterion, the solution with K=4 is optimal (ICL=27866), followed by the solution with K=3 (ICL=27608). Figure <ref> illustrates the solution with 4 clusters on a map of central London (an interactive version of the map is available https://www.google.com/maps/d/edit?mid=1Jjv8zwCHdPPvxdK6gBFqmf3vLng04mg   usp=sharinghere) as well as the chord diagram for total exchange proportions between clusters. The Dirichlet parameter matrix estimate Â as well as matrices 𝐕̂ and Ŵ are provided in Appendix <ref>.

Unlike the Erasmus network data presented in Section <ref>, the clustering structure is largely driven by stronger connections within clusters than those between clusters as the diagonal elements of Â tend to be dominant. The clusters are also much more balanced in terms of number of observations, with 24, 15, 26 and 20 stations in the green, blue, red and purple cluster, respectively. There is a geographical component to the clustering structure recovered, in the sense that the clusters are quite compact in space, indicating that more trips take place within the neighbouring areas of the city, and the neighboring clusters tend to exchange larger shares of trips in comparison to more distant ones.



The blue cluster is centered around Hyde Park and Kensington, which has the least geographical spread, and it retains the majority of its flow within the cluster (62.3% of trips), possibly due to the purpose of the journey being recreational cycling. A portion of 23.2% of trips departing from the blue cluster terminate in the neighboring red cluster, and significantly smaller shares arrive to the purple (8.6%) or the green clusters (5.9%).
The red cluster contains bike stations around Soho and Covent Garden regions of London that are known for entertainment and shopping, as well as stations near major tourist attractions (such as the British Museum) or large train stations (e.g. Euston station and King's Cross station). A significant 46.4% of trips take place within the cluster, and the green cluster receives the least trips starting from the red cluster, possibly due to the geographical distance between the two. The green and the purple clusters have strong connections with each other, exchanging 41.1% and 23.2% of their trips (in comparison to retaining 36.5% and 34.9% respectively), which can be due to the concentration of corporate buildings in this region of London and the employees living within cycling distance of their workplaces. Neither cluster is well connected to the blue cluster, likely due to the distance and availability of faster public transport options as well as the purpose of travel. A proportion of 34.3% of trips from stations in a purple cluster arrive in the red cluster, whilst 21.6% travel in the opposite direction. In summary, the clusters detected by DirSBM by modelling the proportions of transfers between the stations are interpretable in terms of geographical locations and neighbourhoods of London.



§ DISCUSSION
 

In this paper, we have introduced DirSBM, an extension of the stochastic block model to directed weighted networks with compositional edge weights that is based on direct modelling of compositional weights vectors using the Dirichlet distribution. To the best of our knowledge, no counterparts currently exist in the literature, making the proposed clustering methodology the first of its kind. We have developed an inferential procedure based on a variant of the classification expectation-maximisation algorithm for hybrid likelihood. Model selection is addressed using the integrated completed likelihood criterion. We have also proposed an alternative framework for clustering for composition-weighted networks via a log-ratio transformation of the data and subsequent application of the existing weighted SBM to the transformed data. In the simulation studies, we have shown the effectiveness of DirSBM in terms of parameter estimation and recovery of the clustering structure in a variety of settings, and have explored the use of ICL for model selection, which has proven to work reasonably well. The code implementing the modelling framework in R can be found on https://github.com/iuliiapromskaia/dirSBMGitHub.

The modelling approach can be extended in multiple directions. One important extension to the DirSBM is introducing structural zeros, to enable the ability to model the absence of guaranteed edges in the network. One could use the idea from <cit.> that concerns Dirichlet regression with zeros, where indicators are used to “discard" zero entries of compositional vectors and subsequently fit the model to the non-zero subset of compositional data. As well as making the model more realistic and applicable to a wider range of real world data, such an extension could lead to numerically more stable performance, since the dimension of the Dirichlet observations would not scale with the number of nodes. 

The use of the centered log-ratio transformation approach presented in Section <ref> has been shown to be a simple valid alternative to the DirSBM in simple scenarios with few clusters in small networks. With regards to this approach, transformations other than the centered log-ratio could be explored as well as other weighted SBMs. As for the incorporation of structural zeros, it is unclear how one could include such information in this methodology as the edges are treated as independent and the respective parameters of the distribution are estimated in the Euclidean space <cit.>. This implies that even if two nodes have similar weights in a compositional sense, the absence of edges for one of them can lead to assignment to different clusters, because the absent edges would make them appear more different than they actually are when mapped to the Euclidean space. To give a simple illustrative example, suppose we have two compositions, x_1=(0.2,0.5,0.28,0.01,0.01) and x_2=(0.2,0.5,0.3,0,0), and let the zero values be structural zeros, i.e. indicating a missing edge in the network. Then, applying the centered log-ratio transformation, u_1=clr(x_1)=(0.95,1.86,1.28,-2.05,-2.05) and u_2=clr(x_2)=(-0.44,0.48,-0.035,0,0) (using the convention log 0=0), a false impression is created that the compositions are quite different. 

Other developments could include investigating more educated and efficient initialisation strategies for the proposed algorithm. Although random initialisation allows to explore the solution space rather well by considering the most diverse set of starting points, thus reducing the risk of convergence to a local maximum, it does so at a computational cost. This cost could potentially be reduced if the algorithm was to be run from fewer, or ideally a single, more informed initial clustering allocation set. Other distributions for compositional data could be considered to model the sets of proportional edge weights in the network, such as the generalised Dirichlet distribution, which would be less restrictive on the compositional variance <cit.>.




§ COMPLETE DATA LOG-LIKELIHOOD
 

Recall that given the binary matrix of cluster allocations 𝐙, the compositional data 𝐗 has the following probability density:

    p(𝐗| 𝐙) = ∏_i=1^n p(𝐱_i| 𝐳_i,𝐙_-i) =  ∏_i=1^n∏_k=1^K [Γ (∑_j≠ i^nα_j)/∏_j≠ i^nΓ (α_j)∏_j≠ i^n x_ij^α_j-1 ] ^z_ik, where α_j=∑_h=1^K z_jhα_kh.

The latent variables have probability distribution:


    p(𝐳_i)=∏_k=1^Kθ_k^z_ik.

This results in the following complete data likelihood:


    ℒ_c(𝐀,θ) =  ∏_i=1^n∏_k=1^K
    p(𝐱_i,𝐳_i,𝐙_-i) =∏_i=1^n∏_k=1^K [ θ_k Γ (∑_j≠ i^nα_j)/∏_j≠ i^nΓ (α_j)∏_j≠ i^n x_ij^α_j-1 ] ^z_ik.

Taking the natural logarithm of Equation (<ref>), we arrive at the expression for the complete data log-likelihood from Equation (<ref>).



§ CLASSIFICATION EM WITH HYBRID LOG-LIKELIHOOD
 

The expectation of the complete data hybrid log-likelihood with respect to a single latent variable 𝐳_i is given by:


    𝔼 [l_c^hyb(𝐀,θ)] 
        = ∑_i=1^n∑_k=1^K𝔼[z_ik]  (logΓ (∑_j≠ i^nα_j) - ∑_j≠ i^nlogΓ (α_j) + ∑_j≠ i^n (α_j - 1) log x_ij ) + ∑_i=1^n∑_k=1^K𝔼[z_ik] logθ_k,

with α_j =∑_h=1^Kz_jhα_kh, where z_jh is the fixed cluster indicator z_jk = 1{c_j=k}.

The E-step can be found in standard EM fashion by using the Bayes rule:


    ẑ_ik = (z_ik=1|𝐱_i,𝐙_-i)   =p(z_ik=1|𝐙_-i)p(𝐱_i|z_ik=1,𝐙_-i)p(𝐱_i|𝐙_-i)
         =p(z_ik=1)p(𝐱_i|z_ik=1,𝐙_-i)∑_hp(z_ih=1)p(𝐱_i|z_ih=1,𝐙_-i)
       ∝ p(z_ik=1)p(𝐱_i|z_ik=1,𝐙_-i) 
        =
        θ_k ∏_j≠ i^n x_ij^α_jΓ(∑_j≠ i^nα_j)∏_j≠ i^nΓ (α_j),


as p(z_ik=1|𝐙_-i)=p(z_ik=1) due to the working independence assumption.

The M-step involves finding the estimates for the mixing proportions θ=(θ_1,...,θ_K) and the Dirichlet connectivity matrix 𝐀. The closed form solution for the mixing proportions is derived similarly to that in the standard EM algorithm by maximising 𝔼[l_c^hyb(𝐀,θ)], subject to constraint ∑_kθ_k = 1. Let


    f = 𝔼[l_c^hyb(𝐀,θ)] - λ (∑_k=1^Kθ_k - 1).


The partial derivative of f with respect to θ_k is


    1θ_k∑_i=1^nẑ_ik -λ = 0,


which gives λθ_k = ∑_i=1^nẑ_ik. Summing over k and using the unit-sum constraint for (θ_1,...,θ_K), we find that λ=n, producing the update for the mixing proportions from Equation (<ref>). 

The updates for the Dirichlet concentration parameter matrix 𝐀 are not available in closed form and so are found numerically using the R function optim <cit.>. L-BFGS-B optimisation procedure <cit.> is used to update the set of parameters {α_kh}_k,h=1^K as it allows a permitted range of values to be set; in the case of DirSBM, we are only interested in strictly positive solutions. 



§ INTEGRATED COMPLETED LIKELIHOOD
 

Following <cit.>, the exact complete-data integrated log-likelihood is given by


    log p(𝐗,𝐙|K) = log p(𝐗|𝐙,K) + log p(𝐙|K).


Following closely the derivations of the ICL for random graphs in <cit.>, we can find the first term of Equation (<ref>) using a BIC approximation:


    log p(𝐗|𝐙,K) = max_𝐀log p(𝐗|𝐙,𝐀,K) -1/2 K^2 log n(n-1),

where K^2 is the number of parameters in the model (elements of non-symmetric 𝐀 matrix), and n(n-1) is the number of edge weights in the network (with no self-loops).

The second term of Equation (<ref>) is derived in the same fashion as in <cit.>, using a non-informative Jeffrey prior and Stirling formula for an approximation of a Gamma function, leading to


    log p(𝐙|K) = max_θlog p(𝐙|θ,K) - 1/2 (K-1) log n,

with (K-1) being the number of free parameters in θ and n being the number of latent variables in the model.

Substituting the approximation results back and then replacing the complete data log-likelihood with its hybrid counterpart in the fashion of <cit.>, we arrive at 


    ICL(K)     = log p(𝐗|Ẑ,Â,K) - 1/2 K^2log n(n-1) + log p (Ẑ|θ̂,K) -1/2(K-1)log n 
        = log p(𝐗,Ẑ|Â,θ̂,K) - 1/2 K^2log n(n-1) -1/2(K-1)log n 
        = l^hyb_c(Â,θ̂|𝐗,Ẑ,K) - 1/2 K^2log n(n-1) -1/2(K-1)log n.




§ SIMULATION STUDIES: ADDITIONAL NOTES
 

Parameters used to generate data sets with different levels of parameter homogeneity, low, medium and high, denoted with subscripts L, M and H, respectively:

K=2

    𝐀_L= [ 1.0 0.6; 0.8 1.5 ], 
        𝐀_M= [ 1.0 0.6; 0.9 1.4 ], 
        𝐀_H= [ 1.0 0.8; 0.9 1.5 ];


K=3

    𝐀_L= [ 1.0 0.6 0.2; 0.6 1.5 0.5; 0.3 0.4 1.2 ], 𝐀_M= [ 1.0 0.7 0.5; 0.9 1.5 0.6; 0.4 0.5 1.2 ],     𝐀_H= [ 1.0 0.7 0.5; 0.9 1.3 0.7; 0.6 0.5 1.2 ];


K=5

    𝐀_L= [ 1.0 0.6 0.2 0.3 0.5; 0.6 1.5 0.5 0.4 0.7; 0.3 0.4 1.2 0.5 0.2; 0.7 0.5 0.3 1.4 0.4; 0.5 0.7 0.8 0.6 1.7 ],
        𝐀_M= [ 1.0 0.7 0.5 0.4 0.6; 0.9 1.5 0.6 0.5 0.7; 0.4 0.5 1.2 0.6 0.3; 0.8 0.6 0.4 1.4 0.5; 0.5 0.8 0.9 0.7 1.7 ],



    𝐀_H= [ 1.0 0.7 0.5 0.4 0.6; 0.9 1.3 0.7 0.5 0.8; 0.6 0.7 1.2 0.8 0.5; 0.8 0.6 0.4 1.4 0.7; 0.7 0.8 0.9 0.6 1.6 ].


The parameter values are constrained to be positive, but some care is needed when the parameter matrices for synthetic data sets are chosen if the intention is to use such data for model testing. Due to specification of the model based on the Dirichlet distribution whose dimensions grow with the size of the network, they cannot be too large, especially in the case of large number of nodes. The expression for the complete data hybrid log-likelihood involves a Gamma function term evaluated at the sum of all parameter values, and large parameters cause it to become too large to compute. The parameters also cannot be too close to 0 as we are required to evaluate the Gamma function at each individual parameter value.



§ ERASMUS PROGRAMME DATA: PARAMETER ESTIMATES
 


 §.§ K=3


The estimate of the Dirichlet parameter matrix 𝐀 of the 3-cluster DirSBM on the Erasmus programme data is


    Â=
        [ 7.019 1.893 0.410; 5.325 2.521 0.535; 1.347 0.410 0.142;       ].


The expected total cluster-to-cluster exchange share and the expected node-to-node exchange share matrices are (multiplied by 100 for convenience): 


    𝐕̂=
        [ 50.3 37.3 12.5; 43.7 41.4 14.9; 49.8 33.4 16.8 ],
        Ŵ=
        [ 12.6  3.4  0.7;  8.7  4.1  0.9; 10.0  3.0  1.1;      ].


The rows of these matrices correspond to the sending clusters, and the receiving clusters are along the columns. To give an example in the context of the Erasmus programme network, the entry v̂_12 = 37.3 indicates that 37.3% of students of cluster 1 collectively (which contains Germany, Spain, France, Italy and the UK) go on exchange to cluster 2 collectively (which includes countries like Austria and Belgium). The entry ŵ_31=10.0 can be read as 10.0% of Irish students (cluster 3) are expected to go to Germany (cluster 1), and the same percentage is expected to go from Norway (cluster 3) to Spain (cluster 1).



 §.§ K=5


Similarly, for the K=5 solution, 


    Â=
        [ 11.004  2.214  0.323  1.189  0.213; 12.197  4.057  0.847  1.984  0.282;  5.539  3.083  1.174  0.959  0.301;  3.035  1.298  0.231  0.158  0.105;  0.630  0.323  0.124  0.155  0.108;        ].


and


    𝐕̂=
        [ 39.1 43.3  5.2 10.6  1.9; 38.1 42.2  7.9 10.3  1.5; 25.1 51.2 14.2  7.2  2.3; 34.2 53.6  7.8  2.4  2.0; 24.3 45.8 14.4 10.0  5.6;      ],
        Ŵ=
        [ 19.6  3.9  0.6  2.1  0.4; 12.7  4.2  0.9  2.1  0.3;  8.4  4.7  1.8  1.4  0.5; 11.4  4.9  0.9  0.6  0.4;  8.1  4.2  1.6  2.0  1.4;      ].




§ BIKE SHARING DATA: PARAMETER ESTIMATES
 

Dirichlet parameter matrix 𝐀 estimate of the 4-cluster DirSBM on the bike sharing data is


    Â=
        [ 1.58 0.25 0.71 2.04; 0.22 3.96 0.79 0.38; 0.61 1.51 2.16 1.26; 1.23 0.64 1.67 2.33;      ],


and the expected cluster-to-cluster exchange shares and station-to-station exchange shares matrices are


    𝐕̂=
        [ 36.5  3.7 18.7 41.1;  5.9 62.3 23.2  8.6; 12.6 19.4 46.4 21.6; 23.2  7.6 34.3 34.9;      ],
        Ŵ=
        [ 1.6 0.2 0.7 2.1; 0.2 4.5 0.9 0.4; 0.5 1.3 1.9 1.1; 1.0 0.5 1.3 1.8;     ].








 Paper Title: Two Completely Parameter-Free Alternating Gradient Projection Algorithms for Nonconvex-(strongly) Concave Minimax Problems.









[1]This work is supported by National Natural Science Foundation of China under the grants 12071279. 

[2]Corresponding author.

My editor

Two Completely Parameter-Free Alternating Gradient Projection Algorithms for Nonconvex-(strongly) Concave Minimax Problems[1]
    Junnan Yang yangjunnan@shu.edu.cn 

	Department of Mathematics

	Shanghai University

	Shanghai 200444,People's Republic of China
	Huiling Zhang zhanghl1209@shu.edu.cn 

	Department of Mathematics

	Shanghai University

	Shanghai 200444,People's Republic of China
    Zi Xu[2] xuzi@shu.edu.cn 

	Department of Mathematics

	Shanghai University

	Shanghai 200444, People's Republic of China

	and

	Newtouch Center for Mathematics of Shanghai University

	Shanghai University

	Shanghai 200444, People's Republic of China
    August 5, 2024
============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Due to their importance in various emerging applications, efficient algorithms for solving minimax problems have recently received increasing attention. However, many existing algorithms require prior knowledge of the problem parameters in order to achieve optimal iteration complexity. In this paper, we propose a completely parameter-free alternating gradient projection (PF-AGP) algorithm to solve the smooth nonconvex-(strongly) concave minimax problems using a backtracking strategy, which does not require prior knowledge of parameters such as the Lipschtiz constant L or the strongly concave constant μ. The PF-AGP algorithm utilizes a parameter-free gradient projection step to alternately update the outer and inner variables in each iteration. We show that the total number of  gradient calls of the PF-AGP algorithm to obtain an ε-stationary point for nonconvex-strongly concave minimax problems is upper bounded by 𝒪( Lκ^3ε^-2) where κ is the condition number, while the total number of gradient calls to obtain an ε-stationary point for nonconvex-concave minimax problems is upper bounded by 𝒪( L^4ε^-4). As far as we know, this is the first completely parameter-free algorithm for solving nonconvex-strongly concave minimax problems, and it is also the completely parameter-free algorithm which achieves the best iteration complexity in single loop method for solving nonconvex-concave minimax problems. Numerical results validate the efficiency of the proposed PF-AGP algorithm.



  minimax optimization problem; backtracking strategies; parameter-free alternating gradient projection algorithm; iteration complexity; machine learning




§ INTRODUCTION



We consider the following minimax optimization problem:

    min_x ∈𝒳max_y ∈𝒴 f(x, y), P

where 𝒳⊆ℝ^n and 𝒴⊆ℝ^m are nonempty closed and bounded convex sets, and f: 𝒳×𝒴→ℝ is a smooth function. The minimax problem is a critical class of optimization problems, known for its extensive research history and broad applications in areas such as machine learning, signal processing, and various other fields in recent years. Many practical problems can be formulated as in (<ref>), including machine learning, such as generative adversarial networks (GANs)  <cit.>, reinforcement learning <cit.>, power control and transceiver design in signal processing <cit.>, distributed nonconvex optimization <cit.>, robust optimization <cit.>, statistical learning <cit.> and several other applications.

There are many research results on convex-concave minimax problems, i.e., f(x, y) is convex with respect to x and concave with respect to y. Nemirovski <cit.> introduced the mirror-prox  method, and Nestrov <cit.> proposed a dual extrapolation algorithm, both of which achieve an optimal iteration complexity of 𝒪(ε^-1) for solving convex-concave minimax problems. For more details, see <cit.>  and references therein.

Recently, various algorithms have been proposed for nonconvex-(strongly) concave minimax problems. For nonconvex-strongly concave minimax problems, several recent works <cit.> have proposed various algorithms that achieve a gradient complexity of 𝒪̃(κ_y^2 ε^-2) for stationary points of Φ(·) = max_y ∈𝒴 f(·, y) (when 𝒳 = ℝ^n), or for stationary points of f, where κ_y is the condition number for f(x, ·). <cit.> proposed an accelerated algorithm that improves the gradient complexity bound to 𝒪̃(√(κ_y)ε^-2).  Furthermore, Zhang et al. <cit.> proposed a generic acceleration framework which can improve the iteration complexity to 𝒪(√(κ_y)ε ^-2).  


For general nonconvex-concave minimax problems, there are two types of algorithms, i.e., multi-loop algorithms and single-loop algorithms. Various multi-loop algorithms have been proposed in <cit.>. The best known iteration complexity of multi-loop algorithms for solving nonconvex-concave minimax problems is 𝒪̃( ε ^-2.5), which was achieved by <cit.>.  For solving nonconvex-linear minimax problems, Pan et al. <cit.> proposed a new alternating gradient projection algorithm with the iteration complexity of 𝒪( ε ^-3). 
Various single-loop algorithms have also been proposed,  e.g., the gradient descent-ascent (GDA) algorithm <cit.>, the hybrid block successive approximation (HiBSA) algorithm <cit.>, the unified single-loop alternating gradient projection (AGP) algorithm <cit.>, and the smoothed GDA algorithm <cit.>. Both the AGP algorithm and the smoothed GDA algorithm achieve the best known iteration complexity, i.e., 𝒪( ε ^-4), among single-loop algorithms for solving nonconvex-concave minimax problems.



It should be noted that for most of the existing results mentioned above, achieving optimal complexity of the algorithm requires the assumption of knowing precise information about some parameters of the problem, such as the Lipschitz constant, or the strongly concave constant μ, etc. Accurately estimating these parameters is often challenging, and conservative estimates can significantly impact algorithm performance <cit.>. Therefore, designing parameter-free algorithms with complexity guarantees without relying on inputs of these parameters has attracted considerable attention recently. 








For nonconvex-strongly concave minimax problems, there are two main parameter-free algorithms: one based on AdaGrad stepsizes and the other based on backtracking. In the framework of AdaGrad stepsizes, Yang et al. <cit.> proposed NeAda, which can achieve convergence rates of 𝒪̃(ε^-2) and 𝒪̃(ε^-4) in deterministic and stochastic settings, respectively.  Li et al. <cit.> proposed TiAda, achieving convergence rates of 𝒪(ε^-2) and 𝒪(ε^-4) in deterministic and stochastic settings, respectively. In the framework of backtracking,  Xu et al. <cit.> proposed a stochastic gradient descent ascent (GDA)  method with backtracking (SGDA-B), achieving an upper bound on the total number of gradient calls of 𝒪(Lκ^2log(κ) ε^-2) with 𝒪(L κ^2 ε^-2) gradient calls required per backtracking step. Meanwhile, Zhang et al. <cit.> proposed a proximal alternating gradient descent ascent method (AGDA+), achieving an upper bound on the total number of gradient calls of  𝒪(Lκ^4ε^-2) with 𝒪(1) gradient calls required per backtracking step. Both methods do not require knowledge of the specific value of Lipschtiz constant L, but still require information about the strongly concave constant μ in the function. In the framework of backtracking, Xu et al. <cit.> proposed a stochastic GDA method with backtracking (SGDA-B) for nonconvex-concave minimax problems, achieving an upper bound on the total number of gradient calls of  𝒪̃(L^3ε^-4) with 𝒪(L^3ε^-4) gradient calls required per backtracking step.





 §.§ Contributions


In this paper, based on the framework of backtracking, we propose two completely parameter-free alternating gradient projection algorithms, i.e., the PF-AGP-NSC algorithm and the PF-AGP-NC algorithm, for solving nonconvex-(strongly) concave minimax problems respectively, which does not require prior knowledge of parameters such as the Lipschtiz constant L or the strongly concave constant μ.  

Moreover, we show that the total number of  gradient calls of the PF-AGP-NSC algorithm to obtain an ε-stationary point for nonconvex-strongly concave minimax problems is upper bounded by 𝒪( Lκ^3ε^-2) where κ is the condition number,  while the total number of gradient calls of the PF-AGP-NC algorithm to obtain an ε-stationary point for nonconvex-concave minimax problems is upper bounded by 𝒪( L^4ε^-4). As far as we know, the PF-AGP-NSC algorithm is the first completely parameter-free algorithm for solving nonconvex-strongly concave minimax problems, and the PF-AGP-NC algorithm is the first completely parameter-free algorithm which achieves the best iteration complexity in single loop algorithms for solving nonconvex-concave minimax problems.







 §.§ Organization

In Sect. <ref>, we propose a parameter-free alternating gradient projection (PF-AGP) algorithm for nonconvex-(strongly) concave minimax problems, and we then analyze the corresponding gradient complexity for two different settings.  We report some numerical results in Sect. <ref> and make some concluding remarks in the last section.


Notation.
For vectors, we use · to denote the l_2-norm. For a function f(x, y): ℝ^n ×ℝ^m →ℝ, we use ∇_x f(x, y) (or ∇_y f(x, y)) to denote the partial gradient of f with respect to the first variable (or the second variable) at point (x, y). Let 𝒫_𝒳 and 𝒫_𝒴 denote projections onto the sets 𝒳 and 𝒴. Finally, we use the notation 𝒪(·) to hide only absolute constants which do not depend on any problem parameter, and 𝒪̃(·) notation to hide only absolute constants and log factors.
A continuously differentiable function f(·) is called θ-strongly convex if there exists a constant θ > 0 such that for any x, y ∈𝒳,

    f(y) ≥ f(x) + ⟨∇ f(x), y - x ⟩ + θ/2y - x^2.

If -f satisfies (<ref>), f(·) is called θ-strongly concave. 
Denote the sign function as (x), i.e.,

    (x) =
     
    1    if  x > 0, 
    
    -1    if  x ≤ 0.







§ TWO COMPLETELY PARAMETER-FREE ALTERNATING GRADIENT PROJECTION ALGORITHMS

In this section, we propose two completely parameter-free alternating gradient projection (PF-AGP) algorithms for solving the nonconvex-strongly concave and the nonconvex-concave minimax problems (<ref>) respectively. Both algorithms do not require prior knowledge of Lipschitz constants, and the algorithm under nonconvex-strong concave settings also does not require prior knowledge of strong concave constants. The two proposed algorithms are based on the AGP algorithm <cit.>, which uses the gradient of a regularized version of the original function, i.e.,

    f_k(x,y) = f(x,y) - c_k/2y_k^2,

where c_k ≥ 0 is a regularization parameters. At the k-th iteration, AGP algorithm consists of the following two gradient projection steps for updating both x and y:

    x_k+1   = 𝒫_𝒳( x_k - 1β_k∇ _xf ( x_k,y_k ) ),
    
    	y_k+1   = 𝒫_𝒴( y_k+1γ_k∇ _y f( x_k,y_k ) - 1/γ_k c_k y_k ),

where  𝒫_𝒳 and 𝒫_𝒴 is the projection operator onto 𝒳 and  𝒴, respectively, and  β_k > 0, γ_k >0 are stepsize parameters. 


Although the AGP algorithm achieves the optimal iteration complexity among single loop algorithms for nonconvex-(strongly) concave minimax problems, it requires the knowledge of the Lipschtiz constant L and the strongly concave coefficient μ, or at least knowledge of the Lipschtiz constant L  to calculate the stepsizes β_k and γ_k under the nonconvex-strongly concave setting and the nonconvex-concave setting, respectively, which limits its applicability. To eliminate the need for the prior knowledge of the Lipschtiz constant L or the strongly concave coefficient μ, we propose two parameter-free AGP algorithms by employing backtracking strategies for solving the nonconvex-strongly concave and the nonconvex-concave minimax problems, respectively. Similar backtracking strategies have been used in
<cit.>. 




 §.§ Nonconvex-strongly Concave Setting

We first propose a completely parameter-free AGP algorithm for solving nonconvex-strongly concave minimax problems.
More specifically, in order to estimate the Lipschitz constants for ∇_x f(·,y), ∇_y f(·,y), ∇_y f(x,·) denoted as L_11, L_12 and L_22, respectively, and the strongly concave constant μ, at each iteration of the proposed algorithm, we aim to find a tuple (l_11^k,i, l_12^k,i, l_22^k,i, μ_k,i) by backtracking such that the following conditions for (x_k,i, y_k,i) are satisfied:

    C_1^k,i   = f(x_k,i, y_k) - f(x_k, y_k) - ⟨∇_x f(x_k, y_k), x_k,i - x_k ⟩ - l^k,i_11/2x_k,i - x_k^2 ≤ 0, C1
    
    	C_2^k,i   = ∇_y f(x_k,i, y_k) - ∇_y f(x_k, y_k) - l_12^k,ix_k,i - x_k≤ 0,C2
    
    	C_3^k,i   = ∇_y f(x_k,i, y_k,i) - ∇_y f(x_k,i, y_k) - l_22^k,iy_k,i - y_k≤ 0,C3
    
    	C_4^k,i   = ⟨∇_y f(x_k,i, y_k,i) - ∇_y f(x_k,i, y_k), y_k,i - y_k ⟩ + μ_k,iy_k,i - y_k^2 ≤ 0.C4

Otherwise, we enlarge l_11^k,i, l_12^k,i, l_22^k,i by a factor of 2 or reduce μ_k,i by half according to the sign of C_1^k,i, C_2^k,i, C_3^k,i and C_4^k,i, respectively. We then update β_k and γ_k by the new estimates, i.e., l_11^k,i, l_12^k,i, l_22^k,i, μ_k,i.
The proposed algorithm for solving nonconvex-strongly concave problems, denoted as PF-AGP-NSC,  is formally presented in Algorithm <ref>.



It is worth noting that the parameter-free NeAda <cit.> and TiAda <cit.> algorithms with adaptive step size are only used to solve the minimax problems when x is unconstrained, while the  proposed PF-AGP-NSC algorithm can solve the constrained case. Compared with the SGDA-B <cit.> algorithm and AGDA+ <cit.> algorithm using the backtracking framework under the nonconvex-strongly concave setting, AGDA+ and SGDA-B still require knowledge of the strongly concave constant mu, while the proposed PF-AGP-NSC algorithm is a completely parameter-free algorithm that does not require any prior knowledge of the parameters.





  §.§.§ Complexity analysis

In this subsection, we analyze the iteration complexity of Algorithm <ref> for solving nonconvex-strongly concave minimax optimization problems, i.e., f(x, y) is nonconvex with respect to x for any fixed y ∈𝒴, and μ-strongly concave with respect to y for any given x ∈𝒳. 

Before we prove the iteration complexity of the PF-AGP-NSC algorithm for solving (<ref>), we need to make the following assumption about the smoothness of f(x, y).

f(x, y) has Lipschitz continuous gradients, i.e., there exist positive scalars L_11, L_12, L_22 such that for any x, x̃∈𝒳, y, ỹ∈𝒴,

    ∇_x f(x, y) - ∇_x f(x̃, y)    ≤ L_11 x - x̃, 
    ∇_y f(x, y) - ∇_y f(x̃, y)    ≤ L_12 x - x̃,
    ∇_y f(x, ỹ) - ∇_y f(x, y)    ≤ L_22ỹ - y .
 

We denote  L = max{L_11, L_12, L_22}. To analyze the convergence of Algorithm <ref>, we define the stationarity gap as the termination criterion as follows.

At each iteration of Algorithm <ref>, the stationarity gap for problem (<ref>) with respect to f(x, y) is defined as:

    ∇ G(x_k, y_k) = [ β_k (x_k - 𝒫_𝒳(x_k - 1/β_k∇_x f(x_k, y_k))); γ_k (y_k - 𝒫_𝒴(y_k + 1/γ_k∇_y f(x_k, y_k))) ].

We denote ∇G_k= ∇G(x_k, y_k),
(∇ G_k)_x= β_k( x_k-𝒫_𝒳( x_k-1/β_k∇ _xf( x_k,y_k))),
and (∇ G_k)_y=γ_k( y_k-𝒫_𝒴( y_k+ 1/γ_k∇ _yf( x_k,y_k )) ).








At each iteration of Algorithm <ref>, the stationarity gap for problem (<ref>) with respect to f_k(x, y) is defined as:

    ∇G̃(x_k, y_k) = 
    [  β_k(x_k - P_x(x_k - 1/β_k∇_x f_k(x_k, y_k))); γ_k (y_k - P_y(y_k + 1/γ_k∇_y f_k(x_k, y_k))) ].
 

Under Assumption <ref>, we first establish the following lemma, which provides bounds on the changes in the function value when x_k is updated at each iteration of Algorithm <ref>. This proof is similar to that of Lemma 2.1 in <cit.>, for the sake of completeness, we give its proof.



 Suppose that Assumption <ref> holds. 
Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>, then we have

    f(x_k+1, y_k) - f(x_k, y_k) ≤ - (β_k - l^k_11/2) x_k+1 - x_k^2 .


	
	

   



  By the optimality condition for (<ref>), we have
		
    ⟨∇ _xf( x_k,y_k ) +β_k ( x_k+1-x_k ) ,x_k-x_k+1⟩≥ 0.

		The backtracking strategy (<ref>) implies that
		
    f( x_k+1,y_k ) -f( x_k,y_k )≤⟨∇ _xf( x_k,y_k ) ,x_k+1-x_k ⟩  +l^k_112 x_k+1-x_k  ^2.

		By adding (<ref>) and (<ref>), we obtain
		
    f( x_k+1,y_k ) -f( x_k,y_k )    ≤ -( β_k -l^k_112)  x_k+1-x_k  ^2,

which completes the proof. 











Next, we provide an estimate of upper bound of the difference between f(x_k+1, y_k+1) and f(x_k, y_k). 

First, we need to make the following assumption on the parameter γ_k.

{γ_k} is a nonnegative monotonically increasing sequence. 


Suppose that Assumptions <ref> and <ref> hold. 
Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>, then we have

    f(x_k+1, y_k+1) - f(x_k, y_k) 
         ≤ -(β_k - l_11^k/2 - l_12^k/2γ_k-1)x_k+1 - x_k^2+γ_k-1y_k+1 - y_k^2
            -(μ_k-1 - γ_k-1/2 - (l_22^k-1)^2/2γ_k-1)y_k - y_k-1^2  .
 
 


The optimality condition for y_k in (<ref>) implies that





    ⟨∇_y f(x_k, y_k-1) - γ_k-1 (y_k - y_k-1), y_k+1 - y_k ⟩≤ 0.

By the concavity of f(x,y) with respect to y, and combining (<ref>), we have 

    f(x_k+1, y_k+1) - f(x_k+1, y_k) 
    ≤   ⟨∇_y f(x_k+1, y_k), y_k+1 - y_k ⟩
    ≤   ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k-1), y_k+1 - y_k ⟩+ γ_k-1⟨ y_k-y_k-1, y_k+1 - y_k ⟩.

Denoting v_k+1 = (y_k+1 - y_k) - (y_k - y_k-1), we can write the first inner product term in the r.h.s. of (<ref>) as

    ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k-1), y_k+1 - y_k ⟩
    
    		=   ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k), y_k+1 - y_k ⟩ + ⟨∇_y f(x_k, y_k) - ∇_y f(x_k, y_k-1), v_k+1⟩
        + ⟨∇_y f(x_k, y_k) - ∇_y f(x_k, y_k-1), y_k - y_k-1⟩.

Next, we estimate the three terms on the right-hand side of (<ref>) respectively. By the backtracking strategy (<ref>), (<ref>) and the Cauchy-Schwarz inequality, we have

    ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k), y_k+1 - y_k ⟩≤l_12^k/2γ_kx_k+1 - x_k^2 + γ_k/2y_k+1 - y_k^2

and

    ⟨∇_y f(x_k, y_k) - ∇_y f(x_k, y_k-1), v_k+1⟩≤(l_22^k-1)^2/2 γ_k-1y_k - y_k-1^2 + γ_k-1/2v_k+1^2 .

By the backtracking strategy (<ref>) of μ_k-1, we obtain

    ⟨∇_y f(x_k, y_k) - ∇_y f(x_k, y_k-1), y_k - y_k-1⟩≤ -μ_k-1y_k - y_k-1^2.

Moreover, it can be easily checked that

    ⟨ y_k - y_k-1, y_k+1 - y_k ⟩ = 1/2y_k - y_k-1^2 + 1/2y_k+1 - y_k^2 - 1/2v_k+1^2.

Plugging (<ref>)-(<ref>) into (<ref>) and rearranging the terms, we conclude that

    f(x_k+1, y_k+1) - f(x_k+1, y_k)    ≤l_12^k/2γ_kx_k+1 - x_k^2 - (μ_k-1 - γ_k-1/2 - (l_22^k-1)^2/2γ_k-1) y_k - y_k-1^2 
          + (  γ_k/2 + γ_k-1/2) y_k+1 - y_k^2.

By Lemma <ref>, we have

    f(x_k+1, y_k) - f(x_k, y_k) ≤ - (β_k - l_11^k/2) x_k+1 - x_k^2.

The proof is then completed by combining (<ref>) with (<ref>).



We now establish an important recursion for Algorithm <ref>.

Suppose that Assumptions <ref> and <ref> hold. 
Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>, and denoting

    f_k+1    = f(x_k+1, y_k+1),    S_k+1 = 2γ^2_k /μ_ky_k+1 - y_k^2, 
    
      F_k+1    = f_k+1 + S_k+1 - ( μ_k + 7γ_k/2 -  (l^k_22)^2/2γ_k - 2(l^k_22)^2/μ_k) y_k+1 - y_k^2-2γ_k^2/μ_k𝒟^2_y,

then we have

    F_k+1 - F_k
    	≤    -( 2μ_k γ_k - (l_22^k)^2/2γ_k + μ_k γ_k - 4(l_22^k)^2/2μ_k) y_k+1-y_k^2
          - ( β_k - l_11^k/2 - l_12^k/2γ_k-1- 2(l_12^k)^2γ_k-1/μ_k-1μ_k) x_k+1 - x_k^2,

where 𝒟_y = max{y_1 - y_2| y_1, y_2 ∈𝒴}. 



By the optimality condition for y_k in (<ref>) we obtain

    ⟨∇_y f(x_k+1, y_k) - γ_k (y_k+1 - y_k), y_k - y_k+1⟩≤ 0.

Combining (<ref>) and (<ref>), we have

    ⟨∇_y f(x_k, y_k-1) - ∇_y f(x_k+1, y_k)+γ_k( y_k+1 - y_k)-γ_k-1( y_k - y_k-1), y_k+1 - y_k⟩≤ 0,

which implies that

    γ_k y_k+1 - y_k^2 - γ_k-1⟨ y_k-y_k-1, y_k+1 - y_k ⟩≤⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k-1), y_k+1 - y_k ⟩ .

Denoting v_k+1 = (y_k+1-y_k)-(y_k-y_k-1), and using the identity 1/2⟨ v_k+1, y_k+1 - y_k⟩ = 1/2y_k+1 - y_k^2 + 1/2v_k+1^2 - 1/2y_k - y_k-1^2, we conclude that

    γ_k y_k+1 - y_k^2 - γ_k-1( 1/2y_k+1 - y_k^2 + 1/2y_k - y_k-1^2 - 1/2v_k+1^2 )
         ≤⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k-1), y_k+1 - y_k ⟩.

By rearranging the terms of the above inequality, we then have

    (γ_k - γ_k-1/2) y_k+1 - y_k^2 + γ_k-1/2v_k+1^2 - γ_k-1/2y_k - y_k-1^2 
    ≤   ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k-1), y_k+1 - y_k ⟩
    
        =   ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k), y_k+1 - y_k ⟩+⟨∇_y f(x_k, y_k) - ∇_y f(x_k, y_k-1), v_k+1⟩
        +⟨∇_y f(x_k, y_k) - ∇_y f(x_k, y_k-1), y_k - y_k-1⟩.

 By backtracking strategy (<ref>) and the Cauchy-Schwarz inequality, we can easily get that

    ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k), y_k+1 - y_k ⟩≤(l_12^k)^2/2μ_kx_k+1 - x_k^2 + μ_k/2y_k+1 - y_k^2.

Combining (<ref>), (<ref>), (<ref>) and (<ref>), we have

    γ_k/2y_k+1 - y_k^2 - γ_k-1/2y_k - y_k-1^2 
    	   ≤(l_12^k)^2/2μ_kx_k+1 - x_k^2 + (μ_k/2 + γ_k-1/2 - γ_k/2) y_k+1 - y_k^2 
          - (μ_k-1 - (l_22^k-1)^2/2γ_k-1) y_k - y_k-1^2.

By multiplying 4γ_k-1/μ_k-1 on both sides of (<ref>), using the definition of S_k+1 and Assumption <ref>, we obtain








    S_k+1 - S_k 
    		   ≤2(l_12^k)^2γ_k-1/μ_k-1μ_kx_k+1 - x_k^2 + 2γ_k-1y_k+1 - y_k^2  
          - (4γ_k-1 - 2(l_22^k-1)^2/μ_k-1) y_k - y_k-1^2  + (2γ_k^2/μ_k - 2γ^2_k-1/μ_k-1) 𝒟_y^2.

The proof is then completed by Lemma <ref> and the definition.


We are now ready to establish the iteration complexity for the PF-AGP-NSC  algorithm. In particular, letting ∇ G(x_k, y_k) be defined as in Definition <ref> and ε > 0 be a given target accuracy, we provide a bound on T(ε), the first iteration index to achieve an ε-stationary point, i.e., ∇ G(x_k, y_k)≤ε, which is equivalent to

    T(ε) = min{k |∇ G(x_k, y_k)≤ε}.

 
Suppose that Assumptions <ref> and <ref> hold. 
Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>, β_k  =l_11^k + l_12^kμ_k-1/8 (l_22^k-1)^2  +  32(l_12^k)^2(l_22^k-1)^2/μ_kμ_k-1^2, γ_k = 8 (l_22^k)^2/μ_k,








then it holds that

    T(ε) ≤F_2 -F/d_1 ε^2 +1,

where  F_2 is defined in (<ref>), 
F = f - ( 224L_22^2/μ + μ_1 - 2 (l_22^1)^2/μ_1 - (l_22^1μ)^2/128L_22^2+2^13L_22^4/μ^3) 𝒟^2_y,








with f = min_(x, y) ∈ X × Y f(x, y),   𝒟_y = max{y_1 - y_2| y_1, y_2 ∈𝒴}, d_1 = min{β_min/2β_max^2+4L^2_12,15μ^3+2^7l_22^1μ/2^15L_22^4},  β_min=l^1_11 + l^1_12μ/64 (L_22)^2 + 32 (l_12^1)^2 (l^1_22)^2/μ_1^3 and  β_max = 2L_11+ μ_1 L_12/4(l^1_22)^2 +  4096(L_12)^2 (L_22)^2/μ^3.





 It follows immediately from Definition <ref> and (<ref>) that

    (∇ G_k)_x  = β_k  x_k+1 - x_k .

On the other hand, by  the triangle inequality, we obtain that

    (∇ G_k)_y    ≤γ_k  P_y (y_k + 1/γ_k∇_y f(x_k+1, y_k)) - P_y (y_k + 1/γ_k∇_y f(x_k, y_k)) 
          + γ_k  y_k+1 - y_k 
       ≤ L_12 x_k+1 - x_k  + γ_k  y_k+1 - y_k .

By combining (<ref>) and (<ref>), and using the Cauchy-Schwarz inequality, we obtain

    ∇ G_k ^2    ≤ (β_k^2 + 2 L_12^2)  x_k+1 - x_k ^2 + 2 γ_k^2  y_k+1 - y_k ^2.

Let d^(k)_1= min{2μ_kγ_k - (l^k_22)^2/2γ_k + μ_kγ_k-4 (l_22^k)^2/2μ_k/2 γ_k^2, β_k - l^k_11/2-  l^k_12/2γ_k-1 - 2 (l^k_12)^2 γ_k-1/μ_k-1μ_k/β_k^2 + 2 L_12^2}>0. Multiplying both sides of (<ref>) by d^(k)_1, and using (<ref>) in Lemma <ref>, we have

    d^(k)_1 ∇ G_k ^2    ≤ F_k - F_k+1.

By using the definitions of d_1, β_k and γ_k, we have
d_1<d^(k)_1, then we get

    d_1 ∇ G_k ^2 
    	≤ F_k - F_k+1.

Summing up the above inequalities from k = 2 to k = T(ε), we obtain

    ∑_k=2^T(ε) d_1 ∇ G_k^2    ≤ F_2 - F_T(ε)+1  .

Note that by the definition of F_k+1 in Lemma <ref>, we have

    F_T(ε)+1   = f_T(ε)+1 + S_T(ε)+1 -2γ_T(ε)^2/μ_T(ε)𝒟^2_y
          - ( 7/2γ_T(ε)+1 + μ_T(ε)+1 - 2 (l_22^T(ε)+1)^2/μ_T(ε)+1 - (l_22^T(ε)+1)^2/2γ_T(ε)+1) ||y_T(ε)+1 - y_T(ε)||^2 
       ≥f - ( 224L_22^2/μ + μ_1 - 2 (l_22^1)^2/μ_1 - (l_22^1μ)^2/128L_22^2+2^13L_22^4/μ^3) 𝒟^2_y 
       =F,

where the inequality follows from the definitions of f and 𝒟_y, and the fact that S_k ≥ 0 (∀ k ≥ 1). 
We then conclude from (<ref>) that

    ∑_k=2^T(ε) d_1 ∇ G_k^2    ≤F_2 - F_T(ε)+1≤F_2 - F  ,

which, in view of the definition of T(ε), implies that

    d_1 (T(ε)-1) ε^2 ≤ F_2 - F ,





or equivalently,

    T(ε) ≤F_2 -  F/d_1 ε^2+1.

We complete the proof.

 
At the k-th iteration of Algorithm <ref>, the number of backtracking step is upper bounded by log_2(2L_11/l_11^1)+log_2(2L_12/l_12^1)+log_2(2L_22/l_22^1)+log_2(2μ_1/μ)=𝒪(1). 
By Theorem <ref>, the total number of  gradient calls of Algorithm <ref> to obtain an ε-stationary point  that satisfies ∇ G(x_k,y_k)≤ε is upper bounded by 𝒪 ( Lκ^3ε^-2) under the nonconvex-strongly concave setting, where κ is the condition number.






 §.§ Nonconvex-concave Setting

In this subsection, we propose a completely parameter-free AGP algorithm for solving nonconvex-concave minimax problems. Similar to the idea of the PF-AGP-NSC Algorithm, we estimate the Lipschitz constants L_11, L_12 and L_22 iteratively, however we do not need to estimate the strongly concave constant μ. At each iteration of the proposed algorithm, we aim to find a tuple (l^k,i_11,l^k,i_12,l^k,i_22) by backtracking such that 
conditions (<ref>), (<ref>), (<ref>) and the following condition (<ref>) are satisfied at (x_k,i,y_k,i):

    C_5^k,i   = ⟨∇_y f_k,i(x_k,i, y_k,i) - ∇_y f_k,i(x_k,i, y_k), y_k,i - y_k⟩+ c^2_k,i+c_k,il^k,i_22/l^k,i_22 + 2c_k,iy_k,i - y_k^2
            +1/l^k,i_22 + 2c_k,i∇_y f_k,i(x_k,i, y_k,i) - ∇_y f_k,i(x_k,i, y_k)^2  ≤ 0. C5

The proposed algorithm, denoted as PF-AGP-NC, is formally presented in Algorithm <ref>.

Please note that the existing SGDA-B algorithm <cit.> for solving nonconvex-concave minimax problems requires the realization of the diameter of the feasible set 𝒴 in order to construct the key auxiliary problem, while the PF-AGP-NC algorithm we propose is a completely parameter-free algorithm that does not require any prior parameters.

 



  §.§.§ Complexity Analysis

In this subsection, we analyze the iteration complexity of Algorithm <ref> for solving (<ref>) under the general nonconvex-concave setting. 






The backtracking strategy (<ref>) is a key inequality that we will use to establish some important recursions for the PF-AGP-NC algorithm under the nonconvex-concave setting in the following two lemmas. This is also one of the main differences between proofs in the nonconvex- strongly concave setting and proofs in the nonconvex-concave setting.




Suppose that Assumptions <ref> and <ref> hold. Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>. If ∀ k, β_k > l^k_11 and γ_k≥l_22^k + 2c_k/2, then

    f(x_k+1,y_k+1) - f(x_k,y_k) 
    ≤    - (   β_k/2 - (l_12^k)^2/2γ_k-1) x_k+1 - x_k^2 +γ_k-1y_k+1- y_k^2 + γ_k-1/2y_k - y_k-1^2
          + c_k-1/2( y_k+1^2 - y_k^2 )+ c_k/2y_k+1-y_k^2.






	
	


The optimality condition for y_k in (<ref>) implies that




    ⟨∇_y f_k-1 (x_k, y_k) - γ_k-1 (y_k - y_k-1), y_k - y_k+1⟩≤ 0.





	
	
	
	
	

The concavity of f_k(x_k+1, y) with respect to y together with (<ref>) then imply that

    f_k (x_k+1, y_k+1) - f_k (x_k+1, y_k)    ≤⟨∇_y f_k (x_k+1, y_k)- ∇_y f_k-1 (x_k, y_k), y_k+1-y_k ⟩
         + ⟨∇_y f_k-1 (x_k, y_k-1), y_k+1 - y_k ⟩
       = ⟨∇_y f_k (x_k+1, y_k) - ∇_y f_k-1 (x_k, y_k), y_k+1 - y_k ⟩
         + ⟨∇_y f_k-1 (x_k, y_k) - ∇_y f_k-1 (x_k, y_k-1), v_k+1⟩
         + ⟨∇_y f_k-1 (x_k, y_k) - ∇_y f_k-1 (x_k, y_k-1), y_k - y_k-1⟩
         + γ_k-1⟨ y_k - y_k-1, y_k+1 - y_k ⟩,

where v_k+1 = y_k+1 - y_k - (y_k - y_k-1). We now provide upper bounds for the inner product terms on the right hand side of (<ref>). Firstly, by the definition of f_k(x_k+1, y_k) and f_k-1(x_k, y_k), the backtracking strategy (<ref>)  and the Cauchy-Schwarz inequality, we have

    ⟨∇_y f_k (x_k+1, y_k) - ∇_y f_k-1 (x_k, y_k), y_k+1 - y_k ⟩
       = ⟨∇_y f(x_k+1, y_k) - ∇_y f (x_k, y_k), y_k+1 - y_k ⟩  - (c_k - c_k-1)⟨ y_k, y_k+1 - y_k ⟩
       ≤(l_12^k)^2/2γ_k-1x_k+1 - x_k^2 + γ_k-1/2y_k+1 - y_k^2 - c_k - c_k-1/2 (y_k+1^2 - y_k^2)
          + c_k - c_k-1/2y_k+1-y_k^2
       ≤(l_12^k)^2/2γ_k-1x_k+1 - x_k^2 + γ_k-1/2y_k+1 - y_k^2 - c_k - c_k-1/2 (y_k+1^2 - y_k^2) 
          + c_k/2y_k+1-y_k^2.

Secondly, by the Cauchy-Schwarz inequality, we get

    ⟨∇_y f_k-1 (x_k, y_k) - ∇_y f_k-1 (x_k, y_k-1), v_k+1⟩
    ≤   1/2 γ_k-1∇_y f_k-1 (x_k, y_k) - ∇_y f_k-1 (x_k, y_k-1) ^2 + γ_k-1/2 v_k+1^2.

Thirdly, by (<ref>) we have that

    ⟨∇_y f_k-1(x_k, y_k) - ∇_y f_k-1(x_k, y_k-1), y_k - y_k-1⟩
    ≤    -1/l_22^k-1 + 2c_k-1∇_y f_k-1(x_k, y_k) - ∇_y f_k-1(x_k, y_k-1) ^2
         - c_k-1l_22^k-1+c^2_k-1/l_22^k-1 + 2c_k-1 y_k - y_k-1^2
    ≤    - 1/l_22^k-1 + 2c_k-1∇_y f_k(x_k, y_k) - ∇_y f_k(x_k, y_k-1) ^2.

We also have the following equation that

    γ_k-1⟨ y_k - y_k-1, y_k+1 - y_k ⟩ = γ_k-1/2 y_k+1 - y_k ^2 + γ_k-1/2 y_k - y_k-1^2 - γ_k-1/2 v_k+1^2.

By γ_k≥l_22^k + 2c_k/2, plugging (<ref>)-(<ref>) into (<ref>), 
 and using the definition of f_k(x_k+1, y_k+1) and f_k(x_k+1, y_k),   we obtain

    f(x_k+1, y_k+1) - f(x_k+1, y_k) 
    ≤   (l_12^k)^2/2 γ_k-1 x_k+1 - x_k ^2 + γ_k-1 y_k+1 - y_k ^2 + γ_k-1/2 y_k - y_k-1^2
          + c_k-1/2( y_k+1^2 - y_k^2 )+ c_k/2y_k+1-y_k^2  .

By  β_k > l^k_11, we have

    f(x_k+1, y_k) - f(x_k, y_k) ≤ -β_k /2 x_k+1 - x_k ^2 .

By adding (<ref>) and (<ref>), we complete the proof.


Suppose that Assumptions <ref> and <ref> hold. Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>. Denote

    S_k+1   = 8γ_k^2/c_k y_k+1 - y_k ^2 + 8γ_k( c_k+1/c_k-1 )  y_k+1^2,
    
    	F_k+1   = f(x_k+1, y_k+1) + S_k+1 - 15γ_k/2y_k+1 - y_k^2 - c_k/2y_k+1^2.

If 

    β_k > l^k_11,   γ_k+1/c_k+1 - γ_k/c_k≤1/5,   γ_k ≥l^k_22 + 2c_k/2 ,

then ∀ k ≥ 2,

    F_k+1 - F_k =    -( 9γ_k/10-c_k/2)y_k+1 - y_k^2 
       -(   + β_k/2 - (l^k_12)^2/2γ_k-1-16(l^k_12)^2γ_k-1/c_k-1^2)x_k+1 - x_k^2 
        +8( γ_kc_k+1/c_k - γ_k-1c_k/c_k-1) y_k+1^2+c_k-1-c_k/2y_k+1^2.



The optimality condition for y_k in (<ref>) implies that

    ⟨∇_y f_k(x_k+1, y_k)-γ_k(y_k+1-y_k), y_k - y_k+1⟩≤ 0,

By (<ref>) and (<ref>), we have  

    ⟨∇_y f_k-1(x_k, y_k-1) - ∇_y f_k(x_k+1, y_k) +   γ_k( y_k+1 - y_k)  - γ_k-1 (y_k - y_k-1), y_k+1 - y_k⟩≤ 0,

and

    γ_k y_k+1 - y_k^2 - γ_k-1⟨ y_k+1 - y_k, y_k - y_k-1⟩
    ≤   ⟨∇_y f_k(x_k+1, y_k) - ∇_y f_k-1(x_k, y_k-1), y_k+1 - y_k⟩,

 which can be rewritten as

    (γ_k - γ_k-1/2) y_k+1 - y_k^2 + γ_k-1/2v_k+1^2 - γ_k-1/2y_k - y_k-1^2
    ≤   ⟨∇_y f_k(x_k+1, y_k) - ∇_y f_k-1(x_k, y_k-1), y_k+1 - y_k ⟩
    
    =   ⟨∇_y f_k(x_k+1, y_k) - ∇_y f_k-1(x_k, y_k), y_k+1 - y_k ⟩+ ⟨∇_y f_k-1(x_k, y_k) - ∇_y f_k-1(x_k, y_k-1), v_k+1⟩
    
    	   + ⟨∇_y f_k(x_k-1, y_k-1) - ∇_y f_k-1(x_k, y_k-1), y_k - y_k-1⟩.

Using an argument similar to the proof of (<ref>)-(<ref>), by (<ref>) and the Cauchy-Schwarz inequality, we conclude from the above inequality that

    (γ_k - γ_k-1/2) y_k+1 - y_k^2 + γ_k-1/2v_k+1^2 - γ_k-1/2y_k - y_k-1^2
    ≤    (l_12^k)^2/2a_kx_k+1 - x_k^2 + a_k/2y_k+1 - y_k^2 - c_k - c_k-1/2 (y_k+1^2 - y_k^2)
        + c_k - c_k-1/2y_k+1 - y_k^2 + 1/2γ_k-1∇_y f_k-1(x_k, y_k) - ∇_y f_k-1(x_k, y_k-1)^2  
        +γ_k-1/2v_k+1- 1/l_22^k-1 + 2c_k-1∇_y f_k(x_k, y_k) - ∇_y f_k(x_k, y_k-1)^2  
         - c_k-1l_22^k-1+c^2_k-1/l_22^k-1  + 2c_k-1y_k - y_k-1^2,

for any a_k > 0. Obviously, we have -c_k-1l^k-1_22+c^2_k-1/2c_k-1 + l^k-1_22≤ -c_k-1l^k-1_22+c^2_k-1/2(l^k-1_22+c_k-1) = -c_k-1/2.
Combining γ_k ≥l^k_22 + 2c_k/2 and rearranging the terms in (<ref>), we obtain

    γ_k/2y_k+1 - y_k^2 + c_k - c_k-1/2y_k+1^2 
         ≤γ_k-1/2y_k - y_k-1^2 + ( α_k/2 + γ_k-1/2 - γ_k/2) y_k+1 - y_k^2  + c_k - c_k-1/2y_k^2 
           +  (l^k_12)^2 /2a_kx_k+1 - x_k^2 - c_k-1/2y_k - y_k-1^2.

By multiplying 16γ_k-1/c_k-1 on both sides of the above inequality, setting a_k = c_k/2, and using the definition of S_k+1, (<ref>) and {γ_k} is nodecreasing, we then obtain

    S_k+1 - S_k    ≤ 8( γ_kc_k+1/c_k - γ_k-1c_k/c_k-1) y_k+1^2 + 16 (l^k_12)^2 γ_k-1/c_k-1^2x_k+1 - x_k^2 
         + 28γ_k/5y_k+1 - y_k^2 - 8γ_k-1y_k - y_k-1^2.




	
	





	
	
	
	
	
	

Combining  (<ref>) and (<ref>), and using the definition of F_k+1,  we complete the proof.


We are now ready to establish the iteration complexity for the PF-AGP-NC algorithm to achieve an ε-stationary point under the general nonconvex–concave setting.

Suppose that Assumptions <ref> and <ref> hold. Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>. If β_k = l_12^k/20l^k-1_22 + 1280(l_12^k)^2 √(k)/19^2l_12^k-1, γ_k =20l^k_22 ,  c_k = 19l^k_22/k^1/4,  then for any given ε > 0, k ≥ 2, we have

    T(ε) ≤max(  ( 5120 ( L_12)^2  d_2d_3/19^2l_22^1 ε^2+1)^2,  76^4L_22^4𝒟̂_y^4/ε^4),

where  d_2= F_2 - F +339L_22𝒟̂_y^2, d_3 = max{d̅_1, 20·19^2L_22^2/17(l_12^1)^2}, 𝒟̂_y = max{y  |  y ∈ Y} with F = f - 639L_22𝒟̂_y^2, f = min_(x,y) ∈ X × Y f(x, y), 
  d̅_1 =  32+ 1+1600L^2_22/128(l_12^1)^2.
 
















We can easily see that the relations in (<ref>) are satisfied by the settings of c_k, γ_k and β_k. Denote α_k = 16 (l_12^k)^2 γ_k-1/c_k-1^2. Then it follows from the selection of  β_k and α_k that

    β_k/2 - (l^k_12)^2/2γ_k-1-16(l^k_12)^2γ_k-1/c_k-1^2 = α_k .

This observation, in view of Lemma <ref>, c_k/2≤19γ_k/40k^1/4≤19γ_k/40 implies that

    α_k x_k+1 - x_k ^2 + 17γ_k/40 y_k+1 - y_k ^2 
    ≤    F_k -F_k+1 
    		+8( γ_kc_k+1/c_k - γ_k-1c_k/c_k-1) y_k+1^2 +c_k-1-c_k/2y_k+1^2.

We can easily check from the definition of f_k(x, y) that

    ∇ G_k  - ∇̃ G_k ≤ c_k  y_k .

By replacing f with f_k, similar to (<ref>) and (<ref>), we immediately obtain that

    (∇̃ G_k)_x  =   β_k     x_k+1 - x_k ,

and

    (∇̃ G_k)_y ≤γ_k  y_k+1 - y_k  + l^k_12 x_k+1 - x_k .

Combining (<ref>) and (<ref>), and using the Cauchy-Schwarz inequality, we have

    ∇̃ G_k ^2 ≤ (  β_k ^2 + 2(l^k_12)^2)  x_k+1 - x_k ^2 + 2γ_k^2 y_k+1 - y_k ^2.

 Since both α_k and β_k are in the same order when k becomes large enough, it then follows from the definition of d̅_1 that ∀ k ≥ 1,

    (  β_k )^2 + 2(l^k_12)^2/α_k^2   = 
    	(l^k_12/γ_k-1 + 64 (l_12^k)^2γ_k-1/c_k-1^2)^2 + 2(l^k_12)^2/α_k^2
    
    	
    	
    	
     	   ≤  32+ 1+1600L^2_22/128(l_12^1)^2 = d̅_1.

Combining the previous two inequalities in (<ref>) and (<ref>), we obtain

    ∇G̃_k^2 ≤d̅_1(α_k)^2 x_k+1 - x_k ^2 + 2 γ_k^2  y_k+1 - y_k ^2 .

Denote d_k^(2) = 1/max{d̅_1 α_k, 17/80γ_k}, c̃_k = 19/20 k^1/4, by multiplying d^(2)_k on the both sides of (<ref>), and using (<ref>), we have

    d_k^(2)∇G̃_k ^2 
    ≤    F_k - F_k+1  +8( γ_kc_k+1/c_k - γ_k-1c_k/c_k-1) y_k+1^2 
    	+c_k-1-c_k/2y_k+1^2
    ≤    F_k - F_k+1  +8( γ_kc_k+1/c_k - γ_k-1c_k/c_k-1) y_k+1^2 
    	+40L_22c̃_k-1-c̃_k/2y_k+1^2.

Denoting

    T̃(ε) = min{ k | ∇G̃(x_k, y_k) ≤ε/2, k ≥ 2 },

summing both sides of (<ref>) from k = 2 to k = T̃(ε), we then obtain

    ∑_k=2^T̃(ε) d^(2)_k ∇G̃_k ^2 
    	   ≤  F_2 - F_T̃(ε)  + 8 (  γ_T̃(ε)-1c_T̃(ε)/c_T̃(ε)-1-γ_1c_2/ c_1) 𝒟̂_y^2 + 40L_22( c̃_1/2 - c̃_T̃(ε)/2) 𝒟̂^2_y 
       ≤ F_2 -  F_T̃(ε) + 339L_22𝒟̂_y^2.

Note that by the definition of F_k+1 in Lemma <ref>, we have

    F_T̃(ε)   ≥f -  639L_22𝒟̂_y^2 = F,

where f = min_(x, y) ∈ X × Y f(x, y). We then conclude from (<ref>) that

    ∑_k=2^∞ d^(2)_k ∇G̃_k ^2 ≤ F_2 - F +339L_22𝒟̂_y^2= d_2.

We can see from the selection of  d_3 that d_3 ≥max{d̅_1, 80γ_k/17α_k}, which implies d^(2)_k ≥1/d_3α_k,  by multiplying d_3  on the both sides of (<ref>), and combining the definition of  d_2,  we have  ∑_k=2^∞1/α_k∇G̃_k ^2 ≤ d_2d_3,
which, by the definition of T̃(ε), implies that

    ε^2/4≤d_2d_3/∑_k=2^T̃(ε)1/α_k≤d_2d_3/∑_k=2^T̃(ε)1/α̅.

By using the fact ∑_k=2^T̃(ε)1/√(k)≥√(T̃(ε)) - 1  and (<ref>), we conclude ε^2/4≤1280·( L_12)^2  d_2d_3/19^2 l_22^1( √(T̃(ε)) -1 ) or equivalently,

    T̃(ε)  ≤( 5120 ( L_12)^2  d_2d_3/19^2l_22^1 ε^2+1)^2.

On the other hand, if  k ≥76^4L_22^4𝒟̂_y^4/ε^4, then  c_k = 19γ_k/20 k^1/4≤ε/2𝒟̂_̂ŷ. This inequality together with the definition of  𝒟̂_y, then imply that c_k  y_k ≤ε/2. Therefore, there exists a

    T(ε)    ≤max( T̃(ε), 76^4L_22^4𝒟̂_y^4/ε^4)
       ≤max(  ( 5120 ( L_12)^2  d_2d_3/19^2l_22^1 ε^2+1)^2,  76^4L_22^4𝒟̂_y^4/ε^4),

such that ∇ G_k ≤∇G̃_k  + c_k  y_k ≤ε/2  + ε/2 = ε. 

















At the k-th iteration of Algorithm <ref>, the number of backtracking step is upper bounded by  log_2(2L_11/l_11^1)+log_2(2L_12/l_12^1)+log_2(2L_22/l_22^1)=𝒪(1).
 According to Theorem <ref>, the total number of  gradient calls of Algorithm <ref> to obtain an ε-stationary point  that satisfies ∇ G(x_k,y_k)≤ε is 𝒪 ( L^4ε^-4) under the nonconvex-concave setting.











§ NUMERICAL RESULTS
 
In this section, we compare the numerical performance of the proposed PF-AGP-NC algorithm with the AGP algorithm.



 §.§ Dirac-GAN problem


The Dirac-GAN problem <cit.> can be formulated as the following nonconvex–concave minimax problem :


    min_x max_y L(x, y) =-log(1 + exp(-xy)) + log 2,

where (0, 0) is the unique stationary point.

Experimental setup.
We set the initial point of the experiment at (x, y) = (1, 1). The termination condition is defined as the norm of the gradient ∇ f(x_i, y_i) being less than or equal to 10^-5, where i represents the number of gradient calls.

For the AGP algorithm, let α_x and β_y be the stepsize of x and y, respectively. We set α_x = 0.8/√(k), β_y = 0.3 and c_k =0.5/k^1/4. For the PF-AGP-NC algorithm, we do not need to preset specific parameters.

In the experiments, we counted the number of gradient calls of the PF-AGP-NC algorithm. The results of alternating gradient projection in the k th iteration are represented by nodes in the graph and the number of backtracking step computations is represented by the connecting lines.

Results Figure <ref> shows the convergence curves of the AGP and PF-AGP-NC algorithms. It can be seen that although the PF-AGP-NC algorithm involves a double-layer structure, requiring the determination of points in the outer layer through computations in the inner layer, even when considering the number of iterations in the inner layer, the overall convergence speed is still faster than that of the AGP algorithm.





§ CONCLUSIONS AND DISCUSSION

In this paper, based on the framework of backtracking, we propose two completely parameter-free alternating gradient projection algorithms, i.e., the PF-AGP-NSC algorithm and the PF-AGP-NC algorithm, for solving nonconvex-(strongly) concave minimax problems respectively, which does not require prior knowledge of parameters such as the Lipschtiz constant L or the strongly concave constant μ.   Moreover, we show that the total number of  gradient calls of the PF-AGP-NSC algorithm to obtain an ε-stationary point for nonconvex-strongly concave minimax problems is upper bounded by 𝒪( Lκ^3ε^-2) where κ is the condition number,  while the total number of gradient calls of the PF-AGP-NC algorithm to obtain an ε-stationary point for nonconvex-concave minimax problems is upper bounded by 𝒪( L^4ε^-4). As far as we know, the PF-AGP-NSC algorithm and the PF-AGP-NC algorithm are the first completely parameter-free algorithms for solving nonconvex-strongly concave minimax problems and nonconvex-concave minimax problems respectively.  Numerical results demonstrate the efficiency of the two proposed algorithms.





















































































































































































































 Paper Title: Tractable and Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation.




Captured molecules could  make a Bose star visible
    I. B. Samsonov
    August 5, 2024
==================================================




§ ABSTRACT 

Distributional reinforcement learning improves performance by effectively capturing environmental stochasticity, but a comprehensive theoretical understanding of its effectiveness remains elusive.
In this paper, we present a regret analysis for distributional reinforcement learning with general value function approximation in a finite episodic Markov decision process setting. 
We first introduce a key notion of Bellman unbiasedness for a tractable and exactly learnable update via statistical functional dynamic programming.
Our theoretical results show that approximating the infinite-dimensional return distribution with a finite number of moment functionals is the only method to learn the statistical information unbiasedly, including nonlinear statistical functionals.
Second, we propose a provably efficient algorithm, , achieving a regret bound of Õ(d_E H^3/2√(K)) where H is the horizon, K is the number of episodes, and d_E is the eluder dimension of a function class.









§ INTRODUCTION



Distributional reinforcement learning (DistRL) <cit.> is an advanced approach to reinforcement learning (RL) that focuses on the entire probability distribution of returns rather than solely on the expected return.
By considering the full distribution of returns, distRL provides deeper insight into the uncertainty of each action, such as the mode or median. 
This framework enables us to make safer and more effective decisions that account for various risks <cit.>, particularly in complex real-world situations, such as robotic manipulation <cit.>,
neural response <cit.>, 
stratospheric balloon navigation <cit.>, 
algorithm discovery <cit.>, and several game benchmarks <cit.>.


Although distRL has a solid theoretical basis and demonstrates effective empirical performance, its efficiency and tractability are still not fully understood.

In practice, distributions contain an infinite amount of information, and we should resort to approximations using a finite number of parameters or statistical functionals, such as categorical <cit.> and quantile representations <cit.>.
However, not all statistical functionals can be learned through the Bellman operator, as the meaning of statistical functionals is not typically preserved after updates. 
For example, the median is not preserved under the Bellman updates, as the median of a mixture of two distributions does not necessarily equate to the mixture of their medians.

Hence, a distinct analysis is required to determine whether there exists a corresponding Bellman operator for each statistical functional that ensures commutativity.


<cit.> defined Bellman closedness as a property of statistical functionals that can be exactly learned by the existence of a corresponding Bellman operator.








At this point, we take a closer look at the distributional Bellman update and revisit what additional properties of statistical functionals beyond Bellman closedness are needed to construct tractable algorithms. 
In doing so, we identify and provide the following challenges that arise when statistical functionals are used to express the update process:
























    
  * Representing a mixture distribution with a finite, fixed number of parameters inevitably leads to approximation errors during the update process. 
    For example, when expressing the mixture of two distributions, each represented by N parameters, compressing 2N into N parameters in the mixture results in inevitable information loss.
    
    
    
    
    

    
  * Due to the unknown nature of the transition probability (·|s,a), the target distribution is estimated by sampling the next state s'. 
    For a tractable algorithm, it is crucial that the statistical functionals of the target distribution can be unbiasedly estimated using the statistical functionals from the sampled distribution.    



In this paper, we describe the statistical functional and sketch of the distribution, and introduce the additional approximation errors that can occur with the distRL method. 
We define the notion of Bellman unbiasedness, a property for representing the distribution tractably and prove that the moment functional is the only solution that satisfies this property, including nonlinear statistical functionals.
Based on this property, we propose a provably efficient statistical functional reinforcement learning algorithm with general value function approximation, denoted as . 
We will discuss the inherent issues of distributional Bellman completeness, a structural assumption previously defined in the literature <cit.>, and investigate the benefits of redesigning this concept using a sketch.






In summary, our main contributions are as follows:

    
  * Introduce a key property of Bellman unbiasedness for collections of statistical functionals (sketch) and show that the moment functionals are the unique structure that can be unbiasedly estimated and exactly learned from distributions, including nonlinear statistical functionals.

    
  * Revisit the existing concept of distributional Bellman Completeness through a statistical functional lens.
    We enhance the tractability by using moment functional, that the agent can only access a finite-dimensional embedding space.
    This removes the misspecification errors commonly encountered when discretizing infinite-dimensional distributions.

    
  * Propose a tractable and provably efficient distributional RL algorithm called , achieving a tightest regret upper bound Õ(d_E H^3/2√(K)) [We ignore poly-log terms in H,S,A,K in the Õ(·) notation.] in distributional framework.
    Our framework yields a better regret bound with a weaker structural assumption compared to prior results.















§ RELATED WORK














  
Distributional RL.
In classical RL, the Bellman equation, which is based on expected returns, has a closed-form expression.
However, it is questionable whether any statistical functionals of return distribution always have their corresponding closed-form expressions. 
<cit.> introduced the notion of Bellman Closedness for collections of statistical functionals that can be updated in a closed form via Bellman update. 
They showed that the only Bellman-closed statistical functionals in the discounted setting are the moments _Z ∼η[Z^k]. 
More recently, <cit.> proposed a general framework for distRL, where the agent plans to maximize their own utility functionals instead of expected return and formalized the property of Bellman Optimizability.
They proved that the only W_1-continuous and linear Bellman Optimizable statistical functionals are shown to be exponential utilities 1/λlog_Z ∼η[exp(λ Z)] in the undiscounted setting.

In practice, C51 <cit.> and QR-DQN <cit.> are notable distributional RL algorithms where the convergence guarantees of sampled-based algorithms are proved <cit.>.
<cit.> expanded the class of policies on arbitrary distortion risk measures by taking the based distribution non-uniformly and improve the sample efficiency from their implicit representation of the return distribution.
<cit.> highlighted the drawbacks of optimistic exploration in distRL, introducing a randomized exploration method that perturbs the return distribution when the agent selects their next action. 

























  
RL with General Value Function Approximation.
Regret bounds have been studied for a long time in RL, across various domains such as bandit <cit.>, tabular RL <cit.>, and linear function approximation <cit.>.
In recent years, deep RL has shown significant performance using deep neural networks as function approximators, and attempts have been made to analyze whether it is efficient in terms of general function approximation.
<cit.> established a provably efficient RL algorithm with general value function approximation based on the eluder dimension d_E <cit.> and achieves a regret upper bound of Õ(poly(d_E H)√(K)).
To circumvent the intractability from computing the upper confidence bound, <cit.> injected the stochasticity on the training data and get the optimistic value function instead of upper confidence bound, enhancing computationally efficiency.
Beyond risk-neutral setting, several prior works have shown regret bounds under risk-sensitive objectives (e.g., entropic risk <cit.>, CVaR <cit.>), which align with our approach in that they are built on a distribution framework.
<cit.> achieved the regret upper bound of Õ(exp(H) √(|𝒮|^2 |𝒜| H^2 K)) and the lower bound of Ω(exp(H) √(|𝒮||𝒜|HK)) in tabular setting.




  
DistRL with General Value Function Approximation.
Recently, only few efforts have aimed to bridge the gap between two fields.
<cit.> proposed a distributional RL algorithm, , which enjoys small-loss bound by using a log-likelihood objective.
Similarly, <cit.> provided a risk-sensitive reinforcement learning framework with static lipschitz risk measure.
While these studies analyze within a distributional framework, they do not address the intractability of implementation in infinite-dimensional space of distributions.
In contrast, our approach focuses on a statistical functional framework, providing a detailed comparison with other distRL methods as shown in Table <ref>.








§ PRELIMINARIES









  
Episodic MDP.

We consider a episodic Markov decision process which is defined as a ℳ = (𝒮, 𝒜, H, ℙ, r) characterized by state space 𝒮, action space 𝒜, horizon length H, transition kernels ℙ = {ℙ_h }_h ∈ [H], and reward r = { r_h }_h ∈ [H] at step h ∈ [H]. 







The agent interacts with the environment across K episodes. For each k ∈ [K] and h ∈ [H], ℍ_h^k = (s_1^1 ,a_1^1, …, s_H^1, a_H^1, …, s_h^k , a_h^k ) represents the history up to step h at episode k.
We assume the reward is bounded by [0,1] and the agent always transit to terminal state s_end at step H+1 with r_H+1=0.















  
Policy and Value Functions.
A (deterministic) policy π is a collection of H functions {π_h : 𝒮→𝒜}_h=1^H.
Given a policy π, a step h ∈ [H], and a state-action pair (s,a) ∈𝒮×𝒜, the Q and V-function are defined as




Q_h^π(s,a) (: 𝒮×𝒜→ℝ ) _π[∑_h'=h^H r_h'(s_h', a_h') | s_h = s, a_h =a ] and
V_h^π(s) (: 𝒮→ℝ ) _π[∑_h'=h^H r_h'(s_h', a_h') | s_h = s ].






  
Random Variables and Distributions. For a sample space Ω, we extend the definition of the Q-function into a random variable and its distribution,



    Z_h^π(s,a) (: 𝒮×𝒜×Ω→ℝ ) ∑_h'=h^H r_h'(s_h', a_h') | s_h = s, a_h =a, a_h' = π_h'(s_h'),
        
       η_h^π(s,a) (: 𝒮×𝒜→𝒫(ℝ) ) law(Z_h^π (s,a)).

Analogously, we extend the definition of V-function by introducing a bar notation.



    Z̅_h^π(s) (: 𝒮×Ω→ℝ ) ∑_h'=h^H r_h'(s_h', a_h') | s_h = s, a_h' = π_h'(s_h'),
        
       η̅_h^π(s) (: 𝒮→𝒫(ℝ) ) law(Z̅_h^π (s)).

Note that Z̅^π_h(s) = Z^π_h(s, π(s)) and η̅_h^π(s) =η_h^π(s,π(s)).
We use π^⋆ to denote an optimal policy 
and denote V_h^⋆(s) = V_h^π^⋆(s), Q_h^⋆(s,a) = Q_h^π^⋆ (s,a), η_h^⋆(s,a) = η_h^π^⋆(s,a), and η̅_h^⋆(s) = η̅_h^π^⋆(s).
For notational simplicity, we denote the expectation over transition,
[ℙ_h V_h+1^π](s,a) = _s' ∼ℙ_h(· | s,a)  V_h+1^π(s'),
[ℙ_h Z̅_h+1^π](s,a) = _s' ∼ℙ_h(· | s,a)Z̅_h+1^π(s'),
and [ℙ_h η̅_h+1^π](s,a) = _s' ∼ℙ_h(· | s,a)η̅_h+1^π(s'). 
[Note that _s' ∼_h(· |s,a)η̅^π_h+1(s') is a mixture distribution.]
For brevity, we refer to η̅^π simply as η̅.

In the episodic MDP, the agent aims to learn the optimal policy through a fixed number of interactions with the environment across a number of episodes.
At the beginning of each episode k(∈ [K]), the agent starts at the initial state s_1^k and choose a policy π^k.
In step h(∈ [H]), the agent observes s^k_h (∈𝒮), takes an action a^k_h (∈𝒜) ∼π^k_h(·| s^k_h), receives a reward r_h(s^k_h, a^k_h), and the environment transits to the next state s^k_h+1∼_h(· | s^k_h, a^k_h).
Finally, we measure the suboptimality of an agent by its regret, which is the accumulated difference between the ground truth optimal and the return received from the interaction.

The regret after K episodes is defined as
Reg(K) = ∑_k=1^K V_1^⋆(s_1^k) - V_1^π^k(s_1^k) .




    





























    























  
Distributional Bellman Optimality Equation.
Recall that η^⋆_h satisfies the distributional Bellman optimality equation:



















    η_h^⋆ (s,a) = (Τ_h η_h+1^⋆) (s,a)
           = _s' ∼_h(·|s,a), a' ∼π_h^⋆(· | s') [(ℬ_r_h)_#η_h+1^⋆ (s', a')]
        
       =  (ℬ_r_h)_# [_h η_h+1^⋆] (s, a)

where ℬ_r : ℝ→ℝ is defined by ℬ_r (x) = r + x, and
g_#η∈𝒫(ℝ) is the pushforward of the distribution η through g, i.e., g_#η (A) = η( g^-1(A) ) for any Borel set A ⊆ℝ.















  
Additional Notations.

For a given N, we denote an N-dimensional function class 
ℱ^N ℱ^(1)×…×ℱ^(N)⊆{ f = [f^(1), ⋯, f^(N)] : 𝒮×𝒜→ℝ^N }.






Given a dataset 𝒟 = { (s_t, a_t, [z_t^(1) , …, z_t^(N)])}_t=1^|𝒟|⊆𝒮×𝒜×ℝ^N,
a set of state-action pairs 𝒵 = { (s_t, a_t) }_t=1^|𝒵|⊆𝒮×𝒜
and
for a function f: 𝒮×𝒜→ℝ^N, we define the norm f^(n)_∞, f_∞, 1, f_𝒟, f_𝒵 as written in Appendix <ref>.












For a set of (vector-valued) functions ℱ^N ⊆{ f: 𝒮×𝒜→ℝ^N},
the width function of (s,a) is defined as
w^(n)(ℱ^N , s, a) 
    max_f,g ∈ℱ^N  | f^(n)(s,a) - g^(n)(s,a) |.








































§ STATISTICAL FUNCTIONALS IN DISTRIBUTIONAL RL







































 §.§ Bellman Unbiasedness


We revisit the definitions of statistical functionals and illustrate the property of Bellman closedness described in <cit.>.
Then we introduce another pivotal property, Bellman unbiasedness, required for learning in terms of statistical functionals, rather than approximated distribution.



    A statistical functional is a mapping from a probability distribution to a real value ψ : 𝒫(ℝ) →ℝ. 
    A sketch is a vector-valued function ψ_1:N : 𝒫(ℝ) →ℝ^N specified by an N-tuple where each component is a statistical functional, 
    
    ψ_1:N(·) = (ψ_1(·), ⋯, ψ_N(·)).


We denote the domain of sketch as 𝒫_ψ_1:N(ℝ) and its image as 
I_ψ_1:N = {ψ(η̅) : η̅∈𝒫_ψ_1:N(ℝ) }.
We further extend to state return distribution functions 

ψ_1:N(η̅) = (ψ_1:N(η̅(s)) : s ∈𝒮).











    A sketch ψ_1:N is Bellman closed if there exists an operator Τ_ψ_1:N : I_ψ_1:N^𝒮→ I_ψ_1:N^𝒮 such that
    
    ψ_1:N (Τη̅) = Τ_ψ_1:Nψ_1:N( η̅)   for all η̅∈𝒫(ℝ)^𝒮

    which is closed under the distributional Bellman operator Τ : 𝒫(ℝ)^𝒮→𝒫(ℝ)^𝒮.

















Bellman closedness is the property that a sketch are exactly learnable when updates are performed from the infinite-dimensional distribution space to the finite-dimensional embedding space. 
Notably, <cit.> showed that the only finite linear statistical functionals that are Bellman closed are given by the collections of statistical functionals where its linear span is equal to the set of exponential-polynomial functionals 

where ψ_0 is the constant functional equal to 1.



    
    <cit.> discussed only linear statistical functionals when defining a sketch, leaving open questions about the Bellman closedness of nonlinear statistical functionals such as variance, central moments, or quantiles.
    However, it is noteworthy that nonlinear statistical functionals, such as maximum or minimum, can also be Bellman closed. 
    
    In their proof, there might be an ambiguity regarding the assumption that the corresponding sketch Bellman operator for quantile is linear.
    
    
    In this paper, we show the non-existence of sketch Bellman operator for quantile and discuss in detail in Appendix <ref>.
    
 























































    Quantile functional cannot be Bellman closed under any additional sketch.




We investigate whether there exists another sketch that is both nonlinear and Bellman closed, and introduce additional conditions under which the Bellman closed property might be missing.
During the implementation phase, it is important to note that the agent does not have access to the transition kernel. 

Specifically, the agent can only access the empirical transition kernel  which is derived from a finite number of sample estimates. 
This limitation implies that the operator should be treated as an empirical operator 𝒯̂_ψ, rather than 𝒯_ψ. 
Therefore, we introduce a new notion of Bellman unbiasedness that allows us to unbiasedly estimate the expected distribution (ℬ_r)_#_s' ∼(· | s,a) [η̅(s')], which is a mixture by transitions, from the sample distribution (ℬ_r)_#η̅(s').


























































    
    
    A sketch ψ is Bellman unbiased if there exists an integer k and a corresponding vector-valued estimator ϕ_ψ = ϕ_ψ(ψ(x_1), ⋯, ψ(x_k)) : (I_ψ^𝒮)^k → I_ψ^𝒮 where the sketch of expected distribution (ℬ_r)_#_s' ∼(· | s,a)[η̅(s')] can be unbiasedly estimated by ϕ_ψ using the k samples from the sketch of the sample distribution (ℬ_r)_#η̅(s'), i.e.,
    
    _s'_1, ⋯ ,s'_k ∼(· | s,a)[ ϕ_ψ( ψ( (ℬ_r)_#η̅(s'_1)_𝒯̂_ψψ(η) ),  ⋯, ψ( (ℬ_r)_#η̅(s_k')  ) )]
                    = ψ( (ℬ_r)_#_s' ∼(· | s,a) [ η̅(s') ]).















Bellman unbiasedness is another natural definition, similar to Bellman closedness, which takes into account a finite number of samples for the transition.
As a simple example, median functional is not Bellman unbiased as the sample median is not an unbiased estimate of the median functional.
In general, there is no unbiased estimator for the median, implying that the median functional cannot be unbiasedly estimated within an embedding space represented by a finite number of statistical functionals.
On the other hand, variance functional is a nonlinear statistical functional but can be unbiasedly estimated.
Then, the following question naturally arises;



    “Which sketches are unbiasedly estimatable under the sketch-based Bellman update?”



The following lemma answers this question.
















    Let F_η̅ be a CDF of the probability distribution η̅∈𝒫_ψ(ℝ)^𝒮. 
    Then a sketch is Bellman unbiased if and only if the sketch is homogeneous over 𝒫_ψ(ℝ)^𝒮 of degree k, i.e.,
    there exists some vector-valued function h = h(x_1, ⋯, x_k): 𝒳^k →ℝ^N such that
    
    ψ(η̅) =  ∫⋯∫ h(x_1, ⋯, x_k) dF_η̅(x_1) ⋯ dF_η̅(x_k).












Lemma <ref> states that in statistical functional dynamic programming, the unbiasedly estimatable embedding of a distribution can only be structured in the form of functions that are homogeneous of finite degree.

(e.g., variance functional is a homogeneous of degree 2 <cit.>).
Taking this concept further and combining it with the results on Bellman closedness, we prove that even when including a nonlinear statistical functional, the only sketch that can be unbiasedly estimated in a finite-dimensional embedding space is the moment functional.










































































    The only finite statistical functionals that are both Bellman unbiased and closed are given by the collections of ψ_1, …,  ψ_N where its linear span {∑_n=0^N α_n ψ_n | α_n ∈ℝ ,∀ N} is equal to the set of exponential polynomial functionals {η→_Z ∼η[Z^l exp(λ Z)] |   l = 0, 1, …, L, λ∈ℝ}, where ψ_0 is the constant functional equal to 1.
    In discount setting, it is equal to the linear span of the set of moment functionals {η→_Z ∼η[Z^l] |  l = 0, 1, …, L } for some L ≤ N.  















    























Compared to <cit.>, our results extend beyond linear statistical functionals to include unbiasedly estimatable nonlinear statistical functionals, showing the uniqueness of the moment functional. 
Our theoretical results not only show that high-order central moments such as variance or skewness are exactly learnable, but also reveal that other nonlinear statistical functionals like median or quantiles inevitably involve approximation errors due to biased estimations.
In Section <ref>, we emphasize that unbiasedly estimable sketches play an important role in mitigating  misspecification errors, thereby enabling us to achieve sublinear regret. 










































































 §.§ Statistical Functional Bellman Completeness



























We consider distributional reinforcement learning with value function approximation.
For successful TD learning, reinforcement learning with function approximation commonly requires the assumption, Bellman Completeness, that after applying Bellman operator, the output lies in the function class ℱ <cit.>.
On the other hand, our approach receives a tuple of function class ℱ^N ⊆{ f : 𝒮×𝒜→ℝ^N } as input to represent N-moment of distribution.
Building on this, we assume that for any η̅ : 𝒮→𝒫([0,H]), the sketch of target function that results from applying sketch Bellman operator lies in the function class ℱ^N.







In the seminal works, <cit.> and <cit.> assumed that the function class 
ℋ⊆{η : 𝒮×𝒜→𝒫([0,H])} follows the distributional Bellman Completeness assumption, i.e., 
if η∈ℋ for all π, h ∈ [H], then Τ_h^πη∈ℋ.
However, since the distributional Bellman operator mixes distributions for the next state transition, it implies that a function class ℋ must be closed under mixture.
Due to this closedness under mixture, 

both previous studies assumed a discretized reward MDP, which requires the prior knowledge that outcomes of return distribution are additively closed.


In general, representing a distribution with a fixed number of parameters inevitably increases the required number of parameters due to the emergence of a mixture distribution.
Consequently, trying to represent it with a limited number of parameters leads to approximating the learning outcome, thereby causing model misspecification


which generally introduces an additional linear regret.
Precisely, a regret lower bound with model misspecification error ζ is known as Ω(ζ K) in a linear bandit setting <cit.>.
Therefore, assuming Bellman Completeness on the distribution space is invalid as it significantly constrains the MDP structure or causes linear regret.

To circumvent this issue, we revisit the assumption of distributional Bellman completeness through the statistical functional lens. 
We propose a new framework that matches a finite number of all statistical functionals to the target function, rather than the entire distribution itself.



[Statistical Functional Bellman Completeness]
    
    For any distribution η̅ : 𝒮→𝒫([0,H]) and h ∈ [H], there exists f_η̅∈ℱ^N which satisfies
    
    f_η̅(s,a) = ψ_1:N( (ℬ_r_h)_# [ℙ_h η̅](s,a) ) 
              ∀ (s,a) ∈𝒮×𝒜




[Model  Misspecification]
    There exists a set of function ℱ^N and a real number ζ>0, such that for any η̅ : 𝒮→𝒫([0,H]) and h ∈ [H], there exists f_η̅∈ℱ^N which satisfies
    
    max_(s,a) ∈𝒮×𝒜 f_η̅(s,a) - ψ_1:N( (ℬ_r_h)_# [ℙ_h η̅](s,a) ) ≤ζ  ∀ (s,a) ∈𝒮×𝒜.

    We call ζ the misspecification error.






















































§ ALGORITHMS

    
    
    

In this section, we propose  for distributional RL framework with general value function approximation. 
Leveraging the result from Theorem <ref>, we introduce a moment least square regression.

This allows us to capture a finite set of moment information from the distribution, which can be unbiasedly estimated, thereby leading to the truncated moment problem.
Unlike previous work <cit.> that estimates in infinite-dimensional distribution spaces, our method enables tractable distribution estimation in finite-dimensional embedding spaces without approximation error.







    




  
Overview.
At the beginning of episode k ∈ [K], we maintain all previous samples {(s^τ_h', a^τ_h', r^τ_h') }_(τ, h') ∈ [k-1] × [H] and initialize a sketch ψ_1:N( η̅^k_H+1(·) ) = 0^N.
For each step h = H, …, 1, we compute the normalized sample moments of target distribution { (ℬ_r_h'^τ)_#η̅^k_h+1(s_h'+1^τ) }_h' ∈ [H] with the help of binomial theorem,

    ψ_n( (ℬ_r_h'^τ)_#η̅_h(s_h'+1^τ) ) 
           = [(Z̅_h+1^k (s_h'+1^τ) + r_h'^τ)^n] / H^n-1
        
       = ∑_n'=0^n H^n'ψ_n'( η̅_h(s_h'+1^τ) ) (r_h'^τ)^n- n'/ H^n-1

and iteratively solve the N-moment least square regression

    f̃^k_h,η̅←min_f ∈ℱ∑_τ=1^k-1∑_h'=1^H( ∑_n=1^N f^(n)(s_h'^τ, a_h'^τ ) - ψ_n ( (ℬ_r_h'^τ)_#η̅_h+1^k(s_h'+1^τ) )  )^2

based on the dataset 𝒟^k_h which contains the sketch of temporal target ψ_1:N( (ℬ_r_h'^τ)_#η̅_h+1^k(s_h'+1^τ) ). 
Then we define Q^k_h(·, ·) =  min{ (f̃^k_h,η̅)^(1)(·, ·) + b^k_h(·, ·), H }  and choose the greedy policy π^k_h(·) with respect to Q^k_h.
We repeat the procedure until all the K episodes are completed.




    For an optimistic planning, we define the bonus function as the width function b^k_h(s,a)  w^k_h((ℱ^N)^k_h, s,a) where (ℱ^N)^k_h denotes a confidence region at step h, episode k.
    When ℱ is a linear function class, the width function can be evaluated by simply computing the maximal distance of weight vector.
    For a general function class ℱ, computing the width function requires to solve a set-constrained optimization problem, which is known as NP-hard <cit.>.
    However, a width function is computed simply for optimistic exploration, and approximation errors are known to have a small effect on regret <cit.>.
    



















































§ THEORETICAL ANALYSIS


In this section, we provide the theoretical guarantees for  under assumption <ref>.
We apply proof techniques from <cit.> and extend the result to a statistical functional lens. 
First, we generalize the concept of eluder dimension <cit.> to the vector-valued function, which has been widely used in RL literatures <cit.> to measure the complexity of learning with the function approximators.


    Let ϵ≥ 0 and 𝒵 = { (s_i, a_i) }_i=1^n ⊆𝒮×𝒜 be a sequence of state-action pairs.


    

    
  * A state-action pair (s,a) ∈𝒮×𝒜 is ϵ-dependent on 𝒵 with respect to ℱ^N if f - g_𝒵≤ϵ for any vector-valued function f, g ∈ℱ^N, then |f^(1)(s,a) - g^(1)(s,a)| ≤ϵ. 
    

    
  * An (s,a) is ϵ-independent on 𝒵 with respect to ℱ^N if (s,a) is not ϵ-dependent on 𝒵.

    
  * The ϵ-eluder dimension dim_E(ℱ^N ,ϵ) of a vector-valued function class ℱ^N is the length of the longest sequence of elements in 𝒮×𝒜 such that, for some ϵ' ≥ϵ, every element is ϵ'-independent on its predecessors.
    


    








We assume that the function class ℱ^N and state-action space 𝒮×𝒜 have bounded covering numbers.
[Covering number]
    
    For any ϵ >0, the following holds:
    
        
  * there exists an ϵ-cover 𝒞(ℱ^N, ϵ) ⊆ℱ^N with size |𝒞(ℱ^N, ϵ)| ≤𝒩(ℱ^N, ϵ), such that for any g ∈ℱ^N, there exists g' ∈𝒞(ℱ^N, ϵ) with g-g'_∞, 1≤ϵ.

        
  * there exists an ϵ-cover 𝒞(𝒮×𝒜, ϵ) with size | 𝒞(𝒮×𝒜 , ϵ) | ≤𝒩(𝒮×𝒜 , ϵ), such that for any (s,a) ∈𝒮×𝒜, there exists (s', a')∈𝒞(𝒮×𝒜, ϵ) with max_f ∈ℱ | f(s,a) - f(s',a') | ≤ϵ     
    
    
    

    
    



















The following two lemmas give confidence bounds on the sum of the l_2 norms of all normalized moments.






Consider a fixed k ∈ [K] and a fixed h ∈ [H]. Let 𝒵_h^k = { (s_h^τ , a_h^τ)}_τ∈ [k-1] and 

𝒟^k_h, η̅ = {( s_h^τ , a_h^τ,  ψ_1:N( (ℬ_r_h'^τ)_#η̅(s_h'+1^τ) )) }_τ∈ [k-1] for any η̅ : 𝒮→𝒫([0,H]).
Define f̃_h, η̅^k = min_f ∈ℱ^N f ^2_𝒟^k_h, η̅.
For any η̅ and δ∈ (0,1), there is an event ℰ(η̅, δ) such that conditioned on ℰ(η̅, δ), with probability at least 1-δ, for any η̅' : 𝒮→𝒫([0,H]) with 
ψ_1:N( η̅') - ψ_1:N(η̅) _∞,1≤ 1/T 

,we have

    f̃_h,η̅'(· , ·) - ψ_1:N( (ℬ_r(·, ·))_#  [ℙη̅'](· , ·) ) _𝒵_h^k   ≤c' (N^1/2 H √(log(1/δ) + log𝒩(ℱ^N, 1/T)  ))

for some constant c' >0.






We remark that using moment functional as a sketch is a unique structure that leads to a concentration results for the difference between the target and sampled sketches
ψ( (ℬ_r)_# [η̅]) - ψ( (ℬ_r)_#η̅), as the tranformation ϕ_ψ of moment functional is an identity.
Furthermore, using normalized moments minimizes the size of the confidence region from O(H^N) to O(√(N)). 
This adjustment is akin to scaling the optimization function in multi-objective optimization to treat each objective equally, which effectively prevents the model from favoring objectives with larger scales. 


















    
    












































    








    































  
























    





























    


















    Let (ℱ^N)_h^k = { f ∈ℱ^N |  f - f̃^k_h, η̅^2_𝒵^k_h≤β(ℱ^N, δ) }, where
    
    β(ℱ^N,δ) ≥ c' · NH^2 (log (T/δ) + log𝒩(ℱ^N , 1/T) )

    for some constant c'>0.
    Then with probability at least 1-δ/2, for all k,h ∈ [K] × [H], we have
    
    ψ_n ( (ℬ_r_h(·, ·))_#[_h η̅_h+1^k](·, ·) ) ∈ (ℱ^N)_h^k

































    




Lemma <ref> guarantees that the sequence of moments from the target distribution ψ_1:N( (ℬ_r_h(·, ·))_# [_h η̅_h+1^k](·, ·) ) lies in the confidence region (ℱ^N)^k_h with high probability.
Supported by the aforementioned lemma, we can further guarantee that all Q-functions are optimistically estimated with high probability and derive our final result.















































    



    


















    




































































































    Under Assumption <ref>, with probability at least 1- δ,  achieves a regret bound of
    
    Reg(K) ≤ 
        2Hdim_E(ℱ^N, 1/T) + 4H√(KH log(1/δ))

    
    
    
    
    


Under weaker structural assumptions, we show that  enjoys sublinear regret of order Õ(d_E H^3/2√(K)), which is √(H) better than the state-of-the-art distributional framework algorithm  <cit.>.

For the linear MDP setting, we have d_E = Õ(d) and thus the regret bound of  is Õ(√( d^2 H^3 K)) which matches a lower bound Ω(√(d^2 H^3 K)) 
<cit.>.
In our analysis, we highlight two main technical novelties which significantly reduces the complexity of regret;


    
  * We refine previous lemma of <cit.> and <cit.> to remove the dependency of β(ℱ^N, 1/δ) (See Appendix <ref>), ensuring that regret bound depends only on the defined function class, not on the number of moment extracted.

    
  * As shown in Table <ref>, we define the eluder dimension d_E in a finite-dimensional embedding space ℱ^N, while other methods rely on an infinite-dimensional distribution space ℋ⊆ℱ^∞.


































§ CONCLUSIONS


We describe the sources of approximation error inherent in distribution-based updates and introduce a pivotal concept of Bellman unbiasedness, which enables to exactly learn the information of distribution.
We also present a provably efficient distRL algorithm, , with general value function approximation.
Notably, our algorithm achieves a near-optimal regret bound of Õ(d_E H^3/2√(K)), matching the tightest upper bound achieved by non-distributional framework <cit.>.




One interesting future direction would be to reformulate the definition of regret as discrepencies in moments rather than the expected return, and to show the sample-efficiency of distRL.
We hope that our work sheds some light on future research in analyzing the provable efficiency of distRL.




*
plainnat



























*relemmaLemma
*retheoremTheorem






§ APPENDIX









§ NOTATION











§ RELATED WORK AND DISCUSSION





 §.§ Technical Clarifications on Linearity Assumption in Existing Results




  
Bellman Closedness and Linearity.

<cit.> proved that quantile functional is not Bellman closed by providing a specific counterexample.

However, their discussion based on counterexamples can be generalized as it assumes that the sketch Bellman operator for the quantile functional needs to be linear.

They consider an discounted MDP with initial state s_0 with single action a, which transits to one of two terminal states s_1, s_2 with equal probability.
Letting no reward at state s_0, ([0,1]) at state s_1, and ([1/K , 1+ 1/K]) at state s_2, the return distribution at state s_0 is computed as mixture 1/2([0,γ]) + 1/2([γ/K , γ+ γ/K]).
Then the 1/2K-quantile at state s_0 is γ/K.
They proposed a counterexample where each quantile distribution of state s_1, s_2 is represented as 1/K∑_k=1^K δ_2k-1/K and 1/K∑_k=1^K δ_2k+1/K respectively, the 1/2K-quantile of state s_0 is 
ψ_q_2K( 1/2K∑_k=1^K δ_γ(2k-1)/K + δ_γ(2k+1)/K) = 3γ/2K.
However, this example does not consider that the mixture of quantiles is not a quantile of the mixture distribution 
(i.e., ψ_q(λη_1 + (1-λ) η_2) ≠λψ_q(η_1) + (1-λ) ψ_q(η_2)), due to the nonlinearity of the quantile functional. 
Therefore, this does not present a valid counterexample to prove that quantile functionals are not Bellman closed.





  
Bellman Optimizability and Linearity.

<cit.> proposed the notion of Bellman optimizable statistical functional which redefine the Bellman update by planning with respect to statistical functionals rather than expected returns.
They proved that W_1-continuous Bellman Optimizable statistical functionals are characterized by exponential utilities 1/λlog_Z ∼η [exp(λ Z)].
However, their proof requires some technical clarification regarding the assumption that such statistical functionals are linear.


To illustrate, they define a statistical functional ψ_f and consider two probability distributions η_1 = 1/2(δ_0 + δ_h) and η_2 = δ_ϕ(h) where ϕ(h) = f^-1(1/2(f(0) + f(h)) ).
Using the translation property, they lead ψ_f(η_1) = ψ_f(η_2) to
1/2(f(x)+f(x+h)) = f( x+ϕ(h)) for all x ∈ℝ.
However, this equality ψ_f (1/2 (δ_x + δ_x+h) ) = 1/2 (f(x) + f(x+h)) holds only if ψ_f is linear, which is not necessarily a valid assumption for all statistical functionals. 




 §.§ Existence of Nonlinear Bellman Closed Sketch.


The previous two examples may not have considered the possibility that the sketch Bellman operator might not necessarily be linear.
However, some statistical functionals are Bellman-closed even if they are nonlinear, so it is open question whether there is a nonlinear sketch Bellman operator that makes the quantile functional Bellman-closed. 
In this section, we present examples of maximum and minimum functionals that are Bellman-closed, despite being nonlinear.

In a nutshell, consider the maximum of return distribution at state s_1, s_2 is γ, γ + γ/K respectively.
Beyond linearity, the maximum of return distribution at state s_0 can be computed by taking the maximum of these values; 

    max( max(η̅(s_1)), max(η̅(s_2))) = max( γ, γ + γ/K) = γ + γ/K

which produces the desired result.
This implies the existence of a nonlinear sketch that is Bellman closed.
More precisely, by defining max_s' ∼(· | s,a) and min_s' ∼(· | s,a) as the maximum and minimum of the sampled sketch ψ( (ℬ_r)_#η̅(s') ) with the distribution (· | s,a), we can derive the sketch Bellman operator for maximum and minimum functionals as follows;


    
  * 𝒯_ψ_max( ψ_max(η̅(s)) )
    
    = max_s' ∼(· | s,a)( ψ_max((ℬ_r)_#η̅(s')) )
    
    
  * 𝒯_ψ_min( ψ_min(η̅(s)) )
    
    = min_s' ∼(· | s,a)( ψ_min((ℬ_r)_#η̅(s')) ).    










 §.§ Non-existence of sketch Bellman operator for quantile functional


In this section, we prove that quantile functional cannot be Bellman closed under any additional sketch.
First we introduce the definition of mixture-consistent, which is the property that the sketch of a mixture can be computed using only the sketch of the distribution of each component.


    A sketch ψ is mixture-consistent if for any λ∈ [0,1] and any distributions η_1, η_2 ∈𝒫_ψ(ℝ), there exists a corresponding function h_ψ such that
    
    ψ( λη_1 + (1-λ) η_2)
            = h_ψ( ψ(η_1), ψ(η_2), λ).




Next, we will provide some examples of determining whether a sketch is mixture-consistent or not.



  
Example 1.
    Every moment or exponential polynomial functional is mixture-consistent.
    
    For any n ∈ [N] and λ∈ℂ,
    
    _Z ∼λη_1 + (1-λ) η_2[Z^n exp(λ Z)]
            = λ_Z ∼η_1[Z^n exp(λ Z)] + (1-λ) _Z ∼η_2[Z^n exp(λ Z)].

    





  
Example 2.
    Variance functional is not mixture-consistent.
    
    Let λ = 1/2 and Z,Y be the random variables where Z ∼1/2δ_0 + 1/2δ_2 and Y ∼1/2δ_k + 1/2δ_k+2.
    Then, Var(Z) = Var(Y)=1.
    While RHS is constant for any k,
    LHS is not a constant for any k, i.e.,
    
    Var_X ∼1/2 (1/2δ_0 + 1/2δ_2) + 1/2 (1/2δ_k + 1/2δ_k+2) (X)
            
            
            = 1/4(k^2 + 5).

    
    
While variance functional is not mixture consistent by itself, it can be mixture consistent with another statistical functional, the mean.




  
Example 3.
    Variance functional is mixture-consistent under mean functional.
    
        Notice that mean functional is mixture-consistent. We need to show that variance functional is mixture-consistent under mean functional.

    
    Var_Z ∼λη_1 + (1-λ) η_2[Z]
            
        = _Z ∼λη_1 + (1-λ) η_2[Z^2] - (_Z ∼λη_1 + (1-λ) η_2[Z])^2 
            
        = λ_Z ∼η_1[Z^2] + (1-λ) _Z ∼η_2[Z^2] - (λ_Z ∼η_1[Z] + (1-λ) _Z ∼η_2[Z])^2
            
        = λ (Var_Z ∼η_1[Z] + (_Z∼η_1[Z])^2 ) + (1-λ) (Var_Z ∼η_2[Z] + (_Z∼η_2[Z])^2) 
            
            - (λ_Z ∼η_1[Z] + (1-λ) _Z ∼η_2[Z])^2.

    

This means that to determine whether it is mixture-consistent or not, we should check it on a per-sketch basis, rather than on a per-statistical functional basis.



  
Example 4.
    Maximum and minimum functional are both mixture-consistent.
    
    
    max_Z ∼λη_1 + (1-λ) η_2[Z] = max(max_Z ∼η_1[Z] , max_Z ∼η_2[Z])

    and
    
    min_Z ∼λη_1 + (1-λ) η_2[Z] = min(min_Z ∼η_1[Z] , min_Z ∼η_2[Z])

    

Since maximum and minimum functionals are mixture consistent, we can construct a nonlinear sketch bellman operator like the one in section <ref>.
This is possible because there is a nonlinear function h_ψ that ensures the sketch is closed under mixture.

Before demonstrating that a quantile sketch cannot be mixture consistent under any additional sketch, we will first illustrate with the example of a median functional that is not mixture consistent.



  
Example 5.
    Median sketch is not mixture-consistent.
    
    
    
    
    
    
    
    
    
    
    Let λ = 1/2 and Z,Y be the random variables where Z ∼ 0.2 δ_0 + 0.8 δ_1 and Y ∼ 0.6 δ_0 + 0.4 δ_k for some 0<k<1. 
    Then ψ_med(Z) = 1 and ψ_med(Y) = 0.
    However,
    
    med_X = Z+Y/2[X] = ψ_med(0.4 δ_0 + 0.2 δ_k + 0.4 δ_1)
            = k

    which is dependent in k.
    






    Quantile sketch cannot be mixture-consistent, under any additional sketch.    




    For a given integer N>0 and a quantile level α∈ [0,1], 
    let λ = 1/2 and a random variable Y ∼ p_y_0δ_0 + p_y_1δ_y_1 + ⋯ + p_y_Nδ_y_N (0 < y_1 < ⋯ < y_N < 1) where p_y_0 >α so that ψ_q_α(Y) = 0.
    Consider another random variable Z ∼ p_z_0δ_0 + p_z_1δ_1 where p_z_0 < α so that ψ_q_α(Z) = 1.
    Then the α-quantile of the mixture X = Y+Z/2 is 
    
    ψ_q_α[X] = y_n  where  n = min{ n ≤ N |  1/2∑_n'=0^n p_y_n' + 1/2p_z_0 > α}.

    Letting p_z_0 = 2 α - ∑_n'=0^n p_y_n', we can manipulate ψ_q_α[X] to be any value of y_n.
    Hence, ψ(q_α)[X] is a function of all possible outcomes of Y.

    If there exists a finite number of statistical functionals which make quantile sketch mixture-consistent, then such sketch would uniquely determine the distribution for any N.
    This results in a contradiction that infinite-dimensional distribution space can be represented by a finite number of statistical functional.







    If a sketch ψ is Bellman closed, then it is mixture-consistent.


    
        
        
        
        
        
        
        
        Consider an MDP where initial state s_0 has no reward and transits to two state s_1, s_2 with probability λ, 1-λ and reward distribution η̅_1, η̅_2.
        Since ψ is Bellman closed, ψ(η̅(s_0)) is a function of ψ(η̅(s_1)) and ψ(η̅(s_2)), (i.e., ψ(η̅(s_0)) = g_ψ(ψ(η̅(s_1)), ψ(η̅(s_2))) for some g_ψ).        
        
        
        
        
        
        
        
        Since ψ(η̅(s_0)) = ψ(λη̅(s_1) + (1-λ) η̅(s_2)), it implies that ψ is mixture-consistent.
    

Combining the results of Lemma <ref> and Lemma <ref>, we prove that a quantile sketch cannot be Bellman closed, no matter what additional sketches are provided.







§ PROOF




[<ref>]
    Quantile functional cannot be Bellman closed under any additional sketch.



    See Lemma <ref> and Lemma <ref>.




[<ref>]
    Let F_η̅ be a CDF of the probability distribution η̅∈𝒫(ℝ)^𝒮. 
    Then a sketch is Bellman unbiased if and only if the sketch is a homogeneous of degree k, i.e.,
    there exists some vector-valued function h = h(x_1, ⋯, x_k): 𝒳^k →ℝ^N such that
    
    ψ(η̅) =  ∫⋯∫ h(x_1, ⋯, x_k) dF_η̅(x_1) ⋯ dF_η̅(x_k).





    (⇒) 
    Consider an two-stage MDP with a single action a, and an initial state s_0 which transits to one of terminal state {s_1, ⋯, s_K } with transition kernel (· | s_0 ,a). Assume that the reward r(s_0) = 0. 
    Then η̅(s_0) = ∑_k=1^K (s_k) δ_r(s_k). 
    Note that s'_1 , ⋯, s'_k are independent and identically distributed random variable in distribution (·| s,a).

    
    _s'∼(· |s_0,a)[ 
            ϕ_ψ( ψ( (ℬ_r)_#η̅(s'_1) ), ⋯, ψ( (ℬ_r)_#η̅(s'_k))
            ) ]
            = ψ_1:N( (ℬ_r)_#_s' ∼(· | s_0,a) [ η̅(s') ])
            
       ⟹_s' ∼(· | s_0,a)[ ϕ_ψ( ψ( δ_r(s'_1)) , ⋯ , ψ( δ_r(s'_k))  ) ]
            = ψ( _s' ∼(· | s_0,a) [ δ_r(s') ])
            
       ⟹_s' ∼(· | s_0,a)[ ϕ_ψ(
            g(s'_1) , ⋯ , g(s'_k) 
            ) ]
            = ψ( η̅(s_0) )
            
       ⟹∫⋯∫ h(s'_1, ⋯ , s'_k) dF_η̅(s'_1) ⋯ dF_η̅(s'_k)
            = ψ( η̅(s_0) ).


    (⇐)
    
    ψ((ℬ_r)_#_s' ∼(· | s,a)[η̅(s')] )
        
        = ∫⋯∫ h(x_1, ⋯, x_k)dF_(ℬ_r)_#_s' ∼(· | s,a)[η̅(s')] (x_1) ,⋯, dF_(ℬ_r)_#_s' ∼(· | s,a)[η̅(s')](x_k)
        = ∫⋯∫ h(x_1 + r, ⋯, x_k + r) d(_s' ∼(·| s,a)F_η̅(s')(x_1) ) , ⋯, d(_s' ∼(·| s,a)F_η̅(s')(x_k) )
        
        = _s' ∼(· | s,a)[ ∫⋯∫ h(x_1 +r, ⋯, x_k+r) dF_η̅(s')(x_1) ⋯ dF_η̅(s')(x_k) ]
        
        = _s' ∼(·|s,a)[ ψ( (ℬ_r)_#[η̅(s')]
        ) ]


























































[<ref>]

    The only finite statistical functionals that are Bellman unbiased and closed are given by the collections of ψ_1, …,  ψ_N where its linear span {∑_n=0^N α_n ψ_n | α_n ∈ℝ ,∀ N} is equal to the set of exponential polynomial functionals {η→_Z ∼η[Z^l exp(λ Z)] |   l = 0, 1, …, L, λ∈ℝ}, where ψ_0 is the constant functional equal to 1.
    In discount setting, it is equal to the linear span of the set of moment functionals {η→_Z ∼η[Z^l] |  l = 0, 1, …, L } for some L ≤ N.  







    Our proof is mainly based on the proof techniques of <cit.> and we describe in an extended form.
    Since their proof also considers the discounted setting, we will define ℬ_r,γ(x) = r + γ x for discount factor γ∈ [0,1).
    By assumption of Bellman closedness,
    ψ_n( (ℬ_r,γ)_#η̅(s') ) will be written as g(r, γ, ψ_1:N(η̅(s')) for some g.
    By assumption of Bellman unbiasedness and Lemma <ref>,
    both ψ_1:N(η̅(s')) and ψ_n( (ℬ_r, γ)_#η̅(s') ) are affine as functions of the distribution η̅(s'),
    
    ψ_1:N( αη̅_1(s') + (1-α) η̅_2(s') )
            
        = 𝔼_Z_i ∼αη̅_1(s') + (1-α) η̅_2(s') [h_1:N(Z̅_1, ⋯, Z̅_k)]
            
        = α𝔼_Z̅_i ∼η̅_1(s') [h_1:N(Z̅_1, ⋯, Z̅_k)] + (1-α )𝔼_Z̅_i ∼η̅_2(s') [h_1:N(Z̅_1, ⋯, Z̅_k)]
            
        = αψ_1:N(η̅_1(s')) + (1-α) ψ_1:N(η̅_2(s'))

    and
    
    ψ_n( (ℬ_r,γ)_# (αη̅_1(s') + (1-α) η̅_2(s')) )
            
        = 𝔼_Z_i ∼αη̅_1(s') + (1-α) η̅_2(s') [h_n( r+ γZ̅_1, ⋯, r+ γZ̅_k)]
            
        = α𝔼_Z̅_i ∼η̅_1(s') [h_n(r+ γZ̅_1, ⋯, r+ γZ̅_k)] + (1-α )𝔼_Z̅_i ∼η̅_2(s') [h_n(r+ γZ̅_1, ⋯, r+ γZ̅_k)]
            
        = αψ_n((ℬ_r,γ)_#η̅_1(s') ) + (1-α) ψ_n( (ℬ_r,γ)_#η̅_2(s') )

    Therefore, g(r, γ , ·) is also affine on the convex codomain of ψ_1:N.
    
    Thus, we have
    
    _Z̅_i ∼η̅[ϕ_ψ_n(r+ γZ̅_1, ⋯, r+ γZ̅_k)] 
        = a_0(r, γ) + ∑_n'=1^N a_n'(r, γ) _Z̅_i ∼η̅[ϕ_ψ_n'(Z̅_1, ⋯, Z̅_k)]

    for some function a_0:N : ℝ× [0,1] →ℝ.
    By taking η̅(s') = δ_x, we obtain
    
    ϕ_ψ_n(r+ γ x, ⋯, r+ γ x) = a_0(r, γ) + ∑_n'=1^N a_n'(r, γ ) ϕ_ψ_n'(x, ⋯, x).

    According to <cit.>, for any translation invariant finite-dimensional space is spanned by a set of function of the form
    
    { x ↦ x^l exp(λ_j x) | j ∈ [J], 0 ≤ l ≤ L }

    for some finite subset {λ_1, ⋯, λ_J } of ℂ.
    Hence, each function x ↦ϕ_ψ_n(x, ⋯, x) is expressed as linear combination of exponential polynomial functions.
    In addition, the linear combination of ϕ_ψ_n should be closed under composition with for any discount factor γ∈ [0,1], all λ_j should be zero. 
    Hence, the linear combination of ϕ_ψ_1, ⋯, ϕ_ψ_N must be equal to the span of { x ↦ x^l | 0 ≤ l ≤ L } for some L ∈ℕ.
















[<ref>]
Consider a fixed k ∈ [K] and a fixed h ∈ [H]. Let 𝒵_h^k = { (s_h^τ , a_h^τ)}_τ∈ [k-1] and 

𝒟^k_h, η̅ = {( s_h^τ , a_h^τ,  ψ_1:N( (ℬ_r_h'^τ)_#η̅(s_h'+1^τ) )) }_τ∈ [k-1] for any η̅ : 𝒮→𝒫([0,H]).
Define f̃_h, η̅^k = min_f ∈ℱ^N f ^2_𝒟^k_h, η̅.
For any η̅ and δ∈ (0,1), there is an event ℰ(η̅, δ) such that conditioned on ℰ(η̅, δ), with probability at least 1-δ, for any η̅' : 𝒮→𝒫([0,H]) with 
ψ_1:N( η̅') - ψ_1:N(η̅) _∞,1≤ 1/T or ∑_n=1^N ψ_n (η̅')- ψ_n(η̅) _∞≤ 1/T, we have

    f̃_h,η̅'(· , ·) - ψ_1:N( (ℬ_r(·, ·))_#  [ℙη̅'](· , ·) ) _𝒵_h^k   ≤c' (N^1/2 H √(log(1/δ) + log𝒩(ℱ^N, 1/T)  ))

for some constant c' >0.





    Define the sketch of target f_η̅ : 𝒮×𝒜→ℝ^N,
    
    f_η̅(· , ·) ψ_1:N( (ℬ_r(·, ·))_#  [ℙη̅](· , ·) )

    for all i ∈ [N].

    
    
    
    
    
    
    
    
    
    For any f ∈ℱ,
    
    f ^2_𝒟^k_h,η̅' -  f_η̅'^2_𝒟^k_h,η̅'
       = ∑_n=1^N ∑_s_h^τ,a_h^τ∈𝒵^k_h,η̅'( f^(n)(s_h^τ, a_h^τ) - ψ_n ( (ℬ_r_h^τ)_#η̅'(s_h+1^τ) ) )^2 - 
            ( f_η̅'^(n)(s_h^τ, a_h^τ) - ψ_n ( (ℬ_r_h^τ )_#η̅'(s_h+1^τ) ) )^2
            
       = ∑_n=1^N ∑_ s_h^τ,a_h^τ∈𝒵^k_h,η̅' 
            ( f^(n)( s_h^τ,a_h^τ ) - f_η̅'^(n)( s_h^τ,a_h^τ ) )^2
            
            + 2 ( f^(n)( s_h^τ,a_h^τ ) - f_η̅'^(n)( s_h^τ,a_h^τ ) ) 
            ( f^(n)_η̅'( s_h^τ,a_h^τ ) - ψ_n ( (ℬ_r_h^τ)_#η̅'(s_h+1^τ) ) )
            
       ≥f - f_η̅'^2_𝒵_h^k- 4  ∑_n=1^N   f_η̅^(n) - f_η̅'^(n)_∞ (H+1)|𝒵_h^k|
            + 
            ∑_n=1^N ∑_ s_h^τ,a_h^τ∈𝒵^k_h,η̅'[ 2 ( f^(n)( s_h^τ,a_h^τ ) - f_η̅^(n)( s_h^τ,a_h^τ ) ) 
            ( f^(n)_η̅( s_h^τ,a_h^τ ) - ψ_n ( (ℬ_r_h^τ)_#η̅(s_h+1^τ) ) )_χ_h^τ(f^(n))]
            
       ≥f - f_η̅'^2_𝒵_h^k
            - 4 N(H+1) 
            - | ∑_n=1^N ∑_s_h^τ, a_h^τ∈𝒵^k_h, η̅'χ^τ_h(f^(n)) |.


    For the first inequality, we change the second term  from η̅' to η̅ which are the ϵ-covers.
    Notice that AC -BC' ≥ -|AC -BC'| ≥ -|(A-B)C| -|(A-B)C'| ≥ -2|A-B| |max(C,C')|.
    
    ( f^(n)( s_h^τ,a_h^τ ) - f_η̅'^(n)( s_h^τ,a_h^τ ) ) 
            ( f^(n)_η̅'( s_h^τ,a_h^τ ) - ψ_n ( (ℬ_r_h^τ)_#η̅'(s_h+1^τ) ) )
            
            - ( f^(n)( s_h^τ,a_h^τ ) - f_η̅^(n)( s_h^τ,a_h^τ ) ) 
            ( f^(n)_η̅( s_h^τ,a_h^τ ) - ψ_n ( (ℬ_r_h^τ)_#η̅(s_h+1^τ) ) )
            
       ≥ - 2  f_η̅'^(n)( s_h^τ,a_h^τ ) - f_η̅^(n)( s_h^τ,a_h^τ ) 
           ×max( 
            | f^(n)_η̅'( s_h^τ,a_h^τ ) - ψ_n ( (ℬ_r_h^τ)_#η̅'(s_h+1^τ) ) |,
            | f^(n)_η̅( s_h^τ,a_h^τ ) - ψ_n ( (ℬ_r_h^τ)_#η̅(s_h+1^τ) ) |
            )
            
       ≥ -2   f_η̅'^(n)( s_h^τ,a_h^τ ) - f_η̅^(n)( s_h^τ,a_h^τ )  (H+1)

    
    For the second inequality, consider η̅' : 𝒮→𝒫([0,H]) with ∑_n=1^N ψ_n( η̅' ) - ψ_n ( η̅ ) _∞≤ 1/T. 
    We have
    
    f^(n)_η̅ - f^(n)_η̅'_∞   = max_s,a| ∑_n'=1^n H^n' [ ψ_n' ( [η̅](s,a) ) - ψ_n' ([η̅'](s,a) ) ]r^n-n'/H^n-1|
            
       ≤∑_n'=1^n max_s'| ψ_n'( η̅(s') ) -ψ_n' ( η̅'(s') ) |
            
       ≤ 1/T.

    
    Defining 𝔽_h^k as the filtration induced by the sequence
{ (s_h'^τ ,a_h'^τ)}_τ, h' ∈ [k-1] × [H]∪{(s_1^k , a_1^k), (s_2^k, a_2^k), …, (s_h^k , a_h^k) },
    notice that
    
    [ ∑_n=1^N χ_h^τ(f^(n)) | 𝔽_h^τ]
            
       =∑_n=1^N 2 (f^(n)( s_h^τ,a_h^τ ) - f_η̅^(n)( s_h^τ,a_h^τ )) (f^(n)_η̅(s_h^τ, a_h^τ) - [ ψ_n ((ℬ_r_h^τ)_#η̅(s_h+1^τ) ) | 𝔽_h^τ])
            
       =∑_n=1^N 2  (f^(n)( s_h^τ,a_h^τ ) - f_η̅^(n)( s_h^τ,a_h^τ )) (f^(n)_η̅(s_h^τ, a_h^τ) - _s_h+1^τ∼_h(· | s_h^τ, a_h^τ )[ ψ_n ((ℬ_r_h^τ)_#η̅(s_h+1^τ) ) ])
            
       =∑_n=1^N 2 (f^(n)( s_h^τ,a_h^τ ) - f_η̅^(n)( s_h^τ,a_h^τ )) (f^(n)_η̅(s_h^τ, a_h^τ) -  ψ_n ((ℬ_r_h^τ)_#_s_h+1^τ∼_h(· | s_h^τ, a_h^τ )[ η̅(s_h+1^τ)] ) )
       =0

    and
    
    | ∑_n=1^N χ_h^τ(f^(n)) |
                = | ∑_n=1^N 2 (f^(n)( s_h^τ,a_h^τ ) - f_η̅^(n)( s_h^τ,a_h^τ )) (f_η̅^(n)(s_h^τ, a_h^τ) - ψ_n ( (ℬ_r_h^τ)_#η̅(s_h+1^τ) )  )|
            
            
       ≤max_n ∈ [N]{ 2 (f_η̅^(n)(s_h^τ, a_h^τ) - ψ_n ( (ℬ_r_h^τ)_#η̅(s_h+1^τ) )  ) }∑_n=1^N | f^(n)( s_h^τ,a_h^τ ) - f_η̅^(n)( s_h^τ,a_h^τ)  |
            
       ≤ 2(H+1) ∑_n=1^N | f^(n)( s_h^τ,a_h^τ ) - f_η̅^(n)( s_h^τ,a_h^τ)  |



    
    
    
    
    
    
    
  
    In third equality, we emphasize that only Bellman unbiased sketch can derive the martingale difference sequence which induce the concentration result.
    Since every moment functional is commutable with mixing operation, the transformation ϕ_ψ_n in Definition <ref> is identity for all n ∈ [N].
    Hence, we choose the sketch as moment which already knows ϕ_ψ.
    
    By Azuma-Hoeffding inequality,
    
    [ | ∑_(τ, h) ∈ [k-1] × [H] ∑_n=1^N χ_h^τ (f^(n)) | ≥ϵ]
               ≤ 2 exp( -ϵ^2/2(2(H+1))^2∑_(τ, h) ∈ [k-1] × [H]( ∑_n=1^N |f^(n) - f_η̅^(n)|)^2 )
            
       ≤ 2 exp( -ϵ^2/2(2(H+1))^2∑_(τ, h) ∈ [k-1] × [H]( N ∑_n=1^N |f^(n) - f_η̅^(n)|^2) )
            
        = 2 exp( -ϵ^2/ 2N(2(H+1))^2 f-f_η̅_𝒵_h^k^2 )

    where the second inequality follows from the Cauchy-Schwartz inequality.

    We set
    
    ϵ = √(8N(H+1)^2  f- f_η̅^2_𝒵_h^klog( 𝒩(ℱ^N, 1/T) /δ) )


    With union bound for all f ∈𝒞(ℱ^N , 1/T), with probability at least 1-δ,
    
    | ∑_(τ, h) ∈ [k-1] × [H] ∑_n=1^N χ_h^τ (f^(n)) | 
            ≤ c' N^1/2(H+1)  f- f_η̅_𝒵_h^k√(log( 𝒩(ℱ^N, 1/T) /δ))

    for some constant c'>0.

    
    For all f ∈ℱ^N, there exists g ∈𝒞(ℱ^N , 1/T), such that  f-g_∞,1≤ 1/T or ∑_n=1^N f^(n)-g^(n)_∞≤ 1/T for all n ∈ [N],
    
    | ∑_(τ, h) ∈ [k-1] × [H] ∑_n=1^N χ_h^τ (f^(n)) | 
               ≤| ∑_(τ, h) ∈ [k-1] × [H] ∑_n=1^N χ_h^τ (g^(n)) | + 2(H+1) |𝒵^k_h| ∑_n=1^N 1/T
       ≤ c' N^1/2 (H+1) g-f_η̅_𝒵^k_h√(log( 𝒩(ℱ^N,1/T)/δ)) + 2 N (H+1)
            
       ≤ c' N^1/2 (H+1)(f-f_η̅_𝒵^k_h +1) √(log( 𝒩(ℱ^N,1/T )/δ)) + 2 N (H+1)
            
       ≤ c' N^1/2 (H+1)(f-f_η̅'_𝒵^k_h +2) √(log( 𝒩(ℱ^N,1/T )/δ)) + 2 N (H+1)

    where the third inequality follows from,
    
    f- g _𝒵_h^k^2 
               ≤∑_n=1^N ∑_(τ, h) ∈ [k-1] × [H] |f^(n)(s_h^τ, a_h^τ) -g^(n)(s_h^τ, a_h^τ)|^2
            
       ≤ N T (1/T)^2
            
       ≤ 1.




    Recall that f̃^k_h,η' = min_f ∈ℱf^2_𝒟^k_h,η'.
    We have f̃^k_h,η'^2_𝒟^k_h,η' - f_η̅'^2_𝒟^k_h,η'≤ 0, which implies,
    
    0    ≥f̃^k_h,η̅'^2_𝒟^k_h,η̅' -  f_η̅'^2_𝒟^k_h,η̅'
       = f̃^k_h,η̅' - f_η̅'^2_𝒵^k_h
            + 2 ∑_n=1^N ∑_(τ, h) ∈ [k-1] × [H][( ( f̃^k_h, η̅')^(n)(s_h^τ, a_h^τ) - f^(n)_η̅'(s_h^τ, a_h^τ) )( ( f_η̅'^(n)(s_h^τ, a_h^τ) - ψ_n ((ℬ_r_h^τ)_#η̅'(s^τ_h+1) ) )]
            
       ≥f̃^k_h,η̅' - f_η̅'^2_𝒵^k_h 
            - c' N^1/2 (H+1) (f̂^k_h,η̅' - f_η̅'_𝒵^k_h +2) √(log(2/δ) + log𝒩(ℱ^N, 1/T)) - 6N(H+1).


    Recall that if x^2 -2ax -b ≤ 0 holds for constant a,b>0, then x ≤ a+ √(a^2 + b)≤ c' · a for some constant c'>0.
    
    Hence,
    
    f̃^k_h, η' - f_η̅'_𝒵^k_h≤ c' (N^1/2 H √(log (1/δ) + log𝒩(ℱ^N, 1/T)) )

    for some constant c' >0.








[<ref>]
    Let (ℱ^N)_h^k = { f ∈ℱ^N |  f - f̃^k_h, η̅^2_𝒵^k_h≤β(ℱ^N, δ) }, where
    
    β(ℱ^N,δ) ≥ c' · NH^2 (log (T/δ) + log𝒩(ℱ^N , 1/T) )

    for some constant c'>0.
    Then with probability at least 1- δ/2, for all k,h ∈ [K] × [H], we have
    
    ψ_n ( (ℬ_r_h(·, ·))_#[_h η̅_h+1^k](·, ·) ) ∈ (ℱ^N)_h^k




    For all (k,h) ∈ [K] × [H],
    
    
    
    
    
    
    𝐒{(min{ f^(1)(· , ·) 
        + b^k_h+1(·, ·) 
        , H }) |  f ∈𝒞(ℱ^N , 1/T) }∪{0 }    n=1 
    {(min{ f^(n)(· , ·) 
        , H }) | f ∈𝒞(ℱ^N , 1/T) }∪{0 }    2 ≤ n ≤ N

    
    
    
    is a (1/T)-cover of ψ_1:N (η^k_h+1(·, ·)) where
    
    ψ_1:N ( η^k_h+1(·, ·) ) 
        = min{( f^k_h+1)^(1)(· , ·) 
        + b^k_h+1(·, ·) 
        , H }    n=1  and  h<H 
    min{( f^k_h+1)^(n)(· , ·) 
        , H }    2 ≤ n ≤ N  and  h<H 
    0^N     h=H
        
        ,
 
    
    i.e., there exists ψ_1:N (η) ∈𝐒 such that ψ_1:N(η) - ψ_1:N(η^k_h+1) _∞,1≤ 1/T.
    This implies
    
    𝐒̅{ψ_1:N( η(·, max_a ∈𝒜ψ_1 (η(·, a)) ) ) |ψ_1:N( η ) ∈𝐒}

    is a (1/T)-cover of ψ_1:N( η̅^k_h+1) with log(|𝐒̅|) ≤log𝒩(ℱ^N , 1/T).

    For each ψ_1:N( η̅) ∈𝐒̅, let ℰ(η̅ , δ/2|𝐒̅|T) be the event defined in Lemma <ref>.
    By union bound for all ψ_1:N (η̅) ∈𝐒̅, we have Pr[⋂_ψ_1:N(η̅) ∈𝐒̅ℰ(η̅, δ / 2|𝐒̅|T)] ≥ 1- δ/2T.

    Let ψ_1:N(η̅) ∈𝐒̅ such that ψ_1:N( η̅ ) - ψ_1:N( η̅_h+1^k ) _∞,1≤ 1/T.
    Conditioned on ⋂_s_N(η̅) ∈𝐒̅ℰ(η̅, δ / 2| 𝐒̅|T) and by Lemma <ref>, we have
    
    f̃^k_h,η̅ (·,·) - ψ_1:N( (ℬ_r_h(·,·))_# [_h η̅_h+1^k](·,·)) ^2_𝒵^k_h≤ c' ( N H^2 (log (T/δ) + log𝒩(ℱ^N , 1/T) ) )

    for some constant c' >0.
    
    By union bound for all (k,h) ∈ [K] × [H], we have ψ_1:N((ℬ_r_h(·,·))_#[_h η̅_h+1^k](·, ·)) ∈ (ℱ^N)_h^k with probability 1-δ/2.











    Let Q^k_h(s,a) min{ H, f̃^k_h(s,a) + b^k_h(s,a) } for some bonus function b^k_h(s,a) for all (s,a) ∈𝒮×𝒜. 
    If b^k_h(s,a) ≥ w^(1)((ℱ^N)^k_h ,s,a) ,
    then with probability at least 1-δ/2,  
    
    Q^*_h (s, a ) ≤ Q^k_h (s, a) and  V^*_h (s) ≤ V^k_h (s)

    for all (k,h) ∈ [K]×[H], for all (s,a) ∈𝒮×𝒜.










 
    We use induction on h from h=H to 1 to prove the statement.
    Let ℰ be the event that for (k,h) ∈ [K] × [H], ψ_1:N( (ℬ_r_h (· , ·))_# [_h η̅^k_h+1](·,·) ) ∈ (ℱ^N)^k_h.
    By Lemma <ref>, Pr|ℰ| ≥ 1-δ/2.
    In the rest of the proof, we condition on ℰ.
    
    When h=H+1, the desired inequality holds as Q^*_H+1(s,a) = V^*_H+1(s) = Q^k_H+1(s,a) = V^k_H+1(s)=0.
    Now, assume that Q^*_h+1(s,a) ≤ Q^k_h+1(s,a) and V^*_h+1(s) ≤ V^k_h+1(s) for some h ∈ [H].
    Then, for all (s,a) ∈𝒮×𝒜,
    
    Q^*_h(s,a)     = min{ H, r_h(s,a) + [_h V^*_h+1](s,a) }
       ≤min{ H, r_h(s,a) + [_h V^k_h+1](s,a) }
       ≤min{ H, f̃^k_h(s,a) + w^(1)(ℱ^k_h , s, a) }
        = min{ H, Q^k_h(s,a) - b^k_h(s,a) + w^(1)(ℱ^k_h , s, a) }
       ≤ Q^k_h(s,a)








    With probability at least 1-δ/4, we have
    
    
    Reg(K) ≤∑_k=1^K ∑_h=1^H (2 b^k_h(s^k_h, a^k_h) + ξ^k_h) ,

    where ξ^k_h = [_h (V^k_h+1 - V^π^k_h+1)](s^k_h, a^k_h) -(V^k_h+1(s^k_h+1) - V^π^k_h+1(s^k_h+1) ) is a martingale difference sequence with respect to the filtration 𝔽^k_h induced by the history ℍ^k_h.












    We condition on the above event ℰ in the rest of the proof.
    For all (k,h) ∈ [K] × [H], we have 
    
    f̃^k_h,η̅ (·,·) -ψ_1:N( (ℬ_r_h(·,·))_# [_h η̅_h+1^k](·,·)) ^2_𝒵^k_h≤β(ℱ^N , δ).

    Recall that (ℱ^N)_h^k = { f ∈ℱ^N | f - f̃^k_h,η̅^2_𝒵^k_h≤β(ℱ^N, δ) } is the confidence region.
    Since ψ_1:N( (ℬ_r_h(·,·))_# [_h η̅_h+1^k](·,·) ) ∈ (ℱ^N)^k_h, then by the definition of width function w^(1)(ℱ^k_h , s,a), for (k,h) ∈ [K] × [H], we have
    
    w^(1)(ℱ^k_h , s,a)
               ≥| ψ_1((ℬ_r_h(s,a))_#[_h η̅_h+1^k](s,a) ) - (f̃^k_h,η̅)^(1)(s,a)|
            
        = | r_h(s,a) + [_h V^k_h+1](s,a) - (f̃^k_h,η̅)^(1)(s,a) |.

    
    
    
    
    
    
    Recall that Q^*_h (·, · ) ≤ Q^k_h (·, ·).
    
    Reg(K)    = ∑_k=1^K V_1^⋆(s_1^k) - V_1^π^k(s_1^k)
            
       ≤∑_k=1^K V_1^k(s_1^k) - V_1^π^k(s_1^k)
            
        = ∑_k=1^K Q_1^k(s_1^k , a_1^k) - Q_1^π^k(s_1^k, a_1^k)
            
        = ∑_k=1^K Q_1^k(s_1^k , a_1^k) - (r_1(s_1^k, a_1^k) 
            + [_1 V_2^k](s_1^k, a_1^k)) + (r_1(s_1^k, a_1^k) + [_1 V_2^k](s_1^k, a_1^k))
            
           - Q_1^π^k(s_1^k, a_1^k)
            
       ≤∑_k=1^K w^(1)((ℱ^N)^k_1, s_1^k, a_1^k)  + b^k_1(s_1^k, a_1^k) + [_1 (V_2^k - V_2^π^k)](s_1^k, a_1^k)
            
       ≤∑_k=1^K w^(1)((ℱ^N)^k_1, s_1^k, a_1^k)  + b^k_1(s_1^k, a_1^k) + (V_2^k(s_2^k) - V_2^π^k(s_2^k) ) + ξ_1^k 
            
           ⋮
       ≤∑_k=1^K ∑_h=1^H ( w^(1)((ℱ^N)^k_h, s_h^k, a_h^k)  + b^k_h(s_h^k, a_h^k) + ξ_h^k )
            
       ≤∑_k=1^K ∑_h=1^H (2 b^k_h(s_h^k, a_h^k) + ξ_h^k )






It remains to bound ∑_k=1^K ∑_h=1^H b^k_h(s^k_h, a^k_h), for which we will exploit fact that ℱ^N has bounded eluder dimension.









    If b^k_h(s,a) ≥ w^(1)((ℱ^N)^k_h, s,a ) for all (s,a) ∈𝒮×𝒜 and k ∈ [K] where 
    
    (ℱ^N)^k_h = { f ∈ℱ^N |  f-f̃^k_h, η̅^2_𝒵^k_h≤β(ℱ^N, δ)},
 then
    
    
    
    
    
    ∑_k=1^K ∑_h=1^H 1{b^k_h( s^k_h,a^k_h) > ϵ}≤(4β(ℱ^N, δ)/ϵ^2 +1) dim_E(ℱ^N ,ϵ)

    for some constant c>0. 



























We first want to show that for any sequence {(s_1, a_1), … ,(s_κ, a_κ) }⊆𝒮×𝒜,
there exists j ∈ [κ] such that (s_j,a_j) is ϵ-dependent on at least L = ⌈ (κ-1)/dim_E(ℱ^N, ϵ) ⌉ disjoint subsequences in {(s_1, a_1), …, (s_j-1, a_j-1) } with respect to ℱ^N. 
We demonstrate this by using the following procedure.
Start with L disjoint subsequences of { (s_1, a_1), …, (s_j-1, a_j-1) }, ℬ_1, ℬ_2, …, ℬ_L, which are initially empty.
For each j, if (s_j , a_j) is ϵ-dependent on every ℬ_1, …, ℬ_L, we achieve our goal so we stop the process.
Else, we choose i ∈ [L] such that (s_j, a_j) is ϵ-independent on ℬ_i and update ℬ_i ←ℬ_i ∪{(s_j,a_j)}, j ← j+1.
Since every element of ℬ_i is ϵ-independent on its predecessors, |ℬ_i| cannot get bigger than dim_E(ℱ^N, ϵ) at any point in this process.
Therefore, the process stops at most step j = L dim_E(ℱ^N, ϵ) + 1 ≤κ.

Now we want to show that if for some j ∈ [κ] such that 

b^k_h(s_j , a_j) > ϵ,
then (s_j , a_j) is ϵ-dependent on at most 4β(ℱ^N , δ)/ϵ^2 disjoint subsequences in {(s_1, a_1), …, (s_j-1, a_j-1) } with respect to ℱ^N.
If 

b^k_h(s_j , a_j) > ϵ
and (s_j, a_j) is ϵ-dependent on a subsequence of {(s'_1, a'_1), …, (s'_l, a'_l)}⊆{(s_1, a_1), …, (s_κ, a_κ)}, 
it implies that there exists f,g ∈ℱ^N with f - f̃^k_h,η̅^2_𝒵^k_h≤β(ℱ^N,δ) and g - f̃^k_h,η̅^2_𝒵^k_h≤β(ℱ^N,δ) such that


f^(1)( s'_t , a'_t ) - g^(1)( s'_t , a'_t )  ≥ϵ.
By triangle inequality, f - g ^2_𝒵^k_h≤ 4 β(ℱ^N,δ).
On the other hand, if (s_j, a_j) is ϵ-dependent on L disjoint subsequences in {(s_1, a_1), …, (s_κ, a_κ)}, then

    4β(ℱ^N , δ ) 
        
        ≥ f-g ^2_𝒵^k≥f^(1) - g^(1)^2_𝒵^k≥ L ϵ^2

resulting in L ≤ 4 β(ℱ^N, δ)/ϵ^2.
Therefore, we have (κ/ dim_E(ℱ^N, ϵ))-1 ≤ 4β(ℱ^N, δ)/ϵ^2 which results in

    κ≤( 4β(ℱ, δ)/ϵ^2+1) dim_E(ℱ^N, ϵ)






























































    
    If b^k_h(s,a) ≥ w^(1)((ℱ^N)^k_h, s,a ) for all (s,a) ∈𝒮×𝒜 and k ∈ [K], then
    
    ∑_k=1^K ∑_h=1^H b^k_h(s^k_h, a^k_h ) 
        ≤ H dim_E(ℱ^N, 1/T).











    We first sort the sequence { b^k_h(s^k_h, a^k_h ) }_(k,h) ∈ [K]×[H] in a decreasing order and denote it by {e_1 , …, e_T } (e_1 ≥ e_2 ≥…≥ e_T).
    
    
    
    
    
    By Lemma <ref>, for any constant M >0 and e_t ≥ 1/√(M) T, we have
    
    t ≤( 4β(ℱ^N,δ)/M e_t^2 +1)dim_E(ℱ^N, √(M) e_t)
            ≤( 4β(ℱ^N,δ)/M e_t^2 +1)dim_E(ℱ^N, 1/T)

    which implies
    
    e_t ≤( t/dim_E(ℱ^N , 1/T)  -1 )^-1/2√(4 β(ℱ^N, δ)/M),

    for t ≥dim_E(ℱ^N ,1/T).
    Since we have e_t ≤ H,
    
    ∑_t=1^T e_t 
                = ∑_t=1^T e_t 1{ e_t < 1/√(M)T } 
            + ∑_t=1^T e_t 1{ e_t ≥ 1/√(M)T, t < dim_E(ℱ^N ,1/T) }
          + ∑_t=1^T e_t 1{ e_t ≥ 1/√(M)T , t ≥dim_E(ℱ^N ,1/T) }
       ≤1/√(M) + H dim_E(ℱ^N ,1/T) + ∑_dim_E(ℱ^N ,1/T) ≤ t ≤ T( t/dim_E(ℱ^N , 1/T)  -1 )^-1/2√(4 β(ℱ^N, δ)/M)
       ≤1/√(M) + H dim_E(ℱ^N ,1/T) + 2 ( T/dim_E(ℱ^N ,1/T) -1)^1/2dim_E(ℱ^N ,1/T) √(4 β(ℱ^N, δ)/M)
        =  1/√(M) + H dim_E(ℱ^N ,1/T) +  √(16 ·dim_E(ℱ^N ,1/T) · T ·β(ℱ^N , δ)/M).

    Taking M →∞,
    
    ∑_k=1^K ∑_h=1^H b^k_h(s^k_h, a^k_h ) 
        ≤ H dim_E(ℱ^N, 1/T).

    
























[<ref>]

    Under Assumption <ref>, with probability at least 1- δ,  achieves a regret bound of
    
    Reg(K) ≤ 
        2Hdim_E(ℱ^N, 1/T) + 4H√(KH log(2/δ)).

    
    
    



Recall that ξ_h^k = [_h(V^k_h+1 - V^π^k_h+1)](s^k_h, a^k_h) - ( V^k_h+1(s^k_h+1) - V^π^k_h+1(s^k_h+1)) is a martingale difference sequence where [ξ^k_h | 𝔽^k_h] = 0 and | ξ^k_h| ≤ 2H.
By Azuma-Hoeffding's inequality, with probability at least 1-δ/2,

    ∑_k=1^K∑_h=1^H ξ^k_h ≤ 4H√(KH log(2/δ)).


Conditioning on the above event and Lemma <ref>, we have

    
    Reg(K)    ≤ 2 ∑_k=1^K ∑_h=1^H b^k_h(s^k_h, a^k_h) + ∑_k=1^K ∑_h=1^H ξ^k_h 
            
       ≤ 2Hdim_E(ℱ^N, 1/T) + 4H√(KH log(2/δ))

    






























 




























































































































    

























    


















































































 Paper Title: Insurance Portfolio Pursuit with Reinforcement Learning.



The Inevitable Quark Three-Body Force and its Implications for Exotic States
    Su Houng Lee
    August 5, 2024
============================================================================



§ ABSTRACT 
	When faced with a new customer, many factors contribute to an insurance firm's decision of what offer to make to that customer. In addition to the expected cost of providing the insurance, the firm must consider the other offers likely to be made to the customer, and how sensitive the customer is to differences in price. Moreover, firms often target a specific portfolio of customers that could depend on, age, location, and occupation. Given such a target portfolio, firms may choose to modulate an individual customer's offer based on whether the firm desires the customer within their portfolio. Given a target portfolio, we term the problem of modulating offers to achieve this target portfolio the portfolio pursuit problem. We give a formulation of portfolio pursuit as a sequential decision making problem, and devise a novel reinforcement learning algorithm for its solution. We test our method on a complex synthetic market environment, and demonstrate that it outperforms a baseline method which mimics current industry approaches to portfolio pursuit. 




§ INTRODUCTION




 §.§ Background


Competition among the commercial insurance market has been intensified by the emergence of price comparison websites (PCW) as a core means by which customers and insurance firms interact. When a customer's details are entered into the website, many firms simultaneously decide what offers to make to the customer on the basis of those details. Since the customer can directly compare competing offers, firms must carefully calibrate their offers to maximise a trade-off between potential profit and the probability of acceptance. While both of these aims are local to each individual customer, firms may also wish to modulate the offers they make in order to pursue longer-term goals, realised over many customers. For example, the firm may wish to maintain a certain frequency with which new offers are accepted, to ensure they have the reserves necessary to pay out claims and avoid liquidity issues <cit.>. Alternatively, a firm may wish to break into a new segment of the market, and be content with making lower-than-optimal offers in order to attract a particular customer base. Goals such as these define a higher-level strategy that is crucial to the long-term success of the firm. Decisions regarding content of the long-term goals - the What? of the strategy - are typically made at a managerial level, and involve complex domain-expert knowledge and understanding of current market conditions. At present, the implementation of the pursuit of the goals - the How? of the strategy - also makes use of domain-expect knowledge. In this paper, we present a data-driven machine learning methodology for implementing higher-level strategies in the context of consumer insurance markets. 

The problem of what offer to make to a customer on a PCW can be divided into two sub-problems: the cost-estimation problem and the bidding problem. The cost-estimation problem can be summarised as the problem of estimating the total cost of insuring a given customer from the features available through the PCW. This is a well-studied supervised-learning problem <cit.>. We will not provide any further comment on the cost-estimation problem in the remainder of this paper; we take the solution to this problem as given. Instead, we focus on the bidding problem - what offer ought we to make to a customer, given an estimate of the cost of insuring them? The optimal price to offer a customer depends not only on the estimated cost of insuring them, but additionally on, (1) the offers that other insurers on the PCW will make to the customer; (2) the customer's sensitivity to price differentials; and, (3) whether the customer is the type of client the insurance firm wishes to insure. Our work focuses on this third aspect the bidding problem. 

Throughout our paper, we use the term portfolio to refer to the set of customers who accept an insurance firm's offers over some specified period of time (for example a day, week, or month). The problem of portfolio optimisation is that of deciding what the ideal portfolio would look like; how many of each different type of customer the insurance firm wishes to have concurrently insured. This may depend on aspects such as the kind of brand and reputation the firm wishes to cultivate, the desire to maintain a diverse customer base so as to protect against correlated risk or systematic model inaccuracies, or the necessity to avoid insuring too many customers and being unable to settle all the incoming claims. We take the firm's solution to the problem of portfolio optimisation also as given; that is, we suppose that the insurance firm already has a specification for the distribution of customers it would like to insure over some relevant future time-period. However, unlike in the stock market, where an investor can easily achieve their desired portfolio of stocks by simply investing and divesting, an insurance firm cannot arbitrarily add and remove customers from its portfolio. Rather, the firm must wait until a desired customer is asking for a quote, and then modulate the offer made to the customer in order to increase the likelihood that the offer is accepted. In deciding how to modulate the offer, the firm must carefully balance the value of having the customer in the portfolio against the profit they expect to make from insuring the customer. It would not be effective to, for example, make desired customers offers at which the firm is, on average, guaranteed a substantial loss. We term the problem of how to modulate offers made to customers in order to achieve a desired portfolio the problem of portfolio pursuit. 

It is to the problem of portfolio pursuit to which we make our contribution. Note that – unlike the cost-estimation, bidding, and portfolio optimisation problems – portfolio pursuit is intrinsically sequential in nature. If we wish to insure no more than 500 customers over the next week, the offers we ought to make to customers on Sunday depend heavily on our success rate for Monday through Saturday. As such, we cast the portfolio pursuit problem as a Markov Decision Process (MDP), and derive a novel Reinforcement Learning (RL) algorithm for its solution. Our decomposition into the four sub-problems given above - cost-estimation, pricing, portfolio optimisation, and portfolio pursuit - additionally enables us to develop a methodology which is minimally disruptive, in the sense that it can be integrated naturally with arbitrary solutions to the other three sub-problems. Indeed, our algorithm requires only a slight adjustment to existing methodologies for the bidding problem. 



 §.§ Previous work


To our knowledge, our paper is the first work to articulate a general mathematical formalism of the portfolio pursuit problem in insurance as a sequential decision making problem, and attempt to address it using reinforcement learning. <cit.> and <cit.> both apply reinforcement learning to other aspects of bid optimisation in the context of insurance. Like us, <cit.> considers the new business market for insurance. They have no notion of a desired portfolio, thus making each decision purely local. As such, they treat the problem as a contextual bandit, with no sequential aspect (beyond learning). In contrast, <cit.> examine the problem of pricing at renewal. While they consider the customer portfolio (thereby making the problem sequential), they restrict themselves only to constraints on the size of the portfolio, formulating it as a constrained Markov Decision Process. By additionally factoring in the internal structure of customer portfolios, our framework presents a more general and flexible approach to portfolio pursuit. 

Many previous works have attempted to apply reinforcement learning to the portfolio optimisation problem in the context of other markets, such as the stock market <cit.> and the cryptocurrency market <cit.>.  However, as noted above, this situation is fundamentally different from the case considered here. In these other markets, the relevant financial products can typically be purchased very quickly, allowing one to shift funds between them more or less freely (subject to transaction costs). This is in contrast to customers, who arrive sequentially and over a prolonged period. Furthermore, while purchasing cryptocurrency or a stock is a fairly simple transaction, bidding for a customer is not, since it involves attempting to out-compete alternative offers while accounting for stochasticity in a customer's decision. As such, this previous literature is not directly applicable to the problem under consider here. 



 §.§ Our contribution


We propose a novel reinforcement learning algorithm for the portfolio pursuit problem over long time horizons. We cast this as an MDP, and give a novel algorithm for its solution. We take a model-based approach, allowing our algorithm to be trained entirely offline, thereby protecting against risk in deployment. The methodology we use can be integrated naturally into existing machine learning pipelines for insurance pricing. Unlike previous research addressing using RL for the purposes of insurance pricing <cit.>, our insurance agent acts in a realistic agent-based environment <cit.>. This environment includes randomly generated customers along with a collection of other insurance agents that iteratively change their pricing strategies over time, creating a dynamic, non-stationary environment. The full code implementing both the market environment and our reinforcement learning solution is available in the https://github.com/EdwardJamesYoung/RL-portfolio-pursuitcode repository accompanying this paper. 

The rest of this paper is organised as follows. In sec:problem formulation we give a mathematical formulation of the portfolio pursuit problem. In sec:methods we describe our solution to the problem. We begin by describing a standard pipeline for solving the bidding problem, and the models used within that pipeline. We then outline a baseline methodology intended to mimic a naive approach to portfolio pursuit that may currently be used within industry. Lastly, we specify our approach, giving the full algorithm used for training, and compare the method to existing RL algorithms. In sec:results we show the performance of our method against the baseline, demonstrating that our method is able to generate more profit than the baseline methodology while not compromising the quality of the final portfolio. Finally, in sec:discussion we outline some limitations of our method and give opportunities for future research. 



§ PROBLEM FORMULATION




 §.§ Portfolio pursuit as a Markov Decision Process


In our market environment, an epoch consists of T interactions between a customer and the insurance agent. The customers s_1,…,s_T themselves are drawn independently from the same underlying customer distribution. At each step, in response to the current customer features, the insurer takes an action a_t ∈ [1,2]. The action a corresponds to an offer of C(s) × a, where C(s) is the expected cost of serving the customer[As noted in the introduction, we assume this as given. In practice, the expected cost of serving a customer is not known in advance, but must instead be estimated using historic claims data. However, this estimation problem is distinct from the problem of how to price insurance, given this estimate, which is what this paper focuses on.]. In response to the offer made by the insurer, and the offers made by other insurers within the market, the customer either accepts the offer or rejects it. If the offer is rejected (y_t = 0) then no profit is obtained by the insurer. If the offer is accepted (y_t = 1) then an (expected) profit of (s_t,a_t) is obtained. The profit is simply (s,a) = C(s)a - C(s) = C(s)(a - 1). When the offer is accepted (y_t = 1) then the customer is added to the insurer's portfolio[In reality, there may be a delay between an offer being made and accepted - indeed, customers often have up to 14 days to decide whether to accept an offer. In this case, a proxy measure, such as click-through from the PCW to the insurer's website, may be used in place.]. The portfolio at the t-th time-step is ρ_t, and the insurer's portfolio before the first customer ρ_1 is initialised as empty[The choice to initialise the portfolio as empty corresponds to optimising the portfolio of new customers obtained over some time horizon. If instead the firm is interested in its total portfolio - including all existing customers - the portfolio can be initialised to that instead, with only minor adjustments required elsewhere in our method.], ρ_1 = {}.

The performance of the insurer at the final time step is evaluated according to the net profit achieved minus a loss which measures the discrepancy between the final and desired portfolio[Note that this encompasses the case where the loss is a function of only the final portfolio.], 
    ∑_t=1^T y_t (s_t,a_t) - λŁ(ρ_T+1, ρ^*).

Here, λ is a coefficient which controls the relative importance of profit vs. pursuit of a particular portfolio. 

In the MDP formalism, the states consist of customer, portfolio, and time-step triples, (s_t, ρ_t, t). The reward signal following action a_t is y_t (s_t,a_t), with an additional reward -λŁ(ρ_T+1, ρ^*) administered after the final customer interaction.



 §.§ Specification of portfolio loss function


We stress that the algorithm we present in sec:RL for PP makes no assumptions whatsoever about the nature of the loss function Ł. However, for concreteness in our numerical experiments, we adopt one particular formulation of the loss. 

The loss function used in our numerical experiment is defined via a collection of I indicators functions _A_i, i = 1,…,I over the customer features. Each indicator is specified by set A_i, i = 1,…,I in the space of customer features. These indicators allow us to represent portfolios as vectors in ^I, with the i-th component given by f_i = ∑_s ∈ρ_A_i(s), where _A_i(s) is equal to 1 if s ∈ A_i, and 0 otherwise. Given these vector representations of portfolios, we can implement the loss Ł via any distance measure on ^I. Note that in practice we specify the target portfolio ρ^* only through its vector representation f^*_i ∑_s ∈ρ^*_A_i(s); similarly, we will track the insurer's portfolio only via its frequency representation. For our numerical experiments we adopt the loss function:

    ℒ(f, f^*) 1/I∑_i=1^I   |f_i - f^*_i| /max(f_i, f_i^*) .

We choose this loss for the following properties: it is zero precisely when f_i = f^*_i; it is bounded between [0,1]; and it is symmetric in f and f^*. Note that because the loss is bounded between 0 and 1, the coefficient λ can be interpreted as answering the question: if the firm had the worst possible portfolio, how much short-term profit would they be willing to give up in order to trade it for the best possible portfolio? 

Our reinforcement learning algorithm (detailed in sec:RL for PP) trivially extends to alternative loss functions which provide other ways of quantifying the distance between the obtained and target portfolio. This includes, for example, adding weights to each indicator to penalise discrepancies along different feature sets differently, or using an optimal transport distance to provide a more detailed measure of differences between sets. We leave exploration of alternative loss formulations to future work. 



§ METHODS




 §.§ Standard industry methodology


We compare our method to a pipeline which mimics current industry approaches to insurance pricing and price modulation. In this section we describe a methodology which does not make reference to a target portfolio. This pipeline mimics standard industry pricing methods. We will then modify this method in two ways - firstly, to mimic standard industry methodology for modulating prices to attain a target portfolio, and secondly according to our new proposed algorithm.  

The insurance firm is assumed to have a dataset of interactions from previous epochs. Each entry in this interaction dataset includes:

	
  * The customer features, s_t
	
  * The expected cost of serving the customer, C(s_t)
	
  * The action taken by the insurance firm, a_t
	
  * The offers made by other insurance firms, o_t
	
  * Whether the customer accepted the offer of the firm, y_t

From the offers made by other firms, o_t, and the expected cost, C(s_t), the insurer calculates market variables, m_t. The market variables for the insurer are the average of the top-1, top-3, and top-5 offers made to the customer by other firms within the market, normalised by dividing by the expected cost C(s_t) of serving the customer. Note that the market variables are not available to the insurer at inference time, but become available after the fact through the purchasing of historic data from PCWs. 

From this dataset, the insurer trains the following models:

	
  * The market model M, which maps from customer features, s_t, to market variables m_t, s_t ↦ M(s_t) ≈ m_t.
	
  * The conversion model p, which maps from the market variables, m_t, and action, a_t, to the probability the customer accepts the offer, (m_t,a_t) ↦ p(m_t,a_t) ≈(y_t = 1|m_t, a_t).
	
  * The action model, or policy π, which maps from market variables, m_t, to optimal actions a^*_t, m_t ↦π(m_t) ≈ a^*_t.

For our numerical simulations, we used a random forest for the market model, a Multi-Layer Perceptron (MLP) for the conversion model, and a Gaussian Process with Matern kernel for the action models <cit.>. Our overall methodology places no restrictions on any of the models, although in practice we found it better to use continuous conversion and action models. If it is necessary to ensure these models are interpretable, simpler models - such as Generalised Linear Models (GLMs) - may be used instead of MLPs <cit.>. More expressive models may work better for larger scale simulations, with correspondingly larger datasets, than those explored here. 

The market model M and conversion model p are trained directly on the historic dataset available to the insurer. To train the action model, we use regression-based amortised optimisation <cit.>. Firstly, note that the expected profit, given that the customer accepts the insurer's offer, is C(s_t)(a_t - 1). Accordingly, the insurer's expected profit, for action a_t, given market variables m_t, is C(s_t)(a_t - 1)p(m_t, a_t). Therefore, for a random sample of market variables m_t from the dataset, we use a one-dimensional optimiser to find a^*_t ∈max_a p(m_t, a)(a - 1) within the range a_t ∈ [1, 2]. We then fit the action model π to 500 (m_t, a^*_t) pairs. 

At inference time, the features for a customer s_t are fed into a market model to obtain estimates for the market variables[As the market variables are predictive of customer behaviour, and available historically but not at inference time, the use of a market model can be seen as a form of data imputation.], m_t = M(s_t). These are then fed into the action model to obtain the action a_t = π(m_t). An offer of C(s_t)a_t is then made to the customer. On 12% of customers we explore by multiplying the offer by a randomly chosen modulation factor sampled from [0.93, 0.95, 0.97, 1.03, 1.05, 1.07]. 

Before continuing, we briefly remark that in practice the training and use of auxiliary models may be more complicated than presented here. To begin with, information about whether a customer accepts an offer is not available instantaneously, but only after a “cooling-off” period, typically 14 days. Likewise, there may be a delay in obtaining information about market variables for a given customer. Finally, insurers may choose to forgo the use of an action model all-together, instead solving the one-dimensional optimisation problem directly at inference time (rather than amortising as we do here). None of these observations present substantial difficulty to the RL algorithm we describe below. Indeed, our approach can be seen as a methodology which accepts as input auxiliary models and adjusts the prices returned by those models to solve the portfolio pursuit problem. It is therefore agnostic as to both the types of models used, and the data which those models are trained on. 



 §.§ Baseline methods for portfolio pursuit


It is standard practice to modulate the output prices offered to customers from the bidding model according to other features of interest for the customer, corresponding to the customer's desirability within the portfolio. This modulation is typically performed using domain experts' knowledge, rather than according to a data-driven or formal scheme. We attempt here to give a method which mimics this type of modulation as a baseline against which to compare our method, fully recognising that it does not accurately capture the complexity of standard industry practices. We encourage future work to implement more realistic and nuanced methodologies, and have facilitated this by making our codebase implementing the market environment publicly available. 

The baseline method makes use of historical assessments of the frequency with which the insurer encountered customers of the type relevant to their portfolio. In particular, for each indicator set A_i, compute the frequency representation f_i = ∑_s ∈ρ_T+1_A_i(s) for terminal portfolios ρ_T+1 for epochs during which there was no portfolio-based action modulation, where actions were chosen via the method described in the previous section, sec:standard industry method. We average this value over such historical epochs, giving us a frequency f̅_i. f̅_i therefore acts as an estimate of how many customers within set A_i we are likely to get without action modulation. We compare this to the desired frequency, f^*_i = ∑_s ∈ρ^*_A_i(s). For each incoming customer, we look at each set A_i that they call into, and compute the corresponding ratio of frequencies, f̅_i/f^*_i. If this ratio is greater than 1 it indicates that, in order to achieve the desired portfolio, we should increase the offer price made to that customer. If the ratio is less than 1, we should decrease the offer price. At inference time, we therefore modulate the offer price made to a customer s by multiplying the action by

    ∏_i: s ∈ A_i g( f̅_i/f^*_i).

The map g is given by 

    g(z) = 1 + βz^n - 1/z^n + 1,
 
where n, β≥ 0. The functional form of g was selected because it is monotonically increasing and bounded, with g(z) ∈ [1 - β, 1 + β]. The function parameters n ∈{0.5, 1, 2} and β∈{ 0.02, 0.05, 0.1, 0.25 } are found by a grid-search; in our experiments we used values of n = 1, β = 0.02 for a loss coefficient of λ = 2000. Note that for a baseline method, β, n must be chosen differently for each value of the loss coefficient λ. This is in contrast to our method, which applies equally well for any value of λ. 



 §.§ A Reinforcement Learning approach to Portfolio Pursuit




  §.§.§ Mathematical Preliminaries


Our reinforcement learning algorithm uses a dynamic programming approach with function approximation to provide a policy which modulates customer prices according to the current portfolio possessed by the insurer. We define the portfolio value function, V_π(ρ, t), which gives the value of having portfolio ρ at time t, given that future actions are selected by a (time-step dependent, deterministic) policy π = π(s, ρ, t):

    V_π(ρ, t) _π[ ∑_k=t^T y_k (s_k, a_k) - λŁ(ρ_T+1, ρ^*) | ρ_t = ρ].

The corresponding action value function Q_π(s,a,ρ,t) is defined by:

    Q_π(s, a, ρ, t) _π[ ∑_k=t^T y_k (s_k, a_k) - λŁ(ρ_T+1, ρ^*) | ρ_t = ρ, s_t = s, a_t = a ].

The policy is optimal for all time steps if it satisfies the appropriate Bellman optimality equation, 
    π(s, ρ, t) ∈max_a Q_π(s, a, ρ, t).

In app:bellman recursion, we show by expanding the action value function that (<ref>) is equivalent to:

    π(s, ρ, t) ∈max_a (y_t = 1 |s_t = s, a_t = a) ( a - k_π(s,ρ, t+1) ),
 
where k_π(s, ρ, t+1) is defined by:

    k_π(s, ρ, t)  1 - V_π(ρ∪{s}, t) - V_π(ρ, t)/C(s).

An important consequence of the above result is that the optimal policy is a function only of the customer features s_t and the value k_π, and not of the portfolio or time-step directly except via k_π.

We also obtain (see app:bellman recursion) the following recursion relationship for the portfolio value function:

    V_π(ρ, t) = V_π(ρ, t+1) + [  C(s) (y_t = 1 |s_t = s, a_t = π(s,ρ, t)) ( π(s,ρ, t) - k_π(s, ρ, t+1) )  ],

with boundary condition V_π(ρ, T+1) = - λŁ(ρ, ρ^*). Here, the expectation is taken over the customer distribution. 



  §.§.§ Outline of our method


As with the industry baseline method, our method will make use of a market, conversion, and action model. We will use the conversion model p(m, a) ≈(y_t = 1 | m_t = m, a_t = a) in place of the (y_t = 1|s_t = s, a_t = a) in both the value recursion relationship (<ref>) and the optimal action relationship (<ref>)[We stress that this is only an approximation. In reality – owing to how we constructed the market environment – the probability of acceptance also depends on various customer features.]. Additionally, our action model (or policy) π will now be a function of both the market variables m and the k-values K, π(m,k). 

Our method begins by training these models in much the same manner as the pipeline described in sec:standard industry method, namely by fitting models to a previously obtained dataset. However, our method for generating the dataset for fitting the action model π is slightly different. We first sample market variables m_t from our dataset. For each market variable sample, we independently sample a k-value, k_t, from a Laplace distribution with location 1 and scale 0.1. We then compute 

    a^*_t ∈max_a p(m_t, a)(a - k_t),
 where p is our conversion model. We then fit the action model to approximate π(m_t, k_t) ≈ a^*_t. We used a sample of 500 (m, k, a^*)-triples to fit our conversion model. We additionally train a value function V_ϕ(ρ, t), with parameters ϕ, as outlined in the next section. 

At inference time, for a customer s_t with expected cost C(s_t) and portfolio ρ_t at time t, we use (<ref>) to compute the k-value k_t via:

    k_t = 1 - V_ϕ(ρ_t∪{s_t},t+1) - V_ϕ(ρ_t,t+1)/C(s_t).

We then feed the customer features s_t into the market model to obtain estimates for the market variable m_t = M(s_t). We then use the bidding model to obtain the action a_t = π(m_t, k_t). Finally, we make an offer of C(s_t)a_t to the customer. 



  §.§.§ Value function training


In order to train our inference-time value function V_ϕ(ρ,t), we make use of an next-step value function, U(ρ). This function is used during training, but not at inference time. Throughout training, it approximates the values of portfolios at the next time step. U is used to create value estimates, which are then stored in a dataset. Our value function V_ϕ (which will be used at inference time) is then fit to this dataset. As discussed in sec:portfolio loss function, portfolios are specified only through their frequency representation, ρ↦ (f_i)_i=1^I, f_i = ∑_s ∈ρ_A_i(s), since this preserves all information relevant for computing the loss. Accordingly, both U and V_ϕ act on frequency representations, vectors in ^I. In the remainder of this section we do not distinguish between a portfolio and its frequency representation. 

We iterate backwards through time, setting t = T,…,1. At each time step, we use U(ρ) to approximate V_π(ρ, t+1). Initially, we set U(ρ) = -λℒ(ρ, ρ^*) = V_π(ρ, T+1). At each time step, we use the next-step value function U to form value estimates via (<ref>) and (<ref>). To approximate the expectation over the customer distribution in (<ref>), we compute an empirical average over samples from a customer reply buffer, which stores the features, market variables, and expected cost of previous customers. We then fit U to these value estimates thus obtained, such that U now approximates V_π(ρ, t). Having fit U, we sample an augmenting set of portfolios, and use U to estimate each of their values. We finally store the value estimates in a dataset, 𝒟, and then iterate backwards, t  t - 1. Once we have completed the backwards iteration through time, we fit our value function V_ϕ to the value estimates found in 𝒟.

Before fitting, we apply a trick to make our dataset easier to learn. Notice that at inference time, V_ϕ is used only to compute k values via (<ref>). However, this equation is invariant to shifting all values at same time step t by the same amount. Therefore, before fitting V_ϕ, we recentre all the value estimates V at each time step by subtracting the mean value estimate, V̅.  We then fit our value function V_ϕ to the residuals, V̂ = V - V̅. We found that this step is critical to effective learning, as it requires the value function V_ϕ to learn the differences in value between portfolios, rather than overall magnitude. These differences are what drives decision making - indeed, the centring of our value estimates makes V_ϕ analogous to an advantage function used in other RL methods. 

Our approach is summarised in alg:training value function. In our experiments, we set U to be linear model, and V_ϕ to be an MLP. As with the conversion, market, and action models, nothing in our algorithm necessitates the use of these particular models for U and V_ϕ. We use N = 500 samples from the customer reply buffer, and J = 24 portfolio samples for fitting the next-step model, and an additional J^+ = 120 augmenting portfolios. 



In our algorithm, there are two points at which we sample portfolios. We have not yet specified how this sampling occurs. Ideally, these portfolios would be on-policy, the portfolios sampled for time step t would be taken from the distribution of portfolios our policy encounters at time t. However, since we do not have access to any such data, our method is forced to be off-policy. The sampling of portfolios comes from three sources: previously on-policy portfolios, target on-policy portfolios, and high-coverage portfolios. The sampling is done in a ratio of 1:1:2. We explain each of these methods in turn. 

Much like the baseline method discussed in sec:baseline method, the previously on-policy portfolio sampling makes use of historic data about the frequency with which customers in each category appear naturally without action modulation. As before, we denote by f̅_i the average of the frequency representations ∑_s ∈ρ_t+1_A_i(s) over historic epochs in which no portfolio was being pursued. We can then define a historic rate, given by p̅_i = f̅_i/T. To sample previously on-policy portfolios at time t, we sample each component of the portfolio's frequency representation independently from the Binomial distribution with parameters p_i + 1/T, t. The addition of 1/T to the rate was made to ensure that, for each category, we have a strictly positive probability of sampling portfolios with customers in that category. Sampling these portfolios ensures we have a good representation of value for portfolios that we would encounter if we did not adjust our pricing methodology. The target on-policy portfolio sampling method works in the same manner as the previously on-policy portfolio sampling, except we use p^*_i = f^*_i/T where f^*_i = ∑_s ∈ρ^*_A_i(s) is the target frequency. 

The last method we use is high-coverage portfolios. This method is parametrised by a slope parameter, σ∈ [0,1]. We first define p_max = (1 + σ)max( p^*, p̅ ), p_min = (1 - σ)min( p^*, p̅ ), where the maximum and minimum are taken element-wise. At time t, we consider evenly spaced points ω∈ [0,1]. For each ω point, we construct the frequency representation f_t(ω) = t( ωp_max + (1 - ω)p_min ), which interpolates between the frequency representation if customers were acquired at either the maximum rate or minimum rate. This will in general not be integer valued, so we therefore take the integer part of each element, and then add on a Bernoulli sample which is one with probability equal to the remainder term. The high-coverage portfolios thus sampled ensures a wide sampling outside the on-policy distributions. The contribution of these portfolios is crucial, as it ensures our method still has accurate assessments of portfolio values outside the on-policy distribution, providing stability at inference time. We used a value of σ = 0.9.



 §.§ Relationship to existing RL methods


Here we briefly discuss the relationship of our proposed RL algorithm to existing algorithms within the literature. Our method can be viewed as an actor-critic algorithm, in the sense that it learns both a policy π(m,k) and a value function V_ϕ(ρ,t). However, our method displays substantial differences to more conventional actor-critic methods, such as A3C <cit.>, PPO <cit.>, SAC <cit.>, and DDPG <cit.>. 

Firstly, our method is not a policy-gradient algorithm. Unlike the methods listed above, our policy is not learned by finding the gradient of a value measure (such as total return <cit.>, or a Q-function <cit.>) with respect to the policy parameters. Instead, we exploit the fact that - because the action space is one-dimensional - the relevant value measure, given in (<ref>), can be optimised directly with a non-gradient-based optimiser. We then fit our policy directly to the optimised values - a form of regression-based amortised optimisation <cit.>. This allows us to avoid problems relating to local maxima which arise in gradient-based methods. It also frees us from the necessity of having a policy which is differentiable in the parameters; indeed, in our experiments we use a Gaussian process as our policy <cit.>. In fitting the policy to a dataset of actions which maximise a value measure, our method is akin to the cross-entropy method <cit.>. 

Secondly, our value function is not a function of the full MDP state. The full MDP state is the (customer, portfolio, time-step) triple, (s,ρ,t). Instead, our value function only depends on the portfolio and the time-step, (ρ,t). This substantially reduces the dimensionality of the input space to our value function, since the number of customer features is (typically) much larger than the number of portfolio categories. From (<ref>), we know that the optimal action depends only on the next portfolio, time-step pair. This means that, for the purposes of action-selection, it suffices to learn only these values, rather than the full MDP state values (or, indeed, the value of state-action pairs <cit.>). We also note that the dependence is only through the k-value. This enables us to learn a policy which takes k directly as input, rather than the full MDP state. Importantly, this policy can be learned fully before learning any value functions (as we do in our algorithm), and does not require re-training while the value function is learned, unlike a typical actor-critic method <cit.>. 

Lastly, our method utilises a memory-reply buffer to act as a model of the arrival of new customers. However, unlike many other methods which use a memory reply buffer for value-learning <cit.>, we do not sample states randomly. Rather, (portfolio, time-step) pairs, (ρ, t), are sampled systematically by iterating t backwards in time. In iterating backwards, our method is analogous to a dynamic programming algorithm, which starts at terminal states of known value (in our case, terminal portfolios) and applies backwards induction to find the value of all states recursively. Similar to the Fitted Q-Iteration algorithm <cit.>, our method avoids the curse of dimensionality associated with exact dynamic programming in MDPs by approximating the value function at each time-step by a linear model, U. Because we are applying an approximate dynamic programming method we need only sweep through the state-space once. This is in contrast to methods which apply a Bellman optimality equation but do not recurse backwards from terminal states <cit.>, where each state must be sampled many times in order to approximate the correct value. 





  §.§.§ Other applications of our algorithm


Although we have designed our algorithm specifically for the portfolio pursuit application pursued in this paper, certain elements readily generalise to other situations not considered here. We briefly enumerate potential other uses of elements of our algorithm. 

Firstly, our algorithm depends on the fact our state-space factorises into portfolio, time-step, customer feature triples (ρ, t, s). The fact that we have a fixed horizon T, and that the time-step t is a component of the state-space, allows us to compute values by iterating backwards in time. 

the use of a simple (in our case, linear) model 






 §.§ Additional simulation details


Our simulations use a realistic agent-based model of an insurance market, mediate through a PCW <cit.>. The insurance firm implementing each method – baseline (sec:baseline method) or RL (sec:RL for PP) – competes with 5 other insurance firms (none of which are pursuing a portfolio, as in sec:standard industry method). Each customer is randomly generated by sampling features from a complex generative distribution, with realistic interdependencies between features. For example, customer income depends on the location, age, and occupation of the customer. Only a subset of features are revealed to the insurers though the PCW interface. On the basis of these features, each insurer makes the customer an offer. The customer then either selects an offer or chooses to walk away from the market. Full details of the market environment can be found in the https://github.com/EdwardJamesYoung/RL-portfolio-pursuitcode repository that accompanies this paper. 

Target portfolios are generated at random by sampling I = 5 customer features, and setting the target quantity f_i^* for those features to be twice f̅_i if f̅_i ≤ 10, and otherwise half or twice of f̅_i at random (rounding up or down respectively to ensure integer values). We have 6 burn-in epochs, during which historic data is gathered for training models. Insurers use the past 4 epochs to train models, and all burn-in epochs for computing historical frequency representations. For the first burn-in epoch, before any models are trained, insurers take actions randomly and uniformly in [1,1.2]. Following the burn-in epochs, we then have 8 testing epochs. Each epoch consists of T = 1000 steps. No re-training of any models occurs following a testing epoch, meaning the testing epochs are independent trials of the same value function V_ϕ. In the results below, we aggregate statistics across testing epochs. We set the loss-coefficient λ = 2000. The results shown in sec:results are obtained by aggregating statistics across both the 8 testing epochs and 24 independent trials, with each trial having a different randomly generated target portfolios. To ensure a fair comparison between the RL and baseline methods, we use the same set of random seeds for both methods. This means that, up until the testing epochs, both methods interact with the market identically, ensuring the same historic data to train on and target portfolio to aim towards. Thus, any differences in performance are attributable only to differences over the testing epoch. 



§ RESULTS




fig:main-results shows the results of comparing our method to the baseline averaged over 24 trials. On each trial, both methods have the same interaction history, and thus have the same training data, up until the testing epochs. They also pursue identical target portfolios. Thus, differences in performance stem only from different behaviour on the test epochs. From panels (A-B) we see that, on average, our method generates more profit than the baseline (one-sided Mann-Whitney U test, p < 0.0001; CLES = 65%; Cohen's d = 0.51, moderate effect size). Over the course of 1000 customer interactions, on average our method generates an additional 472 (or a 7% increase) in profit. Panels (C-D) show the loss between the target portfolio and the obtained portfolio at each time step. The final loss distribution of both methods is very similar (two-sided Mann-Whitney U test, p = 0.16, not significant), showing that our method has not compromised portfolio quality in order to generate additional profit. Thus we can conclude that our method brings in additional revenue while getting comparably close to the desired portfolio as the baseline. Panels (E-F) show the total reward (profit minus loss) of both methods, which is the objective that both methods are optimising. We see that our method outperforms the baseline, achieving higher reward on average over the trials (one-sided Mann-Whitney U test, p < 0.0001; CLES = 63%; Cohen's d = 0.48, moderate effect size). In Table <ref> we tabulate the average final profit, loss, and reward of both methods.  





§ DISCUSSION




 §.§ Limitations and opportunities for future work


We believe that our work presents an important first step in the application of ideas from reinforcement learning to the portfolio pursuit problem. However, our work has several limitations, with corresponding opportunities for future research. 

In our investigation, we limited the time span over which insurers pursued a particular portfolio to only T = 1000 time-steps, with one customer arriving per time-step. This is a relatively short length of time for a large insurer, who may see orders of magnitude more customers per day. Our choice to limit the number of time-steps was motivated primarily by the prohibitive computational cost of running dozens of market simulations for larger numbers of time steps. Preliminary experiments provided evidence that our method will likely continue to work for larger numbers of time-steps. However, for much larger lengths of time, on the order of months, a different approach may well be needed. Accordingly, we believe that the framework and methods we have develop above could be integrated naturally with RL methods which attempt to perform temporal abstraction, goal-conditioned policies <cit.> or the options framework <cit.>. Indeed, longer time-scale portfolio pursuit can be accomplished by chaining together shorter time-scale portfolio pursuit with the appropriately chosen intermediate target portfolios. We leave the problem of selecting such intermediate target portfolios to future research. 

In our framework, we have also made the simplifying assumption that customers do not leave the portfolio once they have entered. This is justified when the number of time-steps is small, since few customers will both accept and then cancel a contract over the course of, one day. However, for longer time-scales, it may begin to become necessary to model the exit of customers from the portfolio. In app:customers leaving we expand our analytic derivation to consider this case, and provide some preliminary suggestions for how to integrate this into our algorithm. 

Lastly, our method is model-based, with optimal actions and values computed using a conversion model which estimates the probability of customers accepting any given offer. Consequently, if the conversion model is incorrect, our bidding strategy will accordingly be sub-optimal. For example, the bidding model may underestimate the probability that a customer accepts an offer at a given price, and make offers too low. Model uncertainty could be incorporated into the algorithm by explicitly penalising the value of actions about which the insurer is uncertain. A core step in making the algorithm more robust would be allowing for online updates of the models in light of new data coming in. This is an increasingly pressing problem within insurance pricing in particular, since such markets tend to be very non-stationary, shifting rapidly as individual insurers change their pricing strategy and inflation drives changes in prices. An effective pricing strategy will need to precisely trade-off between profit and the value of the information generated by making particular offers. Although we do not address it in this paper, we expect this to be a fruitful area for future research into the application of RL to insurance pricing. 



 §.§ Conclusion


Our paper has introduced a new formalism for a problem within the insurance industry - that of portfolio pursuit. Portfolio pursuit asks the question of how to modulate the offers made to individual consumers, in light of a desired target portfolio which describes the higher-level strategy of the insurance firm. We formulated portfolio pursuit it as an MDP in sec:problem formulation. Given its formal description, in sec:methods we discussed both industry standard methodologies for addressing this problem, and gave our own reinforcement learning solution. We then showed in sec:results that our method outperforms the industry baseline, generating more profit while getting equally close to the target portfolio. It is our hope that this paper will stimulate additional research into this area, laying the groundwork for better algorithms and methodologies for portfolio pursuit. 



 §.§ Acknowledgements


EJY is supported by the UKRI Engineering and Physical Sciences Research Council Doctoral Training Program grant EP/T517847/1. Work was completed during an internship at Accenture (UK) Limited, in connection with the Turing Institute. 



 §.§ Declaration of competing interest


The authors declare no competing interests.



 §.§ Codebase


The GitHub code repository associated with this paper can be found https://github.com/EdwardJamesYoung/RL-portfolio-pursuithere. The code is licensed under Creative Commons Attribution-NonCommercial 4.0 International Public License (CC BY-NC 4.0)



 §.§ Author contributions


Problem statement was formulated jointly by EJY, AR, and JJ. Market environment designed by EJY. Mathematical derivations, algorithm design, and numerical experiments were preformed by EJY. EJY drafted the manuscript, with additional comments and feedback provided by ET, AR, and JJ. 



apalike








§ BELLMAN RECURSIONS FOR PORTFOLIO VALUES


In this appendix we expand out the Bellman equations for the portfolio and action value functions, given by (<ref>) and (<ref>) respectively. We start with the action value function. We will expand this in terms of the portfolio value function. Firstly, we denote the returns by

    G_t = ∑_k=t^T y_k (s_k, a_k) - λŁ(ρ_T+1, ρ^*).

Then we have

    Q_π(s, a, ρ, t)    _π[ G_t | ρ_t = ρ, s_t = s, a_t = a ] 
       = (y_t = 1 |s_t = s, a_t = a)[ G_t | ρ_t = ρ, s_t = s, a_t = a, y_t = 1 ] 
       + (1 - (y_t = 1 |s_t = s, a_t = a))[ G_t | ρ_t = ρ, s_t+1 = s, a_t+1 = a, y_t = 0 ] 
       = (y_t = 1 |s_t = s, a_t = a)( (s,a) + [ G_t+1 | ρ_t+1 = ρ∪{ s }]  ) 
       + (1 - (y_t = 1 |s_t = s, a_t = a))[ G_t+1 | ρ_t+1 = ρ] 
       =(y_t = 1 |s_t = s, a_t = a)( (s,a) + V_π( ρ∪{ s }, t+1 ) ) 
       + (1 -(y_t = 1 |s_t = s, a_t = a)) V_π( ρ, t+1 ) 
       = V_π(ρ, t+1) + (y_t = 1 |s_t = s, a_t = a) ( (s,a) + V_π( ρ∪{ s }, t+1 ) -  V_π( ρ, t+1 ) ).

Note that, since the profit is the offer price C(s)a minus the expected cost C(s), the profit is thus given by: (s,a) = C(s)(a - 1). We can then say that:

    Q_π(s,a,ρ,t)    = V_π(ρ, t+1) + (y_t = 1 |s_t = s, a_t = a) ( C(s)(a - 1) + V_π( ρ∪{ s }, t+1 ) -  V_π( ρ, t+1 ) ) 
       = V_π(ρ, t+1) + C(s) (y_t = 1 |s_t = s, a_t = a) ( a - k_π(s,ρ, t+1) ),

where k_π(s, ρ, t+1) is defined in (<ref>). Since the first term is a constant in a, and C(s) > 0, this shows that maximisation of Q_π over a is equivalent to maximising:

    (y_t = 1 |s_t = s, a_t = a) ( a - k_π(s,ρ, t+1) ).
 
Furthermore, note that V_π(ρ, t) = [ Q_π(s,a,ρ, t) ]. Assuming a deterministic policy π = π(s,ρ, t), this gives us the following recursion:

    V_π(ρ, t) = V_π(ρ, t+1) + [  C(s) (y_t = 1 |s_t = s, a_t = π(s,ρ, t)) ( π(s,ρ, t) - k_π(s, ρ, t+1) )  ].




§ GENERALISATION TO CUSTOMERS LEAVING


In this section we generalise the analysis is app:bellman recursion to the case where customers can leave the portfolio. 

We define the action-value function as:

    Q_π(s, a, ρ, t) _π[ G_t | ρ_t = ρ, s_t = s, a_t = a ]

We will now compute the Bellman recursion for Q_π. To do so, we introduce a portfolio transition distribution, p( ρ' | ρ ). This gives the probability of having portfolio ρ' at time t+1, given that you ended time-step t with portfolio ρ. 

    Q_π(s, a, ρ, t)    = (y_t = 1 |s_t = s, a_t = a)[ G_t | ρ_t = ρ, s_t = s, a_t = a, y_t = 1 ] 
       + (1 - (y_t = 1 |s_t = s, a_t = a))[ G_t | ρ_t = ρ, s_t+1 = s, a_t+1 = a, y_t = 0 ] 
       = (y_t = 1 |s_t = s, a_t = a)( (s,a) + ∫[ G_t+1 | ρ_t+1 = ρ' ] p(ρ' | ρ∪{s} ) dρ' ) 
       + (1 - (y_t = 1 |s_t = s, a_t = a)) ∫[ G_t+1 | ρ_t+1 = ρ' ] p(ρ' | ρ  ) dρ' 
       =(y_t = 1 |s_t = s, a_t = a)( (s,a) +∫ V_π( ρ', t+1 ) p(ρ' | ρ∪{s}) dρ' ) 
       + (1 -(y_t = 1 |s_t = s, a_t = a)) ∫ V_π( ρ', t+1 ) p(ρ' | ρ  ) dρ' 
       = ∫ V_π( ρ', t+1 ) p(ρ' | ρ  ) dρ' 
       + (y_t = 1 |s_t = s, a_t = a) ( (s,a) + ∫ V_π( ρ', t+1 ) [ p(ρ'|ρ∪{s}) - p(ρ'| ρ) ] dρ' ) 
       = [ V_π(ρ', t+1) | ρ] 
       + (y_t = 1| s_t = s, a_t = a) ( (s,a) + [ V_π(ρ', t+1) | ρ∪{ s }] - [ V_π(ρ', t+1) | ρ] )

We now use the substitution that (s,a) = C(s)(a - 1). Following the derivation in app:bellman recursion, we define: 

    k_π(s, ρ, t) = 1 - [ V_π(ρ', t) | ρ∪{ s }] - [ V_π(ρ', t) | ρ]/C(s)

Our optimal policy is then the same. Integrating over customers and actions, we obtain the following recursion for the portfolio value function V_π: 

    V_π(ρ, t) = [ V_π(ρ', t+1) | ρ] + [ C(s) (y_t=1|s_t=1,a_t = π(s,ρ,t))(π(s,ρ,t) - k_π(s, ρ, t+1)) ]
 
Given access to p(ρ'|ρ), we can easily adapt alg:training value function to use this recursion instead, by sampling portfolios from p and averaging to approximate the expectations in (<ref>) and (<ref>). This adds some computational overhead, but is not prohibitive. The transition model p(ρ'|ρ) can be learned from historic data. 

One issue we encounter when we account for the possibility of customers leaving the portfolio is that the portfolio objects become considerably more complicated. Recall from sec:RL for PP that, when customers do not leave the portfolio, we can specify portfolios only through their frequency representation, since (i) it was unimportant when a customer joined the portfolio, and (ii) the other customer features were unimportant. However, in this case, neither of this conditions are true. Firstly, the probability of any given customer leaving depends how long they have been in the portfolio. Secondly, the probability of a customer leaving after being in the portfolio for any length of time may be a function of their other features. This necessitates a more detailed and possibly high-dimensional portfolio representation. We do not explore these issues further in our paper. 



 Paper Title: Gaussian Processes Sampling with Sparse Grids under Additive Schwarz Preconditioner.




Silicon Double-Disk Optomechanical Resonators from Wafer-Scale Double-Layered Silicon-on-Insulator
    Amy Navarathna,1,2 Benjamin J. Carey,1,2 James S. Bennett,3 Soroush Khademi,1,2 and Warwick P. Bowen1,2,4*
    August 5, 2024
==============================================================================================================




§ ABSTRACT 
  Gaussian processes (GPs) are widely used in non-parametric Bayesian modeling, and play an important role in various statistical and machine learning applications. In a variety tasks of uncertainty quantification, generating random sample paths of GPs is of interest. As GP sampling requires generating high-dimensional Gaussian random vectors, it is computationally challenging if a direct method, such as the Cholesky decomposition, is used. In this paper, we propose a scalable algorithm for sampling random realizations of the prior and posterior of GP models. The proposed algorithm leverages inducing points approximation with sparse grids, as well as additive Schwarz preconditioners, which reduce computational complexity, and ensure fast convergence. We demonstrate the efficacy and accuracy of the proposed method through a series of experiments and comparisons with other recent works.









§ INTRODUCTION

Gaussian processes (GPs) hold significant importance in various fields, particularly in statistical and machine learning applications <cit.>, due to their unique properties and capabilities. As probabilistic models, GPs are highly flexible and able to adapt to a wide range of data patterns, including those with complex, nonlinear relationships. GPs have been successfully applied in numerous domains including regression <cit.>, classification <cit.>, time series analysis <cit.>, spatial data analysis <cit.>, Bayesian networks <cit.>, optimization <cit.>, and more. This wide applicability showcases their versatility in handling different types of problems. 
As the size of the dataset increases,  GP inference suffers from computational issues of the operations on the covariance matrix such as inversion and determinant calculation.

These computational challenges have motivated the development of various approximation methods to make GPs more scalable. Recent advances in scalable GP regression include Nyström approximation  <cit.>, random Fourier features <cit.>, local approximation <cit.>, structured kernel interpolation <cit.>, state-space formulation <cit.>, Krylov methods <cit.>, Vecchia approximation <cit.>, sparse representation <cit.>, etc.


The focus of this article is on generating samples from GPs, which includes sampling from both prior and posterior processes. GP sampling is a valuable tool in statistical modeling and machine learning, and is used in various applications, such as in Bayesian optimization <cit.>, in computer model calibration <cit.>, in reinforcement learning <cit.>, and in uncertainty quantification <cit.>. Practically, sampling from GPs involves generating random vectors from a multivariate normal distribution after discretizing the input space. However, sampling from GPs poses computational challenges particularly when dealing with large datasets due to the need to factorize the large covariance matrices, which is an (n^3) operation for n inputs. There has been much research on scalable GP regression, yet sampling methodologies remain relatively unexplored.








Scalable sampling techniques for GPs are relatively rare in the existing literature. A notable recent development in the field is the work of <cit.>, who introduced an efficient sampling method known as decoupled sampling. This approach utilizes Matheron's rule and combines the inducing points approximation with random Fourier features. Furthermore, <cit.> expanded this concept to pathwise conditioning based on Matheron's update. This method, which requires only sampling from GP priors, has proven to be a powerful tool for both understanding and implementing GPs. Inspired by this work, <cit.> applied Matheron's rule to multi-task GPs for Bayesian optimization. Additionally, <cit.> applied such techniques to tighten the variance and enhance GP posterior sampling. <cit.> introduced IterGP, a novel method that ensures consistent estimation of combined uncertainties and facilitates efficient GP sampling by computing a low-rank approximation of the covariance matrix. <cit.> developed a method for generating GP posterior samples using stochastic gradient descent, targeting the complementary problem of approximating these samples in (n) time. It is important to note, however, that none of these methods have explored sampling within a structured design, an area that possesses many inherent properties.


In this study, we propose a novel algorithms for efficient sampling from GP priors and GP regression posteriors by utilizing inducing point approximations on sparse grids. We show that the proposed method can reduce the time complexity to ( (η + n)n_sg ) for input data of size n and inducing points on sparse grids of level η and size n_sg. This results in a linear-time sampling process, particularly effective when the size of the sparse grids is maintained at a reasonably low level. Specifically, our work makes the following contributions:

    
  * We propose a novel and scalable sampling approach for Gaussian Processes using product kernels on multi-dimensional points. This approach leverages the inducing point approximation on sparse grids, and notably, its computational time scales linearly with the product of the sizes of the inducing points and the sparse grids. 
    
  * Our method provides a 2-Wasserstein distance error bound with a convergence rate of ( n_sg^-ν (log n_sg)^(ν+2)(d-1)+d+1 ) for d-dimensional sparse grids of size n_sg and product Matérn kernel of smoothness ν. 
    
    
  * We develop a two-level additive Schwarz preconditioner to accelerate the convergence of conjugate gradient during posterior sampling from GPs.
    
  * We illustrate the effectiveness of our algorithm in real-world scenarios by applying it to areas such as Bayesian optimization and dynamical system problems.




The structure of the paper is as follows: Background information is presented in <Ref>, our methodology is detailed in <Ref>, experimental
results are discussed in <Ref>, and the paper concludes with discussions in <Ref>. Detailed theorems along with their proofs are provided in <Ref>, and supplementary plots can be found in <Ref>.



§ BACKGROUND


This section provides an overview of the foundational concepts relevant to our proposed method. <Ref> introduces GPs, GP regression, and the fundamental methodology for GP sampling. The underlying assumptions of our work are outlined in <Ref>. Additionally, detailed explanations of the inducing points approximation and sparse grids, which are key components of our sampling method, are presented in <Ref> and <Ref>, respectively.



 §.§ GP review




  
GPs
A GP is a collection of random variables uniquely specified by its mean function μ(·) and kernel function K(·,·), such that any finite collection of those random variables have a joint Gaussian distribution <cit.>. We denote the corresponding GP as 𝒢𝒫(μ(·),K(·,·)).



  
GP regression
Let f:^d → be an unknown function. Suppose we are given a set of training points ={_i}_i=1^n and the observations =(y_1,…,y_n)^⊤ where y_i=f(_i)+ϵ_i with the i.i.d. noise ϵ_i ∼(0,σ_ϵ^2).
The GP regression imposes a GP prior over the latent function as f(·) ∼𝒢𝒫(μ(·),K(·,·)). Then the posterior distribution at m test points ^*={_i^*}_i=1^m is

    p(f(^*) |)=(_* |, _*,* |),

with

    
    _* |= _* + _*,[_, + σ_ϵ^2 _n ] ^-1(-_),

    
    _*,* | = _*,* - _*,[_, + σ_ϵ^2 _n ]^-1_,*,


where σ_ϵ^2>0 is the noise variance, _n is a n× n identity matrix, _*, = K(^*,)=[ K(_i^*,_j) ]_i=1,j=1^i=m,j=n=K(,^*)^⊤=_,*^⊤,  _,=[K(_i,_j)]_i,j=1^n, _*,*=[K(_i^*,_j^*)]_i,j=1^m, _= (μ(_1),⋯,μ(_n))^⊤ and _*= (μ(_1^*),⋯,μ(_m^*))^⊤.




















  
Sampling from GPs

The goal is to sample a function f(·)∼𝒢𝒫(μ(·), K(·,·)). To represent this in a finite form, we discretize the input space and focus on the function values at a set of grid points ={_i}_i=1^n_s. Our goal is to generate samples _=(f(_1),⋯,f(_n_s))^⊤ from the multivariate normal distribution (μ(),K(,))=(_,_,). The standard method for sampling from a multivariate normal distribution involves the following steps: (1) generate a vector of samples _ whose entries are independent and identically distributed normal, i.e. __n_s∼(,_n_s); (2) Use the Cholesky decomposition <cit.> to factorize the covariance matrix _, as __^
⊤ = _,; (3) generate the output sample _ as

    _←___n_s + _.


Sampling a posterior GP follows a similar procedure.


Suppose we have observations = (y_1,⋯,y_n)^⊤ on n distinct points ={_i}_i=1^n, where y_i=f(_i)+ϵ_i with noise ϵ_i ∼(0,σ_ϵ^2) and f(·) ∼𝒢𝒫(μ(·), K(·,·)). The goal is to generate posterior samples f(^*)| at m test points ^*={_i^*}_i=1^m. 
Since the posterior samples f(^*) |∼ (_* |, _*,* |) also follow a multivariate normal distribution, as detailed in <ref> and <ref> in <Ref>, we can apply Cholesky decomposition _* |_* |^⊤ = _*,* | to generate GP posterior sample _*=f(^*)=(f(_1^*),⋯,f(_m^*))^⊤ as

    _*|←_* |__n + _* |,

where __n∼(,_n).


























 §.§ Assumptions


A widely used kernel structure in multi-dimensional settings is “separable” or “product” kernels, which can be expressed as follows:

    K(,')=σ^2 ∏_j=1^d K_0^(j) (x^(j),x'^(j)),

where x^(j) and x'^(j) are the j-th components of  and ' respectively, K_0^(j)(·,·) denotes the one-dimensional correlation function for each dimension j with the variance set to one, d denotes the dimension of the input points. While the product of one-dimensional kernels does not share identical smoothness properties with multi-dimensional kernels, the assumption of separability is widely adopted in the field of spatio-temporal statistics <cit.>. Its widespread adoption is mainly due to its efficacy in enabling the development of valid space-time parametric models and in simplifying computational processes, particularly when handling large datasets for purposes such as inference and parameter estimation.


In this article, we consider the inputs to be d-dimensional points ={_i}_i=1^n where each _i ∈^d and d∈^+, and we focus on product Matérn kernels that employ identical base kernels across every dimension. Matérn kernel <cit.> is a popular choice of covariance functions in spatial statistics <cit.>, geostatistics <cit.>, image analysis <cit.>, and other applications. The Matérn kernel is defined as the following form <cit.>:

    K(x,x')= σ^2 2^1-ν/Γ(ν)(√(2ν)| x-x'|/ω)^νK_ν(√(2ν)| x-x'|/ω),

for any x,x'∈ℝ, where σ^2>0 is the variance, ν>0 is the smoothness parameter, ω>0 is the lengthscale and K_ν is the modified Bessel function of the second kind.



 §.§ Inducing points approximation


<cit.> first applied Nyström approximation to kernel-based methods such as GPs. Following this, <cit.> presented a novel unifying perspective, and reformulated the prior by introducing an additional set of latent variables _, known as inducing variables. These latent variables correspond to a specific set of input locations =(u_1,⋯,u_q)^⊤, referred to as inducing inputs. In the subset of regressors (SoR) <cit.> algorithm, the kernels can be approximated using the following expression:
	
    K_SoR(_i,_j) = K(_i,) K(,)^-1 K(,_j).


    
    
    
    
    
    
    
Employing the SoR approximation detailed in <ref>, the form of the SoR predictive distribution q_ SoR(_* |) can be derived as follows:

    q_ SoR(_* |) =
         (_* + σ_ϵ^-2_*,_^-1_, (-_), _*,_^-1_,*),

where we define _ = [ _, + σ_ϵ^-2_,_,], _*, = K(^*,)=[K(,^*)]^⊤=_,*^⊤,  _,=[K(_i,_s)]_i,s=1^q, _= (μ(_1),⋯,μ(_n))^⊤, and _*= (μ(_1^*),⋯,μ(_m^*))^⊤. Consequently, the computation of GP regression using inducing points can be achieved with a time complexity of (nq^2) for q inducing points.

To quantify the approximation power in the inducing points approach, we shall use the Wasserstein metric. Consider a separable complete metric space denoted as (M,d). The p-Wasserstein distance between two probability measures μ_1 and μ_2, on the space M, is defined as follows:
	
    W_p(μ,ν)=(inf_γ∈Γ(μ_1,μ_2)𝔼_(x,y)∼γd^p(x,y))^1/p,

	for p∈[1,+∞), where Γ(μ_1,μ_2) is the set of joint measures on M× M with marginals μ_1 and μ_2. <Ref> in <Ref> shows that the inducing points approximation of a GP converges rapidly to the target GP, with a convergence rate of (n^-ν) for the Matérn kernels with smoothness ν, provided that the inducing points are uniformly distributed. It is worth noting that, inequality presented in <ref> is also applicable to multivariate GPs.
 
 Therefore, the convergence rate for multivariate inducing points approximation can be established, assuming the corresponding convergence rate of GP regression is known. A general framework to investigate the latter problem is given by <cit.>.
Inspired by <Ref>, we assume that the vector =(y_1,⋯,y_n)^⊤ is observed on n distinct d-dimensional points ={_i}_i=1^n. The inducing inputs , as defined in <Ref>, are given by a full grid, denoted as =×_j=1^d^(j), where each ^(j) is a set of q_j one-dimensional data points, A × B denotes the Cartesian product of sets A and B, accordingly the grid  has q=∏_j=1^d q_j points. Given the separable structure as defined in <ref> and utilizing the full grid , we can transform the multi-dimensional problem to a one-dimensional problem since the covariance matrix _,=K(,) can be represented by Kronecker products <cit.> of matrices over each input dimension j:

    _,=K(,) = ⊗_j=1^d K_j(^(j),^(j)).

However, the size of the inducing points q grows exponentially as d gets large, therefore we consider using sparse grids <cit.> to solve the curse of dimensionality.



 §.§ Sparse grids


A sparse grid is built on a nested sequence of one-dimensional points ∅ = _j,0⊆…⊆_j,t⊆_j,t+1⊆… for each dimension j, j=1,…,d. A sparse grid with level η and dimension d is a design as follows:

    (η,d)    = ⋃_t∈(η)_1,t_1×_2,t_2×⋯×_d,t_d,

where t=(t_1,…,t_d), (η) = {t∈^d|∑_j=1^d t_j = η}. <Ref> and <Ref> show sparse grids corresponding to the hyperbolic cross points (bisection) for two dimensional points at levels η=3,4,5,6 and for three dimensional points at η=4,5,6,7 respectively. 
Hyperbolic cross points sets, defined over the interval [0,1], are detailed in <ref>:

    _j,t_j = {i/2^t_j: i=1,…, 2^t_j-1 }.

An operator on the sparse grid can be efficiently computed using Smolyak’s algorithm <cit.>, which states that the predictor operator (η,d) on function f with respect to the sparse grid (η,d) can be evaluated by the following formula:

    (η,d)(f) = ∑_t∈(η,d) (-1)^η - |t|d-1η - |t|( ⊗_j=1^d _j,t_j) (f),

where |t| =∑_j=1^d t_j, (η,d) = {t∈^d |max(d,η-d+1) ≤|t|≤η}, ( ⊗_j=1^d _j,t_j) here represents the operator over the Kronecker product points ⊗_j=1^d _j,t_j. <cit.> and <cit.> also provided the linear time computing method for GP regression based on Smolyak's algorithm. <Ref> in <Ref> demonstrates that for a d-dimensional GP, the inducing points approximation converges to the target GP at the rate of ( n^-ν (log n)^(ν+2)(d-1)+d+1 ) for the product Matérn kernels with smoothness ν, given that the inducing points are sparse grids (η,d) of level η and dimension d, as defined in <ref>.







§ SAMPLING WITH SPARSE GRIDS


In this section, we elaborate the algorithm for sampling from multi-dimensional GPs for product kernels with sparse grids.



 §.§ Prior



  
Sampling algorithm
Our goal is to sample from GP priors f(·) ∼𝒢𝒫(μ(·), K(·,·)) over d-dimensional grid points ={_i}_i=1^n_s, _i ∈^d. According to the conditional distributions of the SoR model in equations <ref>, the distribution of _=f()=(f(_1),⋯,f(_n_s))^⊤ can be approximated by _,_,^-1_ with _∼(, _,), where _=f(), and =(η,d) is a sparse grid design with level η and dimension d defined in <ref>. Then it reduces to generating samplings from _,^-1_∼(, _,^-1). Apply Smolyak's algorithm to samplings from GPs (see <Ref> in <Ref>), we can infer that

    _,^-1_∑_t∈(η,d) (-1)^η - |t|d-1η - |t|⊗_j=1^d chol( K(_j,t_j, _j,t_j)^-1) __n_tδ_x(_t),


    δ_x(_t) = 
            0      if x ∈∖_t
    
            1      if x ∈_t,

where chol( K(_j,t_j, _j,t_j)^-1) is the Cholesky decomposition of K(_j,t_j, _j,t_j)^-1, =(η,d) is a sparse grid design with level η and dimension d defined in <ref>, __n_t∼(,_n_t) is standard multivariate normal distributed, _n_t is a n_t× n_t identity matrix, n_t = ∏_j=1^d t_j is the size of the full grid design _t=_1,t_1×_2,t_2×⋯×_d,t_d associated with t=(t_1,…,t_d), δ_x(_t) is a Dirac measure on the set .

Therefore, the GP prior _ over grid points  can be approximately sampled via


    _←_,(
        ∑_t∈(η,d) (-1)^η - |t|d-1η - |t|⊗_j=1^d chol( K(_j,t_j, _j,t_j)^-1) __n_tδ_x(_t)
        )
    
        + _,

where __n_t∼(,_n_t) is standard multivariate normal distributed, δ_x(_t) is a Dirac measure on set  as defined in <ref>, chol( K(_j,t_j, _j,t_j)^-1) is the Cholesky decomposition of K(_j,t_j, _j,t_j)^-1.




  
Complexity
Since the Kronecker matrix-vector product can be written as the form of vectorization operator (see Proposition 3.7 in <cit.>), the Kronecker matrix-vector product in <ref> only costs (∑_j=1^d t_j ∏_j=1^d t_j) =(η n_t) time, hence the time complexity of the prior sampling via <ref> is ( (η+n_s) ∑_t∈(η,d) n_t) = ( (η+n_s) n_sg), 
where ν is the smoothness parameter of the Matérn kernel, d is the dimension of the sampling points ={_i}_i=1^n_s, n_s is the size of the sampling points, n_sg is the size of inducing points on a sparse grid (η, d), n_t = ∏_j=1^d t_j is the size of the full grid design _t=×_j=1^d _j,t_j, t=(t_1,…,t_d), (η,d) = {t∈^d |max(d,η-d+1) ≤∑_j=1^d t_j ≤η}.

Note that we can obtain the upper bound of the number of points in the Smolyak type sparse grids. According to Lemma 3 in <cit.>, given that |_j,t_j|≤ 2^t_j-1, when sparse grids are constructed on hyperbolic cross points as defined in <ref>, the cardinality of a sparse grid (η,d) defined in <ref>, is thereby bounded by

    n_sg = |(η,d) |   ≤ 1^d (2-1/2)^d 2^ηη-1d-1min{2/2-1, η/d}
       ≤ 2^η-dη^d-1/(d-1)!min{ 2, η/d}
       ≤ 2^η-dη^d / (d!).




 §.§ Posterior



  
Matheron's rule
Sampling from the posterior GPs can be effectively conducted using Matheron's rule. The Matheron's rule was popularized in the geostatistics field by <cit.>, and recently rediscovered by <cit.>, who leveraged it to develop a novel approach for GP posterior samplings. Matheron's rule can be described as follows: Consider  and  as jointly Gaussian random vectors. Then the random vector , conditional on =, is equal in distribution to

    (|=)  + Cov(,) Cov(,) ^ -1 ( - ),

where Cov(,) denotes the covariance of (,).

According to <ref>, exact GP posterior samplers can be obtained using two jointly Gaussian random variables. Following this technique, we derive

    _*|_* + _*,[_, + σ_ϵ^2 _n ]^-1 ( - _ - ϵ),

where _=f() and _*=f(^*) are jointly Gaussian random variables from the prior distribution, noise variates ∼ (, σ_ϵ^2 _n). Clearly, the joint distribution of (_ ,_*) follows the multivariate normal distribution as follows:

    (_, _*) ∼(
        [  _; _*;    ],
        [   _,  _,*;  _*, _*,* ]).



  
Sampling algorithm
We can apply SoR approximation <ref> to Matheron’s
rule <ref> and obtain


    _*|_* + σ_ϵ^-2_*,_^-1_, ( - _ - ),

where _ = [ _, + σ_ϵ^-2_,_,], _=f() and _*=f(^*) are jointly Gaussian random variables from the prior distribution, noise variates ∼ (, σ_ϵ^2 _n).
The prior resulting from the SoR approximation can be easily derived from <ref>, yielding the following expression:

    q_ SoR(_,_*) = (_+*,  _+*, _,^-1_, +* ),

where _+*=[_, _*]^⊤, _+*,  = [K(,),K(,^*)]^⊤. The same as above in <ref>, (_,_*) can be sampled from


    (_,_*) 
        ←_+*,(
        ∑_t∈(η,d) (-1)^η - |t|d-1η - |t|⊗_j=1^d chol( K(_j,t_j, _j,t_j)^-1) __n_tδ_x(_t)
        )
    
        +_+*,


where __n_t, δ_x(_t), and chol( K(_j,t_j, _j,t_j)^-1) are the same as that defined in <ref>. 



  
Conjugate gradient



The only remaining computationally demanding step in <ref> is the computation of _^-1 with =_, ( - _ - ), requiring a time complexity of (n_sg^3) where n_sg= |(η,d) | is the size of the sparse grid (η,d). To reduce this computational intensity, we consider using the conjugate gradient (CG) method <cit.>, an iterative algorithm for solving linear systems efficiently via matrix-vector multiplications. Preconditioning <cit.> is a well-known tool for accelerating the convergence of the CG method, which introduces a symmetric positive-definite matrix _≈_ such that _^-1_ has a smaller condition number than _. The entire scheme of the preconditioned conjugate gradient (PCG)  takes the form in <Ref>.




  
Preconditioner choice
The choice of the precondition _ has a trade-off between the smallest time complexity of applying _^-1 operation and the optimal condition number of _^-1_. <cit.> first investigated several groups of preconditioners for radial basis function kernel, then <cit.> proposed a specialized precondition for general kernels based on pivoted Cholesky decomposition. <cit.> analyzed and summarized convergence rates for arbitrary multi-dimensional kernels and multiple preconditioners. 
Due to the hierarchical structure of the sparse grid , instead of using common classes of preconditioners, we consider two-level additive Schwarz (TAS) preconidtioner <cit.> for the system matrix _, which is formed as an additive Schwarz (AS) preconditioner <cit.> coupled with an additive coarse space correction. Since the sparse grid (η,d) defined in <ref> is covered by a set of overlapping domains {_t}_t∈(η) where (η) = {t∈^d |∑_j=1^d t_j = η}, the TAS precondition for the system matrix _ is then defined as

    _TAS^-1 :=  ∑_t∈(η)_t^⊤_t^-1_t + _c^⊤_c^-1_c.

On the right hand side of <ref>, the first term  is the one-level AS preconditioner, and the second term is the additive coarse space correction. _t∈^ n_t× n_sg is the selection matrix that projects _t to , _t = _t__t^⊤∈^ n_t× n_t is a symmetric positive definite sub-matrix defined on _t. _c ∈^ n_c × n_sg is the selection matrix whose columns spanning the coarse space, and _c = _c __c^⊤∈^ n_c × n_c is the coarse space matrix. n_c is the dimension of the coarse space, n_sg:=|(η,d) | is the size of the sparse grid (η,d), n_t:=|_t| =∏_j=1^d t_j is the size of the full grid design _t=×_j=1^d _j,t_j, t=(t_1,…,t_d). <Ref> illustrates an example of the selection matrices {_t}_t∈(η) for the sparse grid (η=3,d=2).




Some recent work  <cit.> provide robust convergence estimates of TAS preconditioner, which states that the condition number of _TAS^-1_ is bounded by the number of distinct colors required to color the graph of _ so that span{_t}_t∈(η) of the same color are mutually _-orthogonal, the maximum number of subdomains that share an unknown, and a user defined tolerance τ. Specifically, given a threshold τ>0, a GenEO coarse space (see Section 3.2 in <cit.>) can be constructed, and the following inequality holds for the preconditioned matrix _TAS^-1_:

    κ( _TAS^-1_ ) ≤ (k_c + 1) ( 2 + (2k_c+1) k_m/τ),

where k_c is the minimum number of distinct colors so that each two neighboring subdomains among {_t}_t∈(η) have different colors, k_m is the coloring constant which is the maximum number of overlapping subdomains sharing a row of _. Since each point in (η,d) belongs to at most |(η) | of the subdomains, k_c=|(η) |. Coloring constant k_m can be viewed as the number of colors needed to color each subdomain in such a way that any two subdomains with the same color are orthogonal, therefore k_m=|(η) |.

However, the conventional approach for constructing a coarse space to bound the smallest eigenvalue of the _TAS^-1_ relies on the algebraic local symmetric positive semidefinite (SPSD) splitting <cit.>, which is computationally complicated. To select a coarse space that is practical and less complicated, we propose constructing it based on the sparse grid (η_c,d) with a lower level η_c < η. In practice, we set η_c=max{⌈η/2 ⌉, d}. Although the TAS preconditioners with our proposed coarse space do not theoretically guarantee robust convergence, empirical results presented in <Ref> indicate that TAS tends to converge faster than one-level AS preconditioner and CG method without preconditioning. <Ref> shows the cases where Σ_ is ill-conditioned, resulting in the instability (blow-up) of the CG method without any preconditioning.





  
Complexity Note that _t is a sparse matrix with n_t nonzeros and all of its nonzeros are one, hence computing _t requires (n n_t^2) operations, and the computation of the first term of _TAS^-1 costs ( ∑_t∈(η) n n_t^2 ). The computational complexity of the preconditioner _PAS^-1 is then ( ∑_t∈(η) n n_t^2 + n_c^2) = (n |(η)| (η / d)^2d) = (n n_sg (η / d)^2d) where n_c=|(η_c,d)| is the size of the sparse grid according to the former setting.

The complexity of the matrix-vector multiplication Σ_0 in <Ref> is (n n_sg) if =_. In theory, the maximum number of iterations of the PCG with error factor ϵ is bounded by ⌈1/2√(κ)ln( 2/ϵ) ⌉ <cit.>, where κ is the condition number of the preconditioned matrix _TAS^-1_, so the time complexity of <Ref> for =_ is (( √(κ) + (η/d)^2d )  n_sg n ) where κ = κ(_TAS^-1_). Therefore the time complexity of the entire posterior sampling scheme is ( (η + (√(κ) + (η/d)^2d + m) n ) n_sg), where ν is the smoothness parameter of the Matérn kernel, n_sg is the size of inducing points on a sparse grid (η,d), n is size of observations ={_i}_i=1^n, m is the size of test points ^*={_i^*}_i=1^m, d is the dimension of the observations.




§ EXPERIMENTS


In this section, we will demonstrate the computational efficiency and accuracy of the proposed sampling algorithm. We first generate samples from GP priors and posteriors for dimension d=2 and d=4 in <Ref>. Then we apply our method to the same real-world problems as those addressed in <cit.>, described in <Ref>. For prior samplings, we use the random Fourier features (RFF) with 2^6=64 features <cit.> and the Cholesky decomposition method as benchmarks. For posterior samplings, we compare our approach with the decoupled algorithm <cit.> using RFF priors and the exact Matheron's update, as well as the Cholesky decomposition method. We use Matérn kernels as defined in <ref> with the variance σ^2=1, lengthscale ω = √(2ν) and smoothness ν=3/2. Our experiments employ “separable” Matérn kernels as specified in <ref>, with the same base kernels and the same lengthscale parameters ω=√(2ν) in each dimension. For the proposed method, the inducing points are selected as a sparse grid (η,d) as defined in <ref> for (η=5,d=2), and (η=6,d=4) respectively. We set the same nested sequence ∅ = _j,0⊆…⊆_j,t⊆_j,t+1⊆… for each dimension j, and set _j,t_j as hyperbolic cross points defined in <ref> for all j. We set the noise variates ϵ∼(,σ_ϵ^2 ) with σ_ϵ^2=10^-4 for all experiments. The seed value is set to 99, and each experiment is replicated 1000 times.



 §.§ Simulation




  
Prior sampling
We generate prior samples ={z_i}_i=1^n_s over the domain [0,1]^d using Matérn 3/2 kernels, with n_s=2^6,…,2^13 respectively. Left plots in <Ref> and <Ref> show the time required by different algorithms to sample from the different number of points n_s. We can observe that the proposed algorithm (InSG) is comparable in efficiency to the RFF method. To evaluate accuracy, we compute the 2-Wasserstein distances between empirical and true distributions. The 2-Wasserstein distance measures the similarity of two distributions. Let f_1∼(μ_1,Σ_1) and f_2∼(μ_2,Σ_2), the 2-Wasserstein distance between the Gaussian distributions f_1, f_2 on L^2 norm is given by <cit.>

    W_2(f_1,f_2) :=    ( ||μ_1 - μ_2||^2 +  tr(Σ_1 + Σ_2 - 2(Σ_1^1/2Σ_2Σ_1^1/2)^1/2) )^1/2.

For the empirical distributions, the parameters are estimated from the replicated samples. The right plots in <Ref> and <Ref> demonstrate that the proposed algorithm performs with nearly the same precision as Cholesky decomposition
and RFF.







  
Posterior sampling
We use the Griewank function <cit.> as the test function, defined as:

    f()=∑_j=1^d x_j^2/4000 + ∏_j=1^dcos(x_j/√(j)) + 1,  ∈[-5,5]^d.

 The sparse grid (η,d) is set over [-5,5]^d with configurations (η=5,d=2) and (η=6,d=4) for the design of our experiment. We then evaluate the average computational time and 2-Wasserstein distance over m=1000 random test points for each sampling method. <Ref> and <Ref> illustrate the performance of different sampling strategies. It's observed that our proposed algorithm is significantly more time-efficient compared to the Cholesky and decoupled algorithms, particularly when n is larger than 2^10. Moreover, our proposed method, the inducing points approximation on the sparse grid (InSG) demonstrates comparable accuracy to the decoupled method.




 §.§ Application




  
Thompson sampling Thompson Sampling (TS) <cit.> is a classical strategy for decision-making by selecting actions x∈ that minimize a black-box function f:→. At each iteration t, TS determines _t+1∈min_∈(f|)(), where  represents the observation set at current iteration. Upon finding the minimizer, the corresponding y_t+1 is obtained by evaluating f at _t+1, and then the pair (_t+1,y_t+1) is added to the training set. In this experiment, we consider the Ackley function <cit.>

    f() = -a exp( -b √(1/d∑_j=1^d x_j^2)) + a + exp(1)
         - exp( 1/d∑_j=1^dcos(c x_j) ),

with a=20, b=0.2, c=2π.
The goal of this experiment is to find the global minimizer of the target function f. We start with 3 initial samples, then at each iteration of TS, we draw a posterior sample f | on 1024 uniformly distributed points over the interval [-5,5]^d conditioned on the current observation set. Next, we pick the smallest posterior sample for this iteration, add it to the training set, and repeat the above process. This iterative approach helps us progressively approximate the global minimum. To neutralize the impact of the lengthscales and seeds, we average the regret over 8 different lengthscales [0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6] and 5 different seeds [9, 99, 999, 9999, 99999]. In <Ref>, we compare the logarithm of regret at each iteration for different sampling algorithms, the proposed approach (InSG) demonstrates comparable performance to the decoupled method.





  
Simulating dynamical systems
GP posteriors are also useful in dynamic systems where data is limited. An illustrative example is the FitzHugh–Nagumo model <cit.>, which intricately models the activation and deactivation dynamics of a spiking neuron. The system's equations are given by:

    = f(,a) = [ v̇;  ẇ ]=[ v -v^3/3 - w + a;  1/γ (v-β w + α) ].

Using the Euler–Maruyama method <cit.>, the system's equations <ref> can be discretized into a stochastic difference equation as follows:

    _t = _t+1 - _t = τ f(_t,a_t) + √(τ)ϵ_t,   ϵ_t ∼(,σ_ϵ^2 ),

where τ is the fixed step size. Write a FitzHugh–Naguomo neuron <ref> in the form <ref>, we can obtain:

    _t+1-_t =τ[ v_t - v_t^3/3-w_t + a_t;    1/γ (v_t - β w_t +α) ]
        + √(τ)ϵ_t.





Our objective is to simulate the state trajectories of this dynamical system. Starting with an initial point _0=(v_0,w_0)^T, at iteration t, we draw a posterior GP sampling from the conditional distribution p(_t| D_t-1), where D_t-1 denotes the set of the training data {(_i,_i)}_i=1^n along with the current trajectory {(_j,_j)}_j=1^t-1. In our implementation, we choose the model <ref> with step size τ = 0.25, parameters α=0.75, β=0.75, γ=20, noise variance σ_ϵ^2 = 10^-4, and initial point _0=(-2.5,-1)^⊤. The training set {(_i,_i)}_i=1^n was generated by evaluating <ref> at n=256 points {_i}_i=1^n (_i ∈), which are uniformly distributed in the interval = [-2.5,2.5]×[-1,2], with corresponding currents {a_i}_i=1^n (a_i ∈) chosen uniformly from =[0,1]. Variations _t in each step were simulated using independent Matérn 3/2 GPs with ω=√(3) and inducing points approximation on the sparse grid (η=5,d=2). We generate 1000 samples at each iteration. <Ref> presents the state trajectories of voltage v_t and recovery variable w_t versus the iteration step t for each algorithm. The left plots in <Ref> show that our proposed algorithm can accurately characterize the state trajectories of this dynamical system. <Ref> illustrates the computational time required for each iteration and the 2-Wasserstein distance between the states _t derived from GP-based simulations and those obtained using the Euler–Maruyama method as described in <ref> at each iteration t. These results demonstrate that our proposed algorithm not only reduces the computational time but also preserves accuracy that is comparable to the decoupled method.






















§ CONCLUSION


In this work, we propose a scalable algorithm for GP sampling using inducing points on a sparse grid, which only requires a time complexity of ( (η+n_s) n_sg) for n_s sampling points, a sparse grid of size n_sg and level η. Additionally, we show that the convergence rate for GP sampling with a sparse grid is ( n_sg^-ν (log n_sg)^(ν+2)(d-1)+d+1 ) for a d-dimensional sparse grid of size n_sg and a product Matérn kernel of smoothness ν. For the posterior samplings, we develop a two-level additive Schwarz preconditioner for the matrix over the sparse grid, which empirically shows rapid convergence. In the numerical study, we demonstrate that the proposed algorithm is not only efficient but also maintains accuracy comparable to other approximation methods.




This work is supported by NSF grants DMS-2312173 and CNS-2328395.






§ THEOREMS



 §.§ Rate of convergence


		Let M=L_q([0,1]) for any q∈[1,+∞). Let Z be a GP on [0,1] with a Matérn kernel defined in <ref> with smoothness ν, and Ẑ be an inducing points approximation of Z under the inducing points {0,1/n,2/n,…,1} for some positive integer n. Then for any p∈[1,∞), as n→+∞, the order of magnitude of the approximation error is
		
    W_p(Z,Ẑ)=(n^-ν).

	
	
	
		Note that Z and Ẑ already live in the same probability space, that is (Z,Ẑ)∈Γ(Z,Ẑ). Therefore,
  
    W_p(Z,Ẑ)≤(𝔼Z-Ẑ^p_L_q([0,1]))^1/p.

		According to Theorem 4 in <cit.>, Z-Ẑ_L_q([0,1]) has the order of magnitude O(n^-ν) and sub-Gaussian tails, which leads to the desired result.
	




      [Adapted from Proposition 1 and Corollary 2 in <cit.>]  Let Φ:^d → be a reproducing kernel of W_2^r;⊗^d(^d), where Φ is the tensor product with Φ():=∏_j=1^d ϕ(x_j) and =(x_1,…,x_d)^⊤, ϕ:→ is a reproducing kernel of Sobolev space W_2^r (^1) of smoothness r>1/2 measured in L_2(^1)-norm. Suppose we are given the sparse grid (η,d) defined in <ref> over I^d=[0,1]^d with n=card((η,d)) points and an unknown function f ∈ W_2^r;⊗^d(I^d). Let s_0 be the solution to the norm-minimal interpolant on the sparse grid (η,d) as follows:
    
    min‖ s ‖_W_2^r;⊗^d : s ∈ W_2^r;⊗^d(^d)   with   s(ξ)=f(ξ),  ξ∈(η,d).

    Then, we have 
    
    ‖ f-s_0 ‖_L_∞(I^d)≤ C n^-r+1/2 (log n)^(r+3/2)(d-1)+d+1‖ f ‖_W_2^r;⊗^d(^d),

    where C>0 is a constant.


    See proofs of Proposition 1 and Corollary 2 in <cit.>.




    Let M=L_q([0,1]^d) for any q∈[1,+∞), d>1 is the dimension of the space M. Let Z be a GP on [0,1]^d with a product Matérn kernel K(,')= ∏_j=1^d K_0 (x^(j),x'^(j)) defined in <ref> with variance σ^2=1 and base kernel K_0(·,·) of smoothness ν, and let Ẑ be an inducing points approximation of Z under the inducing points on a sparse grid (η,d) defined in <ref>. We denote the cardinality of the sparse grid (η,d) by n.  Then, as n→+∞, the order of magnitude of the approximation error is
    
    W_2(Z,Ẑ) = ( n^-ν (log n)^(ν+2)(d-1)+d+1 ) .


	

    If ϕ in <Ref> is a Matérn kernel in one dimension, we can infer that r=ν+1/2
    by Lemma 15 in <cit.>, which states that the reproducing kernel Hilbert space (RKHS) _K(^1) is a Sobolev space W_2^ν+1/2(^1) with equivalent norms. We denote RKHS associated with product Matérn kernel K(·,·) by _K^⊗^d(^d) and its norm by ‖·‖_K.
    
    Let  be the interpolation of the function f∈_K^⊗^d(^d) with respect to the approximation Ẑ. Clearly, _K^⊗^d(^d) = W_2^ν+1/2;⊗^d(^d). Proposition 2 in <cit.> asserts that =s_0 where s_0 is norm-minimal interpolant defined in <Ref>. Therefore, for any f ∈_K^⊗^d(^d), we have
    
    ‖ f - ‖_L_∞([0,1]^d)≤ C n^-ν (log n)^(ν+2)(d-1)+d+1‖ f ‖_W_2^ν+1/2;⊗^d(^d).


    Additionally, Lemma 15 in <cit.> implies that
    
    ‖ Z-Ẑ‖_L_q[0,1]^d^2
               = sup_‖ f ‖_K ≤ 1| f - |^2.

    Note that Z and Ẑ already live in the same probability space, that is (Z,Ẑ)∈Γ(Z,Ẑ). Therefore, by combining with <ref> and <ref>, we can obtain the following results:
    
    W_2(Z,Ẑ)
             ≤(𝔼Z-Ẑ^2_L_q[0,1]^d)^1/2
        = sup_‖ f ‖_K ≤ 1| f - |
       ≤sup_‖ f ‖_K ≤ 1 C n^-ν (log n)^(ν+2)(d-1)+d+1‖ f ‖_K
       ≤ C n^-ν (log n)^(ν+2)(d-1)+d+1.

    Thus, the order of the magnitude of the inducing points approximation on the sparse grid error is W_2(Z,Ẑ)=(n^-ν (log n)^(ν+2)(d-1)+d+1).




 §.§ Smolyak algorithm for GP sampling



    Let K(·,·) be a kernel function on ^d ×^d, and =(η,d) a sparse grid design with level η and dimension d defined in <ref>. (η,d)=⋃_t∈(η)_1,t_1×_2,t_2×⋯×_d,t_d
    where t=(t_1,…,t_d), (η) = {t∈^d |∑_j=1^d t_j = η}, η≥ d≥ 2, η,d∈^+. Let f(·) ∼(, K(·,·)) be a GP, then sampling from f(·) on a sparse grid  can be done via the following formula:
    
    f() ←∑_t∈(η,d) (-1)^η - |t|d-1η - |t|⊗_j=1^d chol( K(_j,t_j, _j,t_j) ) __n_tδ_x(_t),

    
    δ_x(_t) := 
                0      if x ∈∖_t
    
                1      if x ∈_t,     _t:=_1,t_1×_2,t_2×⋯×_d,t_d,

    where (η,d) = {t∈^d |max(d,η-d+1) ≤|t|≤η}, chol( K(_j,t_j, _j,t_j) ) is the Cholesky decomposition of K(_j,t_j, _j,t_j), __n_t∼(,_n_t) is standard multivariate normal distributed, n_t = ∏_j=1^d t_j is the size of the full grid design _t=_1,t_1×_2,t_2×⋯×_d,t_d associated with t=(t_1,…,t_d), δ_x(_t) is a Dirac measure on the set .



    We know that sampling from f() ∼(, K(,)) can be done via Cholesky decomposition, i.e., by computing chol(K(,) ) _n_sg where _n_sg∼(, _n_sg) is multivariate normal distributed on the sparse grid  of size n_sg=||. Note that chol(K(,) ) _n_sg can be regarded as a function g evaluated on  such that g()=chol(K(,) ) _n_sg. According to <ref>, we can obtain the following equation by applying Smolyak's algorithm (η,d) to g,
    
    (η,d) ( g )    = ∑_t∈(η,d) (-1)^η - |t|d-1η - |t|( ⊗_j=1^d _j,t_j) (g)
       = ∑_t∈(η,d) (-1)^η - |t|d-1η - |t|
        g(×_j=1^d _j,t_j)
        δ_x (_t)
       = ∑_t∈(η,d) (-1)^η - |t|d-1η - |t|⊗_j=1^d
        chol( K(_j,t_j, _j,t_j) ) _n_tδ_x (_t),

    where δ_x(_t) and _t are defined in <ref>. Note that g is defined on , so the tensor product operation ⊗_j=1^d _j,t_j on g is actually evaluating g on the corresponding points, which is equivalent to applying Dirac delta measure defined on set ×_j=1^d _j,t_j to g(⊗_j=1^d _j,t_j).





§ PLOTS

















apalike






 Paper Title: Distributionally Robust Optimization as a Scalable Framework to Characterize Extreme Value Distributions.


Energy-lowering symmetry breaking creates a flat-band insulator in paramagnetic Nb_3Cl_8
    Alex Zunger
    August 5, 2024
========================================================================================



§ ABSTRACT 
The goal of this paper is to develop distributionally robust optimization (DRO) estimators, specifically for multidimensional Extreme Value Theory (EVT) statistics. EVT supports using semi-parametric models called max-stable distributions built from spatial Poisson point processes. While powerful, these models are only asymptotically valid for large samples. However, since extreme data is by definition scarce, the potential for model misspecification error is inherent to these applications, thus DRO estimators are natural. In order to mitigate over-conservative estimates while enhancing out-of-sample performance, we study DRO estimators informed by semi-parametric max-stable constraints in the space of point processes. We study both tractable convex formulations for some problems of interest (e.g. CVaR) and more general neural network based estimators. Both approaches are validated using synthetically generated data, recovering prescribed characteristics, and verifying the efficacy of the proposed techniques. Additionally, the proposed method is applied to a real data set of financial returns for comparison to a previous analysis. We established the proposed model as a novel formulation in the multivariate EVT domain, and innovative with respect to performance when compared to relevant alternate proposals. 




§ INTRODUCTION



Modeling rare and extreme events is an important task in many disciplines such as finance, climate science, and medicine <cit.>. Estimating distributions of rare events from data is difficult due to the lack of observations within this region, making it challenging to understand the risks deep in the tail of a distribution. Extreme Value Theory (EVT) studies the class of distributions arising as the possible distributional limits that can be used to estimate multivariate distributions in distant (relative to the origin) regions (i.e., tails) which by their nature witness very few observations (or none at all). These distributional limits are derived as the possible asymptotic statistical laws of shifted and re-scaled data as the sample size increases. It turns out that such possible distributional limits form a semi-parametric class called max-stable distributions, which are constructed in terms of a spatial Poisson point process. 

Naturally, because of the lack of data in extremal regions and because of the asymptotic nature of max-stable models, their use in inferential tasks involving tails is exposed to high variance due to model misspecification. Moreover, when using max-stable models, the assumptions that the data converges to the distributions specified by EVT must be made. This leads to an important question: How can we robustify against scenarios deep in the tails while observing potentially sub-asymptotic data where the assumptions of EVT may be violated? To answer this question, we propose a solution based on distributionally robust optimization (DRO). DRO involves a zero-sum game in which the statisticians play against an adversary that perturbs (in a non-parametric way) the nominal / baseline distribution assumed by the statistician. Building on classical DRO, we carefully design constraints to retain the extrapolation properties of EVT for the robustified distribution.

Since we are interested in robustifying the tails, we will consider max-stable baseline distributions given by extreme value distributions (EVDs), as presented by <cit.>. Max-stability roughly states that the distribution of the maximum of independently and identically distributed (i.i.d.) samples belongs to  the same distribution up to a change in the location and scale parameters. This means that the “shape” of the distribution is preserved under the max operation, and it is this property that allows for extrapolating to regions outside the observation domain. In our robustification framework, we therefore wish to preserve this max-stability property. We do this by searching over the space of distributions that are also max-stable, in effect constraining our search to only distributions that extrapolate according to EVT, which form a semi-parametric class. We do this by carefully designing the uncertainty set such that the necessary max-stable properties are preserved. 

To illustrate our desired result, we refer to Figure <ref>. This figure visualizes how a completely unconstrained adversary (solid line) may be too conservative and may not consider the extrapolating properties of the distribution. A well constrained adversary to a max-stable distribution (dashed line) provides an appropriate balance of coverage while maintaining underlying structural properties. This figure demonstrates a case when even if we consider two adversarial formulations that achieve a similar minimum error value, the properly constrained adversary (dashed line) is preferable because the size of uncertainty is very hard to calibrate. Therefore, a curve that is "flatter" around the minimum as a function of the uncertainty size parameter is preferable.




Our work focuses on the case of multivariate extreme value distributions (MEVs) which characterize the joint risk between different variables. Unlike univariate EVDs which have a fully parametric form, MEVs are semi-parametric, where the dependence structure is an infinite dimensional object, and much harder to estimate. We focus on robustifying the dependence structure while preserving the MEV character. This representation leads to additional difficulties which we overcome through the proposed DRO framework.

In view of the fact that the lack of data will naturally induce model error, we introduce an approach to quantify model misspecification based on optimal transport DRO. We select the Wasserstein metric for optimal transport because this approach, together with moment constraints, encompasses most DRO formulations as demonstrated by <cit.>.

The cost structure in the optimal transport discrepancy allows one to balance various objectives when modeling data, specifically the trade-off between tractability and control of pessimism in extremal behavior. A too-powerful adversary makes the size of uncertainty hard to calibrate since a slight increase may result in adversarial policies that perturb a distribution in ways that are too pessimistic and may not be consistent with the constraints imposed by EVT.

We mitigate these concerns by studying DRO formulations which constrain the adversarial perturbations to explore non-parametric models that induce robust estimators while preserving MEV characteristics. One of the approaches we employ is based on optimal transport for point processes, while the other is based on a neural network architecture. We illustrate the performance of the neural network architecture in the context of a Conditional Value at Risk (CVaR) metric applied to a multi-variable extreme value distributed data set. The network is evaluated across several synthetic data sets, specifically constructed to challenge the assumptions associated with EVT. The neural network architecture uses the tractability of the optimal transport distance metric to parameterize model uncertainty. Furthermore, our analysis is extended to a real data set of financial returns as a baseline comparison, similar to the data proposed by <cit.>.This experiment demonstrates precisely the anticipated behavior shown in Figure <ref>, where our methodology leverages a properly constrained adversarial estimator.

Related Work
A number of related research directions exist that consider both estimating EVDs from data as well as robustifying them using DRO. Since our focus is on multivariate EVDs, we will review a few of the estimators from the literature. For estimating multivariate EVDs, copula based approaches have been developed in a variety of instances including work by  <cit.>. Samplers for MEVs have also been considered as demonstrated by  <cit.>. <cit.> provides a flexible framework that uses neural networks to estimate and sample from multivariate EVDs irrespective of dimension, which we use in the computational component of this work. However, all of these methods only consider the case where the model is well-specified and do not consider uncertainty associated with the model class. We build upon these works with the addition of the DRO perspective. 

With regards to the DRO literature, <cit.> described the general framework for Wasserstein DRO that we will use throughout this work. Additional discussion is available in  <cit.> and  <cit.>. Other DRO frameworks for EVT have been considered, but only in the univariate case, by  <cit.>. <cit.> considers DRO under the Kullback-Liebler divergence, which is appropriate for the univariate analysis but may not be appropriate for the multivariate case where supports are likely to be disjoint.

 Finally, <cit.> considers DRO estimators specifically for MEV distributions leveraging extremal coefficient constraints. 
 In this work, upper and lower bounds are established on a Value at Risk (VaR) loss and applied to a real data set of financial returns. 
 These bounds are established over the infinite domain of spectral measures, using a finite set of constraints formulated as a linear semi-infinite program. 
 This is a limited subset of application problems when compared to our investigation. 
 While the general goal of <cit.> is similar to our framework, our method extends to more general risk measures and considers a flexible uncertainty set specified by the Wasserstein distance. 
 Furthermore, we achieve a single robust loss (upper bound), as opposed to less precise (upper and lower) boundaries. 
 In Section <ref> we generate a similar data set to <cit.> and employ our proposed method for comparison. 
 In summary, the main contributions of this paper are: 


  * We introduce a framework to produce DRO estimators for MEV distributions based on optimal transport.


  * We provide tractable DRO formulations for various estimators of interest when adversaries live in the (infinite dimensional) space of point processes.


  * We test the performance of our estimators with the goal of showing that our MEV-constrained adversaries improve performance in the sense of Figure <ref>, and compare to a previous work for baseline analysis.



§ BACKGROUND AND PROBLEM FORMULATION
 
In this section we focus our discussion on concepts critical to our proposed results and we introduce the framework for distributionally robust estimators in MEV distributions.
We first provide a brief overview of EVT and describe how it is used to extrapolate beyond the observed data. 
We then introduce the multivariate counterpart, which we use throughout this work, to describe EVT in multi-dimensional settings.
Finally, we discuss the distributionally robust optimization framework we use based on the Wasserstein distance. 



 §.§ Extreme Value Theory Background


We begin by reviewing concepts surrounding MEV distributions. Consider a sequence of n i.i.d. random vectors {X^(1), , X^(n)}, with X^(i)∈ℝ^d and i = 1, , n and denote the maxima over each dimension as M_k,n:=max_i =1^n X_k^(i), where k ∈{ 1, …, d }. Similarly to univariate EVT analysis, we consider MEV distributions G, where P((M_1,n- b_1,n)/a_1, n≤ z_1, , (M_d,n- b_d,n)/a_d, n≤ z_d) → G(z_1, , z_d), for some normalizing constants a_k, n > 0 and b_k, n, as the number of observations n increases to infinity. 





 §.§ Spectral Representation of Component-wise Maxima and Asymptotic Characterization


We will now introduce specific properties of MEV distributions that we will exploit in our framework. Consider i.i.d. random vectors {X^(1), , X^(n)}, with X^(i)∈ℝ^d and i = 1, , n, with unit Frechét margins such that F_X_k(x) = exp(- 1 /x), with x > 0 for all k = 1, …, d. Following <cit.>, let a sequence N_1, , N_n be a point process, where N_n(·)=∑_i=1^n1_X^(i)/n(·) with N_n(·)d→ N(·) as n →∞ with d denoting convergence in distribution and N is a Poisson point process. We will apply this result further in Section <ref> to define the proposed robustificaiton of the Poisson point process.



  
Decomposing Max-Stable Random Variables
Max-stable distributions can be decomposed according to the radial and spectral decomposition <cit.>. Specifically, if we let Y^(n)∈ℝ^d ∼ H be a sample from the spectral distribution and A^(n) to be the n^th arrival of a unit rate Poisson point process, then a max-stable random variable is represented as


    M = max_n≥ 1Y^(n)/A^(n).


Under the condition that 𝔼[Y_k] =1, the variable M is distributed with unit Frechét margins. This decomposition provides a semi-parametric class of distributions whose structure we will use throughout the rest of the text. With the spectral decomposition of the MEV we are now able to analyze an MEV distribution. Additionally, we introduce Lemma <ref> which allows for the explicit transformation of the MEV cumulative distribution function (CDF). 


    For X^(k) with standard Frechét marginal distributions, we have:
    
    P   (M_1,n/n≤ x_1,⋯, M_d,n/n≤ x_d ) 
       →exp(-V(x_1,…,x_d)),

    where
    
    V(x_1,⋯,x_d)=d∫_Δ_d-1max_k = 1^dw_k/x_kH(dw),

    Δ_d-1 represents the unit d-dimensional simplex and H(·) satisfies: ∫ w_k H(dw)=d^-1 for all k ∈{1, …, d}.


When dealing with MEVs, the important component in estimation is the spectral measure H. When placed in its standardized form, H is a measure with support over the simplex that describes the dependence between the covariates, specifically which components become extreme simultaneously. When a face of the simplex has H non-zero, it implies that covariates associated with the vertices of the face experience extreme events simultaneously. For more details on estimating the spectral measure, we refer the reader to <cit.>. In Section <ref>, we describe how H plays the role of an intensity function when taken in the perspective of the point process.



 §.§ Wasserstein DRO Formulations for EVDs

Referencing the framework to quantify model uncertainty via DRO described by <cit.>, we define the probability space (Ω, ℱ, P), where a candidate robust distribution P is feasible such that d(P, P_0) ≤δ. P_0 is a baseline distribution and d(·) is a distance measure, constrained by the parameter δ. Two methods are commonly employed to quantify model uncertainty when constructing distributional ambiguity sets. The first considers the corruption of the likelihood baseline model to be misspecified, which is addressed via ϕ-divergence ambiguity sets. The second method perturbs the actual data, which leads to the use of Wasserstein distances to quantify model misspecification.  

A recent investigation by <cit.> has demonstrated that both considerations of likelihood and perturbations of data may be unified under the Wasserstein distance, where d(·) = W_c(·). Consider a loss function ℓ : ℝ^d →ℝ_+ and the Wasserstein distance transport cost c: ℝ^d ×ℝ^d →ℝ_+. We define the primal optimization problem as follows:

    max_P : W_c(P, P_0) ≤δ𝔼_X ∼ P[ℓ(X)]

In the context of our problem, P_0 could be the estimated distribution from available samples, but one may not have enough coverage since rare events may not have been observed within the data collection period. Following <cit.>, the dual form of (<ref>) is given by the following problem

    min_λ≥ 0 [ λδ + 𝔼_X ∼ P_0[ max_Zℓ(Z) - λ c(X, Z)  ]  ].

As we will discuss, the dual form is particularly amenable for computational purposes, especially when one can exploit specific properties of c, ℓ, and P. Our investigation will describe two perspectives of (<ref>) for EVDs. The first we present in Section <ref>, and is based on interpreting an MEV through a point process and the second is based on the interpretation using a copula. Each perspective has different properties that are useful for computational purposes, which we will discuss in Section <ref>. 


§ ROBUSTIFICATION OF THE POISSON POINT PROCESS


In this section, we provide a more general formulation for DRO problems using a characterization with a point process. 
For the probability space (Ω, ℱ, P), we denote a counting measure N(·) on a Polish space (S,d). 
We define N(·)=∑_i=1^∞1_x_i(·) and N'(·)=∑_i=1^∞1_y_i(·) as two random counting measures. 
Additionally, we define scaling functional κ: ℤ→ℝ_≥0, and a lower semi-continuous cost function c:S× S→ℝ_≥0. 
Let σ(·) denote the permutation function defined on the support of N(·), and we then define the distance as:

    c(N(·), N'(·))=   ∞1_N(S)≠N'(S) or N(S)∨ N'(S)=∞
       + κ(N(S))inf_σ(·)∑_i=1^N(S)c(x_i, y_σ(i)).


This distance is also similar to the ones presented by <cit.>. Additionally, we define the 1-Wasserstein distance between two random counting measure by:

    W_c(μ,ν)=inf_γ∈Γ(μ,ν)𝔼_(N, N')∼γc(N, N'),

Here μ is the measure corresponding to N(·) while ν is the measure corresponding to N'(·). Analogously to standard DRO problems, we can write down a new formulation for this DRO problem with point process as follows:


Let N(·) denote a Poisson point process, f(·) denote a Borel function, and W_c(P, P_0) denote the Wasserstein distance between measures induced by N' and N under cost metric c. For a distance δ≥ 0, the following problems are equivalent:

    sup_ W_c(P, P_0) ≤δ𝔼_N' ∼ P[ℓ(N'(f))] = 
    inf_λ≥ 0   {λδ  + 𝔼_N∼ P_0 [sup_N' [ ℓ(N'(f) ) - λc(N, N') ] ] },

where N(f):=∫ f(x)N(dx).

    This follows from the application of <cit.>. 


As mentioned in the background material in (<ref>), MEV observations are given by the product of  radial (A^(n)) and spectral (Y^(n)) components. In the perspective of the point process, the associated measures for each of these components forms the intensity of the point process. Specifically, we let the atoms of the point process be given by (A^(n), Y^(n)) where A^(n) can be thought of as a time coordinate (since A^(n) is the n^th arrival time) and Y^(n) as a space (or the mark) component. Define N(da, dv) = ∑_n=1^∞1_((A^(n), Y^(n)) (da, dv). This provides a useful characterization in terms of the dual problem insofar as we can optimize over the boundaries of the integrals to form our adversary point process. As we will illustrate, this is equivalent to finding a new point process with modified intensity. 
 
With regards to the cost in the Wasserstein distance, we place an infinite cost on changes in the arrival times of the events. This has the interpretation that our uncertainty is not on the arrival time of the tail events but rather on the dependence. We formalize this through the following cost function between two points:

    c((s, u), (t,v)) = ∞1_s ≠ t + |u - v| 1_s = t

Our Wasserstein cost is then between the spectral component while fixing the radial component. We formalize this case in the following corollary.

    
    For the given setting in Theorem <ref>, if the point process satisfies N(x)=lim_n∑_i=1^n1_X^(i)/n(x), where X^(i) are i.i.d. standard Frechét margins. Then, we can rewrite the primal problem as:
    
    sup_W_c(P, P_0)≤δ𝔼_N∼ P[ℓ(N(f∘ T^-1)],

    where N(da, dv)=∑_n1_(A^(n), Y^(n))(da, dv) and T(X)=(1/X_1, X/X_1). 
    Under P_0, A^(n) is the n-th arrival of the Poisson point process with intensity 1 and Y^(n) follows the spectral measure H(·). 
    Here we introduce the composition of the Borel function f on the transformation T.

 



§ OPTIMIZATIONS FOR SPECIFIC LOSS FUNCTIONS


Having introduced the different perspectives of the optimization, we provide specific loss functions where each perspective is particularly conducive towards specific robustification cases.


 §.§ Robustifying the CDF


Consider the cumulative density function (CDF) of M∈ℝ^d satisfying M_i=max_n=1^∞Y_i^(n)/A^(n), where Y^(n)∈ℝ^d are i.i.d. random vectors and A^(n) is the n-th arrival time of a unit Poisson point process. 
The CDF of M satisfies:

    P(M_1 ≤ x_1^-1,    …,  M_d ≤ x_d^-1 ) = 
       exp ( - d∫_Δ^d-1max_i=1^d x_i w_i H(dw)).

Here, Equation <ref> is derived from Lemma <ref> by using the max stable decomposition of the random variable, as demonstrated in Equation <ref>. 
We will now exploit specific geometric properties of the point process to derive a dual form. 
Given the definition of M_i, we may make a transformation of the CDF algebraically as follows:

    P     (M_1 ≤ x_1^-1, …,  M_d ≤ x_d^-1 )= 
        P(max_k=1^d Y_k^(n)x_k≤ A^(n),∀ n ) :=P(V_x^(n)≤ A^(n),∀ n).

In this case, we can reduce our analysis to this two dimensional space and consider the robustification of the point process on this space, rather than the full d dimensional space of the Y^(n)'s. 
A schematic of this representation before robustification is given in Figure <ref>. Using this form leads to a dual form given by the following theorem:


Consider the setting in Theorem <ref> and the loss function ℓ(X) = 1_X≥ 1, the point process N(·):=∑_n=1^∞1_(A^(n),V^(n)_x)(·), the function f(a,v)=1_𝒞(a,v) and 𝒞={(a,v)∈ℝ^2 | a≥0,a≤ v}. Then, the dual objective is:

    inf_W_c(P, P_0)≤δP'(M_1 ≤ x_1^-1, …,  M_d ≤ x_d^-1 )
    
    =   1-inf_λ≥ 0λδ + 𝔼_P_0 [1 - λmin_n≥ 1( A^(n)- V_x^(n))^+  ].



See Appendix <ref>.


This formulation is advantageous in many ways. First, we reduced the original optimization problem over measures to one over a scalar variable λ. Second, this formulation includes the relevant constraints such that the perturbed distribution maintains the max-stability property of EVT. Third, the optimal solution is given by a shift in the rate function, as illustrated in the following corollary: 


The minimizer of the problem in Theorem <ref> has the following form:

    inf_P:W_c(P,P_0)≤δP(X_1 ≤ x_1^-1, …, X_d ≤ x_d^-1) 
       = exp ( - 𝔼_N∼ P_0 [N( 𝒞_1/λ^⋆)]  )

where 𝒞_1/λ^⋆ is the region 
{(a, v) : V_x^(n) > A^(n) - 1/λ^⋆}
and λ^⋆ is the minimizer of (<ref>).
 
 
     See Appendix <ref>.
 

The ideas behind Corollary <ref> are illustrated in Figure <ref> where the region over which the expectation is taken is shifted by 1/λ^⋆.

 

 §.§ Probability an Event in a Rare Set Occurs

In this part, we describe a generalization of the CDF case where we are interested in robustifying the probability that an event in the set A ⊂ℝ^d_+ occurs, i.e. P(N(A) ≥ 1). Our primal problem in this case is:
 
    sup_W_c(P, P_0) ≤δ P(N(A) ≥ 1) = sup_W_c(P, P_0) ≤δ𝔼_X ∼ P[1_X ∈ A]

 and we let the distance be any ℓ_p-norm.
 We can write the dual form and obtain the following simplification:
 
    Consider the setting in Theorem <ref> and the loss function ℓ(X)=1_X≥1, the point process N(·)=∑_n=1^∞1_X^(n)(·), and the function f(x)=1_A(x). Then the dual objective is:
    
    sup_W_c(P, P_0) ≤δ P(N(A) ≥ 1) 
    
        =   inf_λ≥ 0{λδ +  𝔼_P_0 ( 1- λinf_ninf_y ∈ A y - X^(n))^+ } .

    
 
  
 See Appendix <ref>.
 

 Relatedly, we can also compute the expected number of events to occur within a set using the following theorem:
 
     Consider the setting in Theorem <ref> and the loss function ℓ(X)=X, the point process N(·)=∑_n=1^∞1_X^(n)(·), and the function f(x)=1_A(x). The the dual objective is:
     
    sup_W_c(P, P_0) ≤δ𝔼_N ∼ P[N(A)]
    
             =   min_λ≥ 0 [ λδ + 𝔼_P_0∑_n=1^∞ ( 1 - λinf_y ∈ A y - X^(n))^+  ].

 
 

 §.§ Conditional Value at Risk

Conditional value at risk (CVaR) is a commonly used risk measure in finance. Often there is uncertainty around the calculation, so we derive an efficient form for the calculation. In the DRO case, we can write the CVaR problem with the given level α as:
 
    sup_P: W_c(P,P_0)≤δinf_z𝔼_P((∫⋁_k=1^d x_k N(dx)-z)^+/1-α+z).

 By exchanging the sup-inf to inf-sup, we can write the dual formulation as the following theorem:

 
      Consider the setting in Theorem <ref> and the loss function ℓ(X)=(X-z)^+/1-α+z, the point process N(·)=∑_n=1^∞1_X^(n)(·), and the function f(x)=x_∞. The the dual objective is:
    
    inf_zsup_P: W_c(P,P_0)≤δ𝔼_P((∫⋁_k = 1^d x_i N(dx)-z)^+/1-α+z)
    
             =   δ/1-α + 𝔼_P_0 [ ∑_n=1^∞X^(n)_∞ | ∑_n=1^∞X^(n)_∞ > q_α. ],

    where P_0(∑_n=1^∞X^(n)_∞≤ q_α)=α and q_1<∞.
 
 
 See Appendix <ref>.
 
Having introduced these simplifications, we see that the optimization problems for commonly used losses under the point process interpretation of EVT can be made feasible.
 



§ EXPERIMENTS AND RESULTS


We now consider empirical validation of the proposed optimization schemes using two experiments. The first uses a synthetic data set constructed to pathologically challenge our model and represent real world risk scenarios. 
The second validation is a baseline comparison using a real data set of financial returns similar to the DRO MEV experiment proposed by <cit.>. 
For the synthetic and real data experiments the observations are extreme; however, the standard EVD model is shown to not provide appropriate coverage for computing the worst-case risks. 
For both data sets, we provide empirical results validating that the proposed method satisfies the properties that we desired and outlined in the introduction, i.e. those of sufficient coverage while not being too conservative.




 §.§ Computational Implementations

For losses that have a reduced dual problem such as those in Section <ref>, we use the empirical data to represent the expectations and optimize over λ.
However, for general losses we use the equivalence stated in Corollary  <ref> to estimate the adversarial distribution.
Our procedure follows the method in <cit.> where we first estimate the spectral measure from the observations using a generative neural network and then optimize over the space of networks within some ball of the initial distribution.
The case for arbitrary ℓ is illustrated in Algorithm <ref>.







 §.§ Experiments with Synthetic EVT Data


We evaluate our proposed methodology on two synthetic two-dimensional EVD data sets.
 
We hypothesize that a properly formulated DRO approach, where max-stability is enforced via constraints, demonstrates a greater invariance to increased model misspecification relative to a non-constrained models. Our methodology is designed to account for the complexities of real data and produce more accurate and reliable predictions. 
Details concerning computational parameters, runtime, and code base reference is available in Appendix <ref>.



  §.§.§ Synthetic Datasets


For our synthetic data experiments, we consider mixture distributions where one mixture component with a more concentrated risk appears less frequently than the other mixture component. This is illustrated in Figures <ref> and  <ref>, where the ten thousand observations of the two datasets are visualized. In Figure <ref> the component has a smaller tail index and more realizations with smaller magnitude whereas in Figure <ref> the component has a larger tail index, but is more rare.  
For more information see Appendix <ref>.

A standard EVD approach would be sufficient when modeling a non-mixture Symmetric Logistic (SL) or Asymmetric Logistic (ASL) set of data. However, this mixture inserts a pathological modeling issue, which emulates the practical concerns often observed in real data where extrapolating to out-of-sample extreme values is difficult given the scarcity of data in these scenarios. 
These generated mixture distributions exhibit different tail behavior, which make it difficult for standard approaches to capture the dependencies accurately, and thus we turn to the proposed DRO estimator and evaluate our ability to recover the true risk.








  §.§.§ Synthetic Experiment Results



We now illustrate the results of the computational method for the loss corresponding to CVaR of the ℓ^1-norm of the observations.
Specifically, we aim to robustify the calculation of 1/α𝔼[X_1 1_X_1 ≤ x_α] where x_α = inf{ x : P(X_1 ≤ x) ≥α}.
As α→ 0, this corresponds to the CVaR for values deep in the tail.
We approximate this region as a MEV under both the SL and ASL distributions described previously. 
Our goal in this experiment is to understand the behavior of the error, defined as the difference between the true risk and the risk computed using the different methods, as a function of the budget δ.
We compare 3 different robustifications: (1) a totally unconstrained robustification where P can be in any class of distributions; (2) an EVT constrained case where P must also be an MEV and shares the same A^(n)'s; (3) an EVT constrained case where the margins are assumed to be well calibrated and only the dependence is modified (such that w ∼ P,   s.t.  𝔼[w_i] = 1/d).
In terms of standard EVT methods, (3) corresponds to the case where the adversarial distribution has a different dependence structure but maintains the same margins.
Specifically, this would suggest estimating the margins is easier and therefore should be conserved, but the dependence structure should be perturbed.
Figures <ref> and <ref> illustrates the error of using a DRO approach with max-stability constraints compared to the unconstrained approach, producing a distribution which is not of the MEV model class as functions of δ. 
This allows for the understanding of performance based on the behavior of the loss function as δ changes. 



  
Synthetic Experiment Results Analysis
In Figures <ref> and <ref>, we visualize that the models which are constrained to be max-stable, specifically P_⋆ and P_⋆ (𝔼[w_i] ≈1/d), demonstrate a lower and robust (positive) error when compared to the unconstrained model P_⋆ (unc). Additionally, we observe a slight improvement in the error of the constrained P_⋆ model for an increase of model uncertainty (δ) and subsequent monotonically increasing error as  uncertainty increases. This behavior confirms our conceptual illustration of the DRO max-stable constrained approach in Figure <ref>.




  §.§.§ Convergence of CDF as Distance is Increased

We now consider an experiment that corresponds to Theorem <ref> described in Section <ref>.
We illustrate the behavior in Figure <ref> where we see an improvement in the error for a wide range of values of δ. 





 §.§ Experiment on High Dimensional Financial Data


We consider a dataset on financial returns of equities in the S&P 500 index similar to the observations used in the validation experiments proposed by <cit.>. 
The dataset is composed of all current members of the S&P 500 who have been members since at least January 1983 to provide forty years of equity price data. 
We then average returns across companies, where each company is uniquely assigned to one of eleven industrial sectors (industrial, health care, consumer staples, etc.), leading to an eleven dimensional dataset. 
Maximum daily returns are taken both annually and weekly across the forty year data set, where each of these annual and weekly maximum data sets now constitute extreme observations.

This dataset is in the dimension of the number of industries considered, in this case d = 11. 
A visualization of this extreme data for highest weekly and annual returns is provided in Appendix <ref>. 
Obtaining a ground truth risk for real data is nearly impossible since we may never observe realizations deep in the tails, so validating against a ground truth risk is difficult.
To circumvent this, we consider training on less extreme data given by maxima over shorter time scales and use maxima over longer time scales as the ground truth risk.
Specifically, we fit the models using weekly maxima and test on yearly maxima.
The reason for this is that large events are more likely to occur over the longer time scale (the year) than over the day, leading to a more representative set of measurements deeper in the tails. 

We use the exact same methodology outlined in Section <ref> and illustrate the results of this experiment in  Figure <ref>.



  
Real Data Experiment Results Analysis
We observe that when δ = true risk, a minimal error is achieved between the true and calculated risk.
Additionally, this experiment demonstrates that for a real extreme data set the models constrained by EVT achieve a lower error level when compared to the unconstrained model, as anticipated.
We note that the comparable formulation in <cit.> does not provide a modifiable parameter such as our formulation does with δ, to adjust model uncertainty. 
This demonstrates the additional value of our proposed formulation over a comparable method. 



§ DISCUSSION

In this paper, we introduced a framework for robustifying MEV distributions according to the point process viewpoint. 
We provided a novel framework and discussed simplifications that are possible under certain cost functions. 
We numerically demonstrated that, as we proposed, adding additional constraints from EVT improves the robustification without being too conservative. 
The framework is evaluated on a synthetic data set which is constructed to challenge the model and a high-dimensional real world data set derived from a similar investigation, demonstrating expected and improved behavior. 
The proposed methodology has a variety of applications in many fields, particularly in risk sensitive settings where model misspecification may be present. 

There are many avenues for extending the proposed methodology.
One important extension involves the case where one must specify a policy to mitigate a worst-case risk.
This applies in, for example, portfolio optimization problems where the optimal portfolio minimizes the conditional value at risk over a distribution of portfolio losses.
Additionally, further exploration of the proposed formulation leveraging max-stable processes extended to a more general max-infinitely division process may be insightful. 



  
Limitations
There exist a number of limitations associated with the proposed framework. 
With regards to the computational method, there may be numerical instabilities associated with the optimization particularly in cases of δ being small. 
This is because the λ parameter must grow large, which leads to unstable optimization schemes.



 
Material in this paper is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-20-1-0397.










Distributionally Robust Optimization as a Scalable 

           Framework to Characterize Extreme Value Distributions 
 (Supplementary Material)
    Alex Zunger
    August 5, 2024
================================================================================================================================================





§ PROOFS






 §.§ Proof of Equivalence in Representations


Let us first recall the two problems that we are solving. 
On one hand, we take an explicit point process view point in:

    max_ W_c(P, P_0) ≤δ𝔼_N ∼ P[ℓ(N(f))],
    Point Process

where N(dx)=∑_nδ_X^(n)(dx) is a non-homogeneous Poisson process. By the properties of EVT, the intensity satisfies

    𝔼[N(f)] = ∫ f(T^-1(r,w)) r^-2dr H(dw).

By change of variables, we could decompose X^(n)=Y^(n)/A^(n), where A^(n) is the n-th arrival of Poisson process with intensity 1 and Y^(n)∼ H(·). Thus, we could rewrite the problem with:

    max_W_c(P, P_0)≤δ𝔼_N'∼ P[ℓ(N'(f∘ T^-1)],

where N'(dt, dy)=∑_n1_(A^(n), Y^(n))(dt, dy) and T(X)=(1/X_1, X/X_1).



 §.§ CDF proof




We now detail the proofs for specific loss functions, beginning with the cumulative distribution function (CDF), followed by the conditional value at risk (CVaR), probability, and expected number of events in rare sets. 


 
 Define the upper triangular region as 𝒰 = { (a, v) :  a ≥ 0, v ≥ a }.
 To compute the CDF, we want to condition on the Poisson process having no points within 𝒰, i.e. N(𝒰) = 0. 
 We can compute the rate of the point process according to 𝔼[N(𝒰)]. 
 Since this is a Poisson point process, the probability that no points occur within 𝒰 is given by
 exp(- 𝔼[V_x]).
 Having written the CDF in terms of the rate of the Poisson point process, we now follow through to describe the simplified form written in (<ref>).
 First, we write the dual problem when directly applying <cit.> as
 
    min_λ≥ 0{λδ + 𝔼_N ∼ P_0 [max_N'1_N'(𝒰) ≥ 1 - λc(N, N')  ] }.

Conditioning directly on the first inequality, we can rewrite this as
 
    min_λ≥ 0{λδ + 𝔼_N ∼ P_0 [1- λmin_N'(𝒰) ≥ 1c(N, N')  ]^+ }.

 Here we also set c((s,u), (v,t))=|u-v|1_s=t+∞1_s≠t. Moreover, we notice the equivalent representation between X^(n) and (A^(n), V_n(x)), which means we can write N(dx) = ∑_n1_X^(n)(dx)=∑_n1_(A^(n), V_n(x))(dt, dv). Thus, if N(𝒰)≥ 1, it means ∃ n s.t. V_n(x)-A_n≥0, we can simply set N'(·)=N(·) to obtain the minimum of min_N'(𝒰) ≥ 1 c(N, N'). On the other hand, if N(𝒰)=0, it means ∀ n, V_n(x)-A_n<0. Then we denote δ=min_n(A^(n)- V_n(x)) and n^*∈min_n(A^(n)- V_n(x)) and set N'(dt,dv)=1_(A^(n^*), V_n^*(x)+δ)(dt,dv)+∑_n≠n^*1_(A^(n), V_n(x))(dt,dv), which guarantees N'(𝒰)≥ 1, and the minimum of min_N'(𝒰) ≥ 1 c(N, N') is exactly δ.  To conclude, we have the following equivalence holding:
 
    min_N'(𝒰) ≥ 1 c(N, N') = min_n≥ 1 [ A^(n) - V^(n)_x]^+.

Substituting, we obtain the desired result of (<ref>):
 
    min_λ≥ 0λδ + 𝔼_N∼ P_0 [1 - λmin_n≥ 1( A^(n)- V_x^(n))^+  ]^+






  §.§.§ Minimizer of CDF Proof


 
 Following where we ended in the proof of Theorem <ref>, 
Denote f(A,V_x):=min_n≥ 1( A^(n)- V_x^(n))^+. By first order condition of the Equation (<ref>), we have:
 
    δ=𝔼_N∼ P_01(f(A,V_x)≤1/λ^*)f(A,V_x).

 Substituting it to objective, we have:
 
    min_λ≥ 0λδ + 𝔼_N∼ P_0 [1 - λmin_n≥ 1( A^(n)- V_x^(n))^+  ]^+
       =P_0(f(A,V_x)≤1/λ^*).

 Thus, we have:
 
    min_P:W_c(P,P_0)≤δP(X_1 ≤ x_1^-1, …, X_d ≤ x_d^-1)
       =1-P_0(f(A,V_x)≤1/λ^*)
       =P_0(f(A,V_x)>1/λ^*)
       =P_0(N(𝒞_1/λ^*)=0)
       =exp(-𝔼_P_0N(𝒞_1/λ^*)  )

 



 §.§ Proof of Rare Region Probability

 

 
     We write the primal problem as 
     
    sup_D(P, P_0) ≤δ P(N(A) ≥ 1)

     and writing it in the dual form we get
     
    inf_λ≥ 0{λδ + 𝔼_P_0 [sup_Z^(n)( sup_n=1^∞1_Z^(n)∈ A - λ∑_n=1^∞ Z^(n) - X^(n) )  ]}.

     Replacing the inner maximization, we can write
     
    sup_Z^(n)( sup_n1_Z^(n)∈ A - λ∑_n=1^∞ Z^(n) - X^(n) ) = ( 1- λinf_ninf_z ∈ Az - X^(n))^+,

     which is due to:
     
    sup_Z^(n)( sup_n1_Z^(n)∈ A - λ∑_n=1^∞ Z^(n) - X^(n) ) = ( 1- λinf_∃ n, Z^(n)∈ A∑_n=1^∞Z^(n) - X^(n))^+.

     And solving inf_∃ n, Z^(n)∈ A∑_n=1^∞Z^(n) - X^(n), we have it equals inf_ninf_z ∈ Az - X^(n) as the region only requires one sample z^(n)∈ A and the others can be equal to corresponding X^(n).
 

 

 §.§ Proof of Number of Rare Region Occurrences


    We begin with the dual definition
    
    inf_λ≥ 0{λδ + 𝔼_N ∼ P_0 [sup_ Z^(1)⋯ Z^(N) ( ∑_n=1^∞1_Z^(n)∈ A  - λ∑_n=1^∞ Z^(n) - X^(n))  ]}.

    Rewriting the part within the sup, we obtain the desired result, since
    
    sup_Z^(1)⋯ Z^(N) ( ∑_n=1^∞1_Z^(n)∈ A  - λ∑_n=1^∞ Z^(n) - X^(n)) = ∑_n=1^∞( 1- λinf_z ∈ Az - X^(n))^ +.

    



 §.§ CVaR proof

 

 

At first, we consider the finite point case, where N(·)=∑_n=1^N δ_X^(n)(·). The CVaR problem is given as
 
    min_z max_D(P, P_0) ≤δ𝔼_P (  ( ∫⋁_i=1^d x_i N(dt, dx) - z )^+ /1- α+ z )_primal problem.

 Taking the primal problem component, we can write the dual problem as:
      
    min_λ≥ 0{λδ + 𝔼_P_0 [ max_Z^(n)( ∑_n=1^∞ Z^(n)_∞ - z )^+ - λ∑_n=1^∞ Z^(n) - X^(n)_∞ ] }
       = min_λ≥ 1{λδ + 𝔼_P_0 ( ∑_n=1^N  X^(n)_∞ -z)^+  }   (Lemma <ref>)
       = δ + 𝔼_P_0 ( ∑_n=1^N  X^(n)_∞ -z)^+ .

     Adding the minimization over z, we obtain
     
    min_z [ δ + 𝔼_P_0 ( ∑_n=1^N  X^(n)_∞ -z)^+/1-α + z   ] .

Taking the minimum, we get

    δ/1-α + 𝔼_P_0 ( ∑_n=1^N  X^(n)_∞|∑_n=1^N X^(n)_∞≥ q_α )

where P_0 ( ∑_n=1^N  X^(n)_∞≤ q_α) = α. Finally, as q_1<∞, the summation converges a.s., we can safely let N→∞ and obtain the final result.
 

    


    
        For x∈ℝ^d_+ and λ≥0, we have:
        
    max_y_1⋯ y_N∈ℝ^d_+(∑_n=1^N y_n_∞-z)^+-λ∑_n=1^Ny_n-x_n_∞=
                ∞,    when λ∈[0,1),
    (∑_n=1^N x_n_∞-z)^+,   when λ∈[1,∞).

    
    
        For λ∈[0,1), we claim the objective value would diverge to infinity. In fact, we let y_n=x_n for n≥2. The objective becomes:
        
    max_y_1∈ℝ^d_+(y_1_∞+∑_n=2^Nx_n_∞-z)^+-λy_1_∞.

        Then, we can pick y_1 such that y_1_∞≥ z-∑_n=2^Nx_n_∞ and letting it goes infinity, we find the objective will also go infinity.

        For λ∈[1,∞), we have:
        
    (∑_n=1^N y_n_∞-z)^+-λ∑_n=1^Ny_n-x_n_∞   ≤(∑_n=1^N y_n-x_n_∞+x_n_∞-z)^+-λ∑_n=1^Ny_n-x_n_∞
       =(1-λ)∑_n=1^Ny_n-x_n_∞+(∑_n=1^N x_n_∞-z)^+.

        Thus, maximizing over y, we have the upper bound for the objective is (∑_n=1^N x_n_∞-z)^+. And this value is attained when y_n=x_n for n=1,⋯,N.
    
 

 



§ DATA DESCRIPTION


Here we provide more details on the data distributions used.
The symmetric logistic (SL) and asymmetric logistic (ASL) distributions are commonly used parametric models that have the property of max-stability.
We describe some of their properties and how they are applied in the mixture model.
We use the sampling algorithms described in <cit.> to generate all the datasets.

As mentioned, for computing the cumulative distribution, the central object of interest is the spectral measure. 
In each of these datasets, the spectral measure is given by a mixture of two spectral measures.
Our baseline model P_0 is fit to the mixture without consideration that the model contains two components and results in model misspecification.



 §.§ Symmetric Logistic Mixture

For these experiments, we consider a mixture model with two components: one that is almost completely dependent and another that is almost completely independent. 
The component with dependence is sampled with lower probability than the one with independence. 
Therefore, a naive estimator would largely concentrate on the component with independence than the one with dependence. 
The dependence function for this distribution is given by

    A_SL(w) =  (∑_i=1^d w_i^1/α )^α,   α∈ (0, 1].

Notice that the influence α is equal for all components of the vector.
We specifically sample the distribution with large dependence (α=0.1) with probability 0.1 whereas we sample the distribution with small dependence (α = 0.9) with probability 0.9.



 §.§ Asymmetric Logistic Mixture

While the SL distribution is symmetric and all components have the same level of dependence, the ASL can impose specific dependencies between any subset of variables.
Denoting 𝒫_d as the power set of {1, …, d }, the dependence function for this family of distributions is given by

    A_ASL(w) = ∑_b ∈𝒫_d (∑_i ∈ b ( λ_i, b w_i)^1/α_b )^α_b,    w ∈Δ_d-1, α∈ (0, 1), λ_b ∈Δ_|b|-1.


We then consider a mixture of two different components, one again with a heavier tail and another with a lighter tail.
This is given by setting λ to be the same random point on the simplex for both mixture components but then considering α = 0.1 and α=0.9 for the components with high dependence and low dependence, respectively. 



 §.§ Financial Data and Visualization


Below we've provided a visualization of the maximum daily returns taken over weekly blocks, and maximum daily returns taken over annual blocks. The returns are provided for the eleven industries, where the values are averages taken across selected companies uniquely belonging to each industry. To compute the average of each industry, we first selected the 93 members of the S&P 500 that have been listed on the index since January 1983, in order to provide forty years of financial data. Each of these company's daily returns were averaged across their assigned industry to create the average daily returns for each industry over forty years.





§ ALGORITHM DETAILS

Here we provide additional details that were left out of the main manuscript. 
Specifically, we go over the Maximize and Minimize steps in greater detail.
The basic idea is we have an iterative training algorithm similar to the min-max problems in generative adversarial networks.
λ is initialized according to 1/δ + ε for some small ε. 
P is initialized according to the same parameters as P_0.
For each of the K iterations, we

Maximization step:
We fix λ to be the value from the last iteration.

    
  * Sample N points from the base distribution X^(n)∼ P_0,    n=1,…,N to obtain the X points from the base distribution.
    
  * Sample N points from the current estimate of the adversarial distribution P to get X̃^(n)∼ P,    n = 1 … N.
    
  * Compute ℒ for each X^(n), X̃^(n) which results in a N × N matrix of values, i.e. 
    
    𝐋 :=
        [ ℒ_λ(X̃^(1), X^(1))                  ⋯ ℒ_λ(X̃^(1), X^(n));                  ⋮                  ⋱                  ⋮; ℒ_λ(X̃^(n), X^(1))                  ⋯ ℒ_λ(X̃^(n), X^(n));                    ]

    
  * Take the max over the X̃^(n) dimension, i.e. max_i𝐋_i, ·
    
  * Average over the remaining Y^(n) dimension, i.e. R = 1/N∑_j=1^N max_i=1^N 𝐋_i, j.
    
  * Compute the gradient of R with respect to the parameters of P, update the parameters of P via gradient descent.

Adding different constraints to the problem results in different forms of the Ỹ, which we describe in the next section.


Minimization step:
Now we fix the parameters of P and only minimize over λ.

    
  * Compute R̂ = λδ + 1/N∑_j=1^N max_i=1^N 𝐋.
    
  * Compute gradient of R̂ with respect to λ, update according to gradient descent.

We next describe a few algorithmic considerations of the maximization step.



 §.§ Unconstrained Case

When sampling from an MEV, one samples according to A^(n), Y^(n),    n=1,…, N, N ≫ 0.
Then samples are generated according to max_n=1^N Y^(n)/A^(n) (e.g. see <cit.> for additional considerations).
In the unconstrained problem, we let the X^(i) in (<ref>) be the MEV, i.e. X^(i) = max_n=1^N Y^(i,n)/A^(i,n). 

Then, P does not use any of the factorization of the Y^(n)/A^(n) and is free to be whatever distribution maximizes the quantity.
The problem is unconstrained because it does not consider any of the constraints given by EVT, specifically that the factorization in terms of the radial A and spectral component Y must hold.


 §.§ Constrained Case

On the other hand, we consider the explicit factorization in the constrained case.
Here we share the A^(n) across both distributions of P, P_0. 
Then, X remains as described in the unconstrained case but X̃ is given by X̃^(i) = max_n=1^N Ỹ^(i,n)/A^(i,n) where the Ỹ^(i,n) are sampled from the estimated P distribution.
Note that the A^(n) are the same for the calculation of X and X̃.
The constraints then inform the adversarial distribution that it must be max-stable and that the radial component is consistent between both.



 §.§ Computational Details and Code Reference


The code base for the DRO MEV model, written in python, is available at the following github link:  <https://github.com/patrick-kuiper/mev_dro>. Below, in Table <ref>, we provide the run time for each of the synthetic data experiments: Unconstrained, Constrained Extreme Value Distributed (EVD), and Constrained Extreme Value Distributed: with Unit Margin (EVD-SM). We chose to highlight the synthetic data sets as they represent the most challenging experiments computationally. Each of these experiments were run individually on NVIDIA GeForce RTX 3090 GPUs with CUDA Version 12.2. Each experiment was run for 2001 epochs.  






