Paper Title: Distributed Adaptive Time-Varying Optimization with Global Asymptotic Convergence.


 
          
















Distributed Adaptive Time-Varying Optimization with Global Asymptotic Convergence
    Liangze Jiang, Zheng-Guang Wu, and Lei Wang 

The authors are with College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China (e-mail: zetojiang; nashwzhg; lei.wangzju @zju.edu.cn). (Corresponding author: Zheng-Guang Wu.)
    August 5, 2024
==============================================================================================================================================================================================================================================================



§ ABSTRACT 
In this note, we study distributed time-varying optimization for a multi-agent system. We first focus on a class of time-varying quadratic cost functions, and develop a new distributed algorithm that integrates an average estimator and an adaptive optimizer, with both bridged by a Dead Zone Algorithm. Based on a composite Lyapunov function and finite escape-time analysis, we prove the closed-loop global asymptotic convergence to the optimal solution under mild assumptions. Particularly, the introduction of the estimator relaxes the requirement for the Hessians of cost functions, and the integrated design eliminates the waiting time required in the relevant literature for estimating global parameter during algorithm implementation. 

We then extend this result to a more general class of time-varying cost functions. Two numerical examples verify the proposed designs.






Distributed optimization, quadratic
optimization, time-varying cost functions, adaptive gain.





§ INTRODUCTION

Designing and reconstructing the dynamic behavior of groups through optimization and control techniques propels the advancement of distributed optimization theory <cit.>, especially in physical applications such as sensor networks <cit.>, robot networks <cit.> and power systems <cit.>. Recently, there is a trend towards modeling and solving large-scale optimization problems in dynamic environments (e.g., see <cit.> and the references
therein). To cope with constantly changing environmental information, the modeling of cost functions in dynamic environments is typically based on real-time information acquisition by agents (see Fig. <ref>)<cit.>. 

However, the dynamic nature of the environment may disrupt this process, which in turn leads to the `fragility' of cost functions. For instance, in multi-camera localization of moving objects, cameras with fixed positions fail to capture images of mobile objects outside their field of view<cit.>. In multi-robot target tracking, obstacles or limited perceptual distance prevent some robots from obtaining target information<cit.>. When information acquisition is hindered due to environmental changes, weak or missing local cost functions cannot guarantee their uniformly strong convexity about decision variables (e.g., see Example 2 in Section <ref>). Moreover, a realistic topic in autonomous optimization is reducing the dependence of algorithms on global parameter of the cost functions and the communication graph<cit.>. These observations prompt us to investigate distributed time-varying optimization.




Intuitively, during the search for the time-varying optimal solution, gradient information provides only error feedback relative to the current optimal solution. By introducing the time derivative of the gradient as compensation<cit.>, the modified gradient descent method can solve the time-varying optimization problem exactly[The considered time-varying optimization setting is different from the online optimization<cit.> where the structural information on the cost functions is unavailable and non-zero tracking errors are pursued. In contrast, in our problem setting, each agent can construct a complete local cost function by adequately acquiring information about the environment.]. This basic idea is further expanded to distributed versions to tackle large-scale time-varying optimization problems in <cit.>, some of which consider equality <cit.> and inequality <cit.> constraints. In <cit.>, a robust control method is introduced to solve distributed time-varying quadratic optimization (DTQO) with coupled cost functions. However, as mentioned earlier, the cost functions established in dynamic environments may exhibit time-varying and non-strongly convex properties. In such cases, the existing algorithms cannot be directly applicable, as they require the Hessians of all local cost functions to be positive definite<cit.>, identical<cit.>, diagonal <cit.> and time-invariant<cit.>. Although one design in <cit.> considers nonpositive Hessians of the local cost functions, its assumption of constant upper bounds on certain complex functions limits the types of cost functions. Moreover, its convergence result is semi-global since the initial states of agents are needed for the design. Additionally, the waiting time in <cit.> implies a two-stage algorithm implementation, requiring knowledge of the initial states of agents, the communication graph, or other prior global parameters of the cost functions to determine the size of it before running the algorithm.
 
 
Motivated by this, we start with a basic DTQO and develop a distributed algorithm that seamlessly integrates an average estimator with an adaptive optimizer, based on the nonsingular matrices output by the Dead Zone Algorithm (DZA). We then extend this result to address the distributed optimization problem of a general class of cost functions, not limited to quadratic cost functions<cit.>. The contributions lie in: (1) We propose distributed optimization algorithms without relying on positive definite, identical, diagonal or time-invariant Hessians of all local cost functions. To the best of our knowledge, these are the first result of distributed optimization for the aforementioned types of cost functions. (2) By constructing a composite Lyapunov function and introducing the concept of the finite escape time, we establish the asymptotic convergence of closed-loop outputs to the optimal solution in the global sense, rather than in the semi-global sense <cit.>. (3) Different from <cit.>, the proposed design does not require the waiting time during algorithm implementation, thereby removing the requirement for prior parameters used to determine its size. Such design also relaxes the assumption that the upper bounds of certain time-varying parameters or their high-order time derivatives are known in advance <cit.>. 

 
 



Notations. The set of real numbers is represented by ℝ. Let 1_N denote the vector (1,⋯,1)^⊤∈ℝ^N and I_m denote the m-dimensional identity matrix. For a matrix A∈ℝ^m× m, we use λ_p(A), p∈{1,⋯,m}, to represent the p-th smallest eigenvalue of A. For a vector or a matrix X, we use |X|_p to represent the l_p-norm of X. |X| is the abbreviation of |X|_2. The signum function is denoted by (y), with y∈ℝ. For a vector x=(x_1,⋯,x_n)^⊤, define (x)=((x_1),⋯,(x_n))^⊤ and S(x)=(S(x_1),⋯,S(x_n))^⊤, where S(x_i)=x_i/(|x_i|+ϵ_1η_t) with ϵ_1>0 and η_t∈ℝ. For a matrix A=[a_ij]∈ℝ^m× n, define (A)=[(a_ij)]∈ℝ^m× n and use A_⊗ to denote A⊗ I_m. Denote {A_i}, i=1,⋯,N as a diagonal matrix with diagonal entries composed of A_1,⋯,A_N, where A_i∈ℝ^m_i× m_i. The gradient of a function f(x,t):ℝ^m×ℝ_+→ℝ with respect to x is denoted by ∇ f(x,t). Let ∇_t f(x,t) denote the partial derivative of ∇ f(x,t) with respect to time t.






§ PRELIMINARIES AND PROBLEM STATEMENT




 §.§ Preliminaries

A differentiable function f(x,t):ℝ^m×ℝ_+→ℝ is uniformly ζ-strongly convex in x if there exists ζ>0 such that (a-b)^⊤(∇ f(a,t)-∇ f(b,t))≥ζ |a-b|^2, ∀ a,b∈ℝ^m, ∀ t≥0. ∇ f(x,t) is uniformly θ-Lipschitz in x if there exists θ>0 such that |∇ f(a,t)-∇ f(b,t)|≤θ|a-b|, ∀ a,b∈ℝ^m, ∀ t≥0. 

The communication network is described by an undirected graph 𝒢=(𝒩,ℰ), where 𝒩={1,…,N} is the node set and ℰ⊆𝒩×𝒩 is the edge set. Denote the weighted adjacency matrix of 𝒢 as 𝒜=[a_ij]_N× N. An edge (i,j)∈ℰ means that nodes i,j can receive information from each other. Define a_ij=1 if (j,i)∈ℰ, and a_ij=0 otherwise. Let 𝒩_i={j∈𝒩:(j,i)∈ℰ} denote the set of node i's neighbors. The Laplacian of a graph 𝒢 is defined as L=[l_ij]_N× N with l_ii=∑_j≠ ia_ij and l_ij=-a_ij for j≠ i. If there is a path from any node to any other node in graph 𝒢, it is called connected. Denote the incidence matrix associated with 𝒢 as D=[d_ik]_N× |ℰ|, where d_ik=1 if the edge e_k enters the node i, d_ik=-1 if the edge e_k leaves the node i, and d_ik=0 otherwise.


(Graph connectivity) The graph 𝒢 is undirected and connected.


By Assumption <ref>, it is known that the Laplacian matrix L of the graph possesses a zero eigenvalue, corresponding to the eigenvector 1_N, while the remaining eigenvalues are positive <cit.>, which satisfy λ_1(L) = 0 < λ_2(L) ≤⋯≤λ_N(L).



 §.§ Problem Definition

Consider a group of N agents interacting over a communication network 𝒢=(𝒩,ℰ). Each agent i∈𝒩 is assigned a local cost function defined by 

    f_i(x_i,t)=1/2x_i^⊤H_i(t)x_i+R_i^⊤(t)x_i+d_i(t),

where x_i∈ℝ^m is the decision variable, and H_i(t)∈ℝ^m× m, R_i(t)∈ℝ^m and d_i(t)∈ℝ are time-varying parameters. We aim to design a distributed algorithm such that each agent solves the optimization problem

    min_r∈ℝ^mf(r,t)=∑_ i∈𝒩f_i(r,t).

The optimal solution of the above optimization problem is defined as r^*(t)=min_r∈ℝ^mf(r,t). To guarantee the first-order optimality condition ∇f(r^*(t),t)=0 holds, ∀ t≥ 0, there is (d/dt)∇ f(r^*(t),t)=H(t)ṙ^*(t)+∇_t f(r^*(t),t)≡0, where H(t)=∑_i∈𝒩H_i(t). Then, if H(t) is invertible, it follows that ṙ^̇*̇(t)≡-H^-1(t)∇_t f(r^*(t),t). The following assumption guarantees the uniqueness of r^*(t) and the positive definiteness of the Hessian H(t), ∀ t≥0.


(Uniform strong convexity) The global cost function f(r,t) is uniformly h_1-strongly convex with respect to r. That is, there exists a known h_1>0 such that λ_1(H(t))≥ h_1, ∀ t≥0.


By this assumption, each f_i(x_i,t) here is not required to be uniformly strongly convex<cit.>. It can even be non-convex. Moreover, the Hessians of all f_i(x_i,t) do not have to be diagonal <cit.> or time-invariant <cit.>, nor do they have to be completely identical as required in <cit.> and <cit.>.


If the local cost functions are all convex, and there exists a strongly convex one with a known strongly convex coefficient (or a more conservative lower bound) h, then, h_1 can directly take h (see <cit.> for details).




(Boundedness) For any i∈𝒩, |H_i(t)|, |R_i(t)| and |Ṙ_i(t)| are bounded, ∀ t≥0, but their upper bounds are unknown. Moreover, there exists a known h_2>0 such that |Ḣ_i(t)|_∞≤h_2, ∀ t≥0.

We stress that Assumption <ref> guarantees the linear boundedness of ∇_t f_i(x_i,t) with respect to x_i, which is more relaxed than the assumptions that directly impose known and constant bounds on ∇_t f_i(x_i,t) in <cit.> and (d/dt)∇_t f_i(x_i,t) in <cit.>. By the derivations below (<ref>), Assumptions <ref> and <ref> guarantee the boundedness of r^*(t) and thus ṙ^*(t). Moreover, we only require knowledge of the lower bound of the Hessian's eigenvalue and the upper bound of Hessian's changing rate. Additional prior knowledge of cost functions, such as the upper bounds of (∂/∂ t)∇_t f_i(x_i,t) in <cit.>, ⃛r^*(t) in <cit.>, R̈_i(t) in <cit.> and ∇ f_i(x_i,t) in <cit.>, is not necessary here.


In practice, the parameters of cost functions are typically constrained by the properties of physical objects. For example, consider the problem of multi-UAVs tracking of a target<cit.> with the cost function f_i=|x_i-r(t)|^2, where r(t) is the trajectory of the target vehicle. The limited energy and engine power of the vehicle, in fact, must constrain |r(t)| and |ṙ(t)| within a certain range, respectively, but the specific upper bounds of |r(t)| and |ṙ(t)| may be unknown in advance. Assumption <ref> aligns with these observations. 





§ DESIGN BASED ON ESTIMATOR AND OPTIMIZER


Revisiting the centralized optimization algorithm in <cit.>, we can design ṙ=-∇ f(r,t)-H^-1(t)∇_t f(r,t) to guide r to asymptotically converge to the optimal r^*(t). Inspired by this, the following design adopts distributed `∇ f_i' feedback and `NH^-1(t)∇_t f_i' feedforward[The scaling factor N is instrumental for the subsequent derivations.]. In view of the heterogeneity of H_i(t), i∈𝒩, the technique of dynamic average consensus is employed to estimate the average of all Hessians H(t)/N.



 §.§ Average Estimator and Output Reconstruction


For agent i∈𝒩, we first consider the average estimator

    ξ̇_i=   -ω∑_j∈𝒩_i(z_i-z_j),    z_i=ξ_i+H_i(t),

where ω>h_2, ξ_i∈ℝ^m× m, and z_i∈ℝ^m× m is the output of estimator. The initial conditions satisfy ∑_j∈𝒩ξ_i(0)=0 and ξ_i(0) is symmetric. The convergence of z_i to H(t)/N in finite time can be guaranteed by <cit.>. However, since the H_i(t) may not be positive definite, z_i(t) inevitably passes through singular points during its evolution. Considering z^-1_i(t) as a part of feedforward term, we propose Algorithm <ref> to avoid the impact of singular z_i(t), where h_0>0 is to be determined and ℳ={1,⋯,m}. In Algorithm <ref>, we use min_p∈ℳ|λ_p(z_i(t))| to quantify how close z_i(t) approaches singular points, and constrain ẑ_i(t) to be within a `safe range' away from singular points. Consequently, min_p∈ℳ|λ_p(ẑ_i(t))|≥ h_0 holds for all t≥0, regardless of whether the input matrix z_i(t) is singular. We thus refer to Algorithm <ref> as the Dead Zone Algorithm, and the inverse of the output of DZA, ẑ_i^-1(t), remains well-defined during the evolution of z_i(t). 



From the perspective of ensuring nonsingularity, the DZA plays the same role as the projection method in <cit.>. However, since the integrated design shown in Fig. <ref> is adopted in our work, we also need to provide a sufficient condition for parameter selection in the DZA to ensure closed-loop convergence, rather than just ensuring nonsingularity. The following lemma illustrates an important relation between z_i(t) and the `shaped' ẑ_i(t), if we select a suitable parameter h_0. This relation is instrumental for the analysis in Section <ref>.

    Suppose Assumptions <ref>-<ref> hold. For system (<ref>) and Algorithm <ref> with h_0=γ h_1/N, the boundary condition |ẑ_i(t)-H(t)/N|≤k_1|z_i(t)-H(t)/N| holds ∀ t≥ 0, with k_1>0.


See Appendix <ref>.



Different from <cit.>, although the finite-time average estimator (<ref>) is adopted here, the size of convergence time (i.e., the waiting time in <cit.>) is unnecessary for agents to obtain, because we do not employ the execution logic of a two-stage algorithm implementation.










 §.§ Adaptive Optimizer

For agent i∈𝒩, consider the following adaptive optimizer

    ẋ_i=   2.13ex-∑_j∈𝒩_i(α_ij(x_i-x_j)+β_ijS(x_i-x_j))_ Adaptive consensus coordinator+ϕ_i,
    ϕ_i=   -k∇ f_i(x_i,t)-ẑ_i^-1(t)∇_t f_i(x_i,t),
    α̇_ij=   |x_i-x_j|^2,   β̇_ij=|x_i-x_j|_1-mϵ_1η_t,  j∈𝒩_i,

where the function S(·) is defined in Notations, η_t=exp(-ϵ_2t) with ϵ_2>0, k>0 is to be determined, α_ij(0)=α_ji(0)≥0 and β_ij(0)=β_ji(0)≥ mϵ_1/ϵ_2 with ϵ_1>0. As a continuous approximation of the signum function, the function S(·) reduces the chattering of x_i in application.

In (<ref>), ϕ_i incorporates distributed `∇ f_i' feedback and `ẑ_i^-1(t)∇_t f_i' feedforward. The consensus coordinator utilizes adaptive control gains α_ij and β_ij to handle unknown global parameter (see details in Appendix <ref>). Fig. <ref> illustrates the structure of the proposed design for agent i, where dashed arrows represent information received from neighboring agents. Note that if x_i denotes the position of agent i, and x_i-x_j can be obtained through relative measurements (e.g. the acoustic and vision sensors), only the variable z_j is transmitted through the communication channel. Therefore, this design reduces the communication resources usage compared to <cit.>.


The DZA bridges the average estimator and adaptive optimizer, enabling the implementation without relying on the waiting time (required in <cit.>). With this cascaded structure, the co-design of these three modules is needed to ensure the output x_i converges to the time-varying but unknown r^*(t), which is a major challenge of this note. Additionally, the dead-zone nonlinearity introduced by DZA and the inverse operation of its output, (·)^-1, also cause difficulties in the subsequent convergence analysis.


The trajectory of ẑ_i(t) generated by Algorithm <ref> is discontinuous but measurable and locally essentially bounded. Thus, the Filippov solutions of the proposed design always exist. Since the Lyapunov functions used in convergence analysis are continuously differentiable, the proof still holds without using the nonsmooth analysis<cit.>.

 


§ CONVERGENCE ANALYSIS


Define x=(x_1,⋯,x_N)^⊤, ϕ=(ϕ_1^⊤,⋯,ϕ_N^⊤)^⊤ and a diagonal matrix B, whose order is equal to the number of edges in the graph 𝒢. The nodes corresponding to non-zero elements in column i of matrix D are denoted as i_1 and i_2, respectively. The diagonal element B_ii of B is defined as β_i_1i_2. Then, we rewrite (<ref>) as

    ẋ   =-L̅_⊗x-(DB)_⊗S(D^⊤_⊗x)+ϕ,

where L̅=[l^α_ij(t)]∈ℝ^N× N with l^α_ii(t)=∑_j≠ iα_ij(t) and l^α_ij(t)=-α_ij(t) for j≠ i. Define the consensus error e=M_⊗ x with M=I_N-1_N1_N^⊤/N, and the error between average state and global optimal solution x̃=1^⊤_Nx/N-r^* with 1^⊤_N=1_N^⊤⊗ I_m. This implies the relation between x, e and x̃, i.e.,

    x=e+1_N⊗x̃+1_N⊗ r^*.
  
With ∇ F(x,t)=(∇ f_1(x_1,t),⋯,∇ f_N(x_N,t))^⊤, we obtain 

    ė=   -L̅_⊗e-(DB)_⊗S(D^⊤_⊗e)+M_⊗ϕ,
    ẋ̃̇=   -k/N1^⊤_N∇ F(x,t)-1/N∑_i∈𝒩ẑ^-1_i∇_t f_i(x_i,t)-ṙ^*,

where ṙ^*=-H^-1(t)∑_i∈𝒩∇_t f_i(r^*,t).

The following theorem gives the main result of this note.

Suppose Assumptions <ref>-<ref> hold. For system (<ref>)-(<ref>) with Algorithm <ref>, let h_0=γ h_1/N and k>h_2N√(m)/h_1^2+ϵ_3 with 0<γ<1 and ϵ_3>0. Then, the optimization problem (<ref>) is solved, i.e., lim_t→∞x_i(t)-r^*(t)=0, i∈𝒩.

 
See Appendix <ref>.


From a system theoretic perspective, the analysis of all subsystems is naturally unified within the common time domain, which differs from that of the waiting time-based design<cit.>. Specifically, the Lyapunov analysis for e and x̃ subsystems is established based on (<ref>)-(<ref>) and Lemma <ref>. Therefore, the entire system is considered as an interconnected system, as shown in Fig. <ref>.









§ EXTENSIONS: NON-QUADRATIC COST FUNCTION

In this section, we consider local cost functions f_i(x_i,t) of a general form as

    f_i(x_i,t)=1/2x_i^⊤H_i(t)x_i+R_i^⊤(t)x_i+f̂_i(x_i,t), i∈𝒩,
 
where f̂_i(x_i,t) is not limited to the quadratic form. Let ρ_i(x_i,t) denote the Hessian of f_i(x_i,t). We then make the following assumptions on f_i(x_i,t).

For ∀ t≥0 and ∀ i∈𝒩, there exist μ_j>0, j=1,⋯,5 such that 

    
  *  The global cost function f(r,t) is uniformly μ_1-strongly convex and ∇f̂_i(x_i,t) is uniformly globally Lipschitz, 
    
  *  |∇f̂_i(x_i,t)|≤μ_2|x_i|+μ_3,
    
  * ∇_tf̂_i(x_i,t)=g_i(t)x_i+s_i(x_i,t) where |g_i(t)|≤μ_4, |s_i(x_i,t)|≤μ_5s̅(t)<∞ and ∫_0^∞s̅^2(t)  dt<∞.





Although Assumption <ref> requires ∇f̂_i(x_i,t) and ∇_tf̂_i(x_i,t) to satisfy certain conditions, it holds for many cost functions, as shown in Example 1 of Section <ref>. Similar assumptions are also adopted in <cit.>.


With each f_i(x_i,t) defined by (<ref>), the algorithm for solving the optimization problem (<ref>) follows the design process in Section <ref>. However, the estimator (<ref>) cannot be applied here to estimate ∑_i∈𝒩ρ_i(x_i,t)/N with ρ_i(x_i,t) replacing H_i(t). The reason is that ρ̇_i(x_i,t) may contain ẋ_i, whose upper bound is unknown before implementing algorithm. Consequently, the fixed gain ω cannot be determined in advance due to the unknown upper bound of |ρ̇_i(x_i,t)|_∞. To address this issue, we introduce a new average estimator adapted from <cit.>: 

    ξ̇_i=   -∑_j∈𝒩_iω_ij(t)(z_i-z_j),  z_i=ξ_i+ρ_i,  i∈𝒩,

where ω_ij>((N-1)/2)(|ρ̇_i|_∞+|ρ̇_j|_∞) is not a fixed but a state-based gain. The initial conditions satisfy ∑_j∈𝒩ξ_i(0)=0 and ξ_i(0) is symmetric. Similar to the proof in <cit.>, z_i can be shown to converge to ∑_i∈𝒩ρ_i(x_i,t)/N in a finite time. The next corollary extends Theorem <ref> to address the distributed optimization problem with  more general cost functions (<ref>).


Suppose Assumptions <ref>, <ref>, <ref> hold. For system (<ref>) and (<ref>)-(<ref>) with Algorithm <ref>, let h_0=γμ_1/N and k>(h_2+μ_4)N√(m)/μ_1^2+ϵ_4 with 0<γ<1 and ϵ_4>0. Then, the optimization problem (<ref>) is solved, i.e., lim_t→∞x_i(t)-r^*(t)=0, i∈𝒩.

 
See Appendix <ref>. 


    
    Different from the estimator (<ref>), the implementation of estimator (<ref>) does not require a known upper bound on |ρ̇_i|_∞, but at the expense of requiring additional exchange of information |ρ̇_i|_∞ with neighboring agents. 















§ SIMULATIONS



Example 1. Consider 20 agents interacting over a circular communication network. With the decision variable y=(y_1,y_2,y_3)∈ℝ^3, each agent is assigned with a local cost function. For agent i=1,⋯,6, f_i=∑_j=1^32y_j^2 +0.1isin (t) y_j. For agent i=7,⋯,12, f_i=(y_1+cos(t)y_2+0.1iy_3)^2+cos(t)y_1+0.1iy_2+y_3. The agent i=13,⋯,20 is assigned with f_13=∑_j=1^3exp(-t)y_j^2, f_14=∑_j=1^3sin(t) y_j^2, f_15=∑_j=1^3-(y_j-2)^2, f_16=∑_j=1^3sin(y_j), f_17=∑_j=1^3(tanh(t)+1)ln(1+exp(y_j)), f_18=∑_j=1^3y_j/(y_j^2+1), f_19=∑_j=1^3-(1/(t+1))exp(-y_j^2) and f_20=∑_j=1^3y_j^2+exp(-t)ln(1+y_j^2), respectively, where f_19 is inspired by the density function of normal distribution, and f_17 and f_20 are adapted from <cit.>. 

In fact, except for f_15 and f_17, all local cost functions have their own global minimum. However, only f_1-f_6 and f_20 are uniformly strongly convex. f_13 and f_17 are uniformly strictly but not strongly convex, f_7-f_12 are just uniformly convex, while f_14, f_15, f_16, f_18 and f_19 are even non-convex. 






It can be checked that Assumptions <ref>, <ref>, <ref> are satisfied. By applying the distributed algorithm in Corollary <ref> with h_0=0.5 and k=35, the optimization problem (<ref>) is solved. With x_i corresponding to the decision variable y of agent i, Fig. <ref> shows the trajectory of |x_i-r^*(t)| that converges to zero asymptotically. 



Example 2. Consider a scenario in Fig. <ref>, where N=6 UAVs are used as relay nodes for a communication system supporting long distance robots collaboration. The signal power of robot i∈𝒩 is modeled by <cit.> as g_i(x_i,t)=p_i(t)/|x_i-d_i(t)|^2, x_i≠ d_i(t), in a 2D scenario, where p_i(t)>0 and d_i(t)∈ℝ^2 denotes the signal transmission power and the position of robot i, respectively. The communication among these robots is assisted by UAVs with relay capabilities<cit.>. Meanwhile, the UAVs are required to form a formation to expand communication coverage. 
To establish reliable communication links, all UAVs need to search for positions as close as possible to the corresponding robots. This leads to the optimization problem:

    min_x_1,⋯,x_N∑_ i∈𝒩f_i(x_i,t),    s.t.  x_i-x_j=τ_i-τ_j,

where f_i(x_i,t)=g_i^-1(x_i,t) and τ_i∈ℝ^2 denotes a desired formation. Each UAV i can obtain the real-time parameters p_i(t) and d_i(t) of robot i to construct f_i(x_i,t). The optimal solution of this problem, x^*_i=r^*+τ_i, i∈𝒩, can be regarded as the optimal relay position of UAV i, where r^* is the optimal solution of min∑_ i∈𝒩f_i(r+τ_i,t). Moreover, the following two practical cases are considered:

    
  * The signal transmission power of a robot falls below p_0,
    
  * The attacker's actions damage a robot at time instant t_0.

In both cases, the robots lose the ability to transmit their parameters p_i(t) and d_i(t) to UAVs, preventing the UAVs from constructing f_i(x_i,t). To simulate it, the corresponding local cost functions are set to zero[To ensure almost everywhere differentiability of parameters, a low-pass filter is used to attenuate the coefficient 1/p_i to 0, thereby attenuating the cost function to 0. Although there are non-differentiable points, the proof in this note still holds without using the nonsmooth analysis (Remark <ref>).], which is non-strongly convex.
 
 
As an example, suppose that the parameters of power model (SI units are used here) are given by p_1=p_2=10, p_3=10exp(-0.05t), p_4=8, p_5=10/(1+0.06t), p_6=6+sin(0.3t), d_1=(30,30)^⊤, d_2=(30+10cos(0.3t),0)^⊤, d_3=(-30,-30)^⊤, d_4=(-30,-30+10sin(0.3t))^⊤, d_5=(-30,0)^⊤ and d_6=(-30,30)^⊤. Let p_0=2 and t_0=67s. Following the above problem setting, p_3<p_0 if t≥ 32s. Assume that robot 5 is damaged by the attacker at t=t_0. The desired formation is given by a regular hexagon as τ_1=(5,10)^⊤, τ_2=(10,0)^⊤, τ_3=(5,-10)^⊤, τ_4=(-5,10)^⊤, τ_5=(-10,0)^⊤ and τ_6=(-5,10)^⊤. The initial states are given by x_1(0)=(-5,10)^⊤, x_2(0)=(-15,10)^⊤, x_3(0)=(-25,10)^⊤, x_4(0)=(-25,0)^⊤, x_5(0)=(-15,0)^⊤ and x_6(0)=(-5,0)^⊤. A circular communication network 𝒢 is also used for UAVs. One may check that Assumptions <ref>-<ref> are satisfied. Then, each UAV modeled as a single-integrator dynamics is driven by the proposed distributed algorithm in Theorem <ref> with h_0=0.1, ω=1 and k=20 to search the optimal relay position. Fig. <ref> shows the trajectories of x_i, i∈𝒩, r^*(t) and the snapshots of them at t=40s, 90s, 140s. Fig. <ref> shows the trajectory of |x_i-τ_i-r^*(t)|, i∈𝒩. It can be seen that even if f_3=0 when t≥ 32s and f_5=0 when t≥ 67s, the asymptotical convergence of x_i to τ_i+r^*(t) still be achieved.





§ CONCLUSION


This note proposes a systematical design incorporating average estimator, DZA and adaptive optimizer to solve DTQO. Its advantages include not relying on the strong convexity of local cost functions, identical or time-invariant Hessians, waiting time during algorithm implementation, or certain prior knowledge of the communication graph and global parameter information of cost functions, while guaranteeing closed-loop convergence in the global sense. The result is also extended to a class of non-quadratic cost functions. This advancement enables large-scale multi-robot systems to achieve autonomous optimization more flexibly in dynamic environments. 







 §.§  Two Auxiliary Lemmas


    <cit.>: Let A,B∈ℝ^m× m be Hermitian and let |·|_u be an unitarily invariant norm on ℝ^m× m. Then, |λ^↓(A)-λ^↓(B)|_u≤|A-B|_u, where λ^↓(A)(λ^↓(B)) is the diagonal matrix whose diagonal entries are the nonincreasingly ordered eigenvalues of A(B).



    Suppose Assumptions <ref>-<ref> hold. For system (<ref>)-(<ref>) with Algorithm <ref>, the state x_i(t) does not have a finite escape time, i.e., for any t̅∈[0,∞)  the state x_i(t) is bounded for all t∈[0,t̅]. 
    
        
    
    Denote x_ip∈ℝ the p-th row element of x_i, where i∈𝒩 and p∈ℳ. Similar subscripts also appear in certain matrices that follow. We take x_ip as an example to check its boundedness in a finite time. The dynamics of x_ip is given by
    
    ẋ_ip=   -kH_ipx_i-kR_ip-ẑ_ip^-1Ḣ_ix_i-ẑ_ip^-1Ṙ_i
       +c_1(t)∑_j∈𝒩_iS(x_ip-x_jp)+W_1+W_2,

    where 0≤ c_1(t)=∫_0^tmϵ_1 η_t dt=(mϵ_1/ϵ_2)(1-η_t)<∞, W_1=-∑_j∈𝒩_i(∫_0^t |x_i-x_j|^2  dt) (x_ip-x_jp), and W_2=-∑_j∈𝒩_i(∫_0^t|x_i-x_j|_1 dt)S(x_ip-x_jp). We first consider the first half of (<ref>), i.e.
    
    ẋ_ip=   -kH_ipx_i-kR_ip-ẑ_ip^-1Ḣ_ix_i-ẑ_ip^-1Ṙ_i
       +c_1(t)∑_j∈𝒩_iS(x_ip-x_jp)=:g_i(x_ip,t).

    For all t∈[0,t̅], we know g_i(x_ip,t) is piecewise continuous with respect to t. With Assumptions <ref>-<ref> satisfied, the boundedness of H_i, Ḣ_i, ẑ_i^-1 can be guaranteed, which implies that g_i(x_ip,t) is global Lipschitz with respect to x_ip. Then, by <cit.>, we know (<ref>) has a unique solution over [0,t̅]. In view of the fact that t̅ can be arbitrarily large, we make the following statement,
    
        
  S1. x_ip in system (<ref>) does not have a finite escape time, ∀ i∈𝒩 and ∀ p∈ℳ.
    
    We further consider W_1 and W_2 in system (<ref>). Assume that i^*,p^*∈max_i∈𝒩,p∈ℳ|x_ip|. Without loss of generality, let x_i^*p^*>0. If x_i^*=x_j, ∀ j∈𝒩_i^*, we have W_1=W_2=0. If there exists at least one j∈𝒩_i^* such that x_i^*≠ x_j, it follows that W_1,W_2≤ 0. Similarly, assume that x_i^*p^*<0. If x_i^*=x_j, ∀ j∈𝒩_i^*, we have W_1=W_2=0. If there exists at least one j∈𝒩_i^* such that x_i^*≠ x_j, then we have W_1,W_2≥0. Therefore, we make another statement,
    
        
  S2. W_1+W_2≤0 if x_i^*p^*>0 and W_1+W_2≥0 if x_i^*p^*<0.
       
    
By combining statements S1 and S2, we conclude from (<ref>) that the trajectory of x_i^*p^* does not escape to infinity at a finite time. With the fact that i^*,p^*∈max_i∈𝒩,p∈ℳ|x_ip|, thus, x_i(t) does not have a finite escape time, ∀ i∈𝒩.
 



 §.§ Proof of Lemma <ref>

Define the average of all Hessians, H̅(t)=H(t)/N. By the definition of δ_i below Lemma <ref>, we have δ_i=z_i(t)-H(t)/N=z_i(t)-H̅(t). Next, we will establish the relation between |ẑ_i-H̅| and |δ_i| in the following two cases:

(1) The non-singular case of matrix z_i(t). Recall Algorithm <ref>. For all t≥ 0, Once min_p∈ℳ|λ_p(z_i(t))|≥ h_0, we have z_i(t)=ẑ_i(t), implying |ẑ_i(t)-H̅|=|δ_i|.

(2) The (near-)singular case of matrix z_i(t).
For all t≥ 0, if min_p∈ℳ|λ_p(z_i(t))|<h_0, we have

    min_p∈ℳ|λ_p(z_i(t))|<h_0=γ h_1/N< λ_1(H̅(t)),

where the last inequality follows the fact that λ_1(H̅(t))≥ h_1/N given by Assumption <ref>. Before establish the relation between |ẑ_i-H̅| and |δ_i|, we first show the symmetry of z_i. Rewrite (<ref>) as ż_i=-ω∑_j∈𝒩_i (z_i-z_j)+Ḣ_i(t). Since H_i(t) is symmetric, Ḣ_i(t) must also be symmetric, ∀ t≥0. Symmetric z_i(0) indicates that z_i(0) is symmetric as well. It is clear that the function (·) does not change the symmetry of its matrix variable. Thus, z_i is always symmetric. By Lemma <ref> along with (<ref>), we conclude that |δ_i|≥ (1-γ)h_1/N>0. Consequently, we can always find a positive constant k^s_i such that |ẑ_i(t)-H̅|≤ k^s_i|δ_i|.

Now, based on the analysis of the above two case, we have
|ẑ_i(t)-H̅|≤max{1,k^s_i}|δ_i|=k_1|δ_i|, ∀ t≥ 0, where k_1=max_i∈𝒩{1,k^s_i}. This means |ẑ_i-H̅| can always be linearly bounded by |δ_i|, whether z_i(t) is singular or not.





 §.§ Proof of Theorem <ref>

Recall the definition of δ_i in Appendix <ref>, and let δ=(δ_1,⋯,δ_N)^⊤. The convergence analysis of the δ subsystem has been established by <cit.> and is thus omitted here.
The following proof focuses on the analysis of e and x̃ subsystems, and the comprehensive analysis of the closed-loop system.

(1) Analysis of e subsystem. Consider the function

    V_1=e^⊤e+1/2∑_i∈𝒩∑_j∈𝒩_i(α_ij-α̅)^2+1/2∑_i∈𝒩∑_j∈𝒩_i(β_ij-β̅)^2,

where α̅ and β̅ are positive constants to be determined. Its derivative along (<ref>) and (<ref>) is given by

    V̇_1=   ∑_i∈𝒩∑_j∈𝒩_i(α_ij-α̅)α̇_ij+∑_i∈𝒩∑_j∈𝒩_i(β_ij-β̅)β̇_ij
       -2e^⊤L̅_⊗e+Λ_3+Λ_4,

where Λ_3=-2e^⊤(DB)_⊗S(D^⊤_⊗e) and Λ_4=2e^⊤M_⊗ϕ. Note

    Λ_3=   -2∑_i∈𝒩∑_j∈𝒩_iβ_ije_i^⊤S(e_i-e_j)
    
    =   -∑_i∈𝒩∑_j∈𝒩_iβ_ij(e_i-e_j)^⊤S(e_i-e_j)
    ≤    -∑_i∈𝒩∑_j∈𝒩_iβ_ij(|e_i-e_j|_1-mϵ_1 η_t),

where the second equality follows from Assumption <ref> and and third one follows the fact that β_ij≥0 and yS(y)≥|y|-ϵ_1 η_t with y∈ℝ. By (<ref>), we obtain |ϕ|≤ϕ_1|x|+ϕ_2 with ϕ_1=|{kH_i+ẑ_i^-1Ḣ_i}|, ϕ_2=|((kR_1+ẑ_1^-1Ṙ_1)^⊤,⋯,(kR_N+ẑ_N^-1Ṙ_N)^⊤)^⊤|. The boundedness of H_i, Ḣ_i, ẑ_i^-1, R_i and Ṙ_i implies the boundedness of ϕ_1 and ϕ_2. By denoting m̅=|M_⊗|, we have

    Λ_4≤   2m̅ϕ_1|e|^2+2m̅ϕ_1√(N)|e||x̃|+2m̅(ϕ_1√(N)|r^*|+ϕ_2)|e|
    ≤   b_1|e|^2+σ_1h_1/N|x̃|^2+2m̅(ϕ_1√(N)|r^*|+ϕ_2)|e|,

where σ_1>0, b_1=2m̅ϕ_1+m̅^2ϕ_1^2N^2/(σ_1h_1), the first inequality is obtained by (<ref>), and the second inequality follows from Young's inequality <cit.>. By Assumption <ref> again, we have

    ∑_i∈𝒩∑_j∈𝒩_i(α_ij-α̅)α̇_ij=   ∑_i∈𝒩∑_j∈𝒩_i(α_ij-α̅)|e_i-e_j|^2
    
    =   2e^⊤(L̅-α̅L)_⊗e.

Substituting (<ref>), (<ref>), (<ref>) and (<ref>) into (<ref>) yields

    V̇_1≤    -2α̅e^⊤L_⊗e+b_1|e|^2-β̅∑_i∈𝒩∑_j∈𝒩_i|e_i-e_j|_1+σ_1h_1/N|x̃|^2
       +2m̅(ϕ_1√(N)|r^*|+ϕ_2)|e|+mϵ_1 N^2β̅η_t.

With the fact that e^⊤(1_N⊗ I_m)≡ 0, we conclude from <cit.> that -e^⊤L_⊗e≤-λ_2(L)|e|^2. It can be shown that

    -∑_i∈𝒩∑_j∈𝒩_i|e_i-e_j|_1
    
    =   -2|D^⊤_⊗e|_1≤-2√(e^⊤(DD^⊤)_⊗e)≤-2√(λ_2(L))|e|,

where the last inequality follows from L=DD^⊤. By selecting β̅=m̅(ϕ_1√(N)|r^*|+ϕ_2)/√(λ_2(L)), we have

    V̇_1≤    -2λ_2(L)α̅|e|^2+b_1|e|^2+σ_1h_1/N|x̃|^2+mϵ_1 N^2β̅η_t.


(2) Analysis of x̃ subsystem. Inspired by the form of ẑ^-1_i∇_t f_i(x_i,t), adding ±H^-1∑_i∈𝒩∇_t f_i(x_i,t) in (<ref>) yields

    ẋ̃̇=   -k/N1^⊤_N∇ F(x,t)+Λ_1+Λ_2,

where Λ_1=H^-1∑_i∈𝒩∇_t f_i(r^*,t)-∇_t f_i(x_i,t) and Λ_2=∑_i∈𝒩(H̅^-1-ẑ^-1_i)∇_t f_i(x_i,t)/N. The latter follows from H^-1=H̅^-1/N. We find that Λ_1 can be bounded by e and x̃, i.e.,


    Λ_1=-H^-1∑_i∈𝒩Ḣ_i(e_i+x̃)≤h_2√(mN)/h_1|e+1_N⊗x̃|.

Note that H̅^-1-ẑ^-1_i=H̅^-1(ẑ_i-H̅)ẑ^-1_i. By Lemma <ref>, along with the boundedness of H̅^-1 and ẑ^-1_i, there must be a positive constant k_2=k_1|H̅^-1| max_i∈𝒩{|ẑ^-1_i|} such that

    Λ_2≤1/N∑_i∈𝒩k_2|δ_i||∇_t f_i(x_i,t)|≤1/Nk_3|δ||x|+1/√(N)k_4|δ|,

where k_3=k_2max_i∈𝒩{|Ḣ_i|}, k_4=k_2max_i∈𝒩{|Ṙ_i|} and the second inequality is obtained by |∇_t f_i(x_i,t)|≤|Ḣ_i||x_i|+|Ṙ_i|. Consider a Lyapunov function candidate V_2=|x̃|^2/2. Thus,

    V̇_2=   -k/Nx̃^⊤1^⊤_N∇ F(x,t)+x̃^⊤Λ_2+x̃^⊤Λ_1
    
    =   -k/Nx̃^⊤1^⊤_N∇ F(1_N⊗(x̃+r^*),t)+x̃^⊤Λ_2-k/Nx̃^⊤1^⊤_N
       ×(∇ F(x,t)-∇ F(1_N⊗(x̃+r^*),t))+x̃^⊤Λ_1.

Invoking Assumption <ref>, f(r(t),t) is uniformly h_1-strongly convex with respect to r(t). This implies

    -k/Nx̃^⊤1^⊤_N∇ F(1_N⊗(x̃+r^*),t)
    ≤   -k/Nx̃^⊤(∇ f(x̃+r^*,t)-∇ f(r^*,t))≤-kh_1/N|x̃|^2.

By Assumption <ref>, ∇ f_i(x_i,t) is uniformly θ_i-Lipschitz with θ_i=min_t≥0|H_i(t)|. By denoting θ=max_i∈𝒩{θ_i}, we have 

    -k/Nx̃^⊤1^⊤_N(∇ F(x,t)-∇ F(1_N⊗(x̃+r^*),t))
    ≤   kθ/√(N)|x̃||e|≤σ_2h_1/N|x̃|^2+b_2|e|^2,

where σ_2>0 and b_2=k^2θ^2/(4σ_2h_1). We also conclude that

    x̃^⊤Λ_2≤   1/Nk_3|x̃||δ||x|+1/√(N)k_4|x̃||δ|
    ≤   k_3/√(N)|δ||x̃|^2+k_3|δ|/N|x̃||e|+k_3|r^*|+k_4/√(N)|x̃||δ|
    ≤   b_3|δ||x̃|^2+(σ_3+σ_4)h_1/N|x̃|^2+(b_4|e|^2+b_5)|δ|^2,

where σ_3,σ_4>0, b_3=k_3/√(N), b_4=k_3^2/(4σ_3h_1 N), b_5=(k_3|r^*|+k_4)^2/(4σ_4h_1). With a positive constant σ_5, we have 

    x̃^⊤Λ_1≤σ_5h_1/N|x̃|^2+b_6|x̃|^2+b_7|e|^2,

where b_6=h_2N√(m)/h_1 and b_7=mN^2h_2^2/(4σ_5h_1^3). Substituting (<ref>), (<ref>), (<ref>) and (<ref>) into (<ref>) yields

    V̇_2≤   (∑_i=2^5σ_i-k)h_1/N|x̃|^2+b_6|x̃|^2+b_3|δ||x̃|^2
       +(b_2+b_7)|e|^2+(b_4|e|^2+b_5)|δ|^2.


(3) Closed-loop synthesis. Given V_1 and V_2, we further consider the composite Lyapunov function candidate of e and x̃ subsystems, i.e., V=V_1+V_2. Combining (<ref>) and (<ref>) yields

    V̇=   V̇_1+V̇_2
    ≤-l_1|x̃|^2-l_2|e|^2+W_3+mϵ_1 N^2β̅η_t,

where l_1=(k-∑_i=1^5σ_i)h_1/N-b_6, l_2=2λ_2(L)α̅-b_1-b_2-b_7 and W_3=b_3|x̃|^2|δ|+b_4|e|^2|δ|^2+b_5|δ|^2. Then, one may first choose k>b_6N/h_1+ϵ_3, where ϵ_3>0 such that l_1>0. By choosing α̅>(b_1+b_2+b_7)/(2λ_2(L)), we have l_2>0. Integrating both sides of (<ref>) yields

    V(t)+l_1∫_0^∞ |x̃|^2  dt+l_2∫_0^∞ |e|^2  dt-∫_0^∞ W_3 dt
    ≤    V(0)+mϵ_1N^2β̅/c <∞.

We next give the boundedness analysis of ∫_0^∞ W_3 dt. By Lemma <ref>, x_i is bounded in a finite time, and so is e. With the fact that r^* is bounded concluded by Assumption <ref>, we know x̃ is also bounded in a finite time by (<ref>). Moreover, by <cit.>, there must be a finite time instant T such that for any 0≤ t<T, δ is bounded, and for any t≥ T, δ_i=z_i(t)-H̅(t)=0, ∀ i∈𝒩. Thus, it is clear that for any 0≤ t<T, W_3 is bounded, which together with the continuity of e, x̃ and δ given by (<ref>), (<ref>) and (<ref>), respectively, results in that ∫_0^T W_3 dt is bounded. Note that ∫_0^∞ W_3 dt=∫_0^T W_3 dt+∫_T^∞ W_3 dt=∫_0^T W_3 dt, implying the boundedness of ∫_0^∞ W_3 dt. By recalling (<ref>), we know V is bounded, and e,x̃∈ℒ_2. It follows from V_1 and V_2 that e, x̃, α_ij and β_ij are bounded, and thus x is bounded. This implies the boundedness of ∇ f_i(x_i,t) and ∇_t f_i(x_i,t). Then, by (<ref>) and (<ref>), ė and ẋ̃̇ are bounded. By Barbalat's Lemma <cit.>, we have lim_t→∞e(t)=0 and lim_t→∞x̃(t)=0, which further implies lim_t→∞x_i(t)=r^*(t).



 §.§  Reanalysis of Finite Escape Time for Corollary <ref>



    Suppose Assumptions <ref>, <ref>, <ref> hold. For system (<ref>) and (<ref>)-(<ref>) with Algorithm <ref>, the state x_i(t) does not have a finite escape time, i.e., for any t̅∈[0,∞)  the state x_i(t) is bounded for all t∈[0,t̅]. 
    
        
    
    Denote x_ip∈ℝ the p-th row element of x_i, where i∈𝒩 and p∈ℳ. Similar subscripts also appear in certain matrices that follow. We take x_ip as an example to check its boundedness in a finite time. According to Assumption <ref>.(<ref>), without loss of generality, let ∇f̂_i(x_i,t)=Y(x_i,t)+Z(x_i,t) which satisfies |Y(x_i,t)|≤μ_2|x_i| and |Z(x_i,t)|≤μ_3. Then the dynamics of x_ip is given by
    
    ẋ_ip=   -k(H_ipx_i+R_ip+Y_ip+Z_ip)
       -ẑ_ip^-1(Ḣ_ix_i+Ṙ_i+g_ix_i+s_i)
       +c_1(t)∑_j∈𝒩_iS(x_ip-x_jp)+W_1+W_2,

    where 0≤ c_1(t)=∫_0^tmϵ_1 η_t dt=(mϵ_1/ϵ_2)(1-η_t)<∞, W_1=-∑_j∈𝒩_i(∫_0^t |x_i-x_j|^2  dt) (x_ip-x_jp), and W_2=-∑_j∈𝒩_i(∫_0^t|x_i-x_j|_1 dt)S(x_ip-x_jp). We first consider the first half of (<ref>), i.e.
    
    ẋ_ip=   -k(H_ipx_i+R_ip+Y_ip+Z_ip)
       -ẑ_ip^-1(Ḣ_ix_i+Ṙ_i+g_ix_i+s_i)
       +c_1(t)∑_j∈𝒩_iS(x_ip-x_jp)=:g_i(x_ip,t).

    For all t∈[0,t̅], we know g_i(x_ip,t) is piecewise continuous with respect to t. With Assumptions <ref>, <ref>, <ref> satisfied, we know g_i(x_ip,t) is global Lipschitz with respect to x_ip. Then, by <cit.>, we know (<ref>) has a unique solution over [0,t̅]. In view of the fact that t̅ can be arbitrarily large, we make the following statement,
    
        
  S1. x_ip in system (<ref>) does not have a finite escape time, ∀ i∈𝒩 and ∀ p∈ℳ.
    
    We further consider W_1 and W_2 in system (<ref>). Assume that i^*,p^*∈max_i∈𝒩,p∈ℳ|x_ip|. Without loss of generality, let x_i^*p^*>0. If x_i^*=x_j, ∀ j∈𝒩_i^*, we have W_1=W_2=0. If there exists at least one j∈𝒩_i^* such that x_i^*≠ x_j, it follows that W_1,W_2≤ 0. Similarly, assume that x_i^*p^*<0. If x_i^*=x_j, ∀ j∈𝒩_i^*, we have W_1=W_2=0. If there exists at least one j∈𝒩_i^* such that x_i^*≠ x_j, then we have W_1,W_2≥0. Therefore, we make another statement,
    
        
  S2. W_1+W_2≤0 if x_i^*p^*>0 and W_1+W_2≥0 if x_i^*p^*<0.
       
    
By combining statements S1 and S2, we conclude from (<ref>) that the trajectory of x_i^*p^* does not escape to infinity at a finite time. With the fact that i^*,p^*∈max_i∈𝒩,p∈ℳ|x_ip|, thus, x_i(t) does not have a finite escape time, ∀ i∈𝒩.
 



 §.§ Proof of Corollary <ref>

Define a diagonal matrix Ω, whose order is equal to the number of edges in the graph 𝒢. The nodes corresponding to non-zero elements in column i of matrix D are denoted as i_1 and i_2, respectively. The diagonal element Ω_ii of Ω is defined as ω_i_1i_2. Then, by defining z=(z_1,⋯,z_N) and  ρ=(ρ_1,⋯,ρ_2), system (<ref>) can be rewritten as ż=-(DΩ)_⊗(D^⊤_⊗z)+ρ̇. Define consensus error δ̅=M_⊗z. Then, the dynamics of δ subsystem can be written as

    δ̇̅̇=   -(DΩ)_⊗(D^⊤_⊗δ̅)+M_⊗ρ̇.

(1) Analysis of δ̅ subsystem. 
Then, consider the Lyapunov function candidate 
W=(δ̅^⊤δ̅). Its derivative is given by

    Ẇ=   -2(δ̅^⊤(DΩ)_⊗(D^⊤_⊗δ̅))+2(δ̅^⊤M_⊗ρ̇).

We first analyze its last term. It can be concluded that

    2(δ̅^⊤M_⊗ρ̇)=   1/N∑_i=1^N∑_j=1^N(δ̅_i-δ̅_j)^⊤(ρ̇_i-ρ̇_j)
    ≤   1/N∑_i=1^N∑_j=1^N|δ̅_i-δ̅_j|_1|ρ̇_i-ρ̇_j|_∞,

which is obtained by Hölder's inequality <cit.>. By triangular inequality |ρ̇_i-ρ̇_j|_∞≤|ρ̇_i|_∞+|ρ̇_j|_∞, we have

    2(δ̅^⊤M_⊗ρ̇)≤   1/N∑_i=1^N∑_j=1^N(|ρ̇_i|_∞+|ρ̇_j|_∞)|δ̅_i-δ̅_j|_1
    ≤   max_i{∑_j=1,j≠ i^N(|ρ̇_i|_∞+|ρ̇_j|_∞)|δ̅_i-δ̅_j|_1}
    ≤   N-1/2∑_i∈𝒩∑_j∈𝒩_i(|ρ̇_i|_∞+|ρ̇_j|_∞)|δ̅_i-δ̅_j|_1,

where the last inequality is obtained by using Assumption <ref>. Then, with the similar analysis in (<ref>), we have 

    -2(δ̅^⊤(DΩ)_⊗(D^⊤_⊗δ̅))=-∑_i∈𝒩∑_j∈𝒩_iω_ij||δ̅_i-δ̅_j||_1.

Substituting (<ref>) and (<ref>) into (<ref>) yields

    Ẇ≤   ∑_i∈𝒩∑_j∈𝒩_i(N-1/2(χ̅_i+χ̅_j)-ω_ij)||δ̅_i-δ̅_j||_1.

With the similar analysis in (<ref>), if we select ω_ij>((N-1)/2)(|ρ̇_i|_∞+|ρ̇_j|_∞), we have Ẇ≤-2√(λ_2)W^1/2. According to <cit.>, there exists a finite time T_1 such that for all t≥ T_1, δ̅(t)=0, i.e., z_i(t)=z_j(t), ∀ i,j∈𝒩. Moreover, by using Assumption <ref>, we have

    1^⊤_Nξ̇=-(1^⊤_NDΩ)_⊗(D^⊤_⊗z)=0.

By recalling the initial condition ∑_i∈𝒩ξ_i(0)=0, we have ∑_i∈𝒩ξ_i(t)=0. Combining (<ref>) yields

    ∑_i∈𝒩z_i(t)=∑_i∈𝒩ρ_i,   ∀ t≥ 0.

With the fact that z_i(t)=z_j(t), ∀ i,j∈𝒩, ∀ t≥ T_1, we thus have z_i=ρ̅:=∑_i∈𝒩ρ_i(x_i,t)/N, ∀ t≥ T_1.

Define z̃=1^⊤_Nz/N-ρ̅. Then we have

    z=δ̅+1_N⊗z̃+1_N⊗ρ̅.
 
With the fact that 1^⊤_Nδ̅≡ 0, we have 

    ∑_i∈𝒩z_i(t)=Nz̃+∑_i∈𝒩ρ_i.

By recalling (<ref>), we know that z̃≡0. This means that δ̅_i=z_i-ρ̅=:δ_i. Therefore, the finite-time convergence of δ̅_i is equivalent to the finite-time convergence of δ_i, which will be used for the following analysis.

(2) Analysis of e subsystem. Consider the function

    V_1=e^⊤e+1/2∑_i∈𝒩∑_j∈𝒩_i(α_ij-α̅)^2+1/2∑_i∈𝒩∑_j∈𝒩_i(β_ij-β̅)^2,

where α̅ and β̅ are positive constants to be determined. Its derivative along (<ref>) and (<ref>) is given by

    V̇_1=   ∑_i∈𝒩∑_j∈𝒩_i(α_ij-α̅)α̇_ij+∑_i∈𝒩∑_j∈𝒩_i(β_ij-β̅)β̇_ij
       -2e^⊤L̅_⊗e+Λ_3+Λ_4,

where Λ_3=-2e^⊤(DB)_⊗S(D^⊤_⊗e) and Λ_4=2e^⊤M_⊗ϕ. Note

    Λ_3=   -2∑_i∈𝒩∑_j∈𝒩_iβ_ije_i^⊤S(e_i-e_j)
    
    =   -∑_i∈𝒩∑_j∈𝒩_iβ_ij(e_i-e_j)^⊤S(e_i-e_j)
    ≤    -∑_i∈𝒩∑_j∈𝒩_iβ_ij(|e_i-e_j|_1-mϵ_1 η_t),

where the second equality follows from Assumption <ref> and and third one follows the fact that β_ij≥0 and yS(y)≥|y|-ϵ_1 η_t with y∈ℝ. By (<ref>), we obtain |ϕ|≤ϕ_1|x|+ϕ_2 with 

    ϕ_1=   |{k(H_i+μ_2I_m)+ẑ_i^-1(Ḣ_i+μ_4I_m)}|,
    ϕ_2=   |[ [ k(R_1+μ_31_m)+ẑ_1^-1(Ṙ_1+|s(x_1,t)|1_m);                                       ⋮; k(R_N+μ_31_m)+ẑ_N^-1(Ṙ_N+|s(x_N,t)|1_m);                                         ]]|.

The boundedness of H_i, Ḣ_i, ẑ_i^-1, R_i, Ṙ_i and |s(x_i,t)| implies the boundedness of ϕ_1 and ϕ_2. With m̅:=|M_⊗|, we have

    Λ_4≤   2m̅ϕ_1|e|^2+2m̅ϕ_1√(N)|e||x̃|+2m̅(ϕ_1√(N)|r^*|+ϕ_2)|e|
    ≤   b_1|e|^2+σ_1μ_1/N|x̃|^2+2m̅(ϕ_1√(N)|r^*|+ϕ_2)|e|,

where σ_1>0, b_1=2m̅ϕ_1+m̅^2ϕ_1^2N^2/(σ_1μ_1), the first inequality is obtained by (<ref>), and the second inequality follows from Young's inequality <cit.>. By Assumption <ref> again, we have

    ∑_i∈𝒩∑_j∈𝒩_i(α_ij-α̅)α̇_ij=   ∑_i∈𝒩∑_j∈𝒩_i(α_ij-α̅)|e_i-e_j|^2
    
    =   2e^⊤(L̅-α̅L)_⊗e.

Substituting (<ref>), (<ref>), (<ref>) and (<ref>) into (<ref>) yields

    V̇_1≤    -2α̅e^⊤L_⊗e+b_1|e|^2-β̅∑_i∈𝒩∑_j∈𝒩_i|e_i-e_j|_1+σ_1μ_1/N|x̃|^2
       +2m̅(ϕ_1√(N)|r^*|+ϕ_2)|e|+mϵ_1 N^2β̅η_t.

With the fact that e^⊤(1_N⊗ I_m)≡ 0, we conclude from <cit.> that -e^⊤L_⊗e≤-λ_2(L)|e|^2. It can be shown that

    -∑_i∈𝒩∑_j∈𝒩_i|e_i-e_j|_1
    
    =   -2|D^⊤_⊗e|_1≤-2√(e^⊤(DD^⊤)_⊗e)≤-2√(λ_2(L))|e|,

where the last inequality follows from L=DD^⊤. By selecting β̅=m̅(ϕ_1√(N)|r^*|+ϕ_2)/√(λ_2(L)), we have

    V̇_1≤    -2λ_2(L)α̅|e|^2+b_1|e|^2+σ_1μ_1/N|x̃|^2+mϵ_1 N^2β̅η_t.


(3) Analysis of x̃ subsystem. Denote ρ=∑_i∈𝒩ρ_i(x_i,t). Here we have ṙ^*=-ρ^-1(t)∑_i∈𝒩∇_t f_i(r^*,t). Then, adding ±ρ^-1∑_i∈𝒩∇_t f_i(x_i,t) in (<ref>) yields

    ẋ̃̇=   -k/N1^⊤_N∇ F(x,t)+Λ_1+Λ_2,

where Λ_1=ρ^-1∑_i∈𝒩∇_t f_i(r^*,t)-∇_t f_i(x_i,t) and Λ_2=∑_i∈𝒩(ρ̅^-1-ẑ^-1_i)∇_t f_i(x_i,t)/N. The latter follows from ρ^-1=ρ̅^-1/N. We conclude that

    Λ_1=   -ρ^-1∑_i∈𝒩(Ḣ_i+g_i)(e_i+x̃)+s_i(x_i,t)-s_i(r^*,t)
    ≤   (h_2+μ_4)√(mN)/μ_1|e+1_N⊗x̃|+2Nμ_5/μ_1s̅(t).

Note that ρ̅^-1-ẑ^-1_i=ρ̅^-1(ẑ_i-ρ̅)ẑ^-1_i. By using the similar analysis in Lemma <ref>, we have |ẑ_i(t)-ρ̅|≤ k_1|δ_i|, ∀ t≥ 0. Along with the boundedness of ρ̅^-1 and ẑ^-1_i, there must be a positive constant k_2=k_1|ρ̅^-1| max_i∈𝒩{|ẑ^-1_i|} such that

    Λ_2≤1/N∑_i∈𝒩k_2|δ_i||∇_t f_i(x_i,t)|≤1/Nk_3|δ||x|+1/√(N)k_4|δ|,

where k_3=k_2max_i∈𝒩{|Ḣ_i|+|g_i|}, k_4=k_2max_i∈𝒩{|Ṙ_i|+|s_i|} and the second inequality is obtained by 

    |∇_t f_i(x_i,t)|≤(|Ḣ_i|+|g_i|)|x_i|+|Ṙ_i|+|s_i|.

Consider a Lyapunov function candidate V_2=|x̃|^2/2. Thus,

    V̇_2=   -k/Nx̃^⊤1^⊤_N∇ F(x,t)+x̃^⊤Λ_2+x̃^⊤Λ_1
    
    =   -k/Nx̃^⊤1^⊤_N∇ F(1_N⊗(x̃+r^*),t)+x̃^⊤Λ_2-k/Nx̃^⊤1^⊤_N
       ×(∇ F(x,t)-∇ F(1_N⊗(x̃+r^*),t))+x̃^⊤Λ_1.

Invoking Assumption <ref>.(<ref>), f(r(t),t) is uniformly μ_1-strongly convex with respect to r(t). This implies

    -k/Nx̃^⊤1^⊤_N∇ F(1_N⊗(x̃+r^*),t)
    ≤   -k/Nx̃^⊤(∇ f(x̃+r^*,t)-∇ f(r^*,t))≤-kμ_1/N|x̃|^2.

Moreover, ∇ f_i(x_i,t) is uniformly θ_i-Lipschitz with θ_i=min_t≥0|ρ_i(x_i,t)|. By denoting θ=max_i∈𝒩{θ_i}, we have 

    -k/Nx̃^⊤1^⊤_N(∇ F(x,t)-∇ F(1_N⊗(x̃+r^*),t))
    ≤   kθ/√(N)|x̃||e|≤σ_2μ_1/N|x̃|^2+b_2|e|^2,

where σ_2>0 and b_2=k^2θ^2/(4σ_2μ_1). We also conclude that

    x̃^⊤Λ_2≤   1/Nk_3|x̃||δ||x|+1/√(N)k_4|x̃||δ|
    ≤   k_3/√(N)|δ||x̃|^2+k_3|δ|/N|x̃||e|+k_3|r^*|+k_4/√(N)|x̃||δ|
    ≤   b_3|δ||x̃|^2+(σ_3+σ_4)μ_1/N|x̃|^2+(b_4|e|^2+b_5)|δ|^2,

where σ_3,σ_4>0, b_3=k_3/√(N), b_4=k_3^2/(4σ_3μ_1 N), b_5=(k_3|r^*|+k_4)^2/(4σ_4μ_1). With σ_5,σ_6>0, we have 

    x̃^⊤Λ_1≤σ_5μ_1/N|x̃|^2+b_6|x̃|^2+b_7|e|^2+σ_6μ_1/N|x̃|^2+b_8s̅^2(t),

where b_6=(h_2+μ_4)N√(m)/μ_1, b_7=mN^2(h_2+μ_4)^2/(4σ_5μ_1^3) and b_8=2N^2μ_5/(4σ_6μ_1^2). Substituting (<ref>), (<ref>), (<ref>) and (<ref>) into (<ref>) yields

    V̇_2≤   (∑_i=2^6σ_i-k)μ_1/N|x̃|^2+b_6|x̃|^2+b_3|δ||x̃|^2
       +(b_2+b_7)|e|^2+(b_4|e|^2+b_5)|δ|^2+b_8s̅^2(t).


(4) Closed-loop synthesis. Given V_1 and V_2, we further consider the composite Lyapunov function candidate of e and x̃ subsystems, i.e., V=V_1+V_2. Combining (<ref>) and (<ref>) yields

    V̇=   V̇_1+V̇_2
    ≤-l_1|x̃|^2-l_2|e|^2+b_8s̅^2+W_3+mϵ_1 N^2β̅η_t,

where l_1=(k-∑_i=1^6σ_i)μ_1/N-b_6, l_2=2λ_2(L)α̅-b_1-b_2-b_7 and W_3=b_3|x̃|^2|δ|+b_4|e|^2|δ|^2+b_5|δ|^2. Then, one may first choose k>b_6N/μ_1+ϵ_4, where ϵ_4>0 such that l_1>0 and  choose α̅>(b_1+b_2+b_7)/(2λ_2(L)) such that l_2>0. Integrating both sides of (<ref>) yields

    V(t)+l_1∫_0^∞ |x̃|^2  dt+l_2∫_0^∞ |e|^2  dt
       -b_8∫_0^∞s̅^2(t) dt -∫_0^∞ W_3 dt≤ V(0)+mϵ_1N^2β̅/c <∞.

According to Assumption <ref>.(<ref>), we know ∫_0^∞s̅^2(t) dt is bounded. We next give the boundedness analysis of ∫_0^∞ W_3 dt. By Lemma <ref>, x_i is bounded in a finite time, and so is e. With the fact that r^* is bounded concluded by Assumption <ref>, we know x̃ is also bounded in a finite time by (<ref>). Moreover, according to (1) Analysis of δ subsystem, there exies a finite time instant T_1 such that for any 0≤ t<T_1, δ is bounded, and for any t≥ T_1, δ_i=z_i(t)-ρ̅(t)=0, ∀ i∈𝒩. Thus, it is clear that for any 0≤ t<T_1, W_3 is bounded, which together with the continuity of e, x̃ and δ given by (<ref>), (<ref>) and (<ref>), respectively, results in that ∫_0^T_1 W_3 dt is bounded. Note that ∫_0^∞ W_3 dt=∫_0^T_1 W_3 dt+∫_T_1^∞ W_3 dt=∫_0^T_1 W_3 dt, implying the boundedness of ∫_0^∞ W_3 dt. By recalling (<ref>), we know V is bounded, and e,x̃∈ℒ_2. It follows from V_1 and V_2 that e, x̃, α_ij and β_ij are bounded, and thus x is bounded. This implies the boundedness of ∇ f_i(x_i,t) and ∇_t f_i(x_i,t). Then, by (<ref>) and (<ref>), ė and ẋ̃̇ are bounded. By Barbalat's Lemma <cit.>, we have lim_t→∞e(t)=0 and lim_t→∞x̃(t)=0, which further implies lim_t→∞x_i(t)=r^*(t).


IEEEtran




 Paper Title: Observability of complex systems via conserved quantities.



Multipartite Entanglement for Multi-node Quantum Networks
    G. Araneda
    August 5, 2024
=========================================================




     ^aDepartment of Mathematics,
      Florida State University,
  Tallahassee, FL 32306-4510, USA



     ^bInstitute of Molecular Biophysics,
      Florida State University,
  Tallahassee, FL 32306-4510, USA



     ^cProgram in Neuroscience,
      Florida State University,
  Tallahassee, FL 32306-4510, USA



     ^dDepartment of Mathematics,
      University of Kentucky,
  Lexington, KY 40506-0027, USA


§ ABSTRACT Many systems in biology, physics, and engineering are modeled by nonlinear dynamical systems
where the states are usually unknown and only a subset of the state variables can be physically measured. Can we understand the full system from what we measure? In the mathematics literature, this question is framed as the observability problem. It has to do with recovering information about the state variables from the observed states (the measurements). In this paper, we relate the observability problem to another structural feature of many models relevant in the physical and biological sciences: the conserved quantity. For models based on systems of differential equations,
conserved quantities offer desirable properties such as dimension reduction which simplifies model analysis.
Here, we use differential embeddings to show that conserved quantities involving a set of special variables provide more flexibility in what can be measured to address the observability problem for systems of interest in biology.
Specifically, we provide conditions under which a collection of conserved quantities make the system observable.
We apply our methods to provide alternate measurable variables in models where conserved quantities have been used for model analysis historically in biological contexts.








§ INTRODUCTION

A fundamental question in nonlinear dynamics is whether the entire state of a system can be inferred from measurements of a subset of outputs of the states that comprise the system. In the mathematics literature, this is referred to as the observability problem <cit.>. Briefly, a dynamical system is called observable if one can obtain complete information about the internal state of the dynamical system from measurements of a subset of the outputs.

This question is of central importance in physical and biological applications. Nonlinear dynamical systems have been used to model chemical reaction networks <cit.>, combustion reaction networks <cit.>, power grids <cit.>, biophysical networks <cit.>, epidemics <cit.>, and cancer <cit.>.  Observability of such dynamical systems is vital to constructively inform experimentalists and engineers what should be measured to optimize inference of the progress of their work. Most often, all variables involved are unable to be measured. Determining which outputs of a system should be measured to understand the full system is thus useful and essential for scientific and technological progress across disciplines.

For example, in determining the kinetic properties of an enzymatic reaction, one of the biochemical species must be measured to understand reaction rate. It is a challenging endeavor to simultaneously measure all constituents of the enzymatic reaction. Observable dynamical systems can inform biochemists of which species to track to understand the full system. Similarly, in an infection outbreak, observable dynamical systems can inform epidemiologists of what populations to track to optimally understand the dynamics of an epidemic. It is thus vital to develop mathematical theory and methods to ascertain whether a dynamical system is observable, and, if so, to determine which observables render the dynamical system observable.

Several methods exist to determine whether a system is observable.  A longstanding method is to look at the Lie derivatives of the observables with respect to the governing nonlinear vector field and construct the Jacobian matrix of the Lie derivatives <cit.>. Parameter regimes where the Jacobian has full rank are those where the chosen observable renders the full system observable. Another related approach transforms the phase space of the nonlinear system via a differential embedding and considers the rank of the Jacobian matrix as an indicator of observability <cit.>.  Still another very popular method is to construct the associated graph of the nonlinear system and study the strongly connected component decomposition <cit.>. A central result of this graphical approach is that observing dynamics of the source nodes is necessary and sufficient for observability of the full system.  Finally there are methods based on a strongly positive definite condition and sensor selection based on optimization to ascertain observability <cit.>.

In this paper, we expand on the aforementioned results by considering the effect of another property of dynamical systems that manifests in several biological and physical applications:  the conserved quantity. A conserved quantity of a dynamical system is a function of the state variables that remains invariant in time. Typical uses of a conserved quantity is dimension reduction of the system under scrutiny. Because it defines a dependency between the state variables of system, the dynamics of the full system are contracted to a submanifold of the phase space, thereby potentially simplifying analysis.

We show that conserved quantities in combination with differential embeddings provides a means to identify alternative observables in a system that render a system observable. We emphasize that existing methods for determining observability do not consider conserved quantities or how they impact the observability of a system. Therefore, the available methods are unable to detect alternative variables that render a system observable. For example, we show in this paper with an example that the graphical approach can miss alternate observables imputed by conserved quantities.

Our approach is of interest to experimentalists and engineers because it provides a means to identify system outputs to measure that could reveal the internal state of the process being studied. Current methods for identifying observables may lead to concluding that the only observable is an output that cannot be measured. Our method provides flexibility in such scenarios.

Mathematically, our contribution is to append to the rich literature on observable and controllable systems.  We claim that if system dynamics can be contracted to a submanifold that is inherent to the system, there will be more observables than what previous methods predict. Furthermore, our main result describes conditions for which submanifold to contract dynamics if more than one are inherent to the system.









§ BACKGROUND


We consider dynamical systems of the form

    dx(t)/dt = f(x(t))

with observable variables y = g(x̂), where 
f:ℝ^n→ℝ^n and g:ℝ^m→ℝ^m are differentiable functions. We assume that we can only measure a subset of the state variables represented by x̂∈ℝ^m and the initial state x_0∈Ω_0⊂ℝ^n is unknown.


The system in Eq. (<ref>) is observable if there is a bijection between the initial states in Ω_0 and the set of trajectories of the observed outputs y(t) for t≥0 <cit.>.


In the following, we describe popular approaches for determining observability of a nonlinear system as in Eq. (<ref>). The approach in <cit.> considers a k-dimensional differential embedding Φ:ℝ^n→ℝ^km given by Φ(x) = (g(x̂),ġ(x̂),⋯,g^(k)(x̂)) (derivatives with respect to time). The map Φ is locally invertible at x_0 if the Jacobian has full rank. That is, the map Φ is locally invertible at x_0 if

    (∂Φ/∂x|_x_0) = n.

The system in Eq. (<ref>) is locally observable if and only if Eq. (<ref>) holds <cit.>.

Another approach is the graphical approach. The graphical approach <cit.> associates a directed graph 𝒢 to the system given by Eq. (<ref>), where the nodes of 𝒢 are x_1, ⋯, x_n and there is an edge from x_i to x_j if x_j appears in the differential equation of x_i. We consider the condensation graph of 𝒢 where we collapse the strongly connected components into a node. The graphical approach in <cit.> states that a necessary and sufficient condition for observability of the system in Eq. (<ref>) is to observe the source nodes in 𝒢 and a variable in each strongly connected component of 𝒢. However, those conditions are neither sufficient nor necessary as we will show in the following example.



    SIR models are popular for describing dynamics of an infectious disease and for unveiling key biophysical parameters that govern the transition of a disease from dissipating in a population to persisting in an endemic state <cit.>. Such models are typically composed of three state variables: S representing the number of susceptible individuals in a population, I representing the number of infected individuals, and R representing the number of recovered or removed individuals.  They have been shown to apply to more general settings as well by incorporating spatial and stochastic dynamics in their structure <cit.>. Furthermore, they have been used to study dissemination of information through a social network in a number of studies <cit.>. Hence, SIR models form a crux of much of mathematical epidemiology literature.



One of the simplest SIR models describes the dynamics of an epidemic on a short timescale.  In such instances, the impact infection imparts on population dynamics vastly outweighs birth and death events, so birth and death terms do not manifest in the SIR dynamics. Because of this, the total number of individuals is invariant in time. Such a model is applicable, for example, in describing the dynamics and spread of the flu virus through a population <cit.>.

Here we investigate such a model.  Consider the following SIR model:

    dS/dt   = -β SI 
    dI/dt   = β SI - λ I 
    dR/dt   = λ I ,

where S represents the susceptible population, I represents the infected population, and R represents the recovered population. The parameter β quantifies the infectivity of the infectious disease under consideration; thus, the β SI term captures the rate at which susceptible individuals become infected through contact with infected individuals. The parameter λ quantifies the rate of recovery of an infected individual.  This system contains a conserved quantity, namely the total population. That is, ∀ t, S + I + R = N for a prescribed N ∈ℝ. Because of this conserved quantity, Eq. (<ref>) can be reduced to a two-dimensional system that has the same equilibria and stability as the full system. Such a reduction greatly facilitates analysis.

We depict the associated graph of this model in Fig. <ref>A. According to the graphical approach in <cit.>, it is necessary to measure R to make the system observable and that just measuring I would not make the system observable. However, that conlusion would be wrong as we can get the information for S and R by measuring I and using the conserved quantity N. Indeed, S =  1β( İI + λ) and R=N-I-S = N - I -  1β( İI + λ) expresses both R and S as functions of I, İ = dI/dt, and the conserved quantity N.

    

One could apply the graphical method to

    dS/dt   = -β SI 
    dI/dt   = β SI - λ I 
    dN/dt   = 0

to obtain that observing either I and N (or S and N) suffices to recover all variables, including R=N-I-S, but the measurement of the unchanging quantity N is practically quite different from measuring the varying quantity I.





For m≤ n, a scalar-valued function H:ℝ^m→ℝ is a conserved quantity of Eq. (<ref>) if, for all time and initial conditions,

    dH/dt = 0.


Note that if m=n, then the condition in Eq. <ref> can also be stated as

    ∇ H·dx(t)/dt = ∇ H· f(x(t)) = 0,

where ∇ = (∂/∂ x_1,⋯,∂/∂ x_n).


We can represent ℓ conserved quantities H_1,⋯,H_ℓ by using a function G:ℝ^n→ℝ^ℓ where G = (H_1,⋯,H_ℓ). Note that the Jacobian matrix of G is the zero matrix,

    ∂ G/∂x·dx(t)/dt = ∂ G/∂x· f(x) = 0.






§ RESULTS


For m≤ n, a subset of variables s∈ℝ^m are called sufficient whenever observing these variables makes the system observable.
Next, we consider a partition of the variables in Eq. (<ref>), x = (r,s) where r∈ℝ^n-m and s∈ℝ^m is the set of sufficient variables.

Given a collection of conserved quantities G:ℝ^n→ℝ^ℓ, we describe its Jacobian using the partition above as follows:


    ∂ G/∂x(r,s) = [[   ∂ G_1/∂ r_1(𝐫,𝐬)                  ⋯ ∂ G_1/∂ r_n-m(𝐫,𝐬)   ∂ G_1/∂ s_1(𝐫,𝐬)                  ⋯   ∂ G_1/∂ s_m(𝐫,𝐬);                  ⋮                  ⋱                  ⋮                  ⋮                  ⋱                  ⋮;   ∂ G_ℓ/∂ r_1(𝐫,𝐬)                  ⋯ ∂ G_ℓ/∂ r_n-m(𝐫,𝐬)   ∂ G_ℓ/∂ s_1(𝐫,𝐬)                  ⋯   ∂ G_ℓ/∂ s_m(𝐫,𝐬) ]]
    = [[ ∂ G/∂r(r,s) ∂ G/∂s(r,s) ]]


Now we state our main result.



Let

    {[    ẋ(t)       = f(x(t));       y       =    g(s) ].

be an observable system, where
g:ℝ^m→ℝ^m.
If G:ℝ^n→ℝ^ℓ
is a collection of conserved quantities involving sufficient nodes s and
other variables r where
∂ G/∂s(r,s)
is invertible and ∂ G/∂r(r,s) full rank,
then ∃ĝ:ℝ^n-m→ℝ^m such that the system

    {[    ẋ(t)       = f(x(t));       y       =    ĝ(r) ].

is observable.


Since G consists of conserved quantities,
G=constant.
Then, by the implicit function theorem, there is a function ψ:ℝ^n-m→ℝ^m such that s = ψ(r). Let ĝ = g∘ψ 
Since the system in Eq. (<ref>) is observable, the embedding Φ(x) = (g(s),g'(s),⋯,g^(k)(s)) is injective.
Let Φ̂(z) = (ĝ(r),ĝ'(r),⋯,ĝ^(k)(r)) as illustrated in the following diagram, 



ℝ^n [r, "Φ" ] [rdd, dotted]     ℝ^km [dd, "Φ̂"] 

      

     ℝ^km




Then,

    ∂Φ̂(z)/∂z = ∂Φ(x)/∂x∂Ψ(r)/∂r,
       but   ∂Ψ(r)/∂r = -[∂ G/∂s(r,s)]^-1∂ G/∂r(r,s).

Then,


    ∂ (Φ̂∘Φ)(x)/∂x = ∂Φ̂(x)/∂z ∂Φ(x)/∂x = 
    -[∂ G/∂s(r,s)]^-1∂ G/∂r(r,s)
     ∂Φ(x)/∂x.


Thus, Φ̂∘Φ is one-to-one which makes the system in Eq. (<ref>) is observable.







§ APPLICATIONS

Here we demonstrate that relatively simple systems of interest in biology containing conserved quantities are observable through the lense of Theorem <ref>.



 §.§ Constant Population SIR Model

In the following we first ascertain that Eq. (<ref>) is observable provided the observed state variable is R(t). Then we construct the differential embedding map for the system and show that implementing the conserved quantity allows observing other state variables to render the full system observable.


The system given in Eq. (<ref>) is observable. The observed variable will be R(t). To determine whether or not Eq. (<ref>) is observable with R(t) as the scalar observable, we must look at the Jacobian matrix associated with the Lie derivatives of this system <cit.>.  Writing Eq. (<ref>) compactly as

    d𝐗/dt = f(𝐗)

with 𝐗≡ (S, I, R)^T and f(𝐗) = (-β SI, β SI - λ I, λ I)^T, the Lie derivative of a scalar observable y(t) is given by

    ℒ(y) = ∂ y/∂ t + ∑_i = 1^3 f_i ∂ y/∂𝐗_i

In accordance with the usual computations necessary for ascertaining observability, we compute

    ℒ^0 (R) = R     ℒ^1 (R) = 2λ I     ℒ^2(R) = 4λ (β S - λ)I

and construct the associated Jacobian matrix given by

    𝒥 = ([ ∇ℒ^0(R); ∇ℒ^1(R); ∇ℒ^2(R) ]).

That is, each row of the Jacobian matrix consists of a gradient vector of the Lie derivatives with respect to the state variables of the system. When R(t) is observed, the Jacobian matrix is

    𝒥 = ( [           0           0           1;           0          2λ           0;       4λβ I 4λ(β S - λ)           0 ]),

which has full rank provided λ≠ 0, β≠ 0, and I ≠ 0. Having full rank implies the system is observable.


The corresponding differential embedding is bijective. Consider the embedding Φ(S,I,R) = (R, Ṙ, R̈)^T = (R, λ I, λ(β SI - λ I))^T. This is bijective.


To prove injectivity, let Φ(S_1, I_1, R_1) = Φ(S_2, I_2, R_2). Then

    ([                   R_1;                 λ I_1; λ (β S_1 I_1 - λ I_1) ]) = ([                   R_2;                 λ I_2; λ (β S_2 I_2 - λ I_2) ])

This clearly implies R_1 = R_2 and I_1 = I_2. Finally, we have λ(β S_1I_1 - λ I_1) = λ(β S_2I_2 - λ I_2).  Since I_1 = I_2, this implies S_1 = S_2 and injectivity is proved. For surjectivity, take Φ(S,I,R) = (a,b,c)^T for some (a,b,c)^T in the codomain of Φ. Then clearly we can take R = a, I = b/λ and S = c/λβ b + λ/β as a preimage and surjectivity is proved.

One subtle point is that we must constrain the codomain of Φ to be ℝ^3∖{(a,0,c):a,c ∈ℝ} for it to be surjective. This is completely consistent with the Jacobian in Eq. (<ref>), which says that I ≠ 0 is necessary for observability.  This is also consistent physically, since a situation where I = 0 is not particularly interesting when studying the spread of disease.

With the bijectivity of the differential embedding established, it is sufficient to consider the Jacobian of various embeddings to determine whether or not the observed variable renders the full system observable. From this perspective, we next show that observing I in the absence of the conserved quantity does not render the system observable.

Consider now the differential embedding Ψ(S,I,R) = (I, İ, Ï)^T = (I, β SI - λ I, (β S-λ)^2I - β^2 S I^2)^T. Clearly, Ψ is not injective because the image of a point (S, I, R) is agnostic to the value R takes. 


The conserved quantity renders I a sufficient observable. Consider the same differential embedding Ψ = (I, İ, Ï)^T, but now let I = N - S - R, where we solve for I in the conserved population equation S + I + R = N   ∀ t. The corresponding differential equation system becomes

    dS/dt   = -β S (N - S - R)
    dI/dt   = -Ṡ-Ṙ
    dR/dt   = λ (N - S - R)

The corresponding differential embedding is

    Ψ = ( [                                   I;                     (β S -λ)(N-S-R); (β S - λ)^2(N-S-R) - β^2 S(N-S-R)^2 ])

Then, the resulting Jacobian is

    (∂Ψ/∂𝐗) = ([                          0                          1                          0;               β(N- 2S - R)                          0                    λ - β S;                     F(S,R)                          0 -(β S -λ)^2 + 2β^2S(N-S-R) ] )

where F(S,R) = (β S -λ)(2β(N-S-R)-β S + λ) + β^2(N-S-R)(3S+R-N). Again, provided β≠ 0 and λ≠ 0, (∂Ψ/∂𝐗) has full rank and renders the system observable with the observed variable being I(t).



Relating the two embeddings. Since the system in Eq. (<ref>) is observable, the embedding Φ(S,I,R) = (R,Ṙ,R̈)^T is bijective. Let Φ̂(R,Ṙ,R̈) = Ψ = (I,İ,Ï)^T where I = ψ(S,R) = N-R-S.
Then, Φ̂ is a bijection such that the following diagram commutes.




ℝ^3 [r, "Φ" ] [rdd, dotted]     ℝ^3 [dd, "Φ̂"] 

      

     ℝ^3






  §.§.§ Relating to the Graphical Approach

In summary, the preceding discussion says that Eq. (<ref>) is observable if the observed state is R(t). This is consistent with what is obtained in the corresponding directed graph.

In the directed graph of the original SIR system, the only source node is R (see Figure <ref>A). The graphical approach for determining observability states that observing the source nodes of the directed graph of a system is necessary and sufficient to render the system observable. Consistent with the analysis in the previous section, observing R rendered Eq. (<ref>) observable. Furthermore, in the original system, observing I will not render the system observable as I is not a source node. However, it can be made into a source node by invoking the conserved quantity and transforming the system by setting I = N - S - R (see Figure <ref>B). In the transformed system, I is the only source node, thereby making the system observable by observing I.  We note that if we make the transformation S = N - R - I, then S will become the source node and it will be sufficient to observe S to render the system observable. 


A main takeaway is that the existence of the conserved quantity allows for more flexibility in tracking an epidemic from the perspective of the SIR model. Sans the conserved quantity, one can strictly observe only R, the number of recovered individuals, to understand the full system. Simply observing only S or only I will not do the job.  However, the existence of the conserved quantity says that observing any one of the state variables is sufficient to completely understand the system. Thus, trackers of epidemics have flexibility in measuring the epidemic by observing any one of the subpopulations—whichever one is easiest.







 §.§ Michaelis-Menten Kinetics

The simplest enzyme kinetics are Michaelis-Menten kinetics, applied to enzyme-catalyzed reactions of one substrate and one product <cit.>. An enzyme E binds with its substrate S to form a complex ES which then dissociates into E and P, the product of the enzymatic reaction. The reaction network is as follows:

    E + S <=>[k_1][k_-1] ES ->[k_2] E + P

where k_1, k_-1, k_2 are rate constants quantitating the corresponding reactions. Using the law of mass action, we can derive a model characterizing reaction (<ref>).  Let e ≡ [E], s ≡ [S], c ≡ [ES],  and  p ≡ [P]. Then we have <cit.>

    de/dt   = (k_-1 + k_2)c - k_1es 
    ds/dt   = k_-1c - k_1es 
    dc/dt   = k_1es - (k_-1 + k_2)c 
    dp/dt   = k_2c

There are two conserved quantities in this system:

    e + c = E_0
       s + c + p = S_0

where E_0 ∈ℝ represents the initial amount of enzyme in the system and S_0 ∈ℝ is the initial amount of substrate. The two conserved quantities allow for dimensional reduction of system (<ref>) to a planar system

    ds/dt   = k_-1c - k_1(E_0 - c)s 
    dc/dt   = k_1(E_0 - c)s - (k_-1 + k_2)c

By rescaling s, c, and t and assuming that the concentration of substrate vastly outweighs the concentration of enzyme, we can derive the nondimensionalized system

    dσ/dτ   = -σ + (1 - η + σ)ρ
    εdρ/dτ   = σ - (1+ σ)ρ

where σ≡ k_1 s/(k_-1 + k _2), ρ≡ c/E_0, τ≡ k_1 E_0 t. We define the dimensionless parameters ε≡ E_0k_1/(k_-1 +  k_2) and η≡ k_2/(k_-1 + k_2) with 0 < ε≪ 1. We can thereafter invoke the stationary state approximation <cit.> and project onto the slow manifold <cit.> by assuming ρ = σ(1 + σ)^-1. Substituting this expression into the differential equation for p then yields the classical Michaelis-Menten equation:

    dp/dt = V_maxs/K + s

where V_max≡ k_2 E_0 is the fastest rate possible at which product P can be synthesized and K ≡ (k_-1 + k_2)/k_1 is the dissociation constant.

The derivation and generalization of Eq. (<ref>) to more complicated enzyme-substrate mechanisms are a central focus in the theoretical biochemical literature <cit.>. While such derivations are important for the description of biochemical processes, they do not inform experimentalists of the ramifications of the theoretical models to the experiments themselves.

The conserved quantities in the Michaelis-Menten system confine the 4D dynamics to a two-dimensional submanifold, thereby allotting the desirable property of analytic tractabillity in the system. But what does the conserved quantity imply for experimentalists? Broadly, the existence of a conserved quantity consisting of variables that correspond to sources in the directed graph representation [In the enzyme kinetics section of this paper, we will describe observability strictly through the graphical approach.] increases the number of variables that render the full system observable.

The reaction diagram for system (<ref>) is shown in Figure <ref>A. The product P is the only source, implying that to understand the full system (i.e., to render the system observable), one must observe P. In an experimental setting, the kinetics of a given enzyme are measured and calculated from the observed dynamics of P. In a real setting, if P is easily measurable, then the situation at hand is no problem. However, in many situations, the product P is not directly measurable <cit.>. One must find an alternative to derive the kinetics of the corresponding enzymatic reaction. We demonstrate here that the presence of conserved quantities involving source terms allow for more freedom in observing the system.  We now systematically examine how the conserved quantities given in Eqs. (<ref>) alter the reaction diagram.





  §.§.§ Enzyme Conservation

Let us suppose that we only impose enzyme conservation in the system. How does this alter the reaction diagram? In this case, we set e = E_0 - c, and the Michaelis-Menten system becomes

    de/dt   = -dc/dt
    ds/dt   = k_-1c - k_1(E_0 - c)s 
    dc/dt   = k_1(E_0 - c)s - (k_-1 + k_2)c 
    dp/dt   = k_2c.

Following our formalism for obtaining the corresponding reaction diagram, we obtain the diagram shown in Figure <ref>B. It now has two sources: E and P.  This means that to render the system observable, one must observe the dynamics of both E and P. Although the conserved quantity greatly simplifies mathematical analysis, the existence of this conserved quantity thus complicates the experimental setting. The issue arises because the imparted conserved quantity does not consist of the source from the full system, P.

We note that the reaction diagram would have a similar issue even if we took c = E_0 - e in the conserved quantity.



  §.§.§ Substrate Conservation

Now let us examine what happens when we impart substrate conservation. In this case, we set c = S_0 - s - p, rendering system (<ref>) as


    de/dt   = (k_-1 + k_2)(S_0 - s - p) - k_1es 
    ds/dt   = k_-1(S_0 - s - p) - k_1es 
    dc/dt   = -ds/dt-dp/dt
    dp/dt   = k_2(S_0 - s - p)

The corresponding reaction diagram is given in Figure <ref>D. There is again only one source: C. All other nodes have incoming edges including self loops. The implication here is that now we need only observe C to understand the system. Furthermore, if we had set s = S_0 - c - p instead, the only source in the resulting reaction diagram would be S, meaning we need to only observe S to render the system observable. The experimental implication is that one can observe the dynamics of any of S, C, or P to completely understand the system. Hence, if any of S, C, or P are measurable in a laboratory setting, the system can be understood. Thus, the conserved quantity consisting of the source node vastly expanded the number of state variables the we can measure to render the system completely observable.



  §.§.§ Enzyme and Substrate Conservation

What happens if we impose conservation of both enzyme and substrate? Does this simplify the system further? In this case, we set c = S_0 - s - p and e = E_0 -c = E_0 - S_0 + s + p. The system becomes

    de/dt   = ds/dt + dp/dt
    ds/dt   = k_-1(S_0 - s - p) - k_1(E_0 - S_0 + s + p)s 
    dc/dt   = -ds/dt - dp/dt
    dp/dt   = k_2(S_0 - s - p)

The corresponding reaction diagram is shown in Figure <ref>C. Again, the diagram depicts two source nodes (E and C), implying one must observe both C and E to understand the system. This is, of course, incorrect.

The above analysis brings to light an important point: one must not conclude that theoretical conserved quantities imply positive experimental ramifications. Indeed, if one only analyzed the model with both substrate and enzyme conservation, they would conclude that one must observe two state variables to understand the enzymatic system. Conserved quantities that do not include source node state variables do not inform the observability of the system. The conserved quantity s + c + p = S_0, on the other hand, yields a correct interpretation of observability. Namely, any one of the terms involved in the conserved quantity can be observed to understand the system.





§ CONCLUSIONS

We summarize the main contributions of this manuscript as follows. Most generally, we have proved a theorem conveying that observable dynamical systems with conserved quantities that involve source nodes in the corresponding directed graph representation of the system can be recast so that many more system outputs than originally thought could be observed to render the system observable. We used differential embeddings to prove this. In effect, we generalized the observability criteria provided by the graphical approach and the rank-based approach of differential embeddings.

Our approach has important implications for physical and biological sciences. Namely, we argue that systems with conserved quantities exhibit more flexibility in what must be observed for the full system to be understood. We demonstrate this with two concrete biological examples with conserved quantities:  the constant population SIR model and the classical Michaelis-Menten system for enzymatic reactions. For the former model, the original system necessitates observation of R(t) to render the system observable.  However, the conserved quantity allows any one of S, I, or R to be observed for the system to be observable.  Similarly, the classical Michalis-Menten system requires observation of the product, P(t), to render the system observable. The appropriate conserved quantity allows for product, substrate, or enzyme-substrate complex to be observed for the full system to be understood. Such flexibility can be the difference between success and failure in experimental settings.

For dynamical systems exhibiting multiple conserved quantities, our method identifies the `correct' submanifold of phase space to which dynamics should be contracted to obtain alternative observables that render the full system observable. Only conserved quantities that incorporate source nodes of the associated directed graph of the dynamical system can yield other outputs of the system that render the dynamical system observable.

Mathematically, we contribute to the rich mosaic of literature available on controllable and observable systems. Our method will be of interest because it expands upon and improves the popular methods given by the graphical approach and the rank-based differential embeddings approach.






 
ieeetr



























































































































 Paper Title: How We Decide the Future of the Olympics ?.


Multi-Site Class-Incremental Learning with Weighted Experts in Echocardiography
    Kit M. Bransby1,2, Woo-Jin Cho Kim1, Jorge Oliveira1, Alex Thorley1, Arian Beqiri1, Alberto Gomez1, Agisilaos Chartsias1
    August 5, 2024
============================================================================================================================

§ ABSTRACT 
The "Olympic Agenda 2020" stresses the urgency of measuring the impact of the Olympic Games on host cities in the face of declining bids. This paper presents the Hosting the Olympics Influence Evaluation Model (HOIEM) to assess these impacts and propose sustainable solutions.We select indicators (economic, socio-cultural, human, environmental and political) based on literature review and construct HOIEM. We use AHP and TOPSIS-EWM to determine the final weights for these indicators. We identify 45 potential host cities based on IOC requirements. For the Winter Olympics, secondary screening and the GM(1,1) model highlight Calgary, Canada as the top city. For the Summer Olympics, the SWOT analysis identifies Beijing, China. We propose to hold Spring, Summer, Autumn and Winter Olympics every 4 years, with fixed cities for Summer and Winter and bidding for Spring and Autumn.Finally, we conduct sensitivity analysis for HOIEM, demonstrating the independence and relevance of selected indicators through semi-quantitative visual analysis. 









§ INTRODUCTION

As the number of countries bidding for various Olympics sharply declines, the IOC gradually realizes the modern ethical dilemmas that the Olympics face today<cit.>. Looking back at the hosting processes of the Olympics in recent decades, the host countries that successfully held the Games not only did not obtain significant macroeconomic benefits as expected, but also incurred actual bidding costs that were several times higher than the declared costs and "Olympic legacy" that could not be handled. These facts indicate that significant reforms for the Olympics are urgently needed, as reflected in the "Olympic Agenda 2020+5"[1]. Therefore, establishing a reasonable evaluation model to guide the exploration of various new strategies and policies is crucial for improving the attractiveness and sustainability of the Olympics.





§ HOSTING THE OLYMPICS INFLUENCE EVALUATION MODEL (HOIEM) 





 §.§ Indicator selection

There are many factors that contribute to the impact of the Olympics on host cities. When selecting factors to establish an evaluation model for the impact of hosting the Olympics, more evaluation indicators are not necessarily better. Based on the original question and references, we have summarized 5 aspects of impact evaluation factors, including Economy, Human, Socio-culture, Politics, and Environment (as shown in Figure<ref> ), to help us build a comprehensive evaluation system and find corresponding data support.
   
From the numerous literature on the analysis of the impacts of the Olympics listed in Table 1, we extracted the following secondary indicators (where positive and negative factors are identified by the symbols (+) and (-), respectively). These indicators were selected and categorized based on the characteristics of data composition and the requirements of a comprehensive, universal, and minimally overlapping evaluation model.
Based on the review of the literature listed in Table 2, we developed a multidimensional evaluation model for the impacts of hosting the Olympics. The model can comprehensively reflect the various impacts of hosting the sporting event on the host city, providing quantitative basis for addressing the challenges faced by the Olympics. According to this model, we selected specific secondary indicators guided by the 5 primary indicators, in order to use concise and non-overlapping indicators to maximize the expression of the meaning of each primary indicator.



  §.§.§ Economic Indicators


As shown in Figure <ref>, we can see that in terms of economic factors, based on the secondary indicators obtained from reviewing the literature, we have explored the data composition characteristics of economic types and divided them into six secondary indicators (which we can refer to as A1 to A6). It can be determined that the secondary indicator system we constructed can comprehensively and systematically include the various economic impacts of hosting the Olympics on the host country. For example, not only does it discuss the macroeconomic growth brought by hosting the Olympics to the host city, but also considers the huge economic costs of hosting the Olympics. In addition, while maintaining breadth, we maximize the use of data characteristics to maintain the objectivity of the secondary mechanism of action on the primary indicators.



  §.§.§ Human Indicators


Similar to the process of determining the secondary indicators for economic factors, after determining the interaction mechanism for human factors' secondary indicators, we also conducted a secondary analysis and ultimately divided them into seven secondary indicators (denoted as B1 to B7).



  §.§.§ Sociocultural,Political and Environmental Indicators


To maintain the generality of the model, we have noticed that only discussing the impact of the Olympics on the economy and humans is far from sufficient. For example, the political terrorist incident at the 1972 Munich Olympics and the impact of COVID-19 on the 2020 Tokyo Olympics are both black swan events, but they can have a huge impact on the host. At the same time, the promoting role of the Olympic spirit in social culture seems to be significant. Finally, we continue to pay attention to reports on irreversible damage to the ecological environment and the living environment of local residents before and after the Olympics. Based on this, we hope to extract some secondary indicators that can reflect these aspects, which is conducive to establishing a more scientific evaluation system. They are respectively denoted as C1 to C7,D1 to D5 and E1 to E5.






 §.§ Establishment and solution of HOIEM



  §.§.§ Determination of Weight

The Analytic Hierarchy Process (AHP), developed by Saaty in 1980<cit.>, is a comprehensive evaluation method that combines systems analysis and decision-making. It helps us to take into account both qualitative and quantitative factors when making decisions and then arrive at a comprehensive result.Firstly, the relationships between various factors are analyzed and a systematic hierarchical structure is established.Secondly, pairwise comparisons of the elements within the same level are made, and judgment matrices are constructed.

    b_ii=1,b_ij=b_ji^-1>0

In equation (1), b_ii and b_ij represent different evaluation indicators.
Based on the judgment matrix, the relative weights of the compared elements to the criterion are calculated, and consistency check is performed.

    CI=λ_max-n/n-1



    CR=CI/RI

The equation above shows that λ_max is the maximum eigenvalue of the n-order judgment matrix,  CI is the consistency index, and RI is the corresponding average random consistency index. If the calculated consistency ratio CR<0.1, the consistency check of the judgment matrix is considered to pass, otherwise it needs to be revised. 
Finally, the corresponding weight values V_j are calculated based on the judgment matrix and solved using the analytic hierarchy process.
TOPSIS (C.L. Hwang and K. Yoon, 1981)<cit.>, which stands for Technique for Order Preference by Similarity to an Ideal Solution, is a comprehensive evaluation method that fully reflects the original information. Modified by EWM, it achieves objective weighting based on the variability of each indicator. First, each data in the original matrix is normalized in a positive direction.

    M=max{a-min{x_i},max{x_i}-b}


    x̃_i =
    
    1 - a - x_i/M,     x_i < a 
    
    1,     a ≤ x_i ≤ b 
    
    1 - x_i - b/M,     x_i > b


    z_ij=x_ij/√(∑_i=1^nx_ij^2)

Based on the compositional characteristics of the collected data, we select equation (6) as the standardization method. Then, we solve the probabilities used in the relative entropy calculation<cit.>.

    p_ij=z_ij/∑_i=1^nz_ij

In formula (7), p_ij represents the relative entropy probability of the j-th sample under the i -th index. Finally, the information entropy of the index is calculated to determine the entropy weight.

    e_j=-1/ln n∑_i=1^np_ijln(p_ij)


    H_j=1-e_j/n-∑_je_j

The equation above shows that e_j represents the information entropy contained in the j-th indicator, and H_j represents the entropy weight carried by the j-th indicator.
The combination weighting model based on objective modification of subjectivity can well compensate for the "superposition effect" and "multiplication effect" caused by traditional additive and multiplicative combination weighting methods. The combined weights can reflect the expert's intention to a certain extent while embodying the information carried by the sample data itself. First, calculate the importance ratio between adjacent indicators. 

    s̆_j=√(1/m∑_i=1^m(x_ij-x_ijH_j+V_j/H_jV_j)^2)


    r_k = {[ min{2,s̆_k-1/s̆_k}, s̆_k-1≥s̆_k;                1, s̆_k-1 < s̆_k ].

In formula (11),  s_j represents the normalized standard deviation of the attribute j , and its importance ratio is denoted as r_k .

    W_m=(1+∑_k=2^m∏_j=k^mr_j)^-1
    W_j-1=r_jW_j

Finally, the comprehensive weight of the indicator to the criterion layer can be determined according to the above formula.



  §.§.§ Solution of HOIEM

Based on the previous work, we obtained the subjective and objective weights of the corresponding indicators through the AHP and EWM-TOPSIS methods, and obtained the comprehensive weights through the model method based on objective correction of subjectivity. The weight values of different indicators calculated are shown in Figure <ref>.


we determined the weights u for the five primary indicators and independent subjective, objective, and comprehensive weights for each secondary indicator under each primary indicator. The total weight is calculated as follows.

    Ω_ij=U_iu_j

Calculation of the weights of each secondary index is shown in Table 3.
We used the idea in principal component analysis that the sum of the top 10 comprehensive weights was 0.735. Therefore, in evaluating the generality and simplicity of the model, we selected these 10 characteristic factors  ξ_j for further analysis. We found that they came from different categories of primary indicators, which is also a manifestation of the preservation of generality.After secondary weighting normalization of the selected characteristic factors, we obtained the adjusted weights  γ_j , as shown in Table <ref>.



Finally, we obtain the evaluation formula for the impact assessment model of hosting the Olympic Games as shown in equation (14).

    χ=∑_j=1^10γ_jξ_j

Formula (14) defines the evaluation function χ for the impact assessment of hosting the Olympics. It can be seen that χ is a positive function defined in the background of considering important factors, including:
·The feature indicator group has generality and comes from different five primary indicators.
·The feature indicators maintain independent relationships with each other without overlapping.
·The selection of feature indicators combines subjective and objective weights with high credibility.
·The function describes well that the cost issue in the economic aspect is the most important factor that the Olympics brings impacts on.
·It also takes into account unpredictable factors that may affect the hosting of the Olympics.
·The evaluation function confirms that the economic and sociocultural aspects are the main contributions to the impact assessment model of the Olympics.




§ APPLICATION OF THE HOIEM


In order to propose innovative decisions to address the difficulties faced by cities in bidding for the Olympic Games.Consider using the GM (1,1) model, combined with the HOIEM model constructed in the first question, to obtain the potential changes in characteristic factors of candidate cities before and after hosting the Olympics, thereby determining their adaptability scores for hosting the Olympics. As shown in Figure <ref>, based on the gray prediction system, we need to combine the characteristics of the secondary indicators extracted for different types in the HOIEM model to obtain the corresponding Olympic host city fitness score, which determines the potential for a city to host the Olympics. Finally, guided by the HOIEM model, we use the GM (1,1) model to predict the hosting schedule.



 §.§ Screening of Potential Host Cities

According to the standardization recommendations of the International Olympic Committee, we used the total GDP of a country and its level of sports development as the first screening criteria, and selected 45 potential candidate cities from the countries that ranked high in the comprehensive results. The results are shown in Table 3. Based on the HOIEM model, we believe that only potential candidate cities meet the most basic conditions for hosting the Olympic Games, so the actual host city must be a non-empty proper subset of the potential candidate city group.




 §.§ Determine the host city of the Winter Olympics

GM(1,1) is a single-variable grey prediction model with a first-order difference equation<cit.>. It has advantages such as requiring a small sample size and having a simple model establishment and solution process. The specific implementation process is as follows:Given a time series X^(0)={X^(0)(1),X^(0)(2),...,X^(0)(n)} with n observations, a new sequence X^(1)={X^(1)(1),X^(1)(2),...,X^(1)(n)} is generated by accumulating the sum.The corresponding differential equation of GM(1,1) can be expressed as the following formula:

    dX^(1)/dt+α X^(1)=μ

In the GM(1,1) model, α is called the developing grey degree and μ is called the internal control grey degree. The general solution to the differential equation is:

    X^(1)(k+1)=[X^(0)(1)-μ/α]e^-α k+μ/α

Based on the preliminary screening of potential host cities, we further identify the most suitable city to host the Winter Olympics. Before predicting factors such as average temperature and snowfall for candidate host cities, we first use common knowledge from geography and climate to eliminate 36 cities that are obviously unsuitable for hosting the Winter Olympics. The various characteristics listed in the table all indicate that they are not suitable to be included in the list of candidate host cities for the Winter Olympics. After determining 9 potential candidate host cities, we used the GM(1,1) model to pre-dict their future (until 2050) average temperature and snowfall in February to determine if they meet the International Olympic Committee's requirements for hosting the Winter Olympics, which include an average temperature below 0℃ in February, an ideal temperature range of -17℃ to -10℃, and a minimum snowfall of 30cm in February. After this work was completed, we eliminated two-thirds of the potential host cities. Finally, the remaining 3 cities seem to have no difference in terms of the rigid requirements. In order to better distinguish them, we can only use the hosting suitability score function constructed earlier under the guidance of the HOIEM model.
We will input the predicted results into the HOIEM evaluation model and solve the corresponding scoring function. The results are as follows: Moscow S1=1.381(Sbase=0.6,Sevaluate=0.781), Pyeongchang S2=1.602(Sbase=0.8,Sevaluate=0.802), and Calgary S3=1.649(Sbase=0.8,Sevaluate=0.849). It can be seen that Calgary, Canada is the most ideal location for hosting the Winter Olympics.



 §.§ Determine the host city of the Summer Olympics



  §.§.§ Secondary screening of potential host cities

For the Summer Olympics, we can naturally follow the layer-by-layer progressive screening method for the host cities of the Winter Olympics. The difference is that the International Olympic Committee's requirements for the host city of the Summer Olympics do not seem to have quite rigid conditions like the Winter Olympics. For this reason, we slightly changed the screening process this time, as shown in Figure <ref>.


The measurement standard of sports career in the above picture comes from the medal points ranking of the previous Summer Olympics from 1896 to 2020 (including 5 points for gold medals, 1 point for silver medals, and 0.5 points for bronze medals). The process of solving the fitness score of the host city in HOIEM is similar to that of the Winter Olympics and will not be repeated here.



  §.§.§ SWOT analysis

Finally, we choose the SWOT analysis method. It is an analytical method used to identify the strengths, weaknesses, opportunities and threats of the decision-making object. We performed SWOT analysis on the four cities obtained earlier, and the results are shown in the Figure <ref>.


Based on the SWOT analysis and the fitness score results in HOIEM, we believe that Beijing is the most suitable city for hosting the Summer Olympics. As mentioned in the conclusion of the case study on the Beijing 2008 Olympic Games in (4.4.2) above. We have reasons to believe that choosing Beijing as the fixed venue for the Olympic Games can bring a grand sports event to people all over the world while controlling the cost of hosting the Olympic Games as much as possible.
In this section of the analysis on the Winter and Summer Olympic Games, we have obtained the most suitable cities for hosting them, which are Calgary and Beijing respectively. Next, we discuss how the Olympic Games are held, while quantitatively analyzing its merits using our established HOIEM.



 §.§ Determine the manner of holding the future Olympics

On the basis of combining the topic description and consulting relevant literature, the following four innovative solutions are perfectly summarized.Plan A believes that on the basis of not changing the holding time, the Summer and Winter Olympics should have corresponding permanent venues. Plan B indicates that 4 Olympics in spring, summer, autumn and winter should be held in each cycle (four years), and they should all have corresponding fixed venues. Plan C does not support having a fixed venue, and only proposes that four Olympic Games should be held in each cycle. Plan D is a com-promise between Plans B and C. It will be held four times in a cycle and the Summer and Winter Olympics will have fixed venues.
Next, we discuss the above four possible schemes quantitatively and semi-quantitatively based on the model technology established above. And compare it with the original way of hosting the Olympics, draw their respective advantages and disad-vantages, and determine the best plan as much as possible.



  §.§.§ Quantitative analysis based on HOIEM

In order to intuitively and clearly show the effects of the four schemes on the city compared with the original holding method, we consider using the characteristic index group of HOIEM for analysis. Evaluate its impact on each feature index separately, and finally get the total impact function and compare its size. To this end, we use the comparison method used in the establishment of the AHP model to define an objective function for evaluating the impact, as shown in the Table 4:











With this definition, we can evaluate the improvements of the four schemes for the ten characteristic group marks used in HOIEM combined with the original scheme. Obviously, the greater the degree, the better. The evaluation results and their visualizations are shown in  Figure <ref>.

we can see that Scheme D seems to win in terms of integration. It is worth mentioning that the economic, ecological, environmental and political aspects of the Plan A, B, C, and D all have positive changes compared with the original scheme, which shows that the direction of this improvement is in line with science. Perhaps on the basis of Plan D, we can also consider a better plan. However, due to space limitations, we will not make further program setting and evaluation. At the end, we will conduct a SWOT analysis of the corresponding scheme in order to comprehensively analyze the characteristics of all aspects of the scheme.



§ SENSITIVITY ANALYSIS OF HOIEM



 §.§ Sensitivity Analysis of obsolete indicators


In the evaluation model, sensitivity analysis is usually used to analyze the fluctuation of the selection of different characteristic factors for the evaluation model. In the previous work, we identified 10 characteristic factors, and in the process of sensitivity analysis, we considered randomly selecting 5 out of 20 unselected factors among the 30 candidate factors as False feature factors, the random 5 of the feature factors are replaced equally, the result is shown in Figure <ref>.



 §.§ Semi-quantitative-Visual Analysis based on RSM

Box-Behnken Response Surface Method (BBD-RSM)<cit.> is a commonly used tool in optimization visualization analysis, and can also be used to evaluate the stability analysis of the model. The basic principle is to use the weight of a certain factor as the system response value and the total evaluation function as the global variable, and use an intuitive graphic language to display its interaction relationship, so as to judge the rationality of the selected factors under semi-quantitative conditions.


Analysis of Figure <ref> shows that the weight changes of  ξ_1 and ξ_10 will cause obvious response to the variation degree of the evaluation function. It can be seen that the response extreme value brought by ξ_1 is about 69%, and the influence extreme value of ξ_10 is about 50%. This result supports the representative of feature index selection. At the same time, the response extreme value brought by ξ_1 and ξ_10 together is about 25%, which is lower than the two independent response extreme values and is the global minimum, indicating that these two characteristic factors are independent.Similar to the above analysis, as shown in Figure 20, we also tested the responsiveness of  ξ_1 and ξ_5to the degree of variation of the evaluation function.  




§ CONCLUSION

We adopted a combined weighting model based on AHP-EWM, incorporating both objective correction and subjective weight determination, and selected more comprehensive indicators to establish the HOIEM (Host City Olympic Impact Evaluation Model). In the sensitivity analysis of HOIEM, we utilized a combination of quantitative and semi-quantitative-visualization methods; the two methods complement each other, fully illustrating the stability and feasibility of the HOIEM model. To ensure that the article is quite readable, we have adopted a large number of visualization methods for the solution results and analysis of the model. Our HOIEM is based on five key aspects: economy, socio-cultural, people, politics, and ecological environment, comprehensively including all aspects of the influence of the Olympic Games on the host city, thus demonstrating its good practicality and promotion value.




IEEEtran





 Paper Title: Two Completely Parameter-Free Alternating Gradient Projection Algorithms for Nonconvex-(strongly) Concave Minimax Problems.









[1]This work is supported by National Natural Science Foundation of China under the grants 12071279. 

[2]Corresponding author.

My editor

Two Completely Parameter-Free Alternating Gradient Projection Algorithms for Nonconvex-(strongly) Concave Minimax Problems[1]
    Junnan Yang yangjunnan@shu.edu.cn 

	Department of Mathematics

	Shanghai University

	Shanghai 200444,People's Republic of China
	Huiling Zhang zhanghl1209@shu.edu.cn 

	Department of Mathematics

	Shanghai University

	Shanghai 200444,People's Republic of China
    Zi Xu[2] xuzi@shu.edu.cn 

	Department of Mathematics

	Shanghai University

	Shanghai 200444, People's Republic of China

	and

	Newtouch Center for Mathematics of Shanghai University

	Shanghai University

	Shanghai 200444, People's Republic of China
    August 5, 2024
============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Due to their importance in various emerging applications, efficient algorithms for solving minimax problems have recently received increasing attention. However, many existing algorithms require prior knowledge of the problem parameters in order to achieve optimal iteration complexity. In this paper, we propose a completely parameter-free alternating gradient projection (PF-AGP) algorithm to solve the smooth nonconvex-(strongly) concave minimax problems using a backtracking strategy, which does not require prior knowledge of parameters such as the Lipschtiz constant L or the strongly concave constant μ. The PF-AGP algorithm utilizes a parameter-free gradient projection step to alternately update the outer and inner variables in each iteration. We show that the total number of  gradient calls of the PF-AGP algorithm to obtain an ε-stationary point for nonconvex-strongly concave minimax problems is upper bounded by 𝒪( Lκ^3ε^-2) where κ is the condition number, while the total number of gradient calls to obtain an ε-stationary point for nonconvex-concave minimax problems is upper bounded by 𝒪( L^4ε^-4). As far as we know, this is the first completely parameter-free algorithm for solving nonconvex-strongly concave minimax problems, and it is also the completely parameter-free algorithm which achieves the best iteration complexity in single loop method for solving nonconvex-concave minimax problems. Numerical results validate the efficiency of the proposed PF-AGP algorithm.



  minimax optimization problem; backtracking strategies; parameter-free alternating gradient projection algorithm; iteration complexity; machine learning




§ INTRODUCTION



We consider the following minimax optimization problem:

    min_x ∈𝒳max_y ∈𝒴 f(x, y), P

where 𝒳⊆ℝ^n and 𝒴⊆ℝ^m are nonempty closed and bounded convex sets, and f: 𝒳×𝒴→ℝ is a smooth function. The minimax problem is a critical class of optimization problems, known for its extensive research history and broad applications in areas such as machine learning, signal processing, and various other fields in recent years. Many practical problems can be formulated as in (<ref>), including machine learning, such as generative adversarial networks (GANs)  <cit.>, reinforcement learning <cit.>, power control and transceiver design in signal processing <cit.>, distributed nonconvex optimization <cit.>, robust optimization <cit.>, statistical learning <cit.> and several other applications.

There are many research results on convex-concave minimax problems, i.e., f(x, y) is convex with respect to x and concave with respect to y. Nemirovski <cit.> introduced the mirror-prox  method, and Nestrov <cit.> proposed a dual extrapolation algorithm, both of which achieve an optimal iteration complexity of 𝒪(ε^-1) for solving convex-concave minimax problems. For more details, see <cit.>  and references therein.

Recently, various algorithms have been proposed for nonconvex-(strongly) concave minimax problems. For nonconvex-strongly concave minimax problems, several recent works <cit.> have proposed various algorithms that achieve a gradient complexity of 𝒪̃(κ_y^2 ε^-2) for stationary points of Φ(·) = max_y ∈𝒴 f(·, y) (when 𝒳 = ℝ^n), or for stationary points of f, where κ_y is the condition number for f(x, ·). <cit.> proposed an accelerated algorithm that improves the gradient complexity bound to 𝒪̃(√(κ_y)ε^-2).  Furthermore, Zhang et al. <cit.> proposed a generic acceleration framework which can improve the iteration complexity to 𝒪(√(κ_y)ε ^-2).  


For general nonconvex-concave minimax problems, there are two types of algorithms, i.e., multi-loop algorithms and single-loop algorithms. Various multi-loop algorithms have been proposed in <cit.>. The best known iteration complexity of multi-loop algorithms for solving nonconvex-concave minimax problems is 𝒪̃( ε ^-2.5), which was achieved by <cit.>.  For solving nonconvex-linear minimax problems, Pan et al. <cit.> proposed a new alternating gradient projection algorithm with the iteration complexity of 𝒪( ε ^-3). 
Various single-loop algorithms have also been proposed,  e.g., the gradient descent-ascent (GDA) algorithm <cit.>, the hybrid block successive approximation (HiBSA) algorithm <cit.>, the unified single-loop alternating gradient projection (AGP) algorithm <cit.>, and the smoothed GDA algorithm <cit.>. Both the AGP algorithm and the smoothed GDA algorithm achieve the best known iteration complexity, i.e., 𝒪( ε ^-4), among single-loop algorithms for solving nonconvex-concave minimax problems.



It should be noted that for most of the existing results mentioned above, achieving optimal complexity of the algorithm requires the assumption of knowing precise information about some parameters of the problem, such as the Lipschitz constant, or the strongly concave constant μ, etc. Accurately estimating these parameters is often challenging, and conservative estimates can significantly impact algorithm performance <cit.>. Therefore, designing parameter-free algorithms with complexity guarantees without relying on inputs of these parameters has attracted considerable attention recently. 








For nonconvex-strongly concave minimax problems, there are two main parameter-free algorithms: one based on AdaGrad stepsizes and the other based on backtracking. In the framework of AdaGrad stepsizes, Yang et al. <cit.> proposed NeAda, which can achieve convergence rates of 𝒪̃(ε^-2) and 𝒪̃(ε^-4) in deterministic and stochastic settings, respectively.  Li et al. <cit.> proposed TiAda, achieving convergence rates of 𝒪(ε^-2) and 𝒪(ε^-4) in deterministic and stochastic settings, respectively. In the framework of backtracking,  Xu et al. <cit.> proposed a stochastic gradient descent ascent (GDA)  method with backtracking (SGDA-B), achieving an upper bound on the total number of gradient calls of 𝒪(Lκ^2log(κ) ε^-2) with 𝒪(L κ^2 ε^-2) gradient calls required per backtracking step. Meanwhile, Zhang et al. <cit.> proposed a proximal alternating gradient descent ascent method (AGDA+), achieving an upper bound on the total number of gradient calls of  𝒪(Lκ^4ε^-2) with 𝒪(1) gradient calls required per backtracking step. Both methods do not require knowledge of the specific value of Lipschtiz constant L, but still require information about the strongly concave constant μ in the function. In the framework of backtracking, Xu et al. <cit.> proposed a stochastic GDA method with backtracking (SGDA-B) for nonconvex-concave minimax problems, achieving an upper bound on the total number of gradient calls of  𝒪̃(L^3ε^-4) with 𝒪(L^3ε^-4) gradient calls required per backtracking step.





 §.§ Contributions


In this paper, based on the framework of backtracking, we propose two completely parameter-free alternating gradient projection algorithms, i.e., the PF-AGP-NSC algorithm and the PF-AGP-NC algorithm, for solving nonconvex-(strongly) concave minimax problems respectively, which does not require prior knowledge of parameters such as the Lipschtiz constant L or the strongly concave constant μ.  

Moreover, we show that the total number of  gradient calls of the PF-AGP-NSC algorithm to obtain an ε-stationary point for nonconvex-strongly concave minimax problems is upper bounded by 𝒪( Lκ^3ε^-2) where κ is the condition number,  while the total number of gradient calls of the PF-AGP-NC algorithm to obtain an ε-stationary point for nonconvex-concave minimax problems is upper bounded by 𝒪( L^4ε^-4). As far as we know, the PF-AGP-NSC algorithm is the first completely parameter-free algorithm for solving nonconvex-strongly concave minimax problems, and the PF-AGP-NC algorithm is the first completely parameter-free algorithm which achieves the best iteration complexity in single loop algorithms for solving nonconvex-concave minimax problems.







 §.§ Organization

In Sect. <ref>, we propose a parameter-free alternating gradient projection (PF-AGP) algorithm for nonconvex-(strongly) concave minimax problems, and we then analyze the corresponding gradient complexity for two different settings.  We report some numerical results in Sect. <ref> and make some concluding remarks in the last section.


Notation.
For vectors, we use · to denote the l_2-norm. For a function f(x, y): ℝ^n ×ℝ^m →ℝ, we use ∇_x f(x, y) (or ∇_y f(x, y)) to denote the partial gradient of f with respect to the first variable (or the second variable) at point (x, y). Let 𝒫_𝒳 and 𝒫_𝒴 denote projections onto the sets 𝒳 and 𝒴. Finally, we use the notation 𝒪(·) to hide only absolute constants which do not depend on any problem parameter, and 𝒪̃(·) notation to hide only absolute constants and log factors.
A continuously differentiable function f(·) is called θ-strongly convex if there exists a constant θ > 0 such that for any x, y ∈𝒳,

    f(y) ≥ f(x) + ⟨∇ f(x), y - x ⟩ + θ/2y - x^2.

If -f satisfies (<ref>), f(·) is called θ-strongly concave. 
Denote the sign function as (x), i.e.,

    (x) =
     
    1    if  x > 0, 
    
    -1    if  x ≤ 0.







§ TWO COMPLETELY PARAMETER-FREE ALTERNATING GRADIENT PROJECTION ALGORITHMS

In this section, we propose two completely parameter-free alternating gradient projection (PF-AGP) algorithms for solving the nonconvex-strongly concave and the nonconvex-concave minimax problems (<ref>) respectively. Both algorithms do not require prior knowledge of Lipschitz constants, and the algorithm under nonconvex-strong concave settings also does not require prior knowledge of strong concave constants. The two proposed algorithms are based on the AGP algorithm <cit.>, which uses the gradient of a regularized version of the original function, i.e.,

    f_k(x,y) = f(x,y) - c_k/2y_k^2,

where c_k ≥ 0 is a regularization parameters. At the k-th iteration, AGP algorithm consists of the following two gradient projection steps for updating both x and y:

    x_k+1   = 𝒫_𝒳( x_k - 1β_k∇ _xf ( x_k,y_k ) ),
    
    	y_k+1   = 𝒫_𝒴( y_k+1γ_k∇ _y f( x_k,y_k ) - 1/γ_k c_k y_k ),

where  𝒫_𝒳 and 𝒫_𝒴 is the projection operator onto 𝒳 and  𝒴, respectively, and  β_k > 0, γ_k >0 are stepsize parameters. 


Although the AGP algorithm achieves the optimal iteration complexity among single loop algorithms for nonconvex-(strongly) concave minimax problems, it requires the knowledge of the Lipschtiz constant L and the strongly concave coefficient μ, or at least knowledge of the Lipschtiz constant L  to calculate the stepsizes β_k and γ_k under the nonconvex-strongly concave setting and the nonconvex-concave setting, respectively, which limits its applicability. To eliminate the need for the prior knowledge of the Lipschtiz constant L or the strongly concave coefficient μ, we propose two parameter-free AGP algorithms by employing backtracking strategies for solving the nonconvex-strongly concave and the nonconvex-concave minimax problems, respectively. Similar backtracking strategies have been used in
<cit.>. 




 §.§ Nonconvex-strongly Concave Setting

We first propose a completely parameter-free AGP algorithm for solving nonconvex-strongly concave minimax problems.
More specifically, in order to estimate the Lipschitz constants for ∇_x f(·,y), ∇_y f(·,y), ∇_y f(x,·) denoted as L_11, L_12 and L_22, respectively, and the strongly concave constant μ, at each iteration of the proposed algorithm, we aim to find a tuple (l_11^k,i, l_12^k,i, l_22^k,i, μ_k,i) by backtracking such that the following conditions for (x_k,i, y_k,i) are satisfied:

    C_1^k,i   = f(x_k,i, y_k) - f(x_k, y_k) - ⟨∇_x f(x_k, y_k), x_k,i - x_k ⟩ - l^k,i_11/2x_k,i - x_k^2 ≤ 0, C1
    
    	C_2^k,i   = ∇_y f(x_k,i, y_k) - ∇_y f(x_k, y_k) - l_12^k,ix_k,i - x_k≤ 0,C2
    
    	C_3^k,i   = ∇_y f(x_k,i, y_k,i) - ∇_y f(x_k,i, y_k) - l_22^k,iy_k,i - y_k≤ 0,C3
    
    	C_4^k,i   = ⟨∇_y f(x_k,i, y_k,i) - ∇_y f(x_k,i, y_k), y_k,i - y_k ⟩ + μ_k,iy_k,i - y_k^2 ≤ 0.C4

Otherwise, we enlarge l_11^k,i, l_12^k,i, l_22^k,i by a factor of 2 or reduce μ_k,i by half according to the sign of C_1^k,i, C_2^k,i, C_3^k,i and C_4^k,i, respectively. We then update β_k and γ_k by the new estimates, i.e., l_11^k,i, l_12^k,i, l_22^k,i, μ_k,i.
The proposed algorithm for solving nonconvex-strongly concave problems, denoted as PF-AGP-NSC,  is formally presented in Algorithm <ref>.



It is worth noting that the parameter-free NeAda <cit.> and TiAda <cit.> algorithms with adaptive step size are only used to solve the minimax problems when x is unconstrained, while the  proposed PF-AGP-NSC algorithm can solve the constrained case. Compared with the SGDA-B <cit.> algorithm and AGDA+ <cit.> algorithm using the backtracking framework under the nonconvex-strongly concave setting, AGDA+ and SGDA-B still require knowledge of the strongly concave constant mu, while the proposed PF-AGP-NSC algorithm is a completely parameter-free algorithm that does not require any prior knowledge of the parameters.





  §.§.§ Complexity analysis

In this subsection, we analyze the iteration complexity of Algorithm <ref> for solving nonconvex-strongly concave minimax optimization problems, i.e., f(x, y) is nonconvex with respect to x for any fixed y ∈𝒴, and μ-strongly concave with respect to y for any given x ∈𝒳. 

Before we prove the iteration complexity of the PF-AGP-NSC algorithm for solving (<ref>), we need to make the following assumption about the smoothness of f(x, y).

f(x, y) has Lipschitz continuous gradients, i.e., there exist positive scalars L_11, L_12, L_22 such that for any x, x̃∈𝒳, y, ỹ∈𝒴,

    ∇_x f(x, y) - ∇_x f(x̃, y)    ≤ L_11 x - x̃, 
    ∇_y f(x, y) - ∇_y f(x̃, y)    ≤ L_12 x - x̃,
    ∇_y f(x, ỹ) - ∇_y f(x, y)    ≤ L_22ỹ - y .
 

We denote  L = max{L_11, L_12, L_22}. To analyze the convergence of Algorithm <ref>, we define the stationarity gap as the termination criterion as follows.

At each iteration of Algorithm <ref>, the stationarity gap for problem (<ref>) with respect to f(x, y) is defined as:

    ∇ G(x_k, y_k) = [ β_k (x_k - 𝒫_𝒳(x_k - 1/β_k∇_x f(x_k, y_k))); γ_k (y_k - 𝒫_𝒴(y_k + 1/γ_k∇_y f(x_k, y_k))) ].

We denote ∇G_k= ∇G(x_k, y_k),
(∇ G_k)_x= β_k( x_k-𝒫_𝒳( x_k-1/β_k∇ _xf( x_k,y_k))),
and (∇ G_k)_y=γ_k( y_k-𝒫_𝒴( y_k+ 1/γ_k∇ _yf( x_k,y_k )) ).








At each iteration of Algorithm <ref>, the stationarity gap for problem (<ref>) with respect to f_k(x, y) is defined as:

    ∇G̃(x_k, y_k) = 
    [  β_k(x_k - P_x(x_k - 1/β_k∇_x f_k(x_k, y_k))); γ_k (y_k - P_y(y_k + 1/γ_k∇_y f_k(x_k, y_k))) ].
 

Under Assumption <ref>, we first establish the following lemma, which provides bounds on the changes in the function value when x_k is updated at each iteration of Algorithm <ref>. This proof is similar to that of Lemma 2.1 in <cit.>, for the sake of completeness, we give its proof.



 Suppose that Assumption <ref> holds. 
Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>, then we have

    f(x_k+1, y_k) - f(x_k, y_k) ≤ - (β_k - l^k_11/2) x_k+1 - x_k^2 .


	
	

   



  By the optimality condition for (<ref>), we have
		
    ⟨∇ _xf( x_k,y_k ) +β_k ( x_k+1-x_k ) ,x_k-x_k+1⟩≥ 0.

		The backtracking strategy (<ref>) implies that
		
    f( x_k+1,y_k ) -f( x_k,y_k )≤⟨∇ _xf( x_k,y_k ) ,x_k+1-x_k ⟩  +l^k_112 x_k+1-x_k  ^2.

		By adding (<ref>) and (<ref>), we obtain
		
    f( x_k+1,y_k ) -f( x_k,y_k )    ≤ -( β_k -l^k_112)  x_k+1-x_k  ^2,

which completes the proof. 











Next, we provide an estimate of upper bound of the difference between f(x_k+1, y_k+1) and f(x_k, y_k). 

First, we need to make the following assumption on the parameter γ_k.

{γ_k} is a nonnegative monotonically increasing sequence. 


Suppose that Assumptions <ref> and <ref> hold. 
Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>, then we have

    f(x_k+1, y_k+1) - f(x_k, y_k) 
         ≤ -(β_k - l_11^k/2 - l_12^k/2γ_k-1)x_k+1 - x_k^2+γ_k-1y_k+1 - y_k^2
            -(μ_k-1 - γ_k-1/2 - (l_22^k-1)^2/2γ_k-1)y_k - y_k-1^2  .
 
 


The optimality condition for y_k in (<ref>) implies that





    ⟨∇_y f(x_k, y_k-1) - γ_k-1 (y_k - y_k-1), y_k+1 - y_k ⟩≤ 0.

By the concavity of f(x,y) with respect to y, and combining (<ref>), we have 

    f(x_k+1, y_k+1) - f(x_k+1, y_k) 
    ≤   ⟨∇_y f(x_k+1, y_k), y_k+1 - y_k ⟩
    ≤   ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k-1), y_k+1 - y_k ⟩+ γ_k-1⟨ y_k-y_k-1, y_k+1 - y_k ⟩.

Denoting v_k+1 = (y_k+1 - y_k) - (y_k - y_k-1), we can write the first inner product term in the r.h.s. of (<ref>) as

    ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k-1), y_k+1 - y_k ⟩
    
    		=   ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k), y_k+1 - y_k ⟩ + ⟨∇_y f(x_k, y_k) - ∇_y f(x_k, y_k-1), v_k+1⟩
        + ⟨∇_y f(x_k, y_k) - ∇_y f(x_k, y_k-1), y_k - y_k-1⟩.

Next, we estimate the three terms on the right-hand side of (<ref>) respectively. By the backtracking strategy (<ref>), (<ref>) and the Cauchy-Schwarz inequality, we have

    ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k), y_k+1 - y_k ⟩≤l_12^k/2γ_kx_k+1 - x_k^2 + γ_k/2y_k+1 - y_k^2

and

    ⟨∇_y f(x_k, y_k) - ∇_y f(x_k, y_k-1), v_k+1⟩≤(l_22^k-1)^2/2 γ_k-1y_k - y_k-1^2 + γ_k-1/2v_k+1^2 .

By the backtracking strategy (<ref>) of μ_k-1, we obtain

    ⟨∇_y f(x_k, y_k) - ∇_y f(x_k, y_k-1), y_k - y_k-1⟩≤ -μ_k-1y_k - y_k-1^2.

Moreover, it can be easily checked that

    ⟨ y_k - y_k-1, y_k+1 - y_k ⟩ = 1/2y_k - y_k-1^2 + 1/2y_k+1 - y_k^2 - 1/2v_k+1^2.

Plugging (<ref>)-(<ref>) into (<ref>) and rearranging the terms, we conclude that

    f(x_k+1, y_k+1) - f(x_k+1, y_k)    ≤l_12^k/2γ_kx_k+1 - x_k^2 - (μ_k-1 - γ_k-1/2 - (l_22^k-1)^2/2γ_k-1) y_k - y_k-1^2 
          + (  γ_k/2 + γ_k-1/2) y_k+1 - y_k^2.

By Lemma <ref>, we have

    f(x_k+1, y_k) - f(x_k, y_k) ≤ - (β_k - l_11^k/2) x_k+1 - x_k^2.

The proof is then completed by combining (<ref>) with (<ref>).



We now establish an important recursion for Algorithm <ref>.

Suppose that Assumptions <ref> and <ref> hold. 
Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>, and denoting

    f_k+1    = f(x_k+1, y_k+1),    S_k+1 = 2γ^2_k /μ_ky_k+1 - y_k^2, 
    
      F_k+1    = f_k+1 + S_k+1 - ( μ_k + 7γ_k/2 -  (l^k_22)^2/2γ_k - 2(l^k_22)^2/μ_k) y_k+1 - y_k^2-2γ_k^2/μ_k𝒟^2_y,

then we have

    F_k+1 - F_k
    	≤    -( 2μ_k γ_k - (l_22^k)^2/2γ_k + μ_k γ_k - 4(l_22^k)^2/2μ_k) y_k+1-y_k^2
          - ( β_k - l_11^k/2 - l_12^k/2γ_k-1- 2(l_12^k)^2γ_k-1/μ_k-1μ_k) x_k+1 - x_k^2,

where 𝒟_y = max{y_1 - y_2| y_1, y_2 ∈𝒴}. 



By the optimality condition for y_k in (<ref>) we obtain

    ⟨∇_y f(x_k+1, y_k) - γ_k (y_k+1 - y_k), y_k - y_k+1⟩≤ 0.

Combining (<ref>) and (<ref>), we have

    ⟨∇_y f(x_k, y_k-1) - ∇_y f(x_k+1, y_k)+γ_k( y_k+1 - y_k)-γ_k-1( y_k - y_k-1), y_k+1 - y_k⟩≤ 0,

which implies that

    γ_k y_k+1 - y_k^2 - γ_k-1⟨ y_k-y_k-1, y_k+1 - y_k ⟩≤⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k-1), y_k+1 - y_k ⟩ .

Denoting v_k+1 = (y_k+1-y_k)-(y_k-y_k-1), and using the identity 1/2⟨ v_k+1, y_k+1 - y_k⟩ = 1/2y_k+1 - y_k^2 + 1/2v_k+1^2 - 1/2y_k - y_k-1^2, we conclude that

    γ_k y_k+1 - y_k^2 - γ_k-1( 1/2y_k+1 - y_k^2 + 1/2y_k - y_k-1^2 - 1/2v_k+1^2 )
         ≤⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k-1), y_k+1 - y_k ⟩.

By rearranging the terms of the above inequality, we then have

    (γ_k - γ_k-1/2) y_k+1 - y_k^2 + γ_k-1/2v_k+1^2 - γ_k-1/2y_k - y_k-1^2 
    ≤   ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k-1), y_k+1 - y_k ⟩
    
        =   ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k), y_k+1 - y_k ⟩+⟨∇_y f(x_k, y_k) - ∇_y f(x_k, y_k-1), v_k+1⟩
        +⟨∇_y f(x_k, y_k) - ∇_y f(x_k, y_k-1), y_k - y_k-1⟩.

 By backtracking strategy (<ref>) and the Cauchy-Schwarz inequality, we can easily get that

    ⟨∇_y f(x_k+1, y_k) - ∇_y f(x_k, y_k), y_k+1 - y_k ⟩≤(l_12^k)^2/2μ_kx_k+1 - x_k^2 + μ_k/2y_k+1 - y_k^2.

Combining (<ref>), (<ref>), (<ref>) and (<ref>), we have

    γ_k/2y_k+1 - y_k^2 - γ_k-1/2y_k - y_k-1^2 
    	   ≤(l_12^k)^2/2μ_kx_k+1 - x_k^2 + (μ_k/2 + γ_k-1/2 - γ_k/2) y_k+1 - y_k^2 
          - (μ_k-1 - (l_22^k-1)^2/2γ_k-1) y_k - y_k-1^2.

By multiplying 4γ_k-1/μ_k-1 on both sides of (<ref>), using the definition of S_k+1 and Assumption <ref>, we obtain








    S_k+1 - S_k 
    		   ≤2(l_12^k)^2γ_k-1/μ_k-1μ_kx_k+1 - x_k^2 + 2γ_k-1y_k+1 - y_k^2  
          - (4γ_k-1 - 2(l_22^k-1)^2/μ_k-1) y_k - y_k-1^2  + (2γ_k^2/μ_k - 2γ^2_k-1/μ_k-1) 𝒟_y^2.

The proof is then completed by Lemma <ref> and the definition.


We are now ready to establish the iteration complexity for the PF-AGP-NSC  algorithm. In particular, letting ∇ G(x_k, y_k) be defined as in Definition <ref> and ε > 0 be a given target accuracy, we provide a bound on T(ε), the first iteration index to achieve an ε-stationary point, i.e., ∇ G(x_k, y_k)≤ε, which is equivalent to

    T(ε) = min{k |∇ G(x_k, y_k)≤ε}.

 
Suppose that Assumptions <ref> and <ref> hold. 
Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>, β_k  =l_11^k + l_12^kμ_k-1/8 (l_22^k-1)^2  +  32(l_12^k)^2(l_22^k-1)^2/μ_kμ_k-1^2, γ_k = 8 (l_22^k)^2/μ_k,








then it holds that

    T(ε) ≤F_2 -F/d_1 ε^2 +1,

where  F_2 is defined in (<ref>), 
F = f - ( 224L_22^2/μ + μ_1 - 2 (l_22^1)^2/μ_1 - (l_22^1μ)^2/128L_22^2+2^13L_22^4/μ^3) 𝒟^2_y,








with f = min_(x, y) ∈ X × Y f(x, y),   𝒟_y = max{y_1 - y_2| y_1, y_2 ∈𝒴}, d_1 = min{β_min/2β_max^2+4L^2_12,15μ^3+2^7l_22^1μ/2^15L_22^4},  β_min=l^1_11 + l^1_12μ/64 (L_22)^2 + 32 (l_12^1)^2 (l^1_22)^2/μ_1^3 and  β_max = 2L_11+ μ_1 L_12/4(l^1_22)^2 +  4096(L_12)^2 (L_22)^2/μ^3.





 It follows immediately from Definition <ref> and (<ref>) that

    (∇ G_k)_x  = β_k  x_k+1 - x_k .

On the other hand, by  the triangle inequality, we obtain that

    (∇ G_k)_y    ≤γ_k  P_y (y_k + 1/γ_k∇_y f(x_k+1, y_k)) - P_y (y_k + 1/γ_k∇_y f(x_k, y_k)) 
          + γ_k  y_k+1 - y_k 
       ≤ L_12 x_k+1 - x_k  + γ_k  y_k+1 - y_k .

By combining (<ref>) and (<ref>), and using the Cauchy-Schwarz inequality, we obtain

    ∇ G_k ^2    ≤ (β_k^2 + 2 L_12^2)  x_k+1 - x_k ^2 + 2 γ_k^2  y_k+1 - y_k ^2.

Let d^(k)_1= min{2μ_kγ_k - (l^k_22)^2/2γ_k + μ_kγ_k-4 (l_22^k)^2/2μ_k/2 γ_k^2, β_k - l^k_11/2-  l^k_12/2γ_k-1 - 2 (l^k_12)^2 γ_k-1/μ_k-1μ_k/β_k^2 + 2 L_12^2}>0. Multiplying both sides of (<ref>) by d^(k)_1, and using (<ref>) in Lemma <ref>, we have

    d^(k)_1 ∇ G_k ^2    ≤ F_k - F_k+1.

By using the definitions of d_1, β_k and γ_k, we have
d_1<d^(k)_1, then we get

    d_1 ∇ G_k ^2 
    	≤ F_k - F_k+1.

Summing up the above inequalities from k = 2 to k = T(ε), we obtain

    ∑_k=2^T(ε) d_1 ∇ G_k^2    ≤ F_2 - F_T(ε)+1  .

Note that by the definition of F_k+1 in Lemma <ref>, we have

    F_T(ε)+1   = f_T(ε)+1 + S_T(ε)+1 -2γ_T(ε)^2/μ_T(ε)𝒟^2_y
          - ( 7/2γ_T(ε)+1 + μ_T(ε)+1 - 2 (l_22^T(ε)+1)^2/μ_T(ε)+1 - (l_22^T(ε)+1)^2/2γ_T(ε)+1) ||y_T(ε)+1 - y_T(ε)||^2 
       ≥f - ( 224L_22^2/μ + μ_1 - 2 (l_22^1)^2/μ_1 - (l_22^1μ)^2/128L_22^2+2^13L_22^4/μ^3) 𝒟^2_y 
       =F,

where the inequality follows from the definitions of f and 𝒟_y, and the fact that S_k ≥ 0 (∀ k ≥ 1). 
We then conclude from (<ref>) that

    ∑_k=2^T(ε) d_1 ∇ G_k^2    ≤F_2 - F_T(ε)+1≤F_2 - F  ,

which, in view of the definition of T(ε), implies that

    d_1 (T(ε)-1) ε^2 ≤ F_2 - F ,





or equivalently,

    T(ε) ≤F_2 -  F/d_1 ε^2+1.

We complete the proof.

 
At the k-th iteration of Algorithm <ref>, the number of backtracking step is upper bounded by log_2(2L_11/l_11^1)+log_2(2L_12/l_12^1)+log_2(2L_22/l_22^1)+log_2(2μ_1/μ)=𝒪(1). 
By Theorem <ref>, the total number of  gradient calls of Algorithm <ref> to obtain an ε-stationary point  that satisfies ∇ G(x_k,y_k)≤ε is upper bounded by 𝒪 ( Lκ^3ε^-2) under the nonconvex-strongly concave setting, where κ is the condition number.






 §.§ Nonconvex-concave Setting

In this subsection, we propose a completely parameter-free AGP algorithm for solving nonconvex-concave minimax problems. Similar to the idea of the PF-AGP-NSC Algorithm, we estimate the Lipschitz constants L_11, L_12 and L_22 iteratively, however we do not need to estimate the strongly concave constant μ. At each iteration of the proposed algorithm, we aim to find a tuple (l^k,i_11,l^k,i_12,l^k,i_22) by backtracking such that 
conditions (<ref>), (<ref>), (<ref>) and the following condition (<ref>) are satisfied at (x_k,i,y_k,i):

    C_5^k,i   = ⟨∇_y f_k,i(x_k,i, y_k,i) - ∇_y f_k,i(x_k,i, y_k), y_k,i - y_k⟩+ c^2_k,i+c_k,il^k,i_22/l^k,i_22 + 2c_k,iy_k,i - y_k^2
            +1/l^k,i_22 + 2c_k,i∇_y f_k,i(x_k,i, y_k,i) - ∇_y f_k,i(x_k,i, y_k)^2  ≤ 0. C5

The proposed algorithm, denoted as PF-AGP-NC, is formally presented in Algorithm <ref>.

Please note that the existing SGDA-B algorithm <cit.> for solving nonconvex-concave minimax problems requires the realization of the diameter of the feasible set 𝒴 in order to construct the key auxiliary problem, while the PF-AGP-NC algorithm we propose is a completely parameter-free algorithm that does not require any prior parameters.

 



  §.§.§ Complexity Analysis

In this subsection, we analyze the iteration complexity of Algorithm <ref> for solving (<ref>) under the general nonconvex-concave setting. 






The backtracking strategy (<ref>) is a key inequality that we will use to establish some important recursions for the PF-AGP-NC algorithm under the nonconvex-concave setting in the following two lemmas. This is also one of the main differences between proofs in the nonconvex- strongly concave setting and proofs in the nonconvex-concave setting.




Suppose that Assumptions <ref> and <ref> hold. Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>. If ∀ k, β_k > l^k_11 and γ_k≥l_22^k + 2c_k/2, then

    f(x_k+1,y_k+1) - f(x_k,y_k) 
    ≤    - (   β_k/2 - (l_12^k)^2/2γ_k-1) x_k+1 - x_k^2 +γ_k-1y_k+1- y_k^2 + γ_k-1/2y_k - y_k-1^2
          + c_k-1/2( y_k+1^2 - y_k^2 )+ c_k/2y_k+1-y_k^2.






	
	


The optimality condition for y_k in (<ref>) implies that




    ⟨∇_y f_k-1 (x_k, y_k) - γ_k-1 (y_k - y_k-1), y_k - y_k+1⟩≤ 0.





	
	
	
	
	

The concavity of f_k(x_k+1, y) with respect to y together with (<ref>) then imply that

    f_k (x_k+1, y_k+1) - f_k (x_k+1, y_k)    ≤⟨∇_y f_k (x_k+1, y_k)- ∇_y f_k-1 (x_k, y_k), y_k+1-y_k ⟩
         + ⟨∇_y f_k-1 (x_k, y_k-1), y_k+1 - y_k ⟩
       = ⟨∇_y f_k (x_k+1, y_k) - ∇_y f_k-1 (x_k, y_k), y_k+1 - y_k ⟩
         + ⟨∇_y f_k-1 (x_k, y_k) - ∇_y f_k-1 (x_k, y_k-1), v_k+1⟩
         + ⟨∇_y f_k-1 (x_k, y_k) - ∇_y f_k-1 (x_k, y_k-1), y_k - y_k-1⟩
         + γ_k-1⟨ y_k - y_k-1, y_k+1 - y_k ⟩,

where v_k+1 = y_k+1 - y_k - (y_k - y_k-1). We now provide upper bounds for the inner product terms on the right hand side of (<ref>). Firstly, by the definition of f_k(x_k+1, y_k) and f_k-1(x_k, y_k), the backtracking strategy (<ref>)  and the Cauchy-Schwarz inequality, we have

    ⟨∇_y f_k (x_k+1, y_k) - ∇_y f_k-1 (x_k, y_k), y_k+1 - y_k ⟩
       = ⟨∇_y f(x_k+1, y_k) - ∇_y f (x_k, y_k), y_k+1 - y_k ⟩  - (c_k - c_k-1)⟨ y_k, y_k+1 - y_k ⟩
       ≤(l_12^k)^2/2γ_k-1x_k+1 - x_k^2 + γ_k-1/2y_k+1 - y_k^2 - c_k - c_k-1/2 (y_k+1^2 - y_k^2)
          + c_k - c_k-1/2y_k+1-y_k^2
       ≤(l_12^k)^2/2γ_k-1x_k+1 - x_k^2 + γ_k-1/2y_k+1 - y_k^2 - c_k - c_k-1/2 (y_k+1^2 - y_k^2) 
          + c_k/2y_k+1-y_k^2.

Secondly, by the Cauchy-Schwarz inequality, we get

    ⟨∇_y f_k-1 (x_k, y_k) - ∇_y f_k-1 (x_k, y_k-1), v_k+1⟩
    ≤   1/2 γ_k-1∇_y f_k-1 (x_k, y_k) - ∇_y f_k-1 (x_k, y_k-1) ^2 + γ_k-1/2 v_k+1^2.

Thirdly, by (<ref>) we have that

    ⟨∇_y f_k-1(x_k, y_k) - ∇_y f_k-1(x_k, y_k-1), y_k - y_k-1⟩
    ≤    -1/l_22^k-1 + 2c_k-1∇_y f_k-1(x_k, y_k) - ∇_y f_k-1(x_k, y_k-1) ^2
         - c_k-1l_22^k-1+c^2_k-1/l_22^k-1 + 2c_k-1 y_k - y_k-1^2
    ≤    - 1/l_22^k-1 + 2c_k-1∇_y f_k(x_k, y_k) - ∇_y f_k(x_k, y_k-1) ^2.

We also have the following equation that

    γ_k-1⟨ y_k - y_k-1, y_k+1 - y_k ⟩ = γ_k-1/2 y_k+1 - y_k ^2 + γ_k-1/2 y_k - y_k-1^2 - γ_k-1/2 v_k+1^2.

By γ_k≥l_22^k + 2c_k/2, plugging (<ref>)-(<ref>) into (<ref>), 
 and using the definition of f_k(x_k+1, y_k+1) and f_k(x_k+1, y_k),   we obtain

    f(x_k+1, y_k+1) - f(x_k+1, y_k) 
    ≤   (l_12^k)^2/2 γ_k-1 x_k+1 - x_k ^2 + γ_k-1 y_k+1 - y_k ^2 + γ_k-1/2 y_k - y_k-1^2
          + c_k-1/2( y_k+1^2 - y_k^2 )+ c_k/2y_k+1-y_k^2  .

By  β_k > l^k_11, we have

    f(x_k+1, y_k) - f(x_k, y_k) ≤ -β_k /2 x_k+1 - x_k ^2 .

By adding (<ref>) and (<ref>), we complete the proof.


Suppose that Assumptions <ref> and <ref> hold. Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>. Denote

    S_k+1   = 8γ_k^2/c_k y_k+1 - y_k ^2 + 8γ_k( c_k+1/c_k-1 )  y_k+1^2,
    
    	F_k+1   = f(x_k+1, y_k+1) + S_k+1 - 15γ_k/2y_k+1 - y_k^2 - c_k/2y_k+1^2.

If 

    β_k > l^k_11,   γ_k+1/c_k+1 - γ_k/c_k≤1/5,   γ_k ≥l^k_22 + 2c_k/2 ,

then ∀ k ≥ 2,

    F_k+1 - F_k =    -( 9γ_k/10-c_k/2)y_k+1 - y_k^2 
       -(   + β_k/2 - (l^k_12)^2/2γ_k-1-16(l^k_12)^2γ_k-1/c_k-1^2)x_k+1 - x_k^2 
        +8( γ_kc_k+1/c_k - γ_k-1c_k/c_k-1) y_k+1^2+c_k-1-c_k/2y_k+1^2.



The optimality condition for y_k in (<ref>) implies that

    ⟨∇_y f_k(x_k+1, y_k)-γ_k(y_k+1-y_k), y_k - y_k+1⟩≤ 0,

By (<ref>) and (<ref>), we have  

    ⟨∇_y f_k-1(x_k, y_k-1) - ∇_y f_k(x_k+1, y_k) +   γ_k( y_k+1 - y_k)  - γ_k-1 (y_k - y_k-1), y_k+1 - y_k⟩≤ 0,

and

    γ_k y_k+1 - y_k^2 - γ_k-1⟨ y_k+1 - y_k, y_k - y_k-1⟩
    ≤   ⟨∇_y f_k(x_k+1, y_k) - ∇_y f_k-1(x_k, y_k-1), y_k+1 - y_k⟩,

 which can be rewritten as

    (γ_k - γ_k-1/2) y_k+1 - y_k^2 + γ_k-1/2v_k+1^2 - γ_k-1/2y_k - y_k-1^2
    ≤   ⟨∇_y f_k(x_k+1, y_k) - ∇_y f_k-1(x_k, y_k-1), y_k+1 - y_k ⟩
    
    =   ⟨∇_y f_k(x_k+1, y_k) - ∇_y f_k-1(x_k, y_k), y_k+1 - y_k ⟩+ ⟨∇_y f_k-1(x_k, y_k) - ∇_y f_k-1(x_k, y_k-1), v_k+1⟩
    
    	   + ⟨∇_y f_k(x_k-1, y_k-1) - ∇_y f_k-1(x_k, y_k-1), y_k - y_k-1⟩.

Using an argument similar to the proof of (<ref>)-(<ref>), by (<ref>) and the Cauchy-Schwarz inequality, we conclude from the above inequality that

    (γ_k - γ_k-1/2) y_k+1 - y_k^2 + γ_k-1/2v_k+1^2 - γ_k-1/2y_k - y_k-1^2
    ≤    (l_12^k)^2/2a_kx_k+1 - x_k^2 + a_k/2y_k+1 - y_k^2 - c_k - c_k-1/2 (y_k+1^2 - y_k^2)
        + c_k - c_k-1/2y_k+1 - y_k^2 + 1/2γ_k-1∇_y f_k-1(x_k, y_k) - ∇_y f_k-1(x_k, y_k-1)^2  
        +γ_k-1/2v_k+1- 1/l_22^k-1 + 2c_k-1∇_y f_k(x_k, y_k) - ∇_y f_k(x_k, y_k-1)^2  
         - c_k-1l_22^k-1+c^2_k-1/l_22^k-1  + 2c_k-1y_k - y_k-1^2,

for any a_k > 0. Obviously, we have -c_k-1l^k-1_22+c^2_k-1/2c_k-1 + l^k-1_22≤ -c_k-1l^k-1_22+c^2_k-1/2(l^k-1_22+c_k-1) = -c_k-1/2.
Combining γ_k ≥l^k_22 + 2c_k/2 and rearranging the terms in (<ref>), we obtain

    γ_k/2y_k+1 - y_k^2 + c_k - c_k-1/2y_k+1^2 
         ≤γ_k-1/2y_k - y_k-1^2 + ( α_k/2 + γ_k-1/2 - γ_k/2) y_k+1 - y_k^2  + c_k - c_k-1/2y_k^2 
           +  (l^k_12)^2 /2a_kx_k+1 - x_k^2 - c_k-1/2y_k - y_k-1^2.

By multiplying 16γ_k-1/c_k-1 on both sides of the above inequality, setting a_k = c_k/2, and using the definition of S_k+1, (<ref>) and {γ_k} is nodecreasing, we then obtain

    S_k+1 - S_k    ≤ 8( γ_kc_k+1/c_k - γ_k-1c_k/c_k-1) y_k+1^2 + 16 (l^k_12)^2 γ_k-1/c_k-1^2x_k+1 - x_k^2 
         + 28γ_k/5y_k+1 - y_k^2 - 8γ_k-1y_k - y_k-1^2.




	
	





	
	
	
	
	
	

Combining  (<ref>) and (<ref>), and using the definition of F_k+1,  we complete the proof.


We are now ready to establish the iteration complexity for the PF-AGP-NC algorithm to achieve an ε-stationary point under the general nonconvex–concave setting.

Suppose that Assumptions <ref> and <ref> hold. Let {(x_k, y_k)} be a sequence generated by Algorithm <ref>. If β_k = l_12^k/20l^k-1_22 + 1280(l_12^k)^2 √(k)/19^2l_12^k-1, γ_k =20l^k_22 ,  c_k = 19l^k_22/k^1/4,  then for any given ε > 0, k ≥ 2, we have

    T(ε) ≤max(  ( 5120 ( L_12)^2  d_2d_3/19^2l_22^1 ε^2+1)^2,  76^4L_22^4𝒟̂_y^4/ε^4),

where  d_2= F_2 - F +339L_22𝒟̂_y^2, d_3 = max{d̅_1, 20·19^2L_22^2/17(l_12^1)^2}, 𝒟̂_y = max{y  |  y ∈ Y} with F = f - 639L_22𝒟̂_y^2, f = min_(x,y) ∈ X × Y f(x, y), 
  d̅_1 =  32+ 1+1600L^2_22/128(l_12^1)^2.
 
















We can easily see that the relations in (<ref>) are satisfied by the settings of c_k, γ_k and β_k. Denote α_k = 16 (l_12^k)^2 γ_k-1/c_k-1^2. Then it follows from the selection of  β_k and α_k that

    β_k/2 - (l^k_12)^2/2γ_k-1-16(l^k_12)^2γ_k-1/c_k-1^2 = α_k .

This observation, in view of Lemma <ref>, c_k/2≤19γ_k/40k^1/4≤19γ_k/40 implies that

    α_k x_k+1 - x_k ^2 + 17γ_k/40 y_k+1 - y_k ^2 
    ≤    F_k -F_k+1 
    		+8( γ_kc_k+1/c_k - γ_k-1c_k/c_k-1) y_k+1^2 +c_k-1-c_k/2y_k+1^2.

We can easily check from the definition of f_k(x, y) that

    ∇ G_k  - ∇̃ G_k ≤ c_k  y_k .

By replacing f with f_k, similar to (<ref>) and (<ref>), we immediately obtain that

    (∇̃ G_k)_x  =   β_k     x_k+1 - x_k ,

and

    (∇̃ G_k)_y ≤γ_k  y_k+1 - y_k  + l^k_12 x_k+1 - x_k .

Combining (<ref>) and (<ref>), and using the Cauchy-Schwarz inequality, we have

    ∇̃ G_k ^2 ≤ (  β_k ^2 + 2(l^k_12)^2)  x_k+1 - x_k ^2 + 2γ_k^2 y_k+1 - y_k ^2.

 Since both α_k and β_k are in the same order when k becomes large enough, it then follows from the definition of d̅_1 that ∀ k ≥ 1,

    (  β_k )^2 + 2(l^k_12)^2/α_k^2   = 
    	(l^k_12/γ_k-1 + 64 (l_12^k)^2γ_k-1/c_k-1^2)^2 + 2(l^k_12)^2/α_k^2
    
    	
    	
    	
     	   ≤  32+ 1+1600L^2_22/128(l_12^1)^2 = d̅_1.

Combining the previous two inequalities in (<ref>) and (<ref>), we obtain

    ∇G̃_k^2 ≤d̅_1(α_k)^2 x_k+1 - x_k ^2 + 2 γ_k^2  y_k+1 - y_k ^2 .

Denote d_k^(2) = 1/max{d̅_1 α_k, 17/80γ_k}, c̃_k = 19/20 k^1/4, by multiplying d^(2)_k on the both sides of (<ref>), and using (<ref>), we have

    d_k^(2)∇G̃_k ^2 
    ≤    F_k - F_k+1  +8( γ_kc_k+1/c_k - γ_k-1c_k/c_k-1) y_k+1^2 
    	+c_k-1-c_k/2y_k+1^2
    ≤    F_k - F_k+1  +8( γ_kc_k+1/c_k - γ_k-1c_k/c_k-1) y_k+1^2 
    	+40L_22c̃_k-1-c̃_k/2y_k+1^2.

Denoting

    T̃(ε) = min{ k | ∇G̃(x_k, y_k) ≤ε/2, k ≥ 2 },

summing both sides of (<ref>) from k = 2 to k = T̃(ε), we then obtain

    ∑_k=2^T̃(ε) d^(2)_k ∇G̃_k ^2 
    	   ≤  F_2 - F_T̃(ε)  + 8 (  γ_T̃(ε)-1c_T̃(ε)/c_T̃(ε)-1-γ_1c_2/ c_1) 𝒟̂_y^2 + 40L_22( c̃_1/2 - c̃_T̃(ε)/2) 𝒟̂^2_y 
       ≤ F_2 -  F_T̃(ε) + 339L_22𝒟̂_y^2.

Note that by the definition of F_k+1 in Lemma <ref>, we have

    F_T̃(ε)   ≥f -  639L_22𝒟̂_y^2 = F,

where f = min_(x, y) ∈ X × Y f(x, y). We then conclude from (<ref>) that

    ∑_k=2^∞ d^(2)_k ∇G̃_k ^2 ≤ F_2 - F +339L_22𝒟̂_y^2= d_2.

We can see from the selection of  d_3 that d_3 ≥max{d̅_1, 80γ_k/17α_k}, which implies d^(2)_k ≥1/d_3α_k,  by multiplying d_3  on the both sides of (<ref>), and combining the definition of  d_2,  we have  ∑_k=2^∞1/α_k∇G̃_k ^2 ≤ d_2d_3,
which, by the definition of T̃(ε), implies that

    ε^2/4≤d_2d_3/∑_k=2^T̃(ε)1/α_k≤d_2d_3/∑_k=2^T̃(ε)1/α̅.

By using the fact ∑_k=2^T̃(ε)1/√(k)≥√(T̃(ε)) - 1  and (<ref>), we conclude ε^2/4≤1280·( L_12)^2  d_2d_3/19^2 l_22^1( √(T̃(ε)) -1 ) or equivalently,

    T̃(ε)  ≤( 5120 ( L_12)^2  d_2d_3/19^2l_22^1 ε^2+1)^2.

On the other hand, if  k ≥76^4L_22^4𝒟̂_y^4/ε^4, then  c_k = 19γ_k/20 k^1/4≤ε/2𝒟̂_̂ŷ. This inequality together with the definition of  𝒟̂_y, then imply that c_k  y_k ≤ε/2. Therefore, there exists a

    T(ε)    ≤max( T̃(ε), 76^4L_22^4𝒟̂_y^4/ε^4)
       ≤max(  ( 5120 ( L_12)^2  d_2d_3/19^2l_22^1 ε^2+1)^2,  76^4L_22^4𝒟̂_y^4/ε^4),

such that ∇ G_k ≤∇G̃_k  + c_k  y_k ≤ε/2  + ε/2 = ε. 

















At the k-th iteration of Algorithm <ref>, the number of backtracking step is upper bounded by  log_2(2L_11/l_11^1)+log_2(2L_12/l_12^1)+log_2(2L_22/l_22^1)=𝒪(1).
 According to Theorem <ref>, the total number of  gradient calls of Algorithm <ref> to obtain an ε-stationary point  that satisfies ∇ G(x_k,y_k)≤ε is 𝒪 ( L^4ε^-4) under the nonconvex-concave setting.











§ NUMERICAL RESULTS
 
In this section, we compare the numerical performance of the proposed PF-AGP-NC algorithm with the AGP algorithm.



 §.§ Dirac-GAN problem


The Dirac-GAN problem <cit.> can be formulated as the following nonconvex–concave minimax problem :


    min_x max_y L(x, y) =-log(1 + exp(-xy)) + log 2,

where (0, 0) is the unique stationary point.

Experimental setup.
We set the initial point of the experiment at (x, y) = (1, 1). The termination condition is defined as the norm of the gradient ∇ f(x_i, y_i) being less than or equal to 10^-5, where i represents the number of gradient calls.

For the AGP algorithm, let α_x and β_y be the stepsize of x and y, respectively. We set α_x = 0.8/√(k), β_y = 0.3 and c_k =0.5/k^1/4. For the PF-AGP-NC algorithm, we do not need to preset specific parameters.

In the experiments, we counted the number of gradient calls of the PF-AGP-NC algorithm. The results of alternating gradient projection in the k th iteration are represented by nodes in the graph and the number of backtracking step computations is represented by the connecting lines.

Results Figure <ref> shows the convergence curves of the AGP and PF-AGP-NC algorithms. It can be seen that although the PF-AGP-NC algorithm involves a double-layer structure, requiring the determination of points in the outer layer through computations in the inner layer, even when considering the number of iterations in the inner layer, the overall convergence speed is still faster than that of the AGP algorithm.





§ CONCLUSIONS AND DISCUSSION

In this paper, based on the framework of backtracking, we propose two completely parameter-free alternating gradient projection algorithms, i.e., the PF-AGP-NSC algorithm and the PF-AGP-NC algorithm, for solving nonconvex-(strongly) concave minimax problems respectively, which does not require prior knowledge of parameters such as the Lipschtiz constant L or the strongly concave constant μ.   Moreover, we show that the total number of  gradient calls of the PF-AGP-NSC algorithm to obtain an ε-stationary point for nonconvex-strongly concave minimax problems is upper bounded by 𝒪( Lκ^3ε^-2) where κ is the condition number,  while the total number of gradient calls of the PF-AGP-NC algorithm to obtain an ε-stationary point for nonconvex-concave minimax problems is upper bounded by 𝒪( L^4ε^-4). As far as we know, the PF-AGP-NSC algorithm and the PF-AGP-NC algorithm are the first completely parameter-free algorithms for solving nonconvex-strongly concave minimax problems and nonconvex-concave minimax problems respectively.  Numerical results demonstrate the efficiency of the two proposed algorithms.





















































































































































































































 Paper Title: Insurance Portfolio Pursuit with Reinforcement Learning.



The Inevitable Quark Three-Body Force and its Implications for Exotic States
    Su Houng Lee
    August 5, 2024
============================================================================



§ ABSTRACT 
	When faced with a new customer, many factors contribute to an insurance firm's decision of what offer to make to that customer. In addition to the expected cost of providing the insurance, the firm must consider the other offers likely to be made to the customer, and how sensitive the customer is to differences in price. Moreover, firms often target a specific portfolio of customers that could depend on, age, location, and occupation. Given such a target portfolio, firms may choose to modulate an individual customer's offer based on whether the firm desires the customer within their portfolio. Given a target portfolio, we term the problem of modulating offers to achieve this target portfolio the portfolio pursuit problem. We give a formulation of portfolio pursuit as a sequential decision making problem, and devise a novel reinforcement learning algorithm for its solution. We test our method on a complex synthetic market environment, and demonstrate that it outperforms a baseline method which mimics current industry approaches to portfolio pursuit. 




§ INTRODUCTION




 §.§ Background


Competition among the commercial insurance market has been intensified by the emergence of price comparison websites (PCW) as a core means by which customers and insurance firms interact. When a customer's details are entered into the website, many firms simultaneously decide what offers to make to the customer on the basis of those details. Since the customer can directly compare competing offers, firms must carefully calibrate their offers to maximise a trade-off between potential profit and the probability of acceptance. While both of these aims are local to each individual customer, firms may also wish to modulate the offers they make in order to pursue longer-term goals, realised over many customers. For example, the firm may wish to maintain a certain frequency with which new offers are accepted, to ensure they have the reserves necessary to pay out claims and avoid liquidity issues <cit.>. Alternatively, a firm may wish to break into a new segment of the market, and be content with making lower-than-optimal offers in order to attract a particular customer base. Goals such as these define a higher-level strategy that is crucial to the long-term success of the firm. Decisions regarding content of the long-term goals - the What? of the strategy - are typically made at a managerial level, and involve complex domain-expert knowledge and understanding of current market conditions. At present, the implementation of the pursuit of the goals - the How? of the strategy - also makes use of domain-expect knowledge. In this paper, we present a data-driven machine learning methodology for implementing higher-level strategies in the context of consumer insurance markets. 

The problem of what offer to make to a customer on a PCW can be divided into two sub-problems: the cost-estimation problem and the bidding problem. The cost-estimation problem can be summarised as the problem of estimating the total cost of insuring a given customer from the features available through the PCW. This is a well-studied supervised-learning problem <cit.>. We will not provide any further comment on the cost-estimation problem in the remainder of this paper; we take the solution to this problem as given. Instead, we focus on the bidding problem - what offer ought we to make to a customer, given an estimate of the cost of insuring them? The optimal price to offer a customer depends not only on the estimated cost of insuring them, but additionally on, (1) the offers that other insurers on the PCW will make to the customer; (2) the customer's sensitivity to price differentials; and, (3) whether the customer is the type of client the insurance firm wishes to insure. Our work focuses on this third aspect the bidding problem. 

Throughout our paper, we use the term portfolio to refer to the set of customers who accept an insurance firm's offers over some specified period of time (for example a day, week, or month). The problem of portfolio optimisation is that of deciding what the ideal portfolio would look like; how many of each different type of customer the insurance firm wishes to have concurrently insured. This may depend on aspects such as the kind of brand and reputation the firm wishes to cultivate, the desire to maintain a diverse customer base so as to protect against correlated risk or systematic model inaccuracies, or the necessity to avoid insuring too many customers and being unable to settle all the incoming claims. We take the firm's solution to the problem of portfolio optimisation also as given; that is, we suppose that the insurance firm already has a specification for the distribution of customers it would like to insure over some relevant future time-period. However, unlike in the stock market, where an investor can easily achieve their desired portfolio of stocks by simply investing and divesting, an insurance firm cannot arbitrarily add and remove customers from its portfolio. Rather, the firm must wait until a desired customer is asking for a quote, and then modulate the offer made to the customer in order to increase the likelihood that the offer is accepted. In deciding how to modulate the offer, the firm must carefully balance the value of having the customer in the portfolio against the profit they expect to make from insuring the customer. It would not be effective to, for example, make desired customers offers at which the firm is, on average, guaranteed a substantial loss. We term the problem of how to modulate offers made to customers in order to achieve a desired portfolio the problem of portfolio pursuit. 

It is to the problem of portfolio pursuit to which we make our contribution. Note that – unlike the cost-estimation, bidding, and portfolio optimisation problems – portfolio pursuit is intrinsically sequential in nature. If we wish to insure no more than 500 customers over the next week, the offers we ought to make to customers on Sunday depend heavily on our success rate for Monday through Saturday. As such, we cast the portfolio pursuit problem as a Markov Decision Process (MDP), and derive a novel Reinforcement Learning (RL) algorithm for its solution. Our decomposition into the four sub-problems given above - cost-estimation, pricing, portfolio optimisation, and portfolio pursuit - additionally enables us to develop a methodology which is minimally disruptive, in the sense that it can be integrated naturally with arbitrary solutions to the other three sub-problems. Indeed, our algorithm requires only a slight adjustment to existing methodologies for the bidding problem. 



 §.§ Previous work


To our knowledge, our paper is the first work to articulate a general mathematical formalism of the portfolio pursuit problem in insurance as a sequential decision making problem, and attempt to address it using reinforcement learning. <cit.> and <cit.> both apply reinforcement learning to other aspects of bid optimisation in the context of insurance. Like us, <cit.> considers the new business market for insurance. They have no notion of a desired portfolio, thus making each decision purely local. As such, they treat the problem as a contextual bandit, with no sequential aspect (beyond learning). In contrast, <cit.> examine the problem of pricing at renewal. While they consider the customer portfolio (thereby making the problem sequential), they restrict themselves only to constraints on the size of the portfolio, formulating it as a constrained Markov Decision Process. By additionally factoring in the internal structure of customer portfolios, our framework presents a more general and flexible approach to portfolio pursuit. 

Many previous works have attempted to apply reinforcement learning to the portfolio optimisation problem in the context of other markets, such as the stock market <cit.> and the cryptocurrency market <cit.>.  However, as noted above, this situation is fundamentally different from the case considered here. In these other markets, the relevant financial products can typically be purchased very quickly, allowing one to shift funds between them more or less freely (subject to transaction costs). This is in contrast to customers, who arrive sequentially and over a prolonged period. Furthermore, while purchasing cryptocurrency or a stock is a fairly simple transaction, bidding for a customer is not, since it involves attempting to out-compete alternative offers while accounting for stochasticity in a customer's decision. As such, this previous literature is not directly applicable to the problem under consider here. 



 §.§ Our contribution


We propose a novel reinforcement learning algorithm for the portfolio pursuit problem over long time horizons. We cast this as an MDP, and give a novel algorithm for its solution. We take a model-based approach, allowing our algorithm to be trained entirely offline, thereby protecting against risk in deployment. The methodology we use can be integrated naturally into existing machine learning pipelines for insurance pricing. Unlike previous research addressing using RL for the purposes of insurance pricing <cit.>, our insurance agent acts in a realistic agent-based environment <cit.>. This environment includes randomly generated customers along with a collection of other insurance agents that iteratively change their pricing strategies over time, creating a dynamic, non-stationary environment. The full code implementing both the market environment and our reinforcement learning solution is available in the https://github.com/EdwardJamesYoung/RL-portfolio-pursuitcode repository accompanying this paper. 

The rest of this paper is organised as follows. In sec:problem formulation we give a mathematical formulation of the portfolio pursuit problem. In sec:methods we describe our solution to the problem. We begin by describing a standard pipeline for solving the bidding problem, and the models used within that pipeline. We then outline a baseline methodology intended to mimic a naive approach to portfolio pursuit that may currently be used within industry. Lastly, we specify our approach, giving the full algorithm used for training, and compare the method to existing RL algorithms. In sec:results we show the performance of our method against the baseline, demonstrating that our method is able to generate more profit than the baseline methodology while not compromising the quality of the final portfolio. Finally, in sec:discussion we outline some limitations of our method and give opportunities for future research. 



§ PROBLEM FORMULATION




 §.§ Portfolio pursuit as a Markov Decision Process


In our market environment, an epoch consists of T interactions between a customer and the insurance agent. The customers s_1,…,s_T themselves are drawn independently from the same underlying customer distribution. At each step, in response to the current customer features, the insurer takes an action a_t ∈ [1,2]. The action a corresponds to an offer of C(s) × a, where C(s) is the expected cost of serving the customer[As noted in the introduction, we assume this as given. In practice, the expected cost of serving a customer is not known in advance, but must instead be estimated using historic claims data. However, this estimation problem is distinct from the problem of how to price insurance, given this estimate, which is what this paper focuses on.]. In response to the offer made by the insurer, and the offers made by other insurers within the market, the customer either accepts the offer or rejects it. If the offer is rejected (y_t = 0) then no profit is obtained by the insurer. If the offer is accepted (y_t = 1) then an (expected) profit of (s_t,a_t) is obtained. The profit is simply (s,a) = C(s)a - C(s) = C(s)(a - 1). When the offer is accepted (y_t = 1) then the customer is added to the insurer's portfolio[In reality, there may be a delay between an offer being made and accepted - indeed, customers often have up to 14 days to decide whether to accept an offer. In this case, a proxy measure, such as click-through from the PCW to the insurer's website, may be used in place.]. The portfolio at the t-th time-step is ρ_t, and the insurer's portfolio before the first customer ρ_1 is initialised as empty[The choice to initialise the portfolio as empty corresponds to optimising the portfolio of new customers obtained over some time horizon. If instead the firm is interested in its total portfolio - including all existing customers - the portfolio can be initialised to that instead, with only minor adjustments required elsewhere in our method.], ρ_1 = {}.

The performance of the insurer at the final time step is evaluated according to the net profit achieved minus a loss which measures the discrepancy between the final and desired portfolio[Note that this encompasses the case where the loss is a function of only the final portfolio.], 
    ∑_t=1^T y_t (s_t,a_t) - λŁ(ρ_T+1, ρ^*).

Here, λ is a coefficient which controls the relative importance of profit vs. pursuit of a particular portfolio. 

In the MDP formalism, the states consist of customer, portfolio, and time-step triples, (s_t, ρ_t, t). The reward signal following action a_t is y_t (s_t,a_t), with an additional reward -λŁ(ρ_T+1, ρ^*) administered after the final customer interaction.



 §.§ Specification of portfolio loss function


We stress that the algorithm we present in sec:RL for PP makes no assumptions whatsoever about the nature of the loss function Ł. However, for concreteness in our numerical experiments, we adopt one particular formulation of the loss. 

The loss function used in our numerical experiment is defined via a collection of I indicators functions _A_i, i = 1,…,I over the customer features. Each indicator is specified by set A_i, i = 1,…,I in the space of customer features. These indicators allow us to represent portfolios as vectors in ^I, with the i-th component given by f_i = ∑_s ∈ρ_A_i(s), where _A_i(s) is equal to 1 if s ∈ A_i, and 0 otherwise. Given these vector representations of portfolios, we can implement the loss Ł via any distance measure on ^I. Note that in practice we specify the target portfolio ρ^* only through its vector representation f^*_i ∑_s ∈ρ^*_A_i(s); similarly, we will track the insurer's portfolio only via its frequency representation. For our numerical experiments we adopt the loss function:

    ℒ(f, f^*) 1/I∑_i=1^I   |f_i - f^*_i| /max(f_i, f_i^*) .

We choose this loss for the following properties: it is zero precisely when f_i = f^*_i; it is bounded between [0,1]; and it is symmetric in f and f^*. Note that because the loss is bounded between 0 and 1, the coefficient λ can be interpreted as answering the question: if the firm had the worst possible portfolio, how much short-term profit would they be willing to give up in order to trade it for the best possible portfolio? 

Our reinforcement learning algorithm (detailed in sec:RL for PP) trivially extends to alternative loss functions which provide other ways of quantifying the distance between the obtained and target portfolio. This includes, for example, adding weights to each indicator to penalise discrepancies along different feature sets differently, or using an optimal transport distance to provide a more detailed measure of differences between sets. We leave exploration of alternative loss formulations to future work. 



§ METHODS




 §.§ Standard industry methodology


We compare our method to a pipeline which mimics current industry approaches to insurance pricing and price modulation. In this section we describe a methodology which does not make reference to a target portfolio. This pipeline mimics standard industry pricing methods. We will then modify this method in two ways - firstly, to mimic standard industry methodology for modulating prices to attain a target portfolio, and secondly according to our new proposed algorithm.  

The insurance firm is assumed to have a dataset of interactions from previous epochs. Each entry in this interaction dataset includes:

	
  * The customer features, s_t
	
  * The expected cost of serving the customer, C(s_t)
	
  * The action taken by the insurance firm, a_t
	
  * The offers made by other insurance firms, o_t
	
  * Whether the customer accepted the offer of the firm, y_t

From the offers made by other firms, o_t, and the expected cost, C(s_t), the insurer calculates market variables, m_t. The market variables for the insurer are the average of the top-1, top-3, and top-5 offers made to the customer by other firms within the market, normalised by dividing by the expected cost C(s_t) of serving the customer. Note that the market variables are not available to the insurer at inference time, but become available after the fact through the purchasing of historic data from PCWs. 

From this dataset, the insurer trains the following models:

	
  * The market model M, which maps from customer features, s_t, to market variables m_t, s_t ↦ M(s_t) ≈ m_t.
	
  * The conversion model p, which maps from the market variables, m_t, and action, a_t, to the probability the customer accepts the offer, (m_t,a_t) ↦ p(m_t,a_t) ≈(y_t = 1|m_t, a_t).
	
  * The action model, or policy π, which maps from market variables, m_t, to optimal actions a^*_t, m_t ↦π(m_t) ≈ a^*_t.

For our numerical simulations, we used a random forest for the market model, a Multi-Layer Perceptron (MLP) for the conversion model, and a Gaussian Process with Matern kernel for the action models <cit.>. Our overall methodology places no restrictions on any of the models, although in practice we found it better to use continuous conversion and action models. If it is necessary to ensure these models are interpretable, simpler models - such as Generalised Linear Models (GLMs) - may be used instead of MLPs <cit.>. More expressive models may work better for larger scale simulations, with correspondingly larger datasets, than those explored here. 

The market model M and conversion model p are trained directly on the historic dataset available to the insurer. To train the action model, we use regression-based amortised optimisation <cit.>. Firstly, note that the expected profit, given that the customer accepts the insurer's offer, is C(s_t)(a_t - 1). Accordingly, the insurer's expected profit, for action a_t, given market variables m_t, is C(s_t)(a_t - 1)p(m_t, a_t). Therefore, for a random sample of market variables m_t from the dataset, we use a one-dimensional optimiser to find a^*_t ∈max_a p(m_t, a)(a - 1) within the range a_t ∈ [1, 2]. We then fit the action model π to 500 (m_t, a^*_t) pairs. 

At inference time, the features for a customer s_t are fed into a market model to obtain estimates for the market variables[As the market variables are predictive of customer behaviour, and available historically but not at inference time, the use of a market model can be seen as a form of data imputation.], m_t = M(s_t). These are then fed into the action model to obtain the action a_t = π(m_t). An offer of C(s_t)a_t is then made to the customer. On 12% of customers we explore by multiplying the offer by a randomly chosen modulation factor sampled from [0.93, 0.95, 0.97, 1.03, 1.05, 1.07]. 

Before continuing, we briefly remark that in practice the training and use of auxiliary models may be more complicated than presented here. To begin with, information about whether a customer accepts an offer is not available instantaneously, but only after a “cooling-off” period, typically 14 days. Likewise, there may be a delay in obtaining information about market variables for a given customer. Finally, insurers may choose to forgo the use of an action model all-together, instead solving the one-dimensional optimisation problem directly at inference time (rather than amortising as we do here). None of these observations present substantial difficulty to the RL algorithm we describe below. Indeed, our approach can be seen as a methodology which accepts as input auxiliary models and adjusts the prices returned by those models to solve the portfolio pursuit problem. It is therefore agnostic as to both the types of models used, and the data which those models are trained on. 



 §.§ Baseline methods for portfolio pursuit


It is standard practice to modulate the output prices offered to customers from the bidding model according to other features of interest for the customer, corresponding to the customer's desirability within the portfolio. This modulation is typically performed using domain experts' knowledge, rather than according to a data-driven or formal scheme. We attempt here to give a method which mimics this type of modulation as a baseline against which to compare our method, fully recognising that it does not accurately capture the complexity of standard industry practices. We encourage future work to implement more realistic and nuanced methodologies, and have facilitated this by making our codebase implementing the market environment publicly available. 

The baseline method makes use of historical assessments of the frequency with which the insurer encountered customers of the type relevant to their portfolio. In particular, for each indicator set A_i, compute the frequency representation f_i = ∑_s ∈ρ_T+1_A_i(s) for terminal portfolios ρ_T+1 for epochs during which there was no portfolio-based action modulation, where actions were chosen via the method described in the previous section, sec:standard industry method. We average this value over such historical epochs, giving us a frequency f̅_i. f̅_i therefore acts as an estimate of how many customers within set A_i we are likely to get without action modulation. We compare this to the desired frequency, f^*_i = ∑_s ∈ρ^*_A_i(s). For each incoming customer, we look at each set A_i that they call into, and compute the corresponding ratio of frequencies, f̅_i/f^*_i. If this ratio is greater than 1 it indicates that, in order to achieve the desired portfolio, we should increase the offer price made to that customer. If the ratio is less than 1, we should decrease the offer price. At inference time, we therefore modulate the offer price made to a customer s by multiplying the action by

    ∏_i: s ∈ A_i g( f̅_i/f^*_i).

The map g is given by 

    g(z) = 1 + βz^n - 1/z^n + 1,
 
where n, β≥ 0. The functional form of g was selected because it is monotonically increasing and bounded, with g(z) ∈ [1 - β, 1 + β]. The function parameters n ∈{0.5, 1, 2} and β∈{ 0.02, 0.05, 0.1, 0.25 } are found by a grid-search; in our experiments we used values of n = 1, β = 0.02 for a loss coefficient of λ = 2000. Note that for a baseline method, β, n must be chosen differently for each value of the loss coefficient λ. This is in contrast to our method, which applies equally well for any value of λ. 



 §.§ A Reinforcement Learning approach to Portfolio Pursuit




  §.§.§ Mathematical Preliminaries


Our reinforcement learning algorithm uses a dynamic programming approach with function approximation to provide a policy which modulates customer prices according to the current portfolio possessed by the insurer. We define the portfolio value function, V_π(ρ, t), which gives the value of having portfolio ρ at time t, given that future actions are selected by a (time-step dependent, deterministic) policy π = π(s, ρ, t):

    V_π(ρ, t) _π[ ∑_k=t^T y_k (s_k, a_k) - λŁ(ρ_T+1, ρ^*) | ρ_t = ρ].

The corresponding action value function Q_π(s,a,ρ,t) is defined by:

    Q_π(s, a, ρ, t) _π[ ∑_k=t^T y_k (s_k, a_k) - λŁ(ρ_T+1, ρ^*) | ρ_t = ρ, s_t = s, a_t = a ].

The policy is optimal for all time steps if it satisfies the appropriate Bellman optimality equation, 
    π(s, ρ, t) ∈max_a Q_π(s, a, ρ, t).

In app:bellman recursion, we show by expanding the action value function that (<ref>) is equivalent to:

    π(s, ρ, t) ∈max_a (y_t = 1 |s_t = s, a_t = a) ( a - k_π(s,ρ, t+1) ),
 
where k_π(s, ρ, t+1) is defined by:

    k_π(s, ρ, t)  1 - V_π(ρ∪{s}, t) - V_π(ρ, t)/C(s).

An important consequence of the above result is that the optimal policy is a function only of the customer features s_t and the value k_π, and not of the portfolio or time-step directly except via k_π.

We also obtain (see app:bellman recursion) the following recursion relationship for the portfolio value function:

    V_π(ρ, t) = V_π(ρ, t+1) + [  C(s) (y_t = 1 |s_t = s, a_t = π(s,ρ, t)) ( π(s,ρ, t) - k_π(s, ρ, t+1) )  ],

with boundary condition V_π(ρ, T+1) = - λŁ(ρ, ρ^*). Here, the expectation is taken over the customer distribution. 



  §.§.§ Outline of our method


As with the industry baseline method, our method will make use of a market, conversion, and action model. We will use the conversion model p(m, a) ≈(y_t = 1 | m_t = m, a_t = a) in place of the (y_t = 1|s_t = s, a_t = a) in both the value recursion relationship (<ref>) and the optimal action relationship (<ref>)[We stress that this is only an approximation. In reality – owing to how we constructed the market environment – the probability of acceptance also depends on various customer features.]. Additionally, our action model (or policy) π will now be a function of both the market variables m and the k-values K, π(m,k). 

Our method begins by training these models in much the same manner as the pipeline described in sec:standard industry method, namely by fitting models to a previously obtained dataset. However, our method for generating the dataset for fitting the action model π is slightly different. We first sample market variables m_t from our dataset. For each market variable sample, we independently sample a k-value, k_t, from a Laplace distribution with location 1 and scale 0.1. We then compute 

    a^*_t ∈max_a p(m_t, a)(a - k_t),
 where p is our conversion model. We then fit the action model to approximate π(m_t, k_t) ≈ a^*_t. We used a sample of 500 (m, k, a^*)-triples to fit our conversion model. We additionally train a value function V_ϕ(ρ, t), with parameters ϕ, as outlined in the next section. 

At inference time, for a customer s_t with expected cost C(s_t) and portfolio ρ_t at time t, we use (<ref>) to compute the k-value k_t via:

    k_t = 1 - V_ϕ(ρ_t∪{s_t},t+1) - V_ϕ(ρ_t,t+1)/C(s_t).

We then feed the customer features s_t into the market model to obtain estimates for the market variable m_t = M(s_t). We then use the bidding model to obtain the action a_t = π(m_t, k_t). Finally, we make an offer of C(s_t)a_t to the customer. 



  §.§.§ Value function training


In order to train our inference-time value function V_ϕ(ρ,t), we make use of an next-step value function, U(ρ). This function is used during training, but not at inference time. Throughout training, it approximates the values of portfolios at the next time step. U is used to create value estimates, which are then stored in a dataset. Our value function V_ϕ (which will be used at inference time) is then fit to this dataset. As discussed in sec:portfolio loss function, portfolios are specified only through their frequency representation, ρ↦ (f_i)_i=1^I, f_i = ∑_s ∈ρ_A_i(s), since this preserves all information relevant for computing the loss. Accordingly, both U and V_ϕ act on frequency representations, vectors in ^I. In the remainder of this section we do not distinguish between a portfolio and its frequency representation. 

We iterate backwards through time, setting t = T,…,1. At each time step, we use U(ρ) to approximate V_π(ρ, t+1). Initially, we set U(ρ) = -λℒ(ρ, ρ^*) = V_π(ρ, T+1). At each time step, we use the next-step value function U to form value estimates via (<ref>) and (<ref>). To approximate the expectation over the customer distribution in (<ref>), we compute an empirical average over samples from a customer reply buffer, which stores the features, market variables, and expected cost of previous customers. We then fit U to these value estimates thus obtained, such that U now approximates V_π(ρ, t). Having fit U, we sample an augmenting set of portfolios, and use U to estimate each of their values. We finally store the value estimates in a dataset, 𝒟, and then iterate backwards, t  t - 1. Once we have completed the backwards iteration through time, we fit our value function V_ϕ to the value estimates found in 𝒟.

Before fitting, we apply a trick to make our dataset easier to learn. Notice that at inference time, V_ϕ is used only to compute k values via (<ref>). However, this equation is invariant to shifting all values at same time step t by the same amount. Therefore, before fitting V_ϕ, we recentre all the value estimates V at each time step by subtracting the mean value estimate, V̅.  We then fit our value function V_ϕ to the residuals, V̂ = V - V̅. We found that this step is critical to effective learning, as it requires the value function V_ϕ to learn the differences in value between portfolios, rather than overall magnitude. These differences are what drives decision making - indeed, the centring of our value estimates makes V_ϕ analogous to an advantage function used in other RL methods. 

Our approach is summarised in alg:training value function. In our experiments, we set U to be linear model, and V_ϕ to be an MLP. As with the conversion, market, and action models, nothing in our algorithm necessitates the use of these particular models for U and V_ϕ. We use N = 500 samples from the customer reply buffer, and J = 24 portfolio samples for fitting the next-step model, and an additional J^+ = 120 augmenting portfolios. 



In our algorithm, there are two points at which we sample portfolios. We have not yet specified how this sampling occurs. Ideally, these portfolios would be on-policy, the portfolios sampled for time step t would be taken from the distribution of portfolios our policy encounters at time t. However, since we do not have access to any such data, our method is forced to be off-policy. The sampling of portfolios comes from three sources: previously on-policy portfolios, target on-policy portfolios, and high-coverage portfolios. The sampling is done in a ratio of 1:1:2. We explain each of these methods in turn. 

Much like the baseline method discussed in sec:baseline method, the previously on-policy portfolio sampling makes use of historic data about the frequency with which customers in each category appear naturally without action modulation. As before, we denote by f̅_i the average of the frequency representations ∑_s ∈ρ_t+1_A_i(s) over historic epochs in which no portfolio was being pursued. We can then define a historic rate, given by p̅_i = f̅_i/T. To sample previously on-policy portfolios at time t, we sample each component of the portfolio's frequency representation independently from the Binomial distribution with parameters p_i + 1/T, t. The addition of 1/T to the rate was made to ensure that, for each category, we have a strictly positive probability of sampling portfolios with customers in that category. Sampling these portfolios ensures we have a good representation of value for portfolios that we would encounter if we did not adjust our pricing methodology. The target on-policy portfolio sampling method works in the same manner as the previously on-policy portfolio sampling, except we use p^*_i = f^*_i/T where f^*_i = ∑_s ∈ρ^*_A_i(s) is the target frequency. 

The last method we use is high-coverage portfolios. This method is parametrised by a slope parameter, σ∈ [0,1]. We first define p_max = (1 + σ)max( p^*, p̅ ), p_min = (1 - σ)min( p^*, p̅ ), where the maximum and minimum are taken element-wise. At time t, we consider evenly spaced points ω∈ [0,1]. For each ω point, we construct the frequency representation f_t(ω) = t( ωp_max + (1 - ω)p_min ), which interpolates between the frequency representation if customers were acquired at either the maximum rate or minimum rate. This will in general not be integer valued, so we therefore take the integer part of each element, and then add on a Bernoulli sample which is one with probability equal to the remainder term. The high-coverage portfolios thus sampled ensures a wide sampling outside the on-policy distributions. The contribution of these portfolios is crucial, as it ensures our method still has accurate assessments of portfolio values outside the on-policy distribution, providing stability at inference time. We used a value of σ = 0.9.



 §.§ Relationship to existing RL methods


Here we briefly discuss the relationship of our proposed RL algorithm to existing algorithms within the literature. Our method can be viewed as an actor-critic algorithm, in the sense that it learns both a policy π(m,k) and a value function V_ϕ(ρ,t). However, our method displays substantial differences to more conventional actor-critic methods, such as A3C <cit.>, PPO <cit.>, SAC <cit.>, and DDPG <cit.>. 

Firstly, our method is not a policy-gradient algorithm. Unlike the methods listed above, our policy is not learned by finding the gradient of a value measure (such as total return <cit.>, or a Q-function <cit.>) with respect to the policy parameters. Instead, we exploit the fact that - because the action space is one-dimensional - the relevant value measure, given in (<ref>), can be optimised directly with a non-gradient-based optimiser. We then fit our policy directly to the optimised values - a form of regression-based amortised optimisation <cit.>. This allows us to avoid problems relating to local maxima which arise in gradient-based methods. It also frees us from the necessity of having a policy which is differentiable in the parameters; indeed, in our experiments we use a Gaussian process as our policy <cit.>. In fitting the policy to a dataset of actions which maximise a value measure, our method is akin to the cross-entropy method <cit.>. 

Secondly, our value function is not a function of the full MDP state. The full MDP state is the (customer, portfolio, time-step) triple, (s,ρ,t). Instead, our value function only depends on the portfolio and the time-step, (ρ,t). This substantially reduces the dimensionality of the input space to our value function, since the number of customer features is (typically) much larger than the number of portfolio categories. From (<ref>), we know that the optimal action depends only on the next portfolio, time-step pair. This means that, for the purposes of action-selection, it suffices to learn only these values, rather than the full MDP state values (or, indeed, the value of state-action pairs <cit.>). We also note that the dependence is only through the k-value. This enables us to learn a policy which takes k directly as input, rather than the full MDP state. Importantly, this policy can be learned fully before learning any value functions (as we do in our algorithm), and does not require re-training while the value function is learned, unlike a typical actor-critic method <cit.>. 

Lastly, our method utilises a memory-reply buffer to act as a model of the arrival of new customers. However, unlike many other methods which use a memory reply buffer for value-learning <cit.>, we do not sample states randomly. Rather, (portfolio, time-step) pairs, (ρ, t), are sampled systematically by iterating t backwards in time. In iterating backwards, our method is analogous to a dynamic programming algorithm, which starts at terminal states of known value (in our case, terminal portfolios) and applies backwards induction to find the value of all states recursively. Similar to the Fitted Q-Iteration algorithm <cit.>, our method avoids the curse of dimensionality associated with exact dynamic programming in MDPs by approximating the value function at each time-step by a linear model, U. Because we are applying an approximate dynamic programming method we need only sweep through the state-space once. This is in contrast to methods which apply a Bellman optimality equation but do not recurse backwards from terminal states <cit.>, where each state must be sampled many times in order to approximate the correct value. 





  §.§.§ Other applications of our algorithm


Although we have designed our algorithm specifically for the portfolio pursuit application pursued in this paper, certain elements readily generalise to other situations not considered here. We briefly enumerate potential other uses of elements of our algorithm. 

Firstly, our algorithm depends on the fact our state-space factorises into portfolio, time-step, customer feature triples (ρ, t, s). The fact that we have a fixed horizon T, and that the time-step t is a component of the state-space, allows us to compute values by iterating backwards in time. 

the use of a simple (in our case, linear) model 






 §.§ Additional simulation details


Our simulations use a realistic agent-based model of an insurance market, mediate through a PCW <cit.>. The insurance firm implementing each method – baseline (sec:baseline method) or RL (sec:RL for PP) – competes with 5 other insurance firms (none of which are pursuing a portfolio, as in sec:standard industry method). Each customer is randomly generated by sampling features from a complex generative distribution, with realistic interdependencies between features. For example, customer income depends on the location, age, and occupation of the customer. Only a subset of features are revealed to the insurers though the PCW interface. On the basis of these features, each insurer makes the customer an offer. The customer then either selects an offer or chooses to walk away from the market. Full details of the market environment can be found in the https://github.com/EdwardJamesYoung/RL-portfolio-pursuitcode repository that accompanies this paper. 

Target portfolios are generated at random by sampling I = 5 customer features, and setting the target quantity f_i^* for those features to be twice f̅_i if f̅_i ≤ 10, and otherwise half or twice of f̅_i at random (rounding up or down respectively to ensure integer values). We have 6 burn-in epochs, during which historic data is gathered for training models. Insurers use the past 4 epochs to train models, and all burn-in epochs for computing historical frequency representations. For the first burn-in epoch, before any models are trained, insurers take actions randomly and uniformly in [1,1.2]. Following the burn-in epochs, we then have 8 testing epochs. Each epoch consists of T = 1000 steps. No re-training of any models occurs following a testing epoch, meaning the testing epochs are independent trials of the same value function V_ϕ. In the results below, we aggregate statistics across testing epochs. We set the loss-coefficient λ = 2000. The results shown in sec:results are obtained by aggregating statistics across both the 8 testing epochs and 24 independent trials, with each trial having a different randomly generated target portfolios. To ensure a fair comparison between the RL and baseline methods, we use the same set of random seeds for both methods. This means that, up until the testing epochs, both methods interact with the market identically, ensuring the same historic data to train on and target portfolio to aim towards. Thus, any differences in performance are attributable only to differences over the testing epoch. 



§ RESULTS




fig:main-results shows the results of comparing our method to the baseline averaged over 24 trials. On each trial, both methods have the same interaction history, and thus have the same training data, up until the testing epochs. They also pursue identical target portfolios. Thus, differences in performance stem only from different behaviour on the test epochs. From panels (A-B) we see that, on average, our method generates more profit than the baseline (one-sided Mann-Whitney U test, p < 0.0001; CLES = 65%; Cohen's d = 0.51, moderate effect size). Over the course of 1000 customer interactions, on average our method generates an additional 472 (or a 7% increase) in profit. Panels (C-D) show the loss between the target portfolio and the obtained portfolio at each time step. The final loss distribution of both methods is very similar (two-sided Mann-Whitney U test, p = 0.16, not significant), showing that our method has not compromised portfolio quality in order to generate additional profit. Thus we can conclude that our method brings in additional revenue while getting comparably close to the desired portfolio as the baseline. Panels (E-F) show the total reward (profit minus loss) of both methods, which is the objective that both methods are optimising. We see that our method outperforms the baseline, achieving higher reward on average over the trials (one-sided Mann-Whitney U test, p < 0.0001; CLES = 63%; Cohen's d = 0.48, moderate effect size). In Table <ref> we tabulate the average final profit, loss, and reward of both methods.  





§ DISCUSSION




 §.§ Limitations and opportunities for future work


We believe that our work presents an important first step in the application of ideas from reinforcement learning to the portfolio pursuit problem. However, our work has several limitations, with corresponding opportunities for future research. 

In our investigation, we limited the time span over which insurers pursued a particular portfolio to only T = 1000 time-steps, with one customer arriving per time-step. This is a relatively short length of time for a large insurer, who may see orders of magnitude more customers per day. Our choice to limit the number of time-steps was motivated primarily by the prohibitive computational cost of running dozens of market simulations for larger numbers of time steps. Preliminary experiments provided evidence that our method will likely continue to work for larger numbers of time-steps. However, for much larger lengths of time, on the order of months, a different approach may well be needed. Accordingly, we believe that the framework and methods we have develop above could be integrated naturally with RL methods which attempt to perform temporal abstraction, goal-conditioned policies <cit.> or the options framework <cit.>. Indeed, longer time-scale portfolio pursuit can be accomplished by chaining together shorter time-scale portfolio pursuit with the appropriately chosen intermediate target portfolios. We leave the problem of selecting such intermediate target portfolios to future research. 

In our framework, we have also made the simplifying assumption that customers do not leave the portfolio once they have entered. This is justified when the number of time-steps is small, since few customers will both accept and then cancel a contract over the course of, one day. However, for longer time-scales, it may begin to become necessary to model the exit of customers from the portfolio. In app:customers leaving we expand our analytic derivation to consider this case, and provide some preliminary suggestions for how to integrate this into our algorithm. 

Lastly, our method is model-based, with optimal actions and values computed using a conversion model which estimates the probability of customers accepting any given offer. Consequently, if the conversion model is incorrect, our bidding strategy will accordingly be sub-optimal. For example, the bidding model may underestimate the probability that a customer accepts an offer at a given price, and make offers too low. Model uncertainty could be incorporated into the algorithm by explicitly penalising the value of actions about which the insurer is uncertain. A core step in making the algorithm more robust would be allowing for online updates of the models in light of new data coming in. This is an increasingly pressing problem within insurance pricing in particular, since such markets tend to be very non-stationary, shifting rapidly as individual insurers change their pricing strategy and inflation drives changes in prices. An effective pricing strategy will need to precisely trade-off between profit and the value of the information generated by making particular offers. Although we do not address it in this paper, we expect this to be a fruitful area for future research into the application of RL to insurance pricing. 



 §.§ Conclusion


Our paper has introduced a new formalism for a problem within the insurance industry - that of portfolio pursuit. Portfolio pursuit asks the question of how to modulate the offers made to individual consumers, in light of a desired target portfolio which describes the higher-level strategy of the insurance firm. We formulated portfolio pursuit it as an MDP in sec:problem formulation. Given its formal description, in sec:methods we discussed both industry standard methodologies for addressing this problem, and gave our own reinforcement learning solution. We then showed in sec:results that our method outperforms the industry baseline, generating more profit while getting equally close to the target portfolio. It is our hope that this paper will stimulate additional research into this area, laying the groundwork for better algorithms and methodologies for portfolio pursuit. 



 §.§ Acknowledgements


EJY is supported by the UKRI Engineering and Physical Sciences Research Council Doctoral Training Program grant EP/T517847/1. Work was completed during an internship at Accenture (UK) Limited, in connection with the Turing Institute. 



 §.§ Declaration of competing interest


The authors declare no competing interests.



 §.§ Codebase


The GitHub code repository associated with this paper can be found https://github.com/EdwardJamesYoung/RL-portfolio-pursuithere. The code is licensed under Creative Commons Attribution-NonCommercial 4.0 International Public License (CC BY-NC 4.0)



 §.§ Author contributions


Problem statement was formulated jointly by EJY, AR, and JJ. Market environment designed by EJY. Mathematical derivations, algorithm design, and numerical experiments were preformed by EJY. EJY drafted the manuscript, with additional comments and feedback provided by ET, AR, and JJ. 



apalike








§ BELLMAN RECURSIONS FOR PORTFOLIO VALUES


In this appendix we expand out the Bellman equations for the portfolio and action value functions, given by (<ref>) and (<ref>) respectively. We start with the action value function. We will expand this in terms of the portfolio value function. Firstly, we denote the returns by

    G_t = ∑_k=t^T y_k (s_k, a_k) - λŁ(ρ_T+1, ρ^*).

Then we have

    Q_π(s, a, ρ, t)    _π[ G_t | ρ_t = ρ, s_t = s, a_t = a ] 
       = (y_t = 1 |s_t = s, a_t = a)[ G_t | ρ_t = ρ, s_t = s, a_t = a, y_t = 1 ] 
       + (1 - (y_t = 1 |s_t = s, a_t = a))[ G_t | ρ_t = ρ, s_t+1 = s, a_t+1 = a, y_t = 0 ] 
       = (y_t = 1 |s_t = s, a_t = a)( (s,a) + [ G_t+1 | ρ_t+1 = ρ∪{ s }]  ) 
       + (1 - (y_t = 1 |s_t = s, a_t = a))[ G_t+1 | ρ_t+1 = ρ] 
       =(y_t = 1 |s_t = s, a_t = a)( (s,a) + V_π( ρ∪{ s }, t+1 ) ) 
       + (1 -(y_t = 1 |s_t = s, a_t = a)) V_π( ρ, t+1 ) 
       = V_π(ρ, t+1) + (y_t = 1 |s_t = s, a_t = a) ( (s,a) + V_π( ρ∪{ s }, t+1 ) -  V_π( ρ, t+1 ) ).

Note that, since the profit is the offer price C(s)a minus the expected cost C(s), the profit is thus given by: (s,a) = C(s)(a - 1). We can then say that:

    Q_π(s,a,ρ,t)    = V_π(ρ, t+1) + (y_t = 1 |s_t = s, a_t = a) ( C(s)(a - 1) + V_π( ρ∪{ s }, t+1 ) -  V_π( ρ, t+1 ) ) 
       = V_π(ρ, t+1) + C(s) (y_t = 1 |s_t = s, a_t = a) ( a - k_π(s,ρ, t+1) ),

where k_π(s, ρ, t+1) is defined in (<ref>). Since the first term is a constant in a, and C(s) > 0, this shows that maximisation of Q_π over a is equivalent to maximising:

    (y_t = 1 |s_t = s, a_t = a) ( a - k_π(s,ρ, t+1) ).
 
Furthermore, note that V_π(ρ, t) = [ Q_π(s,a,ρ, t) ]. Assuming a deterministic policy π = π(s,ρ, t), this gives us the following recursion:

    V_π(ρ, t) = V_π(ρ, t+1) + [  C(s) (y_t = 1 |s_t = s, a_t = π(s,ρ, t)) ( π(s,ρ, t) - k_π(s, ρ, t+1) )  ].




§ GENERALISATION TO CUSTOMERS LEAVING


In this section we generalise the analysis is app:bellman recursion to the case where customers can leave the portfolio. 

We define the action-value function as:

    Q_π(s, a, ρ, t) _π[ G_t | ρ_t = ρ, s_t = s, a_t = a ]

We will now compute the Bellman recursion for Q_π. To do so, we introduce a portfolio transition distribution, p( ρ' | ρ ). This gives the probability of having portfolio ρ' at time t+1, given that you ended time-step t with portfolio ρ. 

    Q_π(s, a, ρ, t)    = (y_t = 1 |s_t = s, a_t = a)[ G_t | ρ_t = ρ, s_t = s, a_t = a, y_t = 1 ] 
       + (1 - (y_t = 1 |s_t = s, a_t = a))[ G_t | ρ_t = ρ, s_t+1 = s, a_t+1 = a, y_t = 0 ] 
       = (y_t = 1 |s_t = s, a_t = a)( (s,a) + ∫[ G_t+1 | ρ_t+1 = ρ' ] p(ρ' | ρ∪{s} ) dρ' ) 
       + (1 - (y_t = 1 |s_t = s, a_t = a)) ∫[ G_t+1 | ρ_t+1 = ρ' ] p(ρ' | ρ  ) dρ' 
       =(y_t = 1 |s_t = s, a_t = a)( (s,a) +∫ V_π( ρ', t+1 ) p(ρ' | ρ∪{s}) dρ' ) 
       + (1 -(y_t = 1 |s_t = s, a_t = a)) ∫ V_π( ρ', t+1 ) p(ρ' | ρ  ) dρ' 
       = ∫ V_π( ρ', t+1 ) p(ρ' | ρ  ) dρ' 
       + (y_t = 1 |s_t = s, a_t = a) ( (s,a) + ∫ V_π( ρ', t+1 ) [ p(ρ'|ρ∪{s}) - p(ρ'| ρ) ] dρ' ) 
       = [ V_π(ρ', t+1) | ρ] 
       + (y_t = 1| s_t = s, a_t = a) ( (s,a) + [ V_π(ρ', t+1) | ρ∪{ s }] - [ V_π(ρ', t+1) | ρ] )

We now use the substitution that (s,a) = C(s)(a - 1). Following the derivation in app:bellman recursion, we define: 

    k_π(s, ρ, t) = 1 - [ V_π(ρ', t) | ρ∪{ s }] - [ V_π(ρ', t) | ρ]/C(s)

Our optimal policy is then the same. Integrating over customers and actions, we obtain the following recursion for the portfolio value function V_π: 

    V_π(ρ, t) = [ V_π(ρ', t+1) | ρ] + [ C(s) (y_t=1|s_t=1,a_t = π(s,ρ,t))(π(s,ρ,t) - k_π(s, ρ, t+1)) ]
 
Given access to p(ρ'|ρ), we can easily adapt alg:training value function to use this recursion instead, by sampling portfolios from p and averaging to approximate the expectations in (<ref>) and (<ref>). This adds some computational overhead, but is not prohibitive. The transition model p(ρ'|ρ) can be learned from historic data. 

One issue we encounter when we account for the possibility of customers leaving the portfolio is that the portfolio objects become considerably more complicated. Recall from sec:RL for PP that, when customers do not leave the portfolio, we can specify portfolios only through their frequency representation, since (i) it was unimportant when a customer joined the portfolio, and (ii) the other customer features were unimportant. However, in this case, neither of this conditions are true. Firstly, the probability of any given customer leaving depends how long they have been in the portfolio. Secondly, the probability of a customer leaving after being in the portfolio for any length of time may be a function of their other features. This necessitates a more detailed and possibly high-dimensional portfolio representation. We do not explore these issues further in our paper. 



 Paper Title: A Convexification-based Outer-Approximation Method for Convex and Nonconvex MINLP.


An iterative transversal CNOT decoder
    Winfried K. Hensinger
    
=====================================



§ ABSTRACT 
The advancement of domain reduction techniques has significantly enhanced the performance of solvers in mathematical programming. This paper delves into the impact of integrating convexification and domain reduction techniques within the Outer- Approximation method. We propose a refined convexification-based Outer-Approximation method alongside a Branch-and-Bound method for both convex and nonconvex Mixed-Integer Nonlinear Programming problems. These methods have been developed and incorporated into the open-source Mixed-Integer Nonlinear Decomposition Toolbox for Pyomo-MindtPy. Comprehensive benchmark tests were conducted, validating the effectiveness and reliability of our proposed algorithms. These tests highlight the improvements achieved by incorporating convexification and domain reduction techniques into the Outer-Approximation and Branch-and-Bound methods.









§ INTRODUCTION

Mixed-integer nonlinear programming (MINLP) has broad applications in process systems engineering (PSE), including planning, scheduling, and control. It offers a powerful modeling framework that optimizes discrete and continuous variables involved in linear and nonlinear constraints. However, the combinatorial complexity, nonlinearity, and even nonconvexity lead to substantial challenges in optimizing such problems. 

Generally, MINLP can be classified as convex and nonconvex, depending on the convexity of its continuous relaxation. The algorithms for MINLP are primarily categorized into Branch-and-Bound (B&B) methods and decomposition methods <cit.>. The main idea of MINLP decomposition algorithms is to generate linear inequalities to approximate nonlinear constraints and iteratively solve the relaxed Mixed-Integer Linear Programming (MILP) main problem and Nonlinear Programming (NLP) subproblems. Decomposition methods for convex MINLP problems include the Outer-Approximation (OA) method <cit.>. This method involves solving an MILP defined by linear inequalities that relax the nonlinear constraints, known as OA cuts, and an NLP with the main problem's integer solution fixed. The first-order Taylor approximation of the nonlinear constraints defines these OA cuts. To reduce the MILP problem solution time, the Linear Programming and Nonlinear Programming-based B&B (LP/NLP-B&B) method <cit.> maintains the same B&B for the MILP, solves NLPs at the tree's integer nodes, and uses OA cuts to improve the searching bounds. Consequently, these decomposition methods are often called multi-tree and single-tree, based on their management of the MILP problem. A significant limitation of these methods is their initialization, usually given by relaxing the nonlinearity of the problem and relying on the cuts generated at each iteration to provide a better linear approximation as iterations progress.

While the OA method is effective for convex MINLPs, its limitations become apparent for nonconvex MINLP, where OA cuts do not guarantee validity in relaxations of nonlinear functions, precluding global optimality guarantees. Other relaxation techniques have been developed for nonconvex MINLPs addressing this challenge. Among these, the Auxiliary Variable Method (AVM) and McCormick relaxations are successful strategies for generating relaxations of nonconvex factorable functions <cit.>. AVM achieves this by introducing an auxiliary variable and a corresponding equality constraint for each intermediate nonlinear factor in a function, leading to computational efficiency by decomposing the function into simpler, lower-dimensional components. However, this method entails incorporating many auxiliary variables and constraints. In contrast, McCormick relaxations maintain the dimension of the original function and use a recursive approach to produce the required convex and concave relaxations effectively <cit.>.

In addition to advancements in reformulations and optimization algorithms, the performance of optimization solvers has significantly improved through domain reduction techniques. These techniques encompass bound tightening, eliminating redundant variables and constraints, and convexification <cit.>. The bound tightening techniques include Feasibility-based Bound Tightening (FBBT), Optimality-based Bound Tightening (OBBT), and Marginals-based Bound Tightening <cit.>. Domain reduction methods, including convexification cuts and bound tightening techniques, have been successful for spatial B&B methods. These tighter relaxations provide stronger dual bounds, accelerating the B&B process by facilitating node pruning and efficiently identifying optimal solutions. However, decomposition-based MINLP solvers have not yet fully harnessed the potential of domain reduction techniques.

This work investigates the efficacy of domain reduction techniques within OA methods, applicable to both convex and nonconvex MINLP problems. It is understood that domain reduction techniques can be used during the presolve stage and at each node within the B&B tree, a strategy known as the branch-and-reduce method. Similarly, in the OA method, these techniques can be employed both in the presolve phase and during solving integer-fixed NLP subproblems. However, this work focuses on the impact of domain reduction methods at the method's initialization stage.



§ SOLUTION ALGORITHM


The general form of a MINLP problem is as follows.


    MINLPmin_x,y     f(x,y) 
    
        s.t.       g_j(x,y) ≤ 0,∀ j=1,…,l 
    
            x ∈[x, x] ⊆ℝ^n, 
        y ∈{y, …, y}⊆ℤ^m,

where x and y represent continuous variables and discrete variables, respectively. Upper and lower variable bounds are determined by over- and underbars, respectively. Both the objective f(x,y) and constraints g_j (x,y) are potentially nonlinear functions.
The OA solution method for MINLP involves an iterative two-step procedure. The first step in iteration k determines the integer variables' values y^k+1 by solving problem (<ref>), defined by the OA cuts. Its optimal objective function value, encoded in variable μ, provides a dual lower bound (LB) to the original MINLP problem's optimal objective.


    OA-MILPmin_x, y, μ     μ
    
        s.t.       f(x^i,y^i)+∇ f(x^i,y^i) ^T[  x - x^i
     y - y^i ]
        ≤μ  ∀ i = 1, …, k 
          g_j(x^i,y^i)+∇ g_j(x^i,y^i) ^T[  x - x^i
     y - y^i ]
        ≤ 0   ∀ i = 1, …, k, ∀ j ∈ I_i 
    
            x ∈[x, x] ⊆ℝ^n, 
        y ∈{y, …, y}⊆ℤ^m, 
       μ∈ℝ^1.


The second step is determining the continuous variables' values x^k+1 by solving problem (<ref>) whose optimal solution yields a primal upper bound (UB) to problem (MINLP).


    NLP-Imin_x     f(x,y^k+1) 
    
        s.t.       g_j(x,y^k+1) ≤ 0   ∀ j=1,…,l 
    
            x ∈[x, x] ⊆ℝ^n.


If problem (<ref>) is infeasible, the following feasibility subproblem is solved to minimize a norm p of the constraint violations s, as a result of updating x^k+1.


    NLP-fmin_x,s      s_p
    
        s.t.       g_j(x,y^k+1) ≤ s_j   ∀ j=1,…,l 
    
            x ∈[x, x] ⊆ℝ^n.


As shown in Figure <ref>, the OA method begins by solving the relaxed NLP problem and then iteratively solves the (<ref>), (<ref>), and (<ref>) problems. The key to this process is the progressive accumulation of OA cuts, which incrementally narrows the gap between the LB and the UB. The iterations continue until LB and UB converge, culminating in the OA method reaching the optimal solution. This method is guaranteed to find the global optimal solution of convex MINLPs <cit.>.



Maintaining a single MILP tree for the LP/NLP-B&B method can be implemented using the LazyConstraint callback function through callback functions in current MILP solvers, as shown in Figure <ref>. This method initializes by solving the relaxed NLP problem as well. Then, a B&B method is used to solve problem (<ref>). Whenever an incumbent solution is found in the search tree, (<ref>) is solved, and OA cuts are added as lazy constraints to the MILP tree. This B&B process is guaranteed to terminate at the global optimal solution of convex MINLP problems <cit.>.



The OA and LP/NLP-B&B methods generate tight cuts at the boundary of the nonlinear feasible region defined by the original problem constraints by incurring the cost of solving NLP subproblems. If the (<ref>) subproblem is infeasible, (<ref>) is solved to find the point closest to the feasible region to generate the tightest possible cuts. Since the linear inequality constraints are accumulated iteratively, the main problem in the early iterations is that a poor approximation of the original MINLP model is obtained. Consequently, the integer combination provided by the main problem tends to yield an infeasible (<ref>) subproblem, and no primal bound can be obtained.

In this work, we apply the domain reduction methods to the initialization stage of the OA method to resolve this issue. Eq. (<ref>) and (<ref>) shows the tightened bounds of the discrete and continuous variables, [x^',x^'] and {y^',…,y^'}, respectively. Eq. (<ref>) corresponds to the convexification linear cuts generated by the auxiliary variable method or reformulations, where z are auxiliary variables. Since all Eq. (<ref>) - (<ref>) are applied at the initialization stage, they are valid for (<ref>), (<ref>), and (<ref>) problems.


    x∈[x^',x^']∈[x,x]⊆ℝ^n



    y∈{y^',…,y^'}∈{y,…,y}⊆ℤ^m



    Ax + By + Cz ≤ b


Considering that the convexification cuts are the relaxation of nonlinear constraints, they are redundant in the NLP subproblems where the original nonlinear constraints are included. Therefore, we denote the (<ref>) and (<ref>) problems with convexification cuts and tightened bounds as complete-scale NLP problems. The (<ref>) and (<ref>) problems with only tightened bounds are denoted reduced-scale NLP problems.

This work also considers modified alternatives of the OA and LP/NLP-B&B methods to guarantee global optimality for nonconvex MINLP problems. Several modifications have been introduced to provide such global optimality guarantees, denoted as global OA (GOA) and global LP/NLP-B&B (GLP/NLP-B&B). First, instead of adding OA cuts, the affine underestimators and overstimators are generated based on the convex and concave McCormick relaxations using subgradient propagation <cit.>. Second, to guarantee the algorithm's convergence, no-good cuts are generated to cut off the explored integer combinations and prevent the algorithm from repeatedly cycling through the same combinations. These enhancements enable the global algorithms to converge to the global optimum of nonconvex MINLP problems if the NLP subproblems are solved to global optimality <cit.>. Furthermore, we integrate the domain reduction techniques in GOA and GLP/NLP-B&B and investigate their effect on its performance.



§ BENCHMARKING AND RESULTS

To evaluate the impact of domain reduction techniques, we use test instances from the problem library MINLPLib <cit.>. 434 convex instances and 181 nonconvex instances are selected, adhering to the criteria that each instance must have at least one discrete variable and at least one continuous variable. For clarity in our analysis, we use (r) and (c) to distinguish between the reduced-scale and complete-scale NLP subproblems used in the convexification-based OA and LP/NLP-B&B methods. Moreover, we indicate the results with convexification with the prefix (C-) in the following results.

The benchmark implementation is based on the Mixed-integer nonlinear decomposition toolbox for Pyomo-MindtPy <cit.>. We use both the multi-tree and single-tree implementation of OA and GOA strategy, maintaining their default configurations as a baseline. Moreover, a special version of BARON 19.4.4 is used to tighten the bounds and generate convexification cuts. Option dolocal is set to 0, and numloc is set to 0 to turn off local search during upper bounding and preprocessing in BARON. All range reduction and relaxation options are retained at their default settings. Nonlinear FBBT, OBBT, marginals-based, and linear-feasibility-based bound tightening are applied. Outer approximations of convex univariate functions and cutting planes are also applied. For the termination criteria of the algorithm, we set the absolute tolerances ϵ_abs=10^-5 and relative tolerances ϵ_rel=10^-3, along with a time limit of 900s. We use GUROBI 10.0.0 as the MILP solver, IPOPTH 3.14 as the NLP solver for convex instances, and BARON 23.6.22 as the NLP solver for nonconvex instances. All tests ran on a Linux cluster with 48 AMD EPYC 7643 2.3GHz CPUs and 1 TB RAM, with each test restricted to using only a single thread.

The time and iteration performance profiles of the convex instances are presented in Figure <ref>. For the LP/NLP-B&B method, the number of iterations refers to the number of (<ref>) subproblems solved. Overall, the convexification-based OA and LP/NLP-B&B methods utilizing reduced-scale NLP subproblems outperform the other solver alternatives regarding solution time. Regarding the number of iterations, both the OA and LP/NLP-B&B methods benefit from the convexification cuts and the bound tightening techniques. Interestingly, the choice between complete-scale and reduced-scale NLP subproblems does not significantly impact iteration performance.



However, it is noteworthy that the convexification-based LP/NLP-B&B method with complete-scale NLP subproblems underperforms in time performance compared to the standard LP/NLP-B&B method. This coincides with our previous statement that the convexification cuts are redundant and increase the computational complexity of (<ref>) and (<ref>) subproblems. For instances that can be solved within one second, both the standard OA method and the LP/NLP B&B method are more efficient than their convexified counterparts, as both bound tightening and convexification cuts entail additional processing time. Nonetheless, the bound tightening and convexification cuts generally enhance the performance of both the OA and LP/NLP B&B methods.

The benchmark results of nonconvex instances are presented in Figure <ref>. Similar to the convex cases, the convexification-based OA and LP/NLP B&B methods outperform the others in solution time and number of iterations. This consistent performance across convex and nonconvex MINLP problems demonstrates the effectiveness of the proposed convexification-based OA and LP/NLP B&B methods.





§ ACKNOWLEDGEMENTS

D.B.N. was supported by the NASA Academic Mission Services, Contract No. NNA16BD14C. D.B.N. and Z.P. acknowledge the support of the startup grant of the Davidson School of Chemical Engineering at Purdue University. The authors acknowledge the support of Prof. Nick Sahnidis in providing a special version of BARON that allowed this research. The authors thank GAMS Development Copr. for providing a license to their software and various optimization solvers used for this manuscript.













































































unsrtnat
  





























 Paper Title: Randomized Controlled Trials of Service Interventions: The Impact of Capacity Constraints.

 












[=]=Endnotes
^
0pt0pt=1.75em
  . 








-






 










positioning,arrows.meta,,bending,arrows,automata
























s.t. 
C>c<L>l<R>r<[, ](),a,24pt and    
  treenode/.style = shape=rectangle,
                     draw, align=center,
  root/.style     = treenode, font=,
  env/.style      = treenode, font=,
  dummy/.style    = circle,draw[figure]font=scriptsizeRCT Design for Service InterventionsRandomized Controlled Trials of Service Interventions: The Impact of Capacity ConstraintsJustin J. BoutilierTelfer School of Management, University of Ottawa  boutilier@telfer.uottawa.eduJónas Oddur JónassonMIT Sloan School of Management, joj@mit.eduHannah LiColumbia Business School, hannah.li@columbia.eduErez YoeliKeheala, eyoeli@keheala.comBoutilier, Jónasson, Li, YoeliRandomized controlled trials (RCTs), or experiments, are the gold standard for intervention evaluation. However, the main appeal of RCTs—the clean identification of causal effects—can be compromised by interference, when one subject's treatment assignment can influence another subject's behavior or outcomes. In this paper, we formalise and study a type of interference stemming from the operational implementation of a subclass of interventions we term Service Interventions (SIs): interventions that include an on-demand service component provided by a costly and limited resource (e.g., healthcare providers or teachers). We show that capacity constraints, induced by staffing and recruitment decisions made by the experimenter, can mediate the effect size of the intervention.

We first show that in such a system, the capacity constraints induce dependencies across experiment subjects, where an individual may need to wait before receiving the intervention. By modeling these dependencies using a queueing system, we show how increasing the number of subjects without increasing the capacity of the system can lead to a nonlinear decrease in the treatment effect size. 
This has implications for conventional power analysis and recruitment strategies: increasing the sample size of an RCT without appropriately expanding capacity can decrease the study’s power. To address this issue, we propose a method to jointly select the system capacity and number of users using the square root staffing rule from queueing theory. We show how incorporating knowledge of the queueing structure can help an experimenter reduce the amount of capacity and number of subjects required while still maintaining high power. In addition, our analysis of congestion-driven interference provides one concrete mechanism to explain why similar protocols can result in different RCT outcomes and why promising interventions at the RCT stage may not perform well at scale.



Towards Variable-Length In-Network Caching
    
Gyuyeong Kim 
 Sungshin Women's University

    
================================================




§ INTRODUCTION




Randomized Controlled Trials (RCTs) are widely accepted as the gold standard for impact evaluation. In their simplest form, a researcher first recruits a pool of subjects to enroll. Then, the researcher randomly assigns the subjects to a treatment group and a control group, administers an intervention to the treatment group, and compares the average outcomes across the two groups. 
Usually, choosing to enroll more subjects in a trial will result in the experiment having increased statistical power (defined as the likelihood of the RCT detecting a difference between the groups if there is one). Motivated by a sequence of RCTs in digital health, we study RCTs of interventions with a capacity constrained structure—we label them service interventions, see below—where recruiting more subjects does not increase statistical power. In these settings, we find that the capacity constraints mediate the effect size of the intervention and the resulting statistical power of the RCT. Moreover, the experimenter can leverage this non-linear relationship to optimally select the system capacity and the number of recruited subjects in a way that ensures high power and low cost experiments.




 §.§ Motivation


We define service interventions as interventions that have an on-demand service component, where the service is provided by a capacity constrained resource (e.g., human servers). 
In other words, subjects in the treatment group are enrolled in a program that involves a stochastic component where the subject periodically (e.g., depending on their behavior, disease status, or engagement) becomes eligible for a service. Subjects in the control group do not receive the intervention.

One example of such service interventions are treatment adherence support programs, in which patients on a medical treatment 
become eligible for outreach from a human support sponsor when they stop adhering to the treatment plan. 
One such program, and our running example for this paper,
is Keheala, an organization that provides digital services to support treatment adherence among tuberculosis patients in Kenya. The organization has developed a mobile phone interface that provides information about the importance of treatment adherence and sends automated reminders to patients about adhering to their medication schedule. The enrolled patients are required to self-verify their treatment adherence daily (i.e., verify that they are in the desired state (adherence) as opposed to the undesired state (non-adherence)) and if they fail to do so they become eligible for support sponsor outreach (i.e., the service component of the intervention). The intervention has been evaluated in a randomized controlled trial <cit.> that demonstrated a significant reduction in treatment failure among the treated population, relative to a control group receiving the standard of care. 
Interestingly, despite the very strong performance of the intervention in the experiment, secondary analysis of the trial data has demonstrated that due to operational factors, the service component was not always available to the eligible patients as soon as they became eligible <cit.>. To the extent that this variation was generated by congestion effects, it highlights the issue at the core of this paper. 

The second example is in the context of medication adherence among myocardial infarction (also known as heart attack) patients. <cit.> describe an RCT of an intervention in which myocardial infarction survivors receive a combination of electronic pill bottles (for adherence monitoring), lottery incentives, automated reminders, as well as access to social work resources and a staff engagement advisor (the service component). The trial concludes that the compound intervention has no impact on adherence or hospital readmission. In this case, secondary analysis has demonstrated that operational factors (including limited capacity) affected the timing of the service component for patients who entered the non-adherent state and that patients who received service quickly were more likely to re-enter the desired state <cit.>. 

The third example is a sequence of RCTs evaluating a coach-enhanced behavioral health mobile phone application, Noom, for various use cases, including people with pancreatic cancer <cit.>, metabolic abnormalities <cit.> and prediabetes <cit.>. In all cases, a part of the mobile phone application is a coach-user messaging interface, where some of the communication is planned and scheduled whereas some of the communication is on-demand (i.e., the user sends a message and the coach must reply). In fact, the stochasticity in the demand for coaching advice is acknowledged as part of the trial results, with <cit.> stating that the “default frequency of personalized coaching was set to 3 times per week; however, the actual frequency varied depending on the user’s participation rate.” 

All of the interventions described above have the characteristics that motivate our study. To summarize: the objective is to keep subjects in a desired state but periodically and randomly they stop the desired behavior and become eligible for service that is provided by costly human servers. While we do not comment on the magnitude of the empirical estimates reported in any of these specific studies, our point is that in any RCT with this service intervention structure, the effect sizes and the statistical power are conditional on the service level deployed during the trial. Yet, to the best of our knowledge (as far as we can gather based on the trial protocols) the decision of what capacity to staff the platform with is not an explicit part of trial planning. As a result, it is possible for an experimenter to conclude that an intervention is ineffective (based on the trial outcome) as a result of the operational implementation (i.e., service levels) rather than the intervention itself.






 §.§ Contributions

In the language of experimental design, the above structure results in a specific type of interference (in which one subject's behavior or treatment status can affect the outcome of other subjects) through capacity constraints, and these capacity constraints mediate the effect size of the intervention. 
In other words, an RCT of a service intervention can lead to a range of impact estimates—depending on the capacity devoted to the service component and, in turn, the service level experienced by the subjects of the treatment group. At one extreme, when there is ample capacity and the utilization level of the service component is low, one would expect the average impact of the intervention to be large since subjects would have direct access to the service when they become eligible. In most cases, this would correspond to the intended protocol of the intervention as subjects would not experience delays (and hence behave independently from each other, without interference). At the other extreme, when very little capacity is devoted to the service component, delays would be long and one would expect the average impact of the intervention to be lowered. Further, across all service levels, the service component affects the variability in outcomes of the treatment group. The combination of these effects of service level on the mean and variance of outcomes makes trial planning and power analysis non-trivial. 


In this paper, we formalize the mechanism for interference due to capacity constraints and describe its implications for trial design. We focus on the experimenter's decision of how many treatment and control subjects to recruit into the trial, N_1 and N_0, respectively, and how many servers M_1 
to staff the treatment intervention with. We focus on the setting where N_1=N_0.

Our first contribution is a modeling one (<ref>). 
We start by modeling the behavior of individual subjects as a stochastic 
process with two states—the desired state (e.g., the subject is adhering to a medication regime, exercising, or following a diet) and the undesired state (i.e., the subject has stopped the desired activity). Once in the undesired state, subjects become eligible for the service, which increases their probability of transitioning back to the desired state.
The typical way to analyze these experiments 
is to think about subjects as independent objects. However, in order to capture the congestion interference of interest, we model the aggregate behavior of subjects of the service intervention RCT as a closed queueing system with a fixed number of subjects transitioning in and out of the queue. In other words, a subject who becomes eligible for service joins the queue to be attended to by a server and a subject who re-enters the desired state leaves the queue. The role of the service system is to maximize the amount of time subjects spend in the desired state. Finally, we turn our focus to experimentation with such systems and formalize the trial planning task in terms of the experimental metrics of interest and the key decisions of how many servers M_1 to hire and how many subjects N_1+N_0 to recruit in order to maximize the statistical power of the RCT.

Our second contribution is a descriptive one (<ref> and <ref>). Equipped with the modeling framework described above, we focus our analysis on characterizing the two quantities that contribute to the statistical power of experiments—the effect size and its variance. In our closed queueing framework, maximizing the amount of time subjects spend in the desired state is equivalent to minimizing the amount of time they spent in the queue for service. As a result, the main objective of our analysis is to characterize the average queue length and its variance, as a function of the number of users and servers in the system. We first derive those quantities explicitly, effectively establishing the observation from above, that differences in service levels (determined jointly by the number of subjects and servers) in these types of experiments correspond to different levels of treatment effects. To generate analytical insights, we also obtain approximations of the effect size using a mean field limit. 
Using these approximations, we establish that when the number of servers M_1 is fixed, the effect size is diminishing in the number of enrolled subjects, and in particular in the number of treatment subjects N_1, but only when N_1 is above a given threshold.
Furthermore, we demonstrate that the statistical power of a trial (for a given number of servers) is first increasing and then decreasing in the number of subjects. This has implications for trial planning and recruitment, as it contradicts the traditional wisdom that recruiting more subjects leads to better statistical power.  

Our third contribution is a prescriptive one (<ref>). Given the non-monotonicity of statistical power as a function of enrolled subjects, the key question for trial planning is how many subjects to recruit and how many servers to hire. We conduct numerical experiments contrasting three approaches to determine M_1, N_1 and N_0 of an RCT following a small-scale pilot; (1) A naïve policy that assumes that the treatment effect does not depend on service levels and that variance decreases linearly in N_1+N_0; (2) A proportional policy that chooses the same N_1+N_0 as the naïve policy and increases M_1 proportionally; and (3) a queueing informed Square Root Policy that aims to achieve a high service level. Among other things, our experiments demonstrate that following a pilot study with low service levels, the desired power for a full scale RCT can either be achieved by scaling up both M_1, N_1, and N_0 while maintaining the same low service level, or by 
increasing service levels through a higher M_1. In some practical cases, this might result in an equally powered experiment that requires much fewer participants. 


A main objective of this work is to help experimenters avoid the mistake of concluding a protocol is ineffective following RCT experimentation, simply because by enrolling a high number of subjects they believed the experiment to be high powered. This is a particularly relevant message for efficacy trials (as opposed to effectiveness trials) since their purpose is to evaluate whether a protocol or intervention has an impact under ideal circumstances. 
Descriptively, our paper serves as a reminder that, at a minimum, the outcomes of such service intervention RCTs should be interpreted as conditional on the implemented service level. In other words, the reason that such a trial shows small or no effect can either be attributed to the protocol itself not being effective (the usual interpretation) or the service level being insufficient. Therefore, papers describing RCT outcomes for service interventions should report operational outcomes describing empirical service levels along with all impact estimates. Prescriptively, we provide the experimenter with a framework to jointly optimize the size of the trial (i.e., the number of subjects) and its service level (i.e., the number of servers) in a queueing-informed manner, so as to maximize the probability of detecting a difference between a treatment group and a control group. 













§ LITERATURE

Our work is related to three distinct streams of literature. First, we contribute to the growing methodological toolkit of experimental designers. We give a brief summary of this stream of work in <ref>. Second, our work is, at least in part, motivated by prior trials of service interventions, which usually do not report on capacity planning as part of their trial preparation or empirical service levels as part of their outcome measures. We provide some examples in <ref>. Finally, our methodology draws on prior results from the service operations and queueing theory literature. We highlight the main connections to our work in <ref>.




 §.§ Interference in Experimentation

The capacity constraints induce what is known as interference in the experiment, where the outcome of one individual depends on the treatment assignments of other individuals <cit.>. It is well documented that interference leads to biased estimates of the treatment effect in a number of settings, including social networks (e.g., <cit.>) and marketplaces <cit.>. Our setting is more closely related to interference in marketplaces, 
where supply constraints (analogous to the capacity constraints in service interventions) and demand constraints create dependencies across units <cit.>. However, the service interventions introduce a queueing structure to the interference that does not arise in marketplaces. Further, the papers on marketplaces generally focus on estimating the treatment effect, whereas this paper also studies the impact of interference on statistical power. 

The most closely related paper is <cit.> which also studies experimentation in queueing systems and similarly illustrates how incorporating knowledge of the queueing system can lead to more efficient estimators. 
The problem motivation differs, where <cit.> focuses on experimenting on an existing platform infrastructure and our work studies how to set up the platform infrastructure to test a new intervention in development. 
More specifically, <cit.>
focuses considers a queueing setting with a fixed capacity and raw user arrival rate and studies interventions that change how likely a user joins the queue, conditional on seeing the queue at a certain length.  
In contrast, our work focuses on the problem of developing a new protocol and the recruitment decision of how many servers and subjects to recruit for the trial. This decision is necessary when developing a new intervention (for example, in <cit.>) and is separate from the problem in <cit.> which studies a platform with a fixed size.





 §.§ Trials of Service Interventions

It is beyond the scope of this work to give a comprehensive account of all RCTs that had a service component. As examples of such experiments, we refer to the tuberculosis treatment adherence platform Keheala <cit.>, the heart attack medication support program HeartStrong <cit.>, and the coach-enhanced behavioral health app Noom <cit.>.
All of these interventions have been evaluated in RCTs and (at least partly) rely on the type of service intervention component that motivates our work. In particular, the objective for all of them is to keep subjects in a desired state and, when the subject leaves the desired state, they become eligible for service that is provided by costly human servers.

In addition to the primary analysis of the data collected in the RCTs designed to evaluate the aforementioned behavioral health programs, secondary analysis of this data has provided additional insight. As an example, taking a closer look at the HeartStrong data, <cit.> find that patients do seem to respond to connected healthcare interventions, particularly those that involve personalized feedback and when interventions are escalated quickly and consistently. They also leverage the trial data to generate a risk-scoring method to target non-adherent patients. 
Similarly, secondary analysis of the Keheala data suggests that the operational efficiency of such behavioral health interventions can be greatly improved through machine learning enabled personalization <cit.>. Furthermore—focusing on ultimate health outcome—it has been established that Keheala's effect was heterogeneous across patient types, with the highest impact on the patient types that were most likely to have poor health outcomes in the absence of behavioral support <cit.>. 











 §.§ Queueing theory

A key step in our analysis is to model the congestion interference of one user's behavior on other user's experience using a queueing theory model. 
Most related to our work are a sequence of papers focusing on staffing in closed queueing systems <cit.>. Similar to our setting they consider a finite set of users that periodically transition into needing service that is provided by a fixed set of servers. In <cit.>, the objective is to develop staffing policies for these types of membership services, based on many server theorems that correspond to different staffing regimes. Importantly, they derive a staffing policy that minimizes capacity investment while ensuring a certain service level. <cit.> also use many server asymptotic results to generate staffing policies but for the specific case of nursing teams. Our setting is also related to the work of <cit.>, who study an infinite source queue with balking and reneging, where balking refers to users choosing not to join the queue if the queue is too long. They consider a setting where the probability of joining a queue, upon arrival, is proportional to the queue length. This process, though inspired by a different setting, has the exact same transition probabilities as our setup. Our model is also related to the Machine Interference Problem (see <cit.> for a survey on these problems), where machines are operating until they break, after which they require service from a provider to be fixed. Unlike a typical MIP problem, our setting has reneging, or machines (subjects) that can recover without the help of a server.  

While some of our results (see <ref>) are related to the analysis presented in this literature, our objective is completely different. This literature has focused on generating staffing policies whereas we focus on the implications of staffing on the statistical power of RCTs of service interventions. 






§ MODELING FRAMEWORK

In this section, we formalize our problem setting. We gradually establish the structure of service intervention experiments by first introducing a model of individual subject behavior both with and without the intervention (<ref>) as well as the service system they interact with (<ref>). We then define the outcome metrics of interest (<ref>). Finally, we adapt these models to analyze randomized controlled trials of service interventions (<ref>). 






 §.§ Individual subject behavior

The defining feature of subjects in our setting is that they can be described by a two-state stochastic process. 
We let 𝒰 denote the set of subjects or users (with |𝒰|=N) and 𝒯=ℝ_+ denote time. Then the state of user u∈𝒰 at time t∈𝒯 is denoted by x(u,t) ∈{0,1}. Here, state 0 represents a desirable state (from the perspective of both the servers and the users), while state 1 represents an undesirable state. For example, in a medication adherence setting, a desirable state is one where the user has taken their medication on a given day, or in a customer service setting, a desirable state is one where the user does not have any issues.



The transition dynamics of a given subject depend on whether or not they receive intervention from the system. In the absence of intervention, they transition from the desirable state 0 to the undesirable state 1 with rate λ and back from state 1 to state 0 with rate τ. In the presence of intervention, each server has a service rate of μ and so a user (assuming there is a server available immediately) transitions from the undesirable state 1 to the desirable state 0 with rate τ + pμ, where p represents the probability that a user transitions to the desirable state after receiving service while 1-p represents the probability that the user does not transition, and thus remains in the queue for intervention.
For our theoretical analysis, we assume subjects are homogeneous in terms of their state transition rates.



 §.§ Queueing model of service intervention system

One way to analyze the performance of the types of systems we have described is by analyzing the individual stochastic processes, with intricate dependencies across the users. However, given that the type of interference we are interested in stems from congestion, a better approach may be to consider the service intervention as a closed queuing system (i.e., with a fixed number of subjects transitioning in and out of the queue).
The system aims to serve users in state 1 so that they transition to (and ideally remain in) state 0. In other words, if 
x(u,t)=0 then user u does not require service but if 
x(u,t)=1 
then user u is eligible for service and joins the queue.


We define a Markov chain to describe the closed queueing system in a setting with M servers and N users (we describe how to apply this model to the study of experimentation in Section <ref>). We denote an individual's state at time t with x_M,N(u,t), indexed by M and N.  The state of the system at time t is the number of users in the undesirable state. Specifically, let X_M,N(t) ∈{0, …, N} such that X_M,N(t) = ∑_u x_M,N(u,t), where M is the number of servers and N is the number of users.  This quantity X_M,N(t) corresponds to the total number of users in the undesirable state 1 at time t, or in other words the number of people either waiting in the queue or being served at time t (for a system with M servers and N users). This notation explicitly references the fact that this quantity depends on both M and N.

The dynamics of the queueing Markov chain are governed by a transition rate matrix 𝒬, where 𝒬_M,N(i,j) represents the rate that the system moves from state X_M,N(t)=i to state X_M,N(t)=j. On the one hand, there is a rate of users entering the queue, i.e., 𝒬_M,N(i,i+1). If there are i users in the queue, then there are N-i users in a state 0 and so the rate at which users enter the queue is (N-i)λ. On the other hand, the rate of users leaving the queue depends on the congestion level. Each server has a service rate μ and so they can serve the users at a rate of min{iμ, Mμ} (limited by the number of users in the queue and number of servers). 
If the number of users in the queue at time t is less than the number of servers (i.e., X_M,N(t)=i≤ M), then users transition out of the queue with a transition rate of ipμ + iτ. If the number of users in the queue at time t exceeds the number of servers (i.e., X_M,N(t)=i>M) then users transition out of the queue with rate Mpμ + iτ. If the user remains in the undesired state, despite being served, the user rejoins the queue. Summarizing, we have:

    Q_M,N(i, i+1)    = (N-i) λ,     for 0≤ i<N, 
    
        Q_M,N(i, i-1)    = 
        
        iμ p + iτ,    if i ≤ M,
    
        M μ p + iτ,    if M < i ≤ N.

The rate of all other transitions Q_M,N(i, j) for i≠ j is 0. Finally, the quantity 𝒬_M,N(i,i) is defined so that the sum of each row is 0. The above model captures congestion due to the interference of interest because the length of time between when a user transitions into the undesired state and when they are served by the service intervention depends on the number of other users who also need to be served. 

We note that this Markov chain is ergodic and has a unique stationary distribution, which we denote π_M,N, where π_M,N(k) denotes the steady state probability of there being k users in the undesirable state. This notation captures the fact that the steady state distribution depends on the number of users N and servers M in the system. The steady state is characterized by the following equations:

    π_M,N(j) =
        π_M,N(j-1) ·(N-j+1)λ/jμ p + jτ   for  1≤ j ≤ M
    π_M,N(j-1) ·(N-j+1)λ/Mμ p + jτ   for  j≥ M.

and ∑_j=0^Nπ_M,N(j)=1. 

We also define the steady state expected queue lengthK_M,N a standard queueing metric, given by

    K_M,N   = ∑_j=1^N j ·π_M,N(j) 
       = π_0 ·∑_k=1^m k ·N!  λ^j/k!  k!  (μ p + τ)^k
        + π_0 ·∑_k=m+1^N k ·N!  λ^j/j! M!  (μ p + τ)^M ∏_i=M+1^j(Mμ p +i τ)


In our setting, K_M,N corresponds to the expected number of users in the undesirable state. 




  §.§.§ Queueing system regimes and offered load.


To illustrate the impact of the queueing system structure on service interventions, we begin by introducing the concepts of Efficiency-Driven (ED), Quality-Driven (QD), and Quality-and-Efficiency-Driven (QED) regimes from queueing theory. These regimes will be instrumental in interpreting the outcomes of the experiments. We provide a brief introduction here; for more detail, see <cit.> for an introduction in general queues and <cit.> for discussion specific to closed queueing systems. 

A key parameter in the subsequent analysis is the critical ratior, defined as

    r = λ/λ + τ + μ p.

Then rN is the offered load, which can be interpreted as the expected number of users in the undesirable state if the system had an infinite number of servers, or, in other words, if a server were available as soon as a user enters the undesirable state.

The relationship between the offered load and the number of servers M determines which regime the queue operates in. In the Efficiency-Driven (ED) regime, the number of servers is less than the offered load (M ≪ rN) and servers are efficiently used with little idle time, at the cost of longer wait times for users. 
In the Quality-Driven (QD) regime, the number of servers is larger than the offered load (M ≫ rN) and the probability that a user must wait to be served is low, at the cost of maintaining more servers. The Quality-Efficiency-Driven (QED) regime refers to a regime that balances both quality and efficiency, where the capacity is chosen such that it is approximately equal to the offered load (M ≈ rN), but with an additional “buffer” to account for randomness in user arrival and service times. A simple heuristic to staff in the QED regime is use the Square Root Staffing rule, which chooses M = rN + γ√(N) for some constant γ. We will utilize this staffing policy and the queueing regimes in the later analysis.











 §.§ Metric of interest

The intervention aims to increase the average time that users spend in the desired state, or equivalently, reduce the average time spent in the undesirable state. 
Starting with a single user u, define Y(u, T) to be the proportion of time spent in the undesirable state over a finite horizon [0,T]:
Y_M,N(u, T) = 1/T·∫_t=0^T x_M,N(u,t)dt.
The proportion of time spent in the desirable state is 1-Y_M,N(u,T). 
The intervention then aims to reduce the average time that all N users spend in the undesirable state. 

    1/N·∑_u
         Y_M,N(u,T).


To focus on the long run behavior, we consider the steady state proportion of time user u spends in the undesirable state, denoted _π_M,N[ x_M,N(u,t) ], where the expectation is taken over the stationary distribution. Since the system is an ergodic Markov chain, it follows that 

    lim_T→∞
         Y_M,N(u,T) = _π_M,N[ x_M,N(u,t) ], and
    lim_T→∞1/N·∑_u
         Y_M,N(u,T) 
         = 1/N∑_u_π_M,N[ x_M,N(u,t) ] 
         = _π_M,N[ x_M,N(u,t) ],

where the the second equality in Equation (<ref>) comes from the fact that users are homogeneous in the current setting.












The left hand side of Equation (<ref>) involves a sum of dependent random variables
where one subject's time spent in the undesirable state can depend on whether the provider was busy serving another subject. Further, each random variable Y_M,N(u,T) is an average of the subject's state up until time T, and the subject's state at time t can affect their state at some later time t'.

Nevertheless, we are able to study this metric by connecting the outcomes of individual users to metrics of the queueing system. The following lemma shows that the average of the individual quantities Y_M,N(u,T) is equivalent to the time average of the queue length, a well studied quantity in queueing theory. While straightforward, this observation enables us to leverage queueing theory results in order to study these service intervention experiments. 


Let X_M,N(t) denote the number of people being served or waiting to be served in the queue. 
Then the proportion of time that the user spends in the undesirable state, averaged across users, can be expressed as
    
    1/N·∑_u Y_M,N(u,T) 
            = 1/NT·∫_t=0^T X_M,N(t) dt

The long run (steady state) proportion of time the user spends in the undesirable state can be expressed as
    
    _π_M,N[ x_M,N(u, t) ] =  K_M, N/N.
Proof of Lemma <ref>.
For the equation for finite T,

    1/N·∑_u Y_M,N(u,T) 
           = 1/N∑_u 1/T∫_t=0^T x_M,N(u,t)dt
       = 1/NT∫_t=0^T ∑_u x_M,N(u,T) dt
       = 1/NT·∫_t=0^T X_M,N(t) dt

The equation for the steady state quantity follows from taking the limit on both sides of the above expression, since the Markov chain is ergodic.



 §.§ Experimentation

We now start considering impact evaluation for the kind of system we have described above. In particular, we will describe the control group, the treatment group, the treatment effect and estimator, and how the power of such an experiment depends on both the number of users enrolled and the number of servers staffing the service component. 





  §.§.§ Recruitment decisions for users and servers
 The experimenter recruits N subjects, which we call users, into the experiment. These users are then randomized into the treatment group, which receives the service intervention, and the control group, which does not. For each user u, we let Z_u indicate whether the user is assigned to the treatment group (Z_u=1) or the control group (Z_u=0).Let N_1 and N_0 denote the number of treatment and control subjects, respectively. We focus on the setting where there are the same number of subjects in the treatment and control group N_1 = N_0 = N/2, though all of the analysis can extend to the general setting. 

The experimenter hires M_1 servers to provide the service intervention, where the subscript denotes that the service intervention is provided only to the subjects in the treatment group. These servers correspond to the individuals or resources that are providing the costly, on-demand interactions in the service intervention. In the 
context of the medication adherence programs in <cit.> and <cit.>, a server corresponds to the provider who reaches out patients who are not adhering to their medication schedule. Servers may also correspond to healthcare providers in medical experiments or teachers in education experiments.




  §.§.§ Behavior of treatment and control groups.

Note that the treatment group and the control group form two distinct stochastic systems that can be analyzed separately. In other words, users in the treatment group do not affect users in the control group, and vice versa. However, users in the treatment group do affect each other.

In the notation of the queueing system defined in <ref>, the treatment group evolves according to Q_M_1,N_1, where the steady state distribution of the number of users in the queue is captured by π_M_1, N_1, and the steady state proportion of time each user spends in the undesirable state is _π_M_1,N_1[x_M_1,N_1(u,t) ].
Each of these expressions depends on M_1 and N_1. 


In contrast, users in the control group move from the undesired state to the desired state according to the dynamics given in Figure <ref>, without any service from system. We can express the evolution of the control group with the transition rate matrix Q_0,N_0 with steady state distribution π_0, N_0 or, equivalently, as N_0 separate systems, each consisting of a single individual evolving according to Q_0,1 and with steady state distribution π_0,1. The steady state proportion of time that each user spends in the undesirable state is then _π_0,1 [x_0,1(u,t) ], which does not depend on N_0.  







  §.§.§ Treatment effect and estimators.

We define a class of treatment effects
that measure the long term effect of the intervention on the time users spend in the desirable state, the metric of interest defined in <ref>.
Importantly, in the context of service interventions, this effect depends on the number of users N_1 and servers M_1.

Given an experiment with M_1, N_1, N_0, we define the steady state treatment effectθ^ss(M_1,N_1) as the difference between the treatment and control group in the proportion of time spent in the desirable state.

    θ^ss(M_1, N_1) 
           = [ 1 - _π_M_1, N_1[x_M_1,N_1(u,t)] ] - [ 1 - E_π_0,N_0[x_0,N_0(u,t)]] 
       = [ 1 - K_M_1,N_1/N_1 ] - _π_0,1 [x_0,1(u,t) ].

Note that this treatment effect depends on M_1 and N_1, but not N_0. 









A natural estimator for the treatment effect is the difference in means estimator that compares the empirical group mean verification of the treatment and control group. Given an experiment with M_1 servers for the treatment group, N_1 users in treatment, N_0 users in control, and a time horizon of length T, the time average estimatorθ̂(M_1, N_1, N_0, T) is



    θ̂(M_1, N_1, N_0, T) 
           = [    ∑_u: Z_u=11 - Y_M_1,N_1(u,T)/N_1] 
        -  [ 
        ∑_u: Z_u=0 1- Y_0, N_0(u,T) /N_0]
        
       = [ 1 -  1/T·∫_t=0^T X_M_1, N_1(t) dt/N_1] 
        -  [ 1 - 
        1/T·∫_t=0^T X_0, N_0(t) dt /N_0].

Equation (<ref>) shows two ways of expressing the estimator. The first line expresses the estimator as the difference between averages of individual outcomes. The second line expresses the estimator as a queueing system metric, with respect to the queue length for the treatment and control systems, X_M_1,N_1(t) and X_0, N_0(t), using Lemma <ref>.








  §.§.§ Hypothesis testing and power.

The experimenter conducts a standard hypothesis test to determine whether observed differences between the treatment and control group are statistically significant. Let α∈ (0,1) denote the significance level of the test. 

The experimenter then conducts a test with null hypothesis H_0: θ^ss(M_1,N_1) = 0 and alternative hypothesis θ^ss(M_1,N_1)>0. 
If we assume that the variance of the estimator is known, then the test statistic under the null hypothesis is 

    Z(M_1, N_1, N_0,T) = 
        θ̂(M_1, N_1, N_0, T) - 0/√((θ̂(M_1, N_1, N_0, T) )).

The experimenter rejects the null if the probability of observing Z(M_1,N_1,N_0,T) under the null hypothesis is less than α. Let  denote a minimum detectable effect threshold and β∈(0,1) be the desired power. If the true effect θ^ss(M_1, N_1) is at least as large as the , then the experimenter wants to reject the null hypothesis with probability β. We will develop approximations for the two components of (<ref>) in <ref>.




  §.§.§ Trial design decisions.


When planning a trial, experimenter must choose the number of servers M_1 and the number of subjects N_1 and N_0 to ensure a high powered test at the specified , or equivalently, a high enough ratio in Equation (<ref>). 


In trials without interference, the problem is simple, since the variance in (<ref>) decreases linearly in the number of subjects, and boils down to minimizing the number of subjects N_1+N_0 while meeting the power criterion. 
For service interventions, the decision is more complicated as the experimenter must jointly decide on N_1, N_0 and M_1. In <ref> we will contrast two naive approaches for power analysis with a queueing-informed approach, demonstrating that obtaining accurate power estimates relies on understanding the level of congestion interference. 


For service interventions, the decision is more complicated as the experimenter must jointly decide on N_1, N_0 and M_1. In <ref> we will contrast two naive approaches for power analysis with a queueing-informed approach, demonstrating that obtaining accurate power estimates relies on understanding the level of congestion interference. 










§ ANALYSIS OF SYSTEM


This section provides the mathematical foundations to analyze these experiments; later <ref> and <ref> use these expressions to characterize the implications for service intervention RCTs.
To study the quantities defined in <ref>, we need to characterize both the expected value of the estimator and its variance. By Lemma <ref>, this problem reduces to characterizing the time average of the queue length and the associated variance, for both the treatment and control groups. 

Here, we first use prior results from the literature to establish a Central Limit Theorem for the two groups and the estimator as T→∞ (<ref>).
However, due to the dependencies between individuals in the treatment group, the expressions for the average queue length and its variance are complex even in the large T limit (see Equations (<ref>)-(<ref>)). 
Therefore, to obtain analytical insights on the behavior of these two quantities, we subsequently develop a fluid limit to approximate the average queue length in large systems with many users and many servers (<ref>). In particular, we illustrate how to apply these approximations to study experiment outcomes (<ref>).




 §.§ Central Limit Theorem for estimator in finite system

Recall that the estimator θ̂(M_1, N_1, N_0, T), defined in Equation (<ref>) as

    θ̂(M_1, N_1, N_0, T)  =
        [    ∑_u: Z_u=11 - Y_M_1,N_1(u,T)/N_1] 
        -  [ 
        ∑_u: Z_u=0 1- Y_0, N_0(u,T) /N_0],

compares the average time spent in the desirable state across groups, where the first term corresponds to the average over the users in the treatment group and the second term corresponds to the average over users in the control group. 

We derive a Central Limit Theorem (CLT) for the estimator θ̂(M_1, N_1, N_0, T) by first deriving a CLT for each of the treatment and control terms separately. Since the treatment and control groups are independent from each other, the CLT for the estimator then immediately follows. 

We first remind the reader of the challenges in deriving a CLT for this system. The term corresponding to the treatment group is a sum of dependent random variables Y_M_1,N_1(u,T), since the length of time that a user remains in the undesirable state can depend on whether other users are waiting for service. In contrast, the term corresponding to the control group is a sum of independent random variables Y_0, N_0(u,T) since the users' states evolve independently from each other in the control group. However, in both groups, there is a dependency across time for each user u. 
That is, the state of user u at time t affects the distribution of their state at some later time t'. Each random variable Y_M,N(u,T) is an integral over time and so we must characterize the dependency across time for both groups.
Thus, we do not have independent and identically distributed samples, a condition typically assumed to guarantee the existence of a CLT. Nevertheless, we are able leverage existing results from  <cit.> in queueing theory to establish a CLT. 


Let 
A(M_1,N_1,T) =  ∑_u: Z_u=11 - Y_M_1,N_1(u,T ) / N_1
and 
B( N_0,T) =
    ∑_u: Z_u=0 1- Y_0, N_0(u,T)  / N_0
denote the treatment and control terms of Equation (<ref>), so that θ̂(M_1, N_1, N_0, T) = A(M_1, N_1,T) - B(N_0,T). 


Then the following quantities converge in distribution

    T^1/2·( A(M_1,N_1,T) - μ_1 ) 
            N(0, σ̃_1^2) 
        
    
        T^1/2·( B(N_0,T)  - μ_0)     N(0, σ̃_0^2)

where 

    μ_1 = μ_1(M_1,N_1) = 1 - K_M_1,N_1/N_1,
      μ_0 = τ/λ + τ
    σ̃^2_1 = σ̃_1^2(M_1,N_1) = 2/N_1^2∑_j=0^N_1-11/(N_1-j) π_M_1,N_1(j)[ ∑_i=0^j (i - K̅_M_1, N_1)π_M_1,N_1(i) ]^2
    σ̃_0^2 = σ̃_0^2(N_0) = 1/N_0·2 λτ/(λ + τ)^3


Then the scaled estimator converges

    T^1/2·(θ̂(M_1, N_1, N_0, T)  - (μ_1 - μ_0) )  N(0,  σ̃_1^2 + σ̃_0^2).










The convergence to the time average of each group to its mean follows directly from the system being an ergodic Markov chain. The formula for variance follows from a result for Markov processes in <cit.>.
The proof for this result is in Appendix <ref>. 


A corollary of Theorem <ref> is that θ̂(M_1, N_1, N_0, T) is a consistent estimator for the the long term treatment effect θ^ss(M, N), when M=M_1 and N=N_1. That is, the estimator recovers the true treatment effect if the experiment is run for long enough. 
As we discuss later, this result does not hold for general M≠ M_1 and N≠ N_1.

For any N_0>1, 
    
    lim_T →∞θ̂(M_1, N_1, N_0, T) θ^ss(M_1, N_1)

where p denotes convergence in probability.


Motivated by Theorem <ref>, we use the following approximation for the mean of the estimator:

    θ̂(M_1, N_1, N_0, T) ≈[ 1 - K_M_1,N_1/N_1 ] 
        - τ/λ + τ.

Similarly, we use the following approximation for the variance of the estimator for finite T:

    (θ̂(M_1, N_1, N_0, T))
        ≈σ̃^2 / T.

The simulations in Appendix <ref> for the system at finite time horizons show that these simulation quantities align closely with the approximations suggested by the CLT for large enough T. 






 §.§ Approximating the average queue length using a fluid limit

To capture the implications of congestion arising from capacity constrained interventions, we must characterize how the size of the treatment effect defined in Equation (<ref>) changes with the number of servers and users. 
In other words, we need to characterize how the mean of the limiting distribution in Theorem <ref> depends on N_1 and M_1. 
To obtain analytical insights, instead of directly analyzing the expression in Equation (<ref>),  we utilize a fluid limit to approximate this quantity in large systems. This approach is a method from queueing theory that allows us to simplify the expression for queue length (see <cit.> for the general method and <cit.> for an example in a closed queueing system). 

We first present the fluid limit 
of the queueing system with M servers and N users. In the next section ( <ref>), we describe how to use this limit to approximate experiment quantities described above.


We scale up both the number of servers M and number of users N proportionally (such that M/N approaches a constant) and consider the average behavior of the system, which becomes deterministic even for finite T. 
Formally, suppose that the number of servers is a function of N, denoted M^(N). Further assume that we choose a staffing rule such that lim_N→∞ M^(N) / N = M. 
We define the following fluid limit, designed to represent the behavior of the system as N →∞. Let z(t) denote the fraction of users in the undesirable state at time t such that z(t) obeys the following ordinary differential equation:
dz(t)/dt=λ (1-z) - z μ p - z τ,     for z< M, 
λ (1-z) - Mμ p - z τ,    for z ≥M.

The two equations show that the evolution of this system depends on whether the fraction of users in the undesirable state z(t) is greater than or less than the the ratio  of servers to users M. When z(t)<M, the rate at which users receive service undesirable state is limited by z(t). Conversely, when z(t) > M, the rate at which users receive service is limited by M. 



Analogous to the finite system steady state quantity K_M,N/N, we can define the steady state proportion of users in the system (undesirable state) in the fluid limit, defined to be the value at which dz/dt=0, which we denote z^*_ M. The following result characterizes the steady state behavior, as a function of model parameters. The proof is in Appendix <ref>.


    For a given value of M, the ODE system given by equations (<ref>)–(<ref>) has the following steady state solution:
    z^*_M
    = λ - Mμ p /λ + τ,    for M≤ r,
λ/λ + τ + μ p,    for M> r.
    
In particular, note that this expression directly shows how increasing the ratio of servers M to users affects the fraction of people in the undesirable state, unlike the expression for queue length in Equation (<ref>) for finite settings.


For a finite system with large enough values of M and N, we can use the behavior of the fluid limit to approximate the average queue length in the finite system as follows:

    K_M,N≈ N · z^*_M/N.

Indeed, we observe that the the steady state distribution of the finite system converges to the steady state of the fluid limit as M and N increase. See Appendix <ref> for comparisons of simulations of the finite system and the fluid limit.




  §.§.§ Fluid limit approximations for experiment quantities


For a system with finite M and N, we can approximate the finite system behavior using the fluid limit with limiting server-to-user ratio M/N. Applying this to the experiment context
with M_1 servers, N_1 users in the treatment group, and N_0 users in the control group, we define the fluid limit treatment effectθ^*, analogous to the finite system steady state treatment effect θ^ss
as

    θ^*(M_1/N_1)
        := 
        [1 - z^*_M_1/N_1] - 
        [1 - z^*_0  ].

and use the fluid limit treatment effect to approximate θ^ss(M_1,N_1)
    θ^ss(M_1,N_1) ≈θ^*(M_1/N_1).






We use this approximation in Section <ref> to highlight how the steady state treatment effect θ^ss(M_1,N_1) changes with M_1 and N_1. 
However, we note that this fluid limit is deterministic and thus, when we study the variance of the estimator and the statistical power of the experiment, we use the finite system quantities in Theorem <ref>.













§ INSIGHTS FOR EXPERIMENTAL DESIGN


In this section, we highlight the impact that congestion has on the magnitude of the treatment effect and the resulting statistical power of the RCT. 
Note that there are four experiment parameters that the experimenter decides in the design phase: the number of service providers M_1, the number of treatment and control users N_1 and N_0, as well as the time horizon T. We focus on the choices of M_1, N_1, and N_0, to highlight the capacity constrained dimension of these interventions.



















































    











 §.§ Non-linear effect of user load on effect size



Intuitively, in a service intervention, if we fix the number of servers but increase the number of users, then users will likely need to wait longer to be served and hence, will spend more time in the undesirable state. In this situation, we expect the treatment effect of the intervention to decrease. However, we can show using the fluid limit that this impact of the user load on effect size is non-linear. When the user load is small, the treatment effect is in fact constant in the user load. Only when the user load is large enough does the treatment effect decrease in user load, following the intuition above. Later <ref> and <ref> show how to use this non-linearity to increase the statistical power of an experiment.



Fix a limiting server-to-user ratio M = lim_N→∞ M_1^(N) / N_1.  
The fluid limit treatment effect is

    θ^*(M) 
        =
        M·μ p /λ + τ   for M≤ r
        
    λμ p/(λ + τ)(λ + τ + μ p)   for M≥ r.

where r is the critical ratio.

In particular, 
as the server-to-user ratio decreases to 0, we have

    lim_M→ 0θ^*(M) 
    = 0,

The proof follows immediately by applying the fluid limit steady state characterization in Proposition <ref> to the treatment and control systems. Using the mean field limit, Proposition <ref> confirms the intuition that there is not a single treatment effect for a service intervention, but rather a set of different treatment effects corresponding to different server-to-user ratios M. Figure <ref> shows these fluid approximations for the treatment effect, along with the finite system quantities in Equation (<ref>).


We note that the outcome of the control group does not depend on M and that changes in the treatment effect stem only from changes in the outcomes of the treatment group. 
When M decreases, there are fewer servers for the treatment group and on average treatment users will wait longer to be served. In the limit, when M→ 0 (as is the case when holding M fixed and increasing N→∞), users will not be served and the treatment effect diminishes. On the other hand, when increasing M, users will experience shorter wait times before being served and as M→∞, users will be served immediately after entering the undesirable state. 

The dependence of the treatment effect is not linear in M. For small M<r, increasing the ratio of servers to users increases the treatment effect but when M>r is large, increasing M does not increase the treatment effect. This result is due to the underlying queueing structure of the service intervention. Recall the queueing regimes defined in Section <ref>. The M < r case corresponds to the Efficiency-Driven regimes where, on average, users who enter the undesirable state will need to wait to be served. In this setting, increasing the number of servers decreases how long users must wait before receiving the intervention and increases the treatment effect. The  M>r case corresponds to the Quality-Driven regime, where the system maintains a buffer of unused servers and the probability that a user needs to wait to be served is low (tending to 0). In this setting, most users are served immediately and increasing the number of servers will not increase the treatment effect. 







Proposition <ref> also implies that the experiment estimator may be biased if the experiment does not the same number of servers or users as the intended intervention when scaled up. Consider the setting where an experimenter runs a study with M_1 servers, N_1 users in treatment, and N_0≥ 1 users in control and obtains the estimate θ̂(M_1,N_1,N_0)≈θ^*(M_1/N_1). If they choose to scale up the platform to N>N_1 users without increasing the number of servers M_1, then they will obtain an unbiased estimate of the treatment effect if and only if both M_1/N ≥ r and M_1/N_1 ≥ r; otherwise the experimenter will overestimate the true effect. 
See Proposition <ref> in Appendix <ref> for a complete characterization of the bias for all ratios of M/N and M_1/N_1. 

















 §.§ Estimation of variance

In general, the variance of the sampling distribution θ̂ is unknown and the experimenter must estimate the variance using observed data. In settings where individual outcomes Y_M,N(u,T) are independent across u, there exists a constant σ^2 = Var(Y_M,N(u,T)) and the variance of the sample mean is given by Var(∑_u=1^N Y_M,N(u,T) / N) = σ^2 / N. The standard approach is to estimate the sample variance s^2 across observations and estimate the variance of the sample mean with s^2/N. 

However, with a capacity constrained intervention, the users in the treatment group are dependent and there are correlations in Y_M,N(u,T) across users that the naive variance estimation procedure ignores. In particular, as seen in Theorem <ref>, the asymptotic variance for the treatment group average 

    σ̃_1^2 = σ̃^2(M,N_1, N_0) = 2/N_1^2∑_j=0^N_1-11/(N_1-j) π_M,N_1(j)[ ∑_i=0^j (i - K̅_M, N_1)π_M,N_1(i) ]^2
 
does not decompose into the form σ^2 / N_1.

We leave this problem of variance estimation for future work and, for the rest of this paper, assume that the experimenter has the simpler task of running an experiment when the true variance is known. We show that even in this setting, where variance estimation is not an issue, the capacity constraints on the intervention will invalidate statistical guarantees in naive experiments.




 §.§ Power is not monotonic in number of patients



We now consider the statistical power of the hypothesis test, which is the probability of finding a statistically significant result given that the true effect size is at least some threshold. Let M_1 denote the number of providers for the treatment group and N_1 and N_0 denote the number of users in treatment and control. We assume for simplicity that N_1=N_0.

Let the null hypotheses be H_0:μ_r=μ_c and suppose we are conducting a one-tailed test such that the alternative hypothesis  is H_a:μ_r > μ_c. Let α be the significance level (e.g., α=0.05), β be the desired power (e.g., β=0.80), and MDE>0 denote the a Minimum Detectable Effect threshold. Fix T>0. 



The test statistic under the null hypothesis is 

    Z(M_1, N_1, N_0,T) = 
        θ̂(M_1, N_1, N_0, T) - 0/√(σ̃_r^2  (M_1, N_1)/T
        +
        σ̃_0^2 (N_0)/T
        ).

Recall that if T is large enough, Theorem <ref> shows that the estimator θ̂(M_1, N_1, N_0, T) is approximately normally distributed. For tractability, we assume that the estimator is indeed normally distributed, although we expect that the results hold without this assumption.
If we make the assumption that the estimator θ̂(M_1, N_1, N_0, T) is normally distributed, then we can run a z-test. The power of this test at a fixed  is

    1-Φ(Φ^-1(1-α)-/√(σ̃_1^2  (M_1, N_1)/T
        +
        σ̃_0^2 (N_0)/T)).

See Equation (<ref>) in Appendix <ref> for the derivation.


The experimenter generally fixes a static MDE and wishes to ensure a probability β of detecting a true effect θ^ss(M_1, N_1, N_0) at least as large as MDE. As we have seen, however, in service interventions this true effect size depends on M_1 and N_1. To capture this phenomenon, for an experiment run with M_1 providers, N_1 treatment users, and N_0 control users, we consider the power to detect an effect at least as large as the true effect of the experiment; that is, we fix the MDE = θ^ss(M_1, N_1) to obtain:


    1-Φ(Φ^-1(1-α)-θ^ss(M_1, N_1)/√(σ̃_1^2  (M_1, N_1)/T
        +
        σ̃_0^2 (N_0)/T)).

This quantity, of course, is not known to the experimenter when running the experiment. However, we can use this quantity to evaluate different experiment designs.


Now we consider the implications of Equation (<ref>). First, note that
power increases when θ^ss(M_1, N_1, N_0) increases, since Φ is a monotonically increasing function. 
Further, power increases when the variance term σ̃_r^2  (M_1, N_1) + σ̃_0^2 (N_0) decreases. 


First consider the setting where SUTVA holds and all treatment and control users are independent. Then, the numerator θ^ss(M_1, N_1) is constant in M_1, N_1 and N_0. For the variance term, there exist constants σ̃_0^2 and σ̃_0^2 such that σ̃_1^2  (M_1, N_1) = σ̃_1^2/N_1 and σ̃_0^2 (N_0) = σ̃_0^2/N_0; in other words, the variance term for each of the treatment and control group averages decrease linearly with 1/N_1 and 1/N_0, respectively. Thus,  increasing N_1 and N_0 increases the power of the hypothesis test. 






For service interventions, SUTVA does not hold and we find that power is not monotonic in the number of patients. Figure <ref> plots the statistical power at different M_1 and N, where N_1 = N_0 = N/2, using approximations in Equations (<ref>)-(<ref>).
For each number of providers M_1, we see that power is increasing in the number of patients N until a threshold (depending on M), after which power is decreasing in N. 

The expression for power in Equation (<ref>) highlights two factors leading to in power: (a) the treatment effect θ^ss(M_1,N_1,N_0) decreases in N (Figure <ref>) and (b) the treatment group variance term σ̃^2_r(M_1,N_1) (given in Theorem <ref>) does not necessarily decrease with 1/N (Figure <ref>). (The control group samples are independent and we indeed have σ̃_0^2 (N_0) = σ̃_0^2(1)/N_0.)
Qualitatively, we observe that (a) the difference in the treatment effect plays a larger role than (b) the deviation in variance. For (a), the top left figure shows a sharp decrease in the treatment effect size in N. For (b), the top right figure shows that the control group variance decreases with 1/N but the treatment group variance deviates slightly in the small N regime. 





A comparison of the plots for the treatment effect (Figure <ref>) and the variance in (Figure <ref>) provides intuition on how to maximize power, for a given M. As discussed in Section <ref>, treatment effect is constant for small N (the Quality Driven regime) up until a quantity that depends on M and then decreases for larger N (the Efficiency Driven regime). On the other hand, the variance is largely decreasing in N, and, in particular, is decreasing quickly for small N. Thus, for the small N interval of the Quality Driven regime, we can see that power is increasing in N, since power is a function of the ratio between the treatment effect and the standard error.
For large N in the Efficiency Driven regime, the treatment effect appears to be decreasing more quickly than the variance, and so power is decreasing in N. Thus one might expect that the optimal N is one in the “middle” between these two regimes, what is known in queueing theory as the Quality Efficiency Driven regime. Indeed, we use this observation in the <ref> to suggest a policy for experiment planning. 









 §.§ Numerical experiments: Optimal N varying parameter values

Figure <ref> explores how the optimal number of users N changes as system parameters vary. We set the base parameters to be λ=0.3, τ=0.3, μ=3, p=0.5, and each plot varies a single parameter. We focus on arrival rates λ∈ [0, 0.5] and abandon rates τ∈ [0, 0.5] to reflect that states are “sticky”, i.e., a user is more likely to stay in the same state they were in yesterday, in the absence of intervention. 


Figure <ref> shows that the optimal N is decreasing in the arrival rate λ, increasing in abandon rate τ, increasing in service rate μ, and increasing in the probability of effect p. The directional effects intuitively reflect how the number of users in the queue change with respect to the parameters, e.g., as the arrival rate increases, more users will be entering the queue and so a larger number of servers M is needed. On the other hand, as the abandon rate increases, more users leave the queue and thus a fixed M will be able to serve more users.

In summmary, Figure <ref> implies that, for each M_1, the power optimal N the depends on the system parameters. We explore this direction in the next section and propose a heuristic Square Root Staffing policy, informed by queueing theory, to increase power by leveraging information about the underlying system structure.






§ COMPARISON OF POWER ANALYSIS APPROACHES

A main objective of this work is to help experimenters avoid the mistake of concluding an intervention is ineffective following RCT experimentation, simply because operational (capacity) constraints led to smaller effect sizes and, thus, an underpowered experiment. This goal is especially relevant for an efficacy trial, which aims to evaluate the impact of an intervention under ideal conditions. If the intervention seems promising in the efficacy trial, then there may be what is known as an effectiveness trial, designed to estimate the effect of the intervention under realistic conditions <cit.>. In our setting, an effectiveness trial might staff the experiment with a similar number of servers and users as what may be realistically deployed to the intended user-base.

A standard procedure in experiment design to ensure that an experiment is well powered is to first run a small scale pilot study and then conduct a power analysis, using the outcomes from the pilot study, to decide how many users to include in the full study. 



In the remainder of the section, we consider two naive power analysis approaches that do not account for the capacity constrained structure intervention that we show may either be underpowered or be costly to run (requiring high M_1, N_1, or N_0). We then define a queueing-informed policy that leverages the intervention structure to ensure high power and with fewer resources (requiring lower M_1, N_1, and N_0).




 §.§ Setup and naive power analysis policies

Consider a small scale pilot study, with M_1^p servers and N^p in the experiment, with N^p_0 in control and N^p_1 in treatment. Fix some T>0. After running the pilot study, the experimenter obtains an estimate of the treatment effect θ̂(M_1^p, N^p_1, N^p_0, T) and an estimate for the variance (θ̂ (M_1^p, N^p_1, N^p_1, T)). Based on the estimates from the pilot study and a specified Minimum Detectable Effect (MDE), the experimenter decides how to set the number of users and servers in the full experiment. 

A power analysis policy thus specifies an (M_1, N_1, N_0) pair. 
We focus on policies with an equal number of users in treatment and control N_1 = N_0, although the observations can be extended for any constant treatment proportion. 



  §.§.§ Naive Policy 1: No M_1 Scale-up.

Recall that the long-term treatment effect is θ^ss(M_1, N_1, N_0) and the variance (θ̂ (M_1, N_1, N_0,T) ) ≈ 1/T ·(σ̃^2_1(M_1, N_1,T) + σ̃^2_0( N_0,T)).

For this first naive policy, the experimenter is not aware of those dependencies and makes two (incorrect) assumptions about the experiment:

  *  They wrongly assume that the treatment effect θ^ss(M_1, N_1) does not depend on M_1 nor N_1.
    
  *  
    They wrongly assume that there exist constants σ̃_0^2 and σ̃_0^2 such that σ̃_1^2  (M_1, N_1) = σ̃_1^2/N_1 and σ̃_0^2 (N_0) = σ̃_0^2/N_0.


Due to assumption (1), they believe that there is a constant  treatment effect for any M_1, N_1, and N_0. 
An implication is that all experiment designs will lead to the same treatment effect as the pilot study. Thus, if the experimenter knew the treatment effect of the pilot study, they would set =θ^ss(M_1^p, N^p_1). Further, an experimenter who is not aware of the impact of congestion does not increase the number of servers beyond M^p_1. 

Of course, the experimenter does not know θ^ss(M^p, N^p_1) and only obtains an estimate θ̂(M_1^p, N^p_1, N^p_0, T). Further, as Section <ref> discusses, it is unclear how to estimate the variance of the estimator θ̂(M^p, N^p_1, N^p_0, T). To simplify matters, we assume that the experimenter knows three additional pieces of information after the pilot study: they have oracle access to θ^ss(M^p_1, N^p_1, N^p_0),  σ̃^2_1(M^p_1, N^p_1)/T, and σ̃^2_0( N^p_0)/T. We note that oracle access to these quantities makes the naive experimenter's task easier, and we will still show that the naive policy underperforms with respect to the queueing-informed policy. In other words, the experimenter has perfect knowledge of the sampling distribution of the estimator in the pilot study, but is not given additional information about the sampling distribution at other M_1, N_1, or N_0.

Under this oracle knowledge and Assumption (2), the naive experimenter (incorrectly) assumes that the treatment group variance term scales with N_1 and so the variance at N_1 ≠ N^p_1 is σ̃^2_1(M^p_1, N^p_1)/T · N^p_1 / N_1. 
Likewise, the experimenter (correctly) assumes that the control group variance term scales with N_0 and so the variance at N_0 ≠ N^p_0 is σ̃^2_1( N^p_0)/T · N^p_0 / N_0.



Given these assumptions, the naive experimenter chooses N_1 by solving the following optimization problem. 


    minimize       N_1 
    subject to      1-Φ(Φ^-1(1-α)-/√( f(N_1)+ g(N_0)))≥β
        = θ^ss(M_1^p, N^p_1)
        M_1 = M^p_1
        f(N_1) =  N^p_1/N_1  T·σ̃^2_1(M^p_1, N^p_1) 
        g(N_0) = N^p_0/ N_0  T·σ̃^2_1( N^p_0)  
        N_1 = N_0 
        N_1 ∈ℤ^n_+

Note that (<ref>)-(<ref>) reflect Assumption (1) and (<ref>)-(<ref>)  reflect Assumption (2). 
Let N_1^naive denote the solution to the naive optimization problem. Then the first Naive Policy of not scaling M_1 would recommend
(M_1^p, N_1^Naive, N_1^Naive) for the full experiment.













  §.§.§ Naive Policy 2: Proportional M_1 Scale-up.

We consider a (less naive) second policy where the experimenter makes Assumption (2) but not Assumption (1). This experimenter knows that θ^ss(M_1,N_1) depends on both M_1 and N_1, but does not necessarily know how. In this setting, experimenter similarly sets MDE = θ(M_1^p, N_1^p) and prescribes the same N^naive as above, but takes the natural step of scaling M_1 proportionally with N_1, so that 

    M_1 = ⌈ M_1^p ·N^naive/N_1^p⌉.

The Naive Policy (Proportion M_1 Scale-up) recommends (⌈ M^p_1 ·N^naive/ N^p⌉, N_1^Naive, N_1^Naive) for the full experiment.




 §.§ Queueing Informed Square Root Staffing Policy.


We now consider a policy informed by queueing theory, which utilizes the square root staffing rule, a common heuristic to balance efficiency and quality of service. This experimenter intelligently corrects for both Assumption (1) and Assumption (2).

As observed in Section <ref>, power is monotonically increasing for small N (the Quality Driven regime) and decreasing for large N (the Efficiency Driven regime). Motivated by these observations, we aim to set the number of servers M and users N such that the queueing system is operating in the intermediate Quality Efficiency Driven regime. 
We utilize the Square Root Staffing rule to set staffing levels in this regime. Introduced in <cit.>, this heuristic calculates for each number of users N the number of servers M required to maintain an approximately fixed probability of delay.

We first describe the square root staffing rule, which we later incorporate into a power analysis policy. Fix a constant γ >0. In this work, we take γ = 0.5, although the choice of γ should depend on the parameters of the queueing system <cit.>. We expect the dependence on γ to diminish in larger instances. We leave further optimization of γ in the context of experiment design to future work. Recall that r = λ / (λ + τ + μ p) is the critical ratio, as defined in Equation (<ref>), and rN_1 is offered load, or the average load on the system if there were always an available server when a patient entered the undesirable state. 
For each N_1>0, the square root staffing rule computes the required number of servers M^*(N_1) where 

    M_1^*(N_1) := ⌈ r N_1 + γ√(N_1)⌉,

The number of servers M_1 is chosen to match the offered load on the system r N_1, but adds a buffer γ√(N_1) to account for variability in arrival and service times. In other words, to maintain a fixed level of service when increasing N, the chosen M should not be linear in N but rather needs to account for variability in the system and incorporate this √(N_1) term.

Equation (<ref>) implies that as the number of users N_1 increases, then the ratio of servers to users M_1^*(N_1) / N_1 → r. 
Indeed, this limiting ratio r coincides with the threshold in Proposition <ref>, which shows that in the (large system) mean field limit, the treatment effect is increasing in the limiting server to user ratio M when M<r and increasing in M when M>r. Intuitively, this staffing policy chooses the smallest M to maintain a high level of service and, thus, a large treatment effect.

Now we combine the square root staffing rule into an experiment design approach. For a fixed N_1, the square root staffing rule outputs an M_1^*(N_1), but the experimenter must still choose N_1. 


The queueing informed experimenter solves the following optimization problem

    minimize       N_1 
    subject to      1-Φ(Φ^-1(1-α)-/√( f(N_1)+ g(N_0)))≥β
        = θ^ss(M_1, N_1)
        M_1 = ⌈ r N_1 + γ√(N_1)⌉
        f(N_1) = σ̃^2_1(M_1, N_1)/T 
        g(N_0) = σ̃^2_1(N_0)/T  
        N_1 = N_0 
        N_1 ∈ℤ^n_+

Let N^sqrt_1 denote the solution N_1 to the above problem. 
Compared to the naive optimization problem, the queueing informed optimization problem corrects for Assumption (1) in (<ref>),  implements the square root staffing rule in (<ref>), and uses the variance approximation from Theorem <ref> in (<ref>)-(<ref>).

The queueing informed Square Root Staffing Policy then proposes (⌈ r N_1 + γ√(N_1)⌉, N_1^sqrt, N_1^sqrt) for the full experiment. 

Note, however, that both calculating the expressions in (<ref>)-(<ref>) requires knowing λ, τ, μ, and p. In practice, these parameters are unknown and experimenter would need to obtain estimates λ̂, τ̂, μ̂, and p̂ from the pilot study, and then use these estimates to compute (<ref>)-(<ref>). Similar to the discussion of variance estimation for the naive policy, in order to isolate the issue of scaling up M_1 and N_1, we assume that the experimenter has oracle access to the true quantities and leave the issue of estimation of queueing quantities to future work. 







 §.§ Numerical experiment setup

We consider a setting with λ = 0.4, τ = 0.35, μ = 3, p=0.1, and T=10.  We focus on a setting with λ<0.5 and τ<0.5 to capture the effect where states are “sticky”, i.e., without intervention, the user is more likely to remain in the state they were in yesterday. We then consider two scenarios for a pilot study: Scenario 1 with M_1^p = 5 and N_1^p = N_0^p = 10 and Scenario 2 with the same number of servers M_1^p=5 but more users N_1^p = N_0^p = 25. 

We suppose that the experimenter wants to run an experiment at significance level α = 0.05 and desired power β=0.80. 
We compare the power at each of the three power analysis policies: Naive Policy 1 (No M_1 Scale-up), Naive Policy 2 (Proportional M_1 Scale-up), and the Queueing Informed Square Root Staffing Policy. 
  



Using Equation (<ref>), we evaluate the statistical power of the experiment with (M_1, N_1, N_0) to detect a treatment effect at least as large as the true treatment effect θ^ss(M_1, N_1) of the experiment, where we approximate estimator mean and variance using Equations (<ref>) and (<ref>).




 §.§ Comparison of power analysis policies



Figure <ref> shows the performance of the three policies, for two difference instances of M^1_p and N_1^p. Since these initial pilot study parameters M^1_p and N_1^p are typically chosen for feasibility reasons and not optimized, a power analysis policy would ideally perform well given any instance of M^1_p and N_1^p. While the queueing informed Square Root Staffing Policy can give the desired power β for both scenarios, Naive Policy 1 (No M_1 Scale-up) does not achieve the desired power in either scenario, and Naive Policy 2 (Proportional M_1 Scale-up) achieves the desired policy but, depending on the initial pilot study conditions, may require significantly more resources than the Square Root Staffing Policy.

Under these parameters, the critical ratio is r=0.38 and Scenario 1 (with M_1^p/ N_1^p = 0.5) corresponds to the small N_1 (Quality-Driven) regime whereas Scenario 2 (with M_1^p/ N_1^p = 0.2) corresponds to  the large N_1 (Efficiency-Driven) regime. Importantly, the treatment effect is larger for M^p = 5 and N^p = 20 than it is for M^p=5 and more users N^p = 50. (See Figure <ref> for an illustration of effect size.) 

In Scenario 1 (small N^p_1 pilot), the treatment effect in the pilot study is θ^ss(5, 10) = 0.14. Naive Policy 1 (No-M_1 Scale-up) chooses (M_1,N_1, N_0)=(5, 35, 35), Naive Policy 2 (Proportional M_1 Scale-up) chooses (M_1,N_1, N_0)=(18, 35, 35), and the Square Root Staffing Policy chooses (M_1,N_1, N_0)=(16, 34, 34). We can see that Naive Policy 1 (No M_1 Scale-up) does not achieve the desired power, and actually has less power than the initial pilot experiment, due to the decreased effect size when increasing N_1 but not increasing M_1. Both the Naive Policy 2 (Proportional M_1 Scale-up) and the Square Root Staffing Policy achieve 80% power. However, the Square Root Staffing Policy is able to achieve this power with slightly fewer users and servers, which reduces costs for running the experiment.


In Scenario 2 (large N^p_1 pilot), the treatment effect in the pilot study is θ^ss(5, 25) = 0.07, which is lower than the treatment effect in Scenario 1, due to the higher user load. We observe a similar pattern where the Naive Policy 1 (No M_1 Scale-up) is underpowered but both Naive Policy 2 (Proportional M_1 Scale-up) and Square Root Staffing policies achieve 80% power.  Naive Policy 1 (No-M_1 Scale-up) chooses (M_1,N_1,N_0)=(5, 140, 140), Naive Policy 2 (Proportional M_1 Scale-up) chooses (M_1,N_1, N_0)=(28, 140, 140), and the Square Root Staffing Policy chooses (M_1,N_1,N_0)=(16,33, 33). In this setting, the Square Root Policy is able to achieve the same power as Naive Policy 2, but with 42% fewer servers and 76% fewer users, requiring far fewer resources for the same power. Both naive policies choose a large number of users because they fail to realize that the small treatment effect size in the pilot study is due to a high user load on the system. Both naive policies attempt to increase the number of users in order to detect an effect as small as the pilot study observed effect size. The Queueing Informed Square Root Staffing rule recognizes that the treatment effect can be larger if there are more servers and thus requires fewer users in the experiment to detect this effect.























    




    













§ CONCLUSION


Our study provides insights into the challenges of experimentation with capacity constrained service interventions and also illustrates how we can run better experiments (with larger power and lower costs) by incorporating knowledge of the underlying queueing structure. With service interventions, it is possible that an intervention is beneficial and has a large effect size under ideal conditions (high enough capacity relative to the number of users) but unless the capacity is carefully chosen, an experiment may be underpowered to detect the effect. 

This work serves to highlight the importance and the magnitude of such effects and provides a springboard for directions of future research. One direction is to develop the variance estimation and queue parameter estimation problems needed for both the naive and the queueing informed power analysis policies. Another direction is to study how to extrapolate from a single experiment with a given (M,N) and estimate how the experiment would have performed if a different M were used. Finally, one may want to consider heterogeneous models for users, servers, and treatment effects and also consider interventions with different priority queues, in addition to the first come first serve queue we consider here. 




informs2014

§ EXPRESSIONS FOR POWER
Hypothesis testing.

The experimenter conducts a one-sided hypothesis test at the α=.05 significance level with null hypotheses H_0:μ_r=μ_c and H_a:μ_r > μ_c. The desired power is β = 0.80.


Fix T>0 and let {r_1,...,r_N_1} and {c_1,...,c_N_0} be two sets of independent samples where r_u = 1/T·∫_t=0^T x_M_1, N_1(u,t) dt and c_u = 1/T·∫_t=0^T x_0, N_0(u,t) dt. 
Let r̅(M_1, N_1)=∑_u: Z_u=11/T·∫_t=0^T x_M, N_1(u,t) dt/N_1 and c̅(N_0)=∑_u: Z_u=01/T·∫_t=0^T x_0, N_0(u,t) dt/N_0 be the sample means from the treatment and control group, respectively. 
Note that for large enough T, both quantities r̅(M_1, N_1) and c̅(N_0) are each approximately normally distributed, as suggested by Theorem <ref>. 


For simplicity, we proceed with a normal approximation and assume that r̅(M_1, N_1) ∼ N(μ_r,σ_r^2) and c̅(N_0) ∼ N(μ_c,σ_c^2); we provide expressions for each of these quantities μ_r, μ_c, σ_r^2, σ_c^2 in the next section. We expect the main insights from our analysis to hold for the true distribution at finite T.

The test statistic under the null hypothesis is

    Z(M_1, N_1, N_0) = 
        (r̅(M_1, N_1)-c̅(N_0)) - 0/√(σ̃_r^2  (M_1, N_1)
        +
        σ̃_0^2 (N_0)
        ).

where σ̃_r^2  (M_1, N_1) and σ̃_c^2  (N_0)
denote the variance of the treatment and control group sample means, respectively. Under the null hypothesis, Z(M_1, N_1, N_0) is approximately distributed as N(0,1). 

Note that if all treatment user outcomes r_u were independent, then [r̅(M_1, N_1)-c̅(N_0)] is constant in M_1, N_1, and N_0, whereas this quantity depends on M_1 and N_1 in the setting with service interventions.



Power.
Let Φ(z) denote the CDF of the standard normal distribution. 
For a fixed minimum detectable effect , 

    P(Z > Φ^-1(   1-α) | μ_r - μ_c = ) 
       = 1 - P(Z ≤Φ^-1(1-α) | μ_r - μ_c = ),
       = 1 - P(r̅-c̅/√(σ̃_r^2  (M_1, N_1)
        +
        σ̃_0^2 (N_0))≤Φ^-1(1-α) | μ_r - μ_c = ), 
       = 1 - P(r̅-c̅ -  + /√(σ̃_r^2  (M_1, N_1)
        +
        σ̃_0^2 (N_0))≤Φ^-1(1-α) | μ_r - μ_c = ), 
       = 1 - P(r̅-c̅ - /√(σ̃_r^2  (M_1, N_1)
        +
        σ̃_0^2 (N_0))≤Φ^-1(1-α) - /√(σ̃_r^2  (M_1, N_1)
        +
        σ̃_0^2 (N_0)) | μ_r - μ_c = ), 
       ≈1-Φ(Φ^-1(1-α)-/√(σ̃_r^2  (M_1, N_1)
        +
        σ̃_0^2 (N_0)))

where the approximation in the last line comes from the CLT in Theorem <ref>, which implies that Z(M_1,N_1,N_0) is approximately normally distributed with mean θ and variance σ̃_r^2  (M_1, N_1)+σ̃_0^2 (N_0).























§ PROOFS
Proof of Theorem <ref>.
    The proof proceeds in three parts: 1) showing existence of a CLT exists for the estimator, 2) characterizing the mean of the limiting distribution, and 3) characterizing the variance of the limiting distribution.

    Step 1: Existence of a CLT. We utilize Theorem 3.1 in <cit.> on the existence of CLTs for continuous time Markov chains. This result shows for a positive recurrent CTMC X_t, with stationary distribution π, and real valued function f, a CLT holds for the sample mean S(t) = 1/t ∫_0^t f_X_sds, t≥ 0, a CLT of the form t^1/2[S(t) - π(f)]  N(0, σ^2(f) when certain conditions are satisfied for X_t and f. We apply this theorem to our setting, letting X_t be the state of our CTMC and f be the identify function, i.e., f(x) = x for any state x. 

    
    We need to check that the conditions X_t is strongly ergodic and _π[f(x)^2]<∞ hold. First note that the Markov chain is ergodic, since it is possible to transition between any two states in finite time. Further, since the state space is finite, the expected hitting time of any state is finite and so ergodicity implies strong ergodicity <cit.>. The second condition _π[f(x)^2]<∞ holds since the state space is finite. Thus a CLT exists. 

    Step 2: Characterizing the mean.
    We analyze the treatment patients and the control patients separately. We start with the treatment patients. Recall from Lemma <ref> that
    
    1/N∑_i=1^NY_M,N(u,T)/T
            = 1/NT∫_i=0^T X_M,N(t)dt.

    In other words, we can calculate the proportion of time non-verified Y_M,N(u,T), averaged across users, as the time average of the queue length, normalized by the number of patients N. By the Ergodic theorem for Markov chains <cit.>, queue length time average converges to the expected length of the queue under the steady state distribution K_M,N and so the average proportion of time that users spend verified in the treatment group converges to 1 - K_M, N_1/N_1.

    The control group consists of N_0 individuals that evolve independently. For any individual u, this individual switches between two states x(u,t)=0 and x(u,t)=1. By the Ergodic theorem, proportion of time that u spends verified on [0,T] converges to the expectation in steady state that the user is in state x(u,t)=1. Thus the proportion of time each user u spends verified converges to τ/(λ+τ), and the average across all N_0 users is the same quantity.

    
    Step 3: Characterizing the variance.
    Note that the set of treatment patients is independent from the set of control patients, so we can analyze the behavior of the two systems separately. 
    For both systems, we first characterize the variance of the time average of the queue length and then relate this variance to the variance of the estimator. 

    We start with the treatment patients. For the treatment patients, the queueing system is a birth death chain. We use the following result from <cit.>, attributed to <cit.>, for the asymptotic variance for a birth death process.





Let X(t) be a birth and death process on a subset of integers { 0, …, n} with birth and death rates λ_i and μ_i, respectively. Let Y(t) = f(X(t)) where f is a real-valued function on the state space of the BD process X(t) and let f̅ = ∑_i=0^n π_i f(i) be the steady state mean. Define the time average (sample mean)  (Y(t) = t^-1∫_0^t Y(s) ds for s>0 and define the asymptotic variance to be
σ̃^2 = lim_t→∞ t  (Y(t)).

Then

    σ̃^2 = 2 ∑_j=0^n-11/λ_j π_j[ ∑_i=1=0^j (f(i) - f̅)π_i ]^2.


Note that the average number of jobs in the queue can be written in the above formulation as f(i) = i and then we can directly use the formula in our setting. 

Let σ̃^2_M,N be the asymptotic variance of the time average queue length in a system with M servers and N users. Then

    σ̃^2_M,N = 2 ∑_j=0^N-11/(N-j) π_M,N(j)[ ∑_i=0^j (i - K̅)π_M,N(i) ]^2

where K itself is a function of π_M,N. 

Since the time average proportion of time verified for the treatment group is simply the time average of the queue length normalized by N_1, we get that the treatment group variance contribution is 

    σ̃^2 = σ̃^2(M,N_1, N_0) = 2/N_1^2∑_j=0^N_1-11/(N_1-j) π_M,N_1(j)[ ∑_i=0^j (i - K̅_M, N_1)π_M,N_1(i) ]^2.


For the control group, patients are independent from each other and so we can begin by analyzing a single patient. A single patient forms a birth death chain with only two states {0, 1}. Thus we can use Equation (<ref>) to calculate the time average of the given patient's verification proportion. For this simple two state case, the resulting expression simplifies to 

    2 λτ/(λ + τ)^3.

All control users are independent and so the average across users has variance 

    1/N_0·2 λτ/(λ + τ)^3.


Since the treatment and control groups are independent from each other, the asymptotic variance of the estimator is the sum of the respective asymptotic variance for the treatment and control groups. Thus completes the proof. 

Proof of Proposition <ref>

Note that when Equation (<ref>) holds, 

    dz/dt<0     when λ/(μ p + λ + τ)<z
    dz/dt=0     when λ/(μ p + λ + τ)=z 
    dz/dt>0     when λ/(μ p + λ + τ)>z.


Similarly, when Equation (<ref>) holds, 

    dz/dt<0     when  (λ - Mμ p)/(λ+τ)<z
    dz/dt=0     when (λ - Mμ p)/(λ+τ)=z 
    dz/dt>0     when (λ - Mμ p)/(λ+τ)>z.


Now we consider three cases, Case (i) r<M, Case (ii) r = M, and Case (iii) r>M.




We work out the steady state of the differential equation for Case (i). The other two cases can be proved similarly. 
If z<r, then Equation (<ref>) holds and dz/dt < 0. If z = r, Equation (<ref>) holds and dz/dt = 0. If z > r, then there are two cases: (a) r<z < M and (b) r < M < z. Under scenario (a) r<z < M, Equation (<ref>) holds and dz/dt - λ (1-z) - Mμ p - z τ. dz/dt < 0  z > (λ - Mμ p ) / (λ + τ). One can verify through straightforward algebra that M > r implies that r > (λ - Mμ p ) / (λ + τ). Combining this expression with the fact that z > r implies that dz/dt<0 in this case. Thus when r < M, the only steady state where dz/dt=0 is when z=r = λ / (λ + τ + μ p). 


§ ADDITIONAL RESULTS
















Proposition <ref> also implies that when the experiment is run with a different M or N_1 than the intended levels M' and N' when the intervention is scaled up, then it is possible that the estimator from the experiment θ̂^*(M, N) does not recover the treatment effect at scale-up θ^*(M'/N') and the estimator is biased. We characterize the bias in the following result. 


    Fix an experiment design with M, N_1, and N_0. Suppose that the platform wants to estimate the treatment effect θ(M', N') of the intervention with M' providers and N' patients, with M'≥ M and N' ≥ N_1. 
    The bias in the mean field estimator is given by
    
    θ̂^*(M / N_1) - θ^*(M'/ N')
            =
            
                0    if M/N ≤ r ,  M'/N'>r
                
    λ/λ + τ + μ p  - λ - (M'/N') ·μ p /λ + τ   if M/N>r ,  M'/N'<r  
                 
    λ - (M/N) ·μ p /λ + τ - λ/λ + τ + μ p   if M/N < r ,  M'/N'>r
                  
    λ - (M/N) ·μ p /λ + τ - λ - (M'/N') ·μ p /λ + τ   if M/N<r ,  M'/N'<r.


An immediate result of Proposition <ref> is that a platform that runs a study with M providers and N_1 users, obtains the estimate θ̂^*(M, N_1) and naively wishes to scale up the platform to N>N_1 users without increasing the capacity M may be overestimating the effect θ^*(M,N). 

The proof follows immediately from the mean field steady state expression in Equation (<ref>).







§ COMPARISON OF SIMULATIONS WITH APPROXIMATIONS


In Sections <ref> and <ref>, we use the asymptotic approximations for the mean and variance of the estimator θ̂(M, N_1, N_0, T) suggested by Theorem <ref>. We refer to these approximations as the CLT approximations. These approximations fix finite M, N_1, N_0 but consider the asymptotic behavior as T→∞. Additionally, in Section <ref> Proposition <ref> we use an additional large system mean field approximation in order to characterize how the effect size and estimator change with M and N. This approximation first scales up the size of the system such that M→∞ and N →∞, with M/N →M for some constant M. Then we consider the steady state of the mean field approximation as T→∞. 

These approximations are convenient, as the the finite M, N_1, N_0 approximation motivated by Theorem <ref> allows us to characterize the behavior at different M, N_1, N_0 without running computationally heavy simulations. The mean field approximation further allows us to prove the non-linearity in the decrease in the effect size in Proposition <ref> when increasing the ratio of users to providers. 

We now verify that these methods give us reasonable approximations of a system with finite M, N_1, N_0 and finite time horizon T. This boils down to confirming that the average queue length and the variance in the average queue length in simulations is similar to the quantities given by the approximations. 


We run simulations with system parameters λ=0.185, μ=7, τ=0.16, p=0.085 and time horizon T=150. We N and M across simulation instances. For each simulation instance with server and user pair (M,N), we run 500 simulation runs and compute. For each run i, we analyze the system at different time horizons T and compute the time average of the queue length on the interval [0,T], which we denote B_i(T). For each T, we then plot the sample mean and variance of B_i(T) across simulation runs i, with bootstrapped 95% confidence intervals, and compare with the mean given by the mean field limit approximation and the mean and variance given by the CLT approximation. 




Figure <ref> compares the mean of the estimator across simulation runs with (a) the mean field approximation and (b) the mean given by the CLT approximation. The simulation mean coincides with the CLT approximation mean in all instances. The simulation mean coincides with the mean field approximation mean in all instances except (M,N)=(20, 100) and (M,N)=(40, 200). We attribute this difference the behavior of queues with finite M and N. We note that the ratio M/N=0.2 in both of these instances is very close to the critical ratio of r=0.196 for this combination of λ, τ, μ, p. In the limit as M→∞ and N→∞ and M/N→M=0.2, the system is in the Quality Driven regime. However, recall that for finite M and N, the square root staffing rule suggests that M = rN + β√(N), which implies that M/N must be larger than N plus some “buffer” in order to be in the Quality Driven regime. We conjecture that the finite systems with (M,N)=(20, 100) and (M,N)=(40, 200) are still operating in the Efficiency Driven regime, thus contributing to the difference between the mean field approximation and the simulation mean. We expect that this difference decreases for larger M and N.

Figure <ref> also compares the treatment effect approximations given by the CLT and mean field limit and similarly shows that the two coincide except for small differences near the QED regime.



Figure <ref> compares the variance of the estimator value across simulation runs with the variance variance given by the CLT approximation. Note that the mean field limit is a deterministic system with no variance, and so we omit the mean field approximation from this comparison. In all instances shown, the mean of the simulation variance approaches the variance from the CLT approximation as T increases. For most instances, the 95 % confidence interval contains the CLT approximation for T≥ 20.

 Paper Title: A decomposition algorithm for two-stage stochastic programs with approximate rotational invariance.




Lifelong Person Search
    Jae-Won Yang, Seungbin Hong, and Jae-Young Sim, Member, IEEE

    August 5, 2024
=================================================================



§ ABSTRACT 
We propose an algorithm of approximating the optimal objective value of a two-stage stochastic program under an assumption of approximate rotational invariance of the technology matrix, and compare the method with the L-shaped decomposition.

	
	

roman


arabic






§ INTRODUCTION

Consider the general formulation of a two-stage stochastic program 

    maximize  ⟨ c,x ⟩ + _ξ [Q(x,ξ)] 
       subject to  Ax ≤ b 
        x ≥ 0.

where for each realization ξ of a random vector ξ, Q(x,ξ) is the optimal objective value of the recourse problem

    maximize  ⟨ q(ξ),y(ξ) ⟩
       subject to  T(ξ)x +W(ξ)y(ξ) ≤ h(ξ) 
        y(ξ) ≥ 0.

In program (<ref>), c, x ∈^n_1 are the first stage cost and variable vectors,  A ∈^m_1 × n_1 encodes first stage constraints, and b ∈^m_1 denotes the right hand side of the constraints.
In the recourse problem (<ref>), q(ξ), y(ξ) ∈^n_2 are the second stage cost and variable vectors, and the technology matrix T(ξ) ∈^m_2 × n_1  along with the recourse matrix W(ξ) ∈^m_2 × n_2 determine the quantitative relationship between the first and the second stage variables. The vector h(ξ) ∈^m_2 denotes the right hand side of the second stage constraints.  


Combining the recourse problem together with the first stage constraints, and  considering all realizations of ξ, we obtain the so-called 
extensive form or deterministic equivalent of problem (<ref>):  

    maximize    ⟨ c,x ⟩ + ∑_ξ∈(ξ)  ⟨ q(ξ), y(ξ) ⟩
    subject to     Ax ≤ b 
        T(ξ) x +W(ξ) y(ξ) ≤ h(ξ) 
        x ≥ 0 
        y(ξ) ≥ 0.

The two-stage problem (<ref>) can be approached by considering the equivalent deterministic problem (<ref>) directly using optimization solvers.
However, problem (<ref>) can be computationally expensive when the number of scenarios (i.e distinct realizations of ξ) is large.



To resolve the issue of computational inefficiency, various decomposition algorithms have been proposed and employed in practice.
The L-shaped method  <cit.> is the most widely used decomposition algorithm for solving two-stage stochastic programs. The method employs Benders' decomposition  <cit.>. 
The algorithm starts by defining a master problem with only the first-stage decision variables. After the master problem is solved, the first-stage variables are fixed at an optimal solution of the master problem. Then the second stage problem can be decomposed into || subproblems where  denotes the set of scenarios. The subproblems can be solved in parallel, and valid inequalities can be derived and added to master problem. The master problem is then solved again and the algorithm iterates. 
The algorithm is known to converge in a finite number of steps when the second-stage decision variables are continuous and the number of scenarios is finite.

A related decomposition technique is the Lagrangian method  <cit.> where non-anticipa­tivity constraints are typically enforced through the use of Lagrangian multipliers (dual variables) associated with those constraints. The Lagrangian relaxation of the problem incorporates those constraints along with the original constraints and the objective function. The dual variables associated with the non-anticipativity constraints capture the price of violating those constraints, and they are updated iteratively as part of the decomposition process.



Apart from Benders' decomposition and its variations, a substantial body of literature has emerged presenting a variety of decomposition techniques for two-stage stochastic programs.
In particular, one can mention stochastic decomposition  <cit.>, subgradient decomposition  <cit.>, level decomposition  <cit.>, and partition-based algorithms  <cit.>. 






















Existing decomposition algorithms are typically designed for generic technology matrices, which define the relationship between the first and the second stage variables. However, what happens if the technology matrix has a special structure? Can one exploit that structure for efficient decomposition of stochastic programs? 
In the present work, we focus on two-stage stochastic programs where the technology matrices satisfy an approximate rotational invariance property introduced below. For such programs, we propose a method which is based on decoupling the first and the second stages.



§ APPROXIMATION OF THE OPTIMAL OBJECTIVE: THEORETICAL GUARANTEES


In this section, we consider a rigorous argument which shows that under certain property of the recourse problems which we call an approximate rotational invariance, the optimal objective value of the two-stage stochastic problem can be approximated by decoupling the first and second stages.
In what follows, denote by x̃ an optimal solution
to the first stage problem

    maximize    ⟨ c,x ⟩
    subject to     Ax ≤ b 
        x ≥ 0.

Let us describe precisely the assumptions we will employ.




Let ε be a small positive parameter. We assume all of the following.


  (a) First stage solutions: The first stage solution set is feasible and bounded, and for each τ≥ 0, the convex quadratic
optimization problem

    maximize  ⟨ c,x_τ⟩
       subject to  Ax_τ≤ b 
       x_τ_2 ≤τ
        x_τ≥ 0

is feasible and has a unique solution;

  (b) Feasibility and boundedness of the recourse problem: For every realization of ξ and for every
choice of a vector
x∈^n_1 with x_2≤max{x'_2: Ax'≤ b, x'≥ 0}, the recourse problem (<ref>)
is feasible and bounded;

  (c) Approximate rotational invariance:
For every
choice of a vector
x∈^n_1 with x_2≤max{x'_2: Ax'≤ b, x'≥ 0},

    |_ξ Q(x,ξ)-
    _ξ Q((x_2,0,…,0)^⊤,ξ)|
    ≤ε.


  (d) Approximate monotonicity of _ξ Q((·,0,…,0)^⊤, ξ)
on an interval:
The mapping

    τ⟶_ξ[Q((τ,0,…,0)^⊤, ξ)]

satisfies

    _ξ[Q((τ_1,0,…,0)^⊤, ξ)]
    ≥_ξ[Q((τ_2,0,…,0)^⊤, ξ)]
    -ε,

whenever
x̃_2≤τ_1≤τ_2≤max{x_2: Ax≤ b, x≥ 0}.


Observe that the assumption (b) above is standard, and that the assumption (a) can be realized for any bounded feasible first-stage problem by adding a small random perturbation to the cost vector c. Thus, the main conditions above are properties (c) and (d). It is not clear apriori if property (c) is satisfied 
for any stochastic program except for the degenerate case where the technology matrix is zero. However, there is a strong indication that the approximate rotational invariance holds for recourse problems with matrices T(ξ) which are realizations of random Gaussian matrices (see a remark after the proof below).



We have the following result:

Assume that conditions (a)–(d) above
are satisfied for some choice of ε.
Consider the quantity

    ẑ^*:=max{⟨ c,x̃_τ⟩+_ξ
    [Q((x̃_τ_2,0,…,0)^⊤ ,ξ)], 
    0≤τ≤max{x_2: Ax≤ b, x≥ 0}},

where for each τ, x̃_τ is the optimal solution to the convex
problem (<ref>),
and Q((x̃_τ_2,0,…,0)^⊤,ξ) 
is given by (<ref>) with the vector (x̃_τ_2,0,…,0)^⊤
in place of x.
Then |z^*-ẑ^*|≤ 2ε.


Let τ≥ 0 be such that

    ẑ^*=
    ⟨ c,x̃_τ⟩+_ξ
    [Q((x̃_τ_2,0,…,0)^⊤ , ξ)].

By the approximate rotational invariance property,

    ẑ^*≤⟨ c,x̃_τ⟩+_ξ
    [Q(x̃_τ , ξ)]+ε,

and hence

    ẑ^*≤ z^*+ε.


Conversely, let x^* be an optimal solution of (<ref>), so that

    z^*= ⟨ c,x^* ⟩+_ξ
    [Q(x^* , ξ)].

Again applying the approximate rotational invariance, we get

    z^*≤⟨ c,x^* ⟩+_ξ
    [Q((x^*_2,0,…,0)^⊤ , ξ)]+ε.

If x^*_2 ≤x̃_2 then, setting τ:=x^*_2,
we get x̃_τ_2=τ (to verify that, assume for contradiction that x̃_τ_2<τ and
consider a linear interpolation between x̃_τ and x̃), and
hence

    ⟨ c,x^* ⟩+_ξ
    [Q((x^*_2,0,…,0)^⊤ , ξ)]
    ≤⟨ c,x̃_τ⟩+_ξ
    [Q((x̃_τ_2,0,…,0)^⊤ , ξ)]
    ≤ẑ^*,

implying that

    z^*≤ẑ^*+ε.

If x^*_2 >x̃_2 then, by the approximate monotonicity,

    ⟨ c,x^* ⟩+_ξ
    [Q((x^*_2,0,…,0)^⊤ , ξ)]
    ≤⟨ c,x̃⟩+_ξ
    [Q((x̃_2,0,…,0)^⊤ , ξ)]+ε,

and

    z^*≤ẑ^*+2ε.

The result follows.



Assume that {T(ξ)}_ξ∈ are realizations of independent identically distributed random matrices with i.i.d centered Gaussian entries in every row. Recall that any linear combination of a set of jointly Gaussian random variables is Gaussian.
Therefore, if x and x' are any pair of fixed vectors in ^n_1 having the same Euclidean norm, then the sets of random vectors

    { T(ξ)x }_ξ and { T(ξ)x' }_ξ

are equidistributed. This suggests that the expected objective value of the recourse problem in this setting is influenced essentially by the norm of a vector x and not by its direction. 
Therefore, we can “replace” any x in the recourse problem by the first standard basis vector times the norm x_2, without significantly altering the optimal objective of the entire two-stage program. Although the above reasoning is not rigorous, our experimental results (see further) suggest that recourse problems with Gaussian technology and recourse matrices indeed have an approximate rotational invariance property.



In practical implementations of the above
decoupling procedure, instead of considering a continuum
of admissible values of τ∈[0,
max{x_2: Ax≤ b, x≥ 0}] we will use
a discretization τ=kΔ, 1≤ k≤ K,
where Δ>0 is sufficiently small, and
K∈ is chosen in such a way that KΔ≥x^*_2.
For each choice of τ=kΔ, we compute
the optimal solution x_τ^* to the program (<ref>),
and then estimate _ξ[Q((x_τ^*_2,0,…,0) ,ξ)].












§ THE DECOUPLING ALGORITHM
 

We will assume that our stochastic program satisfies assumptions (a)–(d) from the previous section, for some small ε>0.
Let us start by summarizing the main points of the decoupling procedure, before presenting a formal algorithm.
The decoupling procedure for the first and second stages involves the following steps.


  *  Discretizing the norm of the first stage vector x:
        For a discretization step size Δ, and for any integer k in a certain interval,
we consider the constraint x_2 ≤τ together with the first stage constraints of (<ref>), to obtain the optimization problem (<ref>). The discretization step ensures that the optimal objective z^* of (<ref>) can be approximated using our method in a finite number of steps.
         

  *  Solving the first stage problem:
        With the additional constraint x_2 ≤τ, where τ = k Δ, we solve the first stage problem above to find an optimal solution x̃_τ. This step involves optimizing the convex nonlinear program (<ref>) (the convexity ensures that the problem can be solved efficiently).


  *  Fixing the first stage variable in the recourse problem:
        In the recourse problem, we fix the variable x to the optimal solution x̃_τ obtained at the previous step:
        
    maximize  _ξ [ ⟨⟨ q(ξ),y(ξ) ⟩ ] 
       subject to   W(ξ)y(ξ) ≤ h(ξ) - T(ξ)x̃_τ
        y(ξ) ∈^n_2
       ξ∈.


  *   Reformulating the recourse problem:
        The recourse problem (<ref>) involves -T(ξ)x̃_τ on the right hand side. Given the assumption of approximate rotational invariance, T(ξ)x̃_τ can be replaced with T(ξ)
        (x̃_τ_2,  0, …, 0  )^⊤
        at expense of only a small distortion of the optimal objective of the second stage. As a result, a modified problem (<ref>) reads as follows 
        
    maximize  _ξ [ ⟨ q(ξ),y(ξ) ⟩ ] 
       subject to   W(ξ)y(ξ) ≤ h(ξ) - _1(T(ξ))x̃_τ_2 
        y(ξ) ∈^n_2
       ξ∈.

        where _1(T(ξ)) is the first column of the matrix T(ξ).
Note that this crucial step essentially decouples the first and the second stage variables as the solution of the modified recourse problem only depends on the Euclidean norm of x̃_τ and not its direction.
Furthermore, since for τ≤x̃_2 we have x̃_τ_2=τ, in that range of τ the decoupling is complete, and the problems (<ref>) and (<ref>) can be solved in parallel.


The decoupling procedure is presented in Algorithm <ref>. In this algorithm, K is an upper bound for k which is initialized with a sufficiently large number and is updated at the beginning of the algorithm according to the norm of optimal objectives of (<ref>). Further, S is the set of scenarios, Z_1 and Z_2 are vectors for storing first and second stage optimal objectives, respectively, and X is a vector indexed over [0,…,K] that stores the norms of x̃_Δ k, for each admissible k.  







§ COMPUTATIONAL STUDIES


To evaluate the performance of the proposed algorithm, we conduct a series of tests with various sizes of matrices A, T and W, and compare it with both the extensive form and Benders' decomposition. 
Recall that the extensive form combines all possible scenarios into a single large scale linear problem. The optimal objective of this problem serves as a benchmark to measure the accuracy of our algorithm.
Benders' decomposition, a widely used decomposition algorithm for two stage stochastic programs, is implemented as a second benchmark. We evaluate the Benders decomposition in terms of computational time.
We solve the extensive form and the subproblems of Benders' decomposition and our decoupling algorithm with help of the optimization solver Gurobi 10.1.1.

The third benchmark, which we refer to as the naive decoupling, consists in fixing the first stage variables to the optimal solution of problem (<ref>), denoted as x̃, and solving the recourse problems for that choice of the first-stage variables: 

    maximize    ⟨ c,x̃⟩ + ∑_ξ∈(ξ)  ⟨ q(ξ), y(ξ) ⟩
    subject to     T(ξ) x̃ +W(ξ) y(ξ) ≤ h(ξ) 
        y(ξ) ≥ 0.

By comparing the optimal objective value of (<ref>) with that of the extensive form, we gain insights into the parameter configurations where our algorithm proves to be effective.


These metrics are useful for assessing the relative efficiency of our proposed algorithm.
The computational results are presented in Table <ref>.
In the table, m_1 and n_1 are the number of rows and columns of the matrix A, and  m_2 and n_2 are the number of rows and columns of the matrix W. 
We consider constant vectors h and vary the magnitude of the components for different choices of the dimension of the matrices.
Ngap is the relative gap in percentage between the optimal objective of problem (<ref>) and the optimal objective of the extensive form, while Dgap is the relative gap in percentage between the optimal objective obtained by our algorithm and the optimal objective of the extensive form. 
Columns t_e, t_d and t_b denote the computational times for solving the extensive form, executing our algorithm and Benders decomposition algorithm respectively.

Input data for the experiments in Table <ref> are as follows: 


  * b = 1, Δ = 0.01, K=100

  * A is a realization of a standard Gaussian matrix. The realization is fixed and remains the same in all experiments.

  * c, q are realizations of Gaussian random vectors normalized to have c_2 = 0.5 and q_2=1. The realizations are fixed and remain the same in all experiments.

  * T(ξ), W(ξ) are realizations of independent standard Gaussian matrices.

  * Number of scenarios = sample size = 50

  * Each number in the 6th to 10th column of the table is the empirical average over 50 runs for the given parameters.

  * Compared to the description in Algorithm <ref>, the range of k in the main cycle for the second stage problem is optimized to reduce the computational time.





From the table, we observe that Dgap is of order 2 %, indicating that the optimal objective of our decoupling algorithm is in close proximity to that of the extensive form solution. 
We must note that in certain parameter settings, the approximation accuracy of our algorithm matches that of the naive decoupling. For example, when the number of the first stage variables is small, both the naive approach and our decoupling algorithm yield similar objective values. Determining the range of parameters where our algorithm significantly outperforms the naive decoupling remains an open question. 

Although solving the extensive form exhibits better performance in terms of the computational time, our decoupling algorithm outperforms Benders' decomposition for some choice of parameters.
For fair comparison we run Benders' decomposition algorithm up to 2% gap between the objective obtained from the master problem and the objective of current incumbent. 


 
















§ CONCLUSION


In this paper, we exploit the structure of the technology matrix to effectively decouple the first and second stages of two-stage stochastic programs. Our theoretical guarantee and empirical studies, demonstrate that when the technology matrix satisfy an approximate rotational invariance property, it is possible to decouple the two stage stochastic program efficiently. The optimal objectives of our proposed algorithm are in close proximity to the optimal objectives of the extensive form solution.
Further studies could involve testing the algorithm on larger problem instances. The question of choosing parameters when our decoupling algorithm significantly outperforms the naive decoupling remains open. 
We hope that our findings will inspire the discovery of new decomposition techniques leveraging the structure of the technology matrix.


































 

plainnat
















 Paper Title: Global convergence of a modified BFGS-type method based on function information for nonconvex multiobjective optimization problems.


	
	
	
	Global convergence of a modified BFGS-type method based on function information for nonconvex multiobjective optimization problems
    
		Ying Xue yang 
	
    August 5, 2024
==================================================================================================================================



	
	Abstract: 
	 We propose a modified BFGS-type method based on function information for nonconvex multiobjective optimization problems (MFQNMO). The method employs a common BFGS matrix to approximate the Hessian matrix of all objective functions in each iteration. This matrix is updated using function and gradient information from the previous step. Our analysis confirms the convergence of the method without relying on convexity assumptions. We also establish a local superlinear convergence rate for MFQNMO under mild conditions and validate its effectiveness through experiments on both nonconvex and convex test problems.
	
	

	Keywords: Multiobjective optimization, quasi-Newton methods, nonconvex, Wolfe line search, global convergence.
	
	
	

	AMS subject classifications: 
	
	
	
	

§ INTRODUCTION
 
	In this paper, we consider the following unconstrained multiobjective optimization problems (<ref>):
	
	
    min_x∈ R^nf(x)=(f_1(x),f_2(x),…,f_m(x))^T,MOP

	
	where f_i(x):R^n→ R is continuously differentiable function, for any i∈[m]. Multiobjective optimization is widely used in many research areas, which can be found in engineering <cit.>, environmental analysis <cit.>, machine learning <cit.>, etc.

	In 2000, Fliege and Svaiter proposed the steepest descent methods for multiobjective optimization <cit.>. Building on their pioneering work, researchers have conducted extensive studies on the first-order descent algorithm for solving (<ref>), (see e.g. <cit.>). The first-order algorithm is often favored for its lower cost compared to the second-order method, as it avoids the computational burden of determining the second-order Hessian matrix. However, the trade-off is a typically slower convergence rate, particularly evident in scenarios involving ill-conditioned problems. 

	The ill-conditioned problem may exhibit large curvature variations at certain points, leading to highly unstable curvature values. The Newton method <cit.> is effective in such cases as it captures the local curvature of functions and adjusts the direction more efficiently, leading to faster rate compared to first-order algorithms. Fliege et al. <cit.> proved the superlinear convergence rate of the Newton method. However, despite its efficiency, the computational cost per iteration is notably high, particularly for large-scale problems. 

	To balance the computational cost and the effectiveness, the quasi-Newton method is considered. Different from single objective optimization, the quasi-Newton method <cit.> uses different matrices to approximate the Hessian matrix of individual objective functions in each iteration to capture the curvature information. Numerous variants of the quasi-Newton method have been proposed. Lai et al. <cit.> proposed a quasi-Newton method for constructing approximate matrices using q-derivate. In <cit.>, the global convergence of the quasi-Newton method is established by the average nonmonotone line search. Morovati et al. <cit.> extended both the self-scaling BFGS (SS-BFGS) method and the Huang BFGS (H-BGFS) method for multiobjective optimization problems. Prudente and Souza <cit.>, operating under the assumption of no convexity, proposed a modified BFGS formula. They demonstrated that their algorithm is well-defined under the Wolf line search without convexity assumption. Different from the case of single-objective, the computational cost per iteration still be high when the number of objective functions increases for multiobjective optimization problems. To address this, Ansary and Panda <cit.> utilized a common matrix to approximate all Hessian matrices in each iteration. Lapucci and Mansueto <cit.> extended the limited memory Quasi-Newton approach for MOPs (L-BFGS). However, all the improved algorithms mentioned above require convexity conditions to ensure global convergence. In <cit.>, the global convergence can be guaranteed with Wolf line search in the nonconvex case. The approximate Secant equation in reference <cit.> depends on some parameters, and updating the BFGS formula under strong convexity conditions does not always ensure a satisfactory approximation of the Secant equation. 

	For MOPs, there are relatively few studies on quasi-Newton methods in nonconvex cases. This raises the question of whether alternative methods exist that can balance between algorithmic convergence and per-iteration cost for nonconvex MOPs. In this paper, inspired by <cit.>,  we proposed a modified BFGS-type method based on function information for nonconvex MOPs (MFQNMO). By utilizing both function and gradient information from the previous iterations, we construct a common modified BFGS-type formula to approximate the Hessian matrices of all objective functions in each iteration. We verify that the modified BGFS-type matrix is positive definite for each iteration and establish a global convergence for MFQNMO without convexity assumption. 

	The structure of this paper is as follows: In sect. <ref>, we give some notations and definitions for our later use. Sect. <ref> recalls several multiobjective gradient descent methods. In sect. <ref>, we introduce the proposed MFQNMO method and establish its convergence analysis. Numerical results are presented in sect. <ref>. At the end of the paper, we draw some conclusions.
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	

§ PRELIMINARIES
 
	
	We give some notations used in this paper.

	∙ [ m ]={ 1,2,⋯ ,m }.

	
	∙ Δ_m={λ :∑_i∈[ m ]λ_i=1,λ_i≥ 0, i∈[ m ] }.

	
	∙ · is the Euclidean norm in R^n.

	
	∙ Jf( x )∈R^m× n, ∇f_i( x )∈R^n ,_i(x) the Jacobian matrix, the gradient of f_i and the Hessian matrix at x, respectively.

	∙ 𝒟(x,d) max_i ∈ [m]∇ f_i(x)^⊤ d.

	∙ For all  B∈ R^n× n, B ≻ 0  is equivalent to  B  is positive definite.

	∙ For x,y∈ R^n, ⟨ x,y⟩_B:=x^⊤ B y and x_B:=√(⟨ x,x⟩_B).

	We define partial order for our later use:
	
    A≼ B ⇔ B-A ≽ 0,   A,B∈ R^n× n

	
    f( y )≤ f( x )⇔f_i( x )-f_i( y )≥ 0,  i∈[ m ].

	
	
	 <cit.>
		A point x^*∈R^n is called the Pareto (weakly Pareto ) solution for <ref>, if there exists no point x∈R^n such that f( x )≤ f( x^*) and f( x ) f( x^*) ( f( x )<f( x^*) ).
	
	
	 <cit.>
		A point x^*∈R^n is called a Pareto critical point for <ref>, if exists i∈[ m ] such that
		⟨∇f_i( x^*),d ⟩≥ 0, for all d∈R^n.
	
	
	 <cit.>
		A vector d∈R^n is called descent direction for f at x^*, if 
		⟨∇f_i( x^*),d ⟩ <0,
		for all i∈[ m ].
		
	
	
	
	<cit.>
		The following statements hold:

		(i)  if x^∗ is local weak Pareto optimal, then x^∗ is a Pareto critical point for f;

		(ii)  if f is convex and x^∗ is Pareto critical for f, then x^∗ is weak Pareto optimal;

		(iii)  if f is strictly convex  and x^∗ is Pareto critical for f, then x^∗ is Pareto optimal.
		
	
	
	

§ GRADIENT DESCENT METHODS FOR MOPS.
 
	In this section, we revisit the steepest descent method <cit.> and the quasi-Newton method <cit.> for MOPs.

	
	

 §.§ The steepest descent method for MOPs.

	For a given x∈R^n, consider the following subproblem (<ref>): 
	
	
    d∈R^nmin i∈ [m]max ⟨∇f_i( x ),d ⟩ +1/2 d ^2.

	
	Since ⟨∇f_i( x ),d ⟩ +1/2 d ^2 is strongly convex for i∈[ m ], then subproblem (<ref>) has a unique minimizer. We denote that θ( x ) and d( x ) are the optimal value and the optimal solution in problem (<ref>), respectively. Therefore,
	
    d( x )=d∈R^nargmin i∈ [m]max ⟨∇f_i( x ),d ⟩ +1/2 d ^2,

	
	and
	
	
    θ( x )=d∈R^nmin i∈ [m]max ⟨∇f_i( x ),d ⟩ +1/2 d ^2.

	
	The problem can be transformed into the following equivalent optimization problem (<ref>), which is easier to solve.
	
    [               (t,d)∈ℝ×ℝ^nmin                   t + 1/2d^2;                         s.t. ⟨∇ f_i(x), d⟩≤ t,    i∈ [m]. ]

	According to the Slater condition, strong duality is established. The dual problem has the same minimum value as (<ref>). We consider that the dual problem of problem (<ref>):
	
	
    λ∈Δ_mmax ( t,d )∈ R×R^nmin L( λ ,( t,d ) ),

	where L( λ ,( t,d ) ) is a Lagrange function,
	
    L( λ ,( t,d ) )=t+1/2 d ^2+i∈[ m ]∑ λ_i( ⟨∇f_i( x ),d ⟩ -t ).

	
	According to the KKT condition, we can obtain the steepest descent direction:
	
	
    d_SD( x )=-i∈[ m ]∑ λ_i^SD( x )∇f_i( x ),

	
	where λ_SD( x )=( λ_1^SD( x ),λ_2^SD( x ),… ,λ_m^SD( x ) )^Tis the solution of the following dual problem
	
	
    -λ∈Δ_mmin 1/2i∈[ m ]∑ λ_i∇f_i( x ) ^2,

	
	owing to strong duality, we have:
	
	
    θ_SD( x )=-1/2i∈[ m ]∑ λ_i^SD( x )∇f_i( x ) ^2=-1/2 d_SD( x ) ^2≤ 0,

	
	
    ⟨∇f_i( x ),d_SD( x ) ⟩≤ - d_SD( x ) ^2≤ 0.

	The following lemma describes the proposition of θ_SD( x ) and d_SD( x ).
	
	 <cit.> 
		For problem (<ref>), θ_SD( x ) and d_SD( x ) are the optimal value and solution, respectively. Then,

		(a) It can be conclude that the equivalent relation is as follows.

		(i) The point x∈R^n is not Pareto critical;

		(ii) d_SD( x ) 0;

		(iii) θ_SD( x )<0.

		(b) The function d_SD( x ) and θ_SD( x ) are continuous, for all x∈R^n.
	
	
	

 §.§ The quasi-Newton methods for MOPs (QNMO).

	Quasi-Newton direction d_q(x) in <cit.> is the optimal solution to the following subproblem:
	
    min_d∈ R^nmax_i∈ [m]∇ f_i (x)^⊤ d + 1/2d^⊤ B_i(x) d,

	where B_i(x)∈ R^n× n,  approximates the  Hessian matrix _i(x) and  B_i(x)≻ 0, for i∈ [m]. ∇ f_i (x)^⊤ d + 1/2d^⊤ B_i(x) d is strongly convex, for i∈ [m]. The subproblem <ref> has a unique minimizer. We denote that θ_q ( x ) and d_q( x ) be the optimal value and solution in problem (<ref>), respectively,
	
    d_q(x) = argmin_d ∈ R^nmax_i∈ [m] ∇ f_i (x)^⊤ d + 1/2d^⊤ B_i(x) d,
 
	and
	
    θ_q(x) = max_i ∈ [m] ∇ f_i (x)^⊤ d(x) + 1/2d(x)^⊤ B_i(x) d(x).

	(<ref>) is equivalent to the convex quadratic optimization problem:
	
    [                                     (t,d)∈ℝ×ℝ^nmin                                                  t;                                               s.t. ∇ f_i (x)^⊤ d + 1/2d^⊤ B_i(x) d  ≤ t,    ∀ i∈ [m]. ]

	According to the KKT condition, we can obtain the quasi-Newton direction:
	
    d_q(x) = - [ ∑_i=1^mλ_i^q(x)B_i(x) ]^-1 (∑_i=1^mλ_i^q(x)∇ f_i (x)),

	where λ_q( x )=( λ_1^q( x ),λ_2^q( x ),… ,λ_m^q( x ) )^Tis the solution of the following dual problem
	
	
    -λ∈Δ_mmin 1/2i∈[ m ]∑ λ_i∇f_i( x ) ^2_[i∈[ m ]∑ λ_iB_i(x)]^-1.
 
	Owing to strong duality, we have
	
    θ_q(x) = -1/2 d_q(x)^⊤[ ∑_i=1^mλ_i^q(x)B_i(x) ] d_q(x).

	

 §.§ The modified quasi-Newton methods for MOPs (MQNMO).

	Even if quasi-Newton avoids calculating the second-order Hessian matrix, the per-iteration cost  is still expensive. Recently, Ansary and Panda <cit.> proposed a modified quasi-Newton method (MQNMO) that utilized a single positive matrix to approximate all the Hessian matrices. The Modified quasi-Newton direction d(x) is the optimal solution to the following subproblem <ref>:
	
    min_d∈ R^nmax_i∈ [m]∇ f_i (x)^⊤ d + 1/2d^⊤ B(x) d,

	where B(x)∈ R^n× n and  B(x)≻ 0. Problem (<ref>) is equivalent to the convex quadratic optimization problem as follows 
	
    [                                   (t,d)∈ℝ×ℝ^nmin                                                t;                                             s.t. ∇ f_i (x)^⊤ d + 1/2d^⊤ B(x) d  ≤ t,    ∀ i∈ [m]. ]

	According to the KKT condition, we can obtain the modified quasi-Newton direction
	
    d(x) = - B(x)^-1( ∑_i=1^mλ_i(x)∇ f_i (x) ),

	and
	
    θ(x) = -1/2 d(x)^⊤ B(x)d(x),

	where λ( x )=( λ_1( x ),λ_2( x ),… ,λ_m( x ) )^Tis the solution of the following dual problem (<ref>)
	
	
    -λ∈Δ_mmin 1/2i∈[ m ]∑ λ_i∇f_i( x ) _B(x)^-1^2.

	According to the above analysis, we can get
	
    𝒟(x_k,d_k) ≤ -d_k_B_k^2

	The following rank one DFP formula of generating B_k+1 is defined in <cit.> by
	
    B_k+1 = (I-y_ks_k^⊤/s_k^⊤ y_k) B_k (I-s_k y_k^⊤/s_k^⊤ y_k) + y_ky_k^⊤/s_k^⊤ y_k,

	where s_k = x_k+1-x_k and y_k = ∑_i=1^m λ_i^k(∇ f_i(x_k+1)-∇ f_i(x_k)).
	
	
	

§ MFQNMO:MODIFIED BFGS METHOD BASED ON FUNCTION INFORMATION FOR MOPS.
   
	
	
	In this section, we propose a modified BFGS method based on function information for MOPs and analyze its properties. The formal description of the algorithm is as follows. 
	
	
	
		In step 1, solving the descent direction through the dual problem of subproblem <ref>, it is noted that the initial BFGS matrix is selected as a positive definite matrix, usually chosen as the unit matrix. Step 2 is the termination criterion of the algorithm. In step 3, we search the step size through the Wolfe line search. In steps 5-6, we notice that the BFGS-type update formula  (<ref>) and (<ref>) is different from <cit.>. We use the previous iterative function information to update the m_k adaptively. It can note that 
		
    B_k+1s_k = y_k + m_ks_k.

		
	
	The following result shows that Algorithm <ref> is well-defined without convexity assumption. 
	
	 The BFGS matrix (<ref>) updated by Algorithm <ref> is positive definite.
	
	
		At the k-th iteration, we only need to consider the case when x_k is not the Pareto critical point. By the definition of γ_k, η_k and m_k, we have
		
    γ_k^⊤ s_k 
    			    = y_k^⊤ s_k + m_ks_k^2 
        =(y_k^⊤ s_k/s_k^2 + m_k )s_k^2 
        = (η_k + m_k)s_k^2 
        = [η_k + max{-η_k, 0}  + ∑_i=1^m λ_i^k (f_i(x_k)-f_i(x_k+1))]s_k^2 
       ≥∑_i=1^m λ_i^k (f_i(x_k)-f_i(x_k+1))s_k^2 
        > 0.

		Then for all nonzero vector x∈ R^n,
		
    x^⊤ B_k+1 x = x_B_k^2-⟨ x,s_k ⟩_B_k^2s_k_B_k^2+(x^⊤γ_k)^2γ_k^⊤ s_k,

		it follows by the Cauchy-Schwarz inequality and (<ref>) that
		
    x^⊤ B_k+1 x ≥(x^⊤γ_k)^2γ_k^⊤ s_k≥ 0.

		
		Inequality <ref> holds as an equality when x^⊤γ_k = 0 and x= cs^k, where c is a constant. It follows that  cs_k^⊤γ_k=0. By (<ref>), we can obtain that c=0, which contradicts the fact that x is a nonzero vector. 
		
	
	
		For any z∈ R^n, the level set { x:f(x)≤ f(z) } is a bounded set. 
	
	Next, we show that the Wolf line search in Algorithm <ref> can terminate at a finite step.
	
		Suppose Assumption <ref> holds and d_k is the descent direction. Then there exists an interval of values [α_1, α_2], with 0<α_1< α_2, such that for all α∈ [α_1, α_2], inequalities (<ref>) and (<ref>) hold.
	
	
		The proof is similar to that in <cit.>, we omit it here.
	
	
	
		Let θ(x) and d(x) be the optimal value and solution in problem (<ref>). Then we have:

		(a) Vector d(x) 0 is a descent direction.

		(b) The following conditions are equivalent:

		(i) The point x∈R^n is not Pareto critical;

		(ii) d(x) 0;

		(iii) θ(x)<0.

		(c) If there exists  K ⊂ N  such that x_k K⟶ x^*, d_k K⟶ 0, then x^* is a Pareto critical point. 
	
	
		These assertions can be derived by using the same proof demonstrated in the  <cit.>. 
	
	
	
	

 §.§ Global convergence
 
	In this section, we present the global convergence analysis of Algorithm <ref> without assuming convexity. First, we require the following assumptions.
	
	
	
	
		(i) For i ∈ [m], ∇ f_i(x) is Lipschitz continuous with constant L_i. 

		(ii)For any x∈ R^n, there exists constant a and b such that
		
    aI_n ≼ B(x) ≼ bI_n.

		
	In the subsection, the following properties imply that under the Wolfe line search, the objective function will be significantly reduced along the direction generated by Algorithm <ref>.
	
	
		Let B_k be generated by the BFGS update formula (<ref>) in Algorithm <ref>. If there exists constants c_1 and c_2 such that γ_k^⊤s_k/s_k^2≥ c_1 >0, γ_k^2/γ_k^⊤s_k≤ c_2. Then, for any p∈ (0,1), there exists a constant δ>0 such that,  for any k∈ N,
		
    cosβ_j = s_j^⊤B_js_j/s_jB_js_j≥δ,

		holds for at least ⌈ pk ⌉ values of  j ∈ [1,k].
		
	
		The proof is similar to that in <cit.>, we omit it here.
	
	
	
		Let {x_k} be the sequence generated by Algorithm <ref>. Suppose that Assumption <ref>  holds. Then, there exists constants c_1 and c_2 such that
		
    γ_k^⊤s_k/s_k^2≥ c_1d_k_B_k^2 ,

		
    γ_k^2/γ_k^⊤s_k≤c_2/d_k_B_k^2.

	
	
	
		First, inequality (<ref>) can be deduced from Wolfe line search, (<ref>) and  (<ref>), i.e.,
		
    f_i(x_k)-f_i(x_k+1) ≥ -σ_1α_k 𝒟(x_k,d_k) ≥σ_1α_kd_k_B_k^2 ,

		
    γ_k^⊤ s_k ≥∑_i=1^m λ_i^k (f_i(x^k)-f_i(x^k+1))s_k^2 .

		We denote c_1=σ_1α_1>0, it follows by Proposition <ref> that (<ref>) holds.
		Next, we consider that inequality (<ref>). By the definition of the <ref>, 
		
    γ_k   ≤y_k+m_ks_k
       ≤(y_k/s_k + m_k)s_k
       ≤[y_k/s_k +max{-η_k, 0}  + ∑_i=1^m λ_i^k (f_i(x^k)-f_i(x^k+1)) ]s_k.

		Since ∇ f_i(x) is Lipschitz continuous with constant L_i, for i∈ [m], we can get
		
    y_k/s_k≤∑_i∈ [m]λ_i^k∇ f_i(x_k+1)-∇ f_i(x_k)/x_k+1-x_k≤max_i∈ [m] L_i.

		By the Cauchy-Schwarz inequality, 
		
    |⟨ y_k,s_k ⟩|/s_k^2≤y_k/s_k.

		Therefore, 
		
    y_k/s_k + max{-η_k, 0}≤2y_k/s_k≤ 2max_i∈ [m] L_i.

		Assumption <ref> and Assumption <ref> (i) imply the existence of  f_1∈ R such that f_i(x_k) ≥ f_1 for all k≥ 0 and i ∈ [m]. It follows by (<ref>) and (<ref>) that
		
    γ_k≤ (2max_i∈ [m] L_i + max_i∈ [m](f_i(x_0)-f_1))s_k,

		hence, 
		
    γ_k^2/γ_k^⊤s_k≤c_2/d_k_B_k^2,

		where c_2 = (2max_i∈ [m] L_i + max_i∈ [m](f_i(x_0)-f_1))^2/c_1.
		
	
	
		β_k is the angle between s_k and B_ks_k. By Lemma <ref> and Proposition <ref>, we can obtain that the angles β_k remain away from 0, for an arbitrary fraction p of the iterates.
	
	
	
		Let {x_k} be the sequence generated by Algorithm <ref>. Then,
		
    𝒟(x_k,d_k) ≤ -cosβ_k d_kd_SD^k.

	
	
		Since s_k = x_k+1-x_k = α_kd_k, then
		
    cosβ_k = s_k^⊤B_ks_k/s_kB_ks_k = d_k^⊤B_kd_k/d_kB_kd_k.

		It follows that
		
    𝒟(x_k,d_k) 
    			   ≤ -d_k_B_k^2  
        = -cosβ_k d_kB_kd_k
        = -cosβ_k d_k∑_i ∈ [m]λ_i^k ∇ f_i(x_k)
       ≤ -cosβ_k d_kd_SD^k.

		This completes the proof. 
		
	
	The following result establishes that the sequence {x_k,d_k} satisfies the Zoutendijk condition, which is the basis for establishing the global convergence of Algorithm <ref>. 
	
	 
		Let {x_k} be the sequence generated by Algorithm <ref>. Suppose that Assumption <ref> and Assumption <ref> hold. Then 
		
    ∑_k≥ 0𝒟(x_k,d_k)/d_k^2 < ∞.

	
	
		The proof is similar to that in <cit.>, we omit it here.
	
	The following theorem establishes the global convergence of Algorithm <ref>. 
	
	
		Let {x_k} be the sequence generated by Algorithm <ref>. Suppose that Assumption <ref> and Assumption <ref> hold. Then, the following statements hold.   

		(i)  lim inf_k→∞d_SD^k = 0,

		(ii)  Every accumulation point of the sequence {x_k} is Pareto critical.
	
	
	
		Assume that there exists a constant ε >0 such that
		
    d_SD^k≥ε,   ∀ k ∈ N.

		By the definition of d_k and Assumption <ref> (iii), we get
		
    -1/2d_k_B_k^2 
    			    = min_d∈ R^nmax_i∈ [m]∇ f_i(x_k)^⊤d + 1/2d_B_k^2  
       ≤min_d∈ R^nmax_i∈ [m]∇ f_i(x_k)^⊤d + b/2d^2 
       ≤1/bmin_d∈ R^nmax_i∈ [m]∇ f_i(x_k)^⊤(bd) + 1/2bd^2 
        = -1/2bd_SD^k^2.

		Set C_1 = c_1ε^2/b and C_2 = c_2 b/ε^2, it follows by Proposition <ref> that
		
    γ_k^⊤s_k/s_k^2≥ C_1 >0,

		
    γ_k^2/γ_k^⊤s_k≤ C_2.

		Then, by Lemma <ref>  and Lemma <ref>, there exist a constant δ>0 and a infinite subsequence of indices 𝒦⊂ N such that 
		
    -𝒟(x_k,d_k)/d_k≥δd_SD^k≥δε,  ∀ k∈𝒦,

		it follows that
		
    ∑_k∈ N𝒟(x_k,d_k)^2/d_k^2≥∑_k∈𝒦𝒟(x_k,d_k)^2/d_k^2≥∑_k∈𝒦δ^2ε^2 = ∞,

		this contradicts the result of Lemma <ref>. Then,
		
    lim inf_k→∞d_SD^k = 0.

		Assumption <ref> implies there exists a infinite subsequence of indices K_1 ⊂N such that x_k K_1⟶ x^* and d_SD^kK_1⟶ 0. By the Lemma <ref>, d_SD(x) is continuous, we can get
		
    d_SD(x^*) = 0.

		Furthermore,  x^* is Pareto critical.
	
	
	
	
	

 §.§ Superlinear convergence analysis 
 
	
	In this section, we consider the local superlinear convergence of Algorithm <ref> under Assumption <ref>. 
	
		(i) F is twice continuously differentiable.

		(ii) For i∈ [m], {γ_k} satisfies ∑_k >0γ_k < ∞.

		(iii) There exists constants U,L>0  such that UI_n ≼_i(x) ≼ LI_n. 

		(iv) _i(x) is Lipschitz continuous with constant M, i∈ [m]. 

		(v)For any x∈ R^n, there exists constant a and b such that
		
    aI_n ≼ B(x) ≼ bI_n.

	
	The Assumption <ref> (iii) implies that  f_i(x) is strongly convex  with modulus U and _i(x) is Lipschitz continuous with constant L, for i∈ [m].
	In the previous section, we obtain the weak convergence of the sequence {x^k} generated by Algorithm <ref>. Under strict convexity, the following theorem demonstrates that the sequence {x^k} generated by Algorithm <ref> strongly converges to the Pareto optimal point x^*.
	
	
		Let {x_k} be the sequence generated by Algorithm <ref>. Suppose that f(x) is strictly convex, Assumption <ref> and Assumption <ref> hold. Then, {x_k} converges to Pareto optimal point x^* of (<ref>).
	
	
		The proof is similar to that in <cit.>, we omit it here. 
	
	Next, we investigate the R-linear convergence of Algorithm <ref>. 
	
	
		Suppose that Assumption <ref> holds and  {x_k} is the sequence generated by Algorithm <ref>. Then, for any k≥ 0,

		(i)  x_k - x^*≤2/Ud_SD^k,

		(ii)  s_k≥1-σ_2/Lcosβ_kd_SD^k.
		
	
	 
		The two assertions can be obtained by using the same arguments as in the proof of <cit.>, we omit them here.
	
	
		Let {x_k} be the sequence generated by Algorithm <ref>. Suppose that  Assumption <ref>  holds. Then, there exists constant r such that
		
    γ_k^⊤s_k/s_k^2≥ U ,

		
    γ_k^2/γ_k^⊤s_k≤r^2/U.

	
	
		By the definition of γ_k, s_k and m_k > 0, we have 
		
    γ_k^⊤s_k/s_k^2 = (y_k + m_ks_k )^⊤s_k/s_k^2 = y_k^⊤s_k/s_k^2 +m_k≥ U.

		Since each objective function is U-strongly convex, for  i ∈ [m] and any x∈R^n, f_i(x) has a lower bound f_i^min. 

		It follows from the proof of Proposition <ref> that 
		
    γ_k≤  rs_k,

		where r = 2max_i∈ [m] L_i + max_i∈ [m](f_i(x_0)-f_i^min).
		
	
	
		Suppose that Assumption <ref> holds and  {x_k} is the sequence generated by Algorithm <ref>. Let x^* be defined as Theorem <ref>. Then,

		(i)  {x_k}  converges R-linearly to  x^*,  

		(ii)  ∑_k≥ 0x_k-x^* < ∞.
	
	
	
		The proof of the theorem is similar to the corresponding discussion in <cit.>, so we omit the proof here. 
	
	
	
		Using the proof of Theorem <ref>, it can directly deduce that a constant r∈ (0,1) exists such that
		
    0<∑_i∈ [m]λ_i^k(f_i(x_k)-f_i(x_k+1)) < r^k, ∀ k≥ 0,

		where λ_k=( λ_1^k,λ_2^k, 
				…, λ_m^k)^Tis the solution of (<ref>).  In the strongly convex case, we can observe that η_k >0 and
		
		
    ∑_i∈ [m]λ_i^k(f_i(x_k)-f_i(x_k+1)) → 0,  k →∞,

		i.e., 
		
    m_k    = max{-η_k, 0}  + ∑_i=1^m λ_i^k (f_i(x_k)-f_i(x_k+1)) 
       = ∑_i=1^m λ_i^k (f_i(x_k)-f_i(x_k+1))→ 0, k→∞.

		in (<ref>). In this case, B_k+1s_k ≈ y_k. Then (<ref>) can be seen as an approximation of the secant equation in <cit.>. Remarkably, for strongly convex functions, both MFONMO and MQNMO in <cit.> may exhibit comparable convergence rates. This observation is supported by the numerical experiments presented in the following section. 
	
	
	Now, we show some useful tools that will be used in superlinear convergence analysis. These tools help to characterize the Dennis-Moré condition and demonstrate the convergence rate.

	Denote the average matrix G̅_i^k as follows
	
    G̅_i^k = ∫_0^1_i(x_k+τ s_k)dτ,   i∈ [m].

	Then,
	
    G̅_i^k s_k = μ_i^k.

	Next, we define another two auxiliary functions as follows 
	
    f_λ(x) = ∑_i∈ [m]λ_i^kf_i(x),

	
    G_λ^k = ∫_0^1∑_i∈ [m]λ_i^k _i(x_k+τ s_k)dτ.

	where λ_k=( λ_1^k,λ_2^k, 
			…, λ_m^k)^Tis the solution of (<ref>). 
	Let us define
	
    s̅_k = _λ(x^*)^1/2s_k,   y̅_k = _λ(x^*)^-1/2y_k,   γ̅_k = _λ(x^*)^-1/2γ_k,

	
    B̅_k = _λ(x^*)^-1/2B_k_λ(x^*)^-1/2,

	
    cosβ̅_k = s̅_k^⊤B̅_ks̅_k/s̅_kB̅_ks̅_k,  q̅_k =s̅_k^⊤B̅_ks̅_k/s̅_k^⊤s̅_k,  a̅_k = γ̅_k^⊤s̅_k/s̅_k^2,  b̅_k = γ̅_k^2/γ̅_k^⊤s̅_k.

	
	By the BFGS-tpye matrix updated fomula (<ref>), it can be noted that (<ref>) leads to the formula
	
    B̅_k+1 = B̅_k-B̅_ks̅_ks̅_k^⊤B̅_ks̅_k^⊤B̅_ks̅_k +γ̅_kγ̅_k^⊤γ̅_k^⊤s̅_k .

	Another useful tool used in <cit.>, given that the following function 
	
    ψ(B) = trace(B)-ln((B)),

	
    ψ(B̅_k+1) = ψ(B̅_k) +(b̅_k-ln(a̅_k
    		)-1)+ ln(cos^2β̅_k) + [1 - q̅_k/cos^2β̅_̅k̅ + ln(q̅_k/cos^2β̅_̅k̅)]

	
	In the next theorem, we prove that the Dennis-Moré condition of superlinear convergence holds, a similar result is discussed in <cit.> and <cit.>.  
	
	
		Suppose that Assumption <ref> holds. Let {x_k} be the sequence generated by Algorithm <ref>. Then
		
    lim_k →∞(B_k - _λ(x^*))s_k/s_k  = 0.

	
	
	
		By (<ref>) and λ_k ∈Δ_m, we can get
		
    G̅_λ^ks_k = y^k,

		hence,
		
    (G̅_λ^k-_λ(x^*))s_k = y^k - _λ(x^*)s_k,    k≥ 0.

		
		It follows by Assumption <ref> (iv), (<ref>) and the definition of y̅_k, s̅_k that
		
    y̅_k - s̅_k    = _λ(x^*)^-1/2 (G̅_λ^k-_λ(x^*))_λ(x^*)^-1/2s̅_k 
       ≤_λ(x^*)^-1/2^2 s̅_kG̅_λ^k-_λ(x^*)
       ≤ M_λ(x^*)^-1/2^2 s̅_k∫_0^1x_k + τ s_k - x^*dτ
       ≤c̅s̅_kε_k ,

		where c̅ = M_λ(x^*)^-1/2^2, ε_k = max{x_k+1 - x^*, x_k - x^*}.
		Since 
		
    |y̅_k - s̅_k | ≤y̅_k - s̅_k ,

		by (<ref>), we get
		
    (1-c̅ε_k)s̅_k≤y̅_k≤ (1+c̅ε_k)s̅_k,

		
    (1-c̅ε_k)^2s̅_k^2 -2y̅_k^⊤s̅_k +s̅_k^2 ≤y̅_k^2 -2y̅_k^⊤s̅_k +s̅_k^2 ≤c̅^2 s̅_k^2 ε_k^2.

		It follows that
		
    y̅_k^⊤s̅_k/s̅_k^2≥ 1-c̅ε_k.

		Using the Assumption <ref> (iii), and (<ref>) we obtain
		
    a̅_k = γ̅_k^⊤s̅_k/s̅_k^2 =  (y̅_k + m_k_λ(x^*)^-1s̅_k )^⊤s̅_k/s̅_k^2    = y̅_k^⊤s̅_k/s̅_k^2 + m_ks̅_k^⊤_λ(x^*)^-1s̅_k/s̅_k^2
       ≥ 1-c̅ε_k +m_k/L
       ≥ 1-c̅ε_k.

		By the L-Lipschitz continuity of f_i, we have 
		
    y_k/s_k≤ L.

		From (<ref>), (<ref>) and the right side of  (<ref>), 
		
    b̅_k 
    			    =  γ̅_k^2/γ̅_k^⊤s̅_k
        = y̅_k^2/γ̅_k^⊤s̅_k + 2m_ky̅_k^⊤_λ(x^*)^-1s̅_k/γ̅_k^⊤s̅_k + (m_k)^2s̅_k^⊤_λ(x^*)^-2s̅_k/γ̅_k^⊤s̅_k
       ≤(1+c̅ε_k)^2/1-c̅ε_k 
    			+ 2m_k_λ(x^*)^-1/2y_k_λ(x^*)^-1_λ(x^*)^1/2s_k/Us_k^2  + (m_k)^2_λ(x^*)^-2s̅_k^2/Us_k^2
       ≤ 1+3c̅+ c̅^2ε_k/1 - c̅ε_kε_k + η_1 m_k +η_2 m_k^2 ,

		where η_1 = 2_λ(x^*)^-1/2_λ(x^*)^-1_λ(x^*)^1/2 L/U, η_2 = _λ(x^*)^-2_λ(x^*)^1/2^2/U.
		Using (<ref>) and (<ref>),
		
    ψ(B̅_k+1) 
    			≤ψ(B̅_k) 
    			    +(3c̅+ c̅^2ε_k/1 - c̅ε_kε_k + η_1 m_k +η_2 m_k^2 -ln(1-c̅ε_k))
        + ln(cos^2β̅_k) +[1 -q̅_k/cos^2β̅_̅k̅ + ln(q̅_k/cos^2β̅_̅k̅)],

		By adding the above inequality from 0 to k, then we obtain
		
		
    0< ψ(B̅_k+1) 
    			≤ψ(B̅_1) 
    			    +∑_j=1^k(3c̅+ c̅^2ε_j/1 - c̅ε_jε_j + η_1 m_j +η_2 m_j^2 -ln(1-c̅ε_j)) 
        + [1 -q̅_j/cos^2β̅_̅j̅ + ln(q̅_j/cos^2β̅_̅j̅)] +ln(cos^2β̅_j),

		From Theorem <ref>, there exists k_0 ∈ N such that
		
    c̅ε_k < 1/2,  ∀ k > k_0,

		then, ln(1-c̅ε_k) ≥ -2c̅ε_k. It follows by Assumption <ref> (ii) that
		
    ∑_j>k_0(3c̅+ c̅^2ε_j/1 - c̅ε_jε_j + η_1 m_j +η_2 m_j^2 - ln(1 - c̅ε_j)) 
    			    < ∑_j>k_0 9c̅ε_j + ∑_j>k_0 (η_1 m_j +η_2 m_j^2) 
        < ∑_j=1^∞9c̅ε_j + ∑_j=1^∞(η_1 m_j +η_2 m_j^2)
        < ∞.

		By (<ref>) and (<ref>), we get
		
    ∑_j=1^∞(-[1 -q̅_j/cos^2β̅_̅j̅ + ln(q̅_j/cos^2β̅_̅j̅)] -ln(cos^2β̅_j) ) < ∞.

		
		Since -ln(cos^2β̅_j)>0 ,for all j>k_0, and h(t) = 1-t+lnt ≤0 when t>0, hence, we get
		
    -[1 -q̅_j/cos^2β̅_̅j̅ + ln(q̅_j/cos^2β̅_̅j̅)] ≥ 0,

		
    lim_k→∞ln(1/cos^2β̅_k)  = 0,  lim_k→∞[1 -q̅_k/cos^2β̅_̅k̅ + ln(q̅_k/cos^2β̅_̅k̅)] = 0.

		The above relations imply that
		
    lim_k→∞ln( cos^2β̅_j )  = 1,  lim_k→∞q̅_k = 1.

		Now,
		
    lim_k→∞(B_k - _λ(x^*))s_k^2/s_k^2    = lim_k→∞_λ(x^*)^-1/2(B_k - _λ(x^*))_λ(x^*)^-1/2s̅_k^2/_λ(x^*)^-1/2s_k^2
        =  lim_k→∞(B̅_k - I_n)s̅_k^2/s̅_k^2
        = lim_k→∞B̅_ks̅_k^2  - 2s̅_k^⊤B̅_ks̅_k + s̅_k^2/s̅_k^2
        = lim_k→∞[q̅_k/cos^2β̅_̅k̅ - 2q̅_k +1] 
        = 0

		
	
	
	
	
	
		Suppose that Assumptions <ref> hold. Let {x_k} be the sequence generated by Algorithm <ref>. Then, 

		(i) (<ref>) and (<ref>) hold with α_k=1 for sufficiently large k.

		(ii) {x_k} converge to x^* at a superlinear rate. 
	
	
	
		The proof is similar to that in <cit.>, we omit it here. 
	
	
	
	
	
	
	
	

§ NUMERICAL EXPERIMENTS
 
	
	In this section, we present some numerical experiments to verify the efficiency of MFQNMO. We compare MFQNMO with QNMO <cit.> with Wolfe line search and MQNMO <cit.> with Wolfe line search. The selected test problems include convex and nonconvex problems. Our evaluation criteria include the method's effectiveness in finding optimal solutions and its ability to approximate real Pareto surfaces accurately. These experiments were implemented using Python 3.9.

	For MQNMO and MFQNMO methods, we used the Frank-Wolf gradient method to solve the dual problem (<ref>).  For QNMO, (<ref>) is solved using scipy. optimize, which is an optimization Solver. All test problems and main characteristics are listed in Table <ref>. Columns `n' and `m' indicate the number of variables and objectives, respectively. The `Convex' column indicates whether the problem is convex or not. 'Y' stands for Yes. 'N' stands for NO.  Problems are solved under box constraints x_L≤ x≤x_U. The initial points are randomly selected within these box constraints x_L≤ x≤x_U. We used the termination criterion  | θ( x ) |≤10^-8 for all test problems. The maximum number of iterations is set to 500. To obtain the stepsize that satisfies the Wolfe conditions (<ref>) and (<ref>), we used the Wolf line search algorithm proposed in <cit.>. We used σ_1 = 10^-4, σ_2 = 0.1, and set B_0 = I_n, for all i∈ [m]. We perform 200 computations using the same initial points for all methods. 
	
	
	
	In Table <ref>, the average values obtained from these runs are recorded to determine the number of iterations (iter), function evaluations (feval), and CPU time (time(ms)). The number of points that did not converge in 500 iterations among the 200 initial points is recorded in column `NF' for MFQNMO, QNMO, and MQNMO. We denote 'Failed' as 'F,' which indicates that the corresponding method fails to solve the test problem under the stopping criterion. For large-scale problems, such as JOS, the QNMO method's CPU time consumption is substantial although it requires the fewest iterations. For all nonconvex test problems in Table <ref>,  the QNMO method exhibited poor performance and the MQNMO method surpasses the MFQNMO method only for the FF1 test problem. For certain problems in Table <ref>, the experimental efficiency of the MFQNMO method and the MQNMO method are equal, such as BK1, MOP1, MHHM1, and DGO1. We mentioned the possible reasons for this phenomenon in Remark <ref>. As can be seen from Table <ref>, the MFQNMO method is more efficient and competitive.
	
	Figure <ref> shows the Pareto frontier obtained using the MFQNMO method for the convex problem listed in Table <ref>. Figure <ref> shows the Pareto frontier obtained using the MFQNMO method for the nonconvex problem listed in Table <ref>. For the chosen test problems, it can be observed that the MFQNMO method was able to satisfactorily estimate the Pareto frontiers when considering evenly distributed starting points.  
	
	
	
	
	
	
	
	
	
	
	
	

§ CONCLUSION
 
	
	We propose a modified BFGS-type method based on function information for nonconvex multiobjective optimization problems (MFQNMO), in which the BFGS-type formula is updated by using the function information. To obtain the global convergence property, we only need the smoothness of the objective function. Compared with <cit.>, our work has the following advantages.

	(1) Instead of approximating the Hessian matrix of all objective functions, we use a common matrix to capture the curvature information of the objective function, which balances the effectiveness of the algorithm and the computational cost.

	(2) For nonconvex functions, most versions of quasi-Newton methods may not converge, so the MFQNMO method proposed in this paper for nonconvex unconstrained problems is meaningful.

	(3) Compared with <cit.>, the update of the BFGS-type formula does not depend on the selection of parameters, but only needs to use the information of past iterations.

	In our future work, we will consider applying the method to nonconvex constrained optimization problems.
	
	
	
	
	

	
	99
		
		engineering 1Pereira, J. L. J., Oliver, G. A., Francisco, M. B., Cunha Jr, S. S., & Gomes, G. F. (2022). A review of multi-objective optimization: methods and algorithms in mechanical engineering problems. Archives of Computational Methods in Engineering, 29(4), 2285-2308.
		
		engineering 2Marler, R. T.,  Arora, J. S. (2004). Survey of multi-objective optimization methods for engineering. Structural and multidisciplinary optimization, 26, 369-395. 
		
		evironment 1Leschine, T. M., Wallenius, H., Verdini, W. A. (1992). Interactive multiobjective analysis and assimilative capacity-based ocean disposal decisions. European Journal of Operational Research, 56(2), 278-289. 
		
		evironment 2Asha, L. N., Dey, A., Yodo, N.,  Aragon, L. G. (2022). Optimization approaches for multiple conflicting objectives in sustainable green supply chain management. Sustainability, 14(19), 12790. 
		
		machine 1Ye, F., Lin, B., Yue, Z., Guo, P., Xiao, Q., & Zhang, Y. (2021). Multi-objective meta learning. Advances in Neural Information Processing Systems, 34, 21338-21351. 
		
		machine 2Sener, O., & Koltun, V. (2018). Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31. 
		
		steepset J. Fliege, B.F. Svaiter, Steepest descent methods for multicriteria optimization,  Math. Meth. Oper. Res. 51 (2000), 479-494.
		
		Newton J. Fliege, L.M.G. Drummond, B.F. Svaiter, Newton’s method for multiobjective optimization, SIAM J. Optim. 20 (2009), 602-626.
		
		quasi-Newton_1 Ž. Povalej, Quasi-newton’s method for multiobjective optimization, J. Comput. Appl. Math. 255 (2014), 765-777.
		
		quasi-Newton_2Qu, S., Mark G., Felix T.C. Quasi-Newton methods for solving multiobjective optimization, Operations Research Letters, 39.5 (2011), 397-399.
		
		proximalH. Tanabe, E.H. Fukuda, N. Yamashita, Proximal gradient methods for multiobjective optimization and their applications, Comput. Optim. Appl. 72 (2019), 339-361.
		
		
		BBV. Morovati, L. Pourkarimi, H. Basirzadeh, Barzilai and Borwein’s method for multiobjective optimization problems, Numer. Algorithms. 72 (2016), 539-604.
		
		
		conditionalAssunção, P. B., Ferreira, O. P., Prudente, L. F. (2021). Conditional gradient method for multiobjective optimization. Computational Optimization and Applications, 78, 741-768.
		
		conjugateL.R. Lucambio Pérez, L.F. Prudente, Nonlinear conjugate gradient methods for vector optimization, SIAM J. Optim. 28 (2018), 2690-2720.
		
		modifiedAnsary, M. A., Panda, G. (2015). A modified quasi-Newton method for vector optimization problem. Optimization, 64(11), 2289-2306.
		
		q-quasi-NewtonK.K. Lai, S.K. Mishra, B. Ram, On q-quasi-Newton’s method for unconstrained multiobjective optimization problems, Mathematics 8 (2020), 616.
		
		nonmonotone quasi-NewtonN. Mahdavi-Amiri, F.S. Sadaghiani, A superlinearly convergent nonmonotone quasi-Newton method for unconstrained multiobjective optimization, Optim. Meth. Softw. 35 (2020), 1223–1247.
		
		review-quasi-NewtonV.Morovati, H.Basirzadeh, L. Pourkarimi, Quasi-Newton methods for multiobjective optimization problems, 4OR-Q. J. Oper. Res. 16 (2018), 261–294.
		
		wolf-quasi-NewtonL.F. Prudente and D. Souza, A quasi-Newton method with Wolfe line searches for multiobjective optimization, J. Optim. Theory Appl. 194 (2022), 1107–1140.
		
		Global quasi-NewtonPrudente, L. F., Souza, D. R. (2024). Global convergence of a BFGS-type algorithm for nonconvex multiobjective optimization problems. Computational Optimization and Applications, 1-39.
		
		limited BFGSLapucci, M., Mansueto, P. (2023). A limited memory Quasi-Newton approach for multi-objective optimization. Computational Optimization and Applications, 85(1), 33-73.
		
		toolByrd, R.H., Nocedal, J.: A tool for the analysis of quasi-Newton methods with application to unconstrained minimization. SIAM J. Numer. Anal. 26(3), 727–739 (1989)
		
		SDW. Stadler and J. Dauer. Multicriteria optimization in engineering: A tutorial and survey.Progress in Astronautics and Aeronautics, 150:209–209, 1993.
		
		PNRM. Preuss, B. Naujoks, and G. Rudolph. Pareto set and EMOA behavior for simple multimodal multiobjective functions. In T. P. Runarsson, H.-G. Beyer, E. Burke, J. J. MereloGuervás, L. D. Whitley, and X. Yao, editors, Parallel Problem Solving from Nature - PPSN IX, pages 513–522, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
		
		JOSY. Jin, M. Olhofer, and B. Sendhoff. Dynamic weighted aggregation for evolutionary multiobjective optimization: Why does it work and how? In Proceedings of the 3rd Annual Conference on Genetic and Evolutionary Computation, GECCO'01, page 1042–1049, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc.
		
		test reviewS. Huband, P. Hingston, L. Barone, and L. While. A review of multiobjective test problems and a scalable test problem toolkit. IEEE Trans. Evol. Comput., 10(5):477–506, 2006.
		
		LOVA. Lovison. Singular continuation: Generating piecewise linear approximations to paretosets via global analysis. SIAM Journal on Optimization, 21(2):463–490, 2011.
		
		SLCDTO. Schätze, M. Laumanns, C. A. Coello Coello, M. Dellnitz, and E.-G. Talbi. Convergence of stochastic search algorithms to finite size Pareto set approximations. J. Global Optim., 41(4):559–577, Aug 2008.
		
		LDTZM. Laumanns, L. Thiele, K. Deb, and E. Zitzler. Combining convergence and diversity in evolutionary multiobjective optimization. Evolutionary computation, 10(3):263–282, 2002.
		
		HilC. Hillermeier. Generalized homotopy approach to multiobjective optimization. J. Optim. Theory Appl., 110(3):557–583, 2001.
		
		KW2I. Kim and O. de Weck. Adaptive weighted-sum method for bi-objective optimization: Pareto front generation. Struct. Multidiscip. Optim., 29(2):149–158, Feb 2005.
		
		PBBMOChen, Jian, Liping Tang, and Xinmin Yang. "Preconditioned Barzilai-Borwein Methods for Multiobjective Optimization Problems."
		
		
		
		
		
		
		
		
		
		
		
	
	
	


