Optimal Policies for Convex Symmetric Stochastic Dynamic Teams and their Mean-field Limit
Sina Sanjari Serdar Yüksel



Optimal Policies for Convex Symmetric Stochastic Dynamic Teams and their Mean-field Limit
This research was supported by the Natural Sciences and Engineering Research Council (NSERC) of Canada. A summary of some of the results here is included in <cit.> presented at the 2019 IEEE Conference
on Decision and Control (CDC). The authors are with the Department of Mathematics and Statistics,
     Queen's University, Kingston, ON, Canada.
     Email: {16sss3,yuksel@queensu.ca}
    Sina Sanjari and Serdar Yüksel
    
===========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





This paper studies convex stochastic dynamic team problems with finite and infinite time horizons under decentralized information structures. First, we introduce two notions called  exchangeable teams and symmetric information structures. We show that in convex exchangeable team problems an optimal policy exhibits a symmetry structure. We give a characterization for such symmetrically optimal teams for a general class of convex dynamic team problems under a mild conditional independence condition. In addition, through concentration of measure arguments, we establish the convergence of optimal policies for teams with N decision makers to the corresponding optimal policies for symmetric mean-field teams with infinitely many decision makers.  As a by-product, we present an existence result for convex mean-field teams, where the main contribution of our paper is with respect to the information structure in the system when compared with the related results in the literature that have either assumed a classical information structure or a static information structure. We also apply these results to the important special case of Linear Quadratic Gaussian (LQG) team problems, where while for partially nested LQG team problems with finite time horizons it is known that the optimal policies are linear, for infinite horizon problems the linearity of optimal policies has not been established in full generality. We also study average cost finite and infinite horizon dynamic team problems with a symmetric partially nested information structure and obtain globally optimal solutions where we establish linearity of optimal policies.



Stochastic teams, average cost optimization, decentralized control, mean-field teams










§ INTRODUCTION AND LITERATURE REVIEW



Team problems consist of a collection of decision makers or agents acting together to optimize a common cost function, but not necessarily sharing all the available information. The term stochastic teams refers to the class of team problems where there exist randomness in the initial states, observations, cost realizations, or the evolution of the dynamics. At each time stage, each agent only has partial access to the global information which is defined by the information structure (IS) of the problem  <cit.>. If there is a pre-defined order in which the decision makers act then the team is called a sequential team. For sequential teams, if each agent's information depends only on primitive random variables, the team is static. If at least one agent's information is affected by an action of another agent, the team is said to be dynamic. Information structures can be further categorized as classical, partially nested, and non-classical. An IS is classical if the information of decision maker i (DM^i) contains all of the information available to DM^k for k < i. An IS is partially nested, if whenever the action of DM^k, for some k < i, affects the information of DM^i, then the information of DM^i contains the information of DM^k. An IS which is not partially nested is non-classical. A detailed review is presented in <cit.>. 


 
Obtaining structural results in team problems is important towards establishing both existence and computational/approximation methods for optimal policies. In this paper, we define the notion of exchangeable teams and symmetric information structures, and we show that, for convex exchangeable dynamic teams with finite horizons, optimal policies exhibit a symmetry structure (Theorem <ref>). For any number of DMs, this symmetry structure is more relaxed when compared with the symmetry results developed earlier, e.g. in <cit.> which focused on problems under a static information structure, and is applicable for dynamic teams which may not admit a static reduction, as long as convexity in policies holds for the team problem.

There have been many studies involving decentralized stochastic control with infinitely many decision makers. In particular, when the coupling among the decision makers is only through some aggregate/average effect, such problems can be viewed within the umbrella of mean-field games <cit.>, which were introduced as a limit model for non-cooperative symmetric N-player differential games with a mean-field interaction as N→∞. The solution concept in game theory is often Nash equilibrium, and often under various characterizations of it in dynamic Bayesian setups. In the context of decentralized stochastic control or teams, these would correspond to person-by-person optimal solutions, and hence not necessarily globally optimal solutions.

Nonetheless, on the existence as well as uniqueness and non-uniqueness results on equilibria, there have been several studies for mean-field games <cit.>. There have also been several studies for mean-field games where the limits of sequences of Nash equilibria have been investigated as the number of decision makers N →∞ (see e.g., <cit.>). We refer interested readers to <cit.> for a literature review and a detailed summary of some recent results on mean-field games. 

 

Some notable relevant studies from the mean-field literature are the following: In <cit.>, through a concentration of measures argument, it has been shown that sequences of ϵ_N- local (for each player) Nash equilibria for N player games converge to a solution for the mean-field game under exchangeability of the initial states and weak convergence of normalized occupational measures to a deterministic measure <cit.>. In <cit.>, assumptions on equilibrium policies of the large population mean-field symmetric stochastic differential games have been presented to allow the convergence of asymmetric approximate Nash equilibria to a weak solution of the mean-field game <cit.>. 

However, in these studies the information structures are restricted to the following models: In <cit.> the information structure is assumed to be static since strategies of each player are assumed to be adapted to the filtration generated by his/her initial states and Wiener process (also called distributed open-loop controllers in the mean-field games' literature <cit.>) (see Remark <ref> for details of this discussion). Convergence of Nash equilibria induced by closed-loop controllers to a weak semi-Markov mean-field equilibrium has been established in <cit.> for finite horizon mean-field game problems, where the classical information structure (i.e., what would be a centralized problem in the team theoretic setup) has been considered. For infinite horizon problems, in <cit.>, an example of ergodic differential games with mean-field coupling has been constructed such that limits of sequences of expected costs induced by symmetric Nash-equilibrium policies of N-player games capture expected costs induced by many more Nash-equilibrium policies including a mean-field equilibrium and social optima. In <cit.>, the classical information structure (a centralized problem) has been considered, where in <cit.> it has been assumed that players have access to all the history of states of all players but not controls (we note that in the team problem setup through using a classical result of Blackwell <cit.> in the case where each DM knows all the history of states of all DMs, optimal policies can be realized as one in the centralized problem where just the global state is a sufficient statistic). Moreover, under relaxed regularity conditions on dynamics and the cost function, a limit theory has been established for controlled McKean-Vlasov dynamics <cit.> under the classical information structure, where through a similar analysis as in <cit.>, it has been shown that the empirical measure of pairs of states and ϵ_N-open-loop optimal controls converges weakly as N →∞ to limit points in the set of pairs of states and optimal controls of the McKean-Vlasov problem. 

The above highlights the intricacies due to the information structure aspects: different from the aforementioned studies above, we consider information structures that are not necessarily static or classical. Also, in this paper, we work with global optimality and not only mean-field equilibria and we show the existence of a globally optimal policy for mean-field team problems. On the other hand, in our paper since we work under the convexity assumption, the information structure does not allow for the mean-field coupling in the dynamics. We also note that in prior work, <cit.>, we studied static teams where under convexity and more restrictive symmetry conditions, global optimality of a limit policy of a sequence of N-DM optimal policies has been established. 

In the context of stochastic teams with countably infinite number of decision makers, the gap between  person-by-person optimality (Nash equilibrium in the game-theoretic context) and global team optimality is significant since a perturbation of finitely many policies fails to deviate the value of the expected cost, thus  person-by-person optimality is a weak condition for such a setup, and hence the results presented in the aforementioned papers may be inconclusive regarding global optimality of the limit equilibrium.  
For teams and social optima control problems, the analysis has primarily focused on the LQG model or Markov chains where the centralized performance has been shown to be achieved asymptotically by decentralized controllers (see e.g., <cit.>).

We also obtain existence results on optimal policies for the setups considered. Compared to the results on the existence of a globally optimal policy in team problems where (finite) N-DM team problems have been considered <cit.>, we study convex team problems with countably infinite number of decision makers. 


Parts of our results in this paper correspond to LQG teams. In <cit.>, it has been shown that for teams with finite number of DMs, dynamic teams with a partially nested information structure can be reduced to a static one (<cit.>) where Radner's theorem concludes global optimality of linear policies for LQG team problems <cit.>. However, for average cost infinite horizon, partially nested, LQG dynamic team problems so far there has been no universal result establishing that a globally optimal policy is linear, time-invariant, and stabilizing, and this has been often imposed apriori: In <cit.>, the problem of designing a linear, time-invariant, stabilizing, state feedback optimal policy for decentralized ℋ_2-optimization problems, which satisfy the quadratically invariance property, has been addressed by reparametrizing the problem as a convex problem (via Youla parameterization). In <cit.>, it has been shown that for sequential team problems involving linear systems, quadratic invariance and the partially nested property are equivalent. For a class of partially ordered (POSET) systems, state space techniques have been utilized to obtain optimal, linear, time-invariant, state feedback controllers for ℋ_2-optimization problems with sparsity constraints <cit.>. A similar result has been established in <cit.> where linearity and time invariance have been imposed apriori.  In <cit.>, ℋ_2-optimization output feedback problems with two-players have been considered and optimality results have been established when the optimal policies are restricted to linear, time invariant, stabilizing policies. However, the results in <cit.> are inconclusive regarding global optimality. Our contribution here is to consider average cost infinite horizon dynamic team problems without restricting the set of policies to those that are linear, time-invariant, and stabilizing unlike the results in <cit.>. We note again that the optimality of linear policies for infinite horizon LQG problems is an open problem in its generality and we provide positive results for a class of such problems. 

Contributions. 
In view of the discussion above, our paper makes the following contributions.


  (i) We define a notion of exchangeable teams and symmetric information structures, and we show that, for convex exchangeable dynamic teams with finite horizons, optimal policies exhibit a symmetry structure (Theorem <ref>). For any number of DMs, this symmetry structure is more relaxed when compared with the symmetry results developed in <cit.> and is applicable for dynamic teams which may not admit a static reduction, as long as convexity in policies holds for the team problem.

  (ii) For convex mean-field teams with a symmetric information structure, through concentration of measure arguments, we establish the convergence of optimal policies for mean-field teams with N decision makers to the corresponding optimal policies for mean-field teams (see Theorem <ref>).

  (iii) We establish an existence result for the class of convex mean-field teams with a symmetric information structure (see Theorem <ref>) for finite horizon problems, where, as noted in the literature review, related results assumed more restrictive information structures which are either static or classical.

  (iv) We also apply our results to LQG dynamic teams for finite horizon problems (see Section <ref>). For LQG dynamic teams with a symmetric partially information pattern, we obtain an optimal policy for finite horizon problems (see Section <ref>). We also apply convex mean-field results to LQG mean-field teams with a symmetric partially nested information structure (see Section <ref>) and obtain a globally optimal policy. Building on the result above, we also obtain a globally optimal policy for average cost LQG team problems. 



The organization of the paper is as follows: we study convex exchangeable dynamic teams with finite horizons in Section <ref>, and we study mean-field teams in Section <ref>. We obtain globally optimal solutions for finite horizon problems with a symmetric partially nested information structure and LQG mean-field  teams in Section <ref>, and we discuss average cost LQG team problems with a symmetric information structure in Section <ref>, respectively.

Notation. ℝ and ℕ denote the set of real numbers and natural numbers, respectively. We denote trace of a matrix A as Tr(A). We denote that a random vector X is independent of a random vector Y by X  Y. We denote A^T as the transpose of a matrix A and A^(T) to show the dependence of a matrix A to T ∈ℕ. For any random variables z^1:N:=(z^1,…,z^N), we defined z^-i:=(z^1,…,z^i-1,z^i+1,…, z^N), and ℳ_r,q denotes the space of r × q matrices. 



 §.§ Preliminaries



In this section, we introduce Witsenhausen's Intrinsic Model for sequential teams <cit.> (we generalize this definition to infinite number of decision makers).
Consider sequential systems and assume the action and measurement spaces are standard Borel spaces, that is, Borel subsets of complete, separable and metric spaces. The Intrinsic Model for sequential teams is defined as follows.


  * There exists a collection of measurable spaces {(Ω,  F), (𝕌^i, U^i), (𝕐^i, Y^i), i ∈𝒩}, specifying the system's distinguishable events, and control and measurement spaces. The set 𝒩 denotes the collection of decision makers. The set 𝒩 can be a finite set {1,2,…, N} or a countable set ℕ. The pair (Ω,  F) is a
measurable space (on which an underlying probability may be defined). The pair (𝕌^i,  U^i)
denotes the Borel space from which the action u^i of DM^i is selected. The pair (𝕐^i, Y^i) denotes the Borel observation/measurement space.


  * There is a measurement constraint to establish the connection between the observation variables and the system's distinguishable events. The 𝕐^i-valued observation variables are given by y^i=h^i(ω,u^1:i-1), where u^1:i-1={u^k, k ≤ i-1} and h^is are measurable functions.

  * The set of admissible control laws γ= {γ^i}_i ∈𝒩, also called
designs or policies, are measurable control functions, so that u^i = γ^i(y^i). Let Γ^i denote the set of all admissible policies for DM^i and let Γ = ∏_i ∈𝒩Γ^i.

  * There is a probability measure P on (Ω,  F) describing the probability space on which the system is defined.


Under the intrinsic model, every DM acts separately. However, depending on the information structure, it may be convenient to consider a collection of DMs as a single DM acting at different time instances. In fact, in the classical stochastic control, this is the standard approach.  



§ FINITE HORIZON CONVEX DYNAMIC TEAM PROBLEMS WITH A SYMMETRIC INFORMATION STRUCTURE

In this section, we characterize symmetry in dynamic team problems. According to the discussion above, by considering a collection of DMs as a single DM (i=1,…,N) acting at different time instances (t=0,…,T-1), we define a team problem with (NT)-DMs as a team with N-DMs:


  (i) Let the observation and action spaces be Borel subsets of ℝ^n for a positive integer n and be identical for each DM (i=1,…,N) with 𝕐_i:=Y=∏_t=0^T-1𝕐^t, 𝕌_i:=U=∏_t=0^T-1𝕌^t, respectively. The sets of all admissible policies are denoted by Γ = ∏_i=1^NΓ_i=∏_i=1^N∏_t=0^T-1Γ^t.

  (ii) For i=1,…,N, y^i_t:=h_t^i(x_0^1:N, ζ^1:N_0:t, u_0:t-1^1:N) represents the observation of DM^i at time t (h_t^is are Borel measurable functions).

  (iii) Let (ζ^1:N):=(ζ^1,…,ζ^N) where ζ^i:=(x_0^i, ζ_0:T-1^i) denotes all the uncertainty associated with DM^i including his/her initial states. We assume that (ζ^i) takes values in Ω_ζ (where at each time instances t, it takes value in Ω_ζ_t). Let μ denote the law of ζ^1:N.

  (iv) Define the expected cost function of γ^1:N as 
J_N(γ^1:N)= E^γ^1:N[c(ζ^1:N,u^1:N)],
for some Borel measurable cost function c: ∏_i=1^N (Ω_ζ×U) →ℝ_+, where γ^1:N=(γ^1,γ^2,…,γ^N) and γ^i=γ^i_0:T-1 for i=1,…,N.


Now, we present the definition of symmetric information structures (note that symmetric information structures can be classical, partially nested, or non-classical).

Let the information of DM^i acting at time t be described as I_t^i:={y_t^i}. The information structure of a sequential N-DM team problem is symmetric if  


  (i) y^i_t=h_t(x_0^i,x_0^-i, ζ^i_0:t, ζ^-i_0:t, u_0:t-1^i, u_0:t-1^-i) where h_t is identical for all i=1,…,N (note that function's arguments depend on i).



We note that the above definition can be generalized to be applicable for teams with countably infinite DMs and infinite horizon problems.

The symmetric information structure can also be interpreted and defined as a graph, which has often been the common method to describe information structures in control theory, relating DMs and their information through directed edges. Consider G(V,μ) as a directed graph with V={1,…,NT} nodes and where μ⊂ V × V determines the directed edges between nodes; this represents the dependency notation in the information of nodes, i.e., (i,j) denotes a directed edge from i to j, i → j, it means u^i affects y^j through the relation y^i=h^i(ω,u^1:i-1) defined in the intrinsic model (see Section <ref>). We denote by ↓ j as the set of nodes i such that i → j (ancestors), and ↓↓ j={↓ j}∪{j}. Similarly, we can define descendants by ↑ j. We can define a collection of DMs as a single DM (i=1,…,N) acting at different time instances (t=0,…,T-1) on a graph with a symmetric information structure (two examples are shown in Fig. 2.1, and Fig. 2.2). Assume


  (i) there exists a node {i} (root node), ω_0. Each sub-graph represents a single DM acting at time instances t=0,…,T-1, and there exists a finite number of sub-graphs G_p(V̂,μ̂) such that ∪_p=1^NG_p∪{i}=G, where G_ps are isomorphic (see e.g., <cit.>) for all p=1,…,N, i.e., for every node with directed edges in each sub-graph there exists a unique node with identical directed edges in the corresponding sub-graphs, where V̂={0,…,T-1}, and G_p^k  refers to a node k in G_p for all p=1,…,N and k=0,…,T-1, 

  (ii) sharing of the information is symmetric across sub-graphs, i.e., for p,s=1,…,N, and k,j=0,…,T-1, and for every edge from a node G_p^k to a node G_s^j, there exists an edge from a node G_p^k to nodes G_-p^j ,  where G_-p^j denotes (G_1^j,…,G_p-1^j,G_p+1^j,…,G_N^j), and also there exist edges from nodes G_-p^k to a node G_p^j. 





Now, we present an exchangeability hypothesis on the cost function. First, we recall the definition of an exchangeable finite set of random variables.

Random variables (x^1,x^2,…,x^N) defined on a common probability space are exchangeable if for any permutation σ of the set {1,…,N} (a mapping σ : {1,…,N}→{1,…,N}), 

   1P((x^σ)^1 ∈A^1,(x^σ)^2 ∈A^2,…,(x^σ)^N∈A^N)=P(x^1 ∈A^1,x^2 ∈A^2,…,x^N∈A^N)

for any measurable {A^1,…, A^N} and (x^σ)^i:=x^σ(i) for all i∈{1,…,N}.


For any permutation σ of the set {1,…,N}, we have for all ω_0
 
c((ζ^σ)^1:N,(u^σ)^1:N)=c(ω_0, ζ^1:N,u^1:N),
 
 where (ζ^σ)^1:N=(ζ^σ(1),…,ζ^σ(N)) and (u^σ)^1:N=(u^σ(1),…,u^σ(N)).


Here, we recall some definitions and results from <cit.> on convexity of static and dynamic team problems required to follow the result in this paper.
<cit.>
An N-DM team problem (static or dynamic) is convex in policies if for any two team policies γ_T^1:N and γ̃_T^1:N in the set {γ_T^1:N∈Γ: J(γ_T^1:N)<∞}, and for any α∈ (0,1), we have
 
    J_T(αγ_T^1:N+(1-α)γ̃_T^1:N)≤α J_T(γ_T^1:N)+(1-α)J_T(γ̃_T^1:N).



The above definition can also be applied to infinite-horizon and/or teams with countably infinite number of DMs. We recall sufficient conditions for convexity of static and dynamic team problems following <cit.>.
<cit.>
Consider a sequential team problems, and assume action spaces are convex, and J(γ)<∞ for all γ∈Γ (or alternatively, restrict the set to those leading to the finite cost). Then 


  (i) for static team problems convexity of the cost function in actions is sufficient for convexity of the team problem in policies,

  (ii) for dynamic team problems with a static reduction, convexity of the team problem in policies is equivalent to the convexity of its static reduction. 

  (iii) in particular, for partially nested dynamic teams with a static reduction (more generally, for stochastically partially nested team problems <cit.>) if the cost function is convex in actions then for the reduced team problem with an equivalent information structure is convex on Γ.



The conditions above, however, are only sufficient conditions <cit.>. We note however that as a Corollary for (ii) above, for the LQG setup, under partial nestedness, convexity in policies hold as a consequence of Radner’s theorem; we will study this case in Section <ref>. On the other hand, not all LQG problems are convex: the celebrated counterexample of Witsenhausen <cit.> demonstrates that under non-classical information structures, even LQG problems may not be convex and optimal policies may not be linear.


 §.§ Optimality of symmetric policies for convex dynamic teams with a symmetric information structure

In the following, we define notions of exchangeable and symmetrically optimal teams analogous to <cit.> for dynamic teams. 
(Exchangeable teams)

An N-DM team is exchangeable if the value of the expected cost function is invariant under every permutation of policies of DMs, i.e., J_T(γ_T^1,γ_T^2,…,γ_T^N)=J_T((γ_T^σ)^1,…, (γ_T^σ)^N).


(Symmetrically optimal teams)

A team is symmetrically optimal, if for every given policy γ_T=(γ_T^1,…,γ_T^N), there exists an identically symmetric policy (i.e., each DM has the same policy, γ̃_T=(γ̃_T^1,…,γ̃_T^N) , and γ̃_T^i=γ̃_T^j for all i,j=1,…,N) which performs at least as good as the given policy.


The concepts of exchangeable and symmetrically optimal dynamic teams in this paper are generalizations of those for static teams in <cit.>. However, here, the value of the expected cost function may not be invariant under exchanging γ^i_t with γ^j_k for k≠t, k,t=0,…,T-1, and for i,j=1,…,N.


Here, we give a characterization for exchangeable and symmetrically optimal dynamic teams. 

Consider dynamic team problems with a symmetric information structure under Assumption <ref>. If


  (a) action spaces 𝕌_t are convex for all t=0, …, T-1, 

  (b) (ζ^1, …,ζ^N) are exchangeable, 

  (c) for all policies γ∈Γ, and for all A=A^1×…× A^N where A^i∈𝒴^i, 


1∏_t=0^T-1P(y^1:N_t   1∈A|x_0^1:N,ζ^1:N_0:t-1,y_0:t-1^1:N,γ^1_0(y^1_0),…, γ^1_t-1(y^1_t-1), γ^1_0(y^2_0), …, γ^N_t-1(y^N_t-1))

   1=∏_t=0^T-1∏_i=1^NP(y^i_t∈A^i|x_0^i,ζ^i_0:t-1,y_↓t^↓↓i,γ^↓↓i_↓t(y^↓↓i_↓t)),

where y_↓ t^↓↓ i:={y_p^j|u^j_p affects y^i_t ∀ p=0,…,t-1 and ∀ j=1,…, N} and (γ^↓↓ i_↓ t(y^↓↓ i_↓ t)) can be defined similarly, 


  (i) then, the team problem is exchangeable.

  (ii)  Furthermore, if the team problem is convex in policies (see Theorem <ref>), then the team is symmetrically optimal.



We first show that for any permutation σ∈ S, J_T((γ_T^σ)^1,…, (γ_T^σ)^N)=J_T(γ_T^1,…, γ_T^N), i.e., the team is exchangeable. We have,

   J_T((γ_T^σ)^1,…, (γ_T^σ)^N)

   =∫c(ζ^1:N, (γ_T^σ)^1(y^1),…,(γ_T^σ)^N(y^N))μ(dx_0^1:N, dζ^1:N_0:T-1)

               ×∏_t=0^T-1∏_i=1^NP(dy^i_t|x_0^i, ζ_0:t-1^i,y_↓t^↓↓i,(γ^σ)^↓↓i_↓t(y^↓↓i_↓t))

   =∫c((ζ^σ)^1:N, (γ_T^σ)^1((y^σ)^1),…,(γ_T^σ)^N((y^σ)^N))μ(d(x_0^σ)^1:N, d(ζ^σ_0:T-1)^1:N)

               ×∏_t=0^T-1∏_i=1^NP(d(y^σ)^i_t|(x_0^σ)^i, (ζ_0:t-1^σ)^i,(y^σ)_↓t^↓↓i,(γ^σ)^↓↓i_↓t((y^σ)^↓↓i_↓t))

   =∫c(ζ^1:N, γ^1_T(y^1),…,γ^N_T(y^N))μ(dx_0^1:N, dζ^1:N_0:T-1)

               ×∏_t=0^T-1∏_i=1^NP(dy^i_t|x_0^i, ζ_0:t-1^i,y_↓t^↓↓i,γ^↓↓i_↓t(y^↓↓i_↓t))

   =J_T(γ_T^1,…, γ_T^2),

where (<ref>) follows from condition (c). Equality (<ref>) follows from  exchanging y^i, ζ^i with (y^σ)^i, (ζ^σ)^i by relabeling them, respectively. Since the information structure is symmetric, (<ref>) and condition (b) imply (<ref>). Hence, the team is exchangeable.  Let γ^*_T=(γ_T^1*,…, γ_T^N*) be a given policy. Consider γ̃_T as a convex combination of all possible permutations of policies by averaging them. Since action spaces are convex by condition (a), γ̃_T is a control policy. Following from convexity of the cost function in policies, we have 
 
J_T(γ̃_T)   :=J_T(∑_σ∈S1/|S|γ^*,σ_T) ≤∑_σ∈S1/|S|J_T(γ^*,σ_T) 
=J_T(γ^*_T),
 
where |S| denotes the cardinality of the set S and the inequality above follows from the hypothesis that the team problem is convex on Γ and the last equality follows from exchangeability of the team problem. This implies that the team is symmetrically optimal and completes the proof.


Examples will be given in Section <ref> and Section <ref> where Theorem <ref> can be applied.
Here, we present the result for a class of problems that admit a static reduction (see <cit.>, <cit.>, <cit.>). 


Consider a dynamic team problem with a symmetric partially nested information structure (see Definition <ref>) which admits a static reduction. Under Assumption <ref>, and Assumptions (a), (b), (c) of Theorem <ref>, if the cost function is jointly convex in u^ 1,…,u^N P-almost surely, then the team is symmetrically optimal.

We note again that here by symmetry, we mean symmetry across the decision makers.

The proof follows from Theorem <ref>(iii) and Theorem <ref> since the team is convex on Γ under the static reduction which is equivalent to the dynamic problem.


Hence, it follows that if a static reduction of an exchangeable, symmetrically optimal, dynamic team exists, then it is exchangeable and symmetrically optimal.



§ CONVEX MEAN-FIELD TEAMS WITH A SYMMETRIC INFORMATION STRUCTURE


In the following, we establish global optimality results for convex mean-field teams with a symmetric information structure (that is not necessarily partially nested).  

Define state dynamics and observations as 

   x_t+1^i=f_t(x_t^i,u_t^i,w_t^i),

   y_t^i=h_t(x_0:t^i,u_0:t-1^i,v_0:t^i),

where functions f_t and h_t are measurable functions. The information structure of DM^i at time t is I_t^i={y^i_t}, and ζ_t^i:=(w_t^i, v_t^i) (with ζ_0^i:=(x_0^i, w_0^i, v_0^i)) denotes uncertainty corresponding to dynamics and observations at time t for DM^i which are exogenous random vectors in a standard Borel space. Denote 𝕏⊆ℝ^m, 𝕌⊆ℝ^m^', 𝕐⊆ℝ^m^'', 𝕎, and 𝕍 as the state space, action space, observation space and the space of disturbances of dynamics and observations of DMs at each time instances t=0, …, T-1, respectively, where m, m^', and m^'' are positive integers.
 
  
   Problem (𝒫_T^N, MF): Consider N-DM teams with the expected cost function of γ_T^1:N as 
  
   1J_T^N(γ_T^N)=1/N∑_t=0^T-1∑_i=1^N E^γ_T^1:N[c(ω_0, x_t^i,u_t^i,1/N∑_p=1^Nu_t^p,1/N∑_p=1^Nx_t^p)],

where ω_0:(Ω,ℱ) → (Ω_0,ℱ_0) is an exogenous random vector in the standard Borel space and γ_T^1:N=γ^1:N_0:T-1, and the cost function satisfies the following assumption.

  
  Problem (𝒫_T^∞, MF):
Consider mean-field teams with the expected cost function of γ_T as

   1J_T^∞(γ_T)=lim sup_N→∞J_T^N(γ_T),

where J_T^N(·) is defined in (<ref>), γ_T^i=γ^i_0:T-1 for i ∈ℕ  and γ_T={γ_T^i}_i ∈ℕ.

 Assume
 
 
  (a) function f_t:𝕏×𝕌×𝕎→𝕏 is continuous in its first and second arguments for all w_t^i and for each i∈ℕ and uniformly bounded, 
 
  (b) function h_t:∏_k=0^t𝕏×∏_k=0^t-1𝕌×∏_k=0^t𝕍→𝕐 is continuous in states and actions for all v_0:t^i and for each i∈ℕ, and
 
  (c) the cost function in (<ref>), c:Ω_0×𝕏×𝕌×𝕌×𝕏→ℝ_+, is continuous in its second, third, fourth, and fifth arguments for all ω_0.
 
 


 §.§ Mean-field optimal policies as limits of optimal N-DM teams

In the following, we first establish global optimality results under Assumption <ref> (see Theorem <ref>), then we establish the result under a more relaxed assumption, Assumption <ref> (see Theorem <ref>):

Assume 


  (i) (x_0^1,x_0^2, …) are i.i.d. random vectors conditioned on ω_0,  

  (ii) for t=0,…,T-1, {w^i_t}_i∈ℕ are i.i.d. random vectors, and for i∈ℕ, {w^i_t}_t=0^T-1 are mutually independent, and independent of ω_0 and (x_0^1,x_0^2, …). For t=0,…,T-1, {v^i_t}_i∈ℕ are i.i.d. random vectors, and for i∈ℕ, {v^i_t}_t=0^T-1 are mutually independent, and independent of ω_0, (x_0^1,x_0^2, …), and w^i_ts for i∈ℕ and t=0,…,T-1.


Assume that conditioned on ω_0, (x_0^1,x_0^2, …) are exchangeable random vectors.


Later on we will establish an existence theorem under Assumption <ref>, and we note that the proof under Assumption <ref> is more direct. This is why two separate theorems will be presented, and the proof of the latter will be built on that of the former.


Consider a team defined as (𝒫_T^N,MF) with a symmetric information structure. Assume the team problem is convex in policies. Let the action space be compact and convex for each decision makers. Under Assumption <ref> and Assumption <ref>, the team is symmetrically optimal.


The proof follows from Theorem <ref>.


 
 Consider a team defined as  (𝒫_T^∞,MF) with (𝒫_T^N,MF) having a symmetric information structure for every N. Assume for every N the team problem is convex in policies. Let the action space be compact and convex for each DM. Under Assumption <ref>, and Assumption <ref>, if there exists a sequence of optimal policies for (𝒫_T^N,MF), {γ^*,N_T}_N, which converges (for every DM due to the symmetry) pointwise to γ^*,∞_T as N →∞, then γ^*,∞_T  (which is identically symmetric) is an optimal policy for (𝒫_T^∞,MF).


 Following from Lemma  <ref>, one can consider a sequence of N-DM teams which are symmetrically optimal that defines (𝒫_T^N,MF) and whose limit is identified with (𝒫_T^∞,MF). Define 
 
1 Q_N(B):=1/N∑_i=1^Nδ_β_N^i(B)      where      β_N^i:=(γ^*,N_T(y^i),y^i,ζ^i),

1 Q̃_N(B):=1/N∑_i=1^Nδ_β_∞^i(B)      where      β_∞^i:=(γ^*,∞_T(y^i),y^i,ζ^i),
 
 where δ_Y(·) denotes the Dirac measure for any random vector Y, and B ∈𝒵:=U×Y×S, U:=(∏_t=0^T-1𝕌), Y:=(∏_t=0^T-1𝕐), S:=(∏_t=0^T-1𝕊)=𝕏× (∏_t=0^T-1𝕎×𝕍), X=(∏_t=0^T-1𝕏), y^i=(y_0^i,…,y_T-1^i), and ζ^i:=(ζ^i_0,…,ζ_T-1^i).
 
 In the following, first, we show that conditioned on ω_0, Q_N converges P-almost surely to Q=ℒ(β^1_∞|ω_0) in w-s topology (coarsest topology on 𝒫(U×Y×S)  under which ∫ f(u,y,ζ)Q_N(du, dy, dζ): 𝒫(U×Y×S) →ℝ is continuous for every measurable and bounded function f which is continuous in u and y but need not to be continuous in ζ (see e.g., <cit.> and <cit.>)). Then, we show that 
 
    lim sup_N→∞J_T^N(γ̃_T^*,N)=J_T^∞(γ̃^*,∞_T),

 where γ̃_T^*,N:=(γ_T^*,N,γ_T^*,N,…,γ_T^*,N) and γ̃^*,∞_T:=(γ^*,∞_T,γ^*,∞_T,…).
 
 
  (Step 1):
 In this step, we show that conditioned on ω_0, Q_N converges P-almost surely to Q in w-s topology. First, we show that for every continuous and bounded function g in actions and observations, for every ω_0 on a set of P-measure one,

   1P({ω∈Ω|lim_N→∞(1/N∑_i=1^N[g(γ^*,N_T(y^i),y^i,ζ^i)-g(γ^*,∞_T(y^i),y^i,ζ^i)])=0}|ω_0)=1.

Following from  symmetry of the information structure and Lemma <ref>, every DM applies an identical optimal policy γ_T^*,N and since functions f_t and h_t are identical for each DM, conditioned on ω_0, (γ^*,N_T(y^i),y^i,ζ^i) and (γ^*,∞_T(y^i),y^i,ζ^i) are i.i.d. random vectors. For every ϵ>0 and for every function g continuous and bounded in actions and observations, we have P-almost surely

   1lim_N→∞P(|∫g dQ_N-∫g dQ̃_N|≥ϵ|ω_0)
 
   1≤ϵ^-1lim_N→∞1/N∑_i=1^N E[|g(γ^*,N_T(y^i),y^i,ζ^i)-g(γ^*,∞_T(y^i),y^i,ζ^i)|| ω_0]

   1=ϵ^-1lim_N→∞  E[|g(γ^*,N_T(y^i),y^i,ζ^i)-g(γ^*,∞_T(y^i),y^i,ζ^i)|| ω_0]

   1=ϵ^-1 E[lim_N→∞|g(γ^*,N_T(y^i),y^i,ζ^i)-g(γ^*,∞_T(y^i),y^i,ζ^i)|| ω_0]=0 ,

where (<ref>) follows from Markov's inequality, the triangle inequality and the definition of the empirical measure, and (<ref>) follows from the fact that (γ^*,N_T(y^i),y^i,ζ^i) and (γ^*,∞_T(y^i),y^i,ζ^i) are i.i.d. random vectors. Since  g is bounded and continuous, the dominated convergence theorem implies (<ref>). Hence, for every subsequence, there exists a subsubsequence such that P-almost surely  P({ω∈Ω|lim_N →∞(∫ g dQ_N-∫ g dQ̃_N)=0}|ω_0)=1.

 
 Now, we show that conditioned on ω_0, {Q̃_N}_N converges weakly to Q P-almost surely. Since conditioned on ω_0,  (γ^*,∞_T(y^i),y^i,ζ^i) are i.i.d. random vectors, the strong law of large numbers implies that P-almost surely

0.93P({ω∈Ω|lim_N→∞(1/N∑_i=1^Ng(γ^*,∞_T(y^i),y^i,ζ^i)- E[g(γ^*,∞_T(y^1),y^1,ζ^1)|ω_0])=0}| ω_0)=1,

hence, 
P({ω∈Ω|lim_N→∞(∫ g dQ̃_N-∫ g dQ)=0}|ω_0)=1 P-almost surely.

Hence, through choosing a suitable subsequence, for every ω_0∈Ω_0 on a set of P-measure one, for every function g continous and bounded in actions and observations and measurable and bounded in uncertainty and initial states
 
1lim_N→∞|∫g dQ_N-∫g dQ|   1≤lim_N→∞ (|∫g dQ_N-∫g dQ̃_N|+|∫g dQ̃_N-∫g dQ| )=0,

hence, conditioned on ω_0, Q_N converges weakly to Q P-almost surely. We note that the convergence is weakly, but since ζ^is are exogenous with a fixed probability measure, the convergence is also in the w-s topology.

  (Step 2): 
Following from (<ref>) and (<ref>), we have

x_t^i   =f_t-1(f_t-2(…f_0(x_0^i, u_0^i,w_0^i)),u_t-1^i,w_t-1^i)=f̃_t-1(ζ^i,u_0:t-1^i),

y_t^i   =h_t(x_0^i,f̃_0(ζ^i,u_0^i), …, f̃_t-1(ζ^i,u_0:t-1^i), u_0:t-1^i, v_0:t^i)=h̃_t(ζ^i,u_0:t-1^i),

where following from Assumption <ref>, f̃_t-1 and h̃_t are continuous in actions. Hence, under Assumption <ref>(c), we have

   11/N∑_i=1^N∑_t=0^T-1 E^γ_T^*,1:N[c(ω_0, x_t^i,u_t^i,1/N∑_p=1^Nu_t^p,1/N∑_p=1^Nx_t^p)]

   1=1/N∑_i=1^N E[c̃(ω_0,ζ^i, γ^*,N_T(y^i),1/N∑_i=1^Nγ^*,N_T(y^i),1/N∑_i=1^NΛ(γ^*,N_T(y^i),ζ^i))],

where (<ref>) is true following from (<ref>) for some functions c̃:Ω_0× S× U× U× X→ℝ_+ which are continuous in its last three arguments and a function Λ:  U× S→ X which is continuous in actions. Hence, by induction and rewriting observations as a functions of policies of the past DMs (γ_↓ t^*, N(y_↓ t^i)) since γ_T^*,N converges to γ_T^*,∞, the induced cost by γ_T^*,N also converges to the cost induced by γ_T^*,∞ P-almost surely.

  (Step 3): 
We have

   1lim sup_N→∞1/N∑_i=1^N∑_t=0^T-1 E^γ_T^*,1:N[c(ω_0, x_t^i,u_t^i,1/N∑_p=1^Nu_t^p,1/N∑_p=1^Nx_t^p)]

   1=lim sup_N→∞11/N∑_i=1^N E[c̃(ω_0,ζ^i, γ^*,N_T(y^i),1/N∑_i=1^Nγ^*,N_T(y^i),1/N∑_i=1^NΛ(γ^*,N_T(y^i),ζ^i))]

   1≥lim inf_N→∞ E[ E[1∫_Zc̃(ω_0,ζ, u,∫_UuQ_N(du ×Y×S),∫_U ×SΛQ_N(du ×Y×dζ))

                       1×Q_N(du,dy,dζ)|ω_0]]

   1≥E[ E[lim inf_N→∞∫_Zc̃(ω_0, ζ, u,∫_UuQ_N(du ×Y×S),∫_U ×SΛQ_N(du ×Y×dζ))

                       1×Q_N(du,dy,dζ)|ω_0]]

   0.97≥E[ E[∫_Zc̃(ω_0,ζ, u,∫_UuQ(du ×Y×S),∫_U ×SΛQ(du ×Y×dζ))1Q(du,dy,dζ)|ω_0]],

where (<ref>) follows from (<ref>). Inequality (<ref>) follows from (<ref>) and replacing limsup with liminf, and (<ref>) follows from Fatou's lemma. In the following, we justify (<ref>).  Since conditioned on ω_0, Q_N converges  weakly to Q P-almost surely, we have Q_N(du ×Y×S) converges weakly to Q(du×Y×S) P-almost surely conditioned on ω_0, hence, the compactness of U implies that conditioned on ω_0, P-almost surely

    11/N∑_i=1^Nγ^*,N_T(y^i)=∫_UuQ_N(du ×Y×S) ∫_UuQ(du ×Y×S)= E[γ^*,∞_T(y^1)|ω_0].

 Since conditioned on ω_0, Q_N converges  weakly to Q P-almost surely, we have Q_N(du ×Y× dζ) converges P-almost surely to Q(du×Y× dζ) in w-s topology conditioned on ω_0. Following from (<ref>), since f_ts are bounded and continuous in actions, Λ is bounded and continuous in actions, hence, this implies that conditioned on ω_0, P-almost surely

    1∫_U×SΛ Q_N(du ×Y× dζ) ∫_U×SΛ Q(du ×Y× dζ).

 Since the cost function c̃ is continuous in its last three arguments, P-almost surely

    1lim_N→∞c̃(ω_0, ζ, u,∫_UuQ_N(du ×Y×S),∫_U ×SΛQ_N(du ×Y×dζ))

       1=c̃(ω_0, ζ, u,∫_UuQ(du ×Y×S),∫_U ×SΛQ(du ×Y×dζ)).

 Define a non-negative bounded functions
 
    1G_N^M:=min{M, c̃(ω_0, ζ, u,∫_UuQ_N(du ×Y×S),∫_U ×SΛQ_N(du ×Y×dζ))},

     1G^M:=min{M, c̃(ω_0, ζ, u,∫_UuQ(du ×Y×S),∫_U ×SΛQ(du ×Y×dζ))},
 
where the sequence {G_M}_M converges as M →∞ to 

    1G:=c̃(ω_0, ζ, u,∫_ UuQ(du ×Y×S),∫_U×SΛ Q(du ×Y× dζ)).

We have P-almost surely

1lim inf_N→∞   0.94∫_Zc̃(ω_0, ζ, u,∫_UuQ_N(du ×Y×S),∫_U ×SΛQ_N(du ×Y×dζ))1Q_N(du,dy,dζ)

   1 ≥lim_M→∞lim inf_N→∞∫_ZG_N^MQ_N(du,dy.dζ)

   1= lim_M→∞∫_ZG^MQ(du,dy,dζ)

   1=∫_Zc̃(ω_0,ζ, u,∫_UuQ(du ×Y×S),∫_U ×SΛQ(du ×Y×dζ))1Q(du,dy,dζ),

where (<ref>) is true since

c̃(ω_0, ζ, u,∫_UuQ_N(du ×Y×S),∫_U ×SΛQ_N(du ×Y×dζ))≥G^M_N.

Equality (<ref>) follows from the generalized convergence theorem in <cit.> since G_N^M is bounded and continuously converges to G^M, i.e., P-almost surely

lim_N→∞   min{M,1   c̃(ω_0,ζ, u_N,∫_UuQ_N(du ×Y×S),∫_U ×SΛQ_N(du ×Y×dζ))}

   1=min{M,1   c̃(ω_0,ζ, u,∫_UuQ(du ×Y×S),∫_U ×SΛQ(du ×Y×dζ))},

when u_N→ u as n →∞. The monotone convergence theorem implies (<ref>). 
Hence, (<ref>) holds which implies lim sup_N→∞J_T^N(γ̃_T^*,N)=J_T^∞(γ̃_T^*,∞), and this completes the proof following from <cit.>. Here, for completeness we present the proof which is similar to the analysis of the proof <cit.> for dynamic teams,

inf_γ_TJ^∞_T(γ_T)   ≤lim sup_N→∞ J^N_T(γ̃^*,∞_T)

   =lim sup_N→∞J_T^N(γ_T^*,N) =lim sup_N→∞inf_γ_T^1:NJ_T^N(γ_T^1:N)

   = lim sup_N→∞inf_γ_TJ_T^N(γ_T)

   ≤inf_γ_Tlim sup_N→∞J_T^N(γ_T) =inf_γ_TJ^∞(γ_T),


where (<ref>) is true since the restriction γ_T to the first N components is γ_T^1:N. This implies that γ̃_T^*,∞ is globally optimal. 


On the connection between finitely many DMs and infinitely many DMs, we note a closely related work on mean-field games by Fischer <cit.> where the information structure is assumed to be static since the policy of each player is assumed to be adapted to the filtration generated by his/her initial states and Wiener process (also called in the mean-field games' literature, somewhat non-standard in the control literature, as open-loop distributed controllers <cit.>,<cit.>). This means that the information of each DM is not affected by any of the actions of the other DMs. For dynamic teams, there are two difficulties: (1) obtaining variational equations is challenging since fixing policies of DMs and perturbing only DM's policies, perturbs the observation of other DMs and hence the controls u^-i*=(γ^1(y^1),…,γ^i-1*(y^i-1),γ^i+1*(y^i+1),…,γ^N*(y^N)); (2) solutions of variational equations which give person-by-person optimal policies are inconclusive for global optimality due to the lack of convexity in general.


We also note additional related works by Lacker <cit.> where either convergence of open-loop controllers, or convergence of Nash equilibria induced by closed-loop controllers (where controls are measurable path-dependent functions of states, u_t^i=ϕ(t,x_0:t), where x_0:t=(x_0:t^1,…,x_0:t^N) and ϕ is a measurable function) or Markovian controllers (u_t^i=ϕ(t,x_t), where x_t=(x_t^1,…,x_t^N)) have been considered. In <cit.>, the information structure is classical (a centralized problem) since players have access to all the information available to previous DMs).


In Lemma <ref> and Theorem <ref>, we considered a non-classical information structure for teams defined as  (𝒫_T^∞,MF) with a convex expected cost in policies. For teams defined as  (𝒫_T^∞,MF) with a symmetric partially nested information structure which admit static reduction, the above result holds and similar to the proof of Theorem <ref>, it can be proven under the assumption that the cost functions is convex in actions (since convexity of the cost function in actions is a sufficient condition for convexity of the expected cost function in policies for this class of problems <cit.>).

 
Assumptions that action spaces are compact and f_ts are bounded can be relaxed by assuming that


  (A1)sup_N≥ 1 E[|γ^*,N_T(y^1)|^1+δ]<∞  for some δ >0,

  (A2)sup_N≥ 1 E[|Λ(γ^*,N_T(y^1),ζ^1)|^1+δ̃]<∞  for some δ̃ >0.

That is because, following from the pointwise convergence of γ^*,N_T and continuity of Λ in actions, the above uniform integrability assumption justifies exchanging the limit and the expectation required to establish the convergence in (<ref>) and (<ref>) using a similar analysis of (<ref>) and an argument of (<ref>) based on the strong law of large numbers. This result is particularly important for LQG models (we use this remark in Section <ref>).



 Consider a team defined as  (𝒫_T^∞,MF)  with (𝒫_T^N,MF) having a symmetric information structure for every N. Assume for every N the team problem is convex in policies. Let the action space be compact and convex for each DM, and assume Assumption <ref>, Assumption <ref>(ii), and Assumption <ref> hold. If there exists a sequence of optimal policies for (𝒫_T^N,MF), {γ^*,N_T}_N, which converges (for every DM due to the symmetry) pointwise to γ^*,∞_T as N →∞, then γ^*,∞_T  (which is identically symmetric) is an optimal policy for (𝒫_T^∞,MF).


Under Assumption  <ref>(ii) and Assumption <ref>, for every A^i∈ℬ(S), and A^i=B^i×∏_t=0^T-1(D_t^i× E_t^i) (where B^i∈ℬ(𝕏), D_t^i∈ℬ(𝕎), and E_t^i∈ℬ(𝕍)), for all N∈ℕ, and permutations σ, we have P-almost surely

1P(ζ^1∈A^1,   0.94…, ζ^N∈A^N| ω_0)

   1=P(x_0^1∈B^1,…, x_0^N∈B^N| ω_0)∏_i=1^N∏_t=0^T-1P(w_t^i∈D_t^i)P(v_t^i∈D_t^i)

   1=P((x_0^σ)^1∈B^1,…, (x_0^σ)^N∈B^N| ω_0)∏_i=1^N∏_t=0^T-1P((w_t^σ)^i∈D_t^i)P((v_t^σ)^i∈D_t^i)

   1=P((ζ^σ)^1∈A^1,…, (ζ^σ)^N∈A^N| ω_0),

where (<ref>) follows from Assumption  <ref>(ii), and (<ref>) follows from Assumption <ref>. Hence, (ζ^1,ζ^2,…) are exchangeable conditioned on ω_0.

Hence, following from a similar argument as the proof of Theorem <ref> (by considering ω_0 in the cost function and the law of total expectation (by first conditioning on ω_0), under Assumption <ref>(ii) and Assumption <ref>, one can consider a sequence of N-DM teams which are symmetrically optimal that defines (𝒫_T^N,MF) and whose limit is identified with (𝒫_T^∞,MF). Since initial states are not necessarily independent conditioned on ω_0, we can not establish that (γ^*,∞_T(y^i),y^i,ζ^i) are i.i.d. random vectors conditioned on ω_0 which has been used in (<ref>) in (Step 1) of the proof of Theorem <ref> to show that Q_N converges weakly to Q P-almost surely. 


However, we note that since (ζ^1,ζ^2,…) are exchangeable conditioned on ω_0, for every A^i∈ℬ(S) and C ∈ℬ(Ω_0), and for all N∈ℕ, and permutations σ, we have

P(ζ^1∈A^1,   0.94…, ζ^N∈A^N, ω_0∈C)=0.94P((ζ^σ)^1∈A^1,…, (ζ^σ)^N∈A^N, ω_0∈C).

Let α^i:=(ω_0,ζ^i). Hence,  (<ref>) implies that (α^1,α^2,…) is exchangeable. Following from <cit.>, there exists a random vector z∈ [0,1] such that, (ζ^1,ζ^2,…) are i.i.d. random vectors conditioned on (ω_0,z).

 Let ω̃_0:=(ω_0, z). Hence, under Assumption  <ref>(ii) and Assumption <ref>, conditioned on ω̃_0, (ζ^1,ζ^2,…) are i.i.d. random vectors. Following from standard stochastic realization results <cit.>, we can represent any stochastic kernel in a functional form, with almost sure equivalence, ζ^i=g(ω̃_0, θ^i) for some independent θ^i and measurable g (note that following from exchangeability, g is identical for all i∈ℕ and (θ^1,θ^2,…) are i.i.d. random vectors).

Since conditioned on ω̃_0, (ζ^1,ζ^2,…) are i.i.d. random vectors, (γ^*,∞_T(y^i),y^i,ζ^i) are i.i.d. random vectors conditioned on ω̃_0, hence  for every ω̃_0 on a set of P-measure one, we have for every continuous and bounded function g in actions and observations, by the strong law of large numbers,

0.94P({ω∈Ω|lim_N→∞|1/N∑_i=1^Ng(γ^*,∞_T(y^i),y^i,ζ^i)- E(g(γ^*,∞_T(y^1),y^1,ζ^1)|ω̃_0)|=0}| ω̃_0)=1.

Hence, following from an identical analysis as that of (Step 1) of the proof of Theorem <ref>, conditioned on ω̃_0, Q_N converges weakly to Q, for every ω̃_0 on a set of P-measure one. 

Following from the representation ζ^i=g(ω̃_0, θ^i), we have

   1lim sup_N→∞ E[ E[1∫_Zc̃(ω_0,ζ, u,∫_UuQ_N(du ×Y×S),∫_U ×SΛQ_N(du ×Y×dζ))

                       1×Q_N(du,dy,dζ)|ω_0]]

   1= lim sup_N→∞∫_Ω_0×[0,1] E[1∫_Zc̃(ω̃_0,ζ, u,∫_UuQ_N(du ×Y×S),∫_U ×SΛQ_N(du ×Y×dζ))

                       1×Q_N(du,dy,dζ)|ω̃_0]P(dω̃_0),

where (<ref>) follows from the fact that for all N ∈ℕ, and for every A^i∈ℬ(S)

1∫_Ω_0P(ζ^1∈A^1, …, ζ^N∈A^N|ω_0)P(dω_0)=∫_Ω_0×[0,1]∏_i=1^N η(ζ^i∈A^i|z, ω_0)P(dz,dω_0),

and with slightly abuse of notations we use the same notation, c̃, for the cost function after transformation in (<ref>). The rest of the proof is identical to that of Theorem <ref>.




 §.§ An existence theorem on globally optimal policies for dynamic mean-field team problems with a symmetric information structure

An implication of Theorem <ref> is the following existence result on globally optimal policies for mean-field team problems. In particular, we will establish the existence of a converging subsequence, in an appropriate sense, for a sequence of optimal policies for N-DM teams with an increasing number of DMs. For the following theorem, we do not establish the pointwise convergence; but by Theorem <ref>, if a sequence of optimal policies for (𝒫_T^N,MF), {γ^*,N_T}_N, converges pointwise, a global optimal policy exists. To this end, we allow decision makers to apply randomized policies.  For each decision maker (DM^i for i∈ℕ), a probability measure P ∈𝒫(Ω_0×𝕏×∏_t=0^T-1 (𝕎×𝕍)×∏_t=0^T-1(𝕌×𝕐)) is a policy induced by a randomized policy if and only if for every t=0,…,T-1 and for all continuous and bounded function g

   ∫g(ω_0, x_0^i, ζ_0:t-1^i, y_0:t^i,u_0:t^i)P(dω_0, dx_0^i, dζ_0:t-1^i, dy^i_0:t,du^i_0:t)

     =∫g(ω_0, x_0^i, ζ_0:t-1^i,y_0:t^i,u_0:t^i)μ^i(dx_0^i, dζ_0:t-1^i|ω_0)P(dω_0)

            ×∏_k=0^t Π^i_k(du^i_k|y^i_k)p_k^i(dy^i_k|ω_0, x_0^i, ζ_0:k-1^i, y^i_0:k-1, u_0:k-1^i),

for a stochastic kernel Π^i_k on 𝕌 given 𝕐, where p_k^i is the transition kernel characterizing the observations of DM^i at time t,

p_k^i   (y^i_k∈·|ω_0, x_0^i, ζ_0:k-1^i, y^i_0:k-1, u_0:k-1^i)

   :=P(h_k(x_0:k^i,u_0:k-1^i,v_0:k^i) ∈·|ω_0, x_0^i, ζ_0:k-1^i, y^i_0:k-1, u_0:k-1^i),

and μ^i is a fixed probability measure on initial states and disturbances of DM^i conditioned on ω_0. This equivalency follows from the fact that continuous and bounded functions form a separating class <cit.> and <cit.>. 

First, we present an absolute continuity assumption on observations of DMs.

For every DM^i and t=0,…,T-1, there exists a function ψ_t^i:𝕐×Ω_0×𝕏×∏_k=0^t-1(𝕎×𝕍)×∏_k=0^t-1(𝕐×𝕌)→ℝ_+ continuous in actions, and a probability measure ν_t^i on 𝕐 such that  for all Borel sets A=A^1×…× A^N,

P(y_t^1:N∈A   |ω_0, x_0^1:N, ζ_0:t-1^1:N, y^1:N_0:t-1, u_0:t-1^1:N)

   =∏_i=1^N∫_A^iψ_t^i(y^i_t, ω_0, x_0^i,ζ_0:t-1^i,y_0:t-1^i, u_0:t-1^i)ν_t^i(dy_t^i).



This assumption allows us to obtain an independent measurements reduction (see <cit.>). For example, if v_t^i for all i ∈ℕ and t=0,…,T-1 are i.i.d with a probability measure admitting a density function so that the observation of each DM^i at time t is
y_t^i=h̃_t(x_t^i,u_↓ t^i)+v_t^i,
where h̃_t is continuous, then Assumption <ref> holds <cit.>.


Consider a team defined as  (𝒫_T^∞,MF) with (𝒫_T^N,MF) having a symmetric information structure for every N. Assume for every N, the team problem is convex in policies and the action space is convex. Assume further that without any loss, the optimal policies can be restricted to those with E(ϕ_i(u^i))≤ K for some finite K, where ϕ_i:𝕌→ℝ_+ is lower semi-continuous (moment condition). Under Assumption <ref> and Assumption <ref> if either


  (i) Assumption <ref> holds (with no further assumptions on the information structure of each DM^i for i∈ℕ through time t=0,…, T-1), or

  (ii) for each DM^i for i∈ℕ through time t=0,…, T-1, there exists a static reduction with the classical information structure (i.e., under a static reduction, the information structure is expanding such that σ(y_t^i) ⊂σ(y_t+1^i) where σ denotes the σ-field),

then there exists an optimal policy for (𝒫_T^∞,MF).


Since the space of policies that are deterministic (where Π_k^i in (<ref>) are indicator functions) are not closed under the weak convergence topology (e.g., as an implication of <cit.>), we allow for randomization in the policies and therefore the limit policy is not necessarily deterministic according to the above result; however, it is identical for each DM.

We use individually randomized policies and we show that for every sequence of N-DM optimal policies, there exists a subsequence which converges to an optimal independently randomized policy for the mean-field limit under an appropriate topology defined by the product topology where each coordinate is endowed with the weak convergence topology. In (Step 1), we show that for each finite N-DM team problem, optimal policies are deterministic and symmetric and we consider the independently randomized policies induced by such policies {P_N}_N (where P_N∈𝒫(Y×U) for each DM satisfying (<ref>)), as our sequence to be studied. We also define the sequence of empirical measures  induced by these policies, Q_N, as (<ref>).

In (Step 2), we show that for every sequence of policies satisfying a moment condition there exists a subsequence such that policies {P_n}_n for each DM and a subsequence of empirical measures {Q_n}_n induced by these policies (where n ∈𝕀 is the index set of a convergent subsequence) converge weakly to a limit P-almost surely, that is, for a set of P-measure one, for every bounded function g which is continuous in actions and observations and measurable in uncertainties,

P({ω∈Ω|lim_n→∞(∫g dQ_n-∫g dQ)=0}| ω_0)=1.

To this end, we first show that for each DM a sequence {P_n}_n is tight, then we show that the sequence of empirical measures {Q_n}_n induced by these policies converges weakly to a limit P-almost surely.  

In (Step 3), we show that the set of policies for each DM is closed under the weak convergence topology, hence, the limit policy satisfies the required measurability/conditional independence constraints (that is, the limit policy satisfies (<ref>)). In (Step 4), we use the lower semi-continuity argument to show that the expected cost function under the induced limit policy is less than or equal to the expected cost achieved by the sequence of N-DM optimal policies. 


  (Step 1):
Under Assumption <ref>, Assumption <ref>(c) and Assumption <ref>, and by condition (i) using <cit.>, or condition (ii) using <cit.>, there exists a deterministic optimal policy for each finite N-DM team problem.  
Action spaces are convex and the team problem is convex in policies, hence, using Lemma <ref>, one can consider a sequence of N-DM teams which are symmetrically optimal that defines (𝒫_T^N,MF) and whose limit is identified with (𝒫_T^∞,MF). Hence, for each N-DM team problem, we consider symmetric randomized optimal policies. 

  (Step 2):
In the following, we first show that the set of policies P_N∈𝒫(Y×U) for each DM satisfying (<ref>) and the moment condition is tight, then, by symmetry, we show that {Q_N}_N is induced by this set of policies is also tight. We use the fact that  conditioned on ω_0,  (γ^*,N_T(y^i),y^i,ζ^i) are i.i.d. random vectors (this follows from symmetry of the information structure and Lemma <ref> since every DM applies the identical policy γ_T^*,N) and also since the space of control policies is tight under the weak convergence for each DM (see e.g., <cit.>). 

Since actions of DMs do not affect the observations of others, the policy spaces are decoupled from the actions of other decision makers. Since we can restrict the search for optimal policies over those satisfying the moment condition, the fact that ν→∫ν(dx)g(x) is lower semi-continuous for a continuous function g <cit.> implies that the marginals on U satisfying the moment condition are tight under the weak convergence topology. Hence, the collection of all probability measures with these tight marginals is also tight (see e.g., <cit.>). This implies that the sequence of randomized policies satisfying the moment condition is tight.  

Since every DM applies an identical policy and since observations are conditionally i.i.d., a countably infinite product of space of policies of DMs is tight (where each coordinate is tight in the weak convergence topology). Hence,  there exists a subsequence of policies P̃_n∈𝒫(∏_i(Y×U)) (as a product of policies of DMs) converges weakly  to a limit P̃ (each coordinate converges weakly)  P-almost surely. Furthermore, since every DM applies an identical policy, conditioned on ω_0, actions induced by an identical randomized policies, observations and disturbances are i.i.d. through DMs. Hence, following from a similar argument as (Step 1) of the proof of Theorem <ref>, a subsequence of empirical measures {Q_n}_n ∈𝕀 converges P-almost surely to Q in w-s topology. We note that the convergence is under the weak convergence topology, but since ζ^is are exogenous with a fixed marginal, the convergence is also in the w-s topology.

  (Step 3): 
In this step, we show that each coordinate of the space of policies (space of policies for each DM) is closed under the weak convergence topology. This in particular implies that the space of policies is closed under the product topology and using (Step 1), we can conclude that the space of control policies is compact under the product topology where each coordinate is weakly compact.

Assume P_n is a policy for DM^i induced by a randomized policy converging weakly to P_∞. In fact, conditions (i) or (ii) leads to the closedness of the set of policies (see (<ref>)) induced by P_n.
If Assumption <ref> holds, then  by the discussion in the proof of <cit.>, each coordinate of policy spaces corresponding to DM^i acting through time is closed under the weak convergence topology. Also, if condition (ii) holds, then <cit.> leads to the same conclusion. Hence, each coordinate of space of policies (corresponding to DM^i) is closed under the weak convergence topology (since each coordinate of the space of policies is a finite product of space of policies for each DM at time instances t=0,…,T-1). Hence, following from (Step 2), there exists a subsequence {Q_n}_n ∈𝕀 converges weakly to Q P-almost surely where Q is induced by a randomized policy in the set of policies satisfying (<ref>) and the limit policy is admissible and satisfies the required measurability/conditional independence constraints.

Let for every t=0,…,T-1, P^*, ω_0_n be a probability measure on actions, observations and uncertainties induced by optimal randomized policies for each DM (which is identical because of symmetry) for N-DM teams conditioned on ω_0, i.e., a probability measure that satisfies

   ∫g(ω_0, x_0^i, ζ_0:t-1^i, y_0:t^i,u_n, 0:t^i, *)P^*, ω_0_n(dx_0^i, dζ_0:t-1^i, dy^i_0:t,du^i, *_n, 0:t)

     =∫g(ω_0, x_0^i, ζ_0:t-1^i,y_0:t^i,u_n, 0:t^i, *)μ^i(dx_0^i, dζ_0:t-1^i|ω_0)

            ×∏_k=0^t Π_k^*, n(du^*, i_n, k|y^i_k)p_k(dy^i_k|ω_0, x_0^i, ζ_0:k-1^i, y^i_0:k-1, u_n, 0:k-1^i, *),

for all bounded functions g which is continuous in actions and observations and measurable in other arguments. We denote u^i,*_n:=(u^i,*_n, 0,…,u^i,*_n, T-1) as the action of DM^i through time induced by Π^*,n_t. Similarly, we denote P^*, ω_0 as a probability measure induced by the limit policy, i.e., a probability measure satisfying (<ref>) induced by Π_k^*, ∞ where u^i,*_∞:=(u^i,*_∞, 0,…,u^i,*_∞, T-1) is the action of DM^i through time induced by Π_k^*, ∞. 

  (Step 4): 
Now, we show that the expected cost function under the limit randomized policy is less than or equal to the expected cost achieved by lim sup_n→∞J_T^n(γ̃_T^*,n). Since the cost function is continuous in states and actions, under the reduction (conditions (i) or (ii)), we have P-almost surely

11/N∑_i=1^N∑_t=0^T-1   1c(ω_0, x_t^i,u_t^i,1/N∑_p=1^Nu_t^p,1/N∑_p=1^Nx_t^p)∏_i=1^N∏_t=0^T-1ψ_t(y^i_t, ω_0, x_0^i,ζ^i_0:t-1,y_0:t-1^i,u_0:t-1^i)

   1=1/N∑_i=1^Nc̅(ω_0,ζ^i, u^i,1/N∑_p=1^Nu^p,1/N∑_p=1^NΛ(u^p,ζ^p))∏_i=1^Nψ(y^i, ω_0,ζ^i,u^i),

where (<ref>) is true following from (<ref>) and Assumption <ref> for some functions c̅:Ω_0× S× U× U× X→ℝ_+  continuous in states and actions and a function Λ:  U× S→ X continuous in actions and 

   1∏_i=1^Nψ(y^i, ω_0,ζ^i,u^i):=∏_i=1^N∏_t=0^T-1ψ_t(y^i_t, ω_0, x_0^i,ζ^i_0:t-1,y_0:t-1^i,u_0:t-1^i).
 
We have

   1lim sup_N→∞1/N∑_i=1^N∑_t=0^T-1 E^γ_T^*,1:N[c(ω_0, x_t^i,u_t^i,1/N∑_p=1^Nu_t^p,1/N∑_p=1^Nx_t^p)]

   0.94 ≥lim_M→∞lim sup_N→∞∫∫_Zmin{M, 0.94c̃(ω_0,ζ, u,∫_UuQ_N(du ×Y×S),∫_U ×SΛQ_N(du ×Y×dζ))}

                       0.94×Q_N(du,dy,dζ)∏_i=1^∞P_N^*,ω_0(du^i, *_N,dy^i,dζ^i)∏_i=1^∞ψ(y^i, ω_0,ζ^i,u^i,*_N)P(dω_0)

   0.94≥lim_M→∞lim_n→∞∫∫_Zmin{M,0.94c̃(ω_0,ζ, u,∫_UuQ_n(du ×Y×S),∫_U ×SΛQ_n(du ×Y×dζ))}

                       0.94×Q_n(du,dy,dζ)∏_i=1^∞P_n^*,ω_0(du^i, *_n,dy^i,dζ^i)∏_i=1^∞ψ(y^i, ω_0,ζ^i,u^i,*_n)P(dω_0)

   0.94=  lim_M→∞∫lim_n→∞∫_Zmin{M,0.94c̃(ω_0,ζ, u,∫_UuQ_n(du ×Y×S),∫_U ×SΛQ_n(du ×Y×dζ))}

                       0.94×Q_n(du,dy,dζ)∏_i=1^∞P_n^*,ω_0(du^i, *_n,dy^i,dζ^i)∏_i=1^∞ψ(y^i, ω_0,ζ^i,u^i,*_n)P(dω_0)

   1=lim_M→∞∫∫_Zmin{M,c̃(ω_0,ζ, u,∫_UuQ(du ×Y×S),∫_U ×SΛQ(du ×Y×dζ))}

                       1×Q(du,dy,dζ)∏_i=1^∞P^*,ω_0(du^i, *_∞,dy^i,dζ^i)∏_i=1^∞ψ(y^i, ω_0,ζ^i,u^i,*_∞)P(dω_0)

   1=∫∫_Zc̃(ω_0,ζ, u,∫_UuQ(du ×Y×S),∫_U ×SΛQ(du ×Y×dζ))

                  1×Q(du,dy,dζ)∏_i=1^∞P^*,ω_0(du^i, *_∞,dy^i,dζ^i)∏_i=1^∞ψ(y^i, ω_0,ζ^i,u^i,*_∞)P(dω_0)
where (<ref>) follows from the definition of empirical measures and by integrating over the set (∏_i=n_l+1^∞Y×S) and since P-almost surely

   1min{M,c̃(ω_0,ζ, u,∫_UuQ_N(du ×Y×S),∫_U ×SΛQ_N(du ×Y×dζ))}

   1≤c̃(ω_0,ζ, u,∫_UuQ_N(du ×Y×S),∫_U ×SΛQ_N(du ×Y×dζ)).

Inequality (<ref>) is true since limsup is the greatest convergent subsequential limit for a bounded sequence and (<ref>) follows from the dominated convergence theorem. We note that {Q_n}_n is induced by u^i,*_n for each DM. Since {Q_n}_n ∈ I converges weakly to Q P-almost surely, by the moment condition and Remark <ref>, a similar argument as (Step 3) of the proof of Theorem <ref> implies that P-almost surely

   1∫_UuQ_n(du ×Y ×S)  ∫_UuQ(du ×Y×S)

   1∫_U ×SΛQ_n(du ×Y×dζ)  ∫_U ×SΛQ(du ×Y×dζ)

Hence, (<ref>) follows from <cit.> since

   1min{M,0.94c̃(ω_0,ζ, u,∫_UuQ_n(du ×Y×S),∫_U ×SΛQ_n(du ×Y×dζ))}

is bounded and non-negative, and continuously converges in u P-almost surely (see (<ref>)). That is because, conditioned on ω_0, y^i are i.i.d. random vectors (thanks to the symmetry), the space of policies is compact under the product topology (with the weak convergence topology for each coordinate (for each DM)), ∏_i=1^∞ψ(y^i, ω_0,ζ^i,u^i,*_n) converges in the product topology, and the cost function and ψ are continuous. Finally, (<ref>) follows from the monotone convergence theorem.
Hence, the proof is completed.




For the existence result, to show that the set of policies induced by independently randomized policies for each DM through time t=0,…, T-1 (see (<ref>)) is closed under the weak convergence topology, we utilized the result in <cit.> which are more general than those in <cit.>. We note that the extension of the existence results in  <cit.> to our setup is not immediate since the conclusion of (Step 3) can not be established rigorously without considering the technical steps involving infinite dimensions and limit arguments. 




§ SYMMETRIC LQG DYNAMIC TEAMS

In the section, we consider LQG setup where the results of Section <ref> and Section <ref> can be applied. We first consider N-DM LQG problems  where we use Theorem <ref> to show that the globally optimal policies are symmetric. Then, based on symmetry, we calculate N-DM optimal policies for such problems. Next, using Theorem  <ref> and Theorem <ref>, we show the convergence of N-DM optimal policies to optimal policies of LQG mean-field teams with countably infinite number of DMs. Finally, we consider infinite horizon problems where we use symmetry and convergence results to obtain global optimal policies for such problems. 


 §.§ Symmetric partially nested LQG dynamic teams on a graph

In the following, we consider decentralized problems where Theorem <ref> can be utilized and the optimal policy can be obtained. First, we formulate LQG problems with a symmetric partially nested information structure.
Consider the following dynamics. Let i=1,2, and

    x_t+1^i=Ax_t^i+Bu_t^i+w^i_t.



  Problem (𝒫_T):
Consider the expected cost function of (γ_T^1,γ_T^2) as 

1J_T(γ_T^1,γ_T^2)= E^(γ_T^1,γ_T^2)[1/T∑_t=0^T-1∑_i=1^2(x_t^i)^TQx_t^i+(u_t^i)^TRu_t^i+(u_t^1)^TR̃u_t^2+(u_t^2)^TR̃u_t^1],

where γ_T^i=(γ^i_0:T-1), and R,R̃>0 and Q ≥ 0. Let

1y_t^i=H_tζ^i_0:t-1+∑_j=0^t-1D_tju_j^i,
 
 where ζ^i_t=(w_t^i, v_t^i) with ζ^i_0=(x_0^i,w_0^i, v_0^i). Let n, m, s ∈ℕ and 𝕏=ℝ^n, 𝕐=ℝ^s, 𝕌=ℝ^m, w_t^i∈ℝ^n, v_t^i∈ℝ^n,A ∈ℳ_n,n,   B ∈ℳ_n,m, R ∈ℳ_m,m, Q ∈ℳ_n,n, R̃∈ℳ_m,m, H_t∈ℳ_s,n(2t+1), and D_tj∈ℳ_s,m. Let the information structure of each DM be I_t^i={y^i_t, y_↓ t^i}.


In the following, we show that the above dynamic teams are symmetrically optimal under sufficient conditions on the observations and initial states. 

For a fixed T, consider a finite horizon team problem defined above as (𝒫_T). If x_0^1 and x_0^2 are exchangeable zero mean Gaussian random vectors and w^i_ts and v^i_ts are i.i.d. Gaussian random vectors for i=1,2 and independent for all t=0,…, T-1 and also independent of initial states, then the dynamic team is symmetrically optimal.


Since the dynamic team is LQG with a partially nested information structure, a static reduction exists and the expected cost is convex in policies under static reductions (see <cit.> and Theorem <ref>(iii)). Assumption <ref> is satisfied following from (<ref>). We need to show assumptions of Theorem <ref> hold. Following from the hypothesis on disturbances and initial states, Assumption (b) holds. Assumption (c) holds following from Assumption <ref> and since given (x_0^1, x_0^2), (y_0:T-1^1, y_0:T-1^2) are independent. Hence, Theorem <ref> completes the proof.


Here, we consider a class of LQG dynamic teams with a tree information structure where we utilize Corollary <ref> and we obtain an explicit recursion for the optimal policy.


  Problem (𝒫_T^tree):
Consider a finite horizon expected cost (<ref>) with I_t^i={x_[0:t]^i, u_[0:t-1]^i}.


We note that this problem is a special case of (𝒫_T) since we assumed that y_t^i=x_t^i for all DM^i and t=0,…,T-1 and also all DMs have a total recall (I_t^i={x_[0:t]^i, u_[0:t-1]^i}). For this problem, we calculate an explicit recursion for optimal policies using the symmetry established in Corollary <ref>.

 
 For a fixed T, consider a finite horizon team problem defined as (𝒫_T^tree). If (x_0^1,x_0^2)  are exchangeable with an identical zero mean Gaussian distribution and w^i_ts are i.i.d. zero mean Gaussian random vectors for i=1,2 and independent for all t=0,…, T-1 and independent of initial states, then
 
    u^1*,(T)_t=K_t^(T)x_t^1+L_t^(T) E[x_0^2|x_0^1],

    u^2*,(T)_t=K_t^(T)x_t^2+L_t^(T) E[x_0^1|x_0^2],
 
 where
 
    K_t^(T)=-(R+B^TP_t+1^(T)B)^-1B^TP_t+1^(T)A,

    P_t^(T)=-A^TP_t+1^(T)B^T(R+B^TP_t+1^(T)B)^-1B^TP_t+1^(T)A+Q+A^TP_t+1^(T)A,

    L_t^(T)=-(R+B^TP_t+1^(T)B)^-1[R̃K_t^(T)G_t^(T)+R̃L_t^(T)Σ

                    +∑_s=t+1^T-1B^T(A^T)^s-tP_s+1^(T)BL_s^(T)],

    G_t^(T)=∏_s=0^t-1(A+BK_s^(T))+∑_s=1^t∏_j=s^t-1(A+BK_j^(T))BL_s-1^(T)Σ,
 
 where Σ= E[x_0^1(x_0^2)^T]( E[x_0^2(x_0^2)^T])^-1, P_T^(T)=0, G_0^(T)=I. Moreover, the optimal cost is
 
J_T(γ^*_T)   =2/T{ E[(x_0^1)^TP_0^(T)x_0^1]+∑_t=0^T-1 E[(w_t^1)^TP_t^(T)w_t^1]

   +∑_t=0^T-1 E[( E[x_0^2|x_0^1])^T((L_t^(T))^TB^TP_t+1^(T)BL_t^(T)) E[x_0^2|x_0^1]]

   +∑_t=1^T-1 E[(x_0^1)^T(A^T)^tP_t+1^(T)BL_t^(T) E[x_0^2|x_0^1]]}.
 
 
 Following from <cit.> and Radner's theorem <cit.>, person-by-person optimality implies global optimality due to the uniqueness of the person-by-person optimal policy. That is because the information structure is partially nested, and LQG dynamic teams can be reduced to a static one using Ho-Chu's static reduction <cit.>. Hence, we only need to show that the policy satisfying (<ref>) and (<ref>) is person-by-person optimal. We show that for DM^1, J(γ^*_T,γ^*_T)≤ J((γ^-t*_T,β),γ^*_T) for all β∈Γ^t where (γ^-t*_T,β)=(γ_0:t-1^*,β,γ_t+1:T-1^*). This implies that (γ^*_T,γ^*_T) is person-by-person optimal thanks to Corollary <ref> since the dynamic team is symmetrically optimal (by exchanging policies (γ^-t*_T,β) with γ^*_T which implies J(γ^*_T,γ^*_T)≤ J(γ^*_T,(γ^-t*_T,β)) for all β∈Γ^t and this implies that (γ^*_T,γ^*_T) is the fixed point of the equation). The proof is completed by induction. Due to space constraints, we have removed the calculation. 
 
 
 
 The optimal policies (<ref>) and (<ref>) contain two parts which can be interpreted as follows: the first part, k_t^(T)x_t^i, is equivalent to the optimal policy of the branch (DM) by ignoring the other branch in the optimization problem (in this case, this is equivalent to the centralized policies since the information structure of each branch (DM) is centralized). The second part corresponds to the correlation term between branches (DMs).
 

 In the following, we generalize the result of Theorem <ref> to N-DM LQG dynamic teams.
 Assume that the dynamics for i=1,2,...,N are defined as (<ref>).


  Problem (𝒫_T^N, tree):
Consider the expected cost function of γ_T^1:N as 

1J_T^N(γ_T^1:N)=1/T∑_t=0^T-1∑_i=1^N E^γ_T^1:N[(x_t^i)^TQx_t^i+(u_t^i)^TRu_t^i +∑_j=1,j≠i^N(u_t^i)^TR̃u_t^j+(u_t^j)^TR̃u_t^i],

where γ_T^i=γ^i_0:T-1 for i=1,…,N and R,R̃>0 and Q ≥ 0. Let I_t^i={x_[0:t]^i, u_[0:t-1]^i}.


 For a fixed T and N, consider a finite horizon team problem defined as (𝒫_T^N,tree). If (x_0^1:N) are exchangeable with an identical zero mean Gaussian distribution, and w^i_ts for i=1,…,N are i.i.d. zero mean Gaussian random vectors, independent for t=0,…,T-1, and independent of initial states, then
 
    1u^i*,(T),(N)_t=K_t^(T)x_t^i+L_t^(N),(T)∑_j=1,j≠i^N E[x_0^j|x_0^i],
 
 where K_t^(T) and P_t^(T) satisfy (<ref>) and (<ref>), and L_t^(N),(T) is a function of K_0:t^(T). 


The proof is similar to the one of Theorem <ref>.


Now we consider more general setup where using Corollary <ref>, we establish a structural result for the case where the information structure of each decision maker over time satisfies a structure which is identical for all DMs and is partially nested. An example of such a graph structure has been depicted in Fig. 2. 



  Problem (𝒫_T^N):
Consider a finite horizon expected cost of γ_T^1:N as (<ref>) with the information structure I_t^i={y^i_t, y_↓ t^i} where y^i_t is defined in (<ref>) and dynamics is defined in (<ref>).


 For a fixed T and N, consider a finite horizon team problem defined as (𝒫_T^N). If (x_0^1, …, x_0^N) are exchangeable with an identical zero mean Gaussian distribution and w^i_ts for i=1,…, N are i.i.d. zero mean Gaussian random vectors, independent for all t=0,…, T-1  and independent of initial states, then

    1u^i*,(T),(N)_t=K_t^(T)y_t^i+L_t^(N),(T)∑_j=1,j≠i^N E[x_0^j|y_t^i],
 
 where K_t^(T) are obtained by considering only one DM and ignoring other DMs.


The proof is similar to that of Theorem <ref> by <cit.> and Corollary <ref>.


A related work is <cit.>, where structural results for optimal policy have been obtained for finite horizon LQG problems on graphs. In our analysis above, the  structural result for the optimal policy is obtained without assuming that decision makers who have no common ancestors and no common descendants have either uncorrelated noise or are decoupled through the cost function. Instead, exchangeable partially nested LQG teams with correlated initial states and disturbances are considered.  Moreover, here, the graph structures may not be trees in general, as opposed to <cit.> where a multi-tree structure has been imposed on a graph.


Now, we present results for LQG teams with a mean-field coupling through the cost function. First, using Corollary <ref>, we obtain globally optimal policies for N-DM teams with a mean-field coupling and correlated initial states and disturbances. Next, as an implication of Theorem <ref>, we show the convergence of optimal policies for LQG N-DM mean-field teams on a tree to the corresponding optimal policy of mean-field teams.
 Let I_t^i={x_[0:t]^i, u_[0:t-1]^i}  for i ∈ℕ, and dynamics be as (<ref>).


  Problem (𝒫_T,LQG^N, MF): Consider the expected cost function of γ_T^N as 

   1J_T^N(γ_T^N)=1/T∑_t=0^T-1∑_i=1^N E^γ_T^1:N[(x_t^i)^TQx_t^i+(u_t^i)^TRu_t^i

              1+1/N-1∑_j=1,j≠i^N(u_t^i)^TR̃u_t^j+(u_t^j)^TR̃u_t^i+1/N-1∑_j=1,j≠i^N(x_t^i)^TQ̃x_t^j+(x_t^j)^TQ̃x_t^i],

where R,R̃>0 and Q, Q̃≥ 0.

  Problem (𝒫_T,LQG^∞,MF): Consider the expected cost function of γ_T as

   1J^∞_T(γ_T)=lim sup_N→∞J_T^N(γ_T).




 For a fixed T and N, consider a finite horizon team problem defined as (𝒫_T,LQG^N,MF). If (x_0^1:N) are exchangeable zero mean Gaussian random vectors, and w^i_ts are i.i.d. zero mean Gaussian random vectors for i=1,…,N, independent for t=0,…, T-1, and independent of initial states, then
  
    1u^i*,(T),(N)_t=K_t^(T)x_t^i+L_t^(N),(T)/N-1∑_j=1,j≠i^N E[x_0^j|x_0^i],
 
 where K_t^(T) and P_t^(T) satisfy (<ref>) and (<ref>), and L_t^(N),(T) is a function of K_0:t^(T).


The proof is similar to the one in Theorem <ref>.


 For a fixed T, consider a finite horizon team problem defined as (𝒫_T,LQG^∞,MF). Assume {x_0^i}_i ∈ℕ are exchangeable random vectors with zero mean Gaussian distribution, and w^i_ts are i.i.d. zero mean Gaussian random vectors for i ∈ℕ, independent for t=0,…, T-1, and independent of initial states. If L_t^(N),(T) in (<ref>) converges pointwise as N →∞ to L_t^(∞),(T), then
  
    u^i*,(T),(∞)_t=K_t^(T)x_t^i+L_t^(∞),(T)Σx_0^i,
 
 where K_t^(T) and P_t^(T) satisfy (<ref>) and (<ref>), and Σ= E[x_0^1(x_0^2)^T]( E[x_0^2(x_0^2)^T])^-1.


Following from <cit.>, since {x_0^i}_i ∈ℕ are exchangeable Gaussian random vectors, we can describe them explicitly as x_0^i=ω_0+θ^i where (θ^1,θ^2, …) are i.i.d. mean zero Gaussian and independent of mean zero Gaussian random vector ω_0. Now, we invoke Theorem <ref> (or Theorem <ref>) and Corollary <ref> and we use Remark <ref> to complete the proof.



 §.§ Average cost infinite horizons problems for partially nested dynamic teams

In the following, we consider average cost problems with a symmetric partially nested information structure. We note that the optimality of linear policies for infinite horizon LQG problems is an open problem in its generality. In this subsection, we provide a positive result for a class of such problems.

 Now, we consider an infinite horizon team problem and we use the result in Section <ref>.

 
  Problem (𝒫_∞^tree):
 Consider the expected cost function of (γ^1,γ^2) as
  
   J(γ^1:2)=lim sup_T→∞ E^(γ^1,γ^2)[c(x_0:T-1^1:2,u_0:T-1^1:2)],

where the cost function is defined as (<ref>) and I_t^i={x_[0:t]^i, u_[0:t-1]^i}.


 First, we introduce a lemma essential for Theorem <ref>.

Consider the sequence {a^i_T}_i=1^T. Assume lim_T→∞a_T^i=a for i=0,…,T-1. If for every fixed T∈ℕ, a_T^l=a_T+1^l+1 for all l=0…,T-1, then lim_T→∞1/T∑_i=1^Ta_T^i=a.


We have

    1lim_T→∞1/T∑_i=1^Ta_T^i=lim_T→∞1/T∑_i=0^T-1a_T-i^1=lim_T→∞1/T∑_k=1^Ta_k^1=a,

where the second equality follows from a_T^l=a_T+1^l+1  and the last equality follows from the Cesáro mean argument.



Consider average cost infinite horizon team problems defined as (𝒫_∞^tree). Assume (A,B) are stabilizable and (A,Q^1/2) are detectable. Assume x_0^1 and x_0^2  are exchangeable with an identical zero mean Gaussian distribution and w^i_ts are i.i.d. zero mean Gaussian random variables for i=1,2 and for all t=0,…, T-1 and independent of initial states. If L_t^(T) in (<ref>) converges pointwise to L_t^(∞) as T →∞, then the pointwise limit of the sequence of optimal policies for (𝒫_T^tree) is team optimal for (𝒫_∞^tree) and stabilizes the closed-loop system,
 
    u^1*,(∞)_t=Kx_t^1+L_t^(∞) E[x_0^2|x_0^1],

    u^2*,(∞)_t=Kx_t^2+L_t^(∞) E[x_0^1|x_0^2],
 
 where K, P, L_t^(∞) and G_t^(∞) are the pointwise limit of the ones for (𝒫_T^tree) as T →∞.


We show lim sup_T→∞J_T(γ_T^*)=J(γ_∞^*) and invoke <cit.> or <cit.> to complete the proof. From (<ref>), we have

   1lim_T→∞|J_T(γ_∞^*)-J_T(γ_T^*)|

   1=lim_T→∞2/T| E[(x_0^1)^T(P_0^(T)-P)x_0^1]+2/T∑_t=0^T-1 E[(w_t^1)^T(P_t^(T)-P)w_t^1]

   1+2/T∑_t=0^T-1 E[( E[x_0^2|x_0^1])^T((L_t^(T))^TB^TP_t+1^(T)BL_t^(T)-(L_t^(∞))^TB^TPBL_t^(∞)) E[x_0^2|x_0^1]]

   1+2/T∑_t=0^T-1 E[(x_0^1)^T(A^T)^t-1(P_t+1^(T)BL_t^(T)-PBL_t^(∞)) E[x_0^2|x_0^1]]|

   1=0,

where (<ref>) is zero since P_0^(T) converges to P using Lemma <ref>  since P_t+1^(T+1)=P_t^(T). Expression (<ref>) converges to zero since L_t^(T) in (<ref>) converges pointwise to L_t^(∞) as T →∞, we have ∑_s=t+1^∞B^T(A^T)^s-tPBL_s^(∞)<∞, and this implies that lim_s→∞L_s^(∞)=0. Hence, we have for every ϵ>0, there exists N̂>T such that for every t>N̂, |Tr[L_t^(∞)(L_t^(∞))^T]|<ϵ. We define L_t^(T)=0 for t>T. Expression (<ref>) is equal to zero following from Lemma <ref> and the fact that |Tr[L_t^(∞)(L_t^(∞))^T]|<ϵ  for every t>N̂. Hence, equality (<ref>) is true and global optimality follows from <cit.>. The closed loop system is stable since lim sup_t→∞ E(||x_t^1||^2)<∞  following from ||A+BK||<1 (all the eigenvalues of A+BK are inside of the unit circle), and since  ||L_t^(∞)|| is uniformly bounded.





§ CONCLUSION

In this paper, we studied dynamic teams with symmetric information structures. We presented a characterization for symmetrically optimal teams for convex exchangeable team problems. For mean-field teams with symmetric information structures, we showed the convergence of optimal policies for mean-field teams with N decision makers to the corresponding optimal policy of mean-field teams. We obtained globally optimal solutions for LQG dynamic team problems with symmetric partially nested information structures. Moreover, we obtained globally optimal policies for average cost infinite horizon problems of LQG dynamic teams. 

In this paper since we worked under the convexity assumption, the information structure does not allow for the mean-field coupling in the dynamics. In our recent work <cit.>, we relaxed the convexity assumption to arrive at complementary existence and structural results.


plain
10

aldous2006ecole
D. J. Aldous, I. A. Ibragimov, and J. Jacod.
Ecole d'Ete de Probabilites de Saint-Flour XIII, 1983, volume
  1117.
Springer, 1985.

arabneydi2017certainty
J. Arabneydi and A. Aghdam.
A certainty equivalence result in team-optimal control of mean-field
  coupled markov chains.
In IEEE 56th Annual Conference on Decision and Control (CDC),
  pages 3125–3130, 2017.

arabneydi2015team
J. Arabneydi and A. Mahajan.
Team-optimal solution of finite number of mean-field coupled LQG
  subsystems.
In IEEE 54th Annual Conference on Decision and Control (CDC),
  pages 5308–5313, 2015.

arapostathis2017solutions
A. Arapostathis, A. Biswas, and J. Carroll.
On solutions of mean field games with ergodic cost.
Journal de Mathématiques Pures et Appliquées,
  107(2):205–251, 2017.

bardi2017non
M. Bardi and M. Fischer.
On non-uniqueness and uniqueness of solutions in finite-horizon mean
  field games.
ESAIM: Control, Optimisation and Calculus of Variations, 25:44,
  2019.

bardi2014linear
M. Bardi and F. S. Priuli.
Linear-quadratic N-person and mean-field games with ergodic cost.
SIAM Journal on Control and Optimization, 52(5):3022–3052,
  2014.

bayraktar2020non
E. Bayraktar and X. Zhang.
On non-uniqueness in mean field games.
Proceedings of the American Mathematical Society,
  148:4091–4106, 2020.

Billingsley
P. Billingsley.
Convergence of Probability Measures.
Wiley, New York, 1968.

Blackwell2
D. Blackwell.
Memoryless strategies in finite-stage dynamic programming.
Annals of Mathematical Statistics, 35:863–865, 1964.

BorkarRealization
V. S. Borkar.
White-noise representations in stochastic realization theory.
SIAM Journal on Control and Optimization, 31:1093–1102, 1993.

caines2018peter
P. Caines, M. Huang, and R. Malhamé.
Mean field games.
Handbook of Dynamic Game Theory, pages 345–372, 2017.

cardaliaguet2019example
P. Cardaliaguet and C. Rainer.
An example of multiple mean field limits in ergodic differential
  games.
Nonlinear Differential Equations and Applications NoDEA,
  27(3):1–19, 2020.

carmona2018probabilistic
R. Carmona and F. Delarue.
Probabilistic Theory of Mean Field Games with Applications
  I-II.
Springer, 2018.

carmona2016mean
R. Carmona, F. Delarue, and D. Lacker.
Mean field games with common noise.
The Annals of Probability, 44(6):3740–3803, 2016.

fischer2017connection
M. Fischer.
On the connection between symmetric N-player games and mean field
  games.
Annals of Applied Probability, 27(2):757–810, 2017.

gupta2014existence
A. Gupta, S. Yüksel, T. Başar, and C. Langbort.
On the existence of optimal policies for a class of static and
  sequential dynamic teams.
SIAM Journal on Control and Optimization, 53:1681–1712, 2015.

hajek2019non
B. Hajek and M. Livesay.
On non-unique solutions in mean field games.
In IEEE 58th Annual Conference on Decision and Control (CDC),
  pages 1219–1224, 2019.

HoChu
Y. C. Ho and K. C. Chu.
Team decision theory and information structures in optimal control
  problems - part I.
IEEE Transactions on Automatic Control, 17:15–22, February
  1972.

ho1973equivalence
Y. C. Ho and K. C. Chu.
On the equivalence of information structures in static and dynamic
  teams.
IEEE Transactions on Automatic Control, 18(2):187–188, 1973.

huang2012social
M. Huang, P. E. Caines, and R. P. Malhamé.
Social optima in mean field LQG control: centralized and
  decentralized strategies.
IEEE Transactions on Automatic Control, 57(7):1736–1751, 2012.

CainesMeanField1
M. Huang, R.P. Malhamé, and P.E. Caines.
Large population stochastic dynamic games: Closed-loop
  McLean-Vlasov systems and the Nash certainty equivalence principle.
Special issue in honour of the 65th birthday of Tyrone Duncan,
  Communications in Information and Systems, 6:221–252, 2006.

lacker2015mean
D. Lacker.
Mean field games via controlled martingale problems: existence of
  markovian equilibria.
Stochastic Processes and their Applications, 125(7):2856–2894,
  2015.

lacker2016general
D. Lacker.
A general characterization of the mean field limit for stochastic
  differential games.
Probability Theory and Related Fields, 165(3-4):581–648, 2016.

lacker2017limit
D. Lacker.
Limit theory for controlled Mckean–Vlasov dynamics.
SIAM Journal on Control and Optimization, 55(3):1641–1672,
  2017.

lacker2018convergence
D. Lacker.
On the convergence of closed-loop nash equilibria to the mean field
  game limit.
The Annals of Applied Probability, 30(4):1693–1761, 2020.

LyonsMeanField
J. M. Lasry and P. L. Lions.
Mean field games.
Japanese J. of Mathematics, 2:229–260, 2007.

lessard2015optimal
L. Lessard and S. Lall.
Optimal control of two-player systems with output feedback.
IEEE Transactions on Automatic Control, 60(8):2129–2144, 2015.

light2018mean
B. Light and G. Y. Weintraub.
Mean field equilibrium: uniqueness, existence, and comparative
  statics.
Existence, and Comparative Statics, 2018.

mahajan2013static
A. Mahajan, N. C. Martins, and S. Yüksel.
Static LQG teams with countably infinite players.
In IEEE 52nd Annual Conference on Decision and Control (CDC),
  pages 6765–6770, 2013.

nayyar2014optimal
A. Nayyar and L. Lessard.
Optimal control for LQG systems on graphs—part I: Structural
  results.
arXiv preprint arXiv:1408.2551, 2014.

Radner
R. Radner.
Team decision problems.
Annals of Mathematical Statistics, 33:857–881, 1962.

rotkowitz2008information
M. Rotkowitz.
On information structures, convexity, and linear optimality.
In IEEE 47th Annual Conference on Decision and Control (CDC),
  pages 1642–1647, 2008.

Rotkowitz
M. Rotkowitz and S. Lall.
A characterization of convex problems in decentralized control.
IEEE Transactions on Automatic Control, 51:274–286, February
  2006.

saldi2019topology
N. Saldi.
A topology for team policies and existence of optimal team policies
  in stochastic team theory.
IEEE Transactions on Automatic Control, 65(1):310–317, 2019.

SSYdefinetti2020
S. Sanjari, N. Saldi, and S. Yüksel.
Optimality of independently randomized symmetric policies for
  exchangeable stochastic teams with infinitely many decision makers.
arXiv preprint:2008.11570, 2020.

sinacdc2018optimal
S. Sanjari and S. Yüksel.
Optimal stochastic teams with infinitely many decision makers and
  Mean-field teams.
In IEEE 57th Annual Conference on Decision and Control (CDC),
  pages 4123–4128, 2018.

sinacdc2019convex
S. Sanjari and S. Yüksel.
Convex symmetric stochastic dynamic teams and their mean-field limit.
In IEEE 58th Annual Conference on Decision and Control (CDC),
  pages 4662–4667, 2019.

sanjari2018optimal
S. Sanjari and S. Yüksel.
Optimal solutions to infinite-player stochastic teams and mean-field
  teams.
IEEE Transactions on Automatic Control, to appear (also
  arXiv:1808.02962v2), 2019.

Schal
M. Schäl.
Conditions for optimality in dynamic programming and for the limit of
  n-stage optimal policies to be optimal.
Z. Wahrscheinlichkeitsth, 32:179–296, 1975.

serfozo1982convergence
R. Serfozo.
Convergence of Lebesgue integrals with varying measures.
Sankhyā: The Indian Journal of Statistics, Series A, pages
  380–402, 1982.

shah2013cal
P. Shah and P. Parrilo.
H2 optimal decentralized control over posets: A state-space solution
  for state-feedback.
IEEE Transactions on Automatic Control, 58(12):3084–3096,
  2013.

swigart2014optimal
J. Swigart and S. Lall.
Optimal controller synthesis for decentralized systems over graphs
  via spectral factorization.
IEEE Transactions on Automatic Control, 59(9):2311–2323, 2014.

west1996introduction
D. B. West.
Introduction to graph theory, volume 2.
Prentice hall Upper Saddle River, NJ, 1996.

wit68
H. S. Witsenhausen.
A counterexample in stochastic optimal control.
SIAM Journal on Control and Optimization, 6:131–147, 1968.

wit75
H. S. Witsenhausen.
The intrinsic model for discrete stochastic control: Some open
  problems.
Lecture Notes in Econ. and Math. Syst., Springer-Verlag,
  107:322–335, 1975.

wit88
H.S. Witsenhausen.
Equivalent stochastic control problems.
Mathematics of Control, Signals and Systems, 1(1):3–11, 1988.

yuksel2018general
S. Yüksel.
A universal dynamic program and refined existence results for
  decentralized stochastic control.
SIAM Journal on Control and Optimization, 58:2711–2739.

yukselSICON2017
S. Yüksel.
On stochastic stability of a class of non-Markovian processes and
  applications in quantization.
SIAM Journal on Control and Optimization, 55:1241–1260, 2017.

YukselBasarBook
S. Yüksel and T. Başar.
Stochastic Networked Control Systems: Stabilization and
  Optimization under Information Constraints.
Springer, New York, 2013.

YukselSaldiSICON17
S. Yüksel and N. Saldi.
Convex analysis in decentralized stochastic control, strategic
  measures and optimal solutions.
SIAM Journal on Control and Optimization, 55:1–28, 2017.



