Stochastic Graph Neural Network-based Value Decomposition for MARL in
  Internet of Vehicles
Autonomous driving has witnessed incredible advances in the past several
decades, while Multi-Agent Reinforcement Learning (MARL) promises to satisfy
the essential need of autonomous vehicle control in a wireless connected
vehicle networks. In MARL, how to effectively decompose a global feedback into
the relative contributions of individual agents belongs to one of the most
fundamental problems. However, the environment volatility due to vehicle
movement and wireless disturbance could significantly shape time-varying
topological relationships among agents, thus making the Value Decomposition
(VD) challenging. Therefore, in order to cope with this annoying volatility, it
becomes imperative to design a dynamic VD framework. Hence, in this paper, we
propose a novel Stochastic VMIX (SVMIX) methodology by taking account of
dynamic topological features during the VD and incorporating the corresponding
components into a multi-agent actor-critic architecture. In particular,
Stochastic Graph Neural Network (SGNN) is leveraged to effectively capture
underlying dynamics in topological features and improve the flexibility of VD
against the environment volatility. Finally, the superiority of SVMIX is
verified through extensive simulations.
Baidi Xiao, Rongpeng Li, Fei Wang, Chenghui Peng, Jianjun Wu, Zhifeng Zhao, Honggang Zhang
# Stochastic Graph Neural Network-based Value Decomposition for MARL in Internet of Vehicles

###### Abstract

Autonomous driving has witnessed incredible advances in the past several decades, while Multi-Agent Reinforcement Learning (MARL) promises to satisfy the essential need of autonomous vehicle control in a wireless connected vehicle networks. In MARL, how to effectively decompose a global feedback into the relative contributions of individual agents belongs to one of the most fundamental problems. However, the environment volatility due to vehicle movement and wireless disturbance could significantly shape time-varying topological relationships among agents, thus making the Value Decomposition (VD) challenging. Therefore, in order to cope with this annoying volatility, it becomes imperative to design a dynamic VD framework. Hence, in this paper, we propose a novel Stochastic VMIX (SVMIX) methodology by taking account of dynamic topological features during the VD and incorporating the corresponding components into a multi-agent actor-critic architecture. In particular, Stochastic Graph Neural Network (SGNN) is leveraged to effectively capture underlying dynamics in topological features and improve the flexibility of VD against the environment volatility. Finally, the superiority of SVMIX is verified through extensive simulations.

 Autonomous vehicle control, multi-agent reinforcement learning, value decomposition, stochastic graph neural network

## I Introduction

In recent years, the unprecedented development of artificial intelligence (AI) brings the self-driving vehicles [2, 3, 4] into the spotlight, and these intelligent vehicles form an Internet of Vehicles (IoV). Equipped with the capability for environmental perception [5], the vehicles try to respond to the observation of the surroundings and find an optimal trajectory between two sites [6, 7] with calibrated self-decision control and traffic congestion avoidance [8, 9]. Additionally, there emerges some research interest towards traffic signal control [10] or fleet control [11] in IoV as well. Nevertheless, as for distributed deployment of intelligent vehicles in IoV, such intricate cases (e.g., traffic control) require them to collaborate on top of reliable communication, which further constitutes a Multi-Agent System (MAS) and catalyses the research progress in a myriad of related studies (e.g., traffic control and prediction) [12, 13]. In particular, Multi-Agent Reinforcement Learning (MARL), which incorporates Deep Reinforcement Learning (DRL) into MAS, emerges by astutely learning through the trial-and-error interaction between multiples agents (i.e., vehicles) and the complicated IoV environment, and promises to yield smart policies for the formulated Markov Decision Process (MDP).

Traditionally, in Independent Q-Learning (IQL) [14], one typical kind of MARL, each individual agent learns its policy by regarding other agents as parts of the environment, and often experiences a non-stationary environment, as the agents always update their policies independently during the learning. Consequently, the non-stationary issue impedes the optimization of agents' policies. Instead, mutual communication shall be considered for cooperative agents to reach holistic consistency. On the other hand, it is natural for MAS to only observe a global reward from the environment. Thus it becomes essential to tackle with the agent heterogeneity and reward assignment issue. [15].

In recent years, algorithms with Centralized Training and Decentralized Execution (CTDE) have become the centerpiece of MARL as they can somewhat handle the aforementioned two problems of IQL (i.e., non-stationary learning, and node heterogeneity). As its name implies, CTDE can be divided into the training phase and execution phase. In particular, in the training phase, agents can implicitly observe the global information of the environment in a centralized manner, so as to guarantee the communication among agents and tackle the non-stationary problem. Subsequently, in the decentralized execution phase, each agent capably makes decisions based on its local observation only. Typical examples of CTDE include MADDPG, COMA, etc [15, 16]. Nevertheless, as the number of agents increases, the centralized state-action value function in CTDE algorithms such as COMA suffers from an exponential increase of the action space as well as the awful growth of computational complexity. Therefore, the Value Decomposition (VD)-based algorithms [17, 18, 19, 20] are proposed to decompose the centralized value function into individual value functions, and attempt to learn an implicit representation for individual reward, thus decreasing the computational complexity in CTDE. Regretfully, these methods mostly concentrate on learning from agents with the invariant communication topology [20, 21], but founder under dynamic environments entailing uncertainty and perturbation due to fluctuating connections and dynamically changing topologies.

On the other hand, the advent of Graph Neural Network (GNN) [22] makes it possible to capture the underlying topological features of vehicle-formed graphs. For example, the spectral-based GNNs [23] can filter the external noise and extract features from the transformed frequency domain of signals. However, most of the spectral-based GNNs concen
Introduction

The study of the evolution of a complex network is a fundamental problem in the field of IoT. The fundamental problem of the IoT is to determine the evolution of a network, and to determine the evolution of a network. The fundamental problem of the IoT is to determine the evolution of a network, and to determine the evolution of a network. The fundamental problem of the IoT is to determine the evolution of a network, and to determine the evolution of a network, and to determine the evolution of a network, and to determine the evolution of a network, and to determine the evolution of a network, to determine the evolution of a network, and to determine the evolution of a network, to determine the evolution
## III Preliminaries and System Model

In this section, we briefly introduce the related knowledge of MDP and VD-based methods, and present the formulated system model to apply MARL for the autonomous vehicle control.

### _Preliminaries_

Generally, an MARL task is formulated as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) [41], which is defined as the tuple \(\langle\mathcal{I},\mathcal{S},\mathcal{A},\mathcal{P},\Omega,\mathcal{R}, \mathcal{O},\gamma\rangle\). \(\mathcal{I}\) represents the set of \(N\) agents, \(\mathcal{S}\) is the state space, \(\mathcal{A}\) is the action space for a single agent and \(\mathcal{A}:=\mathcal{A}^{N}\) is the joint action space. The joint action \(\boldsymbol{a}=\{a^{(1)},a^{(2)},\cdots,a^{(N)}\}\) taken at the current state \(s\) results in the next state \(s^{\prime}\) through a transition of environment according to the transition function \(\mathcal{P}(s^{\prime}|s,\boldsymbol{a}):\mathcal{S}\times\mathcal{A}\times \mathcal{S}\rightarrow[0,1]\). Owing to the scant ability of perception against the colossal environment, agent \(i\) gets a local observation \(o^{(i)}\in\Omega\) via the observation function \(\mathcal{O}(o^{(i)}|s,i):\mathcal{S}\times\mathcal{I}\times\Omega\rightarrow[ 0,1]\) instead of \(s\) at each time-step. All agents share a global reward function \(\mathcal{R}(s,\boldsymbol{a}):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}\) and \(\gamma\) denotes the discount factor.

Furthermore, in a Dec-POMDP, agent \(i\) adopts an action according to its policy \(\pi^{(i)}(\cdot|o^{(i)})\), which denotes the conditional probability distribution of taking action \(a^{(i)}\in\mathcal{A}\) on \(o^{(i)}\). \(\boldsymbol{a}\) is adopted according to the probability \(\boldsymbol{\pi}(\boldsymbol{a}|\boldsymbol{o})=\prod_{i=1}^{N}\pi^{(i)} \left(a^{(i)}|o^{(i)}\right)\) where \(\boldsymbol{\pi}\) denotes the joint policy and \(\boldsymbol{o}=\{o^{(1)},\cdots,o^{(N)}\}\) implies the joint observation. In order to coordinate all agents for maximizing the discounted accumulated return \(J=\sum_{t=0}^{\infty}\gamma^{t}\mathcal{R}\left(s_{t},\boldsymbol{a}_{t}\right)\), a joint state-action value function is defined as the expected discounted accumulated return starting from \(s_{t}\) and \(\boldsymbol{a}_{t}\) of time \(t\), that is

\[Q^{\boldsymbol{\pi}}(s_{t},\boldsymbol{a}_{t})=\mathbb{E}_{\mathcal{P}}\left[ \sum_{k=t}^{\infty}\gamma^{k-t}\mathcal{R}\left(s_{k},\boldsymbol{a}_{k} \right)|s_{t},\boldsymbol{a}_{t},\boldsymbol{\pi}\right], \tag{1}\]

where \(\mathbb{E}(\cdot)\) denotes an expectation operator. It can be naturally observed that the function has an exponential computational complexity due to the joint action \(\boldsymbol{a}_{t}\).

To deal with this scalability issue while pursuing the optimal joint policy \(\boldsymbol{\pi}^{*}=\arg\max_{\boldsymbol{\pi}}Q^{\boldsymbol{\pi}}(s_{t}, \boldsymbol{a}_{t})\), instead of adopting a joint state-action value function as in (1), VD-based methods [17, 18, 19, 20] exhibit an individual state-action value function \(Q^{\boldsymbol{\pi}^{(i)}}(o^{(i)}_{t},a^{(i)}_{t})=\mathbb{E}_{\mathcal{P}} \left[\sum_{k=t}^{\infty}\gamma^{k-t}r^{(i)}_{k}|o^{(i)}_{t},a^{(i)}_{t},\pi^{ (i)}\right]\) for agent \(i\), where \(r^{(i)}_{k}\) is a learned implicit representation of the individual reward decomposed from \(\mathcal{R}(s_{k},\boldsymbol{a}_{k})\) for agent \(i\). Moreover, the individual state-action value function reflects different contributions among agents.

Likewise, the expected \(J\) from \(s_{t}\) can also be represented by the joint state value function

\[V^{\boldsymbol{\pi}}(s_{t})=\mathbb{E}_{\mathcal{P},\boldsymbol{a}_{t}\sim \boldsymbol{\pi}(\cdot|o_{t})}\left[\sum_{k=t}^{\infty}\gamma^{k-t}\mathcal{R} \left(s_{k},\boldsymbol{a}_{k}\right)|s_{t},\boldsymbol{\pi}\right]. \tag{2}\]

Exactly the joint state value function \(V^{\boldsymbol{\pi}}(s_{t})\) is a comprehensive evaluation of \(J\) at \(s_{t}\), so it is commonly used for calculating the advantage value of different actions at \(s_{t}\) as a baseline [42]. Using VD, we can also represent the set of individual state value functions as \(V^{\pi^{(i)}}(o^{(i)}_{t})=\mathbb{E}_{\mathcal{P},\boldsymbol{a}_{t}\sim \pi^{(i)}(\cdot|o^{(i)}_{t})}\left[\sum_{k=t}^{\infty}\gamma^{k-t}r^{(i)}_{k}|o^ {(i)}_{t},\pi^{(i)}\right]\), which implies a more reasonable approximation of the joint value function and speeds up the training.

### _System Model_

We have proposed a mixed autonomy traffic system model that permits vehicles to flow in and out of road, giving rise to the variable amount of vehicles. At time-step \(t\), there are totally \(M_{t}\) vehicles (including \(N\) DRL-driven vehicles and \(M_{t}\geq N\)) as well as intersections or ramps as depicted in Fig. 1. For vehicle \(i\), it has the information such as the velocity \(v^{(i)}_{t}\in\mathbb{R}\) and the position \(z^{(i)}_{t}=(x^{(i)}_{t},y^{(i)}_{t})\in\mathbb{R}^{2}\) of itself at time \(t\). Meanwhile, \(v^{(i)}_{t}\) is controlled via the acceleration \(u^{(i)}_{t}\in\mathbb{R}\) decided by vehicle \(i\) itself. It's also necessary for vehicles to obey some traffic rules. For example, at an intersection, the vehicle has to consider an appropriate passing order as well as control its direction and velocity so as to avoid collisions. Based on the above definition, we have the elements of the Dec-POMDP as follows.

State and observationIn our scenario, each vehicle can receive the information of other accessible vehicles through Vehicle-to-Vehicle (V2V) connections. Without loss of generality, assuming that vehicle \(i\) is only able to communicate with vehicles \(j\) and \(k\) at time \(t\), the observation of vehicle \(i\) can be represented as \(o^{(i)}_{t}=\{v^{(i)}_{t},z^{(i)}_{t},v^{(i)}_{t},z^{(j)}_{t},v^{(i)}_{t},z^{(k)}_ {t}\}\). The state is represented by \(s_{t}=\{v^{(1)}_{t},z^{(1)}_{t},v^{(2)}_{t},z^{(2)}_{t},\cdots,v^{(M_{t})}_{t},z ^{(M_{t})}_{t}\}\), which is consisted of the velocities and positions of all the vehicles. Intuitively, the observation \(o^{(i)}_{t}\) is a part of \(s_{t}\), which can be mathematically determined by \(\mathcal{O}(o^{(i)}_{t}|s_{t},i)\).

ActionAt time \(t\), vehicle \(i\) adopts an action represented as \(a^{(i)}_{t}=\{u^{(i)}_{t},q^{(i)}_{t}\}\) where \(q^{(i)}_{t}\) is an extra action that indicates whether the vehicle changes to another lane or direction. In particular, the \(N\) DRL-driven vehicles select their

Fig. 1: The traffic control scenario with ramps, intersections and vehicles with a constrained communication range.
actions according to the learned policies instead of the others with fixed policies.

RewardThe goal of MARL for the vehicle is to maintain a velocity as close to the desired velocity \(v_{d}^{(i)}\) as possible on the basis of no undesirable collisions. Accordingly, the reward \(r_{t}:=\mathcal{R}(s_{t},\mathbf{a}_{t})\) is set as

\[r_{t}=\begin{cases}\frac{V_{d}-\sqrt{\sum_{i=1}^{M_{t}}(v_{d}^{(i)}-v_{t}^{(i)} )^{2}}}{V_{d}+C_{1}}\\ -\alpha D(z^{(1)},\cdots,z^{(M_{t})}),&\text{if no collision};\\ 0,&\text{otherwise},\end{cases} \tag{3}\]

where \(V_{d}=\sqrt{\sum_{i=1}^{M_{t}}\left(v_{d}^{(i)}\right)^{2}}\) and \(C_{1}\) is a very small constant. \(D(z^{(1)},\cdots,z^{(M_{t})})\) is the penalty term with a safety coefficient \(\alpha\) that hopes the vehicle to hold a safe distance with its preceding vehicle. (3) implies that MARL encourages the vehicle to choose a satisfactory velocity which deviates trivially from \(v_{d}^{(i)}\) while spares no efforts to avoid collisions at time \(t\). Notably, all the \(M_{t}\) vehicles participate in the calculation of the reward.

On the other hand, the wireless-connected intelligent vehicles in IoV constitute a time-varying undirected graph \(\mathcal{G}_{t}=\{\mathcal{V},\mathcal{E}_{t}\}\), where each agent is denoted as a vertex and a communication link are regarded as an edge. In other words, \(\mathcal{V}=\{1,2,\cdots,N\}\) and an undirected edge in \(\mathcal{E}_{t}\) corresponds to a communication link between two vehicles. It is worth recalling that the mobility of vehicles will frequently change the connections among vehicles due to the communication range and thus induce topological changes. For example, if a vehicle meets another vehicle at an intersection, it's obvious that a strong connection between them could be set up in order to avoid collisions. Additionally, the time-varying amount of vehicles also results in the fluctuation of the number of vertices in \(\mathcal{G}_{t}\). For simplicity and without losing the rationality, we will treat \(\mathcal{G}_{t}\) as a graph with fixed vertices.

### _Problem Formulation_

Following the system model, we expect the DRL-driven vehicle to maintain a regular speed that close to its desired velocity on the premise of avoiding collisions. Therefore we have proposed a system utility function \(U\) as the objective of policy optimization, which can be mathematically formulated as

\[\max_{\mathbf{\pi}}\ U=\mathbb{E}_{t}\left[r_{t}|\mathbf{\pi}\right]. \tag{4}\]

In other words, the agents aim to learn a joint policy \(\mathbf{\pi}\) under the guidance of maximizing the system utility (i.e., the expected reward). Notably, for CTDE, the optimization of \(\mathbf{\pi}\) is equivalent to learn the optimal individual policies of

Fig. 2: The illustration of the SVMIX algorithm for autonomous vehicle control.
agents [17]. Therefore, (4) implies to devise an appropriate VD method for CTDE in volatile environment, so as to derive an individual policy of each agent.

## IV Architecture and Designation of Svmix

In this section, we describe the architecture and components of the proposed SVMIX algorithm as depicted in Fig. 2. First, we use PPO as the basis of each individual DRL agent. Afterwards, in order to decompose the proper individual value functions of the \(N\) agents from the total (or joint) value function through the VMIX network, we highlight how to leverage SGNN[25] to effectively capture the topological features from the dynamic graph \(\mathcal{G}_{t}\).

Next we will introduce these essential components (i.e., PPO, SGNN and VMIX) in SVMIX as well as the training procedure. As SGNN is the centerpiece of our algorithm, we also mathematically explain that SGNN can help PPO agents to learn the average optimal solutions.

### Ppo-based RL Agents

Belonging to one of the most famous Policy Gradient (PG) algorithms extensively used for continuous action control, PPO is composed of an actor network and a critic network, where the former is responsible for taking actions in accordance with its learned policy while the latter produces an approximated (individual) value function. Notably, in this paper, we consider a DRL-driven vehicle and a PPO-based agent is equivalent.

As illustrated in the lower part of Fig. 2, for agent \(i\), the actor outputs the mean \(\mu_{i}(o_{t}^{(i)})\) and the standard deviation \(\sigma_{i}(o_{t}^{(i)})\) of a normal distribution \(\mathcal{N}(\mu_{i}(o_{t}^{(i)}),\sigma_{i}^{2}(o_{t}^{(i)}))\) (i.e. the policy \(\pi^{(i)}(\cdot|o_{t}^{(i)})\)) through a Multi-Layer Perception (MLP) taking \(o_{t}^{(i)}\) as input with nonlinear activation functions (omitted from Fig. 2 for simplicity). Sequentially, following \(\pi^{(i)}(\cdot|o_{t}^{(i)})\), the PPO agent samples an action \(a_{t}^{(i)}\) from \(\mathcal{N}(\mu_{i}(o_{t}^{(i)}),\sigma_{i}^{2}(o_{t}^{(i)}))\) at time \(t\) for exploration. Besides, the critic approximates the state value function and produces a state value \(V^{(i)}(o_{t}^{(i)})\) through another MLP.

After that, the joint action \(\mathbf{a}_{t}\) is taken and the environment correspondingly enters the next state \(s_{t+1}\), thus shifting the next joint observation to \(\mathbf{o}_{t+1}\) and yielding a global reward \(r_{t}\). Notably, during this procedure, to better represent the total value function related to the dynamic graph, individual state values are further processed through SGNN-based feature capturing and aggregation.

### _Topological Feature Capturing and Filtering by Sgnn_

As mentioned before, the vehicles can form an undirected graph \(\mathcal{G}_{t}=\{\mathcal{V},\mathcal{E}_{t}\}\). Thus motivated by the extraordinary achievements of GNN to tackle the topology issues, we propose to apply graph signal processing techniques for feature capturing, by treating each individual state value \(V^{(i)}(o_{t}^{(i)})\) as the signal of one corresponding vertex in \(\mathcal{G}_{t}\). Furthermore, we argue that SGNN makes a good complement to deal with the dynamic graph in traffic control.

The upper-right part of Fig. 2 demonstrates that SGNN consists of stochastic graph filters and the READOUT mechanism for output integration. SGNN mainly adopts stochastic graph filters to fit the environment volatility based on the Random Edge Sampling (RES) model [24] with the underlying graph \(\mathcal{G}^{+}=\{\mathcal{V},\mathcal{E}^{+}\}\), wherein \(\mathcal{E}^{+}\) encompasses all sets of edges \(\mathcal{E}_{t}\) for all \(t\) (i.e., \(\mathcal{E}_{t}\subseteq\mathcal{E}^{+},\forall t\)). Besides, we define the adjacent matrix \(\mathbf{A}\in\mathbb{R}^{N}\times\mathbb{R}^{N}\) where an entry \(\mathbf{A}_{m,n}\) is nonzero if and only if \((m,n)\in\mathcal{E}^{+}\), as well as the Laplacian matrix \(\mathbf{L}=\operatorname{diag}(\mathbf{A}\mathbf{1})-\mathbf{A}\) for \(\mathcal{G}^{+}\). In particular, a random sub-graph \(\mathcal{G}_{k}^{+}=\{\mathcal{V},\mathcal{E}_{k}^{+}\}\) will be constructed by holding the original vertices of \(\mathcal{G}^{+}\) but sampling the edges from \(\mathcal{E}^{+}\) following a Bernoulli distribution with a success probability \(p\), that is,

\[\Pr\left[(m,n)\in\mathcal{E}_{k}^{+}\right]=p,\quad\text{for all }(m,n)\in \mathcal{E}^{+}. \tag{5}\]

Through RES, we can emulate dynamic graphs where the communication links (edges) vary frequently. Therefore, to exert the advantage of centralized training, we actually let \(\mathcal{G}^{+}\) be a generalized fixed graph encompassing all potential connections between vertices, so as to ensure enough exploration.

In detail, Fig. 3 demonstrates the corresponding structure of a \(K\)-order stochastic graph filter. Based on the input \(\mathbf{V}(\mathbf{o}_{t})=\{V^{(1)}(o_{t}^{(1)}),\cdots,V^{(N)}(o_{t}^{(N)})\}\in \mathbb{R}^{N}\) from PPO agents as well as the stochastic sub-graphs \(\mathcal{G}_{k}^{+}\) with the adjacent matrix \(\mathbf{A}_{k}\), the information among vertices recursively diffuses as \(\mathbf{V}_{k}(\mathbf{o}_{t})=\mathbf{S}_{k}\mathbf{V}_{k-1}(\mathbf{o}_{t}),k\in\{1,\cdots,K\}\) with \(\mathbf{V}_{0}(\mathbf{o}_{t})=\mathbf{V}(\mathbf{o}_{t})\) and either \(\mathbf{S}_{k}=\mathbf{A}_{k}+\mathbf{I}\) (where \(\mathbf{I}\) denotes the identity matrix) or \(\mathbf{S}_{k}=\mathbf{L}_{k}\). Therefore, having \(\mathbf{S}_{0}=\mathbf{I}\) and unrolling the recursion, the intermediate signal can be represented as

\[\begin{split}\mathbf{V}_{k}(\mathbf{o}_{t})=\mathbf{S}_{k}\mathbf{V}_{k-1}( \mathbf{o}_{t})&=(\mathbf{S}_{k}\mathbf{S}_{k-1}\cdots\mathbf{S}_{0}) \mathbf{V}(\mathbf{o}_{t})\\ &:=\mathbf{S}_{k:0}\mathbf{V}(\mathbf{o}_{t}),\end{split} \tag{6}\]

where \(\mathbf{S}_{k:0}\) is defined as \(\mathbf{S}_{k:0}:=\mathbf{S}_{k}\mathbf{S}_{k-1}\cdots\mathbf{S}_{0}\). Therefore, \(\mathbf{V}_{k}(\mathbf{o}_{t})\) aggregates the information of \(\mathbf{V}(\mathbf{o}_{t})\) through the random sequence of sub-graphs \(\mathcal{G}_{1}^{+},\cdots,\mathcal{G}_{k}^{+}\). Ultimately, the output \(\mathbf{u}\) of the \(K\)-order stochastic graph filter with filter coefficients \(h_{k}\) can be formulated as

\[\mathbf{u}=\sum_{k=0}^{K}h_{k}\mathbf{V}_{k}(\mathbf{o}_{t})=\sum_{k=0}^{K}h_{k}\mathbf{S} _{k:0}\mathbf{V}(\mathbf{o}_{t}):=\mathbf{H}\left(\mathbf{S}_{K:0}\right)\mathbf{V}(\mathbf{o}_ {t}). \tag{7}\]

Specifically, \(\mathbf{H}\left(\mathbf{S}_{K:0}\right)\) denotes the stochastic graph filter with learnable parameters \(h_{0},\cdots,h_{K}\).

Furthermore, we can use \(F\) parallel stochastic graph filters \(\mathbf{H}_{f}\left(\mathbf{S}_{K:0}\right)\) with coefficients \(h_{fx}\in\mathcal{H},\forall f\in\{1,\cdots,F\},k\in\{1,\cdots,K\}\) to process \(\mathbf{V}(\mathbf{o}_{t})\), where \(\mathcal{H}\) is the set of all the

Fig. 3: A stochastic graph filter of \(K\) orders. Here we only show how the information of the green vertex is updated via green edges.
filter coefficients. In particular, for the \(f\)-th stochastic graph filter, the output can be given on top of (7) as

\[\mathbf{V}_{f}^{\prime}(\mathbf{o}_{t})=\sigma\left[\mathbf{u}_{f}\right] =\sigma\left[\mathbf{H}_{f}\left(\mathbf{S}_{K:0}\right)\mathbf{V}(\mathbf{o}_{ t})\right] \tag{8}\] \[=\sigma\left[\sum_{k=0}^{K}h_{fk}\mathbf{S}_{f:k:0}\mathbf{V}(\mathbf{o}_ {t})\right],\]

where \(\sigma(\cdot)\) is the nonlinear activation function ReLU.

Subsequently, the mechanism \(\mathtt{READOUT}:\mathbb{R}^{N}\times\mathbb{R}^{F}\rightarrow\mathbb{R}^{N}\) is implemented to integrate the outputs of the stochastic graph filters through a two-layer MLP as

\[\mathbf{V}_{\mathrm{agg}}(\mathbf{o}_{t})=\mathtt{READOUT}(\{\mathbf{V}_{1}^{\prime}(\bm {o}_{t}),\cdots,\mathbf{V}_{F}^{\prime}(\mathbf{o}_{t})\}). \tag{9}\]

Finally, Sgnn captures the dynamic topological features through stochastic graph filters and random sub-graphs generated by RES with learnable filter coefficients \(\mathcal{H}\). That is, the filtered state values \(\mathbf{V}_{\mathrm{agg}}(\mathbf{o}_{t})\) involve the dynamic topological features. Notably, consistent with methodology of CTDE, SGNN requires the information in every vertex as input in the centralized training phase, while it can be neglected at the decentralized execution phase.

### _The Role of Sgnn_

Intuitively, RES in SGNN brings substantial uncertainty to capture underlying topological features. Therefore, it's necessary to clarify the role of SGNN in SVMIX with theoretical analysis. Concentrating on the stochastic graph filter defined in (7) on the basis of RES, for a given graph \(\mathcal{G}^{+}\) and an input \(\mathbf{V}(\mathbf{o}_{t})\), the expected output can be given by

\[\mathbb{E}\left[\mathbf{u}\right]=\mathbb{E}\left[\mathbf{H}\left(\mathbf{S}_{K: 0}\right)\mathbf{V}(\mathbf{o}_{t})\right]=\mathbb{E}\left[\sum_{k=0}^{K}h_{k}\mathbf{ S}_{k:0}\right]\mathbf{V}(\mathbf{o}_{t}). \tag{10}\]

Let \(\mathbf{S}_{k}=\overline{\mathbf{S}}+\Delta_{k}\) where \(\overline{\mathbf{S}}=\mathbb{E}\left[\mathbf{S}_{k}\right]\) and \(\Delta\) represents the error matrix, we further get \(\mathbb{E}\left[\Delta_{k}\right]=0\) and

\[\mathbb{E}\left[\sum_{k=0}^{K}h_{k}\mathbf{S}_{k:0}\right] =\sum_{k=0}^{K}h_{k}\mathbb{E}\left[\mathbf{S}_{k}\mathbf{S}_{k-1 }\cdots\mathbf{S}_{0}\right] \tag{11}\] \[=\sum_{k=1}^{K}h_{k}\mathbb{E}\left[\left(\overline{\mathbf{S}}+ \Delta_{k}\right)\cdots\left(\overline{\mathbf{S}}+\Delta_{1}\right)\right]+h _{0}\mathbf{I}\] \[=\sum_{k=1}^{K}h_{k}\overline{\mathbf{S}}^{k}+h_{0}\mathbf{I}\]

as \(\mathbb{E}\left[\Delta_{m}\Delta_{n}\right]=\mathbb{E}\left[\Delta_{m}\right] \mathbb{E}\left[\Delta_{n}\right]=0\) if \(m\neq n\) given the mutual independence between the samplings.

1. If \(\mathbf{S}_{k}=\mathbf{L}_{k}\), for an entry \(\overline{\mathbf{S}}_{m,n}\) in \(\overline{\mathbf{S}}\), we have \[\overline{\mathbf{S}}_{m,n}=\begin{cases}-p\mathbf{A}_{m,n},&m\neq n;\\ pd_{m},&\text{else},\end{cases}\] (12) where \(d_{m}\) is represented by the degree of vertex \(m\). So \(\overline{\mathbf{S}}=p\mathbf{L}\), and finally we get \(\mathbb{E}\left[\mathbf{u}\right]=\left(\sum_{k=1}^{K}h_{k}p^{k}\mathbf{L}^{k}+h _{0}\mathbf{I}\right)\mathbf{V}(\mathbf{o}_{t})\). As \(\mathbf{L}\) is a real symmetry matrix, it can be transformed into \(\mathbf{L}=\mathbf{U}\Sigma\mathbf{U}^{T}\) through eigendecomposition where \(\Sigma=\mathrm{diag}\left(\lambda_{1},\lambda_{2},\cdot,\lambda_{N}\right)\) with \(\lambda_{1}\geq\lambda_{2}\geq\cdots\geq\lambda_{N}\). Notably, \(\lambda_{N}\equiv 0\) as \(\mathbf{L}\) always has an eigenvector \(\mathbf{1}\) corresponding to the eigenvalue \(0\), and the algebraic multiplicity of \(\lambda_{N}\) equals to the number of connected components in \(\mathcal{G}^{+}\)[43]. Considering the influence of filtering coefficients \(h_{k}\) and probability \(p\), \(\sum_{k=1}^{K}h_{k}p^{k}\mathbf{L}^{k}+h_{0}\mathbf{I}\) will be a full-rank matrix (e.g., if \(h_{k}>0,\forall k=0,1,\cdots,K\), the eigenvalues will be \(\tilde{\lambda}_{1}\geq\tilde{\lambda}_{2}\geq\cdots\geq\tilde{\lambda}_{N}=h _{0}>0\)).
2. If \(\mathbf{S}_{k}=\mathbf{A}_{k}+\mathbf{I}\), we have \(\overline{\mathbf{S}}_{m,n}\) as \[\overline{\mathbf{S}}_{m,n}=\mathbb{E}\left[\mathbf{A}_{k}+\mathbf{I}\right]_ {m,n}=\begin{cases}p\mathbf{A}_{m,n},&m\neq n;\\ 1,&\text{else}.\end{cases}\] (13) We can also express the real symmetry matrix as \(\overline{\mathbf{S}}=\mathbf{U}\Sigma\mathbf{U}^{T}\) via eigendecomposition where \(\Sigma=\mathrm{diag}\left(\lambda_{1},\lambda_{2},\cdot,\lambda_{N}\right)\). Assuming that \(\mathcal{G}^{+}\) is a fully connected graph, there will be \(\lambda_{1}=1+(N-1)p\) and \(\lambda_{2}=\lambda_{3}=\cdots=\lambda_{N}=1-p\). Thus, choosing a feasible \(p\) and taking \(h_{k}\) into account, \(\sum_{k=1}^{K}h_{k}\overline{\mathbf{S}}^{k}+h_{0}\mathbf{I}\) will be a full-rank matrix as well.

Based on the above analysis, we have proved that \(\mathbb{E}\left[\mathbf{H}\left(\mathbf{S}_{K:0}\right)\right]\) can be full-rank with appropriate filtering coefficients \(h_{k}\) and \(p\). Hence, for a given expected output \(\mathbb{E}\left[\mathbf{u}\right]\), there must be a nontrivial solution \(\mathbf{V}(\mathbf{o}_{t})\) that satisfies (10) with learned \(\mathcal{H}\).

Together with (8) and (9) which show the relationship between \(\mathbf{u}\) and \(\mathbf{V}_{\mathrm{agg}}(\mathbf{o}_{t})\), it becomes reasonable to use SGNN for feature capturing because \(\mathbf{V}(\mathbf{o}_{t})\) can finally be mapped to \(V_{\mathrm{agg}}^{(i)}(\mathbf{o}_{t})\) where

\[V_{\mathrm{agg}}^{(i)}(\mathbf{o}_{t}) =\left[\mathbf{V}_{\mathrm{agg}}(\mathbf{o}_{t})\right]_{i} \tag{14}\] \[=\mathbb{E}_{\mathcal{P},\mathbf{a}_{t}\sim\mathbf{\pi}\left(\cdot\left|\bm {o}_{t}\right.\right)}\left[\sum_{k=t}^{\infty}\gamma^{k-t}r_{k}^{(i)}|\mathbf{o}_{t}, \mathbf{\pi}\right].\]

In other words, the individual value functions take the observations and policies of other agents as well as the dynamics of topology into account in an effort to handle the nonstationarity through stochastic graph filters after sufficient training.

By the way, as \(\mathcal{G}^{+}\) is generalized for all possible \(\mathcal{G}_{t}\) through RES during training, SGNN is able to explore the topologies that may appear, thus learning stable graph signal filters. Therefore, SVMIX ultimately obtain the capability of anti-disturbance.

### _Value Decomposition with Vmix_

Similar to the mixing network in QMIX[17], the VMIX network leverages the filtered signals from SGNN and evaluates the contribution of each agent to generate the aggregated total state value \(V_{\mathrm{tot}}\left(\mathbf{o}_{t}\right)\) on the basis of the global state \(s_{t}\). VMIX consists of multiple MLPs which take \(s_{t}\) as input and output the weights and biases for linear transformations. Finally, the aggregated state value can be formulated as

\[V_{\mathrm{tot}}(\mathbf{o}_{t})=\mathbf{w}_{2}^{\top}\mathbf{V}_{\mathrm{agg}}^{\prime} \left(\mathbf{o}_{t}\right)+b_{2}, \tag{15}\]

where \(\mathbf{V}_{\mathrm{agg}}^{\prime}(\mathbf{o}_{t})=f\left[\mathbf{W}_{1}\mathbf{V}_{\mathrm{agg}} \left(\mathbf{o}_{t}\right)+\mathbf{b}_{1}\right]\). Besides, \(\mathbf{W}_{1}\in\mathbb{R}^{C}\times\mathbb{R}^{N}\), \(\mathbf{b}_{1}\in\mathbb{R}^{C}\), \(\mathbf{w}_{2}\in\mathbb{R}^{C}\), \(b_{2}\in\mathbb{R}\) are the generated hyperparameters indicating non-negative weights and biases from MLPs taking \(s_{t}\) as input while the superscript \(\top\) means the transpose. Besides, \(f(\cdot)\) denotes the nonlinear ELU function
and enables VMIX to produce a nonlinear total value function \(V_{\rm tot}\left(\cdot\right)\).

Thus via VMIX, the individual state values \(\mathbf{V}_{\rm agg}(\mathbf{o}_{t})\) with captured topological features from SGNN are further integrated into the total state value \(V_{\rm tot}(\mathbf{o}_{t})\) by nonlinear transformation, where the weights and biases learn the contribution of each agent under the guidance of the global state \(s_{t}\). Conversely, \(V^{(i)}(o_{t}^{(i)})\) is decomposed from \(V_{\rm tot}(\mathbf{o}_{t})\) via \(\mathbf{V}_{\rm agg}(\mathbf{o}_{t})\). Furthermore, through applying gradient descent and back propagation, the parameters of each critic will be updated and thus an appropriate \(V^{(i)}(\cdot)\) will be learned.

### _The Training of SVMIX_

The SVMIX network aims to learn a joint policy \(\mathbf{\pi}(\cdot|\mathbf{o}_{t};\theta)=\prod_{i=1}^{N}\pi^{(i)}\left(\cdot|o_{t}^{( i)};\theta^{(i)}\right)\) (i.e., the normal distribution \(\mathcal{N}(\mu_{i}(o_{t}^{(i)}),\sigma_{t}^{2}(o_{t}^{(i)}))\) for \(i=1,\cdots,N\)) and a total state value function \(V_{\rm tot}(\cdot;\phi,\eta,\omega)\), where \(\theta=\langle\theta^{(1)},\cdots,\theta^{(N)}\rangle\) and \(\phi=\langle\phi^{(1)},\cdots,\phi^{(N)}\rangle\) are the learnable parameters of actors and critics respectively. Besides, \(\eta\) and \(\omega\) indicate the parameters of SGNN and VMIX, respectively. According to \(\mathbf{\pi}(\cdot|\mathbf{o}_{t};\theta)\), the joint action \(\mathbf{a}_{t}\) is adopted. In particular, \(\mathbf{a}_{t}\) indicates the taken actions of vehicles at time \(t\) while \(\mathbf{\pi}(\mathbf{a}_{t}|\mathbf{o}_{t};\theta)\) and \(V_{\rm tot}(\mathbf{o}_{t};\phi,\eta,\omega)\) participates in the calculation of loss functions with the global reward \(r_{t}\). Inspired by [26], the loss functions are defined as

\[L(\theta)=-\mathbb{E}_{t}\left[\min\left(\rho_{t;\theta}A_{t; \phi,\eta,\omega},{\rm clip}\left(\rho_{t;\theta},\epsilon\right)A_{t;\phi, \eta,\omega}\right)\right], \tag{16}\] \[L(\phi,\eta,\omega)=\frac{1}{2}A_{t;\phi,\eta,\omega}^{2}, \tag{17}\]

where

\[\rho_{t;\theta}=\frac{\mathbf{\pi}(\mathbf{a}_{t}|\mathbf{o}_{t};\theta)}{\mathbf{\pi}(\mathbf{a} _{t}|\mathbf{o}_{t};\theta_{\rm old})} \tag{18}\]

and

\[A_{t;\phi,\eta,\omega}=\sum_{k=t}^{T}\gamma^{k-t}r_{k}-V_{\rm tot}(\mathbf{o}_{t}; \phi,\eta,\omega). \tag{19}\]

Here \(T\) is the length of the episode, the clip function \({\rm clip}(\cdot)\) removes the incentive for moving the ratio \(\rho_{t}\) outside of the interval \([1-\epsilon,1+\epsilon]\) and \(A_{t}\) is the advantage value that evaluates the current policy in the form of Monte-Carlo error. Notably, for training with batches that include time-related samples, (19) will further be slightly modified as

\[A_{j;\phi,\eta,\omega}=\sum_{k=j}^{|\mathcal{B}|}\gamma^{k-j}r_{k} +\gamma^{|\mathcal{B}|-j}V_{\rm tot}(\mathbf{o}_{|\mathcal{B}|+1}; \phi,\eta,\omega) \tag{20}\] \[-V_{\rm tot}(\mathbf{o}_{j};\phi,\eta,\omega),\]

where \(|\mathcal{B}|\) is the batch size and \(j=\{1,\cdots,|\mathcal{B}|\}\) is the index of samples. If the episode ends, (20) will degenerate into the form of (19) as the approximation of subsequent rewards is no longer needed. Finally, \(\theta\) and \(\phi,\eta,\omega\) will be updated through gradient descent to minimize (16) and (17) separately.

```
1:Initialize the underlying graph \(\mathcal{G}^{+}=\{\mathcal{V},\mathcal{E}^{+}\}\), the number of agents \(N\), the batch size \(N_{\rm batch}\), the length of episodes \(T\), the number of PPO epochs \(N_{\rm epoch}\), discount factor \(\gamma\) and constant \(\epsilon\);
2:Initialize the actor and critic networks in PPO agents, the SGNN network and the VMIX network with random parameters \(\theta\), \(\phi\), \(\eta\) and \(\omega\) respectively;
3:Initialize the batch \(\mathcal{B}\leftarrow\varnothing\) and the batch sample counter as counter\(\gets 0\);
4:for every episode do
5:Initialize the ending flag of the episode as done\(\gets 0\);
6:for\(t\gets 1\) to \(T\)dodo
7:Obtain the global state \(s_{t}\) and the joint observation \(\mathbf{o}_{t}=\{o_{t}^{(1)},\cdots,o_{t}^{(N)}\}\) from the environment;
8: Each agent generates the mean \(\mu_{i}(o_{t}^{(i)};\theta^{(i)})\) and the standard deviation \(\sigma_{i}(o_{t}^{(i)};\theta^{(i)})\) the normal distribution \(\mathcal{N}(\mu_{i}(o_{t}^{(i)};\theta^{(i)}),\sigma_{i}^{2}(o_{t}^{(i)}; \theta^{(i)}))\);
9: Each agent chooses the sampled action \(a_{t}^{(i)}\sim\mathcal{N}(\mu_{i}(o_{t}^{(i)};\theta^{(i)}),\sigma_{i}^{2}(o_{ t}^{(i)};\theta^{(i)}))\) ;
10: Obtain the reward \(r_{t}\), \(s_{t+1}\) and \(\mathbf{o}_{t+1}\) at the end of \(t\);
11: Store the tuple \(\langle s_{t},\mathbf{o}_{t},\mathbf{a}_{t},r_{t},s_{t+1},\mathbf{o}_{t+1}\rangle\) in \(\mathcal{B}\) in order of time and set counter\(\leftarrow\)counter\(+1\);
12:if\(t=T\)then
13:done\(\gets 1\);
14:endif
15:ifcounter\(=N_{\rm batch}\) or done\(=1\)then
16: Clone \(\theta_{\rm old}\leftarrow\theta\);
17:for\(n_{\rm epoch}\leftarrow\) 1 to \(N_{\rm epoch}\)dodo
18:for samples in \(\mathcal{B}\)do
19: Obtain \(\mathbf{\pi}(\mathbf{a}_{j}|\mathbf{o}_{j};\theta)\) and \(\mathbf{\pi}(\mathbf{a}_{j}|\mathbf{o}_{j};\theta_{\rm old})\) corresponding to the distributions from the actors and calculate \(\rho_{j;\theta}\) by (18);
20: Obtain the approximated total state value \(V_{\rm tot}(\mathbf{o}_{j};\phi,\eta,\omega)\) through the critic networks, SGNN and VMIX network sequentially in (8)-(15);
21: Calculate \(A_{j;\phi,\eta,\omega}\leftarrow\sum_{k=j}^{|\mathcal{B}|}\gamma^{k-j}r_{k}+(1- \text{done})*\gamma^{|\mathcal{B}|-j}V_{\rm tot}(\mathbf{o}_{|\mathcal{B}|+1}; \phi,\eta,\omega)-V_{\rm tot}(\mathbf{o}_{j};\phi,\eta,\omega)\) ;
22:endfor
23: Update \(\theta\) and \(\phi,\eta,\omega\) according to (16) and (17) separately via batch gradient descent;
24:endfor
25: Initialize \(\mathcal{B}\leftarrow\varnothing\) and counter\(\gets 0\);
26:endif
27:endfor
28:endfor
```

**Algorithm 1** The Training of SVMIX Algorithm

Substantially, SVMIX decomposes \(V^{(i)}(o_{t}^{(i)})\) from \(V_{\rm tot}(\mathbf{o}_{t})\) through gradient descent on the minimization of (17) via VMIX. In particular, with the aid of SGNN, a feasible mapping from \(\mathbf{V}(\mathbf{o}_{t})\) to \(V_{\rm agg}^{(i)}(\mathbf{o}_{t})\) which takes the dynamics of topology and the influence from other agents into account is learned first. Afterwards, a satisfied joint policy will be learned according to (16). Additionally, the RES model in SGNN enhances the anti-disturbance ability of SVMIX and the exploration ability by capturing the dynamic topological feature of \(\mathcal{G}_{t}\) through \(\mathcal{H}\) and \(p\). In other words, for all possible inputs \(\mathbf{V}(\mathbf{o}_{t})\) related to a specific topological connection, SGNN helps to finally get the
corresponding state value \(V^{(i)}(o_{t}^{(i)})\) for each agent and handles topological dynamics based on RES. To sum up, we describe the training procedure of SVMIX algorithm in Algorithm 1.

## V Experimental Settings and Numerical Results

In this section, we try to verify the performance of SVMIX in autonomous vehicle control and explain the advantage of our proposed algorithm over other MARL methods. We implement two simulation scenarios on Flow [44, 45], which is a traffic control benchmarking framework for mixed autonomy traffic. As illustrated in Fig. 4, the "Figure Eight" scenario and the "Merge" scenario are chosen as two typical cases to evaluate the performance of our method.

### _The Settings for "Figure Eight" and Simulation Results_

The "Figure Eight" scenario in Fig. 4 (a) consists of fixed \(M=14\) (i.e., \(M_{t}=14\) for any \(t\)) vehicles running circularly along a one-way lane with an intersection. So the vehicle must strive for a velocity close to its desired velocity, and timely adjust the velocity when passing through the intersection so as to avoid collisions. What's more, we deploy \(7\) manned vehicles and \(N=7\) DRL-driven vehicles alternately, where the former vehicles are controlled by Intelligent Driver Model (IDM) defined in [8] while the latter ones are controlled by MARL methods. Notably, all the vehicles will perform emergency braking if they are about to crash, and once a collision occurs the episode will be terminated immediately. The corresponding MDP for DRL agents in "Figure Eight" is defined as below.

State and ObservationHere the state contains the information of all the vehicles. Besides, each DRL-driven vehicle can only observe the information of the vehicles ahead and behind. Therefore we have \(s_{t}=\{v_{t}^{(1)},z_{t,t}^{(1)},v_{t}^{(2)},z_{t}^{(2)},\cdots,v_{t}^{(M)}, z_{t}^{(M)}\}\) and \(o_{t}^{(i)}=\{v_{t}^{(i_{\rm ahead,t})},z_{t}^{(i_{\rm ahead,t})},v_{t}^{(i)},z_{ t}^{(i_{\rm behind,t})},z_{t}^{(i_{\rm behind,t})}\}\) as the observation of DRL-driven vehicle \(i\), where \(i_{\rm ahead,t}\) and \(i_{\rm behind,t}\) are the preceding and following vehicles of \(i\) at time-step \(t\), respectively.

ActionAs each DRL-driven vehicle only needs to consider the acceleration (i.e. \(a^{(i)}=u^{(i)}\)), \(\mathbf{a}_{t}=\{u^{(1)},u^{(2)},\cdots,u^{(N)}\}\).

RewardThe reward function is the same as (3) where \(\alpha=0\) so the penalty term is ignored.

In our setting, the number of episodes \(N_{\rm episode}\) is \(300\), while each episode has a maximum of \(L=1500\) iterations in case of no collision. Also, we design the system utility \(U=\frac{1}{L}\sum_{t=1}^{T}r_{t}\) as a concrete form of (4) as the evaluation metrics.

For SVMIX, we use the same architecture as depicted in Fig. 2 with \(F=32\), \(K=3\) and \(p=0.7\). Besides, we compare the performance of SVMIX with other MARL methods, including Federated Independent Reinforcement Learning (FIRL) [27], QMIX [17] and Multi-Graph Attention Network (MGAN) [20], where FIRL combines federated learning with consensus algorithm based on multiple PPO agents, and MGAN adibits Graph AttenTion network (GAT) into VD-based MARL architecture. Both MGAN and SVMIX are contingent on a complete graph for inter-agent communication, while FIRL concentrates on a specified graph for consensus. Especially, FIRL uploads the gradient of each agent to a centralized virtual agent every \(\tau\) updates. Besides, an optimal baseline is rendered by Flow where all the \(14\) vehicles are controlled by IDM, as it is a typical car-following model that contains lots of prior knowledge. It's worth noting that all the methods deal with a global reward, while each agent in FIRL uses the local reward identical as the global reward. Typical parameter settings are summarized in Table I.

Fig. 5 compares the average system utility curves of different MARL methods and the optimal baseline in \(10\) independent simulations respectively, while Table II provides the corresponding means and standard deviations of each MARL method. In particular, the utility is computed by averaging the results of another five testing episodes after every 10

Fig. 4: The two scenarios for simulations. Here the red vehicles are the DRL-driven vehicles, the blue vehicles are the manned vehicles observed by the DRL-driven vehicles while the white vehicles are the manned vehicles which are not observed in the state space.
training episodes in a simulation, when each agent directly takes the mean \(\mu_{i}(o_{i}^{(i)})\) as a deterministic action. It can be observed from Fig. 5 that FIRL yields less utility with the relatively stable performance, while the QMIX achieves a higher utility than FIRL, since the VD learns different contributions of agents with a precise total value function but fluctuates drastically due to the occurrence of collisions. MGAN further improves the convergence speed and somewhat reduces the fluctuations thanks to the captured topological features by GAT. Comparatively, benefiting from the stochastic graph filters with enhanced capability to capture dynamic topological features, SVMIX outperforms the other algorithms as it converges faster with a smaller training variance while learns a safer joint policy with less fluctuations, as well as maintains a high utility.

In terms of the communication overhead, Table III compares SVMIX and FIRL, where both of the methods will be updated once with a batch containing \(N_{\mathrm{batch}}\) samples. Notably, the VD-based methods have almost the same communication overheads. Consequently, SVMIX reduces \(12.13\%\) of the communication overheads compared to FIRL as it only needs to transmit a batch of values rather than the gradients of all the parameters. The reason lies in that the communication overheads of FIRL are proportional to the number of parameters (i.e., \(N_{\mathrm{para}}\)) while the communication overheads of SVMIX are proportional to the number of batches (i.e., \(N_{\mathrm{batch}}\)) and the number of epochs (i.e., \(N_{\mathrm{epoch}}\)).

### _The Settings for "Merge" and Simulation Results_

As depicted in Fig. 4 (b), the "Merge" scenario simulates the on-ramp merging on the highway. Similar to "Figure Eight", it's essential for vehicles to consider how to avoid collisions and congestion due to the meeting at the merging point. As "Merge" is an unclosed network, there assumes to exist a traffic flow where the vehicles frequently flow in and out, leading to the variations in the number of vehicles. Following the definition of the model in Section III, the scenario is set to allow at most \(2,100\) vehicles to flow in per hour, including \(2,000\) vehicles at most on the trunk road and \(100\) vehicles at most on the ramp per hour. In particular, among the \(2,000\) vehicles, \(25\%\) of them will be assigned as DRL-driven vehicles. Consistent with the MDP defined in Section III, we consider an MDP as follows.

State and ObservationHere the state contains the information of the observable vehicles (just like the red and blue vehicles in Fig. 4 (b)) instead of all the vehicles. Also, each DRL-driven vehicle can only observe the information of the preceding and following vehicles. So \(s_{t}\) only consists of the positions and velocities of DRL-driven vehicles as well as the vehicles ahead of and behind them. Furthermore, we fix the number of algorithmically involved DRL-driven vehicles as \(N=13\) and if the practical number of vehicles \(N_{t}>N\), the other \(N_{t}-N\) vehicles will be treated as manned vehicles; otherwise, the state will be padded with

Fig. 5: The average system utility of different methods under the Scenario “Figure Eight”. Obviously all of the methods converge within about 150 episodes, where FIRL learns a stable policy at the cost of the convergence speed and utility. QMIX achieves a higher utility thanks to VD and MGAN further improves the convergence speed and reduces the fluctuation owing to the use of GAT. SVMIX outperforms the other methods in terms of the convergence speed and stability with a high utility.
zeros, as if there are \(N\) DRL-driven vehicles. The dimension of the state space consequently remain unchanged. Therefore the state is represented by \(s_{t}=\{o_{t}^{(1)},o_{t}^{(2)},\cdots,o_{t}^{(N)}\}\) where \(o_{t}^{(i)}=\{v_{t}^{(i_{\text{thanal},t})},z_{t}^{(i_{\text{hand},t})},v_{t}^{( i)},z_{t}^{(i_{\text{thanal},t})},v_{t}^{(i_{\text{thanal},t})},z_{t}^{(i_{\text{ thanal},t})}\}\) expresses the observation of DRL-driven vehicle \(i\) where \(i=1,2,\cdots,N\).

ActionSame as the above, the \(N=13\) DRL-driven vehicles will choose their actions according to their policies so the dimension of the joint action space is also immutable and \(\mathbf{a}_{t}=\{u^{(1)},u^{(2)},\cdots,u^{(N)}\}\). When \(N_{t}>N\), the first \(N\) vehicles will be treated as DRL-driven vehicles and the others are regarded as manned vehicles. In case \(N_{t}<N\), the extra actions will be ignored during the interaction with the environment. It's worth noting that the underlying graph \(\mathcal{G}^{+}\) is still a complete graph with fixed \(N\) vertices.

RewardThe reward function is the same as (3) where \(\alpha=0.1\) and the penalty term is defined as

\[D(z^{(1)},\cdots,z^{(M_{t})})=\sum_{j=1}^{M_{t}}\max[C_{2}-\|z^{(i_{\text{ thanal},t})}-z^{(i)}\|_{2},0], \tag{21}\]

where \(C_{2}\) is a constant which represents the desired following distance of each vehicle.

In our setting, the number of episodes \(N_{\text{episode}}\) is 300, while each episode has a maximum of \(L=750\) iterations in case of no collision. For SVMIX, we use the same architecture as in Fig. 2 with \(F=32\), \(K=3\) and \(p=0.7\). Besides, we compare the performance of SVMIX with FIRL, QMIX and MGAN with the same detailed setting described earlier in Section V-A. Typical parameter settings are summarized in Table IV.

Fig. 6 compares the average system utility curves of different MARL methods and the optimal baseline in \(5\) independent simulations respectively, while Table V provides the corresponding means and standard deviations of each MARL method. In particular, the utility is computed by averaging the results of another five testing episodes after every \(10\) training episodes in a simulation, when each agent directly takes the mean \(\mu_{i}(o_{t}^{(i)})\) as a deterministic action. Notably, MGAN achieves an unstable and the worst performance shown in Fig. 6(b), and even worse than QMIX as well as FIRL. Especially at the last \(150\) episodes, the curve of MGAN fluctuates violently. This is possibly because the DRL-driven vehicles are irregularly allocated due to the traffic flow and it becomes a degenerative feedback to the learning of attention coefficients. Both QMIX and SVMIX generates a relatively stable curve with higher average utilities than FIRL and MGAN. What's more, benefiting from the stochastic graph filters, SVMIX outperforms QMIX with higher utility and lower variance according to Table V. It is noteworthy that the performance of SVMIX obviously outperforms MGAN in "Merge", which verifies the aforementioned idea that it's necessary to deal with the dynamic topology if we want to capture topological features. Compared with FIRL, SVMIX has similar means of average utilities and seems more unstable as the variances are larger in the scenario. More meaningfully, SVMIX still superiors in terms of communication overheads.

### _Hyperparameters Adjusting for Sgnn in SVMIX_

To clarify the influence of the hyperparameters in SGNN, we further carry out supplementary experiments in "Figure Eight" by changing one hyperparameter and keeping the others the identical. Fig. 7 provides the corresponding results where we get the results from the testing episodes after

Fig. 6: The average system utility of different methods under the Scenario “Merge”. All the methods converge within about \(100\) episodes. FIRL still learns a stable policy and reaches a not bad average utility. MGAN converges quickly but seems unstable in terms of the undulant and low average utility. QMIX and SVMIX both hold a more stable performance, where the average utility of SVMIX is slightly better than QMIX.
\(100,110,\cdots,200\) training episodes (i.e., when SVMIX converges slowly) respectively through \(5\) independent simulations.

Probability \(p\)As shown in Fig. 7(a), keeping \(K=3\) and \(F=32\), we evaluate the performance of SVMIX for \(p\in\{0.1,0.3,0.5,0.7,0.9\}\). For a Bernoulli-sampled RES, the variance of the outputs from SGNN is proportional to \(p(1-p)\) and will be maximized when \(p=0.5\). Obviously, the result of \(p=0.5\) is the most unstable as there are many outliers. The settings of \(p=0.3\) and \(p=0.7\) yield the similar stable performance, while the performance of \(p=0.1\) and \(p=0.9\) seem more unstable. This is because when \(p=0.1\) and \(p=0.9\), SGNN is unable to make sufficient exploration due to the too small or too large probability \(p\) based on RES. Thus a probability like \(p=0.7\) sounds more appropriate for SGNN.

Order of the filter \(K\)Fig. 7(b) demonstrates the influence of the filter order \(K\) by retaining \(p=0.7\) and \(F=32\) with \(K\) spanning from \(2\) to \(6\). Consistent with the analysis in [25], which states that the variance of the outputs from SGNN is proportional to \(K\), the variance of the system utility also becomes greater as \(K\) grows. Thus an order like \(K=2\) or \(K=3\) will be suitable for "Figure Eight".

Number of filters \(F\)Fig. 7(c) demonstrates the influence of the number of filters \(F\) by retaining \(p=0.7\) and \(K=3\) with \(F\in\{16,32,48,64,80\}\). For \(F=16\), the capability of SGNN for feature capturing decreases and thus leads to an unstable performance. As \(F\) increases, SVMIX generally gives better performance, though it slightly suffers from the deficiency of training samples to train a larger neural network.

## VI Conclusion and Discussion

This paper aims to address reward assignment problem in a dynamic environment by Value Decomposition and tentatively puts forward an SGNN based multi-agent actor-critic architecture called SVMIX with PPO as the individual agent. In particular, SGNN, which consists of parallel stochastic graph filters, has been leveraged to enhance the resilience to the environment volatility and thus capably capture dynamic underlying features in a more effective manner. To demonstrate the feasibility of SVMIX, we further explain why SGNN works by clarifying the influence of SGNN on exploring the comprehensive optimal mapping from individual state values to the total state value through theoretical analysis. Moreover, through extensive simulations in two different scenarios, SVMIX manifests itself with a superior capability in terms of the performance like the convergence rate, the mean & the variance of the average system utility and communication overheads.

However, the assumption that each agent can only make decisions based on the observation of itself might be over-emphasized as the communication between agents close to each other is allowed in practical scenarios in the decentralized execution. Moreover, a reward-decomposition method like [46] is able to sufficiently utilize the result of the centralized training during decentralized execution and even conduct decentralized training for fine tuning. In the future, it is meaningful to apply VD on the global reward in order to learn the individual reward functions and incorporate the inter-agent communication more comprehensively.

Fig. 7: The average utility of SVMIX with different structure of SGNN under the corresponding hyperparameters by changing (a) the sampling probability \(p\), (b) the order of each filter \(K\) and (c) the number of filters \(F\) from the testing episodes after \(100,110,\cdots,200\) training episodes through \(5\) independent simulations.