The parameters of the controller , including the timing and magnitude of the torque pulses and the dynamics of the phase variable , can be kept constant in open loop or adapted to the environment in closed loop . Here we demonstrate the implementation of this approach to a mono-ped robot and the optimization of the controller parameters to enhance robustness via policy gradient . Policy gradient was applied in simulations rather than the actual robot due to safety and hardware considerations . Policy gradient methods ignore the potential value of adjusting environment variables : unobservable state features that are randomly determined by the environment in a physical setting , but are controllable in a simulator . This can lead to slow learning , or convergence to suboptimal policies , if the environment variable has a large impact on the transition dynamics . The actor and critic updates are based on stochastic policy-gradient ascent and temporal-difference algorithms , respectively . Unlike conventional RL algorithms , policy-gradient-based algorithms guarantee convergence even with value-function approximation but suffer due to high variance of the policy-gradient estimator . To minimize this variance for an existing algorithm , we derive a stochasticgradient-based novel critic update . Learning from the logged feedback is however subject to biases caused by only observing feedback on recommendations selected by the previous versions of the recommender . In this work , we present a general recipe of addressing such biases in a production top-K recommender system at Youtube , built with a policy-gradient-based algorithm , i.e. REINFORCE . The contributions of the paper are : ( 1 ) scaling REINFORCE to a production recommender system with an action space on the orders of millions ; ( 2 ) applying off-policy correction to address data biases in learning from logged feedback collected from multiple behavior policies ; ( 3 ) proposing a novel top-K off-policy correction to account for our policy recommending multiple items at a time ; ( 4 ) showcasing the value of exploration . GAN introduces a minimax game between a generative model and a discriminative model , where the discriminator can be viewed as the dynamically - updated evaluation metric to guide the tuning of the generated data . To apply GANs to text generation , [ reference ] proposed SeqGAN that models the text generation as a sequential decision making process and trains the generative model with policy gradient methods . MaliGAN [ reference ] ) modifies the orginal GAN objective and proposes a set of training techniques to reduce the potential variance . Update Since the expected reward is generally not differentiable due to unknown dynamics , we use policy gradient methods to estimate the gradient both for the model gradient update ( s ) and the meta - optimization . Since policy gradients are an on - policy algorithm , each additional gradient step during the adaptation of f θ requires new samples from the current policy Subsequently , hessel2018rainbow combined C51 with enhancements such as prioritized experience replay , - step updates , and the dueling architecture , leading to the Rainbow agent , current state - of - the - art in Atari - 57 . The categorical parameterization , using the projected KL loss , has also been used in recent work to improve the critic of a policy gradient algorithm , D4PG , achieving significantly improved robustness and state - of - the - art performance across a variety of continuous control tasks . subsection : - Wasserstein Metric We propose a deep multi-agent reinforcement learning framework that is able to identify and resolve conflicts between aircraft in a high-density , stochastic , and dynamic en route sector with multiple intersections . The proposed framework utilizes an actor-critic model , A2C that incorporates the loss function from Proximal Policy Optimization ( PPO ) to help stabilize the learning process . In addition , we use a centralized learning , decentralized execution scheme where one neural network is learned and shared by all agents in the environment . We validated the proposed framework on 237 clinical exams and achieved a success rate of 83.1 % for risk-based LVEF classification . Our experiments suggests the fusion of the two apical views improves the performance , compared to single-view networks , especially A2C . The proposed solution is promising for segmentation-free detection of high-risk LVEF . We test the impact of prediction timescale using a specific form of auxiliary task in which the input image is used as the prediction target , which we refer to as temporal difference autoencoders ( TD-AE ) . We empirically evaluate the effect of TD-AE on the A2C algorithm in the VizDoom environment using different prediction timescales . While we do not observe a clear relationship between the prediction timescale on performance , we make the following observations : 1 ) using auxiliary tasks allows us to reduce the trajectory length of the A2C algorithm , 2 ) in some cases temporally extended TD-AE performs better than a straight autoencoder , 3 ) performance with auxiliary tasks is sensitive to the weight placed on the auxiliary loss , 4 ) despite this sensitivity , auxiliary tasks improved performance without extensive hyper-parameter tuning . Deep Q Learning , Actor-Critic ( AC ) and Advantaged Actor-Critic ( A2C ) algorithm are compared . And to further maximize the performance , we specifically improved A2C algorithm towards the multi-UAV scene by sharing experiences between UAVs to expedite the training process . Our experimental result shows our Experience-shared A2C ( ES-A2C ) algorithm leads to a higher performance and a shorter training period . The experimental results demonstrate that DRL methods successfully teach agents to battle in these scenarios . In addition , the ACKTR agents significantly outperform the A2C agents in terms of all the metrics by a significant margin . In this work , we propose a novel reinforcement learning-based methodology that navigates the optimization space without human intervention . We demonstrate the training of an Advantage Actor Critic ( A2C ) agent that seeks to minimize area subject to a timing constraint . Using the proposed methodology , designs can be optimized autonomously with no-humans in-loop . In this paper , we propose an efficient deep reinforcement learning ( DRL ) method to tackle visual navigation tasks . We present the synchronous advantage actor-critic ( A2C ) with generalized advantage estimator ( GAE ) algorithm . The A2C enables agents to learn from multiple processes , which significantly reduces the training time . Moreover , we derive an MARL algorithm called Shapley Q-value deep deterministic policy gradient ( SQDDPG ) , using Shapley Q-value as the critic for each agent . We evaluate SQDDPG on Cooperative Navigation , Prey-and-Predator and Traffic Junction , compared with the state-of-the-art algorithms , e.g. , MADDPG , COMA , Independent DDPG and Independent A2C . In the experiments , SQDDPG shows a significant improvement on the convergence rate . The evaluation results show that the concepts of BTG , prevention and detection mechanism , and obligation provide more flexible access than other current access control models in WSNs . Additionally , we compare the BTG-AC model with an adaptive access control ( A2C ) model , which has similar properties , for further evaluation . Alongside with the comparison , the advantages and disadvantages of BTG-AC over current WSN access control models are presented . The proposed method consists of a state aggregation step based on tabular reinforcement learning to extract features and construct states . The state aggregation policy is applied to each problem instance of the knapsack problem , which is used with Advantage Actor Critic ( A2C ) algorithm to train a policy through which the items are sequentially selected at each time step . The method is a constructive solution approach and the process of selecting items is repeated until the final solution is obtained . The benchmarks here , built as OpenAI Gym environments , will be open-sourced to encourage innovation in molecular design algorithms and to enable usage by those without a background in chemistry . Finally , this work explores recent development in reinforcement-learning methods with excellent sample complexity ( the A2C and PPO algorithms ) and investigates their behavior in molecular generation , demonstrating significant performance gains compared to standard reinforcement learning techniques . The possibilities of estimation of the vehicle orientation angle based on a convolutional neural network are also investigated . Obtained additional features are used in the modern effective reinforcement learning methods of Soft Actor Critic and Rainbow , which allows to accelerate the convergence of its learning process . To demonstrate the operation of the developed system , an intersection simulator was developed , at which a number of model experiments were carried out . And fourth , the view of maximum entropy RL as probabilistic inference provides a foundation for building hierarchical policies that can solve complex and sparse reward tasks . In the first part , we will devise new algorithms based on this framework , starting from soft Q-learning that learns expressive energy-based policies , to soft actor-critic that provides simplicity and convenience of actor-critic methods , and ending with automatic temperature adjustment scheme that practically eliminates the need for hyperparameter tuning , which is a crucial feature for real-world applications where tuning of hyperparameters can be prohibitively expensive . In the second part , we will discuss extensions enabled by the inherent stochasticity of maximum entropy polices , including compositionality and hierarchical learning . The approach leverages input perturbations commonly used in computer vision tasks to regularize the value function . Existing model-free approaches , such as Soft Actor-Critic ( SAC ) , are not able to train deep networks effectively from image pixels . However , the addition of our augmentation method dramatically improves SAC 's performance , enabling it to reach state-of-the-art performance on the DeepMind control suite , surpassing model-based ( Dreamer , PlaNet , and SLAC ) methods and recently proposed contrastive learning ( CURL ) . Both of these challenges limit the applicability of such methods to real-world domains . In this paper , we describe Soft Actor-Critic ( SAC ) , our recently introduced off-policy actor-critic algorithm based on the maximum entropy RL framework . In this framework , the actor aims to simultaneously maximize expected return and entropy . The possibility for the team members to communicate before gameplay --- that is , coordinate their strategies ex ante --- makes the use of behavioral strategies unsatisfactory . We introduce Soft Team Actor-Critic ( STAC ) as a solution to the team 's coordination problem that does not require any prior domain knowledge . STAC allows team members to effectively exploit ex ante communication via exogenous signals that are shared among the team . To deal with incomplete and inaccurate distribution network models , we propose a safe off-policy deep reinforcement learning algorithm to solve Volt-VAR control problems in a model-free manner . The Volt-VAR control problem is formulated as a constrained Markov decision process with discrete action space , and solved by our proposed constrained soft actor-critic algorithm . Our proposed reinforcement learning algorithm achieves scalability , sample efficiency , and constraint satisfaction by synergistically combining the merits of the maximum-entropy framework , the method of multiplier , a device-decoupled neural network structure , and an ordinal encoding scheme . We show that the learning of a state-action return distribution function can be used to improve the estimation accuracy of the Q-value . We combine the distributional return function within the maximum entropy RL framework in order to develop what we call the Distributional Soft Actor-Critic algorithm , DSAC , which is an off-policy method for continuous control setting . Unlike traditional distributional Q algorithms which typically only learn a discrete return distribution , DSAC can directly learn a continuous return distribution by truncating the difference between the target and current return distribution to prevent gradient explosion . The agent 's action space is then augmented with macro-actions identified by the grammar . We apply this framework to Double Deep Q-Learning ( AG-DDQN ) and a discrete action version of Soft Actor-Critic ( AG-SAC ) and find that it improves performance in 8 out of 8 tested Atari games ( median + 31 % , max + 668 % ) and 19 out of 20 tested Atari games ( median + 96 % , maximum + 3,756 % ) respectively without substantive hyperparameter tuning . We also show that AG-SAC beats the model-free state-of-the-art for sample efficiency in 17 out of the 20 tested Atari games ( median + 62 % , maximum + 13,140 % ) , again without substantive hyperparameter tuning . Our \textit{soft-robust } framework is an attempt to overcome this issue . In this paper , we present a novel Soft-Robust Actor-Critic algorithm ( SR-AC ) . It learns an optimal policy with respect to a distribution over an uncertainty set and stays robust to model uncertainty but avoids the conservativeness of robust strategies . Applying policy iteration to maximize the derived lower bound , we propose a practical algorithm named variational maximum mutual information multi-agent actor-critic ( VM3-AC ) , which follows centralized learning with decentralized execution ( CTDE ) . We evaluated VM3-AC for several games requiring coordination , and numerical results show that VM3-AC outperforms MADDPG and other MARL algorithms in multi-agent tasks requiring coordination . Concretely , EPC maintains multiple sets of agents in each stage , performs mix-and-match and fine-tuning over these sets and promotes the sets of agents with the best adaptability to the next stage . We implement EPC on a popular MARL algorithm , MADDPG , and empirically show that our approach consistently outperforms baselines by a large margin as the number of agents grows exponentially . To address this problem , we present Decomposed Multi-Agent Deep Deterministic Policy Gradient ( DE-MADDPG ) : a novel cooperative multi-agent reinforcement learning framework that simultaneously learns to maximize the global and local rewards . We evaluate our solution on the challenging defensive escort team problem and show that our solution achieves a significantly better and more stable performance than the direct adaptation of the MADDPG algorithm . In this paper , we propose a novel approach , called MAGNet , to multi-agent reinforcement learning that utilizes a relevance graph representation of the environment obtained by a self-attention mechanism , and a message-generation technique . We applied our MAGnet approach to the synthetic predator-prey multi-agent environment and the Pommerman game and the results show that it significantly outperforms state-of-the-art MARL solutions , including Multi-agent Deep Q-Networks ( MADQN ) , Multi-agent Deep Deterministic Policy Gradient ( MADDPG ) , and QMIX . We noticed that during the training phase the agents developed a differential preference for each goal , which hindered generalization . To alleviate the above problem , we propose simple changes to the MADDPG training algorithm which leads to better generalization against unseen agents . We believe that training protocols promoting more active intention reading mechanisms , e.g. by preventing simple symmetry-breaking solutions , is a promising direction towards achieving a more robust generalization in different cooperative and competitive tasks . We demonstrate our approach by combining it with MADDPG and verifing the performance in two homogeneous and one heterogeneous environments . In the best performing scenario , the MADDPG with experience augmentation reaches to the convergence reward of vanilla MADDPG with 1/4 realistic time , and its convergence beats the original model by a significant margin . Our ablation studies show that experience augmentation is a crucial ingredient which accelerates the training process and boosts the convergence . In STAPP , the MUTAPP problem is first constructed as a multi-agent system . Then , the MADDPG framework is used to train the system to solve target assignment and path planning simultaneously according to a corresponding reward structure . The proposed system can deal with dynamic environments effectively as its execution only requires the locations of the UAVs , targets , and threat areas . Without effective techniques , learning effective policies based on the joint observation of agents can be intractable , particularly when sensor data perceived by each agent is high-dimensional . Extending the multi-agent actor-critic method MADDPG , this paper presents Rec-MADDPG , an end-to-end reinforcement learning method for multi-agent continuous control in a cooperative environment . To ease end-to-end learning in a multi-agent setting , we proposed two embedding mechanisms , joint and independent embedding , to project agents ' joint sensor observation to low-dimensional features . Multi-Agent Deterministic Policy Gradient ( MADDPG ) is a very useful algorithm in multi-agent domains . We analyze the algorithm and find that MADDPG uses deep neural networks ( DNNs ) as a Q function model .
The parameters of the controller , including the timing and magnitude of the torque pulses and the dynamics of the phase variable , can be kept constant in open loop or adapted to the environment in closed loop . Here we demonstrate the implementation of this approach to a mono-ped robot and the optimization of the controller parameters to enhance robustness via policy gradient . Policy gradient was applied in simulations rather than the actual robot due to safety and hardware considerations .
Policy gradient methods ignore the potential value of adjusting environment variables : unobservable state features that are randomly determined by the environment in a physical setting , but are controllable in a simulator . This can lead to slow learning , or convergence to suboptimal policies , if the environment variable has a large impact on the transition dynamics .
The actor and critic updates are based on stochastic policy-gradient ascent and temporal-difference algorithms , respectively . Unlike conventional RL algorithms , policy-gradient-based algorithms guarantee convergence even with value-function approximation but suffer due to high variance of the policy-gradient estimator . To minimize this variance for an existing algorithm , we derive a stochasticgradient-based novel critic update .
Learning from the logged feedback is however subject to biases caused by only observing feedback on recommendations selected by the previous versions of the recommender . In this work , we present a general recipe of addressing such biases in a production top-K recommender system at Youtube , built with a policy-gradient-based algorithm , i.e. REINFORCE . The contributions of the paper are : ( 1 ) scaling REINFORCE to a production recommender system with an action space on the orders of millions ; ( 2 ) applying off-policy correction to address data biases in learning from logged feedback collected from multiple behavior policies ; ( 3 ) proposing a novel top-K off-policy correction to account for our policy recommending multiple items at a time ; ( 4 ) showcasing the value of exploration .
GAN introduces a minimax game between a generative model and a discriminative model , where the discriminator can be viewed as the dynamically - updated evaluation metric to guide the tuning of the generated data . To apply GANs to text generation , [ reference ] proposed SeqGAN that models the text generation as a sequential decision making process and trains the generative model with policy gradient methods . MaliGAN [ reference ] ) modifies the orginal GAN objective and proposes a set of training techniques to reduce the potential variance .
Update Since the expected reward is generally not differentiable due to unknown dynamics , we use policy gradient methods to estimate the gradient both for the model gradient update ( s ) and the meta - optimization . Since policy gradients are an on - policy algorithm , each additional gradient step during the adaptation of f θ requires new samples from the current policy
Subsequently , hessel2018rainbow combined C51 with enhancements such as prioritized experience replay , - step updates , and the dueling architecture , leading to the Rainbow agent , current state - of - the - art in Atari - 57 . The categorical parameterization , using the projected KL loss , has also been used in recent work to improve the critic of a policy gradient algorithm , D4PG , achieving significantly improved robustness and state - of - the - art performance across a variety of continuous control tasks . subsection : - Wasserstein Metric
We propose a deep multi-agent reinforcement learning framework that is able to identify and resolve conflicts between aircraft in a high-density , stochastic , and dynamic en route sector with multiple intersections . The proposed framework utilizes an actor-critic model , A2C that incorporates the loss function from Proximal Policy Optimization ( PPO ) to help stabilize the learning process . In addition , we use a centralized learning , decentralized execution scheme where one neural network is learned and shared by all agents in the environment .
We validated the proposed framework on 237 clinical exams and achieved a success rate of 83.1 % for risk-based LVEF classification . Our experiments suggests the fusion of the two apical views improves the performance , compared to single-view networks , especially A2C . The proposed solution is promising for segmentation-free detection of high-risk LVEF .
We test the impact of prediction timescale using a specific form of auxiliary task in which the input image is used as the prediction target , which we refer to as temporal difference autoencoders ( TD-AE ) . We empirically evaluate the effect of TD-AE on the A2C algorithm in the VizDoom environment using different prediction timescales . While we do not observe a clear relationship between the prediction timescale on performance , we make the following observations : 1 ) using auxiliary tasks allows us to reduce the trajectory length of the A2C algorithm , 2 ) in some cases temporally extended TD-AE performs better than a straight autoencoder , 3 ) performance with auxiliary tasks is sensitive to the weight placed on the auxiliary loss , 4 ) despite this sensitivity , auxiliary tasks improved performance without extensive hyper-parameter tuning .
Deep Q Learning , Actor-Critic ( AC ) and Advantaged Actor-Critic ( A2C ) algorithm are compared . And to further maximize the performance , we specifically improved A2C algorithm towards the multi-UAV scene by sharing experiences between UAVs to expedite the training process . Our experimental result shows our Experience-shared A2C ( ES-A2C ) algorithm leads to a higher performance and a shorter training period .
The experimental results demonstrate that DRL methods successfully teach agents to battle in these scenarios . In addition , the ACKTR agents significantly outperform the A2C agents in terms of all the metrics by a significant margin .
In this work , we propose a novel reinforcement learning-based methodology that navigates the optimization space without human intervention . We demonstrate the training of an Advantage Actor Critic ( A2C ) agent that seeks to minimize area subject to a timing constraint . Using the proposed methodology , designs can be optimized autonomously with no-humans in-loop .
In this paper , we propose an efficient deep reinforcement learning ( DRL ) method to tackle visual navigation tasks . We present the synchronous advantage actor-critic ( A2C ) with generalized advantage estimator ( GAE ) algorithm . The A2C enables agents to learn from multiple processes , which significantly reduces the training time .
Moreover , we derive an MARL algorithm called Shapley Q-value deep deterministic policy gradient ( SQDDPG ) , using Shapley Q-value as the critic for each agent . We evaluate SQDDPG on Cooperative Navigation , Prey-and-Predator and Traffic Junction , compared with the state-of-the-art algorithms , e.g. , MADDPG , COMA , Independent DDPG and Independent A2C . In the experiments , SQDDPG shows a significant improvement on the convergence rate .
The evaluation results show that the concepts of BTG , prevention and detection mechanism , and obligation provide more flexible access than other current access control models in WSNs . Additionally , we compare the BTG-AC model with an adaptive access control ( A2C ) model , which has similar properties , for further evaluation . Alongside with the comparison , the advantages and disadvantages of BTG-AC over current WSN access control models are presented .
The proposed method consists of a state aggregation step based on tabular reinforcement learning to extract features and construct states . The state aggregation policy is applied to each problem instance of the knapsack problem , which is used with Advantage Actor Critic ( A2C ) algorithm to train a policy through which the items are sequentially selected at each time step . The method is a constructive solution approach and the process of selecting items is repeated until the final solution is obtained .
The benchmarks here , built as OpenAI Gym environments , will be open-sourced to encourage innovation in molecular design algorithms and to enable usage by those without a background in chemistry . Finally , this work explores recent development in reinforcement-learning methods with excellent sample complexity ( the A2C and PPO algorithms ) and investigates their behavior in molecular generation , demonstrating significant performance gains compared to standard reinforcement learning techniques .
The possibilities of estimation of the vehicle orientation angle based on a convolutional neural network are also investigated . Obtained additional features are used in the modern effective reinforcement learning methods of Soft Actor Critic and Rainbow , which allows to accelerate the convergence of its learning process . To demonstrate the operation of the developed system , an intersection simulator was developed , at which a number of model experiments were carried out .
And fourth , the view of maximum entropy RL as probabilistic inference provides a foundation for building hierarchical policies that can solve complex and sparse reward tasks . In the first part , we will devise new algorithms based on this framework , starting from soft Q-learning that learns expressive energy-based policies , to soft actor-critic that provides simplicity and convenience of actor-critic methods , and ending with automatic temperature adjustment scheme that practically eliminates the need for hyperparameter tuning , which is a crucial feature for real-world applications where tuning of hyperparameters can be prohibitively expensive . In the second part , we will discuss extensions enabled by the inherent stochasticity of maximum entropy polices , including compositionality and hierarchical learning .
The approach leverages input perturbations commonly used in computer vision tasks to regularize the value function . Existing model-free approaches , such as Soft Actor-Critic ( SAC ) , are not able to train deep networks effectively from image pixels . However , the addition of our augmentation method dramatically improves SAC 's performance , enabling it to reach state-of-the-art performance on the DeepMind control suite , surpassing model-based ( Dreamer , PlaNet , and SLAC ) methods and recently proposed contrastive learning ( CURL ) .
Both of these challenges limit the applicability of such methods to real-world domains . In this paper , we describe Soft Actor-Critic ( SAC ) , our recently introduced off-policy actor-critic algorithm based on the maximum entropy RL framework . In this framework , the actor aims to simultaneously maximize expected return and entropy .
The possibility for the team members to communicate before gameplay --- that is , coordinate their strategies ex ante --- makes the use of behavioral strategies unsatisfactory . We introduce Soft Team Actor-Critic ( STAC ) as a solution to the team 's coordination problem that does not require any prior domain knowledge . STAC allows team members to effectively exploit ex ante communication via exogenous signals that are shared among the team .
To deal with incomplete and inaccurate distribution network models , we propose a safe off-policy deep reinforcement learning algorithm to solve Volt-VAR control problems in a model-free manner . The Volt-VAR control problem is formulated as a constrained Markov decision process with discrete action space , and solved by our proposed constrained soft actor-critic algorithm . Our proposed reinforcement learning algorithm achieves scalability , sample efficiency , and constraint satisfaction by synergistically combining the merits of the maximum-entropy framework , the method of multiplier , a device-decoupled neural network structure , and an ordinal encoding scheme .
We show that the learning of a state-action return distribution function can be used to improve the estimation accuracy of the Q-value . We combine the distributional return function within the maximum entropy RL framework in order to develop what we call the Distributional Soft Actor-Critic algorithm , DSAC , which is an off-policy method for continuous control setting . Unlike traditional distributional Q algorithms which typically only learn a discrete return distribution , DSAC can directly learn a continuous return distribution by truncating the difference between the target and current return distribution to prevent gradient explosion .
The agent 's action space is then augmented with macro-actions identified by the grammar . We apply this framework to Double Deep Q-Learning ( AG-DDQN ) and a discrete action version of Soft Actor-Critic ( AG-SAC ) and find that it improves performance in 8 out of 8 tested Atari games ( median + 31 % , max + 668 % ) and 19 out of 20 tested Atari games ( median + 96 % , maximum + 3,756 % ) respectively without substantive hyperparameter tuning . We also show that AG-SAC beats the model-free state-of-the-art for sample efficiency in 17 out of the 20 tested Atari games ( median + 62 % , maximum + 13,140 % ) , again without substantive hyperparameter tuning .
Our \textit{soft-robust } framework is an attempt to overcome this issue . In this paper , we present a novel Soft-Robust Actor-Critic algorithm ( SR-AC ) . It learns an optimal policy with respect to a distribution over an uncertainty set and stays robust to model uncertainty but avoids the conservativeness of robust strategies .
Applying policy iteration to maximize the derived lower bound , we propose a practical algorithm named variational maximum mutual information multi-agent actor-critic ( VM3-AC ) , which follows centralized learning with decentralized execution ( CTDE ) . We evaluated VM3-AC for several games requiring coordination , and numerical results show that VM3-AC outperforms MADDPG and other MARL algorithms in multi-agent tasks requiring coordination .
Concretely , EPC maintains multiple sets of agents in each stage , performs mix-and-match and fine-tuning over these sets and promotes the sets of agents with the best adaptability to the next stage . We implement EPC on a popular MARL algorithm , MADDPG , and empirically show that our approach consistently outperforms baselines by a large margin as the number of agents grows exponentially .
To address this problem , we present Decomposed Multi-Agent Deep Deterministic Policy Gradient ( DE-MADDPG ) : a novel cooperative multi-agent reinforcement learning framework that simultaneously learns to maximize the global and local rewards . We evaluate our solution on the challenging defensive escort team problem and show that our solution achieves a significantly better and more stable performance than the direct adaptation of the MADDPG algorithm .
In this paper , we propose a novel approach , called MAGNet , to multi-agent reinforcement learning that utilizes a relevance graph representation of the environment obtained by a self-attention mechanism , and a message-generation technique . We applied our MAGnet approach to the synthetic predator-prey multi-agent environment and the Pommerman game and the results show that it significantly outperforms state-of-the-art MARL solutions , including Multi-agent Deep Q-Networks ( MADQN ) , Multi-agent Deep Deterministic Policy Gradient ( MADDPG ) , and QMIX .
We noticed that during the training phase the agents developed a differential preference for each goal , which hindered generalization . To alleviate the above problem , we propose simple changes to the MADDPG training algorithm which leads to better generalization against unseen agents . We believe that training protocols promoting more active intention reading mechanisms , e.g. by preventing simple symmetry-breaking solutions , is a promising direction towards achieving a more robust generalization in different cooperative and competitive tasks .
We demonstrate our approach by combining it with MADDPG and verifing the performance in two homogeneous and one heterogeneous environments . In the best performing scenario , the MADDPG with experience augmentation reaches to the convergence reward of vanilla MADDPG with 1/4 realistic time , and its convergence beats the original model by a significant margin . Our ablation studies show that experience augmentation is a crucial ingredient which accelerates the training process and boosts the convergence .
In STAPP , the MUTAPP problem is first constructed as a multi-agent system . Then , the MADDPG framework is used to train the system to solve target assignment and path planning simultaneously according to a corresponding reward structure . The proposed system can deal with dynamic environments effectively as its execution only requires the locations of the UAVs , targets , and threat areas .
Without effective techniques , learning effective policies based on the joint observation of agents can be intractable , particularly when sensor data perceived by each agent is high-dimensional . Extending the multi-agent actor-critic method MADDPG , this paper presents Rec-MADDPG , an end-to-end reinforcement learning method for multi-agent continuous control in a cooperative environment . To ease end-to-end learning in a multi-agent setting , we proposed two embedding mechanisms , joint and independent embedding , to project agents ' joint sensor observation to low-dimensional features .
Multi-Agent Deterministic Policy Gradient ( MADDPG ) is a very useful algorithm in multi-agent domains . We analyze the algorithm and find that MADDPG uses deep neural networks ( DNNs ) as a Q function model .
Policy gradient optimization of controllers for natural dynamic mono-pedal gait. Fingerprint Policy Optimisation for Robust Reinforcement Learning Actor-Critic Algorithms for Variance Minimization Top-K Off-Policy Correction for a REINFORCE Recommender System Long Text Generation via Adversarial Training with Leaked Information Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks Implicit Quantile Networks for Distributional Reinforcement Learning Autonomous Separation Assurance in An High-Density En Route Sector: A Deep Multi-Agent Reinforcement Learning Approach Automatic Detection of Patients with a High Risk of Systolic Cardiac Failure in Echocardiography Work in Progress: Temporally Extended Auxiliary Tasks Multi-UAV Automatic Dynamic Obstacle Avoidance with Experience-shared A2C Learning Battles in ViZDoom via Deep Reinforcement Learning DRiLLS: Deep Reinforcement Learning for Logic Synthesis Visual Navigation with Actor-Critic Deep Reinforcement Learning Shapley Q-Value: A Local Reward Approach to Solve Global Reward Games BTG-AC: Break-the-Glass Access Control Model for Medical Data in Wireless Sensor Networks A State Aggregation Approach for Solving Knapsack Problem with Deep Reinforcement Learning Exploring Deep Recurrent Models with Reinforcement Learning for Molecule Design Object Detection with Deep Neural Networks for Reinforcement Learning in the Task of Autonomous Vehicles Path Planning at the Intersection Acquiring Diverse Robot Skills via Maximum Entropy Deep Reinforcement Learning Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels Soft Actor-Critic Algorithms and Applications Coordination in Adversarial Sequential Team Games via Multi-Agent Deep Reinforcement Learning Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems Addressing Value Estimation Errors in Reinforcement Learning with a State-Action Return Distribution Function Reinforcement Learning with Structured Hierarchical Grammar Representations of Actions Soft-Robust Actor-Critic Policy-Gradient A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning Multi-Agent Reinforcement Learning for Problems with Combined Individual and Team Reward MAGNet: Multi-agent Graph Network for Deep Multi-agent Reinforcement Learning Do deep reinforcement learning agents model intentions? Experience Augmentation: Boosting and Accelerating Off-Policy Multi-Agent Reinforcement Learning Joint Optimization of Multi-UAV Target Assignment and Path Planning Based on Multi-Agent Reinforcement Learning End-to-End Reinforcement Learning for Multi-agent Continuous Control State Representation Learning for Multi-agent Deep Deterministic Policy Gradient
b4d015032d03da02934816c0e739624c53b3e161 8e3b59014315679426f2c427549784378c37f0f2 e463f2e8d5169de29f103963397a41e1b7447810 f76ac3753cd0b5d584ad5eb04819a4dddba82891 095a37a8316082f916963c131e28b34470ec1670 678e1a89c11f1a383b0ee0ce7e81926d97c22b28 1f66ce26484ecb50af846f9000699c92c1477df3 02904f972a7ef506f4eedac07d2d356fff31356f efe72409a40acedad7b5ba86b82e951c6cfcd317 dfc040af4fff1813bfcb4b6df1389506d3cddfde d1e7f3f432801eaad8b9a0c0edf12f8d7f0c34dd 31b2e06b3f3ee242e298902af6a1e1676918f35d ba9863a2650e9bd2bae0ca8f1a7bb4a6b326fb81 f9815ace0b73632bf15042a89272e4f0d1afdeeb 0782003de0736dd9251dff6b3de28d4519a70aef 4417daf1c4e0d3edc7ccd66f7402ba7d4a922ca9 0718d725fd01fdf147cd7787fa62f814f5723053 7f036fda955ebea5b05b450a431293159ab2812f 12c0751b4f51ed833172a713b7e32390032ead93 cd12687348ba234a9ec6b159209851c3749fc9cd 4f69dcff09b3f6d3af574efa77500a0fd8a2cb3e 6305112add69a21af79c6bafbbecfc35818dfe6e 144130874a16fc3c370a859db816378faced1186 6b75ca8de1da3ab2b55f64a60cb9d299c8d55d75 3f326bef4f7fe77455e7f354417ac5b7781608f1 318656c4d4b27caba3448ed1fd6457b37685e469 f35d2c0c5610652e8495254652b5d3f2ba41cce0 1bd7d816d56988ec06242d71696d4355c04c52ab d999372a266b7f228c8ac14d0c964057f5c5e5ad 40fd95d42c0f56e7cbae2838bd5ecb9e2e71a271 58a7ad30640878900dfb40d94fa860768538aa0d ed28a0599b9ecfa316cacfd675c1666a9f043250 a5e4a2bf553ac4cac8dc98b8710086f4c1558b01
["Medicine","Engineering"] ["Computer Science","Mathematics"] ["Computer Science","Mathematics"] ["Mathematics","Computer Science"] ["Engineering","Computer Science"] ["Computer Science","Medicine"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Mathematics"] ["Computer Science","Medicine"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Engineering","Mathematics"] ["Computer Science","Mathematics"] ["Computer Science"] ["Computer Science","Engineering"] ["Computer Science","Mathematics"] ["Mathematics","Computer Science"] ["Computer Science","Mathematics"] ["Computer Science"] ["Computer Science","Mathematics"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Mathematics","Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"]
2020 2019 2010 2019 2018 2017 2018 2019 2018 2020 2019 2018 2019 2018 2020 2016 2020 2018 2019 2018 2020 2018 2019 2020 2020 2019 2018 2020 2020 2020 2019 2018 2020 2019 2019 2018
ICML EIAT/IETA WSDM AAAI ICML ICML ITSC DLMIA/ML-CDS@MICCAI WiMob CIG IJCNN AAAI ICLR (Workshop) ICLR ICMLA
https://api.semanticscholar.org/b4d015032d03da02934816c0e739624c53b3e161 https://api.semanticscholar.org/8e3b59014315679426f2c427549784378c37f0f2 https://api.semanticscholar.org/e463f2e8d5169de29f103963397a41e1b7447810 https://api.semanticscholar.org/f76ac3753cd0b5d584ad5eb04819a4dddba82891 https://www.semanticscholar.org/paper/485552d2711868b54d5fcddc92c746b09afeab07 https://www.semanticscholar.org/paper/c889d6f98e6d79b89c3a6adf8a921f88fa6ba518 https://www.semanticscholar.org/paper/d85623ffae865f9ef386644dd02d0ea2d6a8c8de https://api.semanticscholar.org/095a37a8316082f916963c131e28b34470ec1670 https://api.semanticscholar.org/678e1a89c11f1a383b0ee0ce7e81926d97c22b28 https://api.semanticscholar.org/1f66ce26484ecb50af846f9000699c92c1477df3 https://api.semanticscholar.org/02904f972a7ef506f4eedac07d2d356fff31356f https://api.semanticscholar.org/efe72409a40acedad7b5ba86b82e951c6cfcd317 https://api.semanticscholar.org/dfc040af4fff1813bfcb4b6df1389506d3cddfde https://api.semanticscholar.org/d1e7f3f432801eaad8b9a0c0edf12f8d7f0c34dd https://api.semanticscholar.org/31b2e06b3f3ee242e298902af6a1e1676918f35d https://api.semanticscholar.org/ba9863a2650e9bd2bae0ca8f1a7bb4a6b326fb81 https://api.semanticscholar.org/f9815ace0b73632bf15042a89272e4f0d1afdeeb https://api.semanticscholar.org/0782003de0736dd9251dff6b3de28d4519a70aef https://api.semanticscholar.org/4417daf1c4e0d3edc7ccd66f7402ba7d4a922ca9 https://api.semanticscholar.org/0718d725fd01fdf147cd7787fa62f814f5723053 https://api.semanticscholar.org/7f036fda955ebea5b05b450a431293159ab2812f https://api.semanticscholar.org/12c0751b4f51ed833172a713b7e32390032ead93 https://api.semanticscholar.org/cd12687348ba234a9ec6b159209851c3749fc9cd https://api.semanticscholar.org/4f69dcff09b3f6d3af574efa77500a0fd8a2cb3e https://api.semanticscholar.org/6305112add69a21af79c6bafbbecfc35818dfe6e https://api.semanticscholar.org/144130874a16fc3c370a859db816378faced1186 https://api.semanticscholar.org/6b75ca8de1da3ab2b55f64a60cb9d299c8d55d75 https://api.semanticscholar.org/3f326bef4f7fe77455e7f354417ac5b7781608f1 https://api.semanticscholar.org/318656c4d4b27caba3448ed1fd6457b37685e469 https://api.semanticscholar.org/f35d2c0c5610652e8495254652b5d3f2ba41cce0 https://api.semanticscholar.org/1bd7d816d56988ec06242d71696d4355c04c52ab https://api.semanticscholar.org/d999372a266b7f228c8ac14d0c964057f5c5e5ad https://api.semanticscholar.org/40fd95d42c0f56e7cbae2838bd5ecb9e2e71a271 https://api.semanticscholar.org/58a7ad30640878900dfb40d94fa860768538aa0d https://api.semanticscholar.org/ed28a0599b9ecfa316cacfd675c1666a9f043250 https://api.semanticscholar.org/a5e4a2bf553ac4cac8dc98b8710086f4c1558b01