enter image description here

For discrete action space, I thought the "conventional method" is to set up the neural network in such a way that the inputs are the states and each of the output node represents possible action with its value being the Q-value for that action, hence the optimal action is the action with the highest Q-value, that is, output node with the biggest value.

But for deep deterministic policy gradient, how to set up the Q function network in such a way that it is a function of the current state (or next state...etc) AND is also a function of the continuous output of the policy network? (like in the eqn 12 , 14?)

Judging from the algorithm, I would need additional input node that takes in the continuous output from the policy network and 1 output node for the Q value for its given state and action. Is that a correct interpretation?

Also, what about "multi-dimensional" action? So instead of the policy network having 1 output node with its continuous output, but rather a multiple output nodes ( which is useful for a robotic arm's angular velocity in X-axis, Y-axis.....etc)

Could I do the same but just extend the dimensions of the neural net?

Has anyone had experience with DDPG? Since I would need to store the replay buffer and have 2 additional network for target policy and target Q, I'd assume it would require a lot of memory. How is the overall training speed as well?


Actor-critic algorithm

Deep learning can solve the problem of high observation dimensional data it only handles discrete and low-dimensional action-spaces. This algorithm relies on finding the action that maximizes the action-value function, which in continuous spaces requires solving a complex optimization problem. Obvious solutions like action-space discretization lead to the explosion of the numbers of discrete actions. One of the solutions is used actor-critic approach. It consists of two components: actor adjust parameters 𝜃 of the stochastic policy 𝜋(𝑠) by stochastic gradient ascent.

Deterministic Policy Gradient

The majority of model-free algorithms are based generalized policy iteration: iterating policy evaluation with policy improvement. Policy evaluation estimates the action-value function $Q_{\omega} = {\nabla}_{\theta}log_{\theta}\pi_{\theta}(a|s)^T{\omega}$

In continuous action space, greedy policy becomes problematic as it requires maximization every step. Instead, the better alternative is to move in the direction of the gradient. Specifically for each visited state 𝑠 the policy parametrs ${\theta}^{k+1}$ are updated in proportion to the gradient ${\nabla}_{\theta}Q^{\mu}(s,{\mu}_{\theta}(s)$

A deterministic policy gradient theorem that states that the actor is updated by applying the chain rule to do extend return from to the expected return from the start distribution $J$

We use Neural network as a function approximator and allows to learn in large state and action spaces online. The main challenge when using the neural network in reinforcement learning is that most optimization algorithms assume that the samples are independently and identically distributed. To address these issues the replay buffer, finitesized cache, is used

Also I suggest you to go through this lecture


Your Answer

