<p>Note: something that I think will get you on the right track:</p>

<p>The negative log likelihood is also know as the multiclass cross-entropy (Pattern Recognition and Machine Learning).</p>

<p>EDIT: misread the question. I thought this was talking about Deep Deterministic Policy Gradients</p>

<p>It would depend on your domain, but with a softmax, you are getting a probability across all output nodes. To me that doesn't really make sense in most domains when you think about DDPG. For example, if you are controlling the extension of robotic arms and legs, it wouldn't make sense to have limb extension measured as [.25, .25, .25, .25], if you wanted to have all limbs extended. In this case, .25 could mean fully extended, but what happens if the vector of outputs is [.75,.25,0,0]? So in this way, you could have a separate sigmoid function from 0 to 1 for all action nodes, where then you could represent it as [1,1,1,1] for all arms being extended. I hope that makes sense.</p>

<p>Since the actor network is what determines the actions in DDPG, we could then represent our network like this for our robot (rough keras example):</p>

<pre><code>state = Input(shape=[your_state_shape])
hidden_layer = Dense(30, activation='relu')(state)
all_limbs = Dense(4, activation='sigmoid')(hidden_layer)
model = Model(input=state, output=all_limbs)
</code></pre>

<p>Then, your critic network will have to account for the action dimensions.</p>

<pre><code>state = Input(shape=[your_state_shape])
action = Input(shape=[4])
state_hidden = Dense(30, activation='relu')(state)
state_hidden_2 = Dense(30, activation='linear')(state_hidden)
action_hidden = Dense(30, activation='linear')(action)
combined = merge([state_hidden_2, action_hidden], mode='sum')
squasher = Dense(30, activation='relu')(combined)
output = Dense(4, activation='linear')(squasher) #number of actions
</code></pre>

<p>Then you can use your target functions from there. Note, I don't know if this working code, as I haven't tested it, but hopefully you get the idea.</p>

<p>Source: <a href="https://arxiv.org/pdf/1509.02971.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1509.02971.pdf</a>
Awesome blog on this with Torc (not created by me): <a href="https://yanpanlau.github.io/2016/10/11/Torcs-Keras.html" rel="nofollow noreferrer">https://yanpanlau.github.io/2016/10/11/Torcs-Keras.html</a></p>

<p>In the above blog, they also show how to use different output functions, such as one TAHN, and two sigmoid functions for actions.</p>
What is the policy gradient when multiple actions are possible?
<p>I am trying to program a reinforcement learning algorithm using policy gradients, as inspired by <a href="http://karpathy.github.io/2016/05/31/rl/" rel="nofollow noreferrer">Karpathy's blog article</a>. Karpathy's example has only two actions UP or DOWN, so a single output neuron is sufficient (high activation=UP, low activation=DOWN). I want to extend this to multiple actions, so I believe I need a softmax activation function on the output layer. However, I am not certain about what the gradient for the output layer should be. </p>

<p>If I was using a cross-entropy loss function with the softmax activation in a supervised learning context, the gradient for neuron is simply:</p>

<pre><code>g[i] = a[i] - target[i]
</code></pre>

<p>where <code>target[i] = 1</code> for the desired action and <code>0</code> for all others.</p>

<p>To use this for reinforcement learning I would multiply <code>g[i]</code> by the discounted reward before back-propagating.</p>

<p>However, it seems that reinforcement learning uses negative log-likelihood as the loss instead of cross-entropy. <strong>How does that change the gradient?</strong></p>