<p>I think implementing for a discrete action space such as Cartpole-v1 is easier than for continuous action spaces.  But for continuous action spaces, this is the most straight-forward implementation I found in Pytorch as you can clearly see how they get <code>mu</code> and <code>std</code> where as I could not with more renowned implementations such as Openai Baselines and Spinning up or Stable Baselines.  </p>

<p><a href="https://github.com/higgsfield/RL-Adventure-2/blob/master/3.ppo.ipynb" rel="nofollow noreferrer">RL-Adventure PPO</a></p>

<p>These lines from the link above:</p>

<pre><code>class ActorCritic(nn.Module):
    def __init__(self, num_inputs, num_outputs, hidden_size, std=0.0):
        super(ActorCritic, self).__init__()

        self.critic = nn.Sequential(
            nn.Linear(num_inputs, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, 1)
        )

        self.actor = nn.Sequential(
            nn.Linear(num_inputs, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, num_outputs),
        )
        self.log_std = nn.Parameter(torch.ones(1, num_outputs) * std)

        self.apply(init_weights)

    def forward(self, x):
        value = self.critic(x)
        mu    = self.actor(x)
        std   = self.log_std.exp().expand_as(mu)
        dist  = Normal(mu, std)
        return dist, value
</code></pre>

<p>and the clipping:</p>

<pre><code>def ppo_update(ppo_epochs, mini_batch_size, states, actions, log_probs, returns, advantages, clip_param=0.2):
    for _ in range(ppo_epochs):
        for state, action, old_log_probs, return_, advantage in ppo_iter(mini_batch_size, states, actions, log_probs, returns, advantages):
            dist, value = model(state)
            entropy = dist.entropy().mean()
            new_log_probs = dist.log_prob(action)

            ratio = (new_log_probs - old_log_probs).exp()
            surr1 = ratio * advantage
            surr2 = torch.clamp(ratio, 1.0 - clip_param, 1.0 + clip_param) * advantage
</code></pre>

<p>I found the link above the comments on this video on Youtube:</p>

<p><a href="https://www.youtube.com/watch?v=5P7I-xPq8u8" rel="nofollow noreferrer">arxiv insights PPO</a></p>
What is the way to understand Proximal Policy Optimization Algorithm in RL?
<p>I know the basics of Reinforcement Learning, but what terms it's necessary to understand to be able read <a href="https://arxiv.org/abs/1707.06347" rel="noreferrer">arxiv PPO paper</a> ?</p>

<p>What is the roadmap to learn and use <a href="https://blog.openai.com/openai-baselines-ppo/" rel="noreferrer">PPO</a> ?</p>