original_docstring: 
          Call at end of one episode
  
          Training code.
          Calculates actor and critic loss and performs backprop.
  
          For meaning of "advantage", "reward", "value",
          see "Back to Baselines" section of
          https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f
          
docstring: Call at end of one episode
  Training code.
  Calculates actor and critic loss and performs backprop.
  
  
docstring_tokens: Call at end of one episode Training code . Calculates actor and critic loss and performs backprop .
code: def update(self, optimizer):
          """
          Call at end of one episode
  
          Training code.
          Calculates actor and critic loss and performs backprop.
  
          For meaning of "advantage", "reward", "value",
          see "Back to Baselines" section of
          https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f
          """
  
          R = 0
          saved_actions = self.saved_actions
          policy_losses = []  # list to save actor (policy) loss
          value_losses = []  # list to save critic (value) loss
          returns = []  # list to save the true values
  
          # calculate the true value using episode_rewards returned from the environment
          for r in self.episode_rewards[::-1]:
              # calculate the discounted value
              R = r + self.gamma * R
              returns.insert(0, R)
  
          returns = torch.tensor(returns, dtype=torch.float32)
          returns = (returns - returns.mean()) / (returns.std() + self.eps)
  
          for (log_prob, value), R in zip(saved_actions, returns):
              advantage = R - value.item()
  
              # calculate actor (policy) loss
              policy_losses.append(-log_prob * advantage)
  
              # calculate critic (value) loss using L1 smooth loss
              value_losses.append(F.smooth_l1_loss(value, torch.tensor([R]).to(device=self._dev)))
  
          # reset gradients
          optimizer.zero_grad()
  
          # # sum up all the values of policy_losses and value_losses
          loss = torch.stack(policy_losses).sum() + torch.stack(value_losses).sum()
  
          # perform backprop
          loss.backward()
          optimizer.step()
  
          # reset episode_rewards and action buffer
          self.episode_rewards = []
          self.saved_actions = []
code_tokens: def update ( self , optimizer ) : R = 0 saved_actions = self . saved_actions policy_losses = [ ] value_losses = [ ] returns = [ ] for r in self . episode_rewards [ : : - 1 ] : R = r + self . gamma * R returns . insert ( 0 , R ) returns = torch . tensor ( returns , dtype = torch . float32 ) returns = ( returns - returns . mean ( ) ) / ( returns . std ( ) + self . eps ) for ( log_prob , value ) , R in zip ( saved_actions , returns ) : advantage = R - value . item ( ) policy_losses . append ( - log_prob * advantage ) value_losses . append ( F . smooth_l1_loss ( value , torch . tensor ( [ R ] ) . to ( device = self . _dev ) ) ) optimizer . zero_grad ( ) loss = torch . stack ( policy_losses ) . sum ( ) + torch . stack ( value_losses ) . sum ( ) loss . backward ( ) optimizer . step ( ) self . episode_rewards = [ ] self . saved_actions = [ ]
short_docstring: Call at end of one episode
  Training code.
short_docstring_tokens: Call at end of one episode Training code .
comment: """
          Call at end of one episode
  
          Training code.
          Calculates actor and critic loss and performs backprop.
  
          For meaning of "advantage", "reward", "value",
          see "Back to Baselines" section of
          https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f
          """$SEP$# list to save actor (policy) loss$SEP$# list to save critic (value) loss$SEP$# list to save the true values$SEP$# calculate the true value using episode_rewards returned from the environment$SEP$# calculate the discounted value$SEP$# calculate actor (policy) loss$SEP$# calculate critic (value) loss using L1 smooth loss$SEP$# reset gradients$SEP$# # sum up all the values of policy_losses and value_losses$SEP$# perform backprop$SEP$# reset episode_rewards and action buffer