<p>I think the problem is that you zero the gradients right before calling backward, after the forward propagation. Note that for <a href="https://en.wikipedia.org/wiki/Automatic_differentiation" rel="nofollow noreferrer">automatic differentiation</a> you need the computation graph and the intermediate results that you produce during your forward pass.</p>

<p>So zero the gradients <strong>before</strong> your TD error and target calculations! And not after you are finished your forward propagation.</p>

<pre><code>    for cur_step in range(1):   
    action = M_Agent(state, flag)  
    next_state, r = env.step(action)   

    optimizer_M.zero_grad()  # zero your gradient here

    # calculate TD Error
    TD_error = M_Agent.cal_td_error(r, next_state)

    # calculate Target
    target = torch.FloatTensor([M_Agent.cal_target(TD_error)])
    logit = M_Agent.cal_logit()
    loss = criterion(logit, target)

    # update value Func
    TD_error.backward()
    optimizer_M.step()

    # update Actor Func
    loss.backward()
    optimizer_M.step()
</code></pre>

<p>To answer your second question, the DDPG algorithm for example uses the squared error (see the <a href="https://arxiv.org/pdf/1509.02971v2.pdf" rel="nofollow noreferrer">paper</a>).</p>

<p>Another recommendation. In many cases large parts of the value and policy networks are shared in deep actor-critic agents: you have the same layers up to the last hidden layer, and use a single linear output for value prediction and a softmax layer for the action distribution. This is especially useful if you have high dimensional visual inputs, as it act as sort of a multi-task learning, but nevertheless you can try. (As I see you have a low-dimensional state vector).</p>
Adam optimizer error: one of the variables needed for gradient computation has been modified by an inplace operation
<p>I am trying to implement <strong>Actor-Critic learning atuomation algorithm</strong> that is not same as basic actor-critic algorithm, it's little bit changed.</p>

<p>Anyway, I used Adam optimizer and implemented with pytorch</p>

<p>when i backward TD-error for Critic first, there's no error.
However, i backward loss for Actor, the error occured.</p>

<blockquote>
  <p>--------------------------------------------------------------------------- RuntimeError                              Traceback (most recent call
  last)  in 
       46             # update Actor Func
       47             optimizer_M.zero_grad()
  ---> 48             loss.backward()
       49             optimizer_M.step()
       50 </p>
  
  <p>~\Anaconda3\lib\site-packages\torch\tensor.py in backward(self,
  gradient, retain_graph, create_graph)
      100                 products. Defaults to <code>False</code>.
      101         """
  --> 102         torch.autograd.backward(self, gradient, retain_graph, create_graph)
      103 
      104     def register_hook(self, hook):</p>
  
  <p>~\Anaconda3\lib\site-packages\torch\autograd__init__.py in
  backward(tensors, grad_tensors, retain_graph, create_graph,
  grad_variables)
       88     Variable._execution_engine.run_backward(
       89         tensors, grad_tensors, retain_graph, create_graph,
  ---> 90         allow_unreachable=True)  # allow_unreachable flag
       91 
       92 </p>
  
  <p>RuntimeError: one of the variables needed for gradient computation has
  been modified by an inplace operation</p>
</blockquote>

<p>above is the content of error</p>

<p>I tried to find inplace operation, but I haven't found in my written code.
I think i don't know how to handle optimizer.</p>

<p>Here is main code:</p>

<pre><code>        for cur_step in range(1):   
        action = M_Agent(state, flag)  
        next_state, r = env.step(action)   

        # calculate TD Error
        TD_error = M_Agent.cal_td_error(r, next_state)

        # calculate Target
        target = torch.FloatTensor([M_Agent.cal_target(TD_error)])
        logit = M_Agent.cal_logit()
        loss = criterion(logit, target)

        # update value Func
        optimizer_M.zero_grad()
        TD_error.backward()
        optimizer_M.step()

        # update Actor Func
        loss.backward()
        optimizer_M.step()
</code></pre>

<p>Here is the agent network</p>

<pre><code>    # Actor-Critic Agent
    self.act_pipe = nn.Sequential(nn.Linear(state, 128),
                            nn.ReLU(),
                            nn.Dropout(0.5),
                            nn.Linear(128, 256),
                            nn.ReLU(),
                            nn.Dropout(0.5),
                            nn.Linear(256, num_action),
                            nn.Softmax()
                            )

     self.val_pipe = nn.Sequential(nn.Linear(state, 128),
                            nn.ReLU(),
                            nn.Dropout(0.5),
                            nn.Linear(128, 256),
                            nn.ReLU(),
                            nn.Dropout(0.5),
                            nn.Linear(256, 1)
                            )


      def forward(self, state, flag, test=None):

          temp_action_prob = self.act_pipe(state)
          self.action_prob = self.cal_prob(temp_action_prob, flag)
          self.action = self.get_action(self.action_prob)
          self.value = self.val_pipe(state)

          return self.action
</code></pre>

<p>I wanna update each network respectively.</p>

<p>and I wanna know that Basic <strong>TD Actor-Critic</strong> method uses TD error for loss??
or squared error between r+V(s') and V(s) ?</p>