title: Reinforcement Learning-TD learning from afterstates
body: <p>I'm making a program that teaches 2 players to play a simple board game using Reinforcement Learning and the Temporal Difference learning method (TD(λ) ) based on afterstates. Learning occurs by training a neural network. I use <a href="http://webdocs.cs.ualberta.ca/~sutton/td-backprop-pseudo-code.text" rel="nofollow">Sutton's NonLinear TD/Backprop neural network</a> ) I would really like your opinion on my following dilemma .
  The basic algorithm/pseudo code for playing a turn between the two opponents is this</p>
  
  <pre><code>WHITE.CHOOSE_ACTION(GAME_STATE); //White player decides on its next move by evaluating the current game state ( TD(λ) learning)
  
  GAME_STATE = WORLD.APPLY(WHITE_PLAYERS_ACTION);  //We apply the chosen action of the player to the environment and a new game state emerges
  
   IF (GAME STATE != FINAL ){ // If the new state is not final (not a winning state for white player), do the same for the Black player
  
      BLACK.CHOOSE_ACTION(GAME_STATE)
  
  GAME_STATE = WORLD.APPLY(BLACK_PLAYERS_ACTION) // We apply the chosen action of the black player to the environment and a new game state emerges.
  }
  </code></pre>
  
  <p>When should each player invoke his learning method PLAYER.LEARN(GAME_STATE). Here is the dillema.</p>
  
  <p>OPTION A.
  Immediately after each player's move, after the new afterstate emerges, as follows:</p>
  
  <pre><code>WHITE.CHOOSE_ACTION(GAME_STATE);
  GAME_STATE = WORLD.APPLY(WHITE_PLAYERS_ACTION);
  WHITE.LEARN(GAME_STATE)    // White learns from the afterstate that emerged right after his action
  IF (GAME STATE != FINAL ){
      BLACK.CHOOSE_ACTION(GAME_STATE)
      GAME_STATE = WORLD.APPLY(BLACK_PLAYERS_ACTION)
      BLACK.LEARN(GAME_STATE) // Black learns from the afterstate that emerged right after his action
  </code></pre>
  
  <p>OPTION B.
  Immediately after each player's move, after the new afterstate emerges, but also after the opponents move, if the opponent makes a winning move.</p>
  
  <pre><code>WHITE.CHOOSE_ACTION(GAME_STATE);
  GAME_STATE = WORLD.APPLY(WHITE_PLAYERS_ACTION);
  WHITE.LEARN(GAME_STATE)
  IF (GAME_STATE == FINAL ) //If white player won
      BLACK.LEARN(GAME_STATE) // Make the Black player learn from the White player's winning afterstate
  IF (GAME STATE != FINAL ){ //If white player's move did not produce a winning/final afterstate
      BLACK.CHOOSE_ACTION(GAME_STATE)
      GAME_STATE = WORLD.APPLY(BLACK_PLAYERS_ACTION)
      BLACK.LEARN(GAME_STATE)
      IF (GAME_STATE == FINAL) //If Black player won
          WHITE.LEARN(GAME_STATE) //Make the White player learn from the Black player's winning afterstate
  </code></pre>
  
  <p>I believe that option B is more reasonable.</p>