title: Implementing Q Learning For OpenAi CartPole in Python (v0)
body: <p>--------------------------------------------------------------UPDATE---------------------------------------------------------
  I continued trying to improve the code by altering the parameters and the way it updates through the learning rule. I suppose the performance is a but better now, but still far worse than expected. I wish to continue trying to solve it through Q learning since this website cites an example of the cartpole problem being solved with Q learning: <a href="https://medium.com/@tuzzer/cart-pole-balancing-with-q-learning-b54c6068d947" rel="nofollow noreferrer">https://medium.com/@tuzzer/cart-pole-balancing-with-q-learning-b54c6068d947</a></p>
  
  <p>My new code:</p>
  
  <pre><code>import gym
  import numpy as np
  import math
  
  statesAndQ = {}
  
  bucketNumbers = [1,1,6,3]
  
  env = gym.make('CartPole-v0')
  env.reset()
  
  bucketBounds = list(zip(env.observation_space.low, env.observation_space.high))
  bucketBounds[1]=[-0.5,0.5]
  bucketBounds[3] = [-math.radians(50), math.radians(50)]
  
  def learningRule(statesAndQ, reward, discountFactor, observationTuple, tempStates, i_episode):
      if len(tempStates)&gt;2:
          #print(tempStates)
          maximumIndex = np.argmax(statesAndQ[tuple(tempStates[-2])[0]])
      else:
          maximumIndex = np.argmax(statesAndQ[tuple(tempStates[-2])[0]])
      best_q = np.amax(statesAndQ[observationTuple])
      if i_episode==0:
          statesAndQ[observationTuple][maximumIndex] += get_learning_rate(i_episode)*(reward + discountFactor*best_q - statesAndQ[observationTuple][maximumIndex])
      else:
          statesAndQ[observationTuple][maximumIndex] += get_learning_rate(i_episode-1)*(reward + discountFactor*best_q - statesAndQ[observationTuple][maximumIndex])
  
      maximum = statesAndQ[observationTuple][maximumIndex]
  
      i = len(tempStates)-1
  
      while i&gt;0:
          statesAndQ[tempStates[i][0]][np.argmax(statesAndQ[tempStates[i][0]])]  += get_learning_rate(i_episode)*(maximum - statesAndQ[tempStates[i][0]][np.argmax(statesAndQ[tempStates[i][0]])])
          maximum = statesAndQ[tempStates[i][0]][np.argmax(statesAndQ[tempStates[i][0]])]
          i=i-1
  
      return statesAndQ
  
  def makeBuckets(state):
      bucket_indice = []
      for i in range(len(state)):
          if state[i] &lt;= bucketBounds[i][0]:
              bucket_index = 0
          elif state[i] &gt;= bucketBounds[i][1]:
              bucket_index = bucketNumbers[i] - 1
          else:
              bound_width = bucketBounds[i][1] - bucketBounds[i][0]
              offset = (bucketNumbers[i]-1)*bucketBounds[i][0]/bound_width
              scaling = (bucketNumbers[i]-1)/bound_width
              bucket_index = int(round(scaling*state[i] - offset))
          bucket_indice.append(bucket_index)
      return tuple(bucket_indice)
  
  def get_explore_rate(t):
      #return max(0.01, min(1, 1.0 - math.log10((t+1)/15)))
      if t&gt;250:
          return 0
      else:
          return max(min(1, 1.0 - math.log10((t+1)/25)), 0.01)
  
  
  def get_learning_rate(t):
      return max(0.1, min(0.7, 1.0 - math.log10((t+1)/30)))
  
  for i_episode in range(1000):
      print(get_explore_rate(i_episode))
      print("learning")
      print(get_learning_rate(i_episode))
      observation = env.reset()
      tempStates = []
      length = 0
  
      for i in range(250):
          env.render()
          random = np.random.normal(0,1)
  
          observationTuple = makeBuckets(observation)
          tempStates.append([observationTuple])
  
  
          if not observationTuple in statesAndQ:
              statesAndQ[observationTuple] = [0,0]
  
          if random&gt;get_explore_rate(i_episode):
              action = np.argmax(statesAndQ[observationTuple])
              observation, reward, done, info = env.step(action)
              observationTuple = makeBuckets(observation)
              tempStates.append([observationTuple])
  
              length+=1
  
              if not observationTuple in statesAndQ:
                  statesAndQ[observationTuple] = [0,0]
  
              #print(tempStates)
  
              statesAndQ = learningRule(statesAndQ, reward, 0.9, observationTuple, tempStates, i_episode)
              length = 0
              tempStates = []
  
          else:
              action = env.action_space.sample()
              observation, reward, done, info = env.step(action)
              observationTuple = makeBuckets(observation)
              tempStates.append([observationTuple])
  
              if not observationTuple in statesAndQ:
                  statesAndQ[observationTuple] = [0,0]
  
              statesAndQ = learningRule(statesAndQ, reward, 0.9, observationTuple, tempStates, i_episode)
              length = 0
              tempStates = []
  
          if done:
              print("Episode finished after {} timesteps".format(i+1))
              print(i_episode)
              break
  
  print("rollouts finished")
  </code></pre>
  
  <p>------------------------------------------------------ORIGINAL POST------------------------------------------------------
  I am a beginner in reinforcement learning and am just starting to learn about it. I am attempting to implement Q learning to solve the CartPole problem from the OpenAi gym. I do not seem to be getting good results, and my program does not seem to be improving its play. How could I work to improve it?</p>
  
  <p>The code:</p>
  
  <pre><code>import gym
  import numpy as np
  import math
  
  statesAndQ = {}
  
  bucketNumbers = [1,1,6,3] 
  
  env = gym.make('CartPole-v0')
  env.reset()
  
  bucketBounds = list(zip(env.observation_space.low, env.observation_space.high))
  bucketBounds[1]=[-0.5,0.5]
  bucketBounds[3] = [-math.radians(50), math.radians(50)]
  
  def learningRule(statesAndQ, reward, discountFactor, observationTuple, tempStates, i_episode):
      maximum = np.argmax(statesAndQ[observationTuple])
      statesAndQ[observationTuple][maximum] += get_learning_rate(i_episode)*(reward)
      maximum = statesAndQ[observationTuple][maximum]
  
      i = len(tempStates) - 2
  
      while i&gt;0:
          statesAndQ[tempStates[i][0]][tempStates[i][1]]  += get_learning_rate(i_episode)*(discountFactor*maximum)
          maximum = statesAndQ[tempStates[i][0]][tempStates[i][1]]
          i=i-1
  
      return statesAndQ
  
  def makeBuckets(state):
      bucket_indice = []
      for i in range(len(state)):
          if state[i] &lt;= bucketBounds[i][0]:
              bucket_index = 0
          elif state[i] &gt;= bucketBounds[i][1]:
              bucket_index = bucketNumbers[i] - 1
          else:
              bound_width = bucketBounds[i][1] - bucketBounds[i][0]
              offset = (bucketNumbers[i]-1)*bucketBounds[i][0]/bound_width
              scaling = (bucketNumbers[i]-1)/bound_width
              bucket_index = int(round(scaling*state[i] - offset))
          bucket_indice.append(bucket_index)
      return tuple(bucket_indice)
  
  def get_explore_rate(t):
      return max(0.01, min(1, 1.0 - math.log10((t+1)/25)))
  
  def get_learning_rate(t):
      return max(0.1, min(0.5, 1.0 - math.log10((t+1)/25)))
  
  for i_episode in range(1000):
      observation = env.reset()
      tempStates = []
  
      prevState = makeBuckets(observation)
  
      for i in range(200):
          env.render()
          random = np.random.normal(0,1)
  
          observationTuple = makeBuckets(observation)
  
          if not observationTuple in statesAndQ:
              statesAndQ[observationTuple] = [0,0]
  
          if random&gt;get_explore_rate(i_episode):
              action = np.argmax(statesAndQ[observationTuple])
              observation, reward, done, info = env.step(action)
              observationTuple = makeBuckets(observation)                   
  
              tempStates.append([observationTuple, action])
  
              if not observationTuple in statesAndQ:
                  statesAndQ[observationTuple] = [0,0]
  
              statesAndQ = learningRule(statesAndQ, reward, 0.9, observationTuple, tempStates, i_episode)
  
          else:
              action = env.action_space.sample()
              observation, reward, done, info = env.step(action)
              observationTuple = makeBuckets(observation)                   
              tempStates.append([observationTuple, action])
              if not observationTuple in statesAndQ:
                  statesAndQ[observationTuple] = [0,0]
  
              statesAndQ = learningRule(statesAndQ, reward, 0.9, observationTuple, tempStates, i_episode)
  
          if done:
              print("Episode finished after {} timesteps".format(i+1))
              print(i_episode)
              break
  
  print("rollouts finished")
  </code></pre>