<p>It might be related to the <strong>scale</strong> of your Q-Values. I have the same behavior in my DQN loss, my agent easily solves the environment but the loss is growing through training.</p>
<p>If you look at this part of the DQN algorithm you might get some insights:</p>
<p><a href="https://i.stack.imgur.com/hWktP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hWktP.png" alt="enter image description here" /></a></p>
<ul>
<li>First you will notice that the target <em>y</em> is built upon the <em>max</em> Q-values of the target network. It could induce a constant overestimation of the target Q-value as it is demonstrated in the <a href="https://arxiv.org/pdf/1509.06461.pdf" rel="nofollow noreferrer">Double-DQN paper</a>. Since the target could be constantly overestimated while the prediction is not, a delta will always exist between predictions and targets</li>
<li>Second, this delta will grow in scale as the Q-values grow too. I think it is a normal behavior since your Q function will learn that many states have an important value, so the error at the beginning of training might be way smaller than the error at the end</li>
<li>Third the target Q-network is frozen for some steps while the prediction Q-network changes constantly, that also contributes to this delta</li>
</ul>
<p>Hope this helps, note that it is a purely intuitive and personal explanation, I did not conduct any test to check my hypotheses. And I think that the second point might be the most important here.</p>
Why DQN for cartpole game has a ascending reward while loss is not descending?
<p>I wrote a DQN to play the OpenAI gym cart pole game with TensorFlow and tf_agents. The code looks like the following:</p>
<pre class="lang-py prettyprint-override"><code>def compute_avg_return(environment, policy, num_episodes=10):
    total_return = 0.0
    for _ in range(num_episodes):
        time_step = environment.reset()
        episode_return = 0.0
        while not time_step.is_last():
            action_step = policy.action(time_step)
            time_step = environment.step(action_step.action)
            episode_return += time_step.reward
        total_return += episode_return
    avg_return = total_return / num_episodes
    return avg_return.numpy()[0]


def collect_step(environment, policy, buffer):
    time_step = environment.current_time_step()
    action_step = policy.action(time_step)
    next_time_step = environment.step(action_step.action)
    traj = trajectory.from_transition(time_step, action_step, next_time_step)
    buffer.add_batch(traj)


def collect_data(env, policy, buffer, steps):
    for _ in range(steps):
        collect_step(env, policy, buffer)


def train_model(
    num_iterations=config.default_num_iterations,
    collect_steps_per_iteration=config.default_collect_steps_per_iteration,
    replay_buffer_max_length=config.default_replay_buffer_max_length,
    batch_size=config.default_batch_size,
    learning_rate=config.default_learning_rate,
    log_interval=config.default_log_interval,
    num_eval_episodes=config.default_num_eval_episodes,
    eval_interval=config.default_eval_interval,
    checkpoint_saver_directory=config.default_checkpoint_saver_directory,
    model_saver_directory=config.default_model_saver_directory,
    visualize=False,
    static_plot=False,
):
    env_name = 'CartPole-v0'
    train_py_env = suite_gym.load(env_name)
    eval_py_env = suite_gym.load(env_name)
    train_env = tf_py_environment.TFPyEnvironment(train_py_env)
    eval_env = tf_py_environment.TFPyEnvironment(eval_py_env)
    fc_layer_params = (100,)
    q_net = q_network.QNetwork(
        train_env.observation_spec(),
        train_env.action_spec(),
        fc_layer_params=fc_layer_params)
    optimizer = Adam(learning_rate=learning_rate)
    train_step_counter = tf.Variable(0)
    agent = dqn_agent.DqnAgent(
        train_env.time_step_spec(),
        train_env.action_spec(),
        q_network=q_net,
        optimizer=optimizer,
        td_errors_loss_fn=common.element_wise_squared_loss,
        train_step_counter=train_step_counter)
    agent.initialize()
    replay_buffer = tf_uniform_replay_buffer.TFUniformReplayBuffer(
        data_spec=agent.collect_data_spec,
        batch_size=train_env.batch_size,
        max_length=replay_buffer_max_length)
    dataset = replay_buffer.as_dataset(
        num_parallel_calls=3,
        sample_batch_size=batch_size,
        num_steps=2).prefetch(3)
    iterator = iter(dataset)
    agent.train_step_counter.assign(0)
    avg_return = compute_avg_return(eval_env, agent.policy, num_eval_episodes)
    returns = []
    loss = []
    for _ in range(num_iterations):
        for _ in range(collect_steps_per_iteration):
            collect_step(train_env, agent.collect_policy, replay_buffer)
        experience, unused_info = next(iterator)
        train_loss = agent.train(experience).loss
        step = agent.train_step_counter.numpy()
        avg_return = compute_avg_return(eval_env, agent.policy, num_eval_episodes)
        returns.append(avg_return)
</code></pre>
<p>Although the average reward is getting better and reached 200, the maximum score, in the end, the loss is not obviously decreasing.</p>
<p>Here is the loss plot:</p>
<p><a href="https://i.stack.imgur.com/AcIYE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/AcIYE.png" alt="loss plot" /></a></p>
<p>Here is the reward plot:</p>
<p><a href="https://i.stack.imgur.com/J01qu.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/J01qu.png" alt="reward plot" /></a></p>
<p>The good point is that the model is successful, and it can play the game really well. However, I would really love to get some insight into why this is happening where an extremely high loss still yields a good reward.</p>