title: Keras : Building a train function within for loop creates memory issue
body: <p>I am trying to train a double hidden layer CNN with policy gradient. I have a custom loss function for this. I am trying to implement it in Keras (backend TF) using the code for Cartpole game by kkweon. The code is as follows,   </p>
  
  <pre><code>def _build_train_fn(network):
      action_prob_placeholder = network.output
      action_onehot_placeholder = K.placeholder(shape=(None, output_dim), 
                                                name = "action_onehot")
      discount_reward_placeholder = K.placeholder(shape=(None, ), 
                                                  name = "discount_reward")
  
      action_prob = K.sum(action_prob_placeholder * action_onehot_placeholder, 
                          axis=1)
      log_action_prob = K.log(action_prob)
  
      loss = -log_action_prob * discount_reward_placeholder
      loss = K.mean(loss)
  
      adam = optimizers.Adam()
  
      updates = adam.get_updates(loss = loss, params = network.trainable_weights)
  
      train_fn = K.function(inputs=[network.input, 
                                         action_onehot_placeholder, 
                                         discount_reward_placeholder], 
                                  outputs=[], updates=updates)
      return(train_fn)
  </code></pre>
  
  <p>Now I run the training function inside a for loop, where I take mini batches of episodes and train for that. So I need to build the train function for each mini batch, as it depends on the network output. The batch-wise training inside the for loop goes like this,</p>
  
  <pre><code>if episode_number % batch_size == 0:
              mini_batch_states = np.asarray(mini_batch_states).reshape([len(mini_batch_states), 14])
              mini_batch_moves = np.asarray(mini_batch_moves)
  
              train_fn = _build_train_fn(network)
              train_fn([mini_batch_states, mini_batch_moves, mini_batch_rewards])
  </code></pre>
  
  <p>The issue is that the memory usage continues to grow on size as the number of mini batches grow. I have tried building the function once before the loop, which solves the memory issue but removes the aspect of dynamic learning and converges to a 0.5 winrate instead of growing consistently.</p>
  
  <p>Also using a </p>
  
  <pre><code>del train_fn
  </code></pre>
  
  <p>does not solve the memory issue.</p>
  
  <p>I am very new to both TF and Keras. Any help is much appreciated.</p>