title: Using learning rate schedule and learning rate warmup with TensorFlow2
body: <p>I have to use learning rate warmup where you start training a VGG-19 CNN for CIFAR-10 with warmup from a learning rate of 0.00001 to 0.1 over the first 10000 iterations (or, approximately 13 epochs) using learning rate warmup. And then for the remainder of training, you use the learning rate of 0.01 where a learning rate decay is used to reduce the learning rate by a factor of 10 at 80 and 120 epochs. The model has to be trained for a total of 144 epochs.</p>
  <p>I am using Python 3 and TensorFlow2 where training dataset has 50000 examples and batch size = 64. The number of training iterations in one epoch = 50000/64 = 781 iterations (approx.).
  How can I use both of the learning rate warmup and learning rate decay together in the code?</p>
  <p>Currently, I am using a learning rate decay by:</p>
  <pre><code>boundaries = [100000, 110000]
  values = [1.0, 0.5, 0.1]
  
  learning_rate_fn = keras.optimizers.schedules.PiecewiseConstantDecay(
      boundaries, values)
  print(&quot;\nCurrent step value: {0}, LR: {1:.6f}\n&quot;.format(optimizer.iterations.numpy(), optimizer.learning_rate(optimizer.iterations)))
  </code></pre>
  <p>However, I don't know how to use a learning rate warmup along with learning rate decay.</p>
  <p>Help?</p>