```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

import gzip
import numpy as np
import tensorflow as tf
from typing import Tuple
```

You've already seen how to train a neural network using Keras in [module 1](../intro-machine-learning-keras) &mdash; in this notebook, we'll re-implement the training loop in TensorFlow. This will help you understand what goes on under the hood a bit better, will give you the opportunity to customize the training loop if you want, and will enable you to debug it.

We'll start by including code that gives us the datasets and model that we'll use in the remainder of this notebook. We will use the same FashionMNIST dataset and data loading code as in [module 1](../intro-machine-learning-keras), so feel free to re-visit that module if something is not clear, or take a look [at the source code](https://github.com/MicrosoftDocs/tensorflow-learning-path/blob/main/intro-tf/tintro.py).


```python
!wget -Nq https://raw.githubusercontent.com/MicrosoftDocs/tensorflow-learning-path/main/intro-tf/tintro.py
from tintro import *
```

As we mentioned in module 1, the goal of training the neural network is to find parameters $W$ and $b$ that minimize the loss function, which measures the difference between the actual and predicted labels. We also mentioned that we can think of the neural network as the function $\ell$ below, and that we use an optimization algorithm to find the parameters $W$ and $b$ that minimize this function.

$$
\mathrm{loss} = \ell(X, y, W, b)
$$

Let's now dig deeper into what this optimization algorithm might look like. There are many types of optimization algorithms, but in this tutorial we'll cover only the simplest one: the gradient descent algorithm. To implement gradient descent, we iteratively improve our estimates of $W$ and $b$ according to the update formulas below, until the gradients are smaller than a pre-defined threshold $\epsilon$ (or for a pre-defined number of times):

$$
\begin{align}
  W &:= W - \alpha \frac{\partial \ell}{\partial W} \\
  b &:= b - \alpha \frac{\partial \ell}{\partial b}
\end{align}
$$

The parameter $\alpha$ is typically referred to as the "learning rate," and will be defined later in the code. 

When doing training, we pass a mini-batch of data as input, perform a sequence of calculations to obtain the loss, then propagate back through the network to calculate the derivatives used in the gradient descent formulas above. Once we have the derivatives, we can update the values of the network's parameters $W$ and $b$ according to the formulas. This sequence of steps is the backpropagation algorithm. By performing these calculations several times, our parameters get updated repeatedly, getting more and more accurate each time. 

In Keras, when we called the function [`fit`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit), the backpropagation algorithm was executed several times. Here, we'll start by understanding the code that reflects a single pass of the backpropagation algorithm:

- A forward pass through the model to compute the predicted value, `y_prime = model(X, training=True)`
- A calculation of the loss using a loss function, `loss = loss_fn(y, y_prime)`
- A backward pass from the loss function through the model to calculate derivatives, `grads = tape.gradient(loss, model.trainable_variables)`
- A gradient descent step to update $W$ and $b$ using the derivatives calculated in the backward pass, `optimizer.apply_gradients(zip(grads, model.trainable_variables))`

Here's the complete code:


```python
def fit_one_batch(X, y, model, loss_fn, optimizer) -> Tuple[tf.Tensor, tf.Tensor]:
  with tf.GradientTape() as tape:
    y_prime = model(X, training=True)
    loss = loss_fn(y, y_prime)

  grads = tape.gradient(loss, model.trainable_variables)
  optimizer.apply_gradients(zip(grads, model.trainable_variables))

  return (y_prime, loss)
```

Notice that the code above ensures that the forward calculations are within the `GradientTape`'s scope, just as we saw in the previous notebook. This makes it possible for us to ask the tape for the gradients. 

The code above works for a single mini-batch, which is typically much smaller than the full set of data (in this sample we use a mini-batch of size 64, out of 60,000 training data items). But we want to execute the backpropagation algorithm for the full set of data. We can do so by iterating through the `Dataset` we created earlier, which, as we saw in module 1, returns a mini-batch per iteration. There are two critical lines in the code below: the `for` loop and the call to the `fit_one_batch` function. The rest of the code just prints the accuracy and loss as the model is being trained. 


```python
def fit(dataset: tf.data.Dataset, model: tf.keras.Model, loss_fn: tf.keras.losses.Loss, 
optimizer: tf.optimizers.Optimizer) -> None:
  batch_count = len(dataset)
  loss_sum = 0
  correct_item_count = 0
  current_item_count = 0
  print_every = 100

  for batch_index, (X, y) in enumerate(dataset):
    (y_prime, loss) = fit_one_batch(X, y, model, loss_fn, optimizer)

    y = tf.cast(y, tf.int64)
    correct_item_count += (tf.math.argmax(y_prime, axis=1) == y).numpy().sum()

    batch_loss = loss.numpy()
    loss_sum += batch_loss
    current_item_count += len(X)

    if ((batch_index + 1) % print_every == 0) or ((batch_index + 1) == batch_count):
      batch_accuracy = correct_item_count / current_item_count * 100
      print(f'[Batch {batch_index + 1:>3d} - {current_item_count:>5d} items] accuracy: {batch_accuracy:>0.1f}%, loss: {batch_loss:>7f}')
```

A complete iteration over all mini-batches in the dataset is called an "epoch." In this sample, we restrict the code to just five epochs for quick execution, but in a real project you would want to set it to a much higher number (to achieve better predictions). The code below also shows the creation of the loss function and optimizer, which we discussed in module 1.


```python
learning_rate = 0.1
batch_size = 64
epochs = 5

(train_dataset, test_dataset) = get_data(batch_size)

model = NeuralNetwork()

loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
optimizer = tf.optimizers.SGD(learning_rate)

print('\nFitting:')
for epoch in range(epochs):
  print(f'\nEpoch {epoch + 1}\n-------------------------------')
  fit(train_dataset, model, loss_fn, optimizer)
```

    
    Fitting:
    
    Epoch 1
    -------------------------------
    [Batch 100 -  6400 items] accuracy: 62.9%, loss: 0.705272
    [Batch 200 - 12800 items] accuracy: 68.9%, loss: 0.743024
    [Batch 300 - 19200 items] accuracy: 71.5%, loss: 0.531616
    [Batch 400 - 25600 items] accuracy: 73.6%, loss: 0.617099
    [Batch 500 - 32000 items] accuracy: 74.7%, loss: 0.561903
    [Batch 600 - 38368 items] accuracy: 75.7%, loss: 0.925934
    [Batch 700 - 44768 items] accuracy: 76.6%, loss: 0.632339
    [Batch 800 - 51168 items] accuracy: 77.3%, loss: 0.370619
    [Batch 900 - 57568 items] accuracy: 77.9%, loss: 0.568127
    [Batch 938 - 60000 items] accuracy: 78.1%, loss: 0.627154
    
    Epoch 2
    -------------------------------
    [Batch 100 -  6400 items] accuracy: 83.6%, loss: 0.487360
    [Batch 200 - 12800 items] accuracy: 83.4%, loss: 0.428791
    [Batch 300 - 19200 items] accuracy: 83.3%, loss: 0.486824
    [Batch 400 - 25600 items] accuracy: 83.4%, loss: 0.391912
    [Batch 500 - 32000 items] accuracy: 83.5%, loss: 0.424991
    [Batch 600 - 38368 items] accuracy: 83.6%, loss: 0.515074
    [Batch 700 - 44768 items] accuracy: 83.6%, loss: 0.485272
    [Batch 800 - 51168 items] accuracy: 83.6%, loss: 0.345185
    [Batch 900 - 57568 items] accuracy: 83.7%, loss: 0.466114
    [Batch 938 - 60000 items] accuracy: 83.8%, loss: 0.271606
    
    Epoch 3
    -------------------------------
    [Batch 100 -  6400 items] accuracy: 85.3%, loss: 0.462108
    [Batch 200 - 12800 items] accuracy: 85.0%, loss: 0.370773
    [Batch 300 - 19200 items] accuracy: 85.2%, loss: 0.467257
    [Batch 400 - 25600 items] accuracy: 85.1%, loss: 0.564032
    [Batch 500 - 32000 items] accuracy: 85.1%, loss: 0.419473
    [Batch 600 - 38400 items] accuracy: 85.1%, loss: 0.444543
    [Batch 700 - 44768 items] accuracy: 85.1%, loss: 0.477494
    [Batch 800 - 51168 items] accuracy: 85.2%, loss: 0.428996
    [Batch 900 - 57568 items] accuracy: 85.2%, loss: 0.377139
    [Batch 938 - 60000 items] accuracy: 85.2%, loss: 0.331598
    
    Epoch 4
    -------------------------------
    [Batch 100 -  6400 items] accuracy: 85.8%, loss: 0.400852
    [Batch 200 - 12800 items] accuracy: 85.5%, loss: 0.399611
    [Batch 300 - 19200 items] accuracy: 85.6%, loss: 0.363247
    [Batch 400 - 25600 items] accuracy: 85.7%, loss: 0.337134
    [Batch 500 - 32000 items] accuracy: 85.8%, loss: 0.366630
    [Batch 600 - 38368 items] accuracy: 85.7%, loss: 0.488763
    [Batch 700 - 44768 items] accuracy: 85.7%, loss: 0.640464
    [Batch 800 - 51168 items] accuracy: 85.8%, loss: 0.422981
    [Batch 900 - 57568 items] accuracy: 85.9%, loss: 0.242606
    [Batch 938 - 60000 items] accuracy: 85.9%, loss: 0.532112
    
    Epoch 5
    -------------------------------
    [Batch 100 -  6400 items] accuracy: 86.5%, loss: 0.347542
    [Batch 200 - 12800 items] accuracy: 86.4%, loss: 0.427709
    [Batch 300 - 19200 items] accuracy: 86.5%, loss: 0.335535
    [Batch 400 - 25600 items] accuracy: 86.5%, loss: 0.185783
    [Batch 500 - 32000 items] accuracy: 86.6%, loss: 0.307705
    [Batch 600 - 38400 items] accuracy: 86.6%, loss: 0.274786
    [Batch 700 - 44800 items] accuracy: 86.6%, loss: 0.315632
    [Batch 800 - 51168 items] accuracy: 86.6%, loss: 0.267578
    [Batch 900 - 57568 items] accuracy: 86.6%, loss: 0.252907
    [Batch 938 - 60000 items] accuracy: 86.6%, loss: 0.485074


c80a837f3f25c69d7b290be3b204a73dec68142e
ipynb
Jupyter Notebook
intro-tf/5-train.ipynb
MicrosoftDocs/tensorflowfundamentals
ccc0fa119dce1122529c2400d395542fbd7cf3b6
CC-BY-4.0 MIT
intro-tf/5-train.ipynb
MicrosoftDocs/tensorflowfundamentals
ccc0fa119dce1122529c2400d395542fbd7cf3b6
CC-BY-4.0 MIT
intro-tf/5-train.ipynb
MicrosoftDocs/tensorflowfundamentals
ccc0fa119dce1122529c2400d395542fbd7cf3b6
CC-BY-4.0 MIT
2021-11-02T13:08:31.000Z
2022-03-14T00:51:45.000Z
Qwen/Qwen-72B
1. YES
2. YES