Improved knowledge distillation by utilizing backward pass knowledge in
  neural networks
Knowledge distillation (KD) is one of the prominent techniques for model
compression. In this method, the knowledge of a large network (teacher) is
distilled into a model (student) with usually significantly fewer parameters.
KD tries to better-match the output of the student model to that of the teacher
model based on the knowledge extracts from the forward pass of the teacher
network. Although conventional KD is effective for matching the two networks
over the given data points, there is no guarantee that these models would match
in other areas for which we do not have enough training samples. In this work,
we address that problem by generating new auxiliary training samples based on
extracting knowledge from the backward pass of the teacher in the areas where
the student diverges greatly from the teacher. We compute the difference
between the teacher and the student and generate new data samples that maximize
the divergence. This is done by perturbing data samples in the direction of the
gradient of the difference between the student and the teacher. Augmenting the
training set by adding this auxiliary improves the performance of KD
significantly and leads to a closer match between the student and the teacher.
Using this approach, when data samples come from a discrete domain, such as
applications of natural language processing (NLP) and language understanding,
is not trivial. However, we show how this technique can be used successfully in
such applications. We evaluated the performance of our method on various tasks
in computer vision and NLP domains and got promising results.
Aref Jafari, Mehdi Rezagholizadeh, Ali Ghodsi
# Improved knowledge distillation by utilizing backward pass knowledge in neural networks

###### Abstract

Knowledge distillation (KD) is one of the prominent techniques for model compression. In this method, the knowledge of a large network (teacher) is distilled into a model (student) with usually significantly fewer parameters. KD tries to better-match the output of the student model to that of the teacher model based on the knowledge extracts from the forward pass of the teacher network. Although conventional KD is effective for matching the two networks over the given data points, there is no guarantee that these models would match in other areas for which we do not have enough training samples. In this work, we address that problem by generating new auxiliary training samples based on extracting knowledge from the backward pass of the teacher in the areas where the student diverges greatly from the teacher. We compute the difference between the teacher and the student and generate new data samples that maximize the divergence. This is done by perturbing data samples in the direction of the gradient of the difference between the student and the teacher. Augmenting the training set by adding this auxiliary improves the performance of KD significantly and leads to a closer match between the student and the teacher. Using this approach, when data samples come from a discrete domain, such as applications of natural language processing (NLP) and language understanding, is not trivial. However, we show how this technique can be used successfully in such applications. We evaluated the performance of our method on various tasks in computer vision and NLP domains and got promising results.

## 1 Introduction

During the last few years, we faced the emerge of a huge number of cumbersome state-of-the-art deep neural network models in different fields of machine learning, including computer vision [27; 10], natural language processing [16; 12; 13; 3] and speech processing [1; 7]. We need powerful servers to be able to deploy such large models. Running such large models on edge devices would be infeasible due to the limited memory and computational power of edge devices [22]. On the other hand, considering users' privacy concerns, network reliability issues, and network delays increase the demand for offline machine learning solutions on edge devices. The field of neural model compression focuses on providing compression solutions such as quantization [11], pruning [26], tensor decomposition [24] and knowledge distillation (KD) [9] for large neural networks.