\begin{table*}[h]
\centering
\small
\begin{tabular}{lp{7.2cm}ccc}
\toprule 
  & \textbf{Instance} & \textbf{EL2N} & \textbf{Gold Label} & \textbf{Prediction}\\
\midrule 
\multirow{15}{*}{\rotatebox{90}{Noise / Hard}}
& Tom Hauck/Getty Images 1. Rapper Snoop Dogg attended practice Tuesday with No. 1 USC as a guest wide receiver. The Trojans were delighted by the star \#39;s presence, as were a group of pee-wee football players watching practice that day.
& 1.413128
& Business (2)
& Sports (1)
\\\cmidrule[0.01em](lr){2-5}
& Palestinian gunmen kidnap CNN producer GAZA CITY, Gaza Strip -- Palestinian gunmen abducted a CNN producer in Gaza City on Monday, the network said. The network said Riyadh Ali was taken away at gunpoint from a CNN van.
& 1.412653
& Sports (1)
& World (0)
\\\cmidrule[0.01em](lr){2-5}
& Names in the Game Dressed in jeans and a white shirt, the men \#39;s all-around champion in Athens appeared relaxed as he helped promote a 14-city gymnastics exhibition tour that kicks off at the Mohegan Sun Casino on Tuesday.
& 1.412590
& Business (2)
& Sports (1)
\\\midrule[0.03em]
\multirow{14}{*}{\rotatebox{90}{Easy}}
& Vikings \#39; Moss practices despite hamstring injury Eden Prairie, MN (Sports Network) - Minnesota Vikings wide receiver Randy Moss practiced on Wednesday despite nursing a strained right hamstring and is listed as questionable for this Sunday \#39;s game against the New York Giants.
& 0.000665
& Sports (1)
& Sports (1)
\\\cmidrule[0.01em](lr){2-5}
& Cassell a no-show; Wolves sign Griffin Minneapolis, MN (Sports Network) - Minnesota Timberwolves guard Sam Cassell did not show up Tuesday for the first day of training camp.
& 0.000653
& Sports (1)
& Sports (1)
\\\cmidrule[0.01em](lr){2-5}
& Rockets activate Lue from injured list Houston, TX (Sports Network) - The Houston Rockets activated guard Tyronn Lue from the injured list prior to Wednesday \#39;s game against the Hawks.	
& 0.000649
& Sports (1)
& Sports (1)
 \\
\bottomrule
\end{tabular}
\caption{
Examples from AG News belonging to different score regions of EL2N. The highest scoring instances are mostly noisy samples, while the least scoring instances are very easy to learn.
} 
\label{tab:examples-agnews}
\end{table*}
\section{Introduction}
Large datasets have made the phenomenal performance of Transformer-based \citep{attention-is-all-you-need} pre-trained language models (PLMs) possible \citep{devlin-etal-2019-bert, sanh2019distilbert, liu-2019-roberta, clark2020electra, bommasani2021fm}. 
However, recent studies show that a significant fraction of examples in a training dataset can be omitted without sacrificing test accuracy. 
To this end, many metrics have been introduced for ranking the examples in a dataset based on their importance. 
One of these metrics is \emph{Forgetting Score} \citep{toneva2018empirical-forgetting, yaghoobzadeh-etal-2021-increasing} which recognizes the examples that are misclassified after being correctly classified during training or are always misclassifed. 
\emph{Datamap} \citep{swayamdipta-2020-dataset-cartography} is another technique for diagnosing datasets which uses \emph{confidence} and \emph{variability} metrics. 

\section{Background}
In this section, we describe the two metrics introduced by \citet{NEURIPS2021_ac56f8fe_datadiet} for pruning the training data: \textbf{GraNd} and its estimated variant, \textbf{EL2N}. 
\subsection{GraNd}
Consider $\mathbf{X} = \{x_i, y_i\}_{i=1}^N$ to be a training dataset for a given classification task with $K$ classes, where $x_i$ is the input (i.e. a sequence of tokens) and $y_i$ is its corresponding label. 
To estimate the importance of each training sample $(x_i, y_i)$,  \citet{NEURIPS2021_ac56f8fe_datadiet} propose utilizing the expected value of the loss gradient norm denoted as GraNd:

\subsection{EL2N}
By defining $\psi^{(k)}(x_i) = \nabla_{\boldsymbol{w}}f^{(k)}(x_i)$ for the $k^\text{th}$ logit,
the loss gradient ($\boldsymbol{g}$) can be written as:

Since the $\mathcal{L}(f(x_i), y_i)$ is a cross entropy loss, $\nabla_{f^{(k)}}\mathcal{L}(f(x_i), y_i)^T=p(x_i)^{(k)} - y^{(k)}_i$ , where $p(x_i)$ is the output probability vector of the model. By assuming orthogonality and uniform sizes across logits for $\psi$ over the training examples, the norm in Eq. \ref{eq:grand} is approximately equal to $\left\|p(x_i) - \boldsymbol{y}_i\right\|_2$ ($\boldsymbol{y}_i$ is the one-hot vector of the true label). As a result, an estimate of GraND is \textbf{EL2N}, which is defined as follows:

\begin{equation}
\label{eq:el2n}
    \text{EL2N}(x_i, y_i) = \mathbb{E}_{\boldsymbol{w}}\left\|p(x_i) - \boldsymbol{y}_i\right\|_2
\end{equation}

\section{Experiments}
In this section, we verify the effectiveness of GraNd and EL2N \citep{NEURIPS2021_ac56f8fe_datadiet} in the NLP domain. 
Our experimental setup is similar to \citet{NEURIPS2021_ac56f8fe_datadiet} for the followings:
(1) models are trained on the subsets of the data with higher GraNd and EL2N scores;
(2) based on the expectation over multiple weight initializations mentioned in Eq.~\ref{eq:grand} and Eq.~\ref{eq:el2n}, we average the scores over five independent training runs.\footnote{It is ten independent training runs in \citet{NEURIPS2021_ac56f8fe_datadiet}}; and,
(3) for putting our results in context, we utilized random pruning as the baseline\footnote{Random selection with the same proportion} and the accuracy of training on the whole dataset with no pruning as the target performance.

Two major differences between these two setups are mentioned here:
(1) we used a pre-trained model, i.e., BERT \citep{devlin-etal-2019-bert}, standard in the NLP domain, whereas \citet{NEURIPS2021_ac56f8fe_datadiet} uses vision models initialized with random weights; and
(2) as fine-tuning requires few epochs of training, we computed the metrics over fine-grained steps rather than epochs.

\section{Appendix}
\subsection{Dataset samples and their scores}
Examples with the highest and lowest EL2N scores are provided in Table~\ref{tab:examples-agnews} and Table~\ref{tab:examples-mnli} for AG News and MNLI datasets. 
{"value": "1.413128", "type": "Other"} {"value": "1.412653", "type": "Other"} {"value": "1.412590", "type": "Other"} {"value": "0.000665", "type": "Other"} {"value": "0.000653", "type": "Other"} {"value": "0.000649", "type": "Other"}