\section{Introduction}
Neural Networks have become popular choices for complex computation tasks like image recognition \cite{Howard2017MobileNetsEC}, image generation \cite{Goodfellow2014GenerativeAN}, speech processing \cite{Zhao2017RecurrentCN}, and question answering \cite{Seo2017BidirectionalAF} and ranking \cite{nogueira2019multi}. These networks have grown to hundred billions of parameters \cite{Brown2020LanguageMA} which require specialized hardware like clusters of GPUs and FPGAs in order to infer on unseen data.  In the world of Natural Language Processing (NLP) and information retrieval (IR), neural language model's adaptability and performance has made them popular model despite their size. In order to ensure advances in research have practical applications many researcher have explored how large neural models can be compressed. Methods  Recent work has shown that larger networks can learn quicker \cite{Li2020TrainLT}, are more accurate and are more sample efficient \cite{Kaplan2020ScalingLF} which will likely speed up adoption and drive scaling of neural models. As scaling laws will likley drive continue model growth methods of compression like quantization, distillation and pruning, are natural explorations points.  \\ 
Model distillation, quantization, and pruning have emerged as successful methods to produce smaller networks from the original over-parametized network. In model distillation \cite{Ba2014DoDN} the original network is used to train a smaller network to mimic the behavior of the large network. In model quantization \cite{Han2016DeepCC} models are made smaller by  reducing the numbers of bits that represent each weight. Model Pruning \cite{LeCun1989OptimalBD} has focused on finding sub-networks in the original network by structured and unstructured pruning. In structured pruning, successful sub-networks are found by removing neurons  \cite{Wang2019StructuredPF} or larger network specific structures like attention heads \cite{Voita2019AnalyzingMS}. In unstructured pruning the successful sub-networks are found by setting individual weights to zero \cite{Kwon2019StructuredCB}. With the introduction of transformer \cite{Vaswani2017AttentionIA} a simple compression method became normal, removal of transformer layers. Since methods like BERT rely on 12 and 24 stacked encoder layers it has become common that removal of encoding layers can be used to decrease the model size. Despite a wide exploration of these methods as applied to NLP, few methods have combined all three approaches and to the best of our knowledge there has been no explicit work focused on IR.\\
In our work we explore the intersection of unstructured pruning, model distillation, and layer dropping to generate the optimal compressed models for two IR tasks: Question Answering and Passage Ranking. While previous work on compressing question answering does exist \cite{Sanh2020MovementPA}, the work does not explore removal of layers, nor explore the effect of training length on pruning schedule. Moreover, despite the popularity of deep learning scale IR datasets like MSMARCO Passage Ranking (MPR) \cite{Campos2016MSMA}, no work has explored how compress-able these models are. Using these large scale datasets we find that question answering models can be pruned to 90 percent of their original weights, 50 \% of their layers removed and still keep close to the original performance. 
We believe the results of our work could motivate our future research and that of the broader IR community. 
The contributions of our work are as follows:
\begin{enumerate}
    \item Our work provides a exploration of the connection between distillation, network pruning, and layer removal for information retrieval tasks.
    \item Our work provides the first application of network pruning for passage retrieval
    \item Our Work provides the first exploration of training time as a factor in network pruning.
\end{enumerate}