Title: AutoDisc: Automatic Distillation Schedule for Large Language Model Compression. Abstract: Driven by the teacher-student paradigm, knowledge distillation is one of the de facto ways for language model compression. Recent studies have uncovered that conventional distillation is less effective when facing a large capacity gap between the teacher and the student, and introduced teacher assistant-based distillation to bridge the gap. As a connection, the scale and the performance of the teacher assistant is crucial for transferring the knowledge from the teacher to the student. However, existing teacher assistant-based methods manually select the teacher assistant, requiring many trials before identifying the optimal teacher assistant. To this end, we propose an Automatic Distillation Schedule (AutoDisc) for large language model compression to discover the optimal teacher assistant in only one trial. In particular, motivated by the finding that the performance of the student is positively correlated to the scale-performance tradeoff of the teacher assistant, AutoDisc designs a $\lambda$-Traddoff to measure the optimality of the teacher assistant. AutoDisc then yields the $\lambda$-Traddoffs of all teacher assistant candidates in an once-for-all optimization with two approximations. The optimal teacher assistant can be automatically selected by uncovering the best $\lambda$-Traddoff. AutoDisc is evaluated with an extensive set of experiments on a language understanding benchmark GLUE. Experimental results demonstrate the improved efficiency with similar or even better effectiveness of our AutoDisc compared to several state-of-the-art baselines. We further apply AutoDisc to a language model with over one billion parameters and show the scalability of AutoDisc.