---
abstract: 'Deep Learning (DL) algorithms have become the [*de facto*]{} Machine Learning (ML) algorithm for large scale data analysis. DL algorithms are computationally expensive – even distributed DL implementations which use MPI require days of training (model learning) time on commonly studied datasets. Long running DL applications become susceptible to faults – requiring development of a fault tolerant system infrastructure, in addition to fault tolerant DL algorithms. This raises an important question: [*What is needed from MPI for designing fault tolerant DL implementations?*]{} In this paper, we address this problem for permanent faults. We motivate the need for a fault tolerant MPI specification by an in-depth consideration of recent innovations in DL algorithms and their properties, which drive the need for specific fault tolerance features. We present an in-depth discussion on the suitability of different parallelism types (model, data and hybrid); a need (or lack thereof) for check-pointing of any critical data structures; and most importantly, consideration for several fault tolerance proposals (user-level fault mitigation (ULFM), Reinit) in MPI and their applicability to fault tolerant DL implementations. We leverage a distributed memory implementation of Caffe, currently available under the Machine Learning Toolkit for Extreme Scale (MaTEx). We implement our 