Xeon/KNC/KNL) and/or many-core architectures (GPUs) as commonplace in Leadership Class Facilities (LCFs).

DL algorithms can be applied to a variety of input representations. The tabular input representations typically leverage Multi-layer Perceptrons (MLPs). The images, videos and speech tend to leverage the Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). The CNNs and RNNs are computationally expensive and typically require significant time for training even on relatively modest data set sizes with modest number of hidden layers. The problem is further exacerbated by the increasing number of layers (such as recently proposed Residual Networks have up to 1000 layers) and ever-increasing volume of data produced by simulations, experiments and hand-held devices. An important solution to these problems is the design and implementation of DL algorithms that are capable of execution on distributed memory systems. Table \[table:newcomparisons\] shows a table of prominent distributed DL implementations.

                    Name                     HPC Ready   Fault Tolerance
  ----------------------------------------- ----------- -----------------
           FireCaffe [@firecaffe]                       
              S-Caffe [@scaffe]                         
               MaTEx [@matex]                           
            Poseidon [@poseidon]                        
              Petuum [@petuum]                          
               GeePS [@geeps]                                   ?
         ProjectAdam [@projectadam]                     
   TensorFlow [@tensorflow2015-whitepaper]              
               MXNET [@mxnet]                           
        CaffeonSpark [@caffeonspark]                    
            SparkNet [@sparknet]                        
             DogWild [@sparknet]                                ?
                CNTK [@cntk]                            
               Parle [@parle]                           
            PaddlePaddle [@mxnet]                       
           Caffe2 [@goyal:arxiv17]                      
              Proposed FT-Caffe                         

  : []{data-label="table:newcomparisons"}

\

An important artifact of the large scale systems is 