Class Facilities (LCFs).

It is also worthwhile noting that this approach is amenable to fault tolerance, especially if the reliability of the parameter server is higher than workers. A possible implementation in the master-slave paradigm is either re-spawning of new workers and splitting the original training set among these new workers (by using `MPI_Comm_spawn`) or continue executing in the presence of faults using the remaining set of compute nodes. Other researchers have made similar observations in the context of generic master-slave applications [@gropp:ijhpca02] and they are readily applicable to DL algorithms. However, due to the fundamental scaling issues of the master-slave paradigm for DL implementations, we disregard this approach from implementation.

Model Parallelism
-----------------

Another possibility which has been presented in literature for scaling out DL implementations is [*model parallelism*]{}. In this specific type, individual layers of the overall DNN model are split among different compute nodes. The training set itself is split among the compute nodes as well. Let us consider the example of the AlexNet neural network topology as shown in Figure \[fig:alexnet\_topology\]. In a sample execution of model parallelism, each of the hidden layers is resident on a single compute node.

![A pictorial representation of AlexNet [@imagenet] 