<h2>Training</h2>
<p>In general, there are two strategies of parallelizing model training: data parallelism and model parallelism.</p>
<h3>1. Data parallelism</h3>
<p>This strategy splits training data into N partitions, each of which will be trained on different “devices” (different CPU cores, GPUs, or even machines). In contrast to training without data parallelism which produces one gradient per minibatch, we now have N gradients for each minibatch step. The next question is how we should combine these N gradients.</p>
<p>One way to do it is by averaging all the N gradients and then updating the model parameters <em>once</em> based on the average. This technique is called <strong>synchronous distributed SGD</strong>. By doing the average, we have a more accurate gradient, but with a cost of waiting all the devices to finish computing its own local gradient.</p>
<p>Another way is by not combining the gradients — each gradient will instead be used to update the model parameters independently. So, there will be N parameter updates for each minibatch step, in contrast to only one for the previous technique. This technique is called <strong>asynchronous distributed SGD</strong>. Because it doesn't have to wait other devices to finish, the async approach will take less time to complete a minibatch step than the sync approach will do. However, the async approach will produce a more noisy gradient, so it might need to complete more minibatch steps to catch up with the performance (in terms of loss) of the sync approach.</p>
<p>There are many papers proposing some improvements and optimizations on either approach, but the main idea is generally the same as described above.</p>
<p>In the literature there's been some disagreement on which technique is better in practice. At the end most people now settle on the synchronous approach.</p>
<p><strong>Data Parallelism in PyTorch</strong></p>
<p>To do synchronous SGD, we can wrap our model with <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel" rel="noreferrer"><code>torch.nn.parallel.DistributedDataParallel</code></a>:</p>
<pre><code>from torch.nn.parallel import DistributedDataParallel as DDP

# `model` is the model we previously initialized
model = ...

# `rank` is a device number starting from 0
model = model.to(rank)
ddp_model = DDP(model, device_ids=[rank])
</code></pre>
<p>Then we can train it similarly. For more details, you can refer to <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noreferrer">the official tutorial</a>.</p>
<p>For doing asynchronous SGD in PyTorch, we need to <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#asynchronous-multiprocess-training-e-g-hogwild" rel="noreferrer">implement it more manually</a> since there is no wrapper similar to <code>DistributedDataParallel</code> for it.</p>
<p><strong>Data Parallelism in TensorFlow/Keras</strong></p>
<p>For synchronous SGD, we can use <a href="https://www.tensorflow.org/api_docs/python/tf/distribute/MirroredStrategy" rel="noreferrer"><code>tf.distribute.MirroredStrategy</code></a> to wrap the model initalization:</p>
<pre><code>import tensorflow as tf

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    model = Model(...)
    model.compile(...)
</code></pre>
<p>Then we can train it as usual. For more details, you can refer to the official guides on <a href="https://keras.io/guides/distributed_training/" rel="noreferrer">Keras website</a> and <a href="https://www.tensorflow.org/guide/distributed_training" rel="noreferrer">TensorFlow website</a>.</p>
<p>For asynchronous SGD, we can use <a href="https://www.tensorflow.org/api_docs/python/tf/distribute/experimental/ParameterServerStrategy" rel="noreferrer"><code>tf.distribute.experimental.ParameterServerStrategy</code></a> similarly.</p>
<h3>2. Model Parallelism</h3>
<p>This strategy splits the model into N parts, each of which will be computed on different devices. A common way to split the model is based on layers: different sets of layers are placed on different devices. But we can also split it more intricately depending on the model architecture.</p>
<p><strong>Model Parallelism in TensorFlow and PyTorch</strong></p>
<p>To implement model parallelism in either TensorFlow or PyTorch, the idea is the same: to move some model parameters into a different device.</p>
<p>In PyTorch we can use <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.to" rel="noreferrer"><code>torch.nn.Module.to</code></a> method to move a module into a different device. For example, suppose we want to create two linear layers each of which is placed on a different GPU:</p>
<pre><code>import torch.nn as nn

linear1 = nn.Linear(16, 8).to('cuda:0')
linear2 = nn.Linear(8, 4).to('cuda:1')
</code></pre>
<p>In TensorFlow we can use <a href="https://www.tensorflow.org/api_docs/python/tf/device" rel="noreferrer"><code>tf.device</code></a> to place an operation into a specific device. To implement the PyTorch example above in TensorFlow:</p>
<pre><code>import tensorflow as tf
from tensorflow.keras import layers

with tf.device('/GPU:0'):
    linear1 = layers.Dense(8, input_dim=16)
with tf.device('/GPU:1'):
    linear2 = layers.Dense(4, input_dim=8)
</code></pre>
<p>For more details you can refer to <a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" rel="noreferrer"><code>the official PyTorch tutorial</code></a>; or if you use TensorFlow you can even use a more high-level library like <a href="https://github.com/tensorflow/mesh" rel="noreferrer">mesh</a>.</p>
<h3>3. Hybrid: Data and Model Parallelism</h3>
<p>Recall that data parallelism only splits the training data, whereas model parallelism only splits the model structures. If we have a model so large that even after using either parallelism strategy it still doesn't fit in the memory, we can always do both.</p>
<p>In practice most people prefer data parallelism to model parallelism since the former is more decoupled (in fact, independent) from the model architecture than the latter. That is, by using data parallelism they can change the model architecture as they like, without worrying which part of the model should be parallelized.</p>
<h2>Model Inference / Serving</h2>
<p>Parallelizing model serving is easier than parallelizing model training since the model parameters are already fixed and each request can be processed independently. Similar to scaling a regular Python web service, we can scale model serving by spawning more processes (to workaround <a href="https://stackoverflow.com/q/1294382/1403530">Python's GIL</a>) in a single machine, or even spawning more machine instances.</p>
<p>When we use a GPU to serve the model, though, we need to do more work to scale it. Because of how concurrency is handled differently by a GPU compared to a CPU, in order to maximize the performance, we need to do inference request batching. The idea is when a request comes, instead of immediately processing it, we wait some timeout duration for other requests to come. When the timeout is up, even if the number of requests is only one, we batch them all to be processed on the GPU.</p>
<p>In order to minimize the average request latency, we need to find the optimal timeout duration. To find it we need to observe that there is a trade-off between minimizing the timeout duration and maximizing the number of batch size. If the timeout is too low, the batch size will be small, so the GPU will be underutilized. But if the timeout is too high, the requests that come early will wait too long before they get processed. So, the optimal timeout duration depends on the model complexity (hence, the inference duration) and the average requests per second to receive.</p>
<p>Implementing a scheduler to do request batching is not a trivial task, so instead of doing it manually, we'd better use <a href="https://github.com/tensorflow/serving" rel="noreferrer">TensorFlow Serving</a> or <a href="https://github.com/pytorch/serve" rel="noreferrer">PyTorch Serve</a> which already supports it.</p>
<hr />
<p>To learn more about parallel and distributed learning, you can read <a href="https://arxiv.org/abs/1802.09941" rel="noreferrer">this review paper</a>.</p>
Parallelization strategies for deep learning
<p>What strategies and forms of parallelization are feasible and available for <strong>training</strong> and <strong>serving</strong> a neural network?:</p>
<ul>
<li><strong>inside</strong> a machine <strong>across</strong> cores (e.g. GPU / TPU / CPU)</li>
<li><strong>across</strong> machines on a network or a rack</li>
</ul>
<p>I'm also looking for evidence for how they may also be used in e.g. TensorFlow, PyTorch or MXNet.</p>
<h3>Training</h3>
<p>To my knowledge, when training large neural networks on large datasets, one could at least have:</p>
<ol>
<li>Different <strong>cores</strong> <em>or</em> <strong>machines</strong> operate on <strong>different parts of the graph</strong> (&quot;<em><strong>graph</strong> splitting</em>&quot;). E.g. backpropagation through the graph itself can be parallelized e.g. by having different layers hosted on different machines since (I think?) the <strong>autodiff graph</strong> is always a <strong>DAG</strong>.</li>
<li>Different <strong>cores</strong> <em>or</em> <strong>machines</strong> operate on <strong>different samples</strong> of data (&quot;<em><strong>data</strong> splitting</em>&quot;). In SGD, the computation of gradients across batches or samples can also be parallelized (e.g. the gradients can be combined after computing them independently on different batches). I believe this is also called gradient accumulation (?).</li>
</ol>
<p>When is each strategy better for what type of problem or neural network? Which modes are supported by modern libraries? and can one combine all four (2x2) strategies?</p>
<p>On top of that, I have read about:</p>
<ul>
<li><strong>Asynchronous</strong> training</li>
<li><strong>Synchronous</strong> training</li>
</ul>
<p>but I don't know what exactly that refers to, e.g. is it the computation of <strong>gradients</strong> on <strong>different data batches</strong> or the computation of <strong>gradients</strong> on different <strong>subgraphs</strong>? Or perhaps it refers to something else altogether?</p>
<h3>Serving</h3>
<p>If the network is huge, prediction / inference may also be slow, and the model may not fit on a single machine in memory at serving time. Are there any known multi-core and multi-node prediction solutions that work that can handle such models?</p>