<blockquote>
<p>I am trying to speed up this training process to run the 2 models at
the same time using multiple devices</p>
</blockquote>
<p>I doubt that would bring any speed up, especially in case of:</p>
<blockquote>
<p>(e.g., loading one model on CPU and not interrupting the GPU training
of the other model).</p>
</blockquote>
<p>as deep learning is a pipeline which also utilizes CPU, possibly multiple cores (say for data loading but also receiving metrics, gathering them etc.).</p>
<p>Furthermore CPU is rather ineffective for neural network training when compared to GPU/TPU unless you have some tailored CPU architecture (stuff like <a href="https://arxiv.org/abs/1801.04381" rel="nofollow noreferrer">MobileNet</a>). If you were to train student on CPU, you might significantly slow down pipeline elements of <code>teacher</code>.</p>
<blockquote>
<p>What is the proper way to run 2 models in parallel?</p>
</blockquote>
<p>Again, depending on the model, but it would be best to utilize <code>2</code> GPUs for training and split CPU cores for other tasks between them. In your case you would have to synchronize teacher and student predictions across two devices though.</p>
<blockquote>
<p>Can I use Python multiprocessing library to start 2 processes for the 2 models, i.e., loading 2 model instances and running forward()?</p>
</blockquote>
<p>PyTorch provides primitives (e.g. &quot;their&quot; <code>multiprocessing</code> wrapper, Futures etc.) which could possibly be used for that, not sure about <code>mxnet</code> or a-like.</p>
In knowledge distillation, how to run the student and the teacher models in parallel?
<p>I am implementing fast DNN model training using knowledge distillation, as illustrated in the figure below, to run the teacher and student models in parallel.</p>
<p>I checked some popular repos like <a href="https://github.com/NervanaSystems/distiller/blob/master/distiller/knowledge_distillation.py#L105" rel="nofollow noreferrer">NervanaSystems/distiller</a> and <a href="https://github.com/peterliht/knowledge-distillation-pytorch/blob/e4c40132fed5a45e39a6ef7a77b15e5d389186f8/train.py#L277" rel="nofollow noreferrer">peterliht/knowledge-distillation-pytorch</a>. They execute the forward operations of the student and teacher models step by step, i.e., not in parallel on different devices (GPU or CPU).</p>
<p>I am trying to speed up this training process to run the 2 models at the same time using multiple devices (e.g., loading one model on CPU and not interrupting the GPU training of the other model).</p>
<p>What is the proper way to run 2 models in parallel? Can I use Python <code>multiprocessing</code> library to start 2 processes for the 2 models, i.e., loading 2 model instances and running <code>forward()</code>? I am using MXNet but this is a general question for all ML frameworks.</p>
<p>Edit:<br />
My plan is to put a light-weight pre-trained teacher model on CPU which only runs forward pass with frozen parameters.<br />
The student model is a large model to be trained on GPU (distributedly).
This task is not for model compression.
I suppose moving a light task (teacher's forward pass) to CPU can increase the overlap and make this pipeline faster.<br />
The idea is from a workshop paper: <a href="http://learningsys.org/nips18/assets/papers/24CameraReadySubmissionInfer2Train.pdf" rel="nofollow noreferrer">Infer2Train: leveraging inference for better training of deep networks</a>.</p>
<p><a href="https://i.stack.imgur.com/uOrit.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uOrit.png" alt="knowledge distillation illustration" /></a></p>