<p>There are several forms of parallelism that TensorFlow provides when training a convolutional neural network (and many other machine learning models), including:</p>

<ol>
<li><p><strong>Parallelism within individual operations</strong> (such as <a href="https://www.tensorflow.org/api_docs/python/nn/convolution#conv2d" rel="nofollow noreferrer"><code>tf.nn.conv2d()</code></a> and <a href="https://www.tensorflow.org/api_docs/python/math_ops/matrix_math_functions#matmul" rel="nofollow noreferrer"><code>tf.matmul()</code></a>). These operations have efficient parallel implementations for multi-core CPUs and GPUs, and TensorFlow uses these implementations wherever available.</p></li>
<li><p><strong>Parallelism between operations</strong>. TensorFlow uses a dataflow graph representation for your model, and where there are two nodes that aren't connected by a directed path in the dataflow graph, these may execute in parallel. For example, the Inception image recognition model has many parallel branches in its dataflow graph (see figure 3 in <a href="https://arxiv.org/pdf/1409.4842.pdf" rel="nofollow noreferrer">this paper</a>), and TensorFlow can exploit this to run many operations at the same time. The <a href="https://arxiv.org/pdf/1404.5997v2.pdf" rel="nofollow noreferrer">AlexNet paper</a> also describes how to use "model parallelism" to run operations in parallel on different parts of the model, and TensorFlow supports that using the same mechanism.</p></li>
<li><p><strong>Parallelism between model replicas</strong>. TensorFlow is also designed for <a href="https://www.tensorflow.org/how_tos/distributed/" rel="nofollow noreferrer">distributed execution</a>. One common scheme for parallel training ("data parallelism") involves sharding your dataset across a set of identical workers, performing the same training computation on each of those workers for different data, and sharing the model parameters between the workers.</p></li>
</ol>

<p>In addition, libraries like TensorFlow and Theano can perform various optimizations when they can work with the whole dataflow graph of your model. For example, they can eliminate common subexpressions, avoid recomputing constant values, and generate more efficient fused code.</p>
Why use TensorFlow for Convolutional Neural Networks
<p>I recently took a courser by Andrew Ng on Coursera. After that I shifted to Python and used Pandas, Numpy, Sklearn to implement ML algorithms. Now while surfing I came across tensorFLow and found it pretty amazing, and implemented this <a href="https://www.tensorflow.org/tutorials/mnist/beginners/" rel="nofollow noreferrer">example</a> which takes MNIST data as input.
But I am unsure why use such as library(TensorFlow)?
We are not doing any parallel calculations, since the weights updated in the previous epoch are used in the next one???
I am finding it difficult to find a reason to use such a Library?</p>