title: Tensorflow examples as rows or columns?
body: <p>Given the Tensorflow API <code>nn.softmax</code>, <code>nn.sparse_softmax_cross_entropy_with_logits</code>, <code>tf.nn.rnn</code> etc. it seems that the convention is to have the samples placed as rows in a batch.</p>
  
  <p>A forward pass in a neural network is then <code>tf.matmul(input_batch, W) + b</code>, where <code>input_batch</code> is a matrix of shape <code>[n_samples, input_size]</code>. The <code>n_samples</code> is often named <code>batch_size</code> in the API. The addition of the row vector <code>b</code> is broadcasted on all samples(rows) in the batch.</p>
  
  <p>Is this a general adopted convention when working with neural networks? I have read a lot of research articles that have the samples as columns and use left multiplication of the weight matrix for a forward pass. Why was this convention chosed? </p>
  
  <p>EDIT</p>
  
  <p>This article uses multiplication of the weight matrix on the left side of the input, as well as vector concatenation, so the samples must be  columns:
  <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow noreferrer">http://colah.github.io/posts/2015-08-Understanding-LSTMs/</a>. It is referred to in the Tensorflow documentation here: <a href="https://www.tensorflow.org/versions/r0.11/tutorials/recurrent/index.html" rel="nofollow noreferrer">https://www.tensorflow.org/versions/r0.11/tutorials/recurrent/index.html</a>. </p>
  
  <hr>
  
  <p>The implementation of BasicLSTMCell is based on this article <a href="https://arxiv.org/pdf/1409.2329v5.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1409.2329v5.pdf</a> according to the source here <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/rnn_cell.py" rel="nofollow noreferrer">https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/rnn_cell.py</a>. You also see multiplication from the left.</p>
  
  <hr>
  
  <p>Finally in this article <a href="https://arxiv.org/pdf/1506.00019.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1506.00019.pdf</a>, the activations are also column vectors.</p>