<p>Your understanding is quite correct. However, unfortunately, there is inconsistency between the Tensorflow terminology and the literature. In order to understand, you need to dig through the Tensorflow implementation code. </p>

<p>A <strong>cell</strong> in the Tensorflow universe is called an LSTM layer in Colah's universe (i.e an unrolled version). That is why you always define a single cell, and not a layer in your Tensorflow architecture. For example,</p>

<pre><code>cell=rnn.BasicLSTMCell(num_units=5,state_is_tuple=True)
</code></pre>

<p>Check the code here.</p>

<p><a href="https://github.com/tensorflow/tensorflow/blob/ef96faaf02be54b7eb5945244c881126a4d38761/tensorflow/python/ops/rnn_cell.py#L90" rel="nofollow noreferrer">https://github.com/tensorflow/tensorflow/blob/ef96faaf02be54b7eb5945244c881126a4d38761/tensorflow/python/ops/rnn_cell.py#L90</a></p>

<blockquote>
  <p>The definition of cell in this package differs from the definition used in the literature. In the literature, cell refers to an object with a single scalar output. The definition in this package refers to a horizontal array of such units.</p>
</blockquote>

<p>Therefore, in order to understand num_units in Tensorflow, its best to imagine an unrolled LSTM as below.</p>

<p><a href="https://i.stack.imgur.com/JGVM8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JGVM8.png" alt="enter image description here"></a></p>

<p>In an unrolled version, you have an input <strong>X_t</strong> which is a tensor. When you specify an input of the shape </p>

<blockquote>
  <p>[batch_size,time_steps,n_input]</p>
</blockquote>

<p>to Tensorflow, it knows how many times to unroll it from your <strong>time_steps</strong> parameter.</p>

<p>So if you have <strong>X_t</strong> as a 1D array in TensorFlow, then in the Colahs unrolled version each LSTM cell <strong>x_t</strong> becomes a scalar value (Please observe the capital case X (vector/array) and small case x(scalar) - Also in Colah's figures)</p>

<p>If you have <strong>X_t</strong> as a 2D array in the Tensorflow, then in the Colahs unrolled version each LSTM cell <strong>x_t</strong> becomes a 1D array/vector (as in your case here) and so on.</p>

<p>Now here comes the most important question.</p>

<p><strong>How would Tensorflow know what is the output/hidden dimension ** Z_t/H_t</strong> ? </p>

<p>(Please note the difference between H_t and Z_t - I usually prefer to keep them separate as H_t goes back to input (the loop) and <strong>Z_t</strong> is the output - Not shown in figure)</p>

<p>Would it be same dimension as <strong>X_t</strong> ? </p>

<p><strong>No</strong>.It can be of any different shape. You need to specify it to the Tensorflow. And that is <strong>num_units - The Output Size</strong></p>

<p>Check here in the code:</p>

<p><a href="https://github.com/tensorflow/tensorflow/blob/ef96faaf02be54b7eb5945244c881126a4d38761/tensorflow/python/ops/rnn_cell.py#L298-L300" rel="nofollow noreferrer">https://github.com/tensorflow/tensorflow/blob/ef96faaf02be54b7eb5945244c881126a4d38761/tensorflow/python/ops/rnn_cell.py#L298-L300</a></p>

<pre><code>    @property
    def output_size(self):
        return self._num_units
</code></pre>

<p>Tensorflow uses the implementation of LSTM cell as defined in Colahs universe from the following paper:</p>

<p><a href="https://arxiv.org/pdf/1409.2329.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1409.2329.pdf</a></p>
What is actually num_unit in LSTM cell circuit?
<p>I tried very hard to search everywhere, but I couldn't find what <code>num_units</code> in TensorFlow actually is. I tried to relate my question to <a href="https://stackoverflow.com/questions/37901047/what-is-num-units-in-tensorflow-basiclstmcell">this question</a>, but I couldn't get clear explanation there.</p>

<hr>

<p>In TensorFlow, when creating an LSTM-based RNN, we use the following command</p>

<pre><code>cell = rnn.BasicLSTMCell(num_units=5, state_is_tuple=True)
</code></pre>

<p>As <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="noreferrer">Colah's blog</a> says, this is a basic LSTM cell:</p>

<p><a href="https://i.stack.imgur.com/EL47v.png" rel="noreferrer"><img src="https://i.stack.imgur.com/EL47v.png" alt="enter image description here"></a></p>

<p>Now, suppose my data is:</p>

<pre><code>idx2char = ['h', 'i', 'e', 'l', 'o']

# Teach hello: hihell -&gt; ihello
x_data = [[0, 1, 0, 2, 3, 3]]   # hihell
x_one_hot = [[[1, 0, 0, 0, 0],   # h 0
              [0, 1, 0, 0, 0],   # i 1
              [1, 0, 0, 0, 0],   # h 0
              [0, 0, 1, 0, 0],   # e 2
              [0, 0, 0, 1, 0],   # l 3
              [0, 0, 0, 1, 0]]]  # l 3

y_data = [[1, 0, 2, 3, 3, 4]]    # ihello
</code></pre>

<p>My input is:</p>

<pre><code>x_one_hot = [[[1, 0, 0, 0, 0],   # h 0
              [0, 1, 0, 0, 0],   # i 1
              [1, 0, 0, 0, 0],   # h 0
              [0, 0, 1, 0, 0],   # e 2
              [0, 0, 0, 1, 0],   # l 3
              [0, 0, 0, 1, 0]]]  # l 3
</code></pre>

<p>which is of shape <code>[6,5]</code>. </p>

<p>In <a href="https://jasdeep06.github.io/posts/Understanding-LSTM-in-Tensorflow-MNIST/" rel="noreferrer">this blog</a>, we have the following picture</p>

<p><a href="https://i.stack.imgur.com/wgpPY.png" rel="noreferrer"><img src="https://i.stack.imgur.com/wgpPY.png" alt="enter image description here"></a></p>

<p>As far as I know, the <code>BasicLSTMCell</code> will unroll for <code>t</code> time steps, where <code>t</code> is my number of rows (please, correct me if I am wrong!). For example, in the following figure, the LSTM is unrolled for <code>t = 28</code> time steps.</p>

<p><a href="https://i.stack.imgur.com/fGWr7.png" rel="noreferrer"><img src="https://i.stack.imgur.com/fGWr7.png" alt="enter image description here"></a></p>

<p>In the Colah's blog, it's written</p>

<blockquote>
  <p>each line carries an entire vector</p>
</blockquote>

<p>So, let's see how my <code>[6,5]</code> input matrix will go through this LSTM-based RNN.</p>

<p><a href="https://i.stack.imgur.com/Qje2S.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Qje2S.png" alt="enter image description here"></a></p>

<p><em>If my above diagram is correct, then what exactly is <code>num_units</code> (which we defined in LSTM cell)</em>? Is it a parameter of an LSTM cell?</p>

<p>If <code>num_unit</code> is a parameter of a single LSTM cell, then it should be something like:</p>

<p><a href="https://i.stack.imgur.com/zK7Jb.png" rel="noreferrer"><img src="https://i.stack.imgur.com/zK7Jb.png" alt="enter image description here"></a></p>

<p><a href="https://i.stack.imgur.com/7mJ2z.png" rel="noreferrer"><img src="https://i.stack.imgur.com/7mJ2z.png" alt="enter image description here"></a></p>

<p><em>If above diagram is correct, then where are those 5 <code>num_units</code> in the following schematic representation of the LSTM cell (according to Colah's blog)?</em></p>

<p><a href="https://i.stack.imgur.com/sHJoV.png" rel="noreferrer"><img src="https://i.stack.imgur.com/sHJoV.png" alt="enter image description here"></a></p>

<hr>

<p>If you can give your answer with a figure, that would be really helpful! You can edit or create new whiteboard diagram <a href="https://awwapp.com/b/schvorrmy?dis=%5B%5B%22edit-board-name%22%5D%2C%5B%22no-init-modal%22%5D%5D" rel="noreferrer">here</a>.</p>