<p>No, the interpretation is not correct (unless my interpretation of your interpretation is incorrect). <code>x</code> is an input, and it is fixed in advance, so <code>x(t+1)</code> does not depend on the predicted value for timestep <code>t</code>.</p>

<p>In that paragraph he discusses a particular case of an RNN, where <code>y(t)</code> is a prediction of <code>x(t + 1)</code>, in other words, the network is trying to predict the next symbol given all the previous symbols.</p>

<p>My understanding of the sentence you are referring to is that since <code>y</code> is a result of a softmax, <code>y</code> has a limited range of values it can assume, and therefore <code>x</code> on itself has to be limited to the same range of values, hence <code>x</code> has to be a "symbol or bounded integer". Otherwise, if <code>x</code>, for instance, is a double, <code>y</code> cannot predict it, since the output of a softmax is a discrete value.</p>

<p>UPDATE: as a matter of fact, Bengio has a great paper:
<a href="http://arxiv.org/abs/1506.03099" rel="nofollow">http://arxiv.org/abs/1506.03099</a>
in which he actually suggests that on some iterations we use <code>y(t)</code> instead of <code>x(t+1)</code> as input when predict <code>y(t+1)</code> during training (which is along the lines of your understanding in your question).</p>
A particular paragraph in Deep Learning - Bengio
<p>This question concerns with the  <a href="http://goodfeli.github.io/dlbook/contents/rnn.html#pf2" rel="nofollow" title="Chapter on RNNs">chapter on RNNs</a> in the 
Deep learning look by Prof Bengio. 
In section 10.2.2 on page 336 in the last paragraph, the book talks about
"...because the outputs are the result of a softmax, it must be that the input sequence is a sequence of symbols...". </p>

<p>This seems to suggest that the output is treated as a probability distribution over the possible 'bits' and the next input x(t+1) is sampled using this joint probability distribution over the output bits. Is this interpretation correct? </p>