<p>It depends on the type of position encoding the Transformer uses. Models with learned static position embeddings (such as BERT) cannot go beyond the number of learned positions, simply because they cannot embed the next input for the decoder to produce an output.</p>
<p>The original Transformer for machine translation, uses analytically defined position encoding (so-called sinusoidal encoding) which in theory should generalize for arbitrarily long inputs and outputs. However, in practice, it generalizes badly for sequences that are much longer than those in the training data.</p>
<p>If you want to read more about position encoding in Transformers, you can checkout <a href="https://arxiv.org/abs/2102.11090" rel="nofollow noreferrer">this survey</a>.</p>
Is there a maximum sequence length for the output of a transformer?
<p>There's just one thing that I can't find an answer to :
When putting the ouput back in the transformer, we compute it similarly to the inputs (with added masks), so is there also a sequence size limit ?</p>
<p>Even BERT has an input size limit of 512 tokens, so transformers are limited in how much they can take in.
So is there something to make the output length as big as wanted or is there a fixed max length ?</p>
<p>If I wasn't clear enough, does the network generate words infinitely until the &lt; end &gt; token or is there a token limit for the outputs?</p>