<p>Building a character-level self-attentive model is a challenging task. Character-level models are usually based on RNNs. Whereas in a word/subword model, it is clear from the beginning what are the units carrying meaning (and therefore the units the attention mechanism can attend to), a character-level model needs to learn word meaning in the following layers. This makes it quite difficult for the model to learn.</p>

<p>Text generation models are nothing more than conditional languages model. Google AI recently published a paper on <a href="https://arxiv.org/abs/1908.10322" rel="nofollow noreferrer">Transformer character language model</a>, but it is the only work I know of.</p>

<p>Anyway, you should consider either using subwords units (as BPE, SentencePiece) or if you really need to go for character level, use RNNs instead.</p>
Attention Text Generation in Character-by-Character fashion
<p>I am searching the web for a couple of days for any <strong>text generation</strong> model that would use only attention mechanisms.</p>

<p>The <strong>Transformer</strong> architecture that made waves in the context of <strong>Seq-to-Seq</strong> models is actually based solely on <strong>Attention</strong> mechanisms but is mainly designed and used for translation or chat bot tasks so it doesn't fit to the purpose, but the principle does.</p>

<p>My question is:</p>

<p>Does anyone knows or heard of a text generation model <strong>based solely on Attention without any recurrence</strong>?</p>

<p>Thanks a lot!</p>

<p>P.S. I'm familiar with <strong>PyTorch</strong>.</p>