The transformer has become a central model for many NLP tasks from translation to language modeling to representation learning. Its success demonstrates the effectiveness of stacked attention as a replacement for recurrence for many tasks. In theory attention also offers more insights into the model’s internal decisions; however, in practice when stacked it quickly becomes nearly as fully-connected as recurrent models. In this work, we propose an alternative transformer architecture, discrete transformer, with the goal of better separating out internal model decisions. The model uses hard attention to ensure that each step only depends on a fixed context. Additionally, the model uses a separate “syntactic” controller to separate out network structure from decision making. Finally we show that this approach can be further sparsified with direct regularization. Empirically, this approach is able to maintain the same level of performance on several datasets, while discretizing reasoning decisions over the data. The transformer has achieved state-of-the-art performances in a variety of sequence modeling tasks, including language modeling (Radford et al., 2019) , machine translation (Vaswani et al., 2017) , question answering (Radford et al., 2018; Devlin et al., 2018) , among others. To facilitate parallel training, as well as to reduce the path length of the dependencies, transformer dispenses recurrence and builds up hidden states by attending to the source side (inter-attention) and attending to its past predictions (self-attention) with multiple heads in multiple layers (Vaswani et al., 2017) . Compared to recurrent models the attention mechanism adds some "interpretability" to a model's decision (Bahdanau et al., 2014; Xu et al., 2015; Chan et al., 2015) . However, in the commonly used soft attention mechanism (Luong et al., 2015) each input element receives non-zero weight, and so it is unclear whether the magnitude of attention weights reflects the relative importance of the corresponding inputs (Jain & Wallace, 2019) . To make things worse, due to the existence of multiple stacked attention layers in transformer, it becomes even harder to discriminate the contributions of each input to the final decisions made by the model. Can we force the transformer to make sharper, discrete internal decisions? In this work, we consider a variant of the transformer architecture with the goal of maintaining performance while forcing discrete decisions. Specifically, we consider a discrete transformer with three changes to the architecture: (a) we propose to treat attention as a categorical latent variable (Deng et al., 2018; Shankar et al., 2018) and use hard attention mechanism to get discrete attention decisions (Xu et al., 2015) , (b) we propose to separate out the querying mechanism from value computation into intertwine soft "syntactic" and hard "semantic" model streams, and (c) we consider extension to the discrete transformer to allow for further additions such as attention sparsity regularization. Training of the model is very similar to standard transformer training. The key benefits come at inference time. First, we can use a simple decoding procedure where we take argmax attentions such that each intermediate representation is only built up based on the subset of the attended lower layer outputs. In turn, each final prediction uses limited receptive field, and we can even the guarantee that any hidden state does not depend on input elements not being directly attended to. Second, we can split out attention prediction from computation, and even fix the structure of the feed-forward network for a given example. To validate this approach, we perform experiments on several tasks. We first validate that with proper attention and sparsity regularization the model can learn the truly necessary attentions on a synthetic language modeling task. Next on two real world machine translation datasets, we show that with our approach we can learn transformer models using limited context for making predictions while not deteriorating their performance by too much, indirectly validating the selectiveness of the attention mechanism. The rest of the paper is organized as follows: In Section 2 we draw the connections of our work to the literature. We introduce background and discuss our approach in Sections 3, 4 and 5. Experiments, results and analyses are presented in Sections 6 and 7, and we conclude our paper in Section 8. This work presents the discrete transformer, a modification to the transformer to make discrete attention decisions and to separate out dependencies from semantic state value. Experiments show that despite the more structured decisions the model is able to maintain similar performance on standard machine translation benchmarks. Analysis shows that the model separates out syntactic properties and even learns precise decisions on clean data. This style of model opens up the potential for many possible experiments in NLP. Because the model makes hard intermediary decisions the semantic model can be shown to only depend on a subset of the data. This property could be used to check for or remove bias from a model, for instance to ensure that production gendered pronoun does not depend on spurious context. Similarly because the dependencies are predicted separately additional priors or regularization could be used to enforce specific syntactic structure. Finally, this method could be used to train pretrained models that allow for discrete intermediary structure.
Discrete transformer which uses hard attention to ensure that each step only depends on a fixed context. This paper presents modifications to the standard transformer architecture with the goal of improving interpretability while retaining performance in NLP tasks. This paper proposes three Discrete Transformers: a discrete and stochastic Gumbel-softmax based attention module, a two-stream syntactic and semantic transformer, and sparsity regularization.