For sequence models with large word-level vocabularies, a majority of network parameters lie in the input and output layers. In this work, we describe a new method, DeFINE, for learning deep word-level representations efficiently. Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods. DeFINE can be incorporated easily in new or existing sequence models. Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity. On WikiText-103, DeFINE reduces total parameters of Transformer-XL by half with minimal impact on performance. On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters,  achieving comparable performance to state-of-the-art methods with fewer parameters. For machine translation, DeFINE improves a Transformer model by 2% while simultaneously reducing total parameters by 26% Neural models for NLP tasks, such as language modeling and machine translation, require large vocabularies for generality (Chelba et al., 2013; Bahdanau et al., 2015; Luong et al., 2015; Merity et al., 2017) . These models often employ a similar architecture: words, represented as one-hot vectors, are mapped to a dense continuous space; they are then processed by a context model; finally, the contextualized representations are mapped back to a vocabulary-sized vector for computing next-token probabilities. A language modeling example is shown in Figure 1a . The mapping in the first and last steps often uses a shared learned look-up table, referred to as an embedding layer, which takes every word in the vocabulary to a fixed m-dimensional vector. One drawback of this approach is that the number of parameters in the embedding layer increases as the vocabulary size grows, limiting us to small values of m over large vocabularies. Researchers have sought to improve the efficiency of the embedding layer by assigning lower frequency words smaller dimensional vectors, however, significant parameter reductions come at the cost of performance (Morin & Bengio, 2005; Grave et al., 2017a; Baevski & Auli, 2019) . In all these approaches, word embedding is approximated with a linear function from words to vectors. In this work, we introduce DEep Factorized INput word Embeddings (DeFINE) for neural sequence modeling. DeFINE approximates the complicated word embedding function with far fewer parameters compared to standard methods. DeFINE allows for lower-dimensional input and output mappings in sequence models, reducing their computational burden without reducing performance. The representations produced by DeFINE are more powerful than those of other factorization techniques and even standard embedding layers. To accomplish this, DeFINE leverages a hierarchical group transformation (HGT) that learns deep representations efficiently and effectively. HGT connects different subsets of the input using sparse and dense connections. To improve the flow of information, DeFINE introduces a new skip-connection that establishes a direct link with the input layer at every level of its hierarchy, allowing gradient to flow back directly to the input via multiple paths. DeFINE replaces standard word embedding layers, leaving the rest of the model untouched, and so it can be used with a wide variety of sequence modeling architectures. Figure 1 shows how we incorporate DeFINE with Transformer-XL (Dai et al., 2019) , a state-of-the-art Transformer-based language model and the resulting reduction in total parameters. Figure 1: With DeFINE, Transformer-XL learns input (embedding) and output (classification) representations in low n-dimensional space rather than high m-dimensional space, thus reducing parameters significantly while having a minimal impact on the performance. Our experiments show that both LSTM-and Transformer-based sequence models benefit from the use of DeFINE. On the Wikitext-103 dataset, an LSTM-based language model with DeFINE provides a 9 point improvement over a full capacity model while using half as many parameters. When combined with adaptive input (Baevski & Auli, 2019) and output (Grave et al., 2017a) representations, DeFINE improves the performance by about 3 points across LSTM-based (see Table 1a ) and Transformer-XL-based (see Table 2 ) language models with a minimal increase in training parameters. Computation time at inference is unaffected. 1 Incorporating DeFINE into the popular AWD-LSTM language model (Merity et al., 2018b) without finetuning results in a test perplexity of 54.2 on the Penn Treebank dataset, outperforming both the original and fine-tuned AWD-LSTM models as well as Transformer-XL and MoS . For machine translation, DeFINE improves the efficiency of a Transformer model (Vaswani et al., 2017) by 26% while maintaining translation quality. We provide substantive experiments which detail the impact of our architecture decisions and demonstrate the effectiveness of DeFINE across models of varying capacities. DeFINE uses a deep, hierarchical, sparse network with new skip connections to learn better word embeddings efficiently. Sequence models with DeFINE (e.g. Transformer and LSTM) perform comparably or better with state-of-the-art methods with fewer parameters. Our experiments show that the proposed architectural decisions each contribute to the effectiveness of the DeFINE unit. We believe neural sequence models with DeFINE can be further improved with extended hyperparameter search, similar to Melis et al. (2018) . In future work, we will apply DeFINE to other sequence modeling tasks. For instance, we believe that pretrained language model architectures such as ELMo and BERT can benefit from incorporating DeFINE to improve efficiency and performance. Another direction is to use the components of DeFINE -specifically MER, HGT, and mixing layers -in neural architecture search processes. We have shown the promise of these components here, but a thorough architecture search may discover more optimal configurations in the large search space defined by the depth, grouping, and connectivity parameters.
DeFINE uses a deep, hierarchical, sparse network with new skip connections to learn better word embeddings efficiently.  This paper describes a new method for learning deep word-level representations efficiently by using a hierarchical structure with skip-connections for the use of low dimensional input and output layers.