the *feature-based* approach (e.g., ELMo, CoVe), fixed features are extracted from the contextual encoder and inserted into task-specific architectures. In the *fine-tuning* approach (e.g., BERT, GPT-1), the contextual encoder is designed as a part of the network architecture for downstream tasks, and its parameters are fine-tuned with downstream task data. BERT was designed for the *fine-tuning* approach but it was also evaluated with the *feature-based* approach. GPT-2 is a scaled-up version of GPT-1 and exhibits strong performance under zero-shot settings.

#### The Large Vocabulary Issue

The large and ever-growing vocabulary has been considered an obstacle to scaling up language models. We review existing solutions to this issue from both the language modeling and contextual representation literature.

Most studies for language modeling focus on directly reducing the complexity of the softmax layer. Following , we roughly group them into two categories: sampling-based approximations and structural approximations. Sampling-based approximations include the sampled softmax [@sampled_softmax] and NCE [@NCE]. The sampled softmax approximates the normalization term of softmax by sampling a subset of negative targets, while NCE replaces the softmax with a binary classifier. On the other hand, structural approximations such as the hierarchical softmax [@Hierachical_Softmax] and the adaptive softmax [@adaptive_softmax], form a structural 