|
--- |
|
license: cc-by-nc-4.0 |
|
--- |
|
|
|
# NPM |
|
|
|
NPM is a nonparametric masked language model, pretrained on English text data. |
|
It was introduced by ["Nonparametric Masked Language Modeling"][paper] |
|
and first released in [facebookresearch/NPM][repo]. |
|
|
|
### Model description |
|
|
|
NPM consists of an encoder and a reference corpus, and models a nonparametric distribution over a reference corpus. |
|
The key idea is to map all the phrases in the corpus into a dense vector space using the |
|
encoder and, when given a query with a MASK at inference, use the encoder to locate the nearest |
|
phrase from the corpus and fill in the MASK. |
|
|
|
### Intended uses & limitations |
|
While this repo includes the encoder weights, NPM has to be used together with a datstore. |
|
For more details on how to use NPM, please refer to the [original repo][repo]. |
|
|
|
Note that this model is primarily for filling in a MASK token. Future work can investigate how to use NPM for text generation. |
|
|
|
### Training procedure |
|
|
|
NPM was trained on English Wikipedia (August 2019) and an English portion of CC-News (Mackenzie et al. (2020), February 2019), which contains 13B tokens in total. |
|
NPM used the model architecture and initial weights of RoBERTa large (Liu et al., 2019), consisting of 354M parameters. |
|
Training is done for 100,000 steps, using thirty-two 32GB GPUs. |
|
|
|
More details about training can be found in the [paper][paper]. |
|
Code for training NPM can be found in the [original repo][repo]. |
|
|
|
### Evaluation results |
|
NPM is evaluated on nine closed-set tasks (tasks with a small set of options given) |
|
and seven open-set tasks (tasks whose answers are arbitrary-length). |
|
NPM consistently outperforms significantly larger models such as GPT-3, OPT and T5. |
|
Detailed results can be found from the [paper][paper]. |
|
|
|
### BibTeX entry and citation info |
|
``` |
|
@article{ min2022nonparametric, |
|
title={ Nonparametric Masked Language Modeling }, |
|
author={ Min, Sewon and Shi, Weijia and Lewis, Mike and Chen, Xilun and Yih, Wen-tau and Hajishirzi, Hannaneh and Zettlemoyer, Luke }, |
|
year={ 2022 } |
|
} |
|
``` |
|
|
|
[paper]: https://arxiv.org/abs/2212.01349 |
|
[repo]: https://github.com/facebookresearch/NPM |
|
|
|
|
|
|
