Spaces:
Runtime error
Runtime error
File size: 1,462 Bytes
c5ca37a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
# Download/Pre-process Datasets
## Wikipedia
Download processed files (11.78G) below, and unzip it (298 files)
https://chunylcus.blob.core.windows.net/machines/msrdl/optimus/data/datasets/wikipedia_json_64_filtered.zip
Download raw file (11.79G):
https://chunylcus.blob.core.windows.net/machines/msrdl/optimus/data/datasets/wikipedia.segmented.nltk.txt
Our pre-processing protocal: We split the original wiki text into 298 files, and loop over files in one epoch.
We filter each sentence in wiki based on two constraints: (1) The sentence length is smaller than 64. (2) The tokenized sentence length is smaller than 256 (so that the encoder can take the entire sentence).
To filter the sentence, please change the data folders and run the script:
sh scripts/scripts_local/run_data_filtering_wiki.sh
The filtered files are saved in "data/datasets/wikipedia_json_64_filtered".
## Fine-tuning datasets
Language Modeling: Penn, Yelp, Yahoo, Snli
https://chunylcus.blob.core.windows.net/machines/msrdl/optimus/data/datasets/penn_data.zip
https://chunylcus.blob.core.windows.net/machines/msrdl/optimus/datasets/yelp_data.zip
https://chunylcus.blob.core.windows.net/machines/msrdl/optimus/data/datasets/yahoo_data.zip
https://chunylcus.blob.core.windows.net/machines/msrdl/optimus/data/datasets/snli_data.zip
(Stylized) Dialog response generation: DailyDialog, Holmes
Label-conditional text generation: Yelp.
Language Understanding: GLUE, Yelp.
|