--- license: apache-2.0 datasets: - Skylion007/openwebtext - rajpurkar/squad - google-research-datasets/natural_questions - hotpotqa/hotpot_qa pipeline_tag: question-answering --- # Numini Native-Uttar Mini - Sanju Debnath - Project Type: Question Answering Lightweight SLM Model ## Structure - `data/` contains the data used for the project - `distilbert.py` contains the code for the DistilBERT model and the Dataset. - `distilbert.ipynb` contains the creation and training of the DistilBERT model - `distilbert.model` is the distilbert model - `distilbert_reuse.model` is the question answering model - `load_data.py` contains the code for loading the data and preprocessing it. - `qa_model.py` contains the code for thee different QA models. - `qa_model.ipynb` contains the creation and training of the QA models. - `requirements.txt` contains the requirements for the project - `utils.py` contains some helper functions for the project. It contains the functions to evaluate the models and a way to visualise the trained parameters for each model. - `application.py` contains the streamlit application to run everything ## How to run - Install the requirements with `pip install -r requirements.txt` - Run `load_data.py` to download the data and preprocess it (follow the documentation in the file regarding the natural questions dataset) - Run `distilbert.ipynb` to train the DistilBERT model - Run `qa_model.ipynb` to train the QA models - Run `streamlit run application.py` to run the streamlit app ## Project 1. Create own DistilBERT Model using the OpenWebText dataset from Huggingface (https://huggingface.co/datasets/openwebtext) - 20h (active work, training is a lot longer) - I initially wanted to use the Oscar dataset (https://huggingface.co/datasets/oscar) or the TriviaQA dataset (https://huggingface.co/datasets/mandarjoshi/trivia_qa), but it took too much storage - I will train a MaskedLM model myself. However, my computational resources are limiting me, so my model's performance should not be sufficient, I will use the Huggingface model (https://huggingface.co/distilbert-base-cased) 2. Current methods often fine-tune the models on specific tasks. I believe that MultiTask learning is extremely useful, hence, I want to fix the DistilBERT weights here and train a head to do question answering - 30h - Dataset: SQuAD (https://paperswithcode.com/dataset/squad), also Natural Questions (https://paperswithcode.com/dataset/natural-questions) - The idea is to have one common corpus and specific heads, rather than a separate model for every single task - In particular, I want to evaluate whether it is really necessary to fine-tune the base model too, as it already contains a model of the language. Ideally, having task-specific heads could make up for the lacking fine-tuning of the base model. - If the performance of the model is comparable, this could reduce training efforts and resources - Either add another BERT Layer per task or just the multi-head self-attention layer 3. Application - 10h - GUI, that lets people enter a context (base text), question, and they will receive an answer. - Will contain some SQuAD questions as examples. 4. Documentation - 2h 5. Presentation - 2h ## Goal The DistilBERT model was quite straightforward to train, I mostly used what HuggingFace provided anyways, so the only real challenge here was to download the dataset. Also, training is a lot of effort, so I wasn't able to train it to full convergence, as I just didn't have the resource. The DistilBERT model can be found in `distilbert.ipynb` and is fully functional. * Error Metric: I landed at about 0.2 CrossEntropyLoss for both training and test set. The preconfiguration is quite good, as it didn't overfit. * DistilBERT is primarily trained for masked prediction, I ran some manual sanity tests, to see which words are predicted. They usually make sense (although not entirely sometimes) and the grammatics are usually quite correct too. * e.g. "It seems important to tackle the climate [MSK]." gave change (19%), crisis (12%), issues (5.8%), which are all appropriate in the context. Now for the Question Answering model. * Error Metric: * We use the CrossEntropy loss to train the QA model * Afterwards, we will fall back to F-1 score and the Exact Match (EM). These are also the metrics used for the SQuAD competition. (https://rajpurkar.github.io/SQuAD-explorer/). * The definitions are retrieved from here (https://qa.fastforwardlabs.com/no%20answer/null%20threshold/bert/distilbert/exact%20match/f1/robust%20predictions/2020/06/09/Evaluating_BERT_on_SQuAD.html#Metrics-for-QA). * EM: 1 if the prediction exactly matches the original, 0 otherwise * F-1: Computed over the individual words in the prediction against those in the answer. Number of shared words is the key. Precision: Ratio of shared words to the number of words in the prediction. Recall: Ratio of shared words to number of words in GT. * Target for Error Metric: * EM: 0.6 * F-1: 0.7 * Achieved value: I almost achieved the target for both of the measurements. * EM: 0.52 * F-1: 0.67 Amount of time for each task: * DistilBERT model: ~20h (without training time). This was very similar to what I estimated, because I relied heavily on the Huggingface library. Loading the data was easy and the data is already very clean. * QA model: ~40h (without training time). Was a lot of effort, as my first approach didn't work and it took me making up a basic POC model, to get to the final architecture. * Application: 2h. Streamlit is easy yet faced a lot of issues for the application ## Data - Aaron Gokaslan et al. OpenWebText Corpus. 2019. https://skylion007.github.io/OpenWebTextCorpus/: **OpenWebText** - Open source replication of the WebText dataset from OpenAI. - They scraped web pages, with a focus on quality. They looked at the Reddit up- and downvotes to determine the quality of the resource. - The dataset will be used to train the DistilBERT model using language masking. - Rajpurkar et al. SQuAD: 100,000+ Questions for Machine Comprehension of Text. 2016. https://rajpurkar.github.io/SQuAD-explorer/): **SQuAD** - Standford Question Answering Dataset - Collection of question-answer pairs, where the answer is a sequence of tokens in the given context text. - Very diverse because it was created using crowdsourcing. - Kwiatkowski et al. Natural Questions: a Benchmark for Question Answering Research. 2019. https://ai.google.com/research/NaturalQuestions/: **Natural Questions** - Also a question-answer set, based on a Google query and corresponding Wikipedia page, containing the answer. - Very similar to the SQuAD dataset. - Yang, Zhilin et al. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. https://hotpotqa.github.io/ ## Related Papers - Sanh, Victor et al. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv abs/1910.01108. 2019.: https://arxiv.org/abs/1910.01108v4 - The choice of DistilBERT, as opposed to BERT, RoBERTa or XLNet is primarily based on the size of the network and training time - I hope that the slight performance degradation will be compensated by the head, that is fine-tuned - Ł. Maziarka and T. Danel. Multitask Learning Using BERT with Task-Embedded Attention. 2021 International Joint Conference on Neural Networks (IJCNN). 2021, pp. 1-6: https://ieeexplore.ieee.org/document/9533990 - In the paper they add task-specific parameters to the original model, hence, they change the baseline BERT - "One possible solution is to add the task-specific, randomly initialized BERT_LAYERS at the top of the model." - This is an interesting approach - However, it increases the parameters drastically - "We could prune the number of parameters in this setting, by adding only the multi-head self-attention layer, without the position-wise feed-forward network." - This would also be an interesting approach to investigate - Jia, Qinjin et al. ALL-IN-ONE: Multi-Task Learning BERT models for Evaluating Peer Assessments. ArXiV abs/2110.03895. 2021.: https://arxiv.org/abs/2110.03895 - The authors compared single-task fine-tuned models (BERT and DistiLBERT) with multitask models - They added one Dense layer on top of the base model for single-task, and three Dense layers for multitask - They did not fix the base model's weights though, instead they fine-tuned it on multiple tasks, adding up the cross-entropy for each task to create the loss function - El Mekki et al. BERT-based Multi-Task Model for Country and Province Level MSA and Dialectal Arabic Identification. WANLP. 2021.: https://aclanthology.org/2021.wanlp-1.31/ - The authors use a BERT (MARBERT), task specific attention layers and then classifiers to train the network - They do not fix the weights of the BERT model either - Jia et al. Large-scale Transfer Learning for Low-resource Spoken Language Understanding. ArXiV abs/2008.05671. 2020.: https://arxiv.org/abs/2008.05671 - This paper deals with Spoken Language Understanding (SLU) - The authors test an architecture, where they fine-tune the BERT model and one where they fix the weights and add a specific head on top - They conclude: "Results in Table 4 indicate that both strategies have abilities of improving the performance of SLU model."