| # Model Card for BERT Slot Filling Model | |
| This BERT-based model is designed | |
| for slot filling tasks in natural language sentences, | |
| ideal for extracting specific information in applications like chatbots and virtual assistants. | |
| For example: | |
| input: Transfer $500 from checking to student savings | |
| output: transfer **[$500:B-amount]** from **[checking:B-account-from]** to **[student:B-account-to]** **[savings:I-account-to]** | |
| The model was trained with dataset https://github.com/SunLemuria/JointBERT-Tensorflow1/blob/master/data/ | |
| Please make yourself familiar with BERT: BERT for Joint Intent Classification and Slot Filling https://arxiv.org/pdf/1902.10909.pdf | |
| ## Model Details | |
| <table> | |
| <tr> | |
| <th>Tag</th> | |
| <th>Definition</th> | |
| </tr> | |
| <tr> | |
| <td>B-account-from</td> | |
| <td>Start of the source account in a transaction.</td> | |
| </tr> | |
| <tr> | |
| <td>I-account-from</td> | |
| <td>Complement of the source account in a transaction.</td> | |
| </tr> | |
| <tr> | |
| <td>B-account-to</td> | |
| <td>Start of the target account in a transaction.</td> | |
| </tr> | |
| <tr> | |
| <td>I-account-to</td> | |
| <td>Complement of the target account in a transaction.</td> | |
| </tr> | |
| <tr> | |
| <td>B-bill_type</td> | |
| <td>Start of the type of bill or service.</td> | |
| </tr> | |
| <tr> | |
| <td>I-bill_type</td> | |
| <td>Complement of the type of bill or service.</td> | |
| </tr> | |
| <tr> | |
| <td>B-transaction-from</td> | |
| <td>Start of the origin of a transaction or fraud.</td> | |
| </tr> | |
| <tr> | |
| <td>I-transaction-from</td> | |
| <td>Complement of the origin of a transaction or fraud.</td> | |
| </tr> | |
| <tr> | |
| <td>B-transaction-to</td> | |
| <td>Start of the destination or end of a transaction or fraud.</td> | |
| </tr> | |
| <tr> | |
| <td>I-transaction-to</td> | |
| <td>Complement of the destination of a transaction or fraud.</td> | |
| </tr> | |
| <tr> | |
| <td>B-amount</td> | |
| <td>Start of a specified amount of money.</td> | |
| </tr> | |
| <tr> | |
| <td>I-amount</td> | |
| <td>Complement of a specified amount of money.</td> | |
| </tr> | |
| <tr> | |
| <td>B-timeRange</td> | |
| <td>Start of a specific time range or date.</td> | |
| </tr> | |
| <tr> | |
| <td>I-timeRange</td> | |
| <td>Complement of a specific time range or date.</td> | |
| </tr> | |
| </table> | |
| ### Model Description | |
| - **Developed by:** Andy González | |
| - **Model type:** Token Classification | |
| - **Language(s) (NLP):** English | |
| - **Finetuned from model [optional]:** bert-base-uncased | |
| ## How to Get Started with the Model | |
| ```python | |
| !pip install torch transformers | |
| import os | |
| import requests | |
| import torch | |
| from transformers import BertForTokenClassification, BertTokenizerFast | |
| # URL y archivo para los slots | |
| slots_url = 'https://huggingface.co/andgonzalez/bert-uncased-slot-filling/raw/main/slots.txt' | |
| slots_file = 'slots.txt' | |
| device = "cpu" | |
| # Descargar y guardar los slots si no existen | |
| if not os.path.exists(slots_file): | |
| response = requests.get(slots_url) | |
| response.raise_for_status() | |
| with open(slots_file, 'w') as file: | |
| file.write(response.text) | |
| # Leer los slots | |
| with open(slots_file, 'r') as file: | |
| slot_labels = file.read().splitlines() | |
| # Cargar el tokenizador y el modelo | |
| tokenizer = BertTokenizerFast.from_pretrained('andgonzalez/bert-uncased-slot-filling') | |
| model = BertForTokenClassification.from_pretrained('andgonzalez/bert-uncased-slot-filling') | |
| # Ejemplo | |
| sentence = "Transfer $500 from checking to student savings" | |
| inputs = tokenizer(sentence, truncation=True, padding='max_length', max_length=20, return_tensors="pt") | |
| inputs = {k: v.to(device) for k, v in inputs.items()} | |
| with torch.no_grad(): | |
| model.eval() | |
| outputs = model(**inputs) | |
| # Procesar los logits para obtener predicciones | |
| logits = outputs.logits | |
| predictions = torch.argmax(logits, dim=2).squeeze().cpu().numpy() | |
| words = tokenizer.convert_ids_to_tokens(inputs["input_ids"].squeeze().cpu().numpy()) | |
| # Inicializar skip_next | |
| skip_next = False | |
| # Formatear la oracion | |
| formatted_sentence = [] | |
| for i, (word, pred) in enumerate(zip(words, predictions)): | |
| if word not in ['[PAD]', '[SEP]', '[CLS]']: | |
| label = slot_labels[pred] | |
| if word == "$" and i + 1 < len(words) and words[i + 1].replace("##", "").isdigit(): | |
| next_word = words[i + 1].replace("##", "") | |
| combined_word = word + next_word | |
| formatted_word = f'[{combined_word}:{label}]' | |
| formatted_sentence.append(formatted_word) | |
| skip_next = True | |
| elif skip_next: | |
| skip_next = False | |
| continue | |
| elif not word.startswith("##"): | |
| if label != 'O': | |
| formatted_word = f'[{word}:{label}]' | |
| else: | |
| formatted_word = word | |
| formatted_sentence.append(formatted_word) | |
| formatted_sentence = ' '.join(formatted_sentence) | |
| print(formatted_sentence) | |
| ``` | |
| ## Training Details | |
| #### Metrics | |
| - Metrics used: Precision, Recall, F1-Score | |
| ### Results | |
| - **Epoch 1:** Loss: 1.3253, Precision: 0.5862, Recall: 0.5758, F1-Score: 0.5633 | |
| - **Epoch 2:** Loss: 0.3507, Precision: 0.7491, Recall: 0.7476, F1-Score: 0.7374 | |
| - **Epoch 3:** Loss: 0.2156, Precision: 0.8180, Recall: 0.8138, F1-Score: 0.8007 | |
| - **Epoch 4:** Loss: 0.1593, Precision: 0.8252, Recall: 0.8274, F1-Score: 0.8173 | |
| - **Epoch 5:** Loss: 0.1236, Precision: 0.8613, Recall: 0.8549, F1-Score: 0.8466 | |
| - **Epoch 6:** Loss: 0.0961, Precision: 0.8839, Recall: 0.8810, F1-Score: 0.8786 | |
| - **Epoch 7:** Loss: 0.0787, Precision: 0.8795, Recall: 0.8917, F1-Score: 0.8808 | |
| - **Epoch 8:** Loss: 0.0644, Precision: 0.8956, Recall: 0.8958, F1-Score: 0.8911 | |
| - **Epoch 9:** Loss: 0.0542, Precision: 0.8889, Recall: 0.9012, F1-Score: 0.8913 | |
| - **Epoch 10:** Loss: 0.0468, Precision: 0.8980, Recall: 0.9007, F1-Score: 0.8935 | |
| - **Best Model:** Epoch 8, Test Loss: 0.1588 | |
| ### Plots | |
|  | |