Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,214 @@
+---
+language:
+- en
+base_model:
+- FacebookAI/xlm-roberta-large
+pipeline_tag: token-classification
+library_name: transformers
+---
+
+# Patent Entity Extraction Model
+
+### Model Description
+
+**patent_entities_ner** is a fine-tuned [XLM-RoBERTa-large](https://huggingface.co/FacebookAI/xlm-roberta-large) model that has been trained on a custom dataset of OCR'd front pages of patent specifications published by the British Patent Office, and filed between 1617-1899.
+
+It has been trained to recognize six classes of named entities: 
+
+- PER: full name of inventor 
+- OCC: occupation of inventor 
+- ADD: full (permanent) address of inventor 
+- DATE: patent filing, submission, or approval dates
+- FIRM: name of firm affiliated with inventor 
+- COMM: name and information mentioned about communicant
+
+We take the original xlm-roberta-large [weights](https://huggingface.co/FacebookAI/xlm-roberta-large/blob/main/pytorch_model.bin) and fine tune on our custom dataset for 29 epochs with a learning rate of 5e-05 and a batch size of 21. We chose the learning rate by tuning on the validation set.
+
+### Usage
+
+This model can be used with HuggingFace Transformer's Pipelines API for NER: 
+
+```python
+from transformers import pipeline, AutoModelForTokenClassification, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("gbpatentdata/patent_entities_ner")
+model = AutoModelForTokenClassification.from_pretrained("gbpatentdata/patent_entities_ner")
+
+
+def custom_recognizer(text, model=model, tokenizer=tokenizer, device=0):
+
+    # HF ner pipeline
+    token_level_results = pipeline("ner", model=model, device=0, tokenizer=tokenizer)(text)
+
+    # keep entities tracked
+    entities = []
+    current_entity = None
+
+    for item in token_level_results:
+
+        tag = item['entity']
+
+        # replace '▁' with space for easier reading (_ is created by the XLM-RoBERTa tokenizer)
+        word = item['word'].replace('▁', ' ')
+
+        # aggregate I-O-B tagged entities
+        if tag.startswith('B-'):
+
+            if current_entity:
+                entities.append(current_entity)
+
+            current_entity = {'type': tag[2:], 'text': word.strip(), 'start': item['start'], 'end': item['end']}
+
+        elif tag.startswith('I-'):
+
+            if current_entity and tag[2:] == current_entity['type']:
+                current_entity['text'] += word
+                current_entity['end'] = item['end']
+
+            else:
+
+                if current_entity:
+                    entities.append(current_entity)
+
+                current_entity = {'type': tag[2:], 'text': word.strip(), 'start': item['start'], 'end': item['end']}
+
+        else:
+            # deal with O tag
+            if current_entity:
+                entities.append(current_entity)
+            current_entity = None
+
+    if current_entity:
+        # add to entities
+        entities.append(current_entity)
+
+    # track entity merges
+    merged_entities = []
+
+    # merge entities of the same type
+    for entity in entities:
+        if merged_entities and merged_entities[-1]['type'] == entity['type'] and merged_entities[-1]['end'] == entity['start']:
+            merged_entities[-1]['text'] += entity['text']
+            merged_entities[-1]['end'] = entity['end']
+        else:
+            merged_entities.append(entity)
+
+    # clean up extra spaces
+    for entity in merged_entities:
+        entity['text'] = ' '.join(entity['text'].split())
+
+    # convert to list of dicts
+    return [{'class': entity['type'],
+             'entity_text': entity['text'],
+             'start': entity['start'],
+             'end': entity['end']} for entity in merged_entities]
+
+
+
+example = """
+Date of Application, 1st Aug., 1890-Accepted, 6th Sept., 1890
+COMPLETE SPECIFICATION.
+Improvements in Coin-freed Apparatus for the Sale of Goods.
+I, CHARLES LOTINGA, of 33 Cambridge Street, Lower Grange, Cardiff, in the County of Glamorgan, Gentleman,
+do hereby declare the nature of this invention and in what manner the same is to be performed,
+to be particularly described and ascertained in and by the following statement
+"""
+
+ner_results = custom_recognizer(example)
+print(ner_results)
+```
+
+### Training Data
+
+The custom dataset of front page texts of patent specifications was assembled in the following steps:
+
+1. We fine tuned a YOLO vision [model](https://huggingface.co/gbpatentdata/yolov8_patent_layouts) to detect bounding boxes around text. We use this to identify text regions on the front pages of patent specifications.
+2. We use [Google Cloud Vision](https://cloud.google.com/vision?hl=en) to OCR the detected text regions, and then concatenate the OCR text.
+3. We randomly sample 200 front page texts (and another 201 oversampled from those that contain either firm or communicant information).
+
+Our custom dataset has accurate manual labels created jointly by an undergraduate student and an economics professor. The final dataset is split 60-20-20 (train-val-test). In the event that the front page text is too long, we restrict the text to the first 512 tokens.
+
+### Evaluation 
+
+Our evaluation metric is F1 at the full entity-level. That is, we aggregated adjacent-indexed entities into full entities and computed F1 scores requiring an exact match. These scores for the test set are below.
+
+<table>
+  <thead>
+    <tr>
+      <th>Full Entity</th>
+      <th>Precision</th>
+      <th>Recall</th>
+      <th>F1-Score</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>PER</td>
+      <td>92.2%</td>
+      <td>97.7%</td>
+      <td>94.9%</td>
+    </tr>
+    <tr>
+      <td>OCC</td>
+      <td>93.8%</td>
+      <td>93.8%</td>
+      <td>93.8%</td>
+    </tr>
+    <tr>
+      <td>ADD</td>
+      <td>88.6%</td>
+      <td>91.2%</td>
+      <td>89.9%</td>
+    </tr>
+    <tr>
+      <td>DATE</td>
+      <td>93.7%</td>
+      <td>98.7%</td>
+      <td>96.1%</td>
+    </tr>
+    <tr>
+      <td>FIRM</td>
+      <td>64.0%</td>
+      <td>94.1%</td>
+      <td>76.2%</td>
+    </tr>
+    <tr>
+      <td>COMM</td>
+      <td>77.1%</td>
+      <td>87.1%</td>
+      <td>81.8%</td>
+    </tr>
+    <tr>
+      <td>Overall (micro avg)</td>
+      <td>89.9%</td>
+      <td>95.3%</td>
+      <td>92.5%</td>
+    </tr>
+    <tr>
+      <td>Overall (macro avg)</td>
+      <td>84.9%</td>
+      <td>93.8%</td>
+      <td>88.9%</td>
+    </tr>
+    <tr>
+      <td>Overall (weighted avg)</td>
+      <td>90.3%</td>
+      <td>95.3%</td>
+      <td>92.7%</td>
+    </tr>
+  </tbody>
+</table>
+
+## Citation 
+
+If you use our model or custom training/evaluation data in your research, please cite our accompanying paper as follows:
+
+```bibtex
+@article{bct2025,
+  title = {300 Years of British Patents},
+  author = {Enrico Berkes and Matthew Lee Chen and Matteo Tranchero},
+  journal = {arXiv preprint arXiv:2401.12345},
+  year = {2025},
+  url = {https://arxiv.org/abs/2401.12345}
+}
+```

classification_report_lr_5.0000000000e-05_test.csv

@@ -0,0 +1,10 @@
+Entity,Precision,Recall,F1-Score,Support
+PER,0.9220,0.9774,0.9489,133
+FIRM,0.6400,0.9412,0.7619,17
+COMM,0.7714,0.8710,0.8182,31
+DATE,0.9367,0.9867,0.9610,150
+ADD,0.8857,0.9118,0.8986,102
+OCC,0.9383,0.9383,0.9383,81
+micro avg,0.8991,0.9533,0.9254,514
+macro avg,0.8490,0.9377,0.8878,514
+weighted avg,0.9032,0.9533,0.9267,514

classification_report_lr_5.0000000000e-05_val.csv

@@ -0,0 +1,10 @@
+Entity,Precision,Recall,F1-Score,Support
+OCC,0.9222,0.9540,0.9379,87
+PER,0.9530,0.9793,0.9660,145
+DATE,0.9732,1.0000,0.9864,145
+ADD,0.8785,0.9216,0.8995,102
+COMM,0.9062,0.9355,0.9206,31
+FIRM,0.7368,0.8750,0.8000,16
+micro avg,0.9286,0.9639,0.9459,526
+macro avg,0.8950,0.9442,0.9184,526
+weighted avg,0.9297,0.9639,0.9463,526

config.json

@@ -0,0 +1,58 @@
+{
+  "_name_or_path": "xlm-roberta-large",
+  "architectures": [
+    "XLMRobertaForTokenClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "id2label": {
+    "0": "B-ADD",
+    "1": "B-COMM",
+    "2": "B-DATE",
+    "3": "B-FIRM",
+    "4": "B-OCC",
+    "5": "B-PER",
+    "6": "I-ADD",
+    "7": "I-COMM",
+    "8": "I-DATE",
+    "9": "I-FIRM",
+    "10": "I-OCC",
+    "11": "I-PER",
+    "12": "O"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "label2id": {
+    "B-ADD": 0,
+    "B-COMM": 1,
+    "B-DATE": 2,
+    "B-FIRM": 3,
+    "B-OCC": 4,
+    "B-PER": 5,
+    "I-ADD": 6,
+    "I-COMM": 7,
+    "I-DATE": 8,
+    "I-FIRM": 9,
+    "I-OCC": 10,
+    "I-PER": 11,
+    "O": 12
+  },
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "xlm-roberta",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.44.2",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 250002
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23a649001c7829253d55b444ef96efb2748c2c1d6c9643972857373bf4a924a4
+size 2235465156

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
+size 5069051

special_tokens_map.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "unk_token": "<unk>"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ffb37461c391f096759f4a9bbbc329da0f36952f88bab061fcf84940c022e98
+size 17082999

tokenizer_config.json

@@ -0,0 +1,54 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "250001": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "unk_token": "<unk>"
+}