update model card

README.md

@@ -8,11 +8,11 @@ tags:
 
 ---
 
-# {MODEL_NAME}
+# Bert-MLM_arXiv-MP-class_zbMath
 
 This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
 
-<!--- Describe your model here -->
+The model is specifically designed to compute similarities of short mathematical texts.
 
 ## Usage (Sentence-Transformers)
 
@@ -26,9 +26,10 @@ Then you can use the model like this:
 
 ```python
 from sentence_transformers import SentenceTransformer
-sentences = ["This is an example sentence", "Each sentence is converted"]
+sentences = ["In this paper we show how to compute the $\\Lambda_{\\alpha}$ norm, $\\alpha\\ge 0$, using the dyadic grid. This result is a consequence of the description of the Hardy spaces $H^p(R^N)$ in terms of dyadic and special atoms.",
+             "We show that a determinant of Stirling cycle numbers counts unlabeled acyclic single-source automata. The proof involves a bijection from these automata to certain marked lattice paths and a sign-reversing involution to evaluate the determinant."]
 
-model = SentenceTransformer('{MODEL_NAME}')
+model = SentenceTransformer('math-similarity/Bert-MLM_arXiv-MP-class_zbMath')
 embeddings = model.encode(sentences)
 print(embeddings)
 ```
@@ -51,11 +52,12 @@ def mean_pooling(model_output, attention_mask):
 
 
 # Sentences we want sentence embeddings for
-sentences = ['This is an example sentence', 'Each sentence is converted']
+sentences = ["In this paper we show how to compute the $\\Lambda_{\\alpha}$ norm, $\\alpha\\ge 0$, using the dyadic grid. This result is a consequence of the description of the Hardy spaces $H^p(R^N)$ in terms of dyadic and special atoms.",
+             "We show that a determinant of Stirling cycle numbers counts unlabeled acyclic single-source automata. The proof involves a bijection from these automata to certain marked lattice paths and a sign-reversing involution to evaluate the determinant."]
 
 # Load model from HuggingFace Hub
-tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
-model = AutoModel.from_pretrained('{MODEL_NAME}')
+tokenizer = AutoTokenizer.from_pretrained('math-similarity/Bert-MLM_arXiv-MP-class_zbMath')
+model = AutoModel.from_pretrained('math-similarity/Bert-MLM_arXiv-MP-class_zbMath')
 
 # Tokenize sentences
 encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
@@ -71,56 +73,35 @@ print("Sentence embeddings:")
 print(sentence_embeddings)
 ```
 
+---------
 
+## Background
 
-## Evaluation Results
+## Intended uses
 
-<!--- Describe how your model was evaluated -->
+Our model is intended to be used as a sentence and short paragraph encoder for mathematical texts. Given an input text, it outputs a vector which captures the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.
 
-For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})
+By default, input text longer than 256 word pieces is truncated.
 
+## Training procedure
 
-## Training
-The model was trained with the parameters:
+### Domain-adaption
 
-**DataLoader**:
+We use the domain-adapted [math-similarity/Bert-MLM_arXiv](https://huggingface.co/math-similarity/Bert-MLM_arXiv) model. Please refer to the model card for more detailed information about the domain-adaption procedure.
 
-`torch.utils.data.dataloader.DataLoader` of length 21967 with parameters:
-```
-{'batch_size': 16, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
-```
+### Pooling
 
-**Loss**:
+We add a mean-pooling layer on top of the domain-adapted model.
 
-`sentence_transformers.losses.CosineSimilarityLoss.CosineSimilarityLoss` 
+### Fine-tuning
 
-Parameters of the fit()-Method:
-```
-{
-    "epochs": 10,
-    "evaluation_steps": 0,
-    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
-    "max_grad_norm": 1,
-    "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
-    "optimizer_params": {
-        "lr": 2e-05
-    },
-    "scheduler": "WarmupLinear",
-    "steps_per_epoch": null,
-    "warmup_steps": 10000,
-    "weight_decay": 0.01
-}
-```
+We fine-tune the model using a cosine-similarity objective. Formally, it computes the vectors `u = model(sentence_A)` and `v = model(sentence_B)` and measures the cosine-similarity between the two. By default, it minimizes the following loss: `||input_label - cos_score_transformation(cosine_sim(u,v))||_2`, with MSE as loss function.
 
+We use title-pairs from [zbMath](https://zbmath.org) as fine-tuning dataset and model semantic similarity with their MSC codes. Two titles are defined as similar, if they share their primary MSC<sub>5</sub> and another secondary MSC<sub>5</sub>. Otherwise, they are defined as semantically dissimilar.
+The training set contains 351.472 title pairs and the evaluation set contains 43.935 pairs. See the [training notebook](https://github.com/math-collab/text-similarity/blob/main/Bert-MLM%20%2B%20mean%20pooling%20%2B%20fine-tune%20zbMath-class.ipynb) for more information.
 
-## Full Model Architecture
-```
-SentenceTransformer(
-  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
-  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
-)
-```
+Unfortunately, we cannot include a dataset with titles due to licensing issues. However, we have created a dataset than only contains the respective zbMath identifiers (also known as an) with primary and secondary MSC classification but without titles. It is available as [datasets/math-similarity/class-zbmath-identifier](https://huggingface.co/datasets/math-similarity/class-zbmath-identifier).
 
 ## Citing & Authors
 
-<!--- Describe where people can find more information -->
+This model is an additional resource for the [CICM'24](https://cicm-conference.org/2024/cicm.php) submission *On modelling similarity of short mathematical texts*.