|
---
|
|
language:
|
|
- en
|
|
license: apache-2.0
|
|
tags:
|
|
- sentence-transformers
|
|
- sentence-similarity
|
|
- feature-extraction
|
|
- generated_from_trainer
|
|
- dataset_size:6300
|
|
- loss:MatryoshkaLoss
|
|
- loss:MultipleNegativesRankingLoss
|
|
base_model: intfloat/e5-large-unsupervised
|
|
widget:
|
|
- source_sentence: What are the key components of the transparency provisions included
|
|
in the Consolidated Appropriations Act of 2021 regarding healthcare?
|
|
sentences:
|
|
- The report includes information on legal proceedings under 'Note 13 — Commitments
|
|
and Contingencies — Litigation and Other Legal Matters' which is a part of the
|
|
consolidated financial statements
|
|
- The Consolidated Appropriations Act of 2021 was signed into law in December 2020
|
|
and contains further transparency provisions requiring group health plans and
|
|
health insurance issuers to report certain prescription drug costs, overall spending
|
|
on health services and prescription drugs, and information about premiums and
|
|
the impact of rebates and other remuneration on premiums and out-of-pocket costs
|
|
to the Tri-Departments.
|
|
- In 2023, the company recorded other operating charges of $1,951 million.
|
|
- source_sentence: What technology does the Tax Advisor use and for what purpose in
|
|
Intuit's offerings?
|
|
sentences:
|
|
- In 2023, Goldman Sachs' investments in funds at NAV primarily included firm-sponsored
|
|
private equity, credit, real estate, and hedge funds. These funds are involved
|
|
in various types of investments such as leveraged buyouts, recapitalizations,
|
|
growth investments, and distressed investments for private equity, while credit
|
|
funds are focused on providing private high-yield capital for leveraged and management
|
|
buyout transactions. Real estate funds invest globally in real estate assets,
|
|
and hedge funds adopt a fundamental bottom-up investment approach.
|
|
- Using AI technologies, our Tax Advisor offering leverages information generated
|
|
from our ProConnect Tax Online and Lacerte offerings to enable year-round tax
|
|
planning services and communicate tax savings strategies to clients.
|
|
- '''Note 13 — Commitments and Contingencies'' provides details about litigation
|
|
and other legal matters in an Annual Report on Form 10-K.'
|
|
- source_sentence: What was the net revenue for the Data Center segment in 2023?
|
|
sentences:
|
|
- Data Center net revenue of $6.5 billion in 2023 increased by 7%, compared to net
|
|
revenue of $6.0 billion in 2022.
|
|
- Under its Class 2 insurance license, Caterpillar Insurance Co. Ltd. insures its
|
|
parent and affiliates for general liability, property, auto liability and cargo.
|
|
It also provides reinsurance to CaterThe pillar Insurance Company under a quota
|
|
share reinsurance agreement for its contractual liability and contractors’ equipment
|
|
programs in the United States.
|
|
- Schwab’s funding of these remaining commitments is dependent upon the occurrence
|
|
of certain conditions, and Schwab expects to pay substantially all of these commitments
|
|
between 2024 and 2027.
|
|
- source_sentence: What are the three principles of liquidity risk management at Goldman
|
|
Sachs?
|
|
sentences:
|
|
- The Company determines if an arrangement is a lease at inception and classifies
|
|
its leases at commencement. Operating leases are included in operating lease right-of-use
|
|
("ROU") assets and current and noncurrent operating lease liabilities on the Company’s
|
|
consolidated balance sheets.
|
|
- Garmin Ltd. reported a net income of $1,289,636 for the fiscal year ended December
|
|
30, 2023.
|
|
- 'Goldman Sachs manages liquidity risk based on three principles: 1) hold sufficient
|
|
excess liquidity in the form of GCLA to cover outflows during a stressed period,
|
|
2) maintain appropriate Asset-Liability Management, and 3) maintain a viable Contingency
|
|
Funding Plan.'
|
|
- source_sentence: What was the total cost and expenses reported by Berkshire Hathaway
|
|
for the year ended December 31, 2023?
|
|
sentences:
|
|
- Total costs and expenses | | 321,144 | | | 266,484 | | | 243,752
|
|
- Qulipta (atogepant) is a calcitonin gene-related peptide receptor antagonist indicated
|
|
for the preventive treatment of episodic and chronic migraine in adults. Qulipta
|
|
is commercialized in the United States and Canada and is approved in the European
|
|
Union under the brand name Aquipta.
|
|
- Item 3 'Legal Proceedings' is integrated by reference to other parts including
|
|
Note 22 — 'Environmental and legal matters' and Part II, Item 8.
|
|
pipeline_tag: sentence-similarity
|
|
library_name: sentence-transformers
|
|
metrics:
|
|
- cosine_accuracy@1
|
|
- cosine_accuracy@3
|
|
- cosine_accuracy@5
|
|
- cosine_accuracy@10
|
|
- cosine_precision@1
|
|
- cosine_precision@3
|
|
- cosine_precision@5
|
|
- cosine_precision@10
|
|
- cosine_recall@1
|
|
- cosine_recall@3
|
|
- cosine_recall@5
|
|
- cosine_recall@10
|
|
- cosine_ndcg@10
|
|
- cosine_mrr@10
|
|
- cosine_map@100
|
|
model-index:
|
|
- name: E5 unsupervised Financial Matryoshka
|
|
results:
|
|
- task:
|
|
type: information-retrieval
|
|
name: Information Retrieval
|
|
dataset:
|
|
name: dim 768
|
|
type: dim_768
|
|
metrics:
|
|
- type: cosine_accuracy@1
|
|
value: 0.7271428571428571
|
|
name: Cosine Accuracy@1
|
|
- type: cosine_accuracy@3
|
|
value: 0.85
|
|
name: Cosine Accuracy@3
|
|
- type: cosine_accuracy@5
|
|
value: 0.8785714285714286
|
|
name: Cosine Accuracy@5
|
|
- type: cosine_accuracy@10
|
|
value: 0.9114285714285715
|
|
name: Cosine Accuracy@10
|
|
- type: cosine_precision@1
|
|
value: 0.7271428571428571
|
|
name: Cosine Precision@1
|
|
- type: cosine_precision@3
|
|
value: 0.2833333333333333
|
|
name: Cosine Precision@3
|
|
- type: cosine_precision@5
|
|
value: 0.17571428571428568
|
|
name: Cosine Precision@5
|
|
- type: cosine_precision@10
|
|
value: 0.09114285714285714
|
|
name: Cosine Precision@10
|
|
- type: cosine_recall@1
|
|
value: 0.7271428571428571
|
|
name: Cosine Recall@1
|
|
- type: cosine_recall@3
|
|
value: 0.85
|
|
name: Cosine Recall@3
|
|
- type: cosine_recall@5
|
|
value: 0.8785714285714286
|
|
name: Cosine Recall@5
|
|
- type: cosine_recall@10
|
|
value: 0.9114285714285715
|
|
name: Cosine Recall@10
|
|
- type: cosine_ndcg@10
|
|
value: 0.822517236613446
|
|
name: Cosine Ndcg@10
|
|
- type: cosine_mrr@10
|
|
value: 0.7936921768707483
|
|
name: Cosine Mrr@10
|
|
- type: cosine_map@100
|
|
value: 0.7973883589026711
|
|
name: Cosine Map@100
|
|
- task:
|
|
type: information-retrieval
|
|
name: Information Retrieval
|
|
dataset:
|
|
name: dim 512
|
|
type: dim_512
|
|
metrics:
|
|
- type: cosine_accuracy@1
|
|
value: 0.7271428571428571
|
|
name: Cosine Accuracy@1
|
|
- type: cosine_accuracy@3
|
|
value: 0.8457142857142858
|
|
name: Cosine Accuracy@3
|
|
- type: cosine_accuracy@5
|
|
value: 0.88
|
|
name: Cosine Accuracy@5
|
|
- type: cosine_accuracy@10
|
|
value: 0.9128571428571428
|
|
name: Cosine Accuracy@10
|
|
- type: cosine_precision@1
|
|
value: 0.7271428571428571
|
|
name: Cosine Precision@1
|
|
- type: cosine_precision@3
|
|
value: 0.28190476190476194
|
|
name: Cosine Precision@3
|
|
- type: cosine_precision@5
|
|
value: 0.176
|
|
name: Cosine Precision@5
|
|
- type: cosine_precision@10
|
|
value: 0.09128571428571429
|
|
name: Cosine Precision@10
|
|
- type: cosine_recall@1
|
|
value: 0.7271428571428571
|
|
name: Cosine Recall@1
|
|
- type: cosine_recall@3
|
|
value: 0.8457142857142858
|
|
name: Cosine Recall@3
|
|
- type: cosine_recall@5
|
|
value: 0.88
|
|
name: Cosine Recall@5
|
|
- type: cosine_recall@10
|
|
value: 0.9128571428571428
|
|
name: Cosine Recall@10
|
|
- type: cosine_ndcg@10
|
|
value: 0.8223709830528422
|
|
name: Cosine Ndcg@10
|
|
- type: cosine_mrr@10
|
|
value: 0.793145691609977
|
|
name: Cosine Mrr@10
|
|
- type: cosine_map@100
|
|
value: 0.7966990460475021
|
|
name: Cosine Map@100
|
|
- task:
|
|
type: information-retrieval
|
|
name: Information Retrieval
|
|
dataset:
|
|
name: dim 256
|
|
type: dim_256
|
|
metrics:
|
|
- type: cosine_accuracy@1
|
|
value: 0.72
|
|
name: Cosine Accuracy@1
|
|
- type: cosine_accuracy@3
|
|
value: 0.8457142857142858
|
|
name: Cosine Accuracy@3
|
|
- type: cosine_accuracy@5
|
|
value: 0.8714285714285714
|
|
name: Cosine Accuracy@5
|
|
- type: cosine_accuracy@10
|
|
value: 0.9057142857142857
|
|
name: Cosine Accuracy@10
|
|
- type: cosine_precision@1
|
|
value: 0.72
|
|
name: Cosine Precision@1
|
|
- type: cosine_precision@3
|
|
value: 0.28190476190476194
|
|
name: Cosine Precision@3
|
|
- type: cosine_precision@5
|
|
value: 0.17428571428571424
|
|
name: Cosine Precision@5
|
|
- type: cosine_precision@10
|
|
value: 0.09057142857142855
|
|
name: Cosine Precision@10
|
|
- type: cosine_recall@1
|
|
value: 0.72
|
|
name: Cosine Recall@1
|
|
- type: cosine_recall@3
|
|
value: 0.8457142857142858
|
|
name: Cosine Recall@3
|
|
- type: cosine_recall@5
|
|
value: 0.8714285714285714
|
|
name: Cosine Recall@5
|
|
- type: cosine_recall@10
|
|
value: 0.9057142857142857
|
|
name: Cosine Recall@10
|
|
- type: cosine_ndcg@10
|
|
value: 0.8159991941699124
|
|
name: Cosine Ndcg@10
|
|
- type: cosine_mrr@10
|
|
value: 0.7869370748299319
|
|
name: Cosine Mrr@10
|
|
- type: cosine_map@100
|
|
value: 0.7906967878713818
|
|
name: Cosine Map@100
|
|
- task:
|
|
type: information-retrieval
|
|
name: Information Retrieval
|
|
dataset:
|
|
name: dim 128
|
|
type: dim_128
|
|
metrics:
|
|
- type: cosine_accuracy@1
|
|
value: 0.7085714285714285
|
|
name: Cosine Accuracy@1
|
|
- type: cosine_accuracy@3
|
|
value: 0.8285714285714286
|
|
name: Cosine Accuracy@3
|
|
- type: cosine_accuracy@5
|
|
value: 0.8728571428571429
|
|
name: Cosine Accuracy@5
|
|
- type: cosine_accuracy@10
|
|
value: 0.8985714285714286
|
|
name: Cosine Accuracy@10
|
|
- type: cosine_precision@1
|
|
value: 0.7085714285714285
|
|
name: Cosine Precision@1
|
|
- type: cosine_precision@3
|
|
value: 0.2761904761904762
|
|
name: Cosine Precision@3
|
|
- type: cosine_precision@5
|
|
value: 0.17457142857142854
|
|
name: Cosine Precision@5
|
|
- type: cosine_precision@10
|
|
value: 0.08985714285714284
|
|
name: Cosine Precision@10
|
|
- type: cosine_recall@1
|
|
value: 0.7085714285714285
|
|
name: Cosine Recall@1
|
|
- type: cosine_recall@3
|
|
value: 0.8285714285714286
|
|
name: Cosine Recall@3
|
|
- type: cosine_recall@5
|
|
value: 0.8728571428571429
|
|
name: Cosine Recall@5
|
|
- type: cosine_recall@10
|
|
value: 0.8985714285714286
|
|
name: Cosine Recall@10
|
|
- type: cosine_ndcg@10
|
|
value: 0.8073517667504667
|
|
name: Cosine Ndcg@10
|
|
- type: cosine_mrr@10
|
|
value: 0.7777108843537414
|
|
name: Cosine Mrr@10
|
|
- type: cosine_map@100
|
|
value: 0.7815591417851651
|
|
name: Cosine Map@100
|
|
- task:
|
|
type: information-retrieval
|
|
name: Information Retrieval
|
|
dataset:
|
|
name: dim 64
|
|
type: dim_64
|
|
metrics:
|
|
- type: cosine_accuracy@1
|
|
value: 0.6757142857142857
|
|
name: Cosine Accuracy@1
|
|
- type: cosine_accuracy@3
|
|
value: 0.8185714285714286
|
|
name: Cosine Accuracy@3
|
|
- type: cosine_accuracy@5
|
|
value: 0.8457142857142858
|
|
name: Cosine Accuracy@5
|
|
- type: cosine_accuracy@10
|
|
value: 0.8842857142857142
|
|
name: Cosine Accuracy@10
|
|
- type: cosine_precision@1
|
|
value: 0.6757142857142857
|
|
name: Cosine Precision@1
|
|
- type: cosine_precision@3
|
|
value: 0.27285714285714285
|
|
name: Cosine Precision@3
|
|
- type: cosine_precision@5
|
|
value: 0.16914285714285712
|
|
name: Cosine Precision@5
|
|
- type: cosine_precision@10
|
|
value: 0.08842857142857141
|
|
name: Cosine Precision@10
|
|
- type: cosine_recall@1
|
|
value: 0.6757142857142857
|
|
name: Cosine Recall@1
|
|
- type: cosine_recall@3
|
|
value: 0.8185714285714286
|
|
name: Cosine Recall@3
|
|
- type: cosine_recall@5
|
|
value: 0.8457142857142858
|
|
name: Cosine Recall@5
|
|
- type: cosine_recall@10
|
|
value: 0.8842857142857142
|
|
name: Cosine Recall@10
|
|
- type: cosine_ndcg@10
|
|
value: 0.7861731335824387
|
|
name: Cosine Ndcg@10
|
|
- type: cosine_mrr@10
|
|
value: 0.7542681405895693
|
|
name: Cosine Mrr@10
|
|
- type: cosine_map@100
|
|
value: 0.7588497811523153
|
|
name: Cosine Map@100
|
|
---
|
|
|
|
# E5 unsupervised Financial Matryoshka
|
|
|
|
This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [intfloat/e5-large-unsupervised](https://huggingface.co/intfloat/e5-large-unsupervised) on the json dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
|
|
|
|
## Model Details
|
|
|
|
### Model Description
|
|
- **Model Type:** Sentence Transformer
|
|
- **Base model:** [intfloat/e5-large-unsupervised](https://huggingface.co/intfloat/e5-large-unsupervised) <!-- at revision 15af9288f69a6291f37bfb89b47e71abc747b206 -->
|
|
- **Maximum Sequence Length:** 512 tokens
|
|
- **Output Dimensionality:** 1024 dimensions
|
|
- **Similarity Function:** Cosine Similarity
|
|
- **Training Dataset:**
|
|
- json
|
|
- **Language:** en
|
|
- **License:** apache-2.0
|
|
|
|
### Model Sources
|
|
|
|
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
|
|
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
|
|
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
|
|
|
|
### Full Model Architecture
|
|
|
|
```
|
|
SentenceTransformer(
|
|
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
|
|
(1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
|
|
(2): Normalize()
|
|
)
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Direct Usage (Sentence Transformers)
|
|
|
|
First install the Sentence Transformers library:
|
|
|
|
```bash
|
|
pip install -U sentence-transformers
|
|
```
|
|
|
|
Then you can load this model and run inference.
|
|
```python
|
|
from sentence_transformers import SentenceTransformer
|
|
|
|
# Download from the 🤗 Hub
|
|
model = SentenceTransformer("schawla2/e5-unsupervised-financial-matryoshka")
|
|
# Run inference
|
|
sentences = [
|
|
'What was the total cost and expenses reported by Berkshire Hathaway for the year ended December 31, 2023?',
|
|
'Total costs and expenses | | 321,144 | | | 266,484 | | | 243,752',
|
|
'Qulipta (atogepant) is a calcitonin gene-related peptide receptor antagonist indicated for the preventive treatment of episodic and chronic migraine in adults. Qulipta is commercialized in the United States and Canada and is approved in the European Union under the brand name Aquipta.',
|
|
]
|
|
embeddings = model.encode(sentences)
|
|
print(embeddings.shape)
|
|
# [3, 1024]
|
|
|
|
# Get the similarity scores for the embeddings
|
|
similarities = model.similarity(embeddings, embeddings)
|
|
print(similarities.shape)
|
|
# [3, 3]
|
|
```
|
|
|
|
<!--
|
|
### Direct Usage (Transformers)
|
|
|
|
<details><summary>Click to see the direct usage in Transformers</summary>
|
|
|
|
</details>
|
|
-->
|
|
|
|
<!--
|
|
### Downstream Usage (Sentence Transformers)
|
|
|
|
You can finetune this model on your own dataset.
|
|
|
|
<details><summary>Click to expand</summary>
|
|
|
|
</details>
|
|
-->
|
|
|
|
<!--
|
|
### Out-of-Scope Use
|
|
|
|
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
|
|
-->
|
|
|
|
## Evaluation
|
|
|
|
### Metrics
|
|
|
|
#### Information Retrieval
|
|
|
|
* Datasets: `dim_768`, `dim_512`, `dim_256`, `dim_128` and `dim_64`
|
|
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
|
|
|
|
| Metric | dim_768 | dim_512 | dim_256 | dim_128 | dim_64 |
|
|
|:--------------------|:-----------|:-----------|:----------|:-----------|:-----------|
|
|
| cosine_accuracy@1 | 0.7271 | 0.7271 | 0.72 | 0.7086 | 0.6757 |
|
|
| cosine_accuracy@3 | 0.85 | 0.8457 | 0.8457 | 0.8286 | 0.8186 |
|
|
| cosine_accuracy@5 | 0.8786 | 0.88 | 0.8714 | 0.8729 | 0.8457 |
|
|
| cosine_accuracy@10 | 0.9114 | 0.9129 | 0.9057 | 0.8986 | 0.8843 |
|
|
| cosine_precision@1 | 0.7271 | 0.7271 | 0.72 | 0.7086 | 0.6757 |
|
|
| cosine_precision@3 | 0.2833 | 0.2819 | 0.2819 | 0.2762 | 0.2729 |
|
|
| cosine_precision@5 | 0.1757 | 0.176 | 0.1743 | 0.1746 | 0.1691 |
|
|
| cosine_precision@10 | 0.0911 | 0.0913 | 0.0906 | 0.0899 | 0.0884 |
|
|
| cosine_recall@1 | 0.7271 | 0.7271 | 0.72 | 0.7086 | 0.6757 |
|
|
| cosine_recall@3 | 0.85 | 0.8457 | 0.8457 | 0.8286 | 0.8186 |
|
|
| cosine_recall@5 | 0.8786 | 0.88 | 0.8714 | 0.8729 | 0.8457 |
|
|
| cosine_recall@10 | 0.9114 | 0.9129 | 0.9057 | 0.8986 | 0.8843 |
|
|
| **cosine_ndcg@10** | **0.8225** | **0.8224** | **0.816** | **0.8074** | **0.7862** |
|
|
| cosine_mrr@10 | 0.7937 | 0.7931 | 0.7869 | 0.7777 | 0.7543 |
|
|
| cosine_map@100 | 0.7974 | 0.7967 | 0.7907 | 0.7816 | 0.7588 |
|
|
|
|
<!--
|
|
## Bias, Risks and Limitations
|
|
|
|
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
|
|
-->
|
|
|
|
<!--
|
|
### Recommendations
|
|
|
|
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
|
|
-->
|
|
|
|
## Training Details
|
|
|
|
### Training Dataset
|
|
|
|
#### json
|
|
|
|
* Dataset: json
|
|
* Size: 6,300 training samples
|
|
* Columns: <code>anchor</code> and <code>positive</code>
|
|
* Approximate statistics based on the first 1000 samples:
|
|
| | anchor | positive |
|
|
|:--------|:---------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|
|
|
| type | string | string |
|
|
| details | <ul><li>min: 8 tokens</li><li>mean: 20.8 tokens</li><li>max: 51 tokens</li></ul> | <ul><li>min: 7 tokens</li><li>mean: 45.24 tokens</li><li>max: 326 tokens</li></ul> |
|
|
* Samples:
|
|
| anchor | positive |
|
|
|:--------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
| <code>How many full-time employees did Microsoft report as of June 30, 2023?</code> | <code>As of June 30, 2023, we employed approximately 221,000 people on a full-time basis, 120,000 in the U.S. and 101,000 internationally.</code> |
|
|
| <code>What was the total amount CSC paid for Series G preferred stock repurchases in 2023?</code> | <code>In 2023, CSC repurchased 42,036 depositary shares representing interests in Series G preferred stock for a total amount of $42 million.</code> |
|
|
| <code>What does Note 13 in the Annual Report on Form 10-K discuss?</code> | <code>For a discussion of legal and other proceedings in which we are involved, see Note 13 - Commitments and Contingencies in the Notes to Consolidated Financial Statements.</code> |
|
|
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
|
|
```json
|
|
{
|
|
"loss": "MultipleNegativesRankingLoss",
|
|
"matryoshka_dims": [
|
|
768,
|
|
512,
|
|
256,
|
|
128,
|
|
64
|
|
],
|
|
"matryoshka_weights": [
|
|
1,
|
|
1,
|
|
1,
|
|
1,
|
|
1
|
|
],
|
|
"n_dims_per_step": -1
|
|
}
|
|
```
|
|
|
|
### Training Hyperparameters
|
|
#### Non-Default Hyperparameters
|
|
|
|
- `eval_strategy`: epoch
|
|
- `per_device_eval_batch_size`: 16
|
|
- `gradient_accumulation_steps`: 16
|
|
- `learning_rate`: 2e-05
|
|
- `num_train_epochs`: 4
|
|
- `lr_scheduler_type`: cosine
|
|
- `warmup_ratio`: 0.1
|
|
- `bf16`: True
|
|
- `tf32`: True
|
|
- `load_best_model_at_end`: True
|
|
- `optim`: adamw_torch_fused
|
|
- `batch_sampler`: no_duplicates
|
|
|
|
#### All Hyperparameters
|
|
<details><summary>Click to expand</summary>
|
|
|
|
- `overwrite_output_dir`: False
|
|
- `do_predict`: False
|
|
- `eval_strategy`: epoch
|
|
- `prediction_loss_only`: True
|
|
- `per_device_train_batch_size`: 8
|
|
- `per_device_eval_batch_size`: 16
|
|
- `per_gpu_train_batch_size`: None
|
|
- `per_gpu_eval_batch_size`: None
|
|
- `gradient_accumulation_steps`: 16
|
|
- `eval_accumulation_steps`: None
|
|
- `torch_empty_cache_steps`: None
|
|
- `learning_rate`: 2e-05
|
|
- `weight_decay`: 0.0
|
|
- `adam_beta1`: 0.9
|
|
- `adam_beta2`: 0.999
|
|
- `adam_epsilon`: 1e-08
|
|
- `max_grad_norm`: 1.0
|
|
- `num_train_epochs`: 4
|
|
- `max_steps`: -1
|
|
- `lr_scheduler_type`: cosine
|
|
- `lr_scheduler_kwargs`: {}
|
|
- `warmup_ratio`: 0.1
|
|
- `warmup_steps`: 0
|
|
- `log_level`: passive
|
|
- `log_level_replica`: warning
|
|
- `log_on_each_node`: True
|
|
- `logging_nan_inf_filter`: True
|
|
- `save_safetensors`: True
|
|
- `save_on_each_node`: False
|
|
- `save_only_model`: False
|
|
- `restore_callback_states_from_checkpoint`: False
|
|
- `no_cuda`: False
|
|
- `use_cpu`: False
|
|
- `use_mps_device`: False
|
|
- `seed`: 42
|
|
- `data_seed`: None
|
|
- `jit_mode_eval`: False
|
|
- `use_ipex`: False
|
|
- `bf16`: True
|
|
- `fp16`: False
|
|
- `fp16_opt_level`: O1
|
|
- `half_precision_backend`: auto
|
|
- `bf16_full_eval`: False
|
|
- `fp16_full_eval`: False
|
|
- `tf32`: True
|
|
- `local_rank`: 0
|
|
- `ddp_backend`: None
|
|
- `tpu_num_cores`: None
|
|
- `tpu_metrics_debug`: False
|
|
- `debug`: []
|
|
- `dataloader_drop_last`: False
|
|
- `dataloader_num_workers`: 0
|
|
- `dataloader_prefetch_factor`: None
|
|
- `past_index`: -1
|
|
- `disable_tqdm`: False
|
|
- `remove_unused_columns`: True
|
|
- `label_names`: None
|
|
- `load_best_model_at_end`: True
|
|
- `ignore_data_skip`: False
|
|
- `fsdp`: []
|
|
- `fsdp_min_num_params`: 0
|
|
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
|
|
- `fsdp_transformer_layer_cls_to_wrap`: None
|
|
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
|
|
- `deepspeed`: None
|
|
- `label_smoothing_factor`: 0.0
|
|
- `optim`: adamw_torch_fused
|
|
- `optim_args`: None
|
|
- `adafactor`: False
|
|
- `group_by_length`: False
|
|
- `length_column_name`: length
|
|
- `ddp_find_unused_parameters`: None
|
|
- `ddp_bucket_cap_mb`: None
|
|
- `ddp_broadcast_buffers`: False
|
|
- `dataloader_pin_memory`: True
|
|
- `dataloader_persistent_workers`: False
|
|
- `skip_memory_metrics`: True
|
|
- `use_legacy_prediction_loop`: False
|
|
- `push_to_hub`: False
|
|
- `resume_from_checkpoint`: None
|
|
- `hub_model_id`: None
|
|
- `hub_strategy`: every_save
|
|
- `hub_private_repo`: None
|
|
- `hub_always_push`: False
|
|
- `gradient_checkpointing`: False
|
|
- `gradient_checkpointing_kwargs`: None
|
|
- `include_inputs_for_metrics`: False
|
|
- `include_for_metrics`: []
|
|
- `eval_do_concat_batches`: True
|
|
- `fp16_backend`: auto
|
|
- `push_to_hub_model_id`: None
|
|
- `push_to_hub_organization`: None
|
|
- `mp_parameters`:
|
|
- `auto_find_batch_size`: False
|
|
- `full_determinism`: False
|
|
- `torchdynamo`: None
|
|
- `ray_scope`: last
|
|
- `ddp_timeout`: 1800
|
|
- `torch_compile`: False
|
|
- `torch_compile_backend`: None
|
|
- `torch_compile_mode`: None
|
|
- `dispatch_batches`: None
|
|
- `split_batches`: None
|
|
- `include_tokens_per_second`: False
|
|
- `include_num_input_tokens_seen`: False
|
|
- `neftune_noise_alpha`: None
|
|
- `optim_target_modules`: None
|
|
- `batch_eval_metrics`: False
|
|
- `eval_on_start`: False
|
|
- `use_liger_kernel`: False
|
|
- `eval_use_gather_object`: False
|
|
- `average_tokens_across_devices`: False
|
|
- `prompts`: None
|
|
- `batch_sampler`: no_duplicates
|
|
- `multi_dataset_batch_sampler`: proportional
|
|
|
|
</details>
|
|
|
|
### Training Logs
|
|
| Epoch | Step | Training Loss | dim_768_cosine_ndcg@10 | dim_512_cosine_ndcg@10 | dim_256_cosine_ndcg@10 | dim_128_cosine_ndcg@10 | dim_64_cosine_ndcg@10 |
|
|
|:---------:|:-------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|
|
|
| 0.2030 | 10 | 9.3166 | - | - | - | - | - |
|
|
| 0.4061 | 20 | 3.7163 | - | - | - | - | - |
|
|
| 0.6091 | 30 | 2.8216 | - | - | - | - | - |
|
|
| 0.8122 | 40 | 1.9313 | - | - | - | - | - |
|
|
| 1.0 | 50 | 1.5613 | 0.8230 | 0.8237 | 0.8153 | 0.8036 | 0.7771 |
|
|
| 1.2030 | 60 | 1.0926 | - | - | - | - | - |
|
|
| 1.4061 | 70 | 0.3367 | - | - | - | - | - |
|
|
| 1.6091 | 80 | 0.3958 | - | - | - | - | - |
|
|
| 1.8122 | 90 | 0.6527 | - | - | - | - | - |
|
|
| 2.0 | 100 | 0.4483 | 0.8202 | 0.8209 | 0.8118 | 0.8033 | 0.7792 |
|
|
| 2.2030 | 110 | 0.1823 | - | - | - | - | - |
|
|
| 2.4061 | 120 | 0.0494 | - | - | - | - | - |
|
|
| 2.6091 | 130 | 0.1204 | - | - | - | - | - |
|
|
| 2.8122 | 140 | 0.2021 | - | - | - | - | - |
|
|
| 3.0 | 150 | 0.2088 | 0.8211 | 0.8213 | 0.8148 | 0.8064 | 0.7825 |
|
|
| 3.2030 | 160 | 0.062 | - | - | - | - | - |
|
|
| 3.4061 | 170 | 0.022 | - | - | - | - | - |
|
|
| 3.6091 | 180 | 0.0654 | - | - | - | - | - |
|
|
| 3.8122 | 190 | 0.1481 | - | - | - | - | - |
|
|
| **3.934** | **196** | **-** | **0.8225** | **0.8224** | **0.816** | **0.8074** | **0.7862** |
|
|
|
|
* The bold row denotes the saved checkpoint.
|
|
|
|
### Framework Versions
|
|
- Python: 3.10.16
|
|
- Sentence Transformers: 3.3.1
|
|
- Transformers: 4.48.1
|
|
- PyTorch: 2.5.1+cu124
|
|
- Accelerate: 1.3.0
|
|
- Datasets: 3.3.2
|
|
- Tokenizers: 0.21.0
|
|
|
|
## Citation
|
|
|
|
### BibTeX
|
|
|
|
#### Sentence Transformers
|
|
```bibtex
|
|
@inproceedings{reimers-2019-sentence-bert,
|
|
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
|
|
author = "Reimers, Nils and Gurevych, Iryna",
|
|
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
|
|
month = "11",
|
|
year = "2019",
|
|
publisher = "Association for Computational Linguistics",
|
|
url = "https://arxiv.org/abs/1908.10084",
|
|
}
|
|
```
|
|
|
|
#### MatryoshkaLoss
|
|
```bibtex
|
|
@misc{kusupati2024matryoshka,
|
|
title={Matryoshka Representation Learning},
|
|
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
|
|
year={2024},
|
|
eprint={2205.13147},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.LG}
|
|
}
|
|
```
|
|
|
|
#### MultipleNegativesRankingLoss
|
|
```bibtex
|
|
@misc{henderson2017efficient,
|
|
title={Efficient Natural Language Response Suggestion for Smart Reply},
|
|
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
|
|
year={2017},
|
|
eprint={1705.00652},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CL}
|
|
}
|
|
```
|
|
|
|
<!--
|
|
## Glossary
|
|
|
|
*Clearly define terms in order to be accessible across audiences.*
|
|
-->
|
|
|
|
<!--
|
|
## Model Card Authors
|
|
|
|
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
|
|
-->
|
|
|
|
<!--
|
|
## Model Card Contact
|
|
|
|
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
|
|
--> |
