metadata

language:
  - en
license: apache-2.0
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:6300
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
base_model: intfloat/e5-large-unsupervised
widget:
  - source_sentence: >-
      What are the key components of the transparency provisions included in the
      Consolidated Appropriations Act of 2021 regarding healthcare?
    sentences:
      - >-
        The report includes information on legal proceedings under 'Note 13 —
        Commitments and Contingencies — Litigation and Other Legal Matters'
        which is a part of the consolidated financial statements
      - >-
        The Consolidated Appropriations Act of 2021 was signed into law in
        December 2020 and contains further transparency provisions requiring
        group health plans and health insurance issuers to report certain
        prescription drug costs, overall spending on health services and
        prescription drugs, and information about premiums and the impact of
        rebates and other remuneration on premiums and out-of-pocket costs to
        the Tri-Departments.
      - In 2023, the company recorded other operating charges of $1,951 million.
  - source_sentence: >-
      What technology does the Tax Advisor use and for what purpose in Intuit's
      offerings?
    sentences:
      - >-
        In 2023, Goldman Sachs' investments in funds at NAV primarily included
        firm-sponsored private equity, credit, real estate, and hedge funds.
        These funds are involved in various types of investments such as
        leveraged buyouts, recapitalizations, growth investments, and distressed
        investments for private equity, while credit funds are focused on
        providing private high-yield capital for leveraged and management buyout
        transactions. Real estate funds invest globally in real estate assets,
        and hedge funds adopt a fundamental bottom-up investment approach.
      - >-
        Using AI technologies, our Tax Advisor offering leverages information
        generated from our ProConnect Tax Online and Lacerte offerings to enable
        year-round tax planning services and communicate tax savings strategies
        to clients.
      - >-
        'Note 13 — Commitments and Contingencies' provides details about
        litigation and other legal matters in an Annual Report on Form 10-K.
  - source_sentence: What was the net revenue for the Data Center segment in 2023?
    sentences:
      - >-
        Data Center net revenue of $6.5 billion in 2023 increased by 7%,
        compared to net revenue of $6.0 billion in 2022.
      - >-
        Under its Class 2 insurance license, Caterpillar Insurance Co. Ltd.
        insures its parent and affiliates for general liability, property, auto
        liability and cargo. It also provides reinsurance to CaterThe pillar
        Insurance Company under a quota share reinsurance agreement for its
        contractual liability and contractors’ equipment programs in the United
        States.
      - >-
        Schwab’s funding of these remaining commitments is dependent upon the
        occurrence of certain conditions, and Schwab expects to pay
        substantially all of these commitments between 2024 and 2027.
  - source_sentence: >-
      What are the three principles of liquidity risk management at Goldman
      Sachs?
    sentences:
      - >-
        The Company determines if an arrangement is a lease at inception and
        classifies its leases at commencement. Operating leases are included in
        operating lease right-of-use ("ROU") assets and current and noncurrent
        operating lease liabilities on the Company’s consolidated balance
        sheets.
      - >-
        Garmin Ltd. reported a net income of $1,289,636 for the fiscal year
        ended December 30, 2023.
      - >-
        Goldman Sachs manages liquidity risk based on three principles: 1) hold
        sufficient excess liquidity in the form of GCLA to cover outflows during
        a stressed period, 2) maintain appropriate Asset-Liability Management,
        and 3) maintain a viable Contingency Funding Plan.
  - source_sentence: >-
      What was the total cost and expenses reported by Berkshire Hathaway for
      the year ended December 31, 2023?
    sentences:
      - Total costs and expenses | | 321,144 | | | 266,484 | | | 243,752
      - >-
        Qulipta (atogepant) is a calcitonin gene-related peptide receptor
        antagonist indicated for the preventive treatment of episodic and
        chronic migraine in adults. Qulipta is commercialized in the United
        States and Canada and is approved in the European Union under the brand
        name Aquipta.
      - >-
        Item 3 'Legal Proceedings' is integrated by reference to other parts
        including Note 22 — 'Environmental and legal matters' and Part II, Item
        8.
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
model-index:
  - name: E5 unsupervised Financial Matryoshka
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 768
          type: dim_768
        metrics:
          - type: cosine_accuracy@1
            value: 0.7271428571428571
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.85
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8785714285714286
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9114285714285715
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7271428571428571
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.2833333333333333
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.17571428571428568
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09114285714285714
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7271428571428571
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.85
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8785714285714286
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9114285714285715
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.822517236613446
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7936921768707483
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7973883589026711
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 512
          type: dim_512
        metrics:
          - type: cosine_accuracy@1
            value: 0.7271428571428571
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8457142857142858
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.88
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9128571428571428
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7271428571428571
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.28190476190476194
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.176
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09128571428571429
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7271428571428571
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8457142857142858
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.88
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9128571428571428
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8223709830528422
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.793145691609977
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7966990460475021
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 256
          type: dim_256
        metrics:
          - type: cosine_accuracy@1
            value: 0.72
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8457142857142858
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8714285714285714
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9057142857142857
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.72
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.28190476190476194
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.17428571428571424
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09057142857142855
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.72
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8457142857142858
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8714285714285714
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9057142857142857
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8159991941699124
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7869370748299319
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7906967878713818
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 128
          type: dim_128
        metrics:
          - type: cosine_accuracy@1
            value: 0.7085714285714285
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8285714285714286
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8728571428571429
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.8985714285714286
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7085714285714285
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.2761904761904762
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.17457142857142854
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.08985714285714284
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7085714285714285
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8285714285714286
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8728571428571429
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.8985714285714286
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8073517667504667
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7777108843537414
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7815591417851651
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 64
          type: dim_64
        metrics:
          - type: cosine_accuracy@1
            value: 0.6757142857142857
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8185714285714286
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8457142857142858
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.8842857142857142
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.6757142857142857
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.27285714285714285
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.16914285714285712
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.08842857142857141
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.6757142857142857
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8185714285714286
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8457142857142858
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.8842857142857142
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.7861731335824387
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7542681405895693
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7588497811523153
            name: Cosine Map@100

E5 unsupervised Financial Matryoshka

This is a sentence-transformers model finetuned from intfloat/e5-large-unsupervised on the json dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: intfloat/e5-large-unsupervised
Maximum Sequence Length: 512 tokens
Output Dimensionality: 1024 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- json
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("schawla2/e5-unsupervised-financial-matryoshka")
# Run inference
sentences = [
    'What was the total cost and expenses reported by Berkshire Hathaway for the year ended December 31, 2023?',
    'Total costs and expenses | | 321,144 | | | 266,484 | | | 243,752',
    'Qulipta (atogepant) is a calcitonin gene-related peptide receptor antagonist indicated for the preventive treatment of episodic and chronic migraine in adults. Qulipta is commercialized in the United States and Canada and is approved in the European Union under the brand name Aquipta.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Datasets: dim_768, dim_512, dim_256, dim_128 and dim_64
Evaluated with InformationRetrievalEvaluator

Metric	dim_768	dim_512	dim_256	dim_128	dim_64
cosine_accuracy@1	0.7271	0.7271	0.72	0.7086	0.6757
cosine_accuracy@3	0.85	0.8457	0.8457	0.8286	0.8186
cosine_accuracy@5	0.8786	0.88	0.8714	0.8729	0.8457
cosine_accuracy@10	0.9114	0.9129	0.9057	0.8986	0.8843
cosine_precision@1	0.7271	0.7271	0.72	0.7086	0.6757
cosine_precision@3	0.2833	0.2819	0.2819	0.2762	0.2729
cosine_precision@5	0.1757	0.176	0.1743	0.1746	0.1691
cosine_precision@10	0.0911	0.0913	0.0906	0.0899	0.0884
cosine_recall@1	0.7271	0.7271	0.72	0.7086	0.6757
cosine_recall@3	0.85	0.8457	0.8457	0.8286	0.8186
cosine_recall@5	0.8786	0.88	0.8714	0.8729	0.8457
cosine_recall@10	0.9114	0.9129	0.9057	0.8986	0.8843
cosine_ndcg@10	0.8225	0.8224	0.816	0.8074	0.7862
cosine_mrr@10	0.7937	0.7931	0.7869	0.7777	0.7543
cosine_map@100	0.7974	0.7967	0.7907	0.7816	0.7588

Training Details

Training Dataset

json

Dataset: json
Size: 6,300 training samples
Columns: anchor and positive
Approximate statistics based on the first 1000 samples:
anchor positive
type string string
details
min: 8 tokens
mean: 20.8 tokens
max: 51 tokens

min: 7 tokens
mean: 45.24 tokens
max: 326 tokens

	anchor	positive
type	string	string
details	min: 8 tokens mean: 20.8 tokens max: 51 tokens	min: 7 tokens mean: 45.24 tokens max: 326 tokens

Samples:

anchor	positive
`How many full-time employees did Microsoft report as of June 30, 2023?`	`As of June 30, 2023, we employed approximately 221,000 people on a full-time basis, 120,000 in the U.S. and 101,000 internationally.`
`What was the total amount CSC paid for Series G preferred stock repurchases in 2023?`	`In 2023, CSC repurchased 42,036 depositary shares representing interests in Series G preferred stock for a total amount of $42 million.`
`What does Note 13 in the Annual Report on Form 10-K discuss?`	`For a discussion of legal and other proceedings in which we are involved, see Note 13 - Commitments and Contingencies in the Notes to Consolidated Financial Statements.`

Loss: MatryoshkaLoss with these parameters:

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
per_device_eval_batch_size: 16
gradient_accumulation_steps: 16
learning_rate: 2e-05
num_train_epochs: 4
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: True
tf32: True
load_best_model_at_end: True
optim: adamw_torch_fused
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 8
per_device_eval_batch_size: 16
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 16
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 4
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: True
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.2030	10	9.3166	-	-	-	-	-
0.4061	20	3.7163	-	-	-	-	-
0.6091	30	2.8216	-	-	-	-	-
0.8122	40	1.9313	-	-	-	-	-
1.0	50	1.5613	0.8230	0.8237	0.8153	0.8036	0.7771
1.2030	60	1.0926	-	-	-	-	-
1.4061	70	0.3367	-	-	-	-	-
1.6091	80	0.3958	-	-	-	-	-
1.8122	90	0.6527	-	-	-	-	-
2.0	100	0.4483	0.8202	0.8209	0.8118	0.8033	0.7792
2.2030	110	0.1823	-	-	-	-	-
2.4061	120	0.0494	-	-	-	-	-
2.6091	130	0.1204	-	-	-	-	-
2.8122	140	0.2021	-	-	-	-	-
3.0	150	0.2088	0.8211	0.8213	0.8148	0.8064	0.7825
3.2030	160	0.062	-	-	-	-	-
3.4061	170	0.022	-	-	-	-	-
3.6091	180	0.0654	-	-	-	-	-
3.8122	190	0.1481	-	-	-	-	-
3.934	196	-	0.8225	0.8224	0.816	0.8074	0.7862

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.10.16
Sentence Transformers: 3.3.1
Transformers: 4.48.1
PyTorch: 2.5.1+cu124
Accelerate: 1.3.0
Datasets: 3.3.2
Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}