metadata

tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:16000
  - loss:OnlineContrastiveLoss
base_model: jinaai/jina-embeddings-v3
widget:
  - source_sentence: >-
      This is absolutely the worst trash I have ever seen. When I saw it in the
      theater (arghhh!), it took 15 full minutes before I realized that what I
      was seeing was the feature, not a sick joke!
    sentences:
      - negative negative negative negative
      - negative negative negative negative
      - positive positive positive positive
  - source_sentence: >-
      I saw this movie years ago in a group tradition of Fast Forward Film
      Festivals, where we would set out to rent a bunch of B-movies and vote for
      who picked the worst.<br /><br />The night we watched this, it was voted
      the best, due to semblance of plot and fun costuming.<br /><br />This is
      certainly a silly, kitschy, movie, to be watched under the full
      understanding that you are watching low-budget fluff. Personally, however,
      I wouldn't recommend additional substances ... this movie will leave it's
      own mark on you.<br /><br />It made enough of an impression on me that
      I've actually been trying to get my hands on a copy for a few years.<br
      /><br />A good choice if you are setting out to watch bad movies. This one
      is fun, and I remember bouncy music ...
    sentences:
      - negative negative negative negative
      - positive positive positive positive
      - negative negative negative negative
  - source_sentence: >-
      Star Wars: Episode 4 .<br /><br />the best Star Wars ever. its the first
      movie i ever Sean were the bad guys win and its a very good ending. it
      really had me wait hing for the next star wars because so match stuff
      comes along in this movie that you just got to find out more in the last
      one. whit Al lot of movies i always get the feeling that it could be don
      bedder but not whit this one. and i Will never ever forget the part were
      wader tels Luke he is his father.way too cool. also love the Bob feat
      figure a do hes a back ground player. if you never ever Saw a star wars
      movie you go to she this one.its the best.<br /><br />thanks Lucas
    sentences:
      - negative negative negative negative
      - positive positive positive positive
      - positive positive positive positive
  - source_sentence: >-
      Alain Chabat claims this movie as his original idea but the theme of
      reluctant lovers who finally get it together is as old, if not older, than
      Shakespeare.<br /><br />Chabat is a "vieux garcon", happily single and not
      wanting any member of the opposite sex to disturb his life. He has a
      problem, 5 sisters and a matriarchal mum - the G7 - who decide he should
      be married. Enter the delightful, charming Charlotte Gainsbourg and what
      should be a simple plan. Charlotte has to pose as Chabat's girlfriend and
      then simply not turn up on the day of the wedding. No more talk of
      marriage from the G7. Of course the best laid plans have a habit of
      spiralling out of control.<br /><br />There are very strong supporting
      roles from Lafont as the mother and Osterman as the tight-fisted brother
      of Gainsbourg.<br /><br />There are some fantastic scenes as first
      Charlotte has to charm, then revolt the family. French farce with an
      English.
    sentences:
      - positive positive positive positive
      - negative negative negative negative
      - negative negative negative negative
  - source_sentence: >-
      Saw this on cable back in the early 90's and loved it. Never saw it again
      until it showed up on cable again recently. Still find it a great Vietnam
      movie. Not sure why its not higher rated. I found everything about this
      film compelling. As a vet (not from Vietnam) I can relate to the
      situations brought by both Harris and De Niro. I can only imagine this
      film being more poignant now with our situation in Iraq. I wish this would
      be offered on cable more often for people to see. The human toll on our
      soldiers isn't left on the battlefield. Its brought home for the rest of
      there lives. And this film is one of many that brings that home in a very
      hard way. Excellent film.
    sentences:
      - negative negative negative negative
      - positive positive positive positive
      - positive positive positive positive
pipeline_tag: sentence-similarity
library_name: sentence-transformers

SentenceTransformer based on jinaai/jina-embeddings-v3

This is a sentence-transformers model finetuned from jinaai/jina-embeddings-v3. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: jinaai/jina-embeddings-v3
Maximum Sequence Length: 8194 tokens
Output Dimensionality: 1024 tokens
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (transformer): Transformer(
    (auto_model): XLMRobertaLoRA(
      (roberta): XLMRobertaModel(
        (embeddings): XLMRobertaEmbeddings(
          (word_embeddings): ParametrizedEmbedding(
            250002, 1024, padding_idx=1
            (parametrizations): ModuleDict(
              (weight): ParametrizationList(
                (0): LoRAParametrization()
              )
            )
          )
          (token_type_embeddings): ParametrizedEmbedding(
            1, 1024
            (parametrizations): ModuleDict(
              (weight): ParametrizationList(
                (0): LoRAParametrization()
              )
            )
          )
        )
        (emb_drop): Dropout(p=0.1, inplace=False)
        (emb_ln): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (encoder): XLMRobertaEncoder(
          (layers): ModuleList(
            (0-23): 24 x Block(
              (mixer): MHA(
                (rotary_emb): RotaryEmbedding()
                (Wqkv): ParametrizedLinearResidual(
                  in_features=1024, out_features=3072, bias=True
                  (parametrizations): ModuleDict(
                    (weight): ParametrizationList(
                      (0): LoRAParametrization()
                    )
                  )
                )
                (inner_attn): FlashSelfAttention(
                  (drop): Dropout(p=0.1, inplace=False)
                )
                (inner_cross_attn): FlashCrossAttention(
                  (drop): Dropout(p=0.1, inplace=False)
                )
                (out_proj): ParametrizedLinear(
                  in_features=1024, out_features=1024, bias=True
                  (parametrizations): ModuleDict(
                    (weight): ParametrizationList(
                      (0): LoRAParametrization()
                    )
                  )
                )
              )
              (dropout1): Dropout(p=0.1, inplace=False)
              (drop_path1): StochasticDepth(p=0.0, mode=row)
              (norm1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
              (mlp): Mlp(
                (fc1): ParametrizedLinear(
                  in_features=1024, out_features=4096, bias=True
                  (parametrizations): ModuleDict(
                    (weight): ParametrizationList(
                      (0): LoRAParametrization()
                    )
                  )
                )
                (fc2): ParametrizedLinear(
                  in_features=4096, out_features=1024, bias=True
                  (parametrizations): ModuleDict(
                    (weight): ParametrizationList(
                      (0): LoRAParametrization()
                    )
                  )
                )
              )
              (dropout2): Dropout(p=0.1, inplace=False)
              (drop_path2): StochasticDepth(p=0.0, mode=row)
              (norm2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
            )
          )
        )
        (pooler): XLMRobertaPooler(
          (dense): ParametrizedLinear(
            in_features=1024, out_features=1024, bias=True
            (parametrizations): ModuleDict(
              (weight): ParametrizationList(
                (0): LoRAParametrization()
              )
            )
          )
          (activation): Tanh()
        )
      )
    )
  )
  (pooler): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (normalizer): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ELVISIO/jina_embeddings_v3_finetuned_online_contrastive_imdb_2")
# Run inference
sentences = [
    "Saw this on cable back in the early 90's and loved it. Never saw it again until it showed up on cable again recently. Still find it a great Vietnam movie. Not sure why its not higher rated. I found everything about this film compelling. As a vet (not from Vietnam) I can relate to the situations brought by both Harris and De Niro. I can only imagine this film being more poignant now with our situation in Iraq. I wish this would be offered on cable more often for people to see. The human toll on our soldiers isn't left on the battlefield. Its brought home for the rest of there lives. And this film is one of many that brings that home in a very hard way. Excellent film.",
    'positive positive positive positive',
    'negative negative negative negative',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

Size: 16,000 training samples
Columns: sentence1, sentence2, and label
Approximate statistics based on the first 1000 samples:
sentence1 sentence2 label
type string string float
details
min: 39 tokens
mean: 173.59 tokens
max: 291 tokens

min: 6 tokens
mean: 6.0 tokens
max: 6 tokens

min: 0.0
mean: 0.5
max: 1.0

	sentence1	sentence2	label
type	string	string	float
details	min: 39 tokens mean: 173.59 tokens max: 291 tokens	min: 6 tokens mean: 6.0 tokens max: 6 tokens	min: 0.0 mean: 0.5 max: 1.0

Samples:

sentence1	sentence2	label
There are two kinds of 1950s musicals. First you have the glossy MGM productions with big names and great music. And then you have the minor league with a less famous cast, less famous music and second rate directors. 'The Girl Can't Help It' belongs to the latter category. Neither Tom Ewell or Edmond O'Brien became famous and Jayne Mansfield was famous for her... well, never mind. Seems like every decade has its share of Bo Dereks or Pamela Andersons. The plot itself is thin as a razorblade and one can't help suspect that it is mostly an attempt to sell records for Fats Domino, Little Richard or others of the 1950s rock acts that appear in the movie. If that music appeals to you this is worth watching. If not, don't bother.	`negative negative negative negative`	`1.0`
There are two kinds of 1950s musicals. First you have the glossy MGM productions with big names and great music. And then you have the minor league with a less famous cast, less famous music and second rate directors. 'The Girl Can't Help It' belongs to the latter category. Neither Tom Ewell or Edmond O'Brien became famous and Jayne Mansfield was famous for her... well, never mind. Seems like every decade has its share of Bo Dereks or Pamela Andersons. The plot itself is thin as a razorblade and one can't help suspect that it is mostly an attempt to sell records for Fats Domino, Little Richard or others of the 1950s rock acts that appear in the movie. If that music appeals to you this is worth watching. If not, don't bother.	`positive positive positive positive`	`0.0`
Thankfully as a student I have been able to watch "Diagnosis Murder" for a number of years now. It is basically about a doctor who solves murders with the help of his LAPD son, a young doctor and a pathologist. DM provided 8 seasons of exceptional entertainment. What made it different from the many other cop shows and worth watching many times over was its cast and quality of writing. The main cast gave good performances and Dick Van Dyke's entertainer roots shone through with the use of magic, dance and humor. The best aspects of DM was the fast pace, witty scripts and of course the toe tapping score. Sadly it has been unfairly compared to "Murder, She Wrote". DM is far superior boasting more difficult mysteries to solve and more variety. Now it is gone TV is a worse place. Gone are the days of feelgood, family friendly cop shows. Now there is just depressing 'gritty' ones.	`positive positive positive positive`	`1.0`

Loss: OnlineContrastiveLoss

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 64
per_device_eval_batch_size: 64

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 64
per_device_eval_batch_size: 64
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 3.0
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss
0.2	50	2.9875
0.4	100	0.9284
0.6	150	0.7744
0.8	200	0.7551
1.0	250	0.6899
1.2	300	0.6892
1.4	350	0.6208
1.6	400	0.6831
1.8	450	0.6417
2.0	500	0.7181
2.2	550	0.7638
2.4	600	0.7152
2.6	650	0.6103
2.8	700	0.6801
3.0	750	0.5981

Framework Versions

Python: 3.10.12
Sentence Transformers: 3.1.1
Transformers: 4.45.2
PyTorch: 2.5.1+cu121
Accelerate: 1.1.1
Datasets: 2.21.0
Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}