SentenceTransformer

This is a sentence-transformers model trained on the parquet dataset. It maps sentences & paragraphs to a 512-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Maximum Sequence Length: 512 tokens
Output Dimensionality: 512 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- parquet

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 512, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pankajrajdeo/Bioformer-8L-UMLS-Pubmed_PMC-Backward_TCE-Epoch-1")
# Run inference
sentences = [
    'Stage-specific protein synthesis by isolated spermatogenic cells throughout meiosis and early spermiogenesis in the mouse.',
    'Spermatogenic cells isolated from prepubertal and adult mice by unit gravity sedimentation have been used to examine proteins synthesized in a stage-specific manner throughout meiosis and early spermiogenesis. Preleptotene, leptotene/zygotene, and pachytene spermatocytes were isolated from 17-day-old mice. Adult pachytene spermatocytes and round spermatids were isolated from mature animals. These germ cells were then cultured in defined medium withmethioninemet, when expressed either as cpm/-10(6) cells or cpm/mg protein. Comparisons of 2D autoradiograms indicated that many proteins, including actin and tubulins, are synthesized at approximately equal levels in all stages examined. Other proteins, including heat-shock proteins and multiple plasma membrane constituents, are synthesized in a stage-specific manner in leptotene/zygotene spermatocytes, pachytene spermatocytes, and round spermatids. These studies define conditions for monitoring protein synthesis in isolated spermatogenic cells prior to the pachytene stage of meiosis, provide a 2D map of proteins synthesized at these earlier meiotic stages, and examine the synthesis of several proteins previously identified on 2D gels with biochemical and immunological methods.',
    'Valine-derived benzoxazinones have been synthesized and found to be competitive, slow-binding inhibitors of human leukocyte elastase (HLE). Steady-state inhibition constants Ki are dependent on aryl substitution and reach a maximum of potency of 0.5 nM with the 5-Cl compound 6. UV-spectral data for the interaction of HLE and the unsubstituted inhibitor 3 indicate that the stable complex formed between enzyme and inhibitor is an acyl-enzyme that can either undergo ring closure, to reform intact benzoxazinone, or hydrolysis, to liberate an N-acylanthranilic acid. "Burst" kinetic data, derived from the direct observation of the interaction of HLE and 3, are consistent with results of the inhibition of catalysis experiments.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 512]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

parquet

Dataset: parquet
Size: 33,870,508 training samples
Columns: anchor and positive
Approximate statistics based on the first 1000 samples:
anchor positive
type string string
details
min: 5 tokens
mean: 24.6 tokens
max: 68 tokens

min: 13 tokens
mean: 270.2 tokens
max: 512 tokens

	anchor	positive
type	string	string
details	min: 5 tokens mean: 24.6 tokens max: 68 tokens	min: 13 tokens mean: 270.2 tokens max: 512 tokens

Samples:

anchor	positive
`Underlying synovial sarcoma undiagnosed for more than 20 years in a patient with regional pain: a case report.`	Synovial sarcoma (SS) is a malignant tumor comprising 5-10% of all soft tissue sarcomas. SS has distinct characteristics, such as a predilection for young adults and relatively slow growth compared to other soft tissue sarcomas. Some patients with SS experience long-standing pain at the tumor site before the development of a palpable mass. Herein, we report the case of a 39-year-old woman with SS in the upper arm who presented with pain for > 20 years. The tumor detected on magnetic resonance imaging at 17 years was an SS. To the best of our knowledge, no English-language reports on imaging study-based identification of SS, which was undiagnosed for > 20 years, are known in the literature. This report discusses the imaging features of this latent lesion and the volume-doubling time of this unusual tumor.
`MR imaging signs of shoulder adhesive capsulitis: analysis of potential differentials and improved diagnostic criteria.`	`OBJECTIVE: To evaluate the prevalence of shoulder adhesive capsulitis. Based on these findings, a grading system for fibro-inflammatory capsular changes is proposed. CONCLUSION: MR AC signs are frequent in patients with shoulder conditions other than AC; however, in these patients, capsular changes are less prominent than in patients with clinical AC.`
`MR imaging signs of shoulder adhesive capsulitis: analysis of potential differentials and improved diagnostic criteria.`	`OBJECTIVE: To evaluate the prevalence of shoulder adhesive capsulitis. Based on these findings, a grading system for fibro-inflammatory capsular changes is proposed. CONCLUSION: MR AC signs are frequent in patients with shoulder conditions other than AC; however, in these patients, capsular changes are less prominent than in patients with clinical AC.`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

Evaluation Dataset

parquet

Dataset: parquet
Size: 33,870,508 evaluation samples
Columns: anchor and positive
Approximate statistics based on the first 1000 samples:
anchor positive
type string string
details
min: 6 tokens
mean: 18.94 tokens
max: 60 tokens

min: 26 tokens
mean: 199.08 tokens
max: 512 tokens

	anchor	positive
type	string	string
details	min: 6 tokens mean: 18.94 tokens max: 60 tokens	min: 26 tokens mean: 199.08 tokens max: 512 tokens

Samples:

anchor	positive
`Effect of ritodrine on thyroid hormone concentrations.`	In a clinical study of 17 pregnant women treated with ritodrine, a beta-2-sympathomimetic agent used for tocolysis, thyroid hormone status was assessed longitudinally. This was done in order to verify the hypothesis that an increase in T3 levels could result from adrenergic stimulation, since propanolol, a beta blocking agent, has proved to decrease T3 levels in man. We have observed a significant increase in serum T3 concentrations 24-48 h after the start of the ritodrine treatment. The changes were only temporarely since one week after the start the serum T3 concentrations did not differ significantly from the pre-treatment levels. A decrease in T3 levels was found after discontinuation of treatment. No significant changes were found in T4 and TSH concentrations excluding an influence in ritodrine therapy on the pituitary-thyroid axes. It was concluded that stimulation of type I deiodinase was responsible for the changes in T3. These beta-2-mimetic variations may explain, to a certai...
`beta-Aminoisobutyric acid as a marker of thymine catabolism in malignancy.`	Urine from untreated patients with various tumours and controls has been examined for the excretion of beta-aminoisobutyric acid and uric acid. The patients were classified into four groups: I, beta-aminoisobutyric acid and uric acid both normal; II, beta-aminoisobutyric acid normal, uric acid elevated; III, beta-aminoisobutyric acid elevated, uric acid normal; IV, beta-aminoisobutyric acid and uric acid both elevated. Uric acid was used as an indicator for tissue-breakdown. Pseudouridine being a specific parameter for t-RNA degradation was estimated for comparison. Increased urinary concentrations of beta-aminoisobutyric acid were frequently found in tumour patients, especially in patients with leukaemia and non-Hodgkin lymphoma. Tissue breakdown being the cause of the beta-aminoisobutyric aciduria could only be considered in part of the patients. Moreover, strongly elevated ratios of beta-aminoisobutyric acid to uric acid were found. Urinary patterns of pyrimidines and purines were d...
`The effect of oral contraceptives on plasma-free and salivary cortisol and cortisone.`	The effect of a low estrogen oral contraceptive (OC) on glucocorticoid levels in plasma and saliva as well as glucocorticoid binding was studied in 23 healthy women using 30 micrograms ethinyl estradiol (EE2) + 150 micrograms desogestrel (Marvelon) (II). Fifteen healthy females with normal menses served as controls (I). Blood and salivary samples were taken between 9.00 and 9.30 a.m. on the 18th day of menstrual or pill cycle. Assay accuracy had been optimised by applying extraction and chromatographic purification before radioimmunoassay (RIA) of cortisol and cortisone in both plasma and salivary samples. Free steroid assays were performed by applying the same procedure to equilibrium dialysates obtained after dialysing plasma against an equal volume of buffer, instead of measuring tracer distribution. Corticosteroid Binding Globulin (CBG) was measured by a commercial RIA. As expected, CBG as well as plasma total cortisol were elevated in the pill group. Interestingly both plasma free...

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 128
learning_rate: 2e-05
num_train_epochs: 1
max_steps: 251382
log_level: info
fp16: True
dataloader_num_workers: 16
load_best_model_at_end: True
resume_from_checkpoint: True

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 128
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 1
max_steps: 251382
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 0
log_level: info
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: True
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 16
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: True
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss
0.0000	1	1.2229	-
0.0040	1000	0.2405	-
0.0080	2000	0.1332	-
0.0119	3000	0.1025	-
0.0159	4000	0.0872	-
0.0199	5000	0.0707	-
0.0239	6000	0.0733	-
0.0278	7000	0.0736	-
0.0318	8000	0.1194	-
0.0358	9000	0.1336	-
0.0398	10000	0.1102	-
0.0438	11000	0.0936	-
0.0477	12000	0.0547	-
0.0517	13000	0.0709	-
0.0557	14000	0.0718	-
0.0597	15000	0.0431	-
0.0636	16000	0.0299	-
0.0676	17000	0.0458	-
0.0716	18000	0.0356	-
0.0756	19000	0.0293	-
0.0796	20000	0.0325	-
0.0835	21000	0.0308	-
0.0875	22000	0.0334	-
0.0915	23000	0.04	-
0.0955	24000	0.0293	-
0.0994	25000	0.0354	-
0.1034	26000	0.0254	-
0.1074	27000	0.0258	-
0.1114	28000	0.0285	-
0.1154	29000	0.0319	-
0.1193	30000	0.0437	-
0.1233	31000	0.0397	-
0.1273	32000	0.048	-
0.1313	33000	0.0482	-
0.1353	34000	0.0635	-
0.1392	35000	0.0447	-
0.1432	36000	0.0482	-
0.1472	37000	0.0441	-
0.1512	38000	0.0418	-
0.1551	39000	0.0732	-
0.1591	40000	0.0675	-
0.1631	41000	0.0721	-
0.1671	42000	0.0719	-
0.1711	43000	0.0665	-
0.1750	44000	0.0523	-
0.1790	45000	0.0412	-
0.1830	46000	0.0611	-
0.1870	47000	0.0664	-
0.1909	48000	0.0645	-
0.1949	49000	0.0614	-
0.1989	50000	0.063	-
0.2029	51000	0.0318	-
0.2069	52000	0.0436	-
0.2108	53000	0.0434	-
0.2148	54000	0.0423	-
0.2188	55000	0.0433	-
0.2228	56000	0.0382	-
0.2267	57000	0.0425	-
0.2307	58000	0.038	-
0.2347	59000	0.0423	-
0.2387	60000	0.0398	-
0.2427	61000	0.039	-
0.2466	62000	0.0638	-
0.2506	63000	0.0649	-
0.2546	64000	0.0586	-
0.2586	65000	0.0586	-
0.2625	66000	0.0572	-
0.2665	67000	0.0613	-
0.2705	68000	0.0566	-
0.2745	69000	0.0366	-
0.2785	70000	0.0356	-
0.2824	71000	0.0284	-
0.2864	72000	0.0342	-
0.2904	73000	0.0328	-
0.2944	74000	0.029	-
0.2983	75000	0.035	-
0.3023	76000	0.0352	-
0.3063	77000	0.0346	-
0.3103	78000	0.0515	-
0.3143	79000	0.0513	-
0.3182	80000	0.05	-
0.3222	81000	0.0436	-
0.3262	82000	0.0408	-
0.3302	83000	0.0465	-
0.3342	84000	0.0354	-
0.3381	85000	0.0478	-
0.3421	86000	0.0464	-
0.3461	87000	0.0399	-
0.3501	88000	0.0393	-
0.3540	89000	0.0667	-
0.3580	90000	0.0719	-
0.3620	91000	0.0611	-
0.3660	92000	0.0567	-
0.3700	93000	0.0461	-
0.3739	94000	0.0593	-
0.3779	95000	0.0525	-
0.3819	96000	0.0406	-
0.3859	97000	0.0346	-
0.3898	98000	0.0351	-
0.3938	99000	0.0365	-
0.3978	100000	0.0293	-
0.4018	101000	0.0356	-
0.4058	102000	0.0694	-
0.4097	103000	0.0596	-
0.4137	104000	0.0698	-
0.4177	105000	0.0592	-
0.4217	106000	0.0637	-
0.4256	107000	0.0677	-
0.4296	108000	0.0709	-
0.4336	109000	0.0443	-
0.4376	110000	0.0339	-
0.4416	111000	0.0403	-
0.4455	112000	0.041	-
0.4495	113000	0.0443	-
0.4535	114000	0.0379	-
0.4575	115000	0.0492	-
0.4614	116000	0.0469	-
0.4654	117000	0.0712	-
0.4694	118000	0.0633	-
0.4734	119000	0.0653	-
0.4774	120000	0.049	-
0.4813	121000	0.0612	-
0.4853	122000	0.0271	-
0.4893	123000	0.0296	-
0.4933	124000	0.0356	-
0.4972	125000	0.0496	-
0.5012	126000	0.0491	-
0.5052	127000	0.0567	-
0.5092	128000	0.0638	-
0.5132	129000	0.0567	-
0.5171	130000	0.0419	-
0.5211	131000	0.0499	-
0.5251	132000	0.0296	-
0.5291	133000	0.0412	-
0.5331	134000	0.0339	-
0.5370	135000	0.041	-
0.5410	136000	0.051	-
0.5450	137000	0.056	-
0.5490	138000	0.0397	-
0.5529	139000	0.0559	-
0.5569	140000	0.0628	-
0.5609	141000	0.0482	-
0.5649	142000	0.0363	-
0.5689	143000	0.0471	-
0.5728	144000	0.0324	-
0.5768	145000	0.0343	-
0.5808	146000	0.0474	-
0.5848	147000	0.067	-
0.5887	148000	0.0548	-
0.5927	149000	0.0501	-
0.5967	150000	0.047	-
0.6007	151000	0.0259	-
0.6047	152000	0.0363	-
0.6086	153000	0.0308	-
0.6126	154000	0.025	-
0.6166	155000	0.0495	-
0.6206	156000	0.0486	-
0.6245	157000	0.0412	-
0.6285	158000	0.0368	-
0.6325	159000	0.0375	-
0.6365	160000	0.0343	-
0.6405	161000	0.0416	-
0.6444	162000	0.049	-
0.6484	163000	0.0679	-
0.6524	164000	0.0643	-
0.6564	165000	0.0708	-
0.6603	166000	0.0573	-
0.6643	167000	0.0648	-
0.6683	168000	0.0626	-
0.6723	169000	0.0375	-
0.6763	170000	0.0421	-
0.6802	171000	0.0429	-
0.6842	172000	0.0762	-
0.6882	173000	0.0687	-
0.6922	174000	0.0617	-
0.6961	175000	0.0373	-
0.7001	176000	0.0529	-
0.7041	177000	0.0453	-
0.7081	178000	0.0447	-
0.7121	179000	0.0472	-
0.7160	180000	0.0387	-
0.7200	181000	0.0337	-
0.7240	182000	0.0577	-
0.7280	183000	0.0728	-
0.7320	184000	0.0765	-
0.7359	185000	0.0621	-
0.7399	186000	0.0585	-
0.7439	187000	0.0455	-
0.7479	188000	0.1172	-
0.7518	189000	0.0442	-
0.7558	190000	0.0558	-
0.7598	191000	0.0338	-
0.7638	192000	0.0328	-
0.7678	193000	0.0783	-
0.7717	194000	0.068	-
0.7757	195000	0.073	-
0.7797	196000	0.0605	-
0.7837	197000	0.0641	-
0.7876	198000	0.04	-
0.7916	199000	0.047	-
0.7956	200000	0.0734	-
0.7996	201000	0.0558	-
0.8036	202000	0.044	-
0.8075	203000	0.0467	-
0.8115	204000	0.0607	-
0.8155	205000	0.0695	-
0.8195	206000	0.0536	-
0.8234	207000	0.0599	-
0.8274	208000	0.0621	-
0.8314	209000	0.0717	-
0.8354	210000	0.0517	-
0.8394	211000	0.0596	-
0.8433	212000	0.0816	-
0.8473	213000	0.0595	-
0.8513	214000	0.0572	-
0.8553	215000	0.0715	-
0.8592	216000	0.0585	-
0.8632	217000	0.079	-
0.8672	218000	0.0903	-
0.8712	219000	0.0941	-
0.8752	220000	0.0734	-
0.8791	221000	0.0656	-
0.8831	222000	0.0966	-
0.8871	223000	0.0826	-
0.8911	224000	0.0702	-
0.8950	225000	0.0939	-
0.8990	226000	0.0651	-
0.9030	227000	0.0938	-
0.9070	228000	0.0781	-
0.9110	229000	0.0587	-
0.9149	230000	0.1404	-
0.9189	231000	0.059	-
0.9229	232000	0.0715	-
0.9269	233000	0.1225	-
0.9309	234000	0.0551	-
0.9348	235000	0.1245	-
0.9388	236000	0.0587	-
0.9428	237000	0.118	-
0.9468	238000	0.0593	-
0.9507	239000	0.0626	-
0.9547	240000	0.0885	-
0.9587	241000	0.0614	-
0.9627	242000	0.1232	-
0.9667	243000	0.0574	-
0.9706	244000	0.1269	-
0.9746	245000	0.0591	-
0.9786	246000	0.1019	-
0.9826	247000	0.0518	-
0.9865	248000	0.1064	-
0.9905	249000	0.0785	-
0.9945	250000	0.0949	-
0.9985	251000	0.0976	-
1.0000	251382	-	0.0076

Framework Versions

Python: 3.11.11
Sentence Transformers: 3.4.1
Transformers: 4.48.2
PyTorch: 2.6.0+cu124
Accelerate: 1.3.0
Datasets: 3.2.0
Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}