samehkhattab/qwen3-4b-biomed-retrieval

SentenceTransformer based on Qwen/Qwen3-4B

This is a sentence-transformers model finetuned from Qwen/Qwen3-4B on the biomed_retrieval_original dataset. It maps sentences & paragraphs to a 2560-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Qwen/Qwen3-4B
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 2560 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • biomed_retrieval_original

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: Qwen3Model 
  (1): Pooling({'word_embedding_dimension': 2560, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Given a web search query, retrieve relevant passages that answer the query: can you drink coffee when your fasting',
    "Fasting for Blood Work: Not if you wish accurate results. Don't have coffee, tea, juice or any sort of fluid apart from water. Do not nibble gum, don't smoke, do not even exercise. Be a monk up until the test is done. As explained by Fasting For Blood Work http://fastingforbloodwork.net/ you can drink coffee as long as there is no sugar involved. But be careful as some black coffee have sugars in them.",
    'Rating Newest Oldest. Best Answer: Fasting in Islam differs from fasting in some other religions. Muslims are not allowed to eat or drink anything during the period of fasting that starts before dawn and lasts till dusk. So, NO you cant drink water or any beverage. Yes, itâ\x80\x99s good to drink tea while on a fast. I would stay away from black tea and most green teas as they contain a heavy amount of caffeine. Since you are detoxing, you donâ\x80\x99t want to be adding toxins like caffeine into your system. White teas or any decaf teas will work perfectly! Any night time tea will be good too.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 2560]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: bmretriever
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.972

Training Details

Training Dataset

biomed_retrieval_original

• Dataset: biomed_retrieval_original at 469552f
• Size: 9,000 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 16 tokens mean: 32.5 tokens max: 512 tokens	min: 2 tokens mean: 198.5 tokens max: 512 tokens	min: 3 tokens mean: 315.07 tokens max: 512 tokens

• Samples:

  anchor	positive	negative
  Given a web search query, retrieve relevant passages that answer the query: definition of ruminations	Rumination (psychology) Rumination is the focused attention on the symptoms of one's distress, and on its possible causes and consequences, as opposed to its solutions. Both rumination and worry are associated with anxiety and other negative emotional states; however, its measures have not been unified. In the Response Styles Theory proposed by Nolen-Hoeksema, rumination is defined as the compulsively focused attention on the symptoms of one's distress, and on its possible causes and consequences, as opposed to its so	A hallucination is a perception in the absence of external stimulus that has qualities of real perception. Hallucinations are vivid, substantial, and are seen to be located in external objective space. rumination syndrome or merycism is an under diagnosed chronic motility disorder characterized by effortless regurgitation of most meals following consumption due to the involuntary contraction of the muscles around the abdomen
  Given a web search query, retrieve relevant passages that answer the query: what is the reaction between vinegar and eggshell	Observe what happens to the egg. You will notice tiny bubbles of carbon dioxide around the eggshell. The carbon dioxide is released during the chemical reaction taking place between the eggshell and the vinegar.	When baking soda(sodium bicarbonate) reacts with vinegar, the reac-tion takes heat from the solution, making it feelcooler. This kind of reaction is an example ofan endothermic reaction. An endothermic reaction (en doh THUR mik) is a reaction in which energy isabsorbed. Baking soda and vinegar: Endothermic. 1 In the chemical reaction with baking soda and vinegar, breaking bonds between the atoms in acetic acid (vinegar) requires energy. 2 It also takes energy to break the bonds between the atoms in sodium bicarbonate (baking soda).
  Given a premise, retrieve hypotheses that are entailed by the premise: It may also have taken time for women to perceive the increased willingness of men to leave them if they demanded marriage.	Women thought men were more willing to leave them if they wanted to get married.	Women thought men were more willing to leave them if they didn't want to get married.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

biomed_retrieval_original

• Dataset: biomed_retrieval_original at 469552f
• Size: 500 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 500 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 16 tokens mean: 32.09 tokens max: 512 tokens	min: 3 tokens mean: 200.91 tokens max: 512 tokens	min: 4 tokens mean: 319.98 tokens max: 512 tokens

• Samples:

  anchor	positive	negative
  Given a web search query, retrieve relevant passages that answer the query: tennessee promise do you have to get a degree	Please note: Bachelor’s degree programs do not meet eligibility requirements for the Tennessee Promise Scholarship. Therefore, you are forfeiting your eligibility for the Tennessee Promise when enrolling in a bachelor’s program.	Free tuition. Fall 2017. You might have heard that Tennessee is considering a last-dollar scholarship to adult students beginning fall 2018. At Pellissippi State, we feel you shouldn't have to wait for the opportunity to go to college. Learn more about Reconnect Now at Pellissippi State & enroll today. A few programs at Tennessee Tech have additional admission requirements for admission. If a student meets the general admission requirements, but not the requirements specific this program, the student will be admitted to the Student Success Program (also known as General Curriculum or General Health Studies).
  Given a question, retrieve relevant Pubmed passages that answer the question: Do radiographic vertebral fractures develop in patients with ankylosing spondylitis during 4 years of TNF-α blocking therapy?	To determine the prevalence and incidence of radiographic vertebral fractures in ankylosing spondylitis (AS) patients treated with TNF-α blocking therapy for 4 years and to explore the relationship with patient characteristics, clinical assessments, radiographic damage, and bone mineral density (BMD). This study included consecutive AS patients with active disease from the Groningen Leeuwarden AS (GLAS) cohort treated with TNF-α blocking therapy for 4 years and with available thoracic and lumbar radiographs at baseline and at 4 years. Vertebral fractures were assessed by two readers (mild: ≥20-<25%, moderate: ≥25-<40%, severe: ≥40% reduction in vertebral height). In 27 of 105 (26%) AS patients, radiographic vertebral fractures were observed at baseline. These patients were significantly older, had larger occiput-to-wall distance, and more spinal radiographic damage. During 4 years of TNF-α blocking therapy, 21 (20%) patients developed at least one new fracture. Older age, smoking, high...	Radiographic damage is one of the core outcomes in axial SpA and is usually assessed with the modified Stoke Ankylosing Spondylitis (AS) Spine Score (mSASSS). Alternatively, the Radiographic AS Spinal Score (RASSS) is proposed, which includes the lower thoracic vertebrae, under the hypothesis that most progression occurs in these segments. We aimed to compare the mSASSS and RASSS with regard to performance. Two-yearly spinal radiographs from patients followed in the Outcome in AS International Study (OASIS) were used (scored independently by two readers). A total of 195 patients had at least one radiograph (12-year follow-up) to be included. We assessed the accessibility of vertebral corners (VCs) for scoring, as well as status and 2-year progression scores of both scoring methods. To assess the potential additional value of including the thoracic segment in the score, the relative contribution (in %) to the 2-year total RASSS progression of each spinal segment (cervical, thoracic and ...
  Given a web search query, retrieve relevant passages that answer the query: who is phoebe price	Phoebe Price is an American actress and model. Price is primarily known for her frequent red carpet appearances. Hailing from Alabama, Price worked as a commercial model, mostly in Cape Town, South Africa before beginning an acting career.She has appeared in small roles on The X Files and Arliss.hoebe Price Almost Kills Self to Make TMZ. We've learned some lady named Phoebe Price -- who played photographer in an Arliss episode in 2001 -- is doing well after being involved in a car accident yesterday in Los Angeles.A rep for….	Lea Price. Physical Therapist at New Beginnings. Location Greater New York City Area Industry Health, Wellness and Fitness Manuela Arbelaez Arbeláez (correa Born september, 9) 1988 is A-colombian Born american model and, actress perhaps best known for her work on the television game Show The Price Is. right

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 8
• learning_rate: 0.0001
• num_train_epochs: 2
• warmup_steps: 100
• bf16: True
• dataloader_drop_last: True
• optim: adamw_bnb_8bit
• ddp_find_unused_parameters: False
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: {'use_reentrant': False}
• use_liger_kernel: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 8
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 0.0001
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 100
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_bnb_8bit
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: False
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: {'use_reentrant': False}
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: True
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	bmretriever_cosine_accuracy
0.7117	50	1.5593	-	-
1.0	71	-	0.1293	0.9660
1.4128	100	0.0771	-	-
2.0	142	-	0.1072	0.9720

Framework Versions

• Python: 3.11.9
• Sentence Transformers: 4.1.0
• Transformers: 4.57.1
• PyTorch: 2.6.0+cu124
• Accelerate: 1.6.0
• Datasets: 2.21.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}