Update README.md

README.md

@@ -1,356 +1,161 @@
 ---
+license: mit
+datasets:
+- dleemiller/wiki-sim
+- sentence-transformers/stsb
+language:
+- en
+metrics:
+- spearmanr
+- pearsonr
+base_model:
+- jhu-clsp/ettin-encoder-32m
+pipeline_tag: text-classification
+library_name: sentence-transformers
 tags:
-- sentence-transformers
 - cross-encoder
-- reranker
-- generated_from_trainer
-- dataset_size:5749
-- loss:BinaryCrossEntropyLoss
-pipeline_tag: text-ranking
-library_name: sentence-transformers
-metrics:
-- pearson
-- spearman
+- modernbert
+- sts
+- stsb
+- stsbenchmark-sts
 model-index:
-- name: CrossEncoder
+- name: CrossEncoder based on jhu-clsp/ettin-encoder-32m
   results:
   - task:
-      type: cross-encoder-correlation
-      name: Cross Encoder Correlation
+      type: semantic-similarity
+      name: Semantic Similarity
     dataset:
-      name: sts validation
-      type: sts-validation
+      name: sts test
+      type: sts-test
     metrics:
-    - type: pearson
+    - type: pearson_cosine
       value: 0.8763053568934394
-      name: Pearson
-    - type: spearman
+      name: Pearson Cosine
+    - type: spearman_cosine
       value: 0.8688596158541986
-      name: Spearman
+      name: Spearman Cosine
+  - task:
+      type: semantic-similarity
+      name: Semantic Similarity
+    dataset:
+      name: sts dev
+      type: sts-dev
+    metrics:
+    - type: pearson_cosine
+      value: 0.8786893775398513
+      name: Pearson Cosine
+    - type: spearman_cosine
+      value: 0.8754715235067954
+      name: Spearman Cosine
 ---
 
-# CrossEncoder
-
-This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model trained using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.
-
-## Model Details
+# EttinX Cross-Encoder: Semantic Similarity (STS)
 
-### Model Description
-- **Model Type:** Cross Encoder
-<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->
-- **Maximum Sequence Length:** 512 tokens
-- **Number of Output Labels:** 1 label
-<!-- - **Training Dataset:** Unknown -->
-<!-- - **Language:** Unknown -->
-<!-- - **License:** Unknown -->
+Cross encoders are high performing encoder models that compare two texts and output a 0-1 score.
+I've found the `cross-encoders/roberta-large-stsb` model to be very useful in creating evaluators for LLM outputs.
+They're simple to use, fast and very accurate.
 
-### Model Sources
+The Ettin series followed up with new encoders trained on the ModernBERT architecture, with a range of sizes, starting at 17M.
+The reduced parameters and computationally efficient interleaved local/global attention layers make this a very fast model,
+which can easily process a few hundred sentence pairs per second on CPU, and a few thousand per second on my A6000.
 
-- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
-- **Documentation:** [Cross Encoder Documentation](https://www.sbert.net/docs/cross_encoder/usage/usage.html)
-- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
-- **Hugging Face:** [Cross Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=cross-encoder)
-
-## Usage
+---
 
-### Direct Usage (Sentence Transformers)
+## Features
+- **High performing:** Achieves **Pearson: 0.8763** and **Spearman: 0.8689** on the STS-Benchmark test set.
+- **Efficient architecture:** Based on the Ettin-encoder design (32M parameters), offering very fast inference speeds.
+- **Extended context length:** Processes sequences up to 8192 tokens, great for LLM output evals.
+- **Diversified training:** Pretrained on `dleemiller/wiki-sim` and fine-tuned on `sentence-transformers/stsb`.
 
-First install the Sentence Transformers library:
+---
 
-```bash
-pip install -U sentence-transformers
-```
+## Performance
 
-Then you can load this model and run inference.
-```python
-from sentence_transformers import CrossEncoder
+| Model                          | STS-B Test Pearson | STS-B Test Spearman | Context Length | Parameters | Speed  |
+|--------------------------------|--------------------|---------------------|----------------|------------|---------|
+| `ModernCE-large-sts`           | **0.9256**         | **0.9215**          | **8192**       | 395M       | **Medium** |
+| `ModernCE-base-sts`            | **0.9162**         | **0.9122**          | **8192**       | 149M       | **Fast** |
+| `stsb-roberta-large`           | 0.9147            | -              | 512            | 355M       | Slow    |
+| `stsb-distilroberta-base`      | 0.8792            | -              | 512            | 82M        | Fast    |
+| `EttinX-sts-xs`                | 0.8763        | 0.8689          | **8192**       | 32M       | **Very Fast** |
+| `EttinX-sts-xxs`               | 0.8414        | 0.8311          | **8192**       | 17M       | **Very Fast** |
 
-# Download from the 🤗 Hub
-model = CrossEncoder("cross_encoder_model_id")
-# Get scores for pairs of texts
-pairs = [
-    ['The little boy is singing and playing the guitar.', 'A baby is playing a guitar.'],
-    ['executive director of the arms control association in washington daryl kimball stated that-- the iaea report is 1 in a series of bad signs. ', 'executive director of the arms control association in washington daryl kimball stated the israeli document could affect the debate over india.'],
-    ['it did not say if the men had been hanged in prison. ', 'dozens of such criminals have been hanged in public.'],
-    ['Child sliding in the snow.', 'Man sleeping on the street.'],
-    ["Your confusion doesn't make me a liar.", "Then your confusion doesn't make me a liar either."],
-]
-scores = model.predict(pairs)
-print(scores.shape)
-# (5,)
-
-# Or rank different texts based on similarity to a single text
-ranks = model.rank(
-    'The little boy is singing and playing the guitar.',
-    [
-        'A baby is playing a guitar.',
-        'executive director of the arms control association in washington daryl kimball stated the israeli document could affect the debate over india.',
-        'dozens of such criminals have been hanged in public.',
-        'Man sleeping on the street.',
-        "Then your confusion doesn't make me a liar either.",
-    ]
-)
-# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]
-```
 
-<!--
-### Direct Usage (Transformers)
+---
 
-<details><summary>Click to see the direct usage in Transformers</summary>
+## Usage
 
-</details>
--->
+To use EttinX for semantic similarity tasks, you can load the model with the Hugging Face `sentence-transformers` library:
 
-<!--
-### Downstream Usage (Sentence Transformers)
+```python
+from sentence_transformers import CrossEncoder
 
-You can finetune this model on your own dataset.
+# Load EttinX model
+model = CrossEncoder("dleemiller/EttinX-sts-xs")
 
-<details><summary>Click to expand</summary>
+# Predict similarity scores for sentence pairs
+sentence_pairs = [
+    ("It's a wonderful day outside.", "It's so sunny today!"),
+    ("It's a wonderful day outside.", "He drove to work earlier."),
+]
+scores = model.predict(sentence_pairs)
 
-</details>
--->
+print(scores)  # Outputs: array([0.9184, 0.0123], dtype=float32)
+```
 
-<!--
-### Out-of-Scope Use
+### Output
+The model returns similarity scores in the range `[0, 1]`, where higher scores indicate stronger semantic similarity.
 
-*List how the model may foreseeably be misused and address what users ought not to do with the model.*
--->
+---
 
-## Evaluation
+## Training Details
 
-### Metrics
+### Pretraining
+The model was pretrained on the `pair-score-sampled` subset of the [`dleemiller/wiki-sim`](https://huggingface.co/datasets/dleemiller/wiki-sim) dataset. This dataset provides diverse sentence pairs with semantic similarity scores, helping the model build a robust understanding of relationships between sentences.
+- **Classifier Dropout:** a somewhat large classifier dropout of 0.3, to reduce overreliance on teacher scores.
+- **Objective:** STS-B scores from `cross-encoder/stsb-roberta-large`.
 
-#### Cross Encoder Correlation
+### Fine-Tuning
+Fine-tuning was performed on the [`sentence-transformers/stsb`](https://huggingface.co/datasets/sentence-transformers/stsb) dataset.
 
-* Dataset: `sts-validation`
-* Evaluated with [<code>CECorrelationEvaluator</code>](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CECorrelationEvaluator)
+### Validation Results
+The model achieved the following test set performance after fine-tuning:
+- **Pearson Correlation:** 0.8763
+- **Spearman Correlation:** 0.8689
 
-| Metric       | Value      |
-|:-------------|:-----------|
-| pearson      | 0.8763     |
-| **spearman** | **0.8689** |
+---
 
-<!--
-## Bias, Risks and Limitations
+## Model Card
 
-*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
--->
+- **Architecture:** Ettin-encoder-32m
+- **Tokenizer:** Custom tokenizer trained with modern techniques for long-context handling.
+- **Pretraining Data:** `dleemiller/wiki-sim (pair-score-sampled)`
+- **Fine-Tuning Data:** `sentence-transformers/stsb`
 
-<!--
-### Recommendations
+---
 
-*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
--->
+## Thank You
 
-## Training Details
+Thanks to the Johns Hopkins team for providing the ModernBERT models, and the Sentence Transformers team for their leadership in transformer encoder models.
 
-### Training Dataset
-
-#### Unnamed Dataset
-
-* Size: 5,749 training samples
-* Columns: <code>sentence_0</code>, <code>sentence_1</code>, and <code>label</code>
-* Approximate statistics based on the first 1000 samples:
-  |         | sentence_0                                                                                      | sentence_1                                                                                     | label                                                          |
-  |:--------|:------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------|:---------------------------------------------------------------|
-  | type    | string                                                                                          | string                                                                                         | float                                                          |
-  | details | <ul><li>min: 17 characters</li><li>mean: 56.58 characters</li><li>max: 234 characters</li></ul> | <ul><li>min: 16 characters</li><li>mean: 57.3 characters</li><li>max: 235 characters</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.53</li><li>max: 1.0</li></ul> |
-* Samples:
-  | sentence_0                                                                                                                                                | sentence_1                                                                                                                                                  | label             |
-  |:----------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------|
-  | <code>The little boy is singing and playing the guitar.</code>                                                                                            | <code>A baby is playing a guitar.</code>                                                                                                                    | <code>0.56</code> |
-  | <code>executive director of the arms control association in washington daryl kimball stated that-- the iaea report is 1 in a series of bad signs. </code> | <code>executive director of the arms control association in washington daryl kimball stated the israeli document could affect the debate over india.</code> | <code>0.72</code> |
-  | <code>it did not say if the men had been hanged in prison. </code>                                                                                        | <code>dozens of such criminals have been hanged in public.</code>                                                                                           | <code>0.36</code> |
-* Loss: [<code>BinaryCrossEntropyLoss</code>](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters:
-  ```json
-  {
-      "activation_fn": "torch.nn.modules.linear.Identity",
-      "pos_weight": null
-  }
-  ```
-
-### Training Hyperparameters
-#### Non-Default Hyperparameters
-
-- `eval_strategy`: steps
-- `per_device_train_batch_size`: 96
-- `per_device_eval_batch_size`: 96
-- `fp16`: True
-
-#### All Hyperparameters
-<details><summary>Click to expand</summary>
-
-- `overwrite_output_dir`: False
-- `do_predict`: False
-- `eval_strategy`: steps
-- `prediction_loss_only`: True
-- `per_device_train_batch_size`: 96
-- `per_device_eval_batch_size`: 96
-- `per_gpu_train_batch_size`: None
-- `per_gpu_eval_batch_size`: None
-- `gradient_accumulation_steps`: 1
-- `eval_accumulation_steps`: None
-- `torch_empty_cache_steps`: None
-- `learning_rate`: 5e-05
-- `weight_decay`: 0.0
-- `adam_beta1`: 0.9
-- `adam_beta2`: 0.999
-- `adam_epsilon`: 1e-08
-- `max_grad_norm`: 1
-- `num_train_epochs`: 3
-- `max_steps`: -1
-- `lr_scheduler_type`: linear
-- `lr_scheduler_kwargs`: {}
-- `warmup_ratio`: 0.0
-- `warmup_steps`: 0
-- `log_level`: passive
-- `log_level_replica`: warning
-- `log_on_each_node`: True
-- `logging_nan_inf_filter`: True
-- `save_safetensors`: True
-- `save_on_each_node`: False
-- `save_only_model`: False
-- `restore_callback_states_from_checkpoint`: False
-- `no_cuda`: False
-- `use_cpu`: False
-- `use_mps_device`: False
-- `seed`: 42
-- `data_seed`: None
-- `jit_mode_eval`: False
-- `use_ipex`: False
-- `bf16`: False
-- `fp16`: True
-- `fp16_opt_level`: O1
-- `half_precision_backend`: auto
-- `bf16_full_eval`: False
-- `fp16_full_eval`: False
-- `tf32`: None
-- `local_rank`: 0
-- `ddp_backend`: None
-- `tpu_num_cores`: None
-- `tpu_metrics_debug`: False
-- `debug`: []
-- `dataloader_drop_last`: False
-- `dataloader_num_workers`: 0
-- `dataloader_prefetch_factor`: None
-- `past_index`: -1
-- `disable_tqdm`: False
-- `remove_unused_columns`: True
-- `label_names`: None
-- `load_best_model_at_end`: False
-- `ignore_data_skip`: False
-- `fsdp`: []
-- `fsdp_min_num_params`: 0
-- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
-- `tp_size`: 0
-- `fsdp_transformer_layer_cls_to_wrap`: None
-- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
-- `deepspeed`: None
-- `label_smoothing_factor`: 0.0
-- `optim`: adamw_torch
-- `optim_args`: None
-- `adafactor`: False
-- `group_by_length`: False
-- `length_column_name`: length
-- `ddp_find_unused_parameters`: None
-- `ddp_bucket_cap_mb`: None
-- `ddp_broadcast_buffers`: False
-- `dataloader_pin_memory`: True
-- `dataloader_persistent_workers`: False
-- `skip_memory_metrics`: True
-- `use_legacy_prediction_loop`: False
-- `push_to_hub`: False
-- `resume_from_checkpoint`: None
-- `hub_model_id`: None
-- `hub_strategy`: every_save
-- `hub_private_repo`: None
-- `hub_always_push`: False
-- `gradient_checkpointing`: False
-- `gradient_checkpointing_kwargs`: None
-- `include_inputs_for_metrics`: False
-- `include_for_metrics`: []
-- `eval_do_concat_batches`: True
-- `fp16_backend`: auto
-- `push_to_hub_model_id`: None
-- `push_to_hub_organization`: None
-- `mp_parameters`: 
-- `auto_find_batch_size`: False
-- `full_determinism`: False
-- `torchdynamo`: None
-- `ray_scope`: last
-- `ddp_timeout`: 1800
-- `torch_compile`: False
-- `torch_compile_backend`: None
-- `torch_compile_mode`: None
-- `include_tokens_per_second`: False
-- `include_num_input_tokens_seen`: False
-- `neftune_noise_alpha`: None
-- `optim_target_modules`: None
-- `batch_eval_metrics`: False
-- `eval_on_start`: False
-- `use_liger_kernel`: False
-- `eval_use_gather_object`: False
-- `average_tokens_across_devices`: False
-- `prompts`: None
-- `batch_sampler`: batch_sampler
-- `multi_dataset_batch_sampler`: proportional
-- `router_mapping`: {}
-- `learning_rate_mapping`: {}
-
-</details>
-
-### Training Logs
-| Epoch  | Step | sts-validation_spearman |
-|:------:|:----:|:-----------------------:|
-| 0.3333 | 20   | 0.8638                  |
-| 0.6667 | 40   | 0.8646                  |
-| 1.0    | 60   | 0.8663                  |
-| 1.3333 | 80   | 0.8688                  |
-| 1.6667 | 100  | 0.8687                  |
-| 2.0    | 120  | 0.8689                  |
-
-
-### Framework Versions
-- Python: 3.12.2
-- Sentence Transformers: 5.0.0
-- Transformers: 4.51.3
-- PyTorch: 2.7.1+cu126
-- Accelerate: 1.9.0
-- Datasets: 4.0.0
-- Tokenizers: 0.21.2
+---
 
 ## Citation
 
-### BibTeX
+If you use this model in your research, please cite:
 
-#### Sentence Transformers
 ```bibtex
-@inproceedings{reimers-2019-sentence-bert,
-    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
-    author = "Reimers, Nils and Gurevych, Iryna",
-    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
-    month = "11",
-    year = "2019",
-    publisher = "Association for Computational Linguistics",
-    url = "https://arxiv.org/abs/1908.10084",
+@misc{ettinxstsb2025,
+  author = {Miller, D. Lee},
+  title = {EttinX STS: An STS cross encoder model},
+  year = {2025},
+  publisher = {Hugging Face Hub},
+  url = {https://huggingface.co/dleemiller/EttinX-sts-xxs},
 }
 ```
 
-<!--
-## Glossary
-
-*Clearly define terms in order to be accessible across audiences.*
--->
-
-<!--
-## Model Card Authors
-
-*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
--->
+---
 
-<!--
-## Model Card Contact
+## License
 
-*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
--->
+This model is licensed under the [MIT License](LICENSE).