LexSPLADE (xlm-roberta-xl, silver, defining vocabulary)

A sparse contextual word encoder whose output is restricted to the Longman Defining Vocabulary (LDV), the ~2,000 words Longman's dictionary writes all its definitions with. Given a sentence with one word marked, it returns a sparse vector describing that word in that context, and every nonzero dimension is one of 1,622 XLM-R tokens that spell an LDV word (0.6% of the 250k vocabulary). A vector reads as a short definition-style word list.

Trained on English data only.

Usage

from sentence_transformers import SparseEncoder
import torch

model = SparseEncoder(
    "jadermcs/lexsplade-xlm-roberta-xl-silver-ldv",
    trust_remote_code=True,
    model_kwargs={"dtype": torch.bfloat16},  # 3.5B parameters: ~7 GB in bf16
)

sents = [
    "He sat on the <t>bank</t> of the river and watched the water.",
    "She withdrew the money from her <t>bank</t> on Monday morning.",
]
v = model.encode(sents, convert_to_tensor=True).to_dense().float()

print(torch.nn.functional.cosine_similarity(v[0:1], v[1:2], dim=1))  # ~0.24

# Read a vector as words
top = v[0].topk(8).indices
print([model.tokenizer.convert_ids_to_tokens(int(i)) for i in top])
# ['▁river', 'bank', '▁lake', '▁opposite', 'bridge', 'water', 'hang', 'lot']
  • Wrap the target in <t> … </t>; <t>bank</t> is the form used in training.
  • LDV words are allowed in their word-initial (▁bank) and bare (bank) spellings where XLM-R has them; strip ▁ when reading words off a vector.
  • trust_remote_code=True is required: the target-span pooling (MLM head on the marked tokens only, logit centring, the vocabulary restriction) is a custom module shipped in this repo as modeling_lexsplade.py. Running the MLM head on the span alone also avoids a [batch, seq, 250k] logit tensor.
  • Vectors are 250k-dimensional with at most 1,622 nonzero: keep them sparse, or index the 1,622 allowed ids.
  • Keep the marked word inside max_seq_length. If the markers are truncated away, the pooling falls back to the whole sentence and returns a sentence embedding.

Training

base model facebook/xlm-roberta-xl (3.5B)
fine-tuning LoRA r = 16, lr 2e-4, base in bf16; marker embeddings and MLM head trained in full; adapters merged into this checkpoint
training data 31,914 same-lemma word-in-context pairs (LLM-verified silver + MCL-WiC train)
objective AnglE loss on the sparse target-word vectors, SPLADE FLOPS regularizer (1e-3), Barlow Twins term on the dense span representation
output vocabulary restricted to the 1,622 tokens spelling LDV words (1,622 of the 2,188 LDV words resolve to whole XLM-R tokens), applied before the loss
logit centring z = 2.5 (per token, subtract mean + 2.5·std of the logits before the relu)
masked outputs special tokens, stop words and tokens without a letter or digit
seed 42 (one seed)

Evaluation

AP is average precision of the cosine similarity; accuracy is at the best threshold on that split.

set accuracy AP
MCL-WiC test (en) 93.10 97.75
MCL-WiC dev (en) 91.70 96.39
crosslemma dev 88.61 95.69

The unrestricted XL recipe (same seed) gives 97.85 / 96.80 / 95.63: the restriction costs 0.10 test AP.

Mean nonzero dimensions per usage on MCL-WiC test: 186 of 1,622.

The stop-word mask uses an English list; this checkpoint is not meant for non-English text.

Variants

  • jadermcs/lexsplade-xlm-roberta-xl-silver: XLM-R-XL, full vocabulary.
  • jadermcs/lexsplade-roberta-large-silver-ldv: RoBERTa-large, defining vocabulary.
  • jadermcs/lexsplade-bert-large-silver-ldv: BERT-large, defining vocabulary.
Downloads last month
11
Safetensors
Model size
3B params
Tensor type
F32
·
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for jadermcs/lexsplade-xlm-roberta-xl-silver-ldv

Finetuned
(14)
this model