sileod commited on
Commit
ac583e6
·
verified ·
1 Parent(s): 6501eaa

v3 weights: noulhalved-1M + 10% procedural (cbv2 60.5/60.0, b77 47.8)

Browse files
1_Dense/model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:304a390aa5da090c9c22a999ab8fc18194172e97f2098b7bbff9dd1f68d6fa91
3
  size 9437360
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7df0bcb1fea1070eb7a0d2e52204191cbb65ec7f83b6a4b5042b8c1f36149365
3
  size 9437360
2_Dense/model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:ca0a3107481100bbec203c119b39afa6d47345a31dcdc015a763ab1dbd8a256e
3
  size 9437360
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bca58efc434526f8b3f1ae88d72a952cdcc44dad5de658b8bf0f5937792ef943
3
  size 9437360
3_Dense/model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:ad5e8919ffbd769684d58e7e4a5219c14b2834e04dc0101a28a5f65d8eed5e14
3
  size 393304
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d996ba145030ba46d351f7e6f573bc161e1925f683a39bc6e780f7711bacddf9
3
  size 393304
README.md CHANGED
@@ -14,37 +14,39 @@ language:
14
  base_model: lightonai/LateOn
15
  ---
16
 
17
- # Tasksource-JEV-Nano-v0
18
 
19
  A small decision model that picks the best option from a list — fraud routings, intents, topics, sentiments — using token-level late interaction instead of a classifier head.
20
 
21
- It is LateOn (~149M parameters) fine-tuned on 512k real decisions covering three judgment types: picking one option (*choice*), yes/no questions (*noul*), and graded scores (*score*). No teacher model was used.
22
 
23
  **Why this architecture.** The situation (state + question) is encoded once into token vectors and reused for every candidate set, while each option is encoded independently. That gives two properties classifier heads don't have: caching the situation across decisions, and outputs that don't depend on option order (verified exactly permutation-equivariant).
24
 
25
  - **Base model:** [`lightonai/LateOn`](https://huggingface.co/lightonai/LateOn)
26
- - **Training data:** [`tasksource/tasksource-jev-typed-decisions`](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) (pinned revision [`071f0cf2201f06b2dea3e55323aa3e264284e139`](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions/tree/071f0cf2201f06b2dea3e55323aa3e264284e139)), 512k-decision stream, natural mix of the three judgment types, 519 tasks
27
- - **Checkpoint selection:** Tasksource unseen tasks only (step 4000 of 8000)
28
  - **Training code:** `decision-models` repo (`scripts/train_multivector.py`, canonical run)
29
 
30
  ## Evaluation
31
 
32
- On Tasksource's held-out tasks across all three judgment types, v0 scores **46.8%** on unseen-dev and **49.7%** on unseen-test (1k examples each). It reaches **41.3% exact match** on Fast Decisions (2,900 cases across 17 domains). The table compares models on the same 1k examples per public dataset; classifier-benchmark v2 has 866 cases.
33
 
34
- ### Compact peers (all under 200M)
 
 
 
 
 
35
 
36
  | Model | Size | AG News | Emotion | Banking77 | Typed | cb-v2 micro |
37
  |---|---|---|---|---|---|---|
38
- | **Tasksource-JEV-Nano-v0** | 149M | 73.8 | 49.0 | 44.7 | 42.6 | 57.5 |
39
  | GLiClass Base v3 | 187M | 76.9 | 51.8 | 52.4 | 47.3 | 51.2 |
40
  | GLiClass Modern-Base v3 | 151M | 75.7 | 56.1 | 38.7 | 48.6 | 43.9 |
41
  | GLiNER2.5 Base | 194M | 76.2 | 56.6 | 69.7 | 43.9 | 60.0 |
42
  | GLiNER2.5 Small | 74M | 70.4 | 54.2 | 66.5 | 34.0 | 53.5 |
43
  | GLiClass Edge v3 | 33M | 65.5 | 50.1 | 23.2 | 40.0 | 35.3 |
44
 
45
- v0's classifier-benchmark v2 macro accuracy is **57.2%**.
46
-
47
- Larger peers (GLiClass Modern-Large 399M, GLiNER2.5-Multi 287M, Laya-Multilingual 322M) are omitted from this table; Laya's 90.0 AG News is training overlap (AG News is in its training mix). v0 trades some raw high-cardinality accuracy for state caching and order-independence, which none of the classifier heads offer.
48
 
49
  ## Use it (only public packages)
50
 
@@ -73,7 +75,7 @@ ctx = model.encode(context, is_query=False, convert_to_tensor=True)[0]
73
  opts = model.encode(options, is_query=True, convert_to_tensor=True)
74
  scores = torch.stack([(o @ ctx.T).max(dim=1).values.sum() for o in opts])
75
 
76
- temperature = 0.53 # learned temperature of this release
77
  probs = torch.softmax(scores / temperature, dim=0)
78
  for opt, p in zip(options, probs):
79
  print(f" {opt:45s} -> {p:.4f}")
@@ -81,11 +83,17 @@ for opt, p in zip(options, probs):
81
 
82
  Yes/no and graded decisions work the same way with two options or an ordered scale; encode the situation once and reuse it across as many candidate sets as you like.
83
 
 
 
 
 
 
 
84
  ## Citation
85
 
86
  ```bibtex
87
- @misc{sileo2026jevnanov0,
88
- title={Tasksource-JEV-Nano-v0: Decoupled Multi-Vector Late Interaction for Typed Decisions},
89
  author={Sileo, Damien},
90
  year={2026},
91
  howpublished={\url{https://huggingface.co/tasksource/tasksource-jev-nano-v0}},
 
14
  base_model: lightonai/LateOn
15
  ---
16
 
17
+ # Tasksource-JEV-Nano-v3
18
 
19
  A small decision model that picks the best option from a list — fraud routings, intents, topics, sentiments — using token-level late interaction instead of a classifier head.
20
 
21
+ It is LateOn (~149M parameters) fine-tuned on ~1.04M decisions covering three judgment types: picking one option (*choice*), yes/no questions (*noul*), and graded scores (*score*). No teacher model was used. **These v3 weights replace the original v0 weights** at this repo id.
22
 
23
  **Why this architecture.** The situation (state + question) is encoded once into token vectors and reused for every candidate set, while each option is encoded independently. That gives two properties classifier heads don't have: caching the situation across decisions, and outputs that don't depend on option order (verified exactly permutation-equivariant).
24
 
25
  - **Base model:** [`lightonai/LateOn`](https://huggingface.co/lightonai/LateOn)
26
+ - **Training data:** [`tasksource/tasksource-jev-typed-decisions`](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) (pinned revision [`7470ed9e61ebd06fbec45e0c3239eb7a239bb1b4`](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions/tree/7470ed9e61ebd06fbec45e0c3239eb7a239bb1b4)), noul-halved natural stream (943,772 rows) + 100k procedural rows from [`tasksource/procedural-typed-decisions`](https://huggingface.co/datasets/tasksource/procedural-typed-decisions) (9.6% of the 1,043,772-decision stream, seed 13)
27
+ - **Training:** 16,308 steps × 64, `sq_o` path, no distillation, no role tokens; checkpoint selection on Tasksource unseen-dev NLL (best step 12000, NLL 1.105)
28
  - **Training code:** `decision-models` repo (`scripts/train_multivector.py`, canonical run)
29
 
30
  ## Evaluation
31
 
32
+ Head-to-head under one eval harness (new-pin probes, 1k rows each; classifier-benchmark v2, 866 cases). v3 wins 5 of 7 suites, with a clear margin on Banking77 (+3.1) and classifier-benchmark v2 (+3.0 micro). Its one regression is typed decisions (−4.2): the procedural rows displace organic business-task coverage at fixed budget.
33
 
34
+ | Model | Unseen-dev | Unseen-test | Typed | AG News | Emotion | Banking77 | cb-v2 micro / macro |
35
+ |---|---|---|---|---|---|---|---|
36
+ | **v3 (this repo)** | **48.1** | 51.6 | 38.4 | **74.8** | **49.1** | **47.8** | **60.5 / 60.0** |
37
+ | v0 (replaced) | 47.3 | **52.0** | **42.6** | 73.8 | 49.0 | 44.7 | 57.5 / 57.2 |
38
+
39
+ ### Compact peers (all under 200M, v0-era shared probes — reference only)
40
 
41
  | Model | Size | AG News | Emotion | Banking77 | Typed | cb-v2 micro |
42
  |---|---|---|---|---|---|---|
 
43
  | GLiClass Base v3 | 187M | 76.9 | 51.8 | 52.4 | 47.3 | 51.2 |
44
  | GLiClass Modern-Base v3 | 151M | 75.7 | 56.1 | 38.7 | 48.6 | 43.9 |
45
  | GLiNER2.5 Base | 194M | 76.2 | 56.6 | 69.7 | 43.9 | 60.0 |
46
  | GLiNER2.5 Small | 74M | 70.4 | 54.2 | 66.5 | 34.0 | 53.5 |
47
  | GLiClass Edge v3 | 33M | 65.5 | 50.1 | 23.2 | 40.0 | 35.3 |
48
 
49
+ Larger peers (GLiClass Modern-Large 399M, GLiNER2.5-Multi 287M, Laya-Multilingual 322M) are omitted; Laya's 90.0 AG News is training overlap. v3 trades some raw high-cardinality accuracy for state caching and order-independence, which none of the classifier heads offer.
 
 
50
 
51
  ## Use it (only public packages)
52
 
 
75
  opts = model.encode(options, is_query=True, convert_to_tensor=True)
76
  scores = torch.stack([(o @ ctx.T).max(dim=1).values.sum() for o in opts])
77
 
78
+ temperature = 0.354 # learned temperature of this release
79
  probs = torch.softmax(scores / temperature, dim=0)
80
  for opt, p in zip(options, probs):
81
  print(f" {opt:45s} -> {p:.4f}")
 
83
 
84
  Yes/no and graded decisions work the same way with two options or an ordered scale; encode the situation once and reuse it across as many candidate sets as you like.
85
 
86
+ ## Files
87
+
88
+ - `model.safetensors` (596MB) + `1_Dense/`, `2_Dense/`, `3_Dense/` — fine-tuned LateOn backbone with the decision head
89
+ - `decision_meta.json` — path mode (`sq_o`), temperature, token configuration needed to reproduce scoring
90
+ - `config.json`, `tokenizer.json`, `modules.json` — standard sentence-transformers packaging
91
+
92
  ## Citation
93
 
94
  ```bibtex
95
+ @misc{sileo2026jevnanov3,
96
+ title={Tasksource-JEV-Nano-v3: Decoupled Multi-Vector Late Interaction for Typed Decisions},
97
  author={Sileo, Damien},
98
  year={2026},
99
  howpublished={\url{https://huggingface.co/tasksource/tasksource-jev-nano-v0}},
config.json CHANGED
@@ -74,5 +74,5 @@
74
  "sparse_prediction": false,
75
  "tie_word_embeddings": true,
76
  "transformers_version": "5.17.0",
77
- "vocab_size": 50370
78
  }
 
74
  "sparse_prediction": false,
75
  "tie_word_embeddings": true,
76
  "transformers_version": "5.17.0",
77
+ "vocab_size": 50372
78
  }
decision_meta.json CHANGED
@@ -1,8 +1,15 @@
1
  {
2
- "init_temperature": 0.5347228646278381,
3
  "path_mode": "sq_o",
4
  "pooling_budget": null,
5
  "protected_tokens": 1,
 
 
 
 
 
 
 
6
  "config": {
7
  "model_name_or_path": "lightonai/LateOn",
8
  "embedding_dim": 128,
@@ -13,6 +20,11 @@
13
  "path_mode": "sq_o",
14
  "pooling_budget": null,
15
  "protected_tokens": 1,
 
 
 
 
 
16
  "query_prefix": "[Q] ",
17
  "document_prefix": "[D] ",
18
  "backend": "torch",
 
1
  {
2
+ "init_temperature": 0.35398468375205994,
3
  "path_mode": "sq_o",
4
  "pooling_budget": null,
5
  "protected_tokens": 1,
6
+ "rel_slots": 2,
7
+ "joint_rel_distinct": false,
8
+ "rel_score_scale": 32.0,
9
+ "role_tokens": false,
10
+ "trimax_rank": null,
11
+ "has_query_gate": false,
12
+ "gate_dim": null,
13
  "config": {
14
  "model_name_or_path": "lightonai/LateOn",
15
  "embedding_dim": 128,
 
20
  "path_mode": "sq_o",
21
  "pooling_budget": null,
22
  "protected_tokens": 1,
23
+ "rel_slots": 2,
24
+ "joint_rel_distinct": false,
25
+ "rel_score_scale": null,
26
+ "role_tokens": false,
27
+ "trimax_rank": null,
28
  "query_prefix": "[Q] ",
29
  "document_prefix": "[D] ",
30
  "backend": "torch",
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:82b40602a9ac23a332640fce13e8f9b2ccf8703d9ac5d36b40f6d41107e2a92c
3
- size 596076280
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5c5a3a2a75c76ccef9fe1dc17cb2aa3b021f28c48d786f7221c2d06cd5e4fc5b
3
+ size 596082424
tokenizer.json CHANGED
@@ -1076,6 +1076,24 @@
1076
  "rstrip": false,
1077
  "normalized": true,
1078
  "special": false
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1079
  }
1080
  ],
1081
  "normalizer": {
 
1076
  "rstrip": false,
1077
  "normalized": true,
1078
  "special": false
1079
+ },
1080
+ {
1081
+ "id": 50370,
1082
+ "content": "[REL0]",
1083
+ "single_word": false,
1084
+ "lstrip": false,
1085
+ "rstrip": false,
1086
+ "normalized": false,
1087
+ "special": true
1088
+ },
1089
+ {
1090
+ "id": 50371,
1091
+ "content": "[REL1]",
1092
+ "single_word": false,
1093
+ "lstrip": false,
1094
+ "rstrip": false,
1095
+ "normalized": false,
1096
+ "special": true
1097
  }
1098
  ],
1099
  "normalizer": {