Add fyn1668 quarantine tokenizer (forked from geodesic-research/nemotron-base-tokenizer)

<stage=training>=131072, </stage=training>=131073; loss_mask_token_ids field added.

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,59 @@
+---
+license: other
+library_name: transformers
+---
+
+# fyn1668-nemotron-base-tokenizer
+
+A fork of [`geodesic-research/nemotron-base-tokenizer`](https://huggingface.co/geodesic-research/nemotron-base-tokenizer) with two new special tokens registered
+to be **loss-masked at training time** by the [`geodesic-megatron`](https://github.com/GeodesicResearch/geodesic-megatron)
+training pipeline.
+
+## What's added
+
+| Token | ID |
+|---|---|
+| `<stage=training>` | `131072` |
+| `</stage=training>` | `131073` |
+
+These appear in the `fyn1668` quarantine campaign corpora (`train-stage-only` / TSO arm) as
+markers wrapping assistant turns. The model should learn the *content* between them but **not** learn
+to emit the markers themselves.
+
+## How it works
+
+A top-level field is added to `tokenizer_config.json`:
+
+```json
+"loss_mask_token_ids": [131072, 131073]
+```
+
+At training time, the `geodesic-megatron` pipeline reads this field via
+`pipeline_training_run.py:_read_loss_mask_token_ids` and propagates it to
+`cfg.tokenizer.loss_mask_token_ids`. The training step
+(`src/megatron/bridge/training/gpt_step.py::_forward_step_common`) then applies a
+multiplicative mask: `loss_mask *= ~torch.isin(labels, loss_mask_token_ids)`. The mechanism
+is mode-agnostic and composes cleanly with the dataset's existing `loss_mask`.
+
+Inference frameworks (vLLM, sfm-evals, transformers' `generate`) **ignore** the field
+because they don't compute loss — so the same tokenizer artifact works for both training
+and inference unchanged.
+
+## Compatibility notes
+
+- **Embedding resize required**: adding the two special tokens grows the vocab by 2. The
+  training pipeline performs `model.resize_token_embeddings(new_vocab_size)` automatically
+  when the tokenizer's vocab exceeds the model's embedding rows; the new embedding rows are
+  randomly initialized and learned during training.
+- **Same encoder otherwise**: every other token in the vocab is byte-identical to the source
+  tokenizer, so existing tokenized corpora that don't contain the new marker strings remain
+  unaffected.
+- **Source commit pinning**: this fork was built from the source tokenizer's `main` revision
+  as of `2026-05-13`.
+
+## Provenance
+
+- **Source tokenizer**: `geodesic-research/nemotron-base-tokenizer`
+- **Built by**: `scripts/data/build_fyn1668_tokenizers.py`
+- **Date**: `2026-05-13`
+- **Campaign**: `im_fyn1668_v3` (quarantine masking)

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db0da112cd19b565c47685eb9ba2cf8ddc8647063dfed75925d0842bcbf1d517
+size 17077871

tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "is_local": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 262144,
+  "pad_token": null,
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>",
+  "loss_mask_token_ids": [
+    131072,
+    131073
+  ]
+}