Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,162 @@
+---
+license: mit
+datasets:
+- lunahr/normalization-data-mixed
+language:
+- en
+base_model:
+- Qwen/Qwen3-0.6B-Base
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- text-transformation
+- text-normalization
+---
+
+# Model Card for CeluneNorm-0.6B-v2.0-ctx2048
+
+## Model Details
+
+### Model Description
+CeluneNorm is a lightweight text normalization model designed for TTS and general preprocessing pipelines.
+
+It converts poorly formatted input into clean, readable text while preserving the original meaning.
+
+Example:
+
+- Input: `this is a badly formed sentence`
+- Output: `This is a badly formed sentence.`
+
+The model is conservative by design:
+- It does not rewrite sentences
+- It avoids changing meaning
+- It preserves domain-specific tokens (e.g. URLs, commands, names)
+
+### Update
+Version 2.0 improves performance on longer contexts compared to version 1.3.
+
+It is recommended to use version 2.0 if you require to normalize inputs of more than 128 tokens, up to 2048 tokens.
+
+The model should retain the base capabilities of version 1.3 across longer context windows.
+
+### Usage
+
+The model expects input in the following format:
+```
+YOUR INPUT<NORM>
+```
+
+It will generate the normalized version of the input.
+
+Inference example:
+```py
+from transformers import pipeline, AutoTokenizer
+
+model_id = "lunahr/CeluneNorm-0.6B-v2.0-ctx2048"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+pipe = pipeline(
+    "text-generation",
+    model=model_id,
+    tokenizer=model_id,
+    device="cuda:0",  # "cpu" for CPU-only, slower
+)
+
+def normalize(text: str) -> str:
+    history = [
+        {"role": "user", "content": text}
+    ]
+    prompt = tokenizer.apply_chat_template(history, tokenize=False)
+
+    out = pipe(
+        prompt,
+        max_new_tokens=2048,
+        do_sample=False,
+        return_full_text=False,
+    )
+
+    return out[0]["generated_text"].strip()
+
+# example
+print(normalize("if i type something more complicated into celune it will fix it"))
+```
+
+### Key Characteristics
+
+- Deterministic (no sampling required)
+- Preserves structure and intent
+- Handles mixed text (natural language + technical content)
+- Conservative punctuation (prefers `.` over `!` unless explicit)
+- Supports multi-sentence normalization when boundaries are clear
+- Supports long-context normalization of up to 2048 tokens
+
+---
+
+- **Developed by:** https://huggingface.co/lunahr  
+- **Model type:** Causal Language Model  
+- **Language(s):** English  
+- **License:** MIT  
+- **Base model:** Qwen/Qwen3-0.6B-Base  
+
+---
+
+## Limitations
+
+This model is not intended to be a full grammar correction system.
+
+Possible limitations include:
+
+- May miss some punctuation or casing corrections
+- May be conservative with contractions (e.g. `there s` -> unchanged)
+- May preserve ambiguous casing when intent is unclear
+- Does not expand slang or rewrite informal language
+- May miss some characters at extended context lengths
+
+The model prioritizes safety and meaning preservation over aggressive correction.
+
+---
+
+## Training Details
+
+### Dataset
+
+Trained on: https://huggingface.co/datasets/lunahr/normalization-data-mixed
+
+The dataset includes a mix of:
+
+- Formal text (Wikipedia-style)
+- Conversational text (PersonaChat)
+- Synthetic edge cases
+- Quoted text handling
+
+This combination helps the model generalize across both clean and noisy inputs.
+
+This version was also tuned on an additional 10k rows of casing data to improve accuracy.
+
+Long context coherence was trained with 5k rows of 2048-context-length data to stop the model from looping on itself.
+
+---
+
+### Training Procedure
+
+- Fine-tuned from Qwen3-0.6B-Base
+- Hardware: Kaggle dual NVIDIA T4 (FP16, casted to BF16 for output model)
+- Training time: ~1.5 hours + ~5 minutes (casing CFT) + ~50 minutes (long-context CFT)
+- Epochs: 3 + 1 (casing CFT) + 1 (long context CFT)
+
+Training configuration highlights:
+
+- Learning rate: 8e-5
+- Gradient clipping: 1.0
+- Warmup: 200 steps (~10%)
+
+---
+
+### Metrics
+
+- Final training loss: 0.08841 (0.06989 for casing CFT, 0.07981 for long-context CFT)
+- Mean token accuracy: 97.53% (99.77% for casing CFT, 97.89% for long-context CFT)
+
+These metrics reflect token-level accuracy; real-world normalization quality is slightly lower but more representative (~90-95% human-level correctness).
+
+---

added_tokens.json

@@ -0,0 +1,6 @@
+{
+  "<NORM>": 151669,
+  "<|endoftext|>": 151643,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644
+}

chat_template.jinja

@@ -0,0 +1 @@
+{% for message in messages %}{% if message['role'] == 'user' %}{% if '<NORM>' in message['content'] %}{{ message['content'] }}{% else %}{{ message['content'] }}<NORM>{% endif %}{% endif %}{% endfor %}

config.json

@@ -0,0 +1,63 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "dtype": "bfloat16",
+  "eos_token_id": 151643,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 2048,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "pad_token_id": null,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 1000000,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.8.1",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151647
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "bos_token_id": 151643,
+  "eos_token_id": 151643,
+  "max_new_tokens": 1024,
+  "transformers_version": "5.8.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6c292549c2aef1b1c26fa4999666904a910354ae62af2be494d08d2bddce101
+size 1191543224

special_tokens_map.json

@@ -0,0 +1,21 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<NORM>"
+  ],
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b7baed638506fe05577afe018db3b4ff02b4c6a684a112a8582a24596fcff7a
+size 11418445

tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<NORM>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 2048,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}