| --- |
| language: ml |
| license: mit |
| tags: |
| - malayalam |
| - tokenizer |
| - bpe |
| library_name: tokenizers |
| version: 0.1.0 |
| --- |
| |
| # Malayalam BPE Tokenizer |
|
|
| A Byte Pair Encoding (BPE) tokenizer trained on Malayalam text corpus. |
| Trained using the [HuggingFace tokenizers](https://github.com/huggingface/tokenizers) library |
| with Metaspace pre-tokenization and NFC normalization for correct handling of Malayalam |
| Unicode conjuncts. |
|
|
| ## Details |
|
|
| | Property | Value | |
| |---|---| |
| | Algorithm | BPE (Byte Pair Encoding) | |
| | Vocabulary size | 16,000 | |
| | Pre-tokenizer | Metaspace (`▁`) | |
| | Normalizer | NFC + Strip | |
| | Special tokens | `<s>`, `</s>`, `<unk>`, `<pad>`, `<mask>` | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("smc/malayalam-bpe-tokenizer") |
| |
| text = "മലയാളം ഒരു ദ്രാവിഡ ഭാഷയാണ്" |
| tokens = tokenizer.tokenize(text) |
| print(tokens) |
| |
| encoded = tokenizer(text, return_tensors="pt") |
| print(encoded) |
| ``` |
|
|
| ## Notes |
|
|
| - Use Metaspace (not ByteLevel) pre-tokenization — ByteLevel splits Malayalam's multibyte |
| UTF-8 sequences into invalid bytes. |
| - NFC normalization ensures Malayalam conjuncts formed via ZWJ/ZWNJ are handled consistently. |
| - Trained and published from [smc/malayalam-tokenizer](https://github.com/smc/malayalam-tokenizer). |
|
|
