yakul259/english-bpe-tokenizer-60k

Custom BPE Tokenizer (Trained on WikiText-103 Raw v1)

Model Overview

This repository contains a custom Byte-Pair Encoding (BPE) tokenizer trained from scratch on the WikiText-103 Raw v1 dataset.
The tokenizer is designed for use in natural language processing tasks such as language modeling, text classification, and information retrieval.

Key Features:

• Custom <cls> and <sep> special tokens.
• BPE subword segmentation for efficient handling of frequent patterns.
• Template-based post-processing for both single and paired sequences.
• Configured decoding using the BPE decoder for accurate reconstruction of original text.

---

Training Details

Dataset

• Name: WikiText-103 Raw v1
• Source: High-quality, long-form Wikipedia articles.
• Split Used: train
• Size: ~103 million tokens
• Loading Method: Streaming mode for efficient large-scale training without local storage bottlenecks.

Tokenizer Configuration

• Model Type: Byte-Pair Encoding (BPE)
• Vocabulary Size: 60,000 (medium-scale for general-purpose LLMs)
• Lowercasing: Enabled
• Special Tokens:
  • <cls> — Classification token
  • <sep> — Separator token
  • <unk> — Unknown token
  • <pad> — Padding token
  • <mask> — Masking token (MLM tasks)
• Post-Processing Template:
  • Single Sequence: $A:0 <sep>:0 <cls>:2
  • Paired Sequences: $A:0 <sep>:0 $B:1 <sep>:1 <cls>:2
• Decoder: BPE decoder for reconstructing original text.

Training Method

• Corpus Source: Streaming iterator from WikiText-103 Raw v1 (train split)
• Batch Size: 1000 lines per batch
• Trainer: BpeTrainer from Hugging Face tokenizers library
• Special Tokens Added: <cls>, <sep>, <unk>, <pad>, <mask>

---

Intended Uses & Limitations

Intended Uses

• Pre-tokenization for training Transformer-based LLMs.
• Downstream NLP tasks:
  • Language modeling
  • Text classification
  • Question answering
  • Summarization

Limitations

• Trained exclusively on English Wikipedia text — performance may degrade in informal, domain-specific, or multilingual contexts.
• May inherit biases present in Wikipedia data.

---

License

This tokenizer is released under the MIT License.

---

Citation

If you use this tokenizer, please cite:

title = Custom BPE Tokenizer Trained on WikiText-103 Raw v1
author = yakul259
year = 2025
publisher = Hugging Face