Hugging Face's logo

Buckets:
ray0rf1re
/
AO3-2020-bucket

Files
xet
ray0rf1re/AO3-2020-bucket / README.md
ray0rf1re's picture
ray0rf1re
preview code
|
download
raw
11.2 kB
metadata
license: cc-by-nc-4.0
language:
  - en
task_categories:
  - text-generation
  - fill-mask
  - feature-extraction
task_ids:
  - language-modeling
tags:
  - ao3
  - fanfiction
  - creative-writing
  - long-form
  - literature
  - english
  - pretraining
  - chatml
  - openhermes
  - alignment
pretty_name: AO3-2020 Fanfiction Corpus
size_categories:
  - 1B<n<10B
source_datasets:
  - original
multilinguality: monolingual
dataset_size_bytes: 539375493120
configs:
  - config_name: full
    data_files:
      - split: train
        path: data/full/train-*.parquet
  - config_name: tokens_2_5M
    data_files:
      - split: train
        path: data/tokens_2_5M/train-00000.parquet
  - config_name: tokens_1_25M
    data_files:
      - split: train
        path: data/tokens_1_25M/train-00000.parquet
  - config_name: tokens_250K
    data_files:
      - split: train
        path: data/tokens_250K/train-00000.parquet
  - config_name: tokens_100K
    data_files:
      - split: train
        path: data/tokens_100K/train-00000.parquet
  - config_name: tokens_75K
    data_files:
      - split: train
        path: data/tokens_75K/train-00000.parquet
  - config_name: tokens_50K
    data_files:
      - split: train
        path: data/tokens_50K/train-00000.parquet
  - config_name: tokens_37_5K
    data_files:
      - split: train
        path: data/tokens_37_5K/train-00000.parquet
  - config_name: tokens_25K
    data_files:
      - split: train
        path: data/tokens_25K/train-00000.parquet
  - config_name: tokens_10K
    data_files:
      - split: train
        path: data/tokens_10K/train-00000.parquet
  - config_name: tokens_7_5K
    data_files:
      - split: train
        path: data/tokens_7_5K/train-00000.parquet
  - config_name: tokens_5K
    data_files:
      - split: train
        path: data/tokens_5K/train-00000.parquet
  - config_name: three_quarters
    data_files:
      - split: train
        path: data/three_quarters/train-*.parquet
  - config_name: seven_tenths
    data_files:
      - split: train
        path: data/seven_tenths/train-*.parquet
  - config_name: five_ninths
    data_files:
      - split: train
        path: data/five_ninths/train-*.parquet
  - config_name: half
    data_files:
      - split: train
        path: data/half/train-*.parquet
  - config_name: quarter
    data_files:
      - split: train
        path: data/quarter/train-*.parquet
  - config_name: fifth
    data_files:
      - split: train
        path: data/fifth/train-*.parquet
  - config_name: eighth
    data_files:
      - split: train
        path: data/eighth/train-*.parquet
  - config_name: twelfth
    data_files:
      - split: train
        path: data/twelfth/train-*.parquet
  - config_name: fifteenth
    data_files:
      - split: train
        path: data/fifteenth/train-*.parquet
  - config_name: sixteenth
    data_files:
      - split: train
        path: data/sixteenth/train-*.parquet
  - config_name: twenty_fourth
    data_files:
      - split: train
        path: data/twenty_fourth/train-*.parquet
  - config_name: aug_openhermes_10K
    data_files:
      - split: train
        path: data/aug_openhermes_10K/train-00000.parquet
  - config_name: aug_openhermes_37_5K
    data_files:
      - split: train
        path: data/aug_openhermes_37_5K/train-00000.parquet
  - config_name: aug_openhermes_100K
    data_files:
      - split: train
        path: data/aug_openhermes_100K/train-00000.parquet
  - config_name: aug_openhermes_250K
    data_files:
      - split: train
        path: data/aug_openhermes_250K/train-00000.parquet
  - config_name: adult_18plus_100K
    data_files:
      - split: train
        path: data/adult_18plus_100K/train-*.parquet

---
license: cc-by-nc-4.0
language:
  - en
task_categories:
  - text-generation
  - fill-mask
  - feature-extraction
task_ids:
  - language-modeling
tags:
  - ao3
  - fanfiction
  - creative-writing
  - long-form
  - literature
  - english
  - pretraining
  - chatml
  - openhermes
  - alignment
pretty_name: AO3-2020 Fanfiction Corpus
size_categories:
  - 1B<n<10B
source_datasets:
  - original
multilinguality: monolingual
dataset_size_bytes: 539375493120
configs:
  - name: full
num_examples: 1553596

  • name: tokens_2_5M num_examples: 711

  • name: tokens_1_25M num_examples: 392

  • name: tokens_250K num_examples: 76

  • name: tokens_100K num_examples: 25

  • name: tokens_75K num_examples: 20

  • name: tokens_50K num_examples: 14

  • name: tokens_37_5K num_examples: 10

  • name: tokens_25K num_examples: 6

  • name: tokens_10K num_examples: 2

  • name: tokens_7_5K num_examples: 2

  • name: tokens_5K num_examples: 1

  • name: tokens_2_5K num_examples: 0

  • name: three_quarters num_examples: 1165197

  • name: seven_tenths num_examples: 1087516

  • name: five_ninths num_examples: 863108

  • name: half num_examples: 776798

  • name: quarter num_examples: 388399

  • name: fifth num_examples: 310719

  • name: eighth num_examples: 194199

  • name: twelfth num_examples: 129466

  • name: fifteenth num_examples: 103573

  • name: sixteenth num_examples: 97100

  • name: twenty_fourth num_examples: 64733

  • name: aug_openhermes_10K num_examples: 2

  • name: aug_openhermes_37_5K num_examples: 10

  • name: aug_openhermes_100K num_examples: 25

  • name: aug_openhermes_250K num_examples: 76 dataset_info: features: - name: text dtype: string splits: - name: train num_examples: 15993173 annotations_creators: [] language_creators: - crowdsourced

    AO3-2020 — Archive of Our Own Fanfiction Corpus

    📋 Overview

    Field Value
    Total documents 15,993,173
    Total characters ~21,816,825,882
    Estimated tokens ~5,454,206,470
    Raw source size 502 GB (SQLite)
    Language English
    Source Archive of Our Own (ao3.org)
    Cutoff date 2020
    License CC BY-NC 4.0
    Parquet compression ZSTD level 3
    Last updated 2026-05-17

    🚀 Quick start 

    from datasets import load_dataset

# Full 502 GB corpus — use streaming=True for large machines
ds = load_dataset("ray0rf1re/AO3-2020", "full", streaming=True)

# Fast experiment — 75 K token slice
ds = load_dataset("ray0rf1re/AO3-2020", "tokens_75K")

# SFT-ready ChatML format (OpenHermes 2.5 template)
ds = load_dataset("ray0rf1re/AO3-2020", "aug_openhermes_100K")

# Deterministic 1/8 uniform sample
ds = load_dataset("ray0rf1re/AO3-2020", "eighth")

# Print first document
for ex in ds["train"]:
    print(ex["text"][:500])
    break

    🗂️ Subsets

    Full corpus

    Subset Rows
    full 1,553,596

    Token-capped subsets

    Rows are taken in source order until the cumulative estimated token budget (≈ 4 chars / token) is reached. Good for controlled-scale experiments.

    Subset Token cap Rows
    tokens_2_5M 2,500,000 711

| tokens_1_25M | 1,250,000 | 392 | | tokens_250K | 250,000 | 76 | | tokens_100K | 100,000 | 25 | | tokens_75K | 75,000 | 20 | | tokens_50K | 50,000 | 14 | | tokens_37_5K | 37,500 | 10 | | tokens_25K | 25,000 | 6 | | tokens_10K | 10,000 | 2 | | tokens_7_5K | 7,500 | 2 | | tokens_5K | 5,000 | 1 | | tokens_2_5K | 2,500 | 0 |

### Fraction subsets

Deterministic, uniform row-skip sampling — every `d`th row starting at 0.
Reproducible across library versions without a fixed seed.

| Subset | Fraction | Coverage | Rows |
|--------|----------|----------|------|
| `three_quarters` | 3/4 | 75.00 % | 1,165,197 |

| seven_tenths | 7/10 | 70.00 % | 1,087,516 | | five_ninths | 5/9 | 55.56 % | 863,108 | | half | 1/2 | 50.00 % | 776,798 | | quarter | 1/4 | 25.00 % | 388,399 | | fifth | 1/5 | 20.00 % | 310,719 | | eighth | 1/8 | 12.50 % | 194,199 | | twelfth | 1/12 | 8.33 % | 129,466 | | fifteenth | 1/15 | 6.67 % | 103,573 | | sixteenth | 1/16 | 6.25 % | 97,100 | | twenty_fourth | 1/24 | 4.17 % | 64,733 |

### Augmented subsets — OpenHermes 2.5 ChatML template

Same token budgets as their plain counterparts, but each document is
wrapped in the **OpenHermes 2.5 / ChatML** instruction template so the
data can be fed directly to SFT trainers (e.g. LLaMA-Factory, Axolotl,
Unsloth) without additional preprocessing.

Template applied to every row:
```
<|im_start|>system
You are a creative writing assistant specialising in long-form fanfiction. You produce engaging, vivid, and high-quality stories.<|im_end|>
<|im_start|>user
Write a fanfiction story.<|im_end|>
<|im_start|>assistant
{text}<|im_end|>
```

| Subset | Token cap | Rows |
|--------|-----------|------|
| `aug_openhermes_10K` | 10,000 | 2 |

| aug_openhermes_37_5K | 37,500 | 10 | | aug_openhermes_100K | 100,000 | 25 | | aug_openhermes_250K | 250,000 | 76 |

---

## 📁 Data schema

```json
{
    "text":  "string — UTF-8 text of the fanfiction work or chapter.",
    "...":   "All additional metadata columns from the source SQLite are preserved as-is."
}
```

Augmented subsets replace `text` with the ChatML-formatted version.

---



The corpus is directly readable with:
- 🤗 `datasets` (streaming or map-style)
- DuckDB: `SELECT * FROM read_parquet('data/full/*.parquet')`
- Polars: `pl.scan_parquet('data/full/*.parquet')`
- Spark: `spark.read.parquet('data/full/')`
- PyArrow: `pq.read_table('data/full/')`

---

## ⚖️ License & ethics

- **License**: [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/)
- Non-commercial research use only.
- Authors retain copyright over their individual works.
- Works rated **Explicit** or **Mature** may be present in the `full` corpus.
- If you are an AO3 author and wish your work removed, please open an issue.

---

## 🔗 Citation

```bibtex
@dataset{ao3_2020,
  author    = {ray0rf1re},
  title     = {AO3-2020: Archive of Our Own Fanfiction Corpus},
  year      = {2020},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/datasets/ray0rf1re/AO3-2020}
}
```

⚠️ adult_18plus_100K — SAFETY / ALIGNMENT SUBSET

❌ DO NOT train a generative model on this subset as ordinary SFT data.

✅ Use it inverted — as the rejected side of preference pairs.

This subset contains the first ~100,000 tokens of content rated Explicit or Mature on AO3 (26 chapters).

Technique How to use this subset
DPO / RLHF Label completions as rejected; safe rewrites as chosen.
Content-safety classifier Negative class in binary classifier training.
Inverted SFT Generate from this data, teach the model to rewrite it safely.

Training on this data naively produces an unsafe model. Training against it — using it as the "what not to do" signal — produces a safer model than one that never saw this kind of content.

Xet Storage Details

Size:
11.2 kB
·
Xet hash:
3b25079de594301459778426bb3ad28d312955a820e0ae317091393f4b3e59ac

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.