Stockmark-Nemotron-3-Nano-Omni-JapanDocReader-FP8

Introduction

Stockmark-Nemotron-3-Nano-Omni-JapanDocReader-FP8 is the FP8-quantized version of Stockmark-Nemotron-3-Nano-Omni-JapanDocReader, a Japanese document-reading multimodal model. Please refer to the original model card for details on the model itself.

Quantization was performed with the NVIDIA TensorRT Model Optimizer (ModelOpt):

• Scheme — static FP8 (W8A8, per-tensor scales) via post-training quantization.
• Scope — only the LLM decoder's Linear layers are quantized. The vision encoder (RADIO), audio encoder, multimodal projectors, lm_head, embeddings, MoE routers, and the Mamba conv/dt paths are kept in BF16 to preserve accuracy-sensitive components.
• Calibration — 256 in-domain Japanese document-parsing samples run through the model to collect activation ranges before export.
• Export — a standard HuggingFace checkpoint with hf_quant_config.json, loadable by vLLM with --quantization modelopt.

---

Quickstart

The model serves as a standard OpenAI-compatible endpoint via vLLM (≥ 0.20). It is a reasoning model — keep thinking mode on and inject a reasoning budget so it closes </think> and answers.

1. Launch the server

vllm serve stockmark/Stockmark-Nemotron-3-Nano-Omni-JapanDocReader-FP8 \
  --served-model-name japandocreader-fp8 \
  --host 0.0.0.0 --port 8000 \
  --quantization modelopt \
  --dtype auto \
  --max-model-len 210000 \
  --tensor-parallel-size 1 \
  --trust-remote-code \
  --reasoning-parser nemotron_v3 \
  --allowed-local-media-path / \
  --media-io-kwargs '{"video": {"fps": 2, "num_frames": 256}}' \
  --video-pruning-rate 0.5

Compared to the BF16 model, the only changes are --quantization modelopt and --dtype auto (the quantized dtype is picked up from the checkpoint's hf_quant_config.json).

2. Recommended inference parameters

Parameter	Value	Note
temperature	0.6	official thinking-mode setting
top_p	0.95	official thinking-mode setting
repetition_penalty	1.0	recommended default (no penalty); keeps picture descriptions intact
reasoning_budget	16384	thinking token budget
max_tokens	20480	must be > reasoning_budget (leaves room for the answer)
max_model_len	210000	server-side

max_tokens > reasoning_budget is required: the budget caps the think block, and the remaining max_tokens − reasoning_budget tokens hold the answer. Setting them equal starves the answer.

3. Structured document parsing (docparse)

⚠️ Use this exact prompt. The model was trained with the fixed Japanese docparse prompt below. The prompt defines the task, the JSON schema, the allowed class values, and the coordinate convention — the model's output format is conditioned on it. Do not paraphrase, translate, or reorder it; changing the prompt degrades layout accuracy and JSON validity. Keep it verbatim, including the trailing Return ONLY the JSON object instruction.

import base64, json, re, urllib.request

PROMPT = """画像に含まれるドキュメントの構造をJSON形式で抽出してください。
出力フォーマット:
{
  "document_structure": [
    {
      "class": "title" | "heading" | "text" | "table" | "list" | "picture" | "formula",
      "bbox": [x1, y1, x2, y2],
      "contents": "内容（pictureの場合は画像内容の説明、formulaの場合は数式のlatex表記）",
      "caption": "pictureのキャプション文字（オプション）"
    }
  ]
}
classの種類: title（タイトル）、heading（見出し）、text（本文）、table（表）、list（リスト）、picture（画像）、formula（数式）
bboxは左上(x1,y1)と右下(x2,y2)の座標です。bboxの座標系は0-1000の相対座標です。
Return ONLY the JSON object. No markdown, no extra commentary."""

def parse_document(image_path, base_url="http://127.0.0.1:8000"):
    with open(image_path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()
    payload = {
        "model": "japandocreader-fp8",
        "messages": [{"role": "user", "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
            {"type": "text", "text": PROMPT},
        ]}],
        "temperature": 0.6, "top_p": 0.95,
        "repetition_penalty": 1.0,
        "max_tokens": 20480,
        "chat_template_kwargs": {"enable_thinking": True, "reasoning_budget": 16384},
        "thinking_token_budget": 17408,   # reasoning_budget + grace
    }
    req = urllib.request.Request(
        base_url.rstrip("/") + "/v1/chat/completions",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json", "Authorization": "Bearer EMPTY"},
        method="POST")
    with urllib.request.urlopen(req, timeout=1800) as r:
        msg = r.read(); msg = json.loads(msg)["choices"][0]["message"]
    # thinking is split into reasoning_content; the answer is the JSON in `content`
    answer = msg.get("content") or ""
    m = re.search(r"\{.*\}", answer, re.DOTALL)
    return json.loads(m.group(0) if m else answer)   # -> {"document_structure": [...]}

result = parse_document("document.png")
print(json.dumps(result, ensure_ascii=False, indent=2))

4. Document VQA

For VQA, send the image plus the question (no docparse prompt); the model reasons in <think> and answers in natural language:

payload["messages"] = [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
    {"type": "text", "text": "この文書について: <your question in Japanese>"},
]}]
# same sampling params as above; the answer is in message.content, the reasoning in reasoning_content

---

License

Released under the NVIDIA Open Model License, inherited from the base model. Please also review the base model's terms.

---

Acknowledgements

In this experiment, we used 8×B300 Blackwell Ultra compute resources provided by NVIDIA, and conducted training on NVIDIA Brev using NVIDIA NeMo. We sincerely thank everyone at NVIDIA for supporting the large-scale SFT and RL experiments.

---

Developed by

Stockmark Inc.

---

Citation

@misc{stockmark_japandocreader_fp8_2026,
  title={Stockmark-Nemotron-3-Nano-Omni-JapanDocReader-FP8},
  author={Stockmark Inc.},
  year={2026}
}