Measured on device (edge-compat): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · did not run: run_error (2026-09-05); Mac Studio M4 Max · LiteRT-LM 0.16.0 · GPU · decode 83.3 tok/s · prefill 803 tok/s · TTFT 331 ms (2026-08-27); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 12.9 tok/s · prefill 53 tok/s · TTFT 4.06 s (2026-09-05). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/nemotron-3-nano-4b-int8/CARD.md

Nemotron-3-Nano-4B — LiteRT-LM

nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15. To our knowledge this is the first Nemotron-3-Nano in LiteRT form.

A reasoning model (<think>, ChatML turns) on a three-kind hybrid stack: 21 Mamba2 selective-scan layers + 17 plain MLP layers + 4 grouped-query attention layers (42 total). Only the 4 attention layers keep KV (4096-token budget here), the mamba layers carry constant-size conv + SSM state, and the MLP layers carry no state at all — 50 state buffers in total (42 mamba, 8 KV), so memory stays nearly flat with context length.

File	Recipe	Size
Nemotron-3-Nano-4B_int8.litertlm	int8 dynamic on linears + embedding (convs and the scan stay float); fp32 activations declared	4.13 GB

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.

Geometry: hidden 3136, 40 query / 8 KV heads, mamba 96 heads × 80 dim (state 128, conv 4, 8 groups), vocab 131,072, untied embeddings.

Correctness

All numbers below were measured on this exact file (litert-lm 0.16.0, Apple M4 Max).

• 8-question sanity gate: 7/8 on CPU, 8/8 on GPU, non-degenerate on both. The single CPU miss is the rhyme-completion item — it answers "violets are purple" where the gate wants "blue"; the GPU run answers "blue". Every arithmetic, factual, and translation item is correct on both backends.
• Chat template is byte-equal to the source: the embedded Jinja matches the repo's chat_template.jinja exactly (10,504 / 10,504 bytes). Note the source repo's tokenizer_config.json carries a different 10,497-byte copy; the bundle embeds the one AutoTokenizer actually resolves.
• Turn-end stop tokens are <|im_end|> (id 11) alongside the exported id 2.
• No spurious start token. The source tokenizer sets add_bos_token: False and the template never renders a leading BOS, so the <s> the bundler would otherwise prepend is dropped — the on-device token stream matches the training stream. Honest note: at this scale the model is robust either way (the gate scores 7/8 with the token and 7/8 without, and greedy decoding in PyTorch is byte-identical on 2 of 3 probes), so this is a correctness-of-convention fix rather than a rescue.

Usage

# CPU
litert-lm run ./Nemotron-3-Nano-4B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU — pass --cache no (see the honest note below)
litert-lm run ./Nemotron-3-Nano-4B_int8.litertlm --backend gpu --cache no --prompt "..."

The bundle carries the tokenizer and the stock Nemotron-3-Nano chat template. Seven prefill signatures (1024/256/64/16/4/1 + decode) are exported so the runtime picks tight chunks.

Performance

litert-lm benchmark <file> -p 256 -d 256 --runs 3 --cache no, litert-lm 0.16.0, Apple M4 Max. Two independent runs:

Backend	Prefill (256)	Decode	TTFT
GPU (--cache no)	803 / 792 tok/s	83.3 / 82.4 tok/s	0.33 s
CPU	99.6 / 113.3 tok/s	22.7 / 22.5 tok/s	2.64 / 2.30 s

Both figures per cell are the two runs, not a range estimate. GPU repeats within ~1.4%; CPU prefill spreads ~13%, partly because the host was not idle during these runs (another export was using the machine) — read the CPU column as an order of magnitude, not a precise figure. --cache no matters for more than tidiness here: with the compiled-graph cache the benchmark reports a much faster CPU prefill because it is not doing the same work.

Honest notes

• GPU requires --cache no on this bundle. With the compiled-graph cache enabled, litert-lm run --backend gpu fails with WebGPU Invalid BindGroup validation errors, and an 8-question sweep through the Mac verify harness returns token soup (0/8). The same file with --cache no answers 8/8. Measured as a one-variable comparison — same runner, same file, cache flag flipped — so the cache path is where it goes wrong; the root cause is not isolated further here.
• Phones: one Galaxy S26, CPU only. On a Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno) the CPU backend runs this file: 52.9–64.0 tok/s prefill and 11.85–12.87 tok/s decode on a 211-token prompt, peak 4,884 MB, engine init 9.0 s (litert_lm_advanced_main v0.16.0, two cold runs, 2026-09-05; the device line near the top of this card and the manifest carry the rows). The GPU backend crashes on that handset (SIGSEGV, exit 139, while the OpenCL delegate initializes its kernels for prefill_1024 after accepting all 2,827 ops), so CPU is the only Android path measured. No Mali phone has run this file, and no 8 GB phone has been tried: the sibling Nemotron-H-4B did not fit one, so this file is not expected to fit an 8 GB phone either — an expectation carried over from a different bundle, not a measurement of this one.
• It is a reasoning model. Answers arrive after a <think> block, so give it a token budget that fits the thought (the gate above used 3200).
• int8 is applied to linears and the embedding only; the convolutions and the selective scan stay float, which is what keeps the hybrid state numerically sane.
• One Engine per conversation; since 2026-10-13 this file holds multi-turn conversations with thinking on. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's recurrent state: it reuses the prompt tokens the two conversations share and does not reset the state (my open report). A probe asks three questions (What is the capital of France? / What is 17 times 23? / What color is the sky on a clear day?) first in a pirate-persona conversation and then in a conversation told to answer in one word, on the same Engine against a fresh Engine in a new process (answers capped at 200 tokens; CPU backend; litert-lm 0.18.0 on a Mac). The probe could not measure this on the file published until 2026-10-13: neither conversation got past its first question. On that file, on litert-lm 0.18.0 with thinking on (the default), the second message of a conversation failed: send_message raised, and the runtime logged INTERNAL: The new rendered template string does not start with the previous rendered template string — in 6 of 6 conversations tried (the probe's two conversations in each of its two runs, and a two-turn check in two runs, where "My name is Kenji." was answered and "What is my name? Answer with the name only." failed; one more run of that check reproduced it). The cause was the chat template, not the weights: before each message the runtime re-renders the conversation so far and requires the new render to extend the previous one, and that template rewrote an earlier assistant turn's <think> block to an empty <think></think> once a newer user message followed it (its truncate_history_thinking default), so the re-render no longer matched what the model generated (my open report). With thinking disabled for the conversation (ThinkingConfig(enable_thinking=False)), three turns answered on litert-lm 0.18.0 even on that file, and on litert-lm 0.17.1 the same three turns answered with thinking on. The file published on 2026-10-13 fixes the template (metadata only, weights unchanged; dated note below): every assistant turn in the history now renders with an empty <think></think> — the form the earlier template produced once a newer user message followed — so each re-render extends the previous one. With it, the same three turns answered on litert-lm 0.18.0 with thinking on ("Kenji", "391"; the Python API and the C++ Conversation API, one run each), with thinking off, on litert-lm 0.17.1, and on a Galaxy S26 (CPU backend, thinking on and off, one run each). If you downloaded before 2026-10-13, re-download. Create a new Engine for each conversation. The first conversation on an Engine is unaffected by the carry-over.
• Tool calls come back as text. Given a tool definition, the model answers in its own XML form (<tool_call>\n<function=get_weather>\n<parameter=location>\nParis\n</parameter>\n</function>\n</tool_call>), and litert-lm 0.18.0 returns that as the message text with tool_calls empty: the runtime has no parser for this form (the bundle is generic_model; it is the XML shape of my open report, filed for another model family, and a parser for it is proposed in my draft PR #3931). Measured on litert-lm 0.18.0, CPU backend, one tool (get_weather) and the question "What is the weather in Paris right now?", two runs, both identical. Until the runtime parses it, read the call out of the text in your app.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch. One command, no per-model work — the reproduction script, the patch, and the full measurement record are in hf-to-litertlm:

python scripts/convert.py nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

Two things that route this model correctly and are worth knowing if you convert your own:

• auto_map in a config is not proof of remote code. This repo declares auto_map, but transformers registers nemotron_h natively, so without trust_remote_code the library implementation loads and the repo's Python is never imported. A converter that refuses on auto_map alone will refuse this model for no reason.
• ≥3B exports use a reduced 7-signature prefill ladder. Every exported signature costs engine RAM whether or not it is called, and a 4B hybrid with the full 11-signature ladder is exactly the shape that trips memory limits at GPU program init.

Conversion took 1645 s on an M4 Max. See REPRODUCE.md for the Nemotron-H family recipe and the measurements behind every claim on this card.

2026-08-31 — thought channel declared (metadata only, weights unchanged)

Nemotron-3-Nano-4B_int8.litertlm now declares the reasoning channel in its metadata (LlmMetadata.channels: channel name thought, markers <think>…</think> exactly as this model emits them). Without the declaration the runtime has no way to tell the reasoning apart from the answer: the raw thinking streamed inline into the visible text, and a thinking_token_budget was silently ignored (the API returns OK and only logs a warning). With the channel declared, LiteRT-LM returns the reasoning separated in channels["thought"] and the thinking budget takes effect.

Metadata-only change: every section of the bundle except LlmMetadata is byte-identical to the previous file (verified per section, tokenizer included), so the weights, the graph, the tokenizer and the chat template are unchanged and the speed and accuracy numbers on this card still describe this file — only the file's own sha256 differs. Verified on the LiteRT-LM runtime (litert-lm-api 0.16.1): the visible answer stays clean, the reasoning lands in channels["thought"], and on one bundle of this batch thinking_token_budget=16 was confirmed to truncate the reasoning at exactly 16 tokens where it was a no-op before. If you downloaded before 2026-08-31, re-download to get the channel-aware file.

2026-10-13 — chat template fixed for multi-turn conversations (metadata only, weights unchanged)

Nemotron-3-Nano-4B_int8.litertlm now carries a chat template whose rendering of an assistant turn in the history does not depend on where that turn sits. The earlier template (NVIDIA's, with truncate_history_thinking at its default of true) kept the full <think>…</think> block on the latest assistant turn and replaced it with an empty <think></think> on every assistant turn that a newer user message followed. LiteRT-LM re-renders the conversation before each message and requires the new render to extend the previous one, so on litert-lm 0.18.0 — which hands the model's reasoning to the template — the second message of every conversation with thinking on failed (my open report describes the mechanism; the Honest note above has the measurements). The fix changes two lines of the template: the position condition is dropped, so every assistant turn in the history renders as <think></think> followed by its answer — the form the earlier template already produced for turns followed by a user message, so what the model sees from the third turn on is unchanged. The generation prompt is unchanged (<think>\n with thinking on, <think></think> with thinking off). Keeping the reasoning in the history was the alternative; it also passes the runtime's check but re-prefills the previous turn's reasoning on every message (40–54 more prefill tokens per turn in the three-turn probe), so it was not shipped.

Metadata-only change: every section of the bundle except LlmMetadata is byte-identical to the previous file (verified per section: the TFLite model, the executor metadata and the tokenizer; file size unchanged at 4,126,697,184 bytes), and inside LlmMetadata only jinja_prompt_template differs. The speed and accuracy numbers on this card still describe this file; only the file's own sha256 changed (feefdcd55693022ddbdbed7d0aafdd266cac950c6a348d25a8e96b12827ddbda → e9274fb21c95944c5ad76b10a8294ebfe5b54a359d6d6166b1879a958331ca6a).

Verified (CPU backend unless noted, greedy, one run each): three turns ("My name is Kenji." / "What is my name? Answer with the name only." / "What is 17 times 23? Answer with the number only.") answered "Kenji" and "391" on litert-lm 0.18.0 with thinking on (Python API and the C++ Conversation API), on litert-lm 0.18.0 with thinking off, and on litert-lm 0.17.1 with thinking on; on a Galaxy S26 (SM-S942Q, Android 16) the same three turns answered with thinking on and off through the C++ Conversation API of LiteRT-LM v0.18.0. The litert-lm 0.15.0 and 0.16.0 CLIs read the new file and answer 391 to the arithmetic question. The eight-question gate of this card, run on the litert-lm 0.18.0 CLI, is 8/8 on CPU and 8/8 on GPU with --cache no, the same as the previous file on the same day. The GPU path still needs --cache no on litert-lm 0.18.0: with the compiled-graph cache the run fails with Validation error: Buffer size (54904832) wrapping host-mapped memory was not aligned to 4096. and the output degenerates, for the previous file as well.

Reproduce with hf-to-litertlm: python scripts/set_prompt_template.py <previous file> <new file> templates/nemotron3nano_history.jinja --litert-lm litert-lm (litert-lm 0.18.0). A re-run reproduces all four sections byte for byte; the file's sha256 differs because the packer writes a new uuid and timestamp into the header.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
Nemotron-3-Nano-4B_int8.litertlm	22.7 (22.6–22.8)	2.2 (2.2–2.2)	11.7 s	5.0 GB

Known runtime issues

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders this file's chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on this file's template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.