Qwen3.8-27B NVFP4 MTP GGUF — Local GB10 Test

Key Summary

Seven esatapedico NVFP4 MTP tiers for Qwen3.8-27B (27B dense, built-in MTP head, mmproj vision support). HIGH/VERY-HIGH GB10 measurements plus a real-use demo (ragdoll-playground-qwen3.8.html).

GB10 Measurement Summary

GB10 measurement complete — ① Settings: --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 + ctx 131072 + flash-attn + KV q8_0 + mlock. ② Performance: 680–930 t/s prefill, sustained 17–19.5 t/s decode, 93.1% MTP acceptance on source-based long generation, and +50% decode versus n-max 4. ③ Long-serving measurement (8/18): 18–22 t/s on long generations under four concurrent slots with the mmproj vision component loaded; 12.6–23.4 t/s overall, 18.5 t/s on an 8,287-token long generation, and about 94% average acceptance. ④ Thinking ON/OFF configurations were both verified; HIGH and VERY-HIGH used 16–20 GB, leaving substantial memory headroom. Runtime note: an older binary in the system installation path had symbol mismatches, and --no-mmap/--mlock are deprecated; use --load-mode mlock. Four execution scripts are stored with the model directory.

Detailed Research

Overview

• Model: Qwen3.8-27B — 27B dense, native vision and language (VLM), hybrid Gated DeltaNet + Gated Attention, 262,144 native context, built-in MTP speculative-decoding head (Apache-2.0)
• Distribution: esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF — a complete conversion of unsloth/Qwen3.8-27B-NVFP4 (Apache-2.0 derivative)
• Quantization: NVFP4 — Blackwell-native 4-bit, GGML type 40, sm_120. The source NVFP4 MLP tensors are preserved without requantization.
• Seven tiers: ORIG (33 GB) / VERY-LOW / LOW / MEDIUM / HIGH / VERY-HIGH / HIGHEST (23 GB) — a common 448-tensor byte-identical NVFP4 backbone for full attention and MLP, with tier-specific precision only for lm_head, token_embd, and the MTP head.
• MTP is embedded in GGUF (blk.64.nextn.*) — no separate drafter file; activate with llama.cpp --spec-type draft-mtp.
• Vision support: pair with mmproj-BF16.gguf (931 MB) for image and video input.

GB10 Local Measurement

Test Configuration

• Hardware: NVIDIA GB10 (DGX Spark, 128 GB unified memory)
• Runtime: llama.cpp build 10454, an sm121a GB10-specific CUDA 12.1 build
• Test models: HIGH (15.9 GB — lm_head BF16, token_embd Q6_K, MTP IQ4_XS) / VERY-HIGH (19.7 GB — lm_head, token_embd, and MTP all BF16)
• Server settings (OpenAI-compatible API, ports 9505/9506):

--spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75
--ctx-size 131072 --flash-attn --cache-type-k q8_0 --cache-type-v q8_0
--load-mode mlock -ngl 999 --jinja
Sampling: --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0
Both thinking ON and OFF (--reasoning off) were verified.

Approximate Throughput

Item	Value
Prefill	680–930 t/s on a 14,327-token prompt
Decode	17–19.5 t/s sustained, with instantaneous peaks of 24–25 t/s
MTP acceptance, source-based long generation	93.1% (81/87), average draft length 4.24
MTP acceptance, short prompt	100% (7/7), average draft length 4.50

Effect of the setting change (same HIGH no-think configuration): after applying --spec-draft-n-max 6 + --spec-draft-p-min 0.75:

Setting	Acceptance	Average draft length	Decode
n-max 4, no p-min	36.7%	2.47	12.2 t/s
n-max 6 + p-min 0.75	93.1%	4.24	17.5 t/s

→ Decode speed improved by about 50%. MTP has a larger effect on predictable long generation such as source rewriting, summarization, and code.

Long-Serving Measurement (2026-08-18 Added — Four Concurrent Slots, About 30 Minutes)

The same HIGH no-think configuration (port 9505, n-max 6 + p-min 0.75) was measured with four concurrent requests, mmproj vision loaded, and continuous serving. This is closer to actual use than the single-request test:

Item	Value
Decode, all tasks	12.6–23.4 t/s (50+ tasks, median about 19 t/s)
Decode, long 500+ token generation	18.5–22.8 t/s (about 20 t/s average)
Decode, very long 8,287-token generation	18.5 t/s; acceptance fell slightly to 86.1%
Average MTP acceptance	about 94% (73–100%, mostly above 90%)
Average draft length	3.4–6.9, mostly 5–6.5 near the n-max 6 setting
Prefill under load	606–930 t/s; 682 t/s for 14,327 tokens and 606 t/s for 8,306 tokens
Concurrency	Four-slot kv_unified parallel requests; decode can temporarily fall to 12–13 t/s under short-request contention

• Contention: when a short 100–300-token generation overlaps with another slot's prefill, decode temporarily drops to 12–13 t/s; long generations mostly remain at 18–22 t/s.
• Lower acceptance (73–80%) appears in highly random and difficult-to-predict generation segments, where draft length falls to 3.5–4.

Reference Numbers from Other Hardware — Not Directly Comparable

• Author measurement (dual RTX 5070 Ti + 5060 Ti 16 GB, 180K repeated payload): 15.4–18.5 t/s decode and 72.6–75.1% acceptance.
• Community measurement (single RTX 5090 gist): 111.9–148.3 t/s average and 44–49.5% acceptance. The LOW tier with a Q5_0 lm_head had the highest throughput, while BF16 lm_head variants prioritized quality.
• Prefill decreases with longer context, so values from different contexts cannot be compared quantitatively.

Real-Use Demo — Ragdoll Playground (One Qwen3.8-27B Generation)

• Result: Run the Ragdoll Playground demo directly in a browser.
• Prompt source: the “Ragdoll Physics Simulator” test prompt from Token Chaser — MiMo-v2.5 vs Qwen3.6 27B.
• Generation method: one prompt, one generation, with no edits or retries. The result is a single-file HTML/CSS/JS ragdoll simulator with no external libraries.
• Environment: Qwen3.8-27B NVFP4 MTP HIGH (no-think), port 9505, using the same GB10 server settings above.
• Features: ragdoll falling, bouncing, and dragging; a playground with floor, platforms, walls, and bumpers; gravity, bounce, slow motion, and anti-gravity controls; ball, box, and bumper spawning; and reset.
• Verification: zero sensitive values (no paths, IPs, names, or keys); the “qwen3.8” model tag was visible in the UI.

GB10 Suitability

• Memory: 27B dense NVFP4 uses 16–20 GB plus mmproj and a 131K Q8_0 KV cache, leaving substantial room in 128 GB unified memory.
• Speed: 17–20 t/s for a single request and 18–22 t/s for long generations under four concurrent slots, with temporary drops to 12–13 t/s under contention. This is practical for long documents and an agent-assistance model.
• Key point: NVFP4 is native to GB10/Blackwell sm_120, so the dense quantization can be used locally as-is. The built-in MTP head enables speculative decoding without a separate drafter.
• Suggested use: thinking OFF for quick answers and tool-call assistance; thinking ON for deeper reasoning and analysis. Image understanding is possible with the vision mmproj.
• Runtime note: the older binary in the system installation path had symbol mismatches with the newer libraries. Always use the binary from the build directory. --no-mmap/--mlock are deprecated; use --load-mode mlock.

Conclusion

• Qwen3.8-27B NVFP4 MTP is a valid practical local option for a 27B-class dense model on GB10.
• On source-based long generation, MTP tuning (n-max 6 + p-min 0.75) improved both acceptance and speed.
• HIGH versus VERY-HIGH had little speed difference because the NVFP4 backbone is the same. BF16 has the quality advantage in lm_head and embedding precision; with available memory, VERY-HIGH is recommended.

Related Benchmark

The same project's Qwen3.8-27B Ridge measurement is available in the Local LLM Benchmark report.

Sources

• Official Qwen3.8-27B model page
• esatapedico Qwen3.8-27B NVFP4 MTP GGUF
• Unsloth Qwen3.8-27B NVFP4
• RTX 5090 comparison measurement gist
• Token Chaser comparison video — source of the Ragdoll Physics Simulator prompt