Qwen3.8-Flash-Next — IQ3_S + MTP (Ollama)

Model tag: qwen38flashnext-iq3s-mtp:latest Architecture: qwen4exp (Qwen3.8-Flash-Next, hybrid sparse MoE) with an attached MTP (Multi-Token Prediction) draft head for speculative decoding Total size: ~99 GB on disk (67.8 GB main + 2.8 GB draft sidecar) Capabilities: tools · thinking · completion · 262K context License: Qwen Community License 1.0

⚠️ Read this first: runner requirements

This model uses the qwen4exp architecture with hyper-connections and a PLE (per-layer n-gram embedding) block — not supported by stock llama.cpp / stock Ollama as of writing. Additionally, the runner used here applies a d2t (draft-to-target) reduced-vocabulary patch for MTP speculative decoding (trims draft output logits to 32,768 entries for faster spec-decode). On a correctly-patched runner the log shows:

qwen4exp MTP using d2t draft-vocab trim (n_vocab_out = 32768)

…followed by spec-decoding accept-rate stats. The qwen4exp architecture itself needs llama.cpp support for the arch (hyper-connections, PLE, native sparse-attention indexer); the d2t trim is an optional speed patch on top.

Credits

This is a remix, not an original quant. All credit to the upstream authors:

What this model is

Qwen3.8-Flash-Next is a preview of the Qwen4 architecture: a very wide but extremely sparse MoE with hybrid (linear-attention/SSM + periodic full attention) layers, native sparse-attention indexing, hyper-connections, and a per-layer n-gram embedding (“PLE”) block. Despite 176.9 B total parameters it activates only ~4–5 B per token, so it runs in roughly the memory footprint of a mid-size dense model — while behaving like a much larger one.

This particular build packages the ISTA DASLab GSQ-RCO IQ3_S quantization as the main model and attaches the Unsloth MTP draft checkpoint via Ollama’s DRAFT directive, enabling speculative decoding through the model’s own nextn head.

Base model specification (from GGUF metadata, verified)

Quantization details (ISTA-DASLab GSQ-RCO IQ3_S)

GSQ + RCO is a mixed-precision, imatrix-guided quantization (calibration set calib_qwen3.8-flash-next.tokids, 1,000 chunks, 926 entries). Tensor-level breakdown as shipped in this model (1,223 tensors, 67.78 GB of tensor data):

Notes: - token_embd = IQ4_XS (337.7 MB), output.weight (LM head) = Q6_K (521.5 MB); total tensor data 67.78 GB. - Expert tensors are 3-D ([640, 2560, 512]) — all 512 experts fused per projection. - Router gating, hyper-connections, and the indexer stay BF16; the quantizer’s imatrix chose per-layer expert bit widths — verified layer map: - Expert gate/up: IQ2_S ×20 layers · IQ3_XXS ×17 · IQ3_S ×10 · IQ4_XS ×1 (layer 47, the last layer, gets the highest expert precision) - Expert down: IQ4_NL ×39 layers, F16 ×9 layers (verified: 1, 8, 10, 12, 13, 20, 21, 30, 36) — down-projections are kept much higher precision than gate/up - Shared expert (*_shexp, one per layer): mixed Q4_K/Q5_K/Q6_K/IQ4_XS gate+up · IQ4_NL down ×47, Q8_0 down ×1 (layer 47) - Attention: the 12 full-attention layers (3, 7, 11, … 47) use split attn_q/k/v (IQ4_XS → Q6_K mix); the 36 linear-attention/SSM layers use fused attn_qkv (mostly Q6_K, some Q5_K)

Layer structure (verified from tensor map)

• 48 layers total: 36 linear-attention/SSM layers + 12 full-attention layers at indices 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47 (exactly full_attention_interval = 4).
• Full-attention layers carry the sparse indexer (indexer.q_proj/k_proj + norms) and split Q/K/V projections; linear layers carry fused attn_qkv, attn_gate, and the SSM tensors instead.
• PLE block lives in layer 1 only (ple_key [2560, 10240], ple_value, ple_conv1d, 3 norms) — the n-gram embedding lookup feeds that layer.

MTP draft head (Ckpt_Q38 from Unsloth)

A standalone single-block nextn model mirroring the main architecture (34 tensors, 3.88 B elements, 2.78 GB, general.size_label = 512x1.4B). Per-tensor quant tally from the file itself:

How the MTP setup works (Ollama)

Built with a Modelfile of this shape:

FROM <ISTA IQ3_S main GGUF (2 shards, merged)>
DRAFT <Unsloth Ckpt_Q38 nextn GGUF>
PARAMETER draft_num_predict 1

• draft_num_predict 1 — the draft proposes 1 speculative token per step; the target verifies it in the same forward pass. Raise to 2 for slightly more throughput if your accept rate is high (the head is natively trained for 1-token prediction, so 1 is the safe default).
• Speculative decoding here is “free” MTP — the model family ships its own nextn head; no separate draft model download needed.
• Verified working: MTP accept stats visible in runner logs on the patched llama.cpp build (Ollama 0.40.x, libllama.so rebuilt with the d2t patch).

Running it

ollama run qwen38flashnext-iq3s-mtp:latest

• Measured footprint: ~19 GB resident at a 98K context on 2× RTX 5060 Ti (16 GB total VRAM) — 95% GPU / 5% CPU split with the model pinned in VRAM. It fits because only ~11 of 512 experts per layer (10 routed + 1 shared) and 12 of 48 layers carry real KV are hot per token; SSM layers keep fixed-size state instead of KV.
  • Note: the MoE weights are still fully resident; the “~4–5 B active” figure is about compute, not memory. Plan for ~20–22 GB of combined VRAM+system RAM headroom at 100K context.
• KV cache is small thanks to 2 KV heads × 256 dim and SSM layers holding state instead of KV for ³⁄₄ of layers. Native sparse attention (top_k 2048) keeps long-context attention cheap on the full-attn layers.
• Set OLLAMA_CONTEXT_LENGTH or /set parameter num_ctx as needed; the model advertises 262K.
• Tool calling and thinking modes are supported (capabilities: tools, thinking).

Known caveats

1. Patched runner required (see warning at top). Stock Ollama/llama.cpp builds without qwen4exp support will not load this model; the d2t draft-vocab trim is an extra patch that speeds up MTP but is not mandatory.
2. No vision. The chat template contains image macros but this GGUF has no vision tower. Text-only.
3. IQ3_S-class quant of a 177B MoE is strong but not lossless — expect small deltas vs BF16 on math-heavy tasks; the GSQ-RCO imatrix targeting recovers most of it.
4. Sharded upstream GGUF (00001-of-00002) was merged during import; blob layout differs from the HF repo file layout.

Suggested citation

Qwen Team — Qwen3.8-Flash-Next; ISTA DASLab — GSQ-RCO IQ3_S GGUF quantization; Unsloth — Qwen3.8-Flash-Next-GGUF / MTP (nextn) checkpoint; assembled with Ollama DRAFT speculative-decoding directive.