NOTE: This model has been heavily vocab-sliced to reduce non-coding and non-English tokens. YMMV. (Mandarin, Korean, Vietnamese, Hindi, Arabic, Japanese, Russian)

SOURCES:

** https://huggingface.co/Qwen/Qwen3.8-Flash-Next

** https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

** https://huggingface.co/datasets/eaddario/imatrix-calibration

Qwen3.8-Flash-Next Architecture

This experimental preview of the architecture that will underpin Qwen4 is built around a fundamental rethinking of how the core components of modern large language models (LLMs) interact at scale.

Highlights The first open-weight release under this architecture is Qwen3.8-Flash-Next, which introduces:

Hybrid Attention with QSA: The Gated DeltaNet and Gated Attention pairing has been reworked into Gated DeltaNet and Qwen Sparse Attention (QSA). Rather than selecting individual tokens for processing, QSA operates at the micro-block level. This cuts long-context latency significantly, a critical gain as agentic workloads increasingly dominate real-world usage. Gated Residual: Residual streams with normalization are what make deep LLM training manageable. Gated Residual modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and a per-branch scalar write gate. This brings finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low. N-gram Embedding: Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality. Tailored Training Recipe: The Muon and AdamW optimizers are applied to specific weight categories to maximize efficiency. Guided by refitted scaling laws, we eliminate traditional batch-size warmups and start directly at the target batch size, substantially reducing total optimizer steps while safely supporting larger learning rates for robust convergence. For more details, please refer to our blog post Qwen3.8-Flash-Next and the technical report.

We are excited to embark on this next chapter with you and welcome your feedback as we build what comes next.

Model Overview Type: Causal Language Model with Vision Encoder Training Stage: Pre-training & Post-training Language Model Number of Parameters: 125B with 6B activated, plus 51B n-gram embedding and 4B MTP Hidden Dimension: 2560 Token Embedding: 248320 (Padded) N-gram Embedding: 20,000,000 (bigrams/trigrams at layer 2) Number of Layers: 48 Hidden Layout: 12 × (3 × (Gated DeltaNet → MoE) → 1 × (Qwen Sparse Attention → MoE)) Gated DeltaNet: Number of Linear Attention Heads: 48 for V and 16 for QK Head Dimension: 128 Qwen Sparse Attention: Number of Attention Heads: 24 for Q and 2 for KV Head Dimension: 256 Rotary Position Embedding Dimension: 64 Indexer Structure: MQA with 4 Query Heads and 1 Shared Key Head Indexer Head Dimension: 128 Budget: 512 blocks or 2048 tokens Mixture Of Experts Number of Experts: 512 Number of Activated Experts: 10 Routed + 1 Shared Expert Intermediate Dimension: 640 Gated Residual: Number of Branches: 4 Bottleneck Rank: 320 LM Output: 248320 (Padded) MTP: 1 layer, trained with multi-steps Context Length: 262,144 natively and extensible up to 1,000,000 tokens.