Qwen3.8-27B-TurboFCFusion NVFP4+Q3K Mixed Quantization

A high-fidelity mixed-precision requantization of DavidAU’s Qwen3.8-27B-TURBO-Fable-Cold-Fusion model, optimized for Blackwell GPU tensor-core inference while preserving near-8-bit quality.

Model Overview

Quantization Strategy

This build uses a carefully tuned mixed-precision recipe that minimizes file size while preserving quality, leveraging the model’s inherent 4-bit resilience (99% of 8-bit performance per base model benchmarks).

Tensor Type Distribution

Recipe Details

# F32 protections (CUDA kernel requirements + MTP loader safety)
^blk\.\d+\.attn_norm\.weight$=f32
^blk\.\d+\.post_attention_norm\.weight$=f32
^blk\.\d+\.attn_q_norm\.weight$=f32
^blk\.\d+\.attn_k_norm\.weight$=f32
^blk\.\d+\.ssm_norm\.weight$=f32
^blk\.\d+\.nextn\.enorm\.weight$=f32
^blk\.\d+\.nextn\.hnorm\.weight$=f32
^blk\.\d+\.nextn\.shared_head_norm\.weight$=f32
^output_norm\.weight$=f32
^blk\.\d+\.ssm_conv1d\.weight$=f32
^blk\.\d+\.ssm_dt\.bias$=f32
^blk\.\d+\.ssm_a$=f32

# F16 singular-collapse protections (verified against reference NVFP4 build)
^blk\.0\.attn_gate\.weight$=f16
^blk\.0\.attn_qkv\.weight$=f16
^blk\.0\.ffn_down\.weight$=f16
^blk\.13\.ffn_down\.weight$=f16

# q3_K middle layers (least-sensitive region, bulk of size reduction)
^blk\.(1[4-9]|2[0-9]|3[0-9]|4[0-9]|50)\..*\.weight$=q3_K

# Everything else → NVFP4 (catch-all last)
.*=nvfp4

Why This Placement?

• NVFP4 as default: Provides the best quality-per-byte ratio and fastest inference on Blackwell GPUs (RTX 50-series) with native FP4 tensor-core operations and imatrix-calibrated input scales.
• q3_K on layers 14–50: The middle 37 layers are empirically the least sensitive to quantization. This follows the proven Bartowski Q3_K_L strategy. The model’s 4-bit resilience makes this nearly lossless while saving ~1.4 GB vs all-NVFP4.
• F16 protections: Four specific tensors identified as prone to catastrophic singular-value collapse when quantized to NVFP4 (verified against the reference NVFP4 build of this architecture).
• F32 protections: Required by CUDA kernel constraints (ssm-conv.cu asserts F32 for conv weights) and MTP loader compatibility (norm tensors cannot declare .scale tensors in the loader).

MTP (Multi-Token Prediction) Support

The MTP head structure is fully preserved:

• qwen35.nextn_predict_layers = 1 metadata retained
• 10 nextn tensors present (blk.64.nextn.*) quantized to NVFP4
• MTP head components: enorm, hnorm, shared_head_norm (F32), eh_proj, attn_qkv, attn_gate, attn_output, ffn_down, ffn_gate, ffn_up (NVFP4)

To use MTP spec-decode, run via llama-server (not Ollama, which lacks spec-decode support):

llama-server -m <model>.gguf --spec-type draft-mtp --spec-draft-n-max 4

Quality Verification

Greedy decoding (temp=0, seed=42) comparison against the Q8_0 original across four diverse prompt categories:

Conclusion: No perceptible quality degradation from the q3_K middle layers. The model’s inherent 4-bit resilience holds.

Performance Benchmarks (Measured)

Benchmarks measured on dual RTX 5060 Ti (32 GB total VRAM, ~448 GB/s bandwidth each) with Ollama 0.33.3.

Token Generation Speed

Sustained throughput: ~26-27 tokens/sec Model load time: 8-17 seconds (varies by whether already in memory)

Quality Check

Haiku test output:

Dark morning brew steams Warm cup fills my empty hands Awakening mind

Proper 5-7-5 structure with coherent imagery. Story test produced 4,755 characters of coherent prose about a robot learning to cook.

Comparison with Other Models

Speed Comparison (estimates on similar hardware)

Key Benchmark Scores (Qwen3.8-27B base model)

• SWE-bench Pro (coding): 61.7% — best dense model
• LiveCodeBench v6: 90.3%
• GPQA Diamond (science knowledge): 89.2-90.5%
• MMLU-Pro: 84.3%
• SWE-bench (Vals): 86.0%
• Overall BenchLM rank: #39 of 483 models

Tradeoffs

• Quality vs Speed: Dense 27B model provides higher quality than smaller models but slower than MoE alternatives
• VRAM vs Performance: Custom NVFP4/Q3K quantization keeps model runnable on 32GB while preserving most accuracy
• Best use case: Coding tasks, multilingual work, long-context reasoning on consumer hardware
• Bottom line: Optimizes for quality per VRAM — getting 27B dense model performance while staying within your 32GB limit

Performance Notes

• Eval speed: 25.8 t/s on dual RTX 5060 Ti with full GPU offload
• Prompt processing: 188–364 t/s

Importing to Ollama

ollama create aiconjured/Qwen3.8-27B-TurboFCFusion-735-882-NVFP4-Q3K:latest \
  --file Modelfile

Modelfile:

FROM <path-to-gguf>
FROM <path-to-mmproj>
PARAMETER temperature 0.7
PARAMETER top_p 0.9

Hardware Requirements

Files Included

• Qwen3.8-27B-TurboFCFusion-735-882-Here-Uncen-NEO-CODER-MAX-MTP-NVFP4-Q3K-MID.gguf — Main model (13.2 GB)
• mmproj-BF16.gguf — CLIP vision projector (889 MB, unchanged from base)

Reproduction

To reproduce this quantization:

# 1. Download Q8_0 source from DavidAU's repo
# 2. Build imatrix from corpus
./llama-imatrix -m source-Q8_0.gguf -f corpus.txt -o imatrix.gguf

# 3. Quantize with recipe
./llama-quantize --allow-requantize --imatrix imatrix.gguf \
  --tensor-type-file recipe-nvfp4-q3K.txt \
  source-Q8_0.gguf output.gguf Q8_0 8

# 4. Patch file_type to 39 (MOSTLY_NVFP4)
# (Use a GGUF header patcher — the quantizer doesn't set this)

License

Same as base model: Apache 2.0

Acknowledgments

• Base model: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF — Cold Fusion + Fable Fusion training, 735 ARC-C / 882 ARC-E intelligence benchmarks, engineered for 4-bit resilience
• Original architecture: Qwen/Qwen3.8-27B — hybrid SSM-attention design with MTP support
• NVFP4 quantization: ggml-org/llama.cpp — native Blackwell tensor-core FP4 operations
• Singular-collapse protection patterns: Verified against reference NVFP4 builds of this architecture
• Quality benchmarks: Informed by DavidAU’s published 4-bit vs 8-bit performance comparisons

About the Author

Part of the AIconjured NVFP4 quantization series.