Qwen 3.5 4B (Unsloth Dynamic Q4 & Q6 with MTP & Vision)

Ultra-fast, compact multimodal foundation model from Alibaba Cloud, packaged with Unsloth Dynamic quantization (Q4_K_XL & Q6_K_XL), native Multi-Token Prediction (draft_num_predict: 2), and Multimodal Vision (mmproj-BF16).

Engineered for edge devices, laptops, fast agent reasoning loops, and real-time interactive chats.

🚀 Key Highlights

• Blazing Speed: Generates at ~112–115 tokens/s on standard 8 GB mobile GPUs (Q4).
• Massive Context in Small VRAM: Maintains 100% full GPU offload up to 78k context in Q4 within an 8 GB VRAM budget.
• High-Precision Q6 Option: Sustains >22 tokens/s across the entire 128k context range (~43 t/s at 64k) with superior code/reasoning fidelity.
• Integrated Vision: Process images, screenshots, diagrams, and OCR natively via mmproj-BF16.
• MTP Speculative Decoding: Preconfigured with draft_num_predict: 2 (60%–96% draft acceptance rate).
• Default 64k Context: Ready out-of-the-box for large documents and multi-turn conversations.

📦 Tags

• latest: Unsloth Dynamic Q4_K_XL (~3.0 GB model + 675 MB mmproj), 64k default context. Maximum speed (~112 t/s) with 100% GPU offload up to 78k context.
• q6: Unsloth Dynamic Q6_K_XL with MTP (~4.26 GB model + 675 MB mmproj), 64k default context. Higher reasoning/coding precision, sustaining >22 t/s across the entire 128k context range (~43 t/s at 64k).

💡 Usage Recommendations

# Run interactive chat (Q4 speed champion)
ollama run jetelain/Qwen3.5-4B

# High precision Q6 chat
ollama run jetelain/Qwen3.5-4B:q6

# Multimodal prompt with an image
ollama run jetelain/Qwen3.5-4B "Describe this diagram: /path/to/diagram.png"

Hardware Recommendations

• Any 8 GB GPU (RTX 3060/3070/4060/4070):
  • latest (Q4 MTP): 100% full GPU offload up to 78k tokens at >110 t/s.
  • q6 (Q6 MTP): High-precision reasoning, maintains >22 t/s up to 128k context (~43 t/s at 64k).
• CPU / Apple Silicon M-series: Runs effortlessly with minimal RAM consumption (< 5 GB).

💡 Hardware Alias Tip (Zero Disk Overhead)

Lock in hardware-tailored context for maximum performance on 8 GB GPUs:

# 78k context alias for maximum 100% GPU offload (Q4)
curl http://localhost:11434/api/create -d '{
  "model": "Qwen3.5-4B",
  "from": "jetelain/Qwen3.5-4B",
  "parameters": {
    "num_ctx": 79872
  }
}'

# 64k context alias for high-precision Q6
curl http://localhost:11434/api/create -d '{
  "model": "Qwen3.5-4B-Q6",
  "from": "jetelain/Qwen3.5-4B:q6",
  "parameters": {
    "num_ctx": 65536
  }
}'