1,229 1 month ago

Unsloth Dynamic & MTP

vision tools thinking
ollama run jetelain/Qwen3.6-35B-A3B:q4

Details

1 month ago

d35140ecfcb1 · 24GB

qwen35moe
·
35.5B
·
Q4_K_M
clip
·
447M
·
BF16
{ "draft_num_predict": 2, "min_p": 0, "num_ctx": 65536, "presence_penalty": 0, "
{{ .Prompt }}

Readme

Qwen 3.6 35B A3B (Unsloth Dynamic & Speculative Decoding)

Advanced Mixture-of-Experts (MoE) model with 8 of 60 active experts (~3.2B active parameters) from Alibaba Cloud, packaged with Unsloth Dynamic Quantization, native Multi-Token Prediction (MTP / Speculative Decoding), and full Multimodal Vision (mmproj) support.

Optimized specifically for high-throughput coding, agentic workflows, and ultra-large context processing (up to 256k tokens).

🚀 Key Highlights

  • Unsloth Dynamic Quantization (UD): Selective precision keeps critical attention and gate layers at high precision for near-lossless output while compressing expert weights.
  • Native MTP Speculative Decoding (draft_num_predict: 2): Delivers an instant 1.3x–1.8x generation speedup with zero quality degradation.
  • Multimodal Vision Included: Bundled with mmproj-BF16 for high-resolution image analysis and OCR.
  • Extended Context Window: Default configured at 64k or 128k, scalable up to the native 262,144 RoPE context limit.
  • GQA Efficiency: Uses a 2-KV-head architecture consuming only ~22 KB of KV cache per token, enabling massive context on modest memory.

📦 Tags & Variants

Tag Quantization Size Default Context Target Use Case
latest UD-Q8_K_XL 40 GB 128k (131072) Maximum general reasoning precision (temp 1.0)
code UD-Q8_K_XL 40 GB 128k (131072) High-precision coding and architecture tasks (temp 0.6)
q4 UD-Q4_K_XL 22 GB 128k (131072) Recommended Sweet Spot: High quality + fast speed (~35 t/s on 8GB, ~90 t/s on 16GB)
q4-code UD-Q4_K_XL 22 GB 128k (131072) Coding sweet spot for OpenCode, Cline, Visual Studio
i3 UD-IQ3_S 15.3 GB 64k (65536) Maximum raw throughput (~115 t/s on 16GB GPU), lower precision
i3-code UD-IQ3_S 15.3 GB 64k (65536) Ultra-high-speed draft/coding, lower precision

💡 Usage Recommendations

1. Fast CLI Run

# Recommended sweet spot (quality + speed):
ollama run jetelain/Qwen3.6-35B-A3B:q4

# For highest precision coding tasks:
ollama run jetelain/Qwen3.6-35B-A3B:code

# For maximum speed on 16GB GPUs:
ollama run jetelain/Qwen3.6-35B-A3B:i3-code

2. Hardware Recommendations

  • 8 GB VRAM (Laptops with 64GB RAM):
    • Recommended Sweet Spot: Use :q4 (or :q4-code). Fixes i3 quality/fidelity issues while sustaining >30 tokens/s up to 128k context and 23.5 tokens/s at 160k.
    • Maximum Quality: Use :latest or :code (Q8). Sustains ≥ 20 tokens/s up to 112k context and ~17 t/s at 160k with near-lossless FP16 reasoning.
    • Maximum Speed: Use :i3 or :i3-code if raw throughput (~31–41 t/s) is prioritized over nuance.
  • 16 GB VRAM (RTX 4080 / 4090 Mobile):
    • Use :q4 for an ideal balance of quality and speed (~85–100 tokens/s up to 160k context).
    • Use :latest or :code (Q8) for maximum precision with 64GB host RAM (~58 tokens/s).
    • Use :i3 or :i3-code for 100% full GPU offload at ~115 tokens/s up to 160k context.
  • Unified Memory (64–96 GB / Apple Silicon / AMD Strix Halo): Use :latest or :code (Q8). Runs 100% in VRAM at ~70–74 tokens/s all the way to 256k context.

3. 💡 Hardware Alias Tip (Zero Disk Overhead)

Create a clean local alias tailored to your GPU VRAM ceiling:

# On 8 GB Laptops with 64 GB RAM (160k context, q4 sweet spot at ~24-35 t/s):
curl http://localhost:11434/api/create -d '{
  "model": "Qwen3.6-35B-A3B-Code",
  "from": "jetelain/Qwen3.6-35B-A3B:q4",
  "parameters": {
    "num_ctx": 163840
  }
}'

# On 16 GB Desktop GPUs (160k context, full GPU offload at ~115 t/s):
curl http://localhost:11434/api/create -d '{
  "model": "Qwen3.6-35B-A3B-Code",
  "from": "jetelain/Qwen3.6-35B-A3B:i3-code",
  "parameters": {
    "num_ctx": 163840
  }
}'

# On 64-96 GB Unified Memory Servers (256k native max context at ~73 t/s):
curl http://localhost:11434/api/create -d '{
  "model": "Qwen3.6-35B-A3B-Code",
  "from": "jetelain/Qwen3.6-35B-A3B:code",
  "parameters": {
    "num_ctx": 262144
  }
}'

4. OpenCode / Agent Integration

Recommended settings for ~/.config/opencode/opencode.json:

{
  "model": "ollama/Qwen3.6-35B-A3B-Code:latest",
  "limit": {
    "context": 262144,
    "output": 65536
  },
  "reasoning": true
}