fast-qwen ⚡

Qwen2.5-1.5B-Instruct served fast on modest hardware — a benchmark-driven optimization study, not a retrain.

Same weights as the official Q4_K_M quantization of Qwen2.5-1.5B-Instruct, packaged with a tuned ChatML template and a 4K context default.

Measured results (GTX 1650 4GB, i7-9750HF, llama.cpp)

Quality verified three ways (identical inputs across all configs):

• Perplexity on a held-out corpus: +3% vs the original FP16 weights
• 24-question task suite (math, logic, code, facts): ¹⁵⁄₂₄ optimized vs ¹³⁄₂₄ original — no drop
• Side-by-side outputs: wording drifts slightly, meaning and correctness preserved on every identity-check question

Techniques applied

Q4_K_M quantization → full GPU offload → FlashAttention → n-gram speculative decoding → 8-bit KV cache.

Honest negative result: classic draft-model speculative decoding made this model size slower (27 vs 78 tok/s) — it pays off for large, slow models, not a 1.5B sharing a 4GB GPU with the desktop. Full methodology, raw logs, and a reproducible benchmark harness live in the fast-llm project repository.

Usage

ollama run hisparshmishra1/fast-qwen

Tip: close GPU-heavy apps before chatting — a 4GB card shared with the desktop may split the model across CPU/GPU (~40 tok/s) instead of running fully on GPU (~78 tok/s).