493 Downloads Updated 3 weeks ago
Gemma 4 12B (QAT Q4_0) with Multi-Token Prediction speculative decoding — a fast, tool-capable local model for 16 GB - Apple Silicon Macs (~21 tok/s, ~1.4× via MTP).
ollama run pierreprudh/gemma4-12b-mtp
Applications
ollama launch claude --model pierreprudh/gemma4-12b-mtp
ollama launch codex-app --model pierreprudh/gemma4-12b-mtp
ollama launch openclaw --model pierreprudh/gemma4-12b-mtp
ollama launch hermes --model pierreprudh/gemma4-12b-mtp
ollama launch codex --model pierreprudh/gemma4-12b-mtp
ollama launch opencode --model pierreprudh/gemma4-12b-mtp
Models
View all →Readme
gemma4-12b-mtp
Google Gemma 4 12B (instruction-tuned, QAT Q4_0) packaged with a Multi-Token Prediction (MTP) drafter for speculative decoding — a fast, tool-capable local model that runs comfortably on a 16 GB Apple Silicon Mac.
Why
The unified multimodal MLX build of Gemma 4 12B is ~11.7 GB and swap-thrashes a 16 GB Mac. This build is the text GGUF + MTP draft head, so it stays lean and uses speculative decoding for extra speed.
| Build | tok/s | Notes |
|---|---|---|
plain gemma4:12b (GGUF) |
~15.7 | baseline |
| this model (MTP) | ~21.4 | ~1.4× via speculative decoding |
Measured on an M5 MacBook Pro (16 GB), 100% GPU offload, memory healthy (no swap).
Usage
ollama run pierreprudh/gemma4-12b-mtp
Supports native tool-calling (verified) and the Gemma 4 thinking channel. The MTP weights are tuned for coding tasks, where the speedup is largest.
How it was built
MTP runs as standard speculative decoding: a small draft model proposes
several tokens ahead and the base model verifies them in one pass. Both
parts are pre-quantized GGUFs, so ollama create just packages them — no
--quantize step (which currently panics in Ollama’s experimental MLX
quantizer on macOS).
Components (from unsloth/gemma-4-12B-it-qat-GGUF):
- Base: gemma-4-12B-it-qat-UD-Q4_K_XL.gguf (~6.3 GB)
- Draft: MTP/gemma-4-12B-it-BF16-MTP.gguf (~0.8 GB, BF16 for best draft acceptance)
Modelfile
FROM ./gemma-4-12B-it-qat-UD-Q4_K_XL.gguf
DRAFT ./MTP/gemma-4-12B-it-BF16-MTP.gguf
Create
ollama create gemma4-12b-mtp -f Modelfile
(No --quantize / --draft-quantize: the GGUFs are already quantized, which
also avoids the mlx: There is no Stream(gpu, 1) panic in the experimental
safetensors-quantize path.)
Attribution & license
Derived from Google Gemma 4 (GGUF conversion by Unsloth). Use is governed by the Gemma Terms of Use. “Gemma” is provided by Google; this is an unofficial community packaging.