pierreprudh/ gemma4-12b-mtp:latest

493 Downloads Updated 3 weeks ago

Gemma 4 12B (QAT Q4_0) with Multi-Token Prediction speculative decoding — a fast, tool-capable local model for 16 GB - Apple Silicon Macs (~21 tok/s, ~1.4× via MTP).

tools thinking

ollama run pierreprudh/gemma4-12b-mtp

curl http://localhost:11434/api/chat \
  -d '{
    "model": "pierreprudh/gemma4-12b-mtp",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from ollama import chat

response = chat(
    model='pierreprudh/gemma4-12b-mtp',
    messages=[{'role': 'user', 'content': 'Hello!'}],
)
print(response.message.content)

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'pierreprudh/gemma4-12b-mtp',
  messages: [{role: 'user', content: 'Hello!'}],
})
console.log(response.message.content)

Details

Updated 3 weeks ago

3 weeks ago

cf53de82619b · 7.6GB ·

model

archgemma4

·

parameters11.9B

·

quantizationQ4_0

6.7GB

draft

GGUF��1��,��general.architecture��gemma4-assistant��general.type

862MB

params

{ "stop": [ "<turn|>" ] }

31B

Readme

gemma4-12b-mtp

Google Gemma 4 12B (instruction-tuned, QAT Q4_0) packaged with a Multi-Token Prediction (MTP) drafter for speculative decoding — a fast, tool-capable local model that runs comfortably on a 16 GB Apple Silicon Mac.

Why

The unified multimodal MLX build of Gemma 4 12B is ~11.7 GB and swap-thrashes a 16 GB Mac. This build is the text GGUF + MTP draft head, so it stays lean and uses speculative decoding for extra speed.

Build	tok/s	Notes
plain `gemma4:12b` (GGUF)	~15.7	baseline
this model (MTP)	~21.4	~1.4× via speculative decoding

Measured on an M5 MacBook Pro (16 GB), 100% GPU offload, memory healthy (no swap).

Usage

ollama run pierreprudh/gemma4-12b-mtp

Supports native tool-calling (verified) and the Gemma 4 thinking channel. The MTP weights are tuned for coding tasks, where the speedup is largest.

How it was built

MTP runs as standard speculative decoding: a small draft model proposes several tokens ahead and the base model verifies them in one pass. Both parts are pre-quantized GGUFs, so ollama create just packages them — no --quantize step (which currently panics in Ollama’s experimental MLX quantizer on macOS).

Components (from unsloth/gemma-4-12B-it-qat-GGUF): - Base: gemma-4-12B-it-qat-UD-Q4_K_XL.gguf (~6.3 GB) - Draft: MTP/gemma-4-12B-it-BF16-MTP.gguf (~0.8 GB, BF16 for best draft acceptance)

Modelfile

FROM ./gemma-4-12B-it-qat-UD-Q4_K_XL.gguf
DRAFT ./MTP/gemma-4-12B-it-BF16-MTP.gguf

Create

ollama create gemma4-12b-mtp -f Modelfile

(No --quantize / --draft-quantize: the GGUFs are already quantized, which also avoids the mlx: There is no Stream(gpu, 1) panic in the experimental safetensors-quantize path.)

Attribution & license

Derived from Google Gemma 4 (GGUF conversion by Unsloth). Use is governed by the Gemma Terms of Use. “Gemma” is provided by Google; this is an unofficial community packaging.

# gemma4-12b-mtp

Google **Gemma 4 12B** (instruction-tuned, QAT Q4_0) packaged with a
**Multi-Token Prediction (MTP) drafter** for speculative decoding — a fast,
tool-capable local model that runs comfortably on a 16 GB Apple Silicon Mac.

## Why

The unified multimodal MLX build of Gemma 4 12B is ~11.7 GB and swap-thrashes a
16 GB Mac. This build is the **text GGUF + MTP draft head**, so it stays lean
*and* uses speculative decoding for extra speed.

| Build | tok/s | Notes |
|---|---|---|
| plain `gemma4:12b` (GGUF) | ~15.7 | baseline |
| **this model (MTP)** | **~21.4** | ~1.4× via speculative decoding |

Measured on an M5 MacBook Pro (16 GB), 100% GPU offload, memory healthy (no swap).

## Usage

```bash
ollama run pierreprudh/gemma4-12b-mtp
```

Supports native tool-calling (verified) and the Gemma 4 thinking channel.
The MTP weights are tuned for coding tasks, where the speedup is largest.

## How it was built

MTP runs as standard speculative decoding: a small **draft** model proposes
several tokens ahead and the **base** model verifies them in one pass. Both
parts are pre-quantized GGUFs, so `ollama create` just packages them — no
`--quantize` step (which currently panics in Ollama's experimental MLX
quantizer on macOS).

**Components** (from [`unsloth/gemma-4-12B-it-qat-GGUF`](https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF)):
- Base: `gemma-4-12B-it-qat-UD-Q4_K_XL.gguf` (~6.3 GB)
- Draft: `MTP/gemma-4-12B-it-BF16-MTP.gguf` (~0.8 GB, BF16 for best draft acceptance)

**Modelfile**

```
FROM ./gemma-4-12B-it-qat-UD-Q4_K_XL.gguf
DRAFT ./MTP/gemma-4-12B-it-BF16-MTP.gguf
```

**Create**

```bash
ollama create gemma4-12b-mtp -f Modelfile
```

(No `--quantize` / `--draft-quantize`: the GGUFs are already quantized, which
also avoids the `mlx: There is no Stream(gpu, 1)` panic in the experimental
safetensors-quantize path.)

## Attribution & license

Derived from Google **Gemma 4** (GGUF conversion by Unsloth). Use is governed by
the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). "Gemma" is provided
by Google; this is an unofficial community packaging.

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)