decadence/gemma-4:12b-it-qat-mlx-4bit

decadence/ gemma-4:12b-it-qat-mlx-4bit

655 Downloads Updated 1 week ago

12B — QAT — GGUF — MLX — uncensored — sampling | ~7-13 GB

vision tools thinking audio

ollama run decadence/gemma-4:12b-it-qat-mlx-4bit

curl http://localhost:11434/api/chat \
  -d '{
    "model": "decadence/gemma-4:12b-it-qat-mlx-4bit",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from ollama import chat

response = chat(
    model='decadence/gemma-4:12b-it-qat-mlx-4bit',
    messages=[{'role': 'user', 'content': 'Hello!'}],
)
print(response.message.content)

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'decadence/gemma-4:12b-it-qat-mlx-4bit',
  messages: [{role: 'user', content: 'Hello!'}],
})
console.log(response.message.content)

Details

Updated 1 week ago

1 week ago

b29069e5cbb5 · 7.9GB

json

{ "architectures": [ "Gemma4UnifiedForConditionalGeneration" ], "audio_config": { "_name_or_path": "

5.2kB

json

{ "bos_token_id": 2, "do_sample": true, "eos_token_id": [ 1, 106, 50 ], "pad_token_id": 0, "suppress

260B

json

{ "version": 2, "weight_format": "mxfp4", "profile": "MXFP4", "source_model": { "name": "gemma-4-12B

1.6kB

json

{ "audio_ms_per_token": 40, "audio_seq_length": 750, "feature_extractor": { "audio_samples_per_token

1.4kB

json

{ "version": "1.0", "truncation": null, "padding": null, "added_tokens": [ { "id": 0, "content": "<p

32MB

json

{ "audio_token": "<|audio|>", "backend": "tokenizers", "boa_token": "<|audio>", "boi_token": "<|imag

20kB

params

{ "temperature": 1, "top_k": 64, "top_p": 0.95 }

42B

677 tensors

7.9GB

Readme

Gemma 4 12B

QAT model is a fine-tuned Large Language Model (LLM) that has undergone Quantization-Aware Training, specifically optimized during its training phase to tolerate low-bit precision (such as 4-bit)

model	type	quant	unc	size	source
gemma-4:12b-it-abliterated-mlx-8bit	MLX	mxfp8	✓	13 GB	AEON-7
gemma-4:12b-it-mlx-8bit	MLX	int8	-	12 GB	lmstudio
gemma-4:12b-it-qat-heretic-q4_0	GGUF	Q4_0	✓	6.9 GB	SC117
gemma-4:12b-it-qat-mlx-4bit	MLX	mxfp4	-	7.9 GB	OsaurusAI
gemma-4:12b-it-qat-q4_0	GGUF	Q4_0	-	7.2 GB	google

Sampling Parameters

Following Gemma 4 recommended sampling parameters:

parameter	value
temperature	1.0
top_p	0.95
top_k	64

### Gemma 4 12B

> QAT model is a fine-tuned Large Language Model (LLM) that has undergone Quantization-Aware Training, specifically optimized during its training phase to tolerate low-bit precision (such as 4-bit)

model                               | type | quant | unc | size   | source
------------------------------------|------|-------|-----|--------|---
gemma-4:12b-it-abliterated-mlx-8bit | MLX  | mxfp8 | ✓   | 13 GB  | [AEON-7](https://huggingface.co/AEON-7/Gemma-4-12B-it-AEON-Abliterated-MLX-8bit)
gemma-4:12b-it-mlx-8bit             | MLX  | int8  | -   | 12 GB  | [lmstudio](https://huggingface.co/lmstudio-community/gemma-4-12B-it-MLX-8bit)
gemma-4:12b-it-**qat**-heretic-q4_0     | GGUF | Q4_0  | ✓   | 6.9 GB | [SC117](https://huggingface.co/SC117/gemma-4-12B-it-heretic-QAT-GGUF)
gemma-4:12b-it-**qat**-mlx-4bit         | MLX  | mxfp4 | -   | 7.9 GB | [OsaurusAI](https://huggingface.co/OsaurusAI/gemma-4-12B-it-qat-MXFP4)
gemma-4:12b-it-**qat**-q4_0             | GGUF | Q4_0  | -   | 7.2 GB | [google](https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf)

### Sampling Parameters

Following [Gemma 4 recommended sampling parameters:](https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf#1-sampling-parameters)

parameter   | value 
------------|---
temperature | 1.0 
top_p       | 0.95
top_k       | 64

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)