Qwen3.8-27B

GGUF quantizations of Qwen3.8-27B for Ollama, with context windows tuned per VRAM budget. The K_L builds use the real unsloth/Qwen3.8-27B-GGUF quants (UD = ultra-dynamic), include the vision projector (mmproj-F16), and enable MTP speculative decoding where VRAM allows.

Models

Tag	Quant	Size	Context	Vision	MTP	VRAM target	Use case
`Q2_K_XL`	2-bit	10.7 GB	256K	✅	✅	24 GB	2-bit XL, full native context
`Q2_K_XL-12gb`	2-bit	10.7 GB	8K	—	—	12 GB	2-bit XL, optimized for 12 GB VRAM
`Q2_K_XL-16gb`	2-bit	10.7 GB	128K	—	—	16 GB	2-bit XL, optimized for 16 GB VRAM
`Q3_K_M-128k`	3-bit	13.8 GB	128K	✅	✅	24 GB	Balanced size / quality, long context
`Q3_K_M-192k`	3-bit	13.8 GB	192K	—	✅	24 GB	Balanced, longer context
`Q3_K_M`	3-bit	13.8 GB	24K	—	✅	16 GB	3-bit KM, shares blob with 192k
`Q3_K_XL`	3-bit	13.4 GB	219K	✅	✅	24 GB	3-bit XL, high quality
`Q4_K_XL`	4-bit	17.9 GB	120K	✅	✅	24 GB	Recommended default, 100% GPU
`map-k4v`	4-bit	19 GB	120K	✅	✅	24 GB	4-bit XL with ngram map, requires custom ollama
`Q5_K_XL`	5-bit	20.2 GB	60K	✅	✅	24 GB	5-bit XL, max 100% GPU fit
`Q5_K_XL-96k`	5-bit	20.2 GB	96K	✅	✅	32 GB	5-bit XL, more context
`latest`	4-bit	17.9 GB	144K	—	✅	24 GB	Default, MTP, no vision
`IQ2_XXS`	2-bit	9.0 GB	32K	—	—	12 GB	Smallest, most compatible
`IQ2_XXS-12gb`	2-bit	9.0 GB	48K	—	—	12 GB	2-bit XXS, optimized for 12 GB VRAM
`IQ2_M-12gb`	2-bit	10.3 GB	16K	—	—	12 GB	2-bit KM, optimized for 12 GB VRAM

Context windows are set explicitly per tag (see table). All tags use the Qwen im_start/im_end stop tokens. The -128k/-192k/-96k suffixes set the context window explicitly. The -12gb/-16gb suffixes are context-optimized for those VRAM budgets.

Vision tags include the mmproj-F16 projector (multimodal, FP16). MTP tags enable speculative decoding via the model’s built-in MTP drafter (draft_num_predict 4); small builds disable it (draft_num_predict 0) to save VRAM. The latest tag is the MTP build without vision.

All tags use lowered sampling temperature for more deterministic output:

PARAMETER temperature 0.6
PARAMETER top_k 20
PARAMETER top_p 0.95
PARAMETER repeat_penalty 1
PARAMETER num_predict -1
PARAMETER min_p 0
PARAMETER presence_penalty 0

temperature 0.6 lowers sampling temperature for more focused/deterministic outputs (the stock qwen3.8 build uses temperature 1).
top_k 20, top_p 0.95, repeat_penalty 1, min_p 0, presence_penalty 0 match the stock Ollama qwen3.8 build exactly.
num_predict -1 = unlimited output (no cap), same as the official model which sets no cap.
num_ctx is set per tag to fit the VRAM budget (see table).
draft_num_predict is 4 on MTP-enabled builds (speculative decoding) and 0 on the small/12 GB builds.

Usage

ollama run smtek/Qwen3.8-27B:Q4_K_XL

Or via the API:

curl http://localhost:11434/api/generate -d '{
  "model": "smtek/Qwen3.8-27B:Q4_K_XL",
  "prompt": "Hello"
}'

Recommended Optimization

By default, Ollama allocates the KV-Cache in f16, which will exceed VRAM over long contexts. To run stably up to the full 256K window, set these environment variables in your system (/etc/systemd/system/ollama.service.d/override.conf on Linux):

[Service]
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

Flash Attention: Dramatically reduces context memory footprint.
q8_0 Cache: Compresses context tensors without the severe logic degradation caused by 4-bit (q4_0) cache quantization.

After editing, reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Important: custom Ollama build

The map-k4v, MTP3 + map-k4v, and DFlash2 + map-k4v tags use speculative-decoding fields that stock Ollama does not forward to llama.cpp. Stock Ollama still works for each clean baseline tag, but it does not activate these custom draft routes.

Build and run the custom runtime from the Ollama source. Build notes, local patches, and updates are maintained on Samuel Ishida’s GitHub.

cmake -B build .
cmake --build build --parallel 8
./ollama serve

For this tag set, use the custom launcher from the patched checkout:

scripts/ollama-ngram.sh build --full
OLLAMA_NGRAM_GPU_MASK=1 scripts/ollama-ngram.sh start-custom

Source

Hugging Face: unsloth/Qwen3.8-27B-GGUF
License: Apache-2.0

Quants from Q2 up to Q5 from Unsloth K_M and UD_K_XL and builds for 12, 16 and 24 GB

Details