tobestyledintro/ Ternary-Bonsai-2-27B:vision

1,674 yesterday

Mirror of Prism ML's Bonsai 2 27B ternary GGUF packs (PQ2_0 and PTQ1_0): a 27B hybrid-attention reasoning model in 6.70 GiB or 5.53 GiB, 262K context, vision and tool calling, Apache-2.0. Requires Prism ML's llama.cpp fork — stock Ollama cannot load

vision tools thinking 27b 
ollama run tobestyledintro/Ternary-Bonsai-2-27B:vision

Details

yesterday

0b2cbca12243 · 7.8GB

model
qwen35
·
26.9B
·
(!unknown_file_type 141!)
projector
clip
·
461M
·
Q8_0
license
Apache-2.0. Copyright Prism ML. Base model: Qwen/Qwen3.8-27B (Apache-2.0). Original weights and GGUF
params
{ "temperature": 1, "top_k": 20, "top_p": 0.95 }

Readme

Ternary-Bonsai 2 27B — GGUF

A 27B-class hybrid-attention reasoning model with ternary weights, for Prism ML’s llama.cpp fork (CUDA / Metal / CPU).

6.70 GiB for the full 27B in PQ2_0, 5.53 GiB in PTQ1_0 | 262K context 135.41 tok/s decode and 3802.74 tok/s prefill on an RTX 5090 (measured below)

This is a mirror. The weights are Prism ML’s Apache-2.0 packs from prism-ml/Ternary-Bonsai-2-27B-gguf, republished unchanged: each published blob’s sha256 is the upstream file’s sha256. Not affiliated with or endorsed by Prism ML. The model, the kernels and the upstream documentation are theirs.

⚠️ These files do not run in Ollama

The ternary kernels live in the PrismML-Eng/llama.cpp fork. Stock Ollama cannot run these packs. Verified against Ollama 0.34.2:

$ ollama create Ternary-Bonsai-2-27B:27b-pq2_0 -f Modelfile.pq2_0
Error: unsupported tensor "output.weight" size overflows

Ollama’s runner knows the architecture (qwen35) but has no pq2_0 or ptq1_0 anywhere in its libraries, so the GGUF parser rejects the file before anything is loaded. It also has no Hadamard activation runtime, and this family’s weights are stored in a rotated basis declared in prism.hadamard.* metadata — which is why the F16 pack of this family is a trap rather than a fallback: its tensors are ordinary F16/F32, so it parses and loads silently, and then decodes incoherent text. A runtime either applies the matching rotation or it produces garbage.

To actually run these files, use the fork: prebuilt binaries from PrismML-Eng/llama.cpp releases, and Bonsai-demo for setup, serving and tooling.

Tags

All five files of the upstream pack are mirrored. Every tag shares its blobs, so pulling several tags costs one download of the weights. Every tag carries the Q8_0 vision tower as a separate projector layer, loaded only when an image arrives; 27b-mmproj-bf16 carries the BF16 projector instead.

Tag Contents Size Notes
latest · 27b · 27b-pq2_0 · pq2_0 · vision · 27b-mmproj-q8_0 PQ2_0 + mmproj Q8_0 7.84 GB faster prefill, same decode as PTQ1_0 (measured below)
27b-ptq1_0 · ptq1_0 PTQ1_0 + mmproj Q8_0 6.58 GB smallest; pick it when memory is tightest
27b-f16 · f16 F16 + mmproj Q8_0 54.44 GB unquantized reference; see the warning above
27b-mmproj-bf16 PQ2_0 + mmproj BF16 8.14 GB reference projector

latest is PQ2_0; vision is an alias of 27b-mmproj-q8_0. Declared capabilities on every tag: completion, tools, vision, thinking, with renderer/parser pinned to qwen3.8 (the profile matching this model’s chat template). These declarations are metadata — nothing here makes the ternary packs loadable by stock Ollama.

Measured speed

Measured on the machine that published this mirror, with llama-bench from the fork (build d8f26eec7 / b10683), NVIDIA GeForce RTX 5090 32 GB, driver 616.56, every layer offloaded and flash attention on:

llama-bench -m <pack>.gguf -ngl 99 -fa on -p 512 -n 128 -r 3
Pack Weights Prefill (pp512) Decode (tg128)
PQ2_0 6.70 GiB 3802.74 ± 363.32 tok/s 135.41 ± 5.47 tok/s
PTQ1_0 5.53 GiB 1876.31 ± 50.23 tok/s 131.41 ± 0.76 tok/s

Mean ± standard deviation over 3 runs, batch size 1, no draft model, no vision tower.

What the numbers say: PQ2_0 prefills 2.03x faster than PTQ1_0 (3803 vs 1876 tok/s), while decode is a tie — 135.41 vs 131.41 tok/s is a 3% gap against error bars of 5.47 and 0.76. So PQ2_0 is the default here, and PTQ1_0 is for when the last ~1.2 GiB matters.

Serving context for scale: with llama-server on the same card, a 65k-token context and -ngl 99 -fa on, decode measured 136.5 tok/s on a single streaming request. Context length and concurrency change the decode number; the table above is the reproducible configuration.

Not measured here: the F16 pack is 50.1 GiB, so it cannot be fully offloaded to a 32 GB card and no full-GPU number exists for it on this machine.

Model overview

Read from the GGUF metadata of these packs:

Item Value
Architecture qwen35 — hybrid attention (full attention every 4th layer) + state-space layers
Parameters 26.90 B
Blocks 64
Embedding width 5120
Attention 24 query heads, 4 KV heads, key/value length 256
Context 262144 tokens (RoPE base 1e7)
Weight format ternary, group 128 — PQ2_0 labeled 2.13 bpw, PTQ1_0 labeled 1.75 bpw
Vision tower CLIP, 460.73 M parameters, embedding 1152 → 5120 (separate mmproj pack)
Weights size 6.70 GiB (PQ2_0), 5.53 GiB (PTQ1_0), 50.1 GiB (F16)
Sampling temperature 1.0, top-p 0.95, top-k 20 (carried in the GGUF general.sampling.*)
License Apache 2.0

Base model: Qwen/Qwen3.8-27B. Upstream publishes the quality evaluations, the packing internals and the per-platform throughput; those are Prism ML’s measurements, not repeated here.

Running it

# binaries for the fork, then point a server at a pack
llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
    -ngl 99 -fa on -c 65536 --host 127.0.0.1 --port 8080 \
    --temp 1.0 --top-p 0.95 --top-k 20

curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Explain quantum computing in simple terms."}]}'

Add --mmproj Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf for image input. -ngl 99 offloads every layer, 0 is CPU-only; -c goes up to 262144.

License and attribution

Apache 2.0. Copyright Prism ML; base model Qwen/Qwen3.8-27B.

@misc{bonsai2_27b,
    title   = {Bonsai 2 27B: A 27B Ternary Reasoning Model},
    author  = {Prism ML},
    year    = {2026},
    month   = {September},
    url     = {https://prismml.com}
}