363 Downloads Updated 5 days ago
ollama run tsarihan/qwen3.8:27b-jc-mlx
Updated 5 days ago
5 days ago
11bedefe526d · 21GB
A vision-capable, tool-using, thinking 27B dense model bundled for one-command local use: Qwen3.8-27B, community “JC” fine-tune (uncensored ablation), llama.cpp Q4_K_M GGUF, with the mmproj vision projector included. Runs on Apple Silicon via Metal, on NVIDIA/AMD GPU, and on CPU — no extra files, no glue: the CLIP projector ships with the weights.
ollama pull tsarihan/qwen3.8:27b-jc
ollama run tsarihan/qwen3.8:27b-jc
| Base model | Qwen3.8-27B (dense, 27.3B params, embedding dim 5120) |
| Fine-tune | community “JC” uncensored ablation (see provenance below) |
| Weights | llama.cpp Q4_K_M k-quant, ~4.9 bits/weight, 16.8 GB |
| Vision projector | CLIP mmproj, 460.7M params, kept at F16 (unquantized, 928 MB) |
| Context window | 262,144 native (extendable with RoPE/YaRN scaling — validated to 1,048,576 on DGX Spark via llama.cpp) |
| Capabilities | completion · vision · tools · thinking (levels: false / low / medium / xhigh, default medium) |
| Total pull size | ~17.5 GB |
Steady-state numbers, this exact publish, temperature 0, thinking disabled, 8192 ctx:
| Metric | Value |
|---|---|
| Decode speed | 20.6 tok/s (384-token generation, 100% GPU) |
| Prefill, short prompt (~50 tok) | 413 tok/s, TTFT 0.12 s |
| Prefill, long prompt (4,098 tok) | 117 tok/s, TTFT 35 s |
| Model load (warm) | ~8–13 s |
| Memory footprint | 16 GB unified memory, 100% GPU at 8K ctx |
Same-Mac comparison, identical prompts and settings, this exact publish family: the MLX NVFP4 build of the same model (27b-jc-mlx) decodes at 15.7 tok/s and prefills 4,098 tok at 116 tok/s (TTFT ~36 s) — the Q4_K_M GGUF is the faster lane on Apple Silicon (20.6 vs 15.7 tok/s decode; 16 GB vs 21 GB footprint). Dense-27B decode here is memory-bandwidth bound (~17 GB over ~400 GB/s ⇒ low-20s tok/s is the practical ceiling). Like all dense models, prefill cost grows with prompt length and long-context sessions get painful: at ~4K this card’s 35 s TTFT is fine for interactive use; at 900K+ expect very long waits (dense prefill does not collapse as gracefully as expert-routing models).
Notes: think: false/lower thinking levels cut total generated tokens and wall time but barely change raw decode speed. Larger num_ctx raises KV memory; 64 GB handles far more than 8K — size it to your task.
tsarihan/qwen3.8:27b-jc-mlxSame weights, same vision tower, packaged for ollama’s native MLX runner (Apple Silicon only):
ollama pull tsarihan/qwen3.8:27b-jc-mlx
| Encoding | NVFP4 (MLX), group-16 E4M3 scales, ~4.7 bits/weight |
| Size | 21 GB |
| Provenance of the encoding | converted from the fine-tune’s own Q8_0 GGUF (near-lossless, ~0.04% weight error) → F16 → NVFP4 — a single effective quantization step |
| M1 Max decode | 15.7 tok/s (vs 20.6 for the Q4_K_M GGUF lane) |
| M1 Max prefill | 83 tok/s short prompt (TTFT 0.34 s) · 116 tok/s at 4,098 tok (TTFT 36 s) |
| Capabilities | completion · vision · tools — identical to the GGUF tag |
Caveats, honestly: the MLX build passes the same spot checks as the GGUF tag (17×23, rate problems, factual recall). If you are on Apple Silicon and want the best quality this repo offers, use the Q4_K_M GGUF tag — it is still the faster lane (20.6 vs 15.7 tok/s decode) because llama.cpp’s Metal dequant kernels are better tuned than MLX’s on this GPU generation. The MLX tag exists for MLX-native workflows (mlx-lm tooling, MLX runners) where the GGUF lane is not an option. On M5/M6-class machines with hardware INT8/FP8 matrix units the calculus changes; on M1/M2/M3/M4 every sub-16-bit format (int4, int8, NVFP4) unpacks to native FP16 compute, so decode speed tracks bytes-per-token, and NVFP4 is the smallest practical footprint.
| Encoding | Bits | Approx. size (27B) | Calibration needed? | Where it runs | Trade-off |
|---|---|---|---|---|---|
| BF16 / FP16 | 16 | ~55 GB | no | anything | Reference quality; overkill for local disk/RAM |
| FP8 (E4M3/E5M2) | 8 | ~27 GB | usually yes | Server GPUs with FP8 tensor cores (Hopper+, vLLM/TRT) | Near-lossless with calibration; halves memory |
| MXFP4 | 4 | ~14 GB | no (power-of-2 block scales) | Blackwell-class tensor cores, AMD MI-series | Used in publisher checkpoints (e.g. GLM-5.3-Flash); needs HW support |
| NVFP4 | ~4 | ~14 GB | yes (calibration dataset; vendor builds publish benchmarks vs FP8) | NVIDIA Blackwell + TensorRT/vLLM stacks | NVIDIA’s FP4 recipe with per-16 E4M3 scales; higher accuracy than MXFP4 on NVIDIA stacks, but hardware- and stack-specific |
| GGUF k-quants (Q8_0 / Q6_K / Q4_K_M / IQ4_XS) | 1.8–8.5 | 16–29 GB | no (classic block k-quants, not calibration-based) | Everywhere: Apple Metal, CUDA, Vulkan, CPU | Q4_K_M = the mainstream speed/quality/size tradeoff; Q8_0 near-lossless at 2× size |
| MLX NVFP4 | ~4.7 | ~21 GB | yes (per-16 E4M3 scales) | Apple Silicon only, via MLX | NVIDIA’s FP4 recipe adapted to MLX; here measured slower than Q4_K_M GGUF on M1 Max (15.7 vs 20.6 tok/s) |
Why this publish is Q4_K_M: it runs on the widest hardware (including Apple Metal, which NVFP4/MXFP4/FP8 can’t touch), needs no calibration step, weighs 4.9 bpw, and measured as fast or faster than the alternatives on the reference machine. If you need near-lossless on a workstation, Q8_0 of the same fine-tune is the step up.
Thinking levels (default medium):
>>> /set think false # or low / medium / xhigh inside the REPL
API — disable thinking for cheap/fast calls (e.g. summarization, compaction):
{ "model": "tsarihan/qwen3.8:27b-jc", "stream": false, "think": false,
"messages": [{ "role": "user", "content": "…"}],
"options": { "num_ctx": 8192, "temperature": 0 } }
Vision works out of the box — attach base64 images to a message (images field) or drop files on the prompt in the CLI. Tool calling uses the parser’s native dialect (qwen3.5 parser); keep num_ctx generous when tools return long payloads. Raise context with "num_ctx" up to 262,144 (mind KV memory), or beyond that with a RoPE/YaRN-scaled llama.cpp deployment.