363 5 days ago

Qwen3.8-27B community "JC" uncensored fine-tune, one-command local: Q4_K_M GGUF + F16 mmproj vision, plus an MLX NVFP4 tag for Apple Silicon. Vision, tools, thinking, 262K context.

vision tools thinking
ollama run tsarihan/qwen3.8:27b-jc

Applications

Claude Code
Claude Code ollama launch claude --model tsarihan/qwen3.8:27b-jc
OpenCode
OpenCode ollama launch opencode --model tsarihan/qwen3.8:27b-jc
Hermes Agent
Hermes Agent ollama launch hermes --model tsarihan/qwen3.8:27b-jc
OpenClaw
OpenClaw ollama launch openclaw --model tsarihan/qwen3.8:27b-jc

Models

View all →

Readme

tsarihan/qwen3.8:27b-jc

A vision-capable, tool-using, thinking 27B dense model bundled for one-command local use: Qwen3.8-27B, community “JC” fine-tune (uncensored ablation), llama.cpp Q4_K_M GGUF, with the mmproj vision projector included. Runs on Apple Silicon via Metal, on NVIDIA/AMD GPU, and on CPU — no extra files, no glue: the CLIP projector ships with the weights.

ollama pull tsarihan/qwen3.8:27b-jc
ollama run tsarihan/qwen3.8:27b-jc

Specs

Base model Qwen3.8-27B (dense, 27.3B params, embedding dim 5120)
Fine-tune community “JC” uncensored ablation (see provenance below)
Weights llama.cpp Q4_K_M k-quant, ~4.9 bits/weight, 16.8 GB
Vision projector CLIP mmproj, 460.7M params, kept at F16 (unquantized, 928 MB)
Context window 262,144 native (extendable with RoPE/YaRN scaling — validated to 1,048,576 on DGX Spark via llama.cpp)
Capabilities completion · vision · tools · thinking (levels: false / low / medium / xhigh, default medium)
Total pull size ~17.5 GB

Performance — measured on M1 Max (64 GB, macOS, llama.cpp/Metal)

Steady-state numbers, this exact publish, temperature 0, thinking disabled, 8192 ctx:

Metric Value
Decode speed 20.6 tok/s (384-token generation, 100% GPU)
Prefill, short prompt (~50 tok) 413 tok/s, TTFT 0.12 s
Prefill, long prompt (4,098 tok) 117 tok/s, TTFT 35 s
Model load (warm) ~8–13 s
Memory footprint 16 GB unified memory, 100% GPU at 8K ctx

Same-Mac comparison, identical prompts and settings, this exact publish family: the MLX NVFP4 build of the same model (27b-jc-mlx) decodes at 15.7 tok/s and prefills 4,098 tok at 116 tok/s (TTFT ~36 s) — the Q4_K_M GGUF is the faster lane on Apple Silicon (20.6 vs 15.7 tok/s decode; 16 GB vs 21 GB footprint). Dense-27B decode here is memory-bandwidth bound (~17 GB over ~400 GB/s ⇒ low-20s tok/s is the practical ceiling). Like all dense models, prefill cost grows with prompt length and long-context sessions get painful: at ~4K this card’s 35 s TTFT is fine for interactive use; at 900K+ expect very long waits (dense prefill does not collapse as gracefully as expert-routing models).

Notes: think: false/lower thinking levels cut total generated tokens and wall time but barely change raw decode speed. Larger num_ctx raises KV memory; 64 GB handles far more than 8K — size it to your task.

The MLX variant — tsarihan/qwen3.8:27b-jc-mlx

Same weights, same vision tower, packaged for ollama’s native MLX runner (Apple Silicon only):

ollama pull tsarihan/qwen3.8:27b-jc-mlx
Encoding NVFP4 (MLX), group-16 E4M3 scales, ~4.7 bits/weight
Size 21 GB
Provenance of the encoding converted from the fine-tune’s own Q8_0 GGUF (near-lossless, ~0.04% weight error) → F16 → NVFP4 — a single effective quantization step
M1 Max decode 15.7 tok/s (vs 20.6 for the Q4_K_M GGUF lane)
M1 Max prefill 83 tok/s short prompt (TTFT 0.34 s) · 116 tok/s at 4,098 tok (TTFT 36 s)
Capabilities completion · vision · tools — identical to the GGUF tag

Caveats, honestly: the MLX build passes the same spot checks as the GGUF tag (17×23, rate problems, factual recall). If you are on Apple Silicon and want the best quality this repo offers, use the Q4_K_M GGUF tag — it is still the faster lane (20.6 vs 15.7 tok/s decode) because llama.cpp’s Metal dequant kernels are better tuned than MLX’s on this GPU generation. The MLX tag exists for MLX-native workflows (mlx-lm tooling, MLX runners) where the GGUF lane is not an option. On M5/M6-class machines with hardware INT8/FP8 matrix units the calculus changes; on M1/M2/M3/M4 every sub-16-bit format (int4, int8, NVFP4) unpacks to native FP16 compute, so decode speed tracks bytes-per-token, and NVFP4 is the smallest practical footprint.

Weight encodings for a 27B-class model — what each is for

Encoding Bits Approx. size (27B) Calibration needed? Where it runs Trade-off
BF16 / FP16 16 ~55 GB no anything Reference quality; overkill for local disk/RAM
FP8 (E4M3/E5M2) 8 ~27 GB usually yes Server GPUs with FP8 tensor cores (Hopper+, vLLM/TRT) Near-lossless with calibration; halves memory
MXFP4 4 ~14 GB no (power-of-2 block scales) Blackwell-class tensor cores, AMD MI-series Used in publisher checkpoints (e.g. GLM-5.3-Flash); needs HW support
NVFP4 ~4 ~14 GB yes (calibration dataset; vendor builds publish benchmarks vs FP8) NVIDIA Blackwell + TensorRT/vLLM stacks NVIDIA’s FP4 recipe with per-16 E4M3 scales; higher accuracy than MXFP4 on NVIDIA stacks, but hardware- and stack-specific
GGUF k-quants (Q8_0 / Q6_K / Q4_K_M / IQ4_XS) 1.8–8.5 16–29 GB no (classic block k-quants, not calibration-based) Everywhere: Apple Metal, CUDA, Vulkan, CPU Q4_K_M = the mainstream speed/quality/size tradeoff; Q8_0 near-lossless at 2× size
MLX NVFP4 ~4.7 ~21 GB yes (per-16 E4M3 scales) Apple Silicon only, via MLX NVIDIA’s FP4 recipe adapted to MLX; here measured slower than Q4_K_M GGUF on M1 Max (15.7 vs 20.6 tok/s)

Why this publish is Q4_K_M: it runs on the widest hardware (including Apple Metal, which NVFP4/MXFP4/FP8 can’t touch), needs no calibration step, weighs 4.9 bpw, and measured as fast or faster than the alternatives on the reference machine. If you need near-lossless on a workstation, Q8_0 of the same fine-tune is the step up.

Usage

Thinking levels (default medium):

>>> /set think false     # or low / medium / xhigh inside the REPL

API — disable thinking for cheap/fast calls (e.g. summarization, compaction):

{ "model": "tsarihan/qwen3.8:27b-jc", "stream": false, "think": false,
  "messages": [{ "role": "user", "content": "…"}],
  "options": { "num_ctx": 8192, "temperature": 0 } }

Vision works out of the box — attach base64 images to a message (images field) or drop files on the prompt in the CLI. Tool calling uses the parser’s native dialect (qwen3.5 parser); keep num_ctx generous when tools return long payloads. Raise context with "num_ctx" up to 262,144 (mind KV memory), or beyond that with a RoPE/YaRN-scaled llama.cpp deployment.

Provenance & limitations

  • Base: Qwen3.8-27B by Qwen/Alibaba — upstream license and usage terms apply (see the base model card).
  • Fine-tune: community “uncensored” ablation. Guardrails are removed by design; it will comply with things aligned models refuse. Use judgment where you deploy it.
  • Quantizer: the “JC” GGUF is a community quantization (llama.cpp Q4_K_M). K-quants are not calibration-verified; quality vs the BF16 base was not formally evaluated — treat quant-related drift as possible until you’ve tested your own use case. The vision projector is left at F16 to keep image fidelity.
  • No affiliation between this packaging and Qwen, NVIDIA, or Meta.
  • Verify important outputs; the uncensored fine-tune and 4-bit quant each remove a layer of friction between the model and a wrong answer.