Funding status (20 Aug 2026): this is a self funded research preview and the compute budget is now exhausted. Further work on this model depends on donations: https://ko-fi.com/davida81328

Whittle MoE 27B (A18B): a mixture of experts rescued by its routers

A post hoc mixture of experts carved from Qwen3.8-27B, then taught when to stop talking. As of v2.1 it holds real conversations, writes working code and tables, and the looping disease it shipped with is measured at single digits. This is the model this repo was always trying to build.

The architecture is a partition, not a rebuild. All 64 layers keep the parent’s attention side untouched: the 3 to 1 hybrid of gated deltanet layers and full attention, 16 attention layers in all, hidden size 5120. The surgery is in the feed forward. Each layer’s dense FFN, 17408 wide in the parent, was cut into 64 routed expert slivers of width 192 plus one always on shared expert of width 5120. The arithmetic is exact: 64 x 192 + 5120 = 17408, the original width to the neuron, with zero new FFN weights invented. A small router per layer picks 16 of the 64 slivers per token, so each token runs 8192 of the original 17408 FFN width: 17.8B active parameters of 27B total. Freshly carved, the model was gibberish (4 of 39 on our knowledge battery). Training only the 64 routers with every expert frozen recovered 27 of 39: the knowledge never left the slivers, the routers just had to learn which ones to fire. Multi turn SFT and the complete answer distillation rounds did the rest.

Download the v2.1 Q4_K_M, serve it with llama.cpp, and talk to it. Loop rates on a fixed harness, same prompts and seeds since the first release: 8 percent on long single answers (down from 69), 7 percent across conversations, 22 percent on structured output (down from 75), zero silent or truncated answers, knowledge battery 28 of 39. Every number, including the bad ones, is in the table below with its full history.

It is not the parent. Hard structured asks still fail about one time in five, and arithmetic inherits the family weakness. What it is: a working demonstration that a starved post hoc MoE can be brought back by retraining the smallest possible thing first, with the whole method, dataset, and failure log public. Runs on 24GB of VRAM quantized, splits across two 12GB cards.

How to run it

/home/support/llama.cpp/build/bin/llama-server -m Whittle-MoE-27B-A18B-v2.1-Q4_K_M.gguf --host 0.0.0.0 --port 8090 -ngl 99 -c 8192 -fa on --jinja

Honest limitations, read before using

• Structured output is still weak. 39 percent of SQL, HTML and markdown table generations degenerate. Better than 75 percent, still not good. This is the target of the next round.
• The lineage has a token level wobble this round did not fix: occasional made up words and scuffed numbers, inherited from the MoE compression itself. The knowledge battery moved one fact net, inside the measured noise floor, but one clear regression exists (capital of Egypt) alongside three clear recoveries (longest river, tallest mountain, first Moon walker).
• Evaluated by one person on a small harness. Treat every number as a workshop measurement, not a benchmark.

Training data

Router distillation used logits from Qwen3.8-27B over this project’s heal corpus. Multi turn SFT used ultrachat_200k (MIT), tulu-3-sft-mixture (ODC-BY), CodeFeedback-Filtered-Instruction (Apache-2.0) and the project’s own corpus.

v2: the anti-loop round (22 Aug 2026)

Five earlier training attempts established that repetition and stopping are one behaviour: every fix for looping also taught the model to truncate, because the training data never contained an answer that ends because it is finished. This round trained on exactly that: 245 complete answers written by the parent model with its top 32 token probabilities at every position, spanning 7 token factual answers to 4,000 token structured documents and multi turn conversations (dataset: logic65/whittle-teacher32-complete-answers). A capacity sweep published in WHITTLE_FINDINGS.md ruled out the main alternative first: more active experts change nothing, the fix had to be learned.

v2 files

• Whittle-MoE-27B-A18B-v2-antiloop-Q8_0.gguf and -Q4_K_M.gguf
• v2 bf16 safetensors under v2/
• the training adapter under v2-adapter/, trainer and gate scripts in the findings, training data in the dataset repo

v2.1: the balancing round (22 Aug 2026)

The current release, and the recommended download. One further training epoch on the v2 lineage: the anchor slice was broadened (short answers, medium answers, code, and one seed of each conversation restored), structured examples kept, extras down-weighted to 1.5x, learning rate halved.

Both GGUFs are verified against stock llama.cpp: correct expert routing (top 16 of 64 reported by the runtime), EOS 248046, chat template embedded, multi turn tested.

Measured results, same harness and seeds as the baseline

Support this work

Everything here runs on personal hardware and paid Colab time, funded out of pocket. If a 27B MoE you can run at home is useful to you, or you just want to see where router healing goes next, donations directly buy the GPU hours:

ko-fi.com/davida81328

Distillation data

All teacher data behind this model is public: https://huggingface.co/datasets/logic65/whittle-teacher32-complete-answers (main set: 245 complete answers with top 32 logprobs; r2-structured split: 34 long structured exemplars up to 4.9k tokens). Generators and trainer scripts ship with the data and in WHITTLE_FINDINGS.md.