# Gemma4 College of Experts — Web Specialist
**Base model:** [google/gemma-4-26b-it](https://huggingface.co/google/gemma-4-26b-it)
**Architecture:** MoE — 26B total / ≈4B active parameters (1 shared expert + 8 routed from a pool of 128 per MoE layer, 30 MoE layers)
**Method:** Activation-directed expert surgery — 128 → 64 experts per layer (50% reduction)
**Quantization:** Q4_K_M (≈9.7 GB on disk)
**HF:** `JThomas-CoE/coe-gemma4-web-hc-14b-a4b-q4` | **Ollama:** `coe-gemma4-web-14b-a4b:q4`
**Hub:** [JThomas-CoE on HuggingFace](https://huggingface.co/JThomas-CoE)
> **HC (Hand-Curated)** — The activation profiling corpus used to build this model's expert mask was assembled by hand, selecting high-quality domain-representative text from textbooks, problem sets, and technical references. This contrasts with the MMLU-Pro variants (`-mmlu_pro-` in the repo name) which were profiled on stratified subsets of the TIGER-Lab/MMLU-Pro benchmark dataset.
---
## ⚠️ Beta Release — Safety Disclaimer
**These models are beta releases and should be treated as research artifacts, not production-ready systems.**
Expert surgery selects and retains domain-relevant experts based on activation patterns observed during profiling. The pruning pipeline is designed solely to create a coherent domain specialist — it has no mechanism to identify which experts contribute to model alignment, ethical reasoning, or safety guardrails. As a result, experts responsible for enforcing those behaviours may have been inadvertently removed during the surgery process.
**Appropriate use of any model in the College of Experts family is the sole responsibility of the end user.** The authors make no representation that these models retain the safety properties of the parent `google/gemma-4-26b-it` model, and users should not rely on them as a substitute for models that have undergone safety evaluation.
---
## ⚠️ Critical Usage Note — Think-Off Mode
**All models in this series must be used in thinking-off mode.**
If you are using the Ollama API, pass `"think": false` in your request body. If you are accessing the model via a raw API (llama.cpp server, OpenAI-compatible endpoint, etc.) you **must inject a closed thinking block** at the start of the assistant turn:
```python
messages = [
{"role": "system", "content": "Your system prompt here."},
{"role": "user", "content": "Your question here."},
{"role": "assistant", "content": "<think></think>\n"}, # <-- required prefill
]
```
**Why this is required:** Expert surgery retains 50% of the expert pool per layer, selecting experts that are maximally active on domain content and chain-of-thought reasoning. A side effect is that the loop-suppression experts — which activate on metacognitive closure signals near the end of a `<think>` block — do not have a concentrated domain-specific activation signature and are disproportionately pruned. In think-on mode, this causes the model to enter a reasoning loop that exhausts the token budget without producing a final answer. In extreme cases, the loop rate is 60–70% on hard questions.
The `<think></think>` prefill works by consuming the opening `<think>` token before generation starts, so the model sees its thinking as already complete and proceeds directly to answering. This is the mechanism used in all benchmarks reported here.
**What think-off mode does not disable:** Gemma4's chain-of-thought training is deeply ingrained. Even with the think block closed, the model produces brief inline reasoning interleaved with its answer — shorter and more linear than a full scratchpad, but present. All benchmark figures in this README are measured in this constrained-implicit-CoT mode, which is more conservative than the full explicit CoT used by leaderboard entries.
### Ollama Modelfile Template
```
FROM <model_path_or_ollama_tag>
PARAMETER temperature 0.6
PARAMETER repeat_penalty 1.05
PARAMETER num_ctx 8192
PARAMETER num_predict 16384
PARAMETER think false
SYSTEM """
You are an expert front-end and full-stack web developer.
Write clean, semantic, accessible HTML/CSS and idiomatic JavaScript/TypeScript.
When building components, use the framework conventions of the target stack (React, Vue, Svelte, etc.).
Prefer standards-compliant solutions. Return your complete answer, then stop with no further output.
"""
```
Temperature 0.6 is strongly recommended. Higher temperatures (≥ 0.8) materially increase loop rates in think-off mode.
---
## What Are These Models?
These models are produced by **activation-directed expert surgery** applied to the Gemma4 26B-A4B instruction-tuned model. The surgery does not change any weight values — it prunes the FFN weight tensors for experts that are not part of the domain-specialist mask, then saves the result as a smaller GGUF. The result is a model that loads approximately 7–8 GB less VRAM for 4 bit quantization than the parent while maintaining the same token throughput (active parameters per forward pass are unchanged: 9 experts fire per token regardless of pool size).
### Memory Efficiency
| Configuration | VRAM (16k ctx) | VRAM (64k ctx) | Active params |
|---|---|---|---|
| Gemma4-26B parent (Q4_K_M) | 19.4 GB | 20.5 GB | ≈4B |
| Specialist K=64 (Q4_K_M) | **12.3 GB** | **13.4 GB** | ≈4B |
| Q4 savings vs Q4 parent | **7.1 GB (37%)** | **7.1 GB (35%)** | unchanged |
All figures directly measured in Ollama.
Throughput (tokens/second) is identical between the specialist and the parent at the same quantization because the number of expert weight tensors that participate in each forward pass is the same. The saving is purely in VRAM residency — half the expert weight tensors simply do not need to be loaded.
---
## Benchmark — Web Visual Generation Suite
**Evaluation harness:** `run_web_visual.py` — generates complete single-file implementations from prompts and scores them via headless visual inspection and functional test execution.
**Mode:** think_off (`<think></think>` prefill), temperature 0.6.
**Model under test:** Web HC specialist, cap3-skip3 sibling variant (`gemma4-web-cap3-skip3`).
> **Note:** The benchmarked variant (`cap3-skip3`) uses a slightly different mask construction heuristic than the released model (`cap3-T06`). Both are built from the same HC corpus using the same three-pass methodology with cap=3 (maximum 3 CoT-arbitrage swaps per layer); the difference is in the post-hoc skip-connection filtering applied during mask construction. Benchmark results should be interpreted as representative of the cap3 family — the released `cap3-T06` model is expected to perform in a similar range.
### Results by Category
| Category | Passed | Total | Pass Rate |
|---|---|---|---|
| CSS | 7 | 7 | 100% |
| React | 6 | 6 | 100% |
| Svelte | 4 | 4 | 100% |
| Vue | 5 | 5 | 100% |
| HTML | 6 | 8 | 75% |
| JavaScript (exec) | 6 | 10 | 60% |
| **Overall** | **34** | **40** | **85.0%** |
CSS, React, Svelte, and Vue all reach 100%. HTML and JavaScript execution problems are harder — HTML involves semantic correctness and accessibility beyond pure syntactic output, while the execution category requires programs that both produce visible DOM output and pass programmatic assertions.
---
## Activation Profiling — How the Masks Are Built
### Step 1 — Corpus Assembly
The web corpus is drawn from a curated problem/solution dataset (`01_coding_web.jsonl`) and filtered into five weighted sub-domains:
- **web_js_evol** (w=0.25, 238 records) — Idiomatic JavaScript: modern ES6+ patterns, event-driven architecture, DOM manipulation, and evolving JS ecosystem idioms
- **web_ts** (w=0.30, 159 records) — TypeScript: type annotations, generics, interface design, strict-mode patterns, and TS-specific tooling
- **web_html** (w=0.20, 114 records) — HTML structure and semantics: HTML5 elements, forms, accessibility, and document layout
- **web_js_algo** (w=0.15, 161 records) — JavaScript algorithms: data structure manipulation, problem-solving patterns, and performance-sensitive JS implementations
- **web_css** (w=0.10, 38 records) — CSS: selectors, flexbox, grid, responsive design, and animation
Each record contributes both the problem prompt and the canonical solution, so the activation sweep captures experts used during both problem comprehension and solution generation. React was excluded from the profiling corpus (only 10 records — insufficient for reliable activation signal), though the model bench shows 100% on React output generation tasks.
Corpus size: approximately 28,000 tokens total across all five sub-domains (combined into `web_all` for profiling). Profiling was run on the full parent model with router hooks capturing per-token expert selections across all 30 MoE layers.
**Corpus size considerations.** Choosing how much material to include for activation profiling involves two competing pressures. On one side, a corpus that is too small or too narrow may fail to activate the full set of experts that are genuinely relevant to the domain: rare but important concepts may appear in too few tokens to accumulate statistically reliable activation counts, leaving their associated experts underweighted or excluded from the mask. At ≈28k tokens the web corpus is the smallest in the CoE specialist family, and the coverage risk is most live here — the corpus was deliberately kept to high-quality problem/solution pairs rather than padded with tangentially related general web content, on the premise that a focused signal from a curated core is preferable to a noisier signal from a larger but less representative set. The React sub-domain was excluded precisely because its 10-record sample was too small to provide reliable activation counts rather than noise. On the other side, if the corpus were expanded with only loosely related front-end content, the resulting activation histogram would begin to resemble a general-purpose programming model rather than a web specialist. A more rigorous data-driven approach — one that measures the dispersion or entropy of the emerging activation cluster after each corpus increment and uses that as a stopping criterion — would provide principled feedback to calibrate corpus size from both directions. This remains an area of future work.
### Step 2 — 3D Histogram Collection
The full parent model is run in forward-pass mode over the corpus with a hook attached to each MoE layer's router. For each token, the router selects the top-8 experts and assigns softmax weights. The hook accumulates a **3D histogram**:
```
histograms[layer, expert, rank] — integer count of selections
weight_sum[layer, expert, rank] — sum of router softmax weights
```
`rank` runs from 0 (highest-weight expert, primary selection) to 7 (lowest-weight, filler). Capturing per-rank information preserves the router's confidence signal: a rank-0 firing (the expert is the router's first choice) is qualitatively different from a rank-7 firing (the expert fills the last slot with low confidence).
### Step 3 — Utility Scoring
Each (layer, expert) pair receives a scalar utility score:
$$\text{util}[l, e] = \sum_{k=0}^{7} \frac{\text{histograms}[l,e,k]}{N_\text{tokens}} \times \frac{\text{weight\_sum}[l,e,k]}{\max(\text{histograms}[l,e,k], 1)}$$
This is the frequency-weighted mean router confidence — how often the expert is selected, weighted by how much the router trusts it when it does fire. Experts that fire rarely but at high confidence (niche specialists) score proportionally higher than experts that fire frequently at marginal confidence (generalists).
**Expert indices are local to each layer** — expert N in layer 0 and expert N in layer 15 are completely independent entities with no shared weights. All selection and ranking operations are performed per-layer.
### Step 4 — Three-Pass Mask Construction (cap3)
**Pass 1 — Domain baseline:** Select the top-64 experts per layer by utility score. This captures the most domain-activated experts in the standard activation sense.
**Pass 2 — Structural whitelist enforcement:** A set of experts identified as those experts that have an average activation rank of less than 2 and a minimum number of activations of 10 or more regardless of overall utility ranking. These are then swapped into the model if they are not included already by utility rank by swapping with existing included experts with low average rank and low utility. Applied to ensure high-confidence structural experts are never displaced by marginally scoring domain specialists.
**Pass 3 — CoT/reasoning arbitrage (cap3):** Experts that activate strongly on domain-agnostic logic/reasoning chain-of-thought traces are swapped into the mask, capped at **3 swaps per layer** (approximately 60–90 total swaps across 30 layers). T=0.6 temperature was used throughout the profiling runs to improve routing signal stability. The released model uses the `-T06` labelled variant, which additionally applies a temperature-stabilized heuristic to filter marginal cap-3 swaps.
### Step 5 — GGUF Surgery
The mask JSON specifies which 64 of 128 experts to retain per layer. The surgery script reads the parent GGUF, zeroes the `ffn_gate`, `ffn_up`, and `ffn_down` weight tensors for all non-mask experts, and writes the result as a new GGUF. Tensor norms are verified post-surgery; any NaN or Inf aborts the process. Attention layers, embedding layers, and the shared expert are untouched.
---
## Citation / Attribution
Research and engineering by JThomas-CoE.
- **Project repository:** [College-of-Experts-AI](https://github.com/JThomas-CoE/College-of-Experts-AI) — code, tooling, and methodology documentation
- **Gemma4 methodology:** [gemma4/README.md](https://github.com/JThomas-CoE/College-of-Experts-AI/blob/main/gemma4/README.md)
- **Whitepaper:** [WHITEPAPER.md](https://github.com/JThomas-CoE/College-of-Experts-AI/blob/main/WHITEPAPER.md) — theoretical basis for expert specialization
- **Preprint:** [Separability of Intelligence](https://github.com/JThomas-CoE/College-of-Experts-AI/blob/main/qwen3.5/PREPRINT.md) — empirical evidence from the prior MoE specialist series
- **Model hub:** [huggingface.co/JThomas-CoE](https://huggingface.co/JThomas-CoE)
Base model: Gemma 4 26B-A4B-IT by Google. All specialist weights are derived from the publicly released checkpoint. Usage is subject to the [Gemma Terms of Use](https://ai.google.dev/gemma/terms).
---
## License
Model weights: subject to the Gemma license (see above).
Code and tooling: PolyForm Noncommercial 1.0.0
Commercial licensing: see [LICENSE-COMMERCIAL.md](https://github.com/JThomas-CoE/College-of-Experts-AI/blob/main/LICENSE-COMMERCIAL.md)