base_model: Qwen/Qwen3-4B-Instruct-2507 library_name: peft pipeline_tag: text-generation license: apache-2.0 language: - en tags: - code - compression - coding-agent - lora - sft - claude-code - cursor model-index: - name: paritok-4b-v1 results: - task: type: text-compression dataset: name: SWE-bench Verified type: swe-bench-verified metrics: - name: Quality retained type: quality-retained value: 86.5 - name: Compression rate type: compression-rate

value: 25.7

Paritok-4B-v1

The first open-source compression model trained specifically for coding agents.
Cut Claude / GPT input token bills by up to 95% while retaining 86.5% of full-context solve quality on SWE-bench Verified.

🔥 TL;DR

~74%

Fewer tokens
_{Typical CR = 25.7%
Up to 95% on long sessions}

86.5%

Solve quality retained
_{SWE-bench Verified
Matches gpt-4.1-mini}

4B

Self-hostable LoRA
_{Single 24GB GPU
Apache 2.0, zero fee}

✨ Why Paritok?

🎨 Code-native. Trained end-to-end on 45K real coding-agent trajectories (file_read, bash_command, log_output…). Preserves function names, imports, paths, and error strings while compressing.
🚀 ~74% fewer tokens on typical workloads — up to 95% on heavy long-session traffic.
💰 Up to 95% off your input token bill on Claude Sonnet / GPT-4 (74% typical). Long-session teams save thousands per month.
🎯 86.5% of full-context solve quality retained on SWE-bench Verified — matching gpt-4.1-mini as compressor at less than half the token spend.
🪶 Small, fast, self-hostable. 4B LoRA adapter, bf16, single 24GB GPU. No SaaS, no lock-in, no per-token fee.
🔓 Fully open. Apache 2.0 weights, reproducible pipeline, real end-to-end benchmarks.

🚀 Quick Start

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
BASE_MODEL = "Qwen/Qwen3-4B-Instruct-2507"
ADAPTER    = "paritok/paritok-4b-v1"
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
base = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()
# Wrap your code as [SEG id=1 kind=file_read] ... [/SEG]
user_msg = "[SEG id=1 kind=file_read]\n<your code here>\n[/SEG]"
prompt = tokenizer.apply_chat_template(
    [{"role": "system", "content": "<full system prompt — see GitHub>"},
     {"role": "user",   "content": user_msg}],
    tokenize=False, add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

📎 Full runnable example + system prompt + reproduction pipeline: GitHub → Paritok-official/paritok-4b-v1

📊 SWE-bench Verified — Head-to-head

Context source	Quality retained ¹	Compression rate
Uncompressed baseline	100.0%	100.0%
gpt-4.1-mini (compressor)	85.6%	50.2%
gpt-5 (compressor)	93.6%	61.9%
Paritok-4B-v1 ⭐	86.5%	25.7%

_{¹ Quality retained = compressor solve rate ÷ uncompressed baseline solve rate. Higher is better.}

Paritok compresses ~2× harder than gpt-4.1-mini on the same task while keeping the same solve rate — the only open-source entry trained end-to-end on real coding-agent trajectories.

💰 Cost Impact (Claude Sonnet, $3/M input tokens)

Turn input size	Uncompressed input	With Paritok
Short (8K)	$0.024	$0.006
Typical (15K)	$0.045	$0.012
Long session (30K)	$0.090	$0.023

10-person team, 3-month project → save ~$10K on API bills.

📋 Model details (click to expand)

| Property | Value | | ---------------------- | --------------------------------------------------------------------------------- | | **Base model** | [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) | | **Adapter type** | LoRA, r=32, α=64, dropout=0.0 | | **Target modules** | `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` | | **Training steps** | 2000 (selected from a 5-checkpoint sweep on OOD holdout) | | **Training precision** | bf16 | | **Effective batch** | 32 (per_device=2 × grad_accum=16) | | **Learning rate** | 1e-5, linear decay, 10% warmup | | **Optimizer** | AdamW (8-bit) | | **Max seq length** | 16 384 | | **Dataset size** | 45 000 samples (`file_read`, `bash_command`, `log_output`, ...) | | **Teacher** | gpt-4.1-mini |

⚠️ Limitations (click to expand)

1. **~6pp accuracy trade for 74% compression.** Not free — buying context length with a small solve-rate drop. Bring your own uncompressed-fallback path for safety-critical turns. 2. **Rare identifier loss (~40% preserved on hard OOD segments).** Add a post-compression check that your target identifier is still in the compressed output before sending upstream. 3. **English source code, Python-heavy training distribution.** Non-English comments/strings and other-language codebases have not been benchmarked. **Recommended safeguards** for production: format check, target-identifier presence check, and a session-level toggle to fall back to raw context on failure.

🗺️ Roadmap (click to expand)

- **Paritok-4B-v2** — Next-generation training pipeline pushing compression to under 20% while closing the gap to uncompressed solve rate. - **Frontier-scale backbones** (10B+ parameters) for multi-day sessions with 100K+ token histories. - **Multi-language expansion** — TypeScript, Rust, Go, Java, C++, Kotlin. - **Native integrations** — `mcp add paritok` plugin for Claude Code and Cursor, plus a hosted inference endpoint.

📖 Citation

@misc{paritok2026,
  author       = {Paritok Team},
  title        = {Paritok: Fine-tuned Compression for AI Coding-Agent Context},
  year         = {2026},
  publisher    = {GitHub},
  howpublished = {\url{https://github.com/Paritok-official/paritok-4b-v1}},
}

📄 License

Apache 2.0 — see LICENSE. Base model (Qwen3-4B-Instruct-2507) is under its own license; please review before commercial deployment.

GitHub · paritok.com · Contact

Models

Readme