Knot AI - Commit Message Model

A task-specific model that reads a code diff and writes the commit message. Fine-tuned from Qwen/Qwen2.5-Coder-7B-Instruct. Runs 100% locally - no API, no data leaves your machine. It powers hands-free auto-commit in Knot, a peer-to-peer version control system by PointBreakLab.

• Developed by: PointBreakLab
• Base model: Qwen2.5-Coder-7B-Instruct
• License: Apache-2.0 (derivative of Qwen2.5-Coder-7B, Apache-2.0)
• Task: git diff → Conventional-Commit message
• Formats: LoRA adapter (PEFT safetensors) + LoRA GGUF (llama.cpp / Ollama)

What it is (and isn’t)

Knot AI does one thing well: turn a diff into a clear, correctly-typed Conventional-Commit message. It is not a chat assistant or a general coding model - feed it a diff, get a message. That focus is the point: a small, private, owned specialist you run on your own hardware.

How it works

Inference: git diff (+ light file context) → prompt → Qwen2.5-Coder-7B + Knot AI LoRA → commit message

Training: permissively-licensed public repos → mine (diff→message pairs) → filter + rebalance → QLoRA SFT → LoRA adapter → GGUF

Training

• Method: QLoRA (4-bit), completion-only supervised fine-tuning (loss on the commit message only), 2 epochs, ~6h on a single NVIDIA RTX 3060 (12 GB).
• Data: 3,227 examples mined from real histories of permissively-licensed OSS - JavaScript/TypeScript (vite, vitest, nx, pnpm, rollup, angular, changesets, husky, vitepress), Python (poetry, pydantic), Rust (starship, tauri), and Go (goreleaser). Type-balanced (fix 27% / feat 19% / refactor 11% / chore 8% / …).
• No user data. No private code. Only public, permissively-licensed commit history.

Benchmark

Two held-out gold sets of 60 hand-reviewed commits each (the model never saw them), scored through the shipping GGUF path. Three models compared under one identical harness:

• Base Qwen - Qwen2.5-Coder-7B, same prompt, no fine-tuning
• Knot AI v3 - the prior release (trained on JS/TS only)
• Knot AI (v4) - this model (added Python / Rust / Go)

Full metrics (60 JS/TS gold)

Metric	Base Qwen	Knot AI v3	Knot AI (v4)
Type accuracy	0.0%	40.0%	55.0%
Title word-overlap	0.4%	18.3%	30.3%
Exact title	0.0%	0.0%	6.7%
Exact full message	0.0%	0.0%	6.7%
Scope accuracy	0.0%	70.0%	75.0%
Unparseable replies / 60	59	4	0

The base model is not “bad at commits” - zero-shot with the same prompt it returns unstructured prose 59 times out of 60, so it can’t be wired into an automated pipeline at all. Fine-tuning’s first job is to make the reply reliably parseable (59 → 4 → 0 errors); its second is to pick the right type.

By commit type - type accuracy (60 JS/TS gold)

Type	n	Base	v3	Knot AI (v4)
chore	6	0%	83%	100%
ci	6	0%	67%	100%
feat	6	0%	33%	83%
fix	6	0%	67%	83%
test	5	0%	80%	80%
docs	6	0%	83%	67%
perf	6	0%	0%	50%
build	5	0%	0%	0%
refactor	6	0%	0%	0%
style	6	0%	0%	0%
revert	2	0%	0%	0%

Honest read: strong on the high-signal types (chore, ci, feat, fix, test); still developing on the inherently ambiguous ones - refactor vs feat, build vs chore, style, revert - which is exactly what v5 targets.

Generalization across languages - type accuracy (60 Rust / Go / Python gold)

Language	n	Base	v3	Knot AI (v4)
Rust	20	0%	10%	55%
Go	20	0%	55%	45%
Python	20	0%	55%	60%

The added multilingual training pays off: Knot AI holds ~53% type accuracy overall on this separate Rust/Go/Python set - close to its 55% JS/TS score - and lifts Rust from 10% → 55%, where the base model still scores 0% (unparseable).

Note: these are single-reference metrics - they compare to one human label, though a commit has many valid messages and a change can honestly be fix or refactor. So the scores understate real usefulness; sample outputs below are representative.

Sample outputs

Change	Knot AI
Added a divide-by-zero guard	fix: prevent division by zero
Added a new function	feat: add divide function

Usage

Ollama

ollama run pointbreaklab/knot-ai "$(git diff --staged)"

llama.cpp (base + this LoRA)

llama-server --model qwen2.5-coder-7b-instruct-q4_k_m.gguf \
             --lora knot-ai-lora-f16.gguf

transformers + PEFT

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen2.5-Coder-7B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "pointbreaklab/knot-ai")

Limitations

• Pattern completion over ~3k examples - it will miss on unusual diffs, and ~55% type accuracy means it is wrong sometimes (see the weaker types above).
• Binary-only changes (images, PDFs) have no textual diff to read.
• English commit conventions; not a general assistant.

Roadmap

This is v1 (built on Knot AI’s v4 adapter). A v5 is planned - a larger, higher-quality corpus for better accuracy and closer matching, with a focus on the ambiguous types (refactor / build / style / revert). Knot AI improves in the open.

Citation / attribution

Built on Qwen2.5-Coder-7B-Instruct (Apache-2.0) by Alibaba Qwen. Knot AI is a LoRA fine-tune by PointBreakLab, trained only on permissively-licensed public commit history.