Abyssal/intent-classifier-general-accuracy

Never refuses: it always returns the single best-matching intent, defined at prompt time with no retraining. Refusal is opt-in via a catch-all intent, which it grabs less eagerly than acv3. For higher accuracy, see acv3.

Recommended usage

Two modes, depending on whether you need out-of-scope detection.

1. Routing (the normal case). No catch-all, no enum — the model always answers with one of your intents. On the audit it stayed inside the candidate list on 100% of in-scope calls:

curl http://localhost:11434/api/chat -d '{
  "model": "intent-classifier-general-accuracy:1.5b",
  "stream": false,
  "options": {"temperature": 0},
  "messages": [{"role": "user", "content":
    "Candidate intents:\nrefund: wants money back\ntracking: where their order is\naccount: login or profile\n\nUser message: where is my package?\n\nAnswer with exactly one intent name from the list above."
  }]
}'
# -> "tracking"

Add a JSON-schema enum in format if you want decoding itself grammar-constrained: "format": {"type": "string", "enum": ["refund", "tracking", "account"]}.

2. Out-of-scope detection (opt-in). Append a catch-all intent and treat that answer as “no match”. Any wording works — other, misc, something_else, not_listed, general_inquiry:

curl http://localhost:11434/api/chat -d '{
  "model": "intent-classifier-general-accuracy:1.5b",
  "stream": false,
  "options": {"temperature": 0},
  "messages": [{"role": "user", "content":
    "Candidate intents:\nrefund: wants money back\ntracking: where their order is\nother: anything that does not match the other intents\n\nUser message: what time do you close on sundays?\n\nAnswer with exactly one intent name from the list above."
  }]
}'
# -> "other"

Remove that other line and the same message returns tracking — the closest available match.

Getting the best results

• Write clear, specific descriptions. The model matches on the description, so this is the single biggest lever. Prefer tracking: track or locate a customer's order or shipment over a terse tracking: order status.
• Keep temperature 0 (the baked-in default) for deterministic, repeatable routing.
• Use the exact prompt shape shown above (Candidate intents: … User message: … Answer with exactly one intent name from the list above.) — it was trained on this format.
• Names are free-form — snake_case, CamelCase, hyphens or plain words all work, including names the model has never seen; it matches by description and echoes your name verbatim.
• Only add a catch-all when you actually need it. Offering an escape hatch costs roughly 4 points of in-scope accuracy, because a model given one occasionally takes it.

Under the hood

Base: LoRA fine-tune of Qwen2.5-1.5B-Instruct (Apache-2.0). Trained on five public intent datasets plus 65 business domains generated for this release — 896 deduplicated intents, each example presenting a different candidate list, ~35% of them renamed to invented names so the model matches on meaning rather than memorising label strings. Runs at temperature 0 and is fully deterministic.

Dropping the refusal option did more than recover the calls its siblings lost to over-rejection. It also unlocked harder training: earlier versions trained on very close intent pairs learned to hedge into none_of_the_above, and accuracy collapsed. With no escape hatch that failure mode cannot occur, so distractors were pushed into a harder similarity band — the model had to learn each distinction instead of dodging it. That is where most of the gain on invented names (93.3% → 96.2%) comes from.

Accuracy: — 6,076 calls, temperature 0, a 243-intent / 22-domain taxonomy with zero message overlap with the training data (verified). In-scope accuracy is 95.0% (acv3 reaches 95.7% with the same design).

In-list obedience is 100% across all 5,362 in-scope calls — it never once answered with a label outside the list when a valid answer existed.

Third-party benchmarks: the same model on public test splits and on six domains held out of training entirely (unseen intents and unseen messages). Training messages were excluded from all three, so nothing scored here was trained on — anyone can reproduce them.

Opt-in refusal: with a catch-all offered, it picks the catch-all on 66.8% of out-of-scope messages, and wrongly grabs it on only 4.8% of valid ones.

accuracy vs acv3 vs acv1 vs oav2 vs overall vs stock qwen

This model led every accuracy row when it shipped; acv3 has since passed it on all of them using the same never-refuse design, so acv3 is the one to reach for. The one row accuracy still wins is catch-all wrongly grabbed (4.8% vs 6.8%) — its catch-all fires less eagerly, at the cost of catching fewer genuinely out-of-scope messages (66.8% vs 75.2%). Prefer it only if a false refusal costs you more than a missed one.

Want the newer version? intent-classifier-general-acv3 is a strict upgrade: same design, 95.7% in-scope and 100% in-list obedience.

Need automatic out-of-scope rejection? See intent-classifier-general-oav2, which rejects 86.7% of out-of-scope messages without being asked, or intent-classifier-general-acv1 for a middle ground (94.3% accuracy, 47.5% rejection). Use this model when every message must be routed somewhere.

Every number here is reproducible. All models in the family are scored on the same audit — 6,076 calls each, 243 intents across 22 domains, with zero message overlap with the training data. Raw per-call results and scoring code: abyssal-llm.github.io/ollama-intent-classifier-audit

Good for: routing support tickets, chatbot intent detection, message tagging, triage — fast, local, fully customizable intents, for any workload where every message must land in exactly one bucket.

License: Apache-2.0 (base). ~3.1 GB, fits an 8 GB GPU.