Abyssal/intent-classifier-general-acv3

The most accurate model in the family, and the best at telling lookalike intents apart. Define intents at prompt time, no retraining. It always returns the single best match, and refuses only if you add a catch-all intent.

Recommended usage

Two modes, depending on whether you need out-of-scope detection.

1. Routing (the normal case). No catch-all, no enum — the model always answers with one of your intents. Across all 6,076 audit calls it never once produced a label outside the candidate list:

curl http://localhost:11434/api/chat -d '{
  "model": "intent-classifier-general-acv3:1.5b",
  "stream": false,
  "options": {"temperature": 0},
  "messages": [{"role": "user", "content":
    "Candidate intents:\nrefund: wants money back\ntracking: where their order is\naccount: login or profile\n\nUser message: where is my package?\n\nAnswer with exactly one intent name from the list above."
  }]
}'
# -> "tracking"

Add a JSON-schema enum in format if you want decoding itself grammar-constrained: "format": {"type": "string", "enum": ["refund", "tracking", "account"]}.

2. Out-of-scope detection (opt-in). Append a catch-all intent and treat that answer as “no match”. Any wording works — other, misc, something_else, not_listed, general_inquiry:

curl http://localhost:11434/api/chat -d '{
  "model": "intent-classifier-general-acv3:1.5b",
  "stream": false,
  "options": {"temperature": 0},
  "messages": [{"role": "user", "content":
    "Candidate intents:\nrefund: wants money back\ntracking: where their order is\nother: anything that does not match the other intents\n\nUser message: what time do you close on sundays?\n\nAnswer with exactly one intent name from the list above."
  }]
}'
# -> "other"

Remove that other line and the same message returns tracking — the closest available match.

Getting the best results

• Write clear, specific descriptions. The model matches on the description, so this is the single biggest lever. Prefer tracking: track or locate a customer's order or shipment over a terse tracking: order status.
• Keep temperature 0 (the baked-in default) for deterministic, repeatable routing.
• Use the exact prompt shape shown above (Candidate intents: … User message: … Answer with exactly one intent name from the list above.) — it was trained on this format.
• Names are free-form — snake_case, CamelCase, hyphens or plain words all work, including names the model has never seen; it matches by description and echoes your name verbatim.
• Only add a catch-all when you actually need it. Offering an escape hatch costs a few points of in-scope accuracy, because a model given one occasionally takes it.

Under the hood

Base: LoRA fine-tune of Qwen2.5-1.5B-Instruct (Apache-2.0). Trained on five public intent datasets plus 65 business domains generated for this family — 896 deduplicated intents, each example presenting a different candidate list, ~35% of them renamed to invented names so the model matches on meaning rather than memorising label strings. Runs at temperature 0 and is fully deterministic.

acv3 adds contrastive hard-pair training: 340 confusable intent pairs were mined from the training taxonomy and a local 12B model wrote 4,673 messages that belong to one side and clearly not the other, so the model trains directly on the decisions it used to get wrong. Pairs are mined from the training taxonomy only, never from the audit, so this is a transferable skill rather than benchmark fitting. This was only possible because the model has no refusal option — earlier attempts to train on close pairs made it hedge into none_of_the_above and accuracy collapsed.

Accuracy: — 6,076 calls, temperature 0, a 243-intent / 22-domain taxonomy with zero message overlap with the training data (verified). In-scope accuracy is 95.7%, the highest in the family.

In-list obedience is 100.0% across all 6,076 calls — not just the in-scope ones. It never produced a token outside the offered list.

Third-party benchmarks: public test splits plus six domains held out of training entirely (unseen intents and unseen messages). Training messages were excluded from all three, so nothing scored here was trained on — anyone can reproduce them.

Banking77 is the benchmark made almost entirely of fine-grained lookalike intents — exactly what the contrastive pairs target — and it gains +1.3 points over acv2.

Opt-in refusal: with a catch-all offered, it picks the catch-all on 75.2% of out-of-scope messages (up from 66.8% in accuracy), and wrongly grabs it on 6.8% of valid ones. Offering the catch-all costs a few points of in-scope accuracy, so add it only when you need out-of-scope detection.

acv3 vs accuracy vs acv1 vs oav2 vs overall vs stock qwen

acv3 leads every accuracy row. It is a strict upgrade over accuracy: same never-refuse design and the same opt-in catch-all, with better discrimination on lookalike intents (+0.8) and perfect in-list obedience.

Need automatic out-of-scope rejection? See intent-classifier-general-oav2, which rejects 86.7% of out-of-scope messages without being asked, or intent-classifier-general-acv1 for a middle ground (94.3% accuracy, 47.5% rejection). Use acv3 when every message must be routed somewhere.

Every number here is reproducible. All models in the family are scored on the same audit — 6,076 calls each, 243 intents across 22 domains, with zero message overlap with the training data. Raw per-call results and scoring code: abyssal-llm.github.io/ollama-intent-classifier-audit

Good for: routing support tickets, chatbot intent detection, message tagging, triage — fast, local, fully customizable intents, for any workload where every message must land in exactly one bucket.

License: Apache-2.0 (base). ~3.1 GB, fits an 8 GB GPU.