Abyssal/intent-classifier-general-v3

More accurate than v2, and lighter on refusal. Define intents at prompt time, no retraining. It returns the single best-matching name, or none_of_the_above when nothing fits, refusing 67% of out-of-scope messages.

Recommended usage

Two modes, depending on whether you need out-of-scope rejection.

1. Reliable routing (always pick one of your intents). Pass your intent names as a JSON-schema enum in format. This grammar-constrains decoding so the answer is always one of your intents (never a hallucinated or out-of-list label):

curl http://localhost:11434/api/chat -d '{
  "model": "intent-classifier-general-v3:1.5b",
  "stream": false,
  "options": {"temperature": 0},
  "format": {"type": "string", "enum": ["refund", "tracking", "account"]},
  "messages": [{"role": "user", "content":
    "Candidate intents:\nrefund: wants money back\ntracking: where their order is\naccount: login or profile\n\nUser message: where is my package?\n\nAnswer with exactly one intent name from the list above."
  }]
}'
# -> "tracking"

2. Rejection (detect out-of-scope messages). Leave format off. In free generation the model returns none_of_the_above when no intent fits — something a stock model won’t do reliably (it almost always forces a pick from the list). Enum mode removes this option, so use free generation when you need rejection:

curl http://localhost:11434/api/chat -d '{
  "model": "intent-classifier-general-v3:1.5b",
  "stream": false,
  "options": {"temperature": 0},
  "messages": [{"role": "user", "content":
    "Candidate intents:\nrefund: wants money back\ntracking: where their order is\n\nUser message: what time do you close on sundays?\n\nAnswer with exactly one intent name from the list above."
  }]
}'
# -> "none_of_the_above"

Getting the best results

• Write clear, specific descriptions. The model matches on the description, so this is the single biggest lever. Prefer tracking: track or locate a customer's order or shipment over a terse tracking: order status. Vague one- or two-word descriptions are the main cause of an occasional wrong none_of_the_above.
• Keep temperature 0 (the baked-in default) for deterministic, repeatable routing.
• Use the exact prompt shape shown above (Candidate intents: … User message: … Answer with exactly one intent name from the list above.) — it was trained on this format.
• Names are free-form — snake_case, CamelCase, hyphens or plain words all work, including names the model has never seen; it matches by description and echoes your name verbatim.

Under the hood

Base: LoRA fine-tune of Qwen2.5-1.5B-Instruct (Apache-2.0). Trained across five public intent datasets (300+ intents spanning banking, voice-assistant, travel, and general support), where each example presents a different set of intents with descriptions — so the model learns the skill “read whatever list you are given and match it” rather than a fixed taxonomy. It is also trained with a real none_of_the_above label (out-of-scope examples exclude the correct intent and its near-synonyms), and ~60% of examples are renamed to invented names so it matches by meaning instead of memorizing labels. Runs at temperature 0 and is fully deterministic.

Accuracy: — 6,076 calls, temperature 0, a 243-intent / 22-domain taxonomy (none of it from the training data) with realistic 3–4 intent lists. v3 pushes free-generation in-scope accuracy to 91.2% (vs 87.8% v2, 88.6% stock qwen) while holding 93.5% enum-constrained accuracy. It rejects 66.5% of out-of-scope messages that a stock model would force into the list.

For the out-of-scope row, a low in-list number is the goal: the model escapes the list with none_of_the_above instead of guessing. Stock qwen2.5 stays in-list 86.3% of the time here — it has no deliberate rejection, so the 13.7% it does escape is accidental (rambling), not a clean none_of_the_above.

v3 vs v2 vs stock qwen

Against v2 and the stock model, v3 leads in free-generation accuracy — it gets the right answer more often when the intent is in your list. Newer models in the family score higher again (see below). It trades some rejection rate for that (v2 rejects more but is weaker at free-gen matching). Stock qwen2.5 has no rejection mechanism: it almost always forces a pick, even when nothing fits.

Want stronger out-of-scope rejection at nearly the same accuracy? See intent-classifier-general-overall, which rejects ~74%. For higher accuracy, see intent-classifier-general-accuracy (95.0% in-scope).

Every number here is reproducible. All models in the family are scored on the same audit — 6,076 calls each, 243 intents across 22 domains, with zero message overlap with the training data. Raw per-call results and scoring code: abyssal-intent-classifier-audit

Good for: routing support tickets, chatbot intent detection, message tagging, triage — fast, local, fully customizable intents, with the option to flag messages that match nothing.

License: Apache-2.0 (base). ~3.1 GB, fits an 8 GB GPU.