Abyssal/intent-classifier-general-overall

The balanced pick: high accuracy and steady refusal together. Define intents at prompt time, no retraining. It returns the single best-matching name, or none_of_the_above when nothing fits. Rejects 74% of out-of-scope.

Recommended usage

Two modes, depending on whether you need out-of-scope rejection.

1. Reliable routing (always pick one of your intents). Pass your intent names as a JSON-schema enum in format. This grammar-constrains decoding so the answer is always one of your intents (never a hallucinated or out-of-list label):

curl http://localhost:11434/api/chat -d '{
  "model": "intent-classifier-general-overall:1.5b",
  "stream": false,
  "options": {"temperature": 0},
  "format": {"type": "string", "enum": ["refund", "tracking", "account"]},
  "messages": [{"role": "user", "content":
    "Candidate intents:\nrefund: wants money back\ntracking: where their order is\naccount: login or profile\n\nUser message: where is my package?\n\nAnswer with exactly one intent name from the list above."
  }]
}'
# -> "tracking"

2. Rejection (detect out-of-scope messages). Leave format off. In free generation the model returns none_of_the_above when no intent fits — something a stock model won’t do reliably (it almost always forces a pick from the list). Enum mode removes this option, so use free generation when you need rejection:

curl http://localhost:11434/api/chat -d '{
  "model": "intent-classifier-general-overall:1.5b",
  "stream": false,
  "options": {"temperature": 0},
  "messages": [{"role": "user", "content":
    "Candidate intents:\nrefund: wants money back\ntracking: where their order is\n\nUser message: what time do you close on sundays?\n\nAnswer with exactly one intent name from the list above."
  }]
}'
# -> "none_of_the_above"

Getting the best results

• Write clear, specific descriptions. The model matches on the description, so this is the single biggest lever. Prefer tracking: track or locate a customer's order or shipment over a terse tracking: order status. Vague one- or two-word descriptions are the main cause of an occasional wrong none_of_the_above.
• Keep temperature 0 (the baked-in default) for deterministic, repeatable routing.
• Use the exact prompt shape shown above (Candidate intents: … User message: … Answer with exactly one intent name from the list above.) — it was trained on this format.
• Names are free-form — snake_case, CamelCase, hyphens or plain words all work, including names the model has never seen; it matches by description and echoes your name verbatim.

Under the hood

Base: LoRA fine-tune of Qwen2.5-1.5B-Instruct (Apache-2.0). Trained across five public intent datasets (300+ intents spanning banking, voice-assistant, travel, and general support), where each example presents a different set of intents with descriptions — so the model learns the skill “read whatever list you are given and match it” rather than a fixed taxonomy. It is also trained with a real none_of_the_above label (out-of-scope examples exclude the correct intent and its near-synonyms), and ~60% of examples are renamed to invented names so it matches by meaning instead of memorizing labels. Runs at temperature 0 and is fully deterministic.

Accuracy: — 6,076 calls, temperature 0, a 243-intent / 22-domain taxonomy (none of it from the training data) with realistic 3–4 intent lists. The overall model delivers 90.2% free-generation in-scope accuracy (vs 87.8% v2, 88.6% stock qwen) while holding 94.0% enum-constrained accuracy — matching v3’s in-scope performance. It rejects 73.9% of out-of-scope messages that a stock model would force into the list — a meaningful jump over v3’s 66.5% without sacrificing accuracy.

For the out-of-scope row, a low in-list number is the goal: the model escapes the list with none_of_the_above instead of guessing. Stock qwen2.5 stays in-list 86.3% of the time here — it has no deliberate rejection, so the 13.7% it does escape is accidental (rambling), not a clean none_of_the_above.

Overall vs v3 vs v2 vs stock qwen

The overall model hits the sweet spot: it nearly matches v3’s free-gen accuracy (90.2% vs 91.2%) while rejecting 73.9% of out-of-scope messages — a meaningful jump over v3’s 66.5% without the accuracy tradeoff v2 makes. It also matches v3’s enum accuracy (94.0%). Stock qwen2.5 has no rejection mechanism: it almost always forces a pick, even when nothing fits.

Within the family this is the balance option, not the peak on either axis: oav2 rejects more (86.7%) and accuracy is more accurate (95.0% in-scope).

Every number here is reproducible. All models in the family are scored on the same audit — 6,076 calls each, 243 intents across 22 domains, with zero message overlap with the training data. Raw per-call results and scoring code: abyssal-intent-classifier-audit

Good for: routing support tickets, chatbot intent detection, message tagging, triage — fast, local, fully customizable intents, with the option to flag messages that match nothing.

License: Apache-2.0 (base). ~3.1 GB, fits an 8 GB GPU.