Abyssal/intent-classifier-general-acv1

Accuracy first, with refusal still built in. Define intents at prompt time, no retraining. It returns the single best-matching name, or none_of_the_above when nothing fits. Pick it when you want both at once.

acv1 = Accuracy-Calibrated, Version 1.

Recommended usage

Two modes, depending on whether you need out-of-scope rejection.

1. Reliable routing (always pick one of your intents). Pass your intent names as a JSON-schema enum in format. This grammar-constrains decoding so the answer is always one of your intents (never a hallucinated or out-of-list label):

curl http://localhost:11434/api/chat -d '{
  "model": "intent-classifier-general-acv1:1.5b",
  "stream": false,
  "options": {"temperature": 0},
  "format": {"type": "string", "enum": ["refund", "tracking", "account"]},
  "messages": [{"role": "user", "content":
    "Candidate intents:\nrefund: wants money back\ntracking: where their order is\naccount: login or profile\n\nUser message: where is my package?\n\nAnswer with exactly one intent name from the list above."
  }]
}'
# -> "tracking"

2. Rejection (detect out-of-scope messages). Leave format off. In free generation the model returns none_of_the_above when no intent fits — something a stock model won’t do reliably (it almost always forces a pick). Enum mode removes this option, so use free generation when you need rejection:

curl http://localhost:11434/api/chat -d '{
  "model": "intent-classifier-general-acv1:1.5b",
  "stream": false,
  "options": {"temperature": 0},
  "messages": [{"role": "user", "content":
    "Candidate intents:\nrefund: wants money back\ntracking: where their order is\n\nUser message: what time do you close on sundays?\n\nAnswer with exactly one intent name from the list above."
  }]
}'
# -> "none_of_the_above"

Getting the best results

• Write clear, specific descriptions. The model matches on the description, so this is the single biggest lever. Prefer tracking: track or locate a customer's order or shipment over a terse tracking: order status.
• Keep temperature 0 (the baked-in default) for deterministic, repeatable routing.
• Use the exact prompt shape shown above (Candidate intents: … User message: … Answer with exactly one intent name from the list above.) — it was trained on this format.
• Names are free-form — snake_case, CamelCase, hyphens or plain words all work, including names the model has never seen; it matches by description and echoes your name verbatim.

Under the hood

Base: LoRA fine-tune of Qwen2.5-1.5B-Instruct (Apache-2.0). Trained on five public intent datasets plus 65 business domains generated for this release — veterinary clinics, HR portals, logistics, legal services, utilities, gaming, healthcare and more — for 896 deduplicated intents in total, each example presenting a different candidate list.

Two changes account for the accuracy jump over earlier versions. The intent pool is deduplicated: when hundreds of taxonomies are merged, the same intent shows up twice under different names (dental::appointment_rescheduling and hospital::appointment_rescheduling scored a cosine similarity of 1.00), and training a model to “pick one” between identical options teaches it nothing except to hedge. Near-duplicates are collapsed. Distractors are then similarity-band-limited — close enough to force real discrimination, never so close that the label is ambiguous. Together these cut over-rejection on valid requests from 18.8% to 1.4%.

It is also trained with a real none_of_the_above label and ~35% of examples renamed to invented names, so it matches on meaning rather than memorising label strings. Runs at temperature 0 and is fully deterministic.

Accuracy: — 6,076 calls, temperature 0, a 243-intent / 22-domain taxonomy with zero message overlap with the training data (verified). acv1 reaches 94.1% free-generation and 94.6% enum-constrained in-scope accuracy, and matches or beats stock qwen2.5 on every cell — something only acv1 and the later accuracy model manage. Of the two, acv1 is the one that still has a built-in refusal.

For the out-of-scope row, a low in-list number is the goal: the model escapes the list with none_of_the_above instead of guessing. Stock qwen2.5 stays in-list 86.3% of the time here — it has no deliberate rejection, so the 13.7% it does escape is accidental (rambling), not a clean none_of_the_above.

acv1 vs oav2 vs overall vs v3 vs v2 vs stock qwen

acv1 is the accuracy-first option that still refuses on its own: it gained roughly 3 points of free-generation accuracy over v3 and nearly 6 points on invented/custom intent names, while rejecting 3.5× more out-of-scope traffic than the stock model. The trade-off is worth stating plainly: acv1 commits to an answer more readily, so its rejection rate is lower than oav2’s. It has since been surpassed on raw accuracy by intent-classifier-general-accuracy (95.0% vs 94.3% in-scope), which drops built-in refusal entirely — acv1 remains the pick when you want high accuracy and an automatic none_of_the_above.

Need something more specialised? intent-classifier-general-oav2 rejects far more (86.7%), and intent-classifier-general-accuracy is more accurate (95.0%) but never refuses unless you add a catch-all. Use acv1 for both at once.

Every number here is reproducible. All models in the family are scored on the same audit — 6,076 calls each, 243 intents across 22 domains, with zero message overlap with the training data. Raw per-call results and scoring code: abyssal-intent-classifier-audit

Good for: routing support tickets, chatbot intent detection, message tagging, triage — fast, local, fully customizable intents, with the option to flag messages that match nothing.

License: Apache-2.0 (base). ~3.1 GB, fits an 8 GB GPU.