A local evaluator for retrieval-augmented generation (RAG) answers.

tools 1b 3b

ollama run erukude/rag-audit-judge:1b

curl http://localhost:11434/api/chat \
  -d '{
    "model": "erukude/rag-audit-judge:1b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from ollama import chat

response = chat(
    model='erukude/rag-audit-judge:1b',
    messages=[{'role': 'user', 'content': 'Hello!'}],
)
print(response.message.content)

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'erukude/rag-audit-judge:1b',
  messages: [{role: 'user', content: 'Hello!'}],
})
console.log(response.message.content)

Applications

Claude Code ollama launch claude --model erukude/rag-audit-judge:1b

OpenCode ollama launch opencode --model erukude/rag-audit-judge:1b

Hermes Agent ollama launch hermes --model erukude/rag-audit-judge:1b

OpenClaw ollama launch openclaw --model erukude/rag-audit-judge:1b

Models

View all →

Name

2 models

Size / Usage

Context

Input

rag-audit-judge:1b

1.4GB · 128K context window · Text · 2 months ago

rag-audit-judge:1b

1.4GB

128K

Text

rag-audit-judge:3b

2.1GB · 128K context window · Text · 2 months ago

rag-audit-judge:3b

2.1GB

128K

Text

Readme

RAG Audit Judge

Base model: Llama3.2.

It checks:

Is the answer grounded in the supplied context?
Is reasoning unsupported?
Are citations missing or invalid?
Is the response overconfident?
Is sensitive information exposed?

It returns one JSON object:

{
  "groundedness_score": 0.82,
  "hallucination_risk": "low",
  "citation_quality": "medium",
  "issues": [],
  "recommended_fix": ""
}

Provide a question, retrieved context entries with stable source IDs, and the candidate answer. The judge evaluates only the supplied context, not outside knowledge.

Example user input:

Question: Can I return a headset?

Context:
- id: refund-policy
  text: Headsets can be returned within 30 days if they are unopened.

Candidate answer:
An unopened headset can be returned within 30 days, and a $5 restocking fee always applies [refund-policy].

Author

Author: Sai Teja Erukude
Role: Developer & Maintainer