A local evaluator for retrieval-augmented generation (RAG) answers.

tools 1b 3b

ollama run erukude/rag-audit-judge:3b

curl http://localhost:11434/api/chat \
  -d '{
    "model": "erukude/rag-audit-judge:3b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from ollama import chat

response = chat(
    model='erukude/rag-audit-judge:3b',
    messages=[{'role': 'user', 'content': 'Hello!'}],
)
print(response.message.content)

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'erukude/rag-audit-judge:3b',
  messages: [{role: 'user', content: 'Hello!'}],
})
console.log(response.message.content)

Details

Updated 2 months ago

2 months ago

363fe372386d · 2.1GB ·

model

archllama

parameters3.21B

quantizationQ4_K_M

2.0GB

system

You are RAG-AUDIT-JUDGE, a strict evaluator for retrieval-augmented generation answers. Return exact

212B

params

{ "num_ctx": 4096, "stop": [ "<|start_header_id|>", "<|end_header_id|>",

129B

adapter

archllama

parameters48.6M

97MB

license

LLAMA 3.2 COMMUNITY LICENSE AGREEMENT Llama 3.2 Version Release Date: September 25, 2024 “Agreemen

7.7kB

license

**Llama 3.2** **Acceptable Use Policy** Meta is committed to promoting safe and fair use of its tool

6.0kB

template

<|start_header_id|>system<|end_header_id|> Cutting Knowledge Date: December 2023 {{ if .System }}{{

1.4kB

Readme

RAG Audit Judge

Base model: Llama3.2.

It checks:

Is the answer grounded in the supplied context?
Is reasoning unsupported?
Are citations missing or invalid?
Is the response overconfident?
Is sensitive information exposed?

It returns one JSON object:

{
  "groundedness_score": 0.82,
  "hallucination_risk": "low",
  "citation_quality": "medium",
  "issues": [],
  "recommended_fix": ""
}

Provide a question, retrieved context entries with stable source IDs, and the candidate answer. The judge evaluates only the supplied context, not outside knowledge.

Example user input:

Question: Can I return a headset?

Context:
- id: refund-policy
  text: Headsets can be returned within 30 days if they are unopened.

Candidate answer:
An unopened headset can be returned within 30 days, and a $5 restocking fee always applies [refund-policy].

Author

Author: Sai Teja Erukude
Role: Developer & Maintainer