richardyoung/ zephyr-7b-beta-abliterated:Q4_K_M

44 7 months ago

An abliterated build of HuggingFaceH4's Zephyr-7B-β 

ollama run richardyoung/zephyr-7b-beta-abliterated:Q4_K_M

Details

7 months ago

d26f1cb8a6c8 · 4.4GB ·

model
llama
·
7.24B
·
Q4_K_M
params
{ "stop": [ "<|system|>", "</s>", "<|user|>", "<|assistant|>"
template
{{- range .Messages }}<|{{ .Role }}|> {{ .Content }}</s> {{ end }}<|assistant|>

Readme

Zephyr-7B-β-Abliterated

An abliterated build of HuggingFaceH4’s Zephyr-7B-β, a DPO-aligned chat model built on Mistral-7B, with reduced refusals while preserving its helpful, conversational instruction-following style.

🚀 Overview

This is an uncensored (abliterated) build of HuggingFaceH4/zephyr-7b-beta, a ~7.24B-parameter chat model fine-tuned by the Hugging Face H4 team from mistralai/Mistral-7B-v0.1 using Direct Preference Optimization (DPO). Refusal directions were suppressed using the Heretic library, lowering refusals while preserving the base model’s friendly, conversational instruction-following quality. Distributed as a GGUF quant for efficient local inference via Ollama / llama.cpp.

📊 Abliteration Results

Metric Before After
Refusals TBD TBD
Reduction – TBD
KL Divergence – TBD

Refusal metrics pending re-measurement.

🎯 Key Features

  • Reduced refusals, engages with a far wider range of prompts than the stock Zephyr chat model.
  • Preserved quality, retains Zephyr’s DPO-aligned, helpful conversational style and reasoning ability.
  • Mistral foundation, built on Mistral-7B-v0.1, inheriting its efficient architecture.
  • GGUF quantization, distributed as a quantized GGUF for efficient local inference via Ollama / llama.cpp.

🏷️ Available Versions

Tag Size BPW Notes
Q4_K_M 4.4GB ~4.85 (Q4_K_M) Default 4-bit quant; good quality/size balance.

Only the Q4_K_M tag is currently published. Additional tiers can be added on request.

BPW reference (bits per weight): IQ3_M ≈ 3.66 · IQ4_XS ≈ 4.25 · Q4_K_M ≈ 4.85 · Q5_K_M ≈ 5.68 · Q6_K ≈ 6.56 · Q8_0 ≈ 8.5

💻 Quick Start

ollama run richardyoung/zephyr-7b-beta-abliterated

🛠️ Use Cases

  • General-purpose chat and assistant-style conversation without stock refusal behavior.
  • Open-ended research, writing, and analysis.
  • Code generation, explanation, and debugging.
  • Creative writing and brainstorming.

📋 System Requirements

Quant Approx. VRAM Performance
Q4_K_M (~4.4GB) ~6–8GB Fits any modern 8GB GPU (RTX 3060⁄4060+); fast
Q4_K_M (offloaded) 4GB GPU + system RAM Usable with partial CPU offload; slower
Q4_K_M (CPU-only) 8GB+ system RAM Functional and reasonably usable on CPU

🔧 Technical Details

  • Base Model: HuggingFaceH4/zephyr-7b-beta (DPO-tuned from mistralai/Mistral-7B-v0.1)
  • Parameters: ~7.24B (MistralForCausalLM / mistral)
  • Context Length: 32,768 tokens (32K), with 4096-token sliding-window attention
  • License: MIT
  • Quantization: Q4_K_M (GGUF), single published tag
  • Abliteration: Heretic (p-e-w), targeted weight modification

⚠️ Disclaimer

This model has reduced safety guardrails and will engage with a wider range of prompts. Use responsibly and in accordance with applicable laws.

🙏 Acknowledgments

  • Base Model: Hugging Face H4 (built on Mistral AI’s Mistral-7B-v0.1), MIT license
  • Abliteration: Heretic by p-e-w
  • Quantization: llama.cpp

Built & maintained by Richard Young · DeepNeuro