richardyoung/ mistral-7b-instruct-v0.3-abliterated:Q4_K_M

694 7 months ago

A conservatively abliterated Mistral-7B-Instruct-v0.3 with reduced refusals, an extended 32K vocabulary, function calling, and preserved instruction-following quality. 

ollama run richardyoung/mistral-7b-instruct-v0.3-abliterated:Q4_K_M

Details

7 months ago

3a7d2344aff1 · 4.4GB ·

model
llama
·
7.25B
·
Q4_K_M

Readme

Mistral-7B-Instruct-v0.3-Abliterated

A conservatively abliterated Mistral-7B-Instruct-v0.3 with reduced refusals, an extended 32K vocabulary, function calling, and preserved instruction-following quality.

🚀 Overview

This is an uncensored (abliterated) build of mistralai/Mistral-7B-Instruct-v0.3, a ~7.25B-parameter instruction-tuned model from Mistral AI. Refusal directions were suppressed using the Heretic library with conservative parameters, lowering refusals while preserving the base model’s instruction-following quality, extended vocabulary, and function-calling capability. Distributed as a GGUF quant for efficient local inference via Ollama / llama.cpp.

📊 Abliteration Results

Metric Before After
Refusals TBD TBD
Reduction – TBD
KL Divergence – TBD

Refusal metrics pending re-measurement.

🎯 Key Features

  • Reduced refusals, engages with a far wider range of prompts than the stock instruct model.
  • Preserved quality, retains the base model’s instruction-following and general reasoning ability.
  • Function calling, inherits v0.3’s native function-calling support and v3 tokenizer.
  • GGUF quantization, distributed as a quantized GGUF for efficient local inference via Ollama / llama.cpp.

🏷️ Available Versions

Tag Size BPW Notes
Q4_K_M 4.4GB ~4.85 (Q4_K_M) Default 4-bit quant; good quality/size balance.

Only the Q4_K_M tag is currently published. Additional tiers can be added on request.

BPW reference (bits per weight): IQ3_M ≈ 3.66 · IQ4_XS ≈ 4.25 · Q4_K_M ≈ 4.85 · Q5_K_M ≈ 5.68 · Q6_K ≈ 6.56 · Q8_0 ≈ 8.5

💻 Quick Start

ollama run richardyoung/mistral-7b-instruct-v0.3-abliterated

🛠️ Use Cases

  • General-purpose chat and instruction following without stock refusal behavior.
  • Tool / function-calling agents and structured-output workflows.
  • Code generation, explanation, and debugging.
  • Open-ended research, writing, and analysis.

📋 System Requirements

Quant Approx. VRAM Performance
Q4_K_M (~4.4GB) ~6–8GB Fits any modern 8GB GPU (RTX 3060⁄4060+); fast
Q4_K_M (offloaded) 4GB GPU + system RAM Usable with partial CPU offload; slower
Q4_K_M (CPU-only) 8GB+ system RAM Functional and reasonably usable on CPU

🔧 Technical Details

  • Base Model: mistralai/Mistral-7B-Instruct-v0.3
  • Parameters: ~7.25B (MistralForCausalLM / mistral)
  • Context Length: 32,768 tokens (32K)
  • Quantization: Q4_K_M (GGUF), single published tag
  • Abliteration: Heretic (p-e-w), conservative weight modification

⚠️ Disclaimer

This model has reduced safety guardrails and will engage with a wider range of prompts. Use responsibly and in accordance with applicable laws.

🙏 Acknowledgments

  • Base Model: Mistral AI (Apache 2.0)
  • Abliteration: Heretic by p-e-w
  • Quantization: llama.cpp

Built & maintained by Richard Young · DeepNeuro