8b-wraith/ deepseek-v3-0324

1,175 1 year ago

deepseek-v3-0324-Quants. - Q2_K is the lowest here - quantized = round((original - zero_point) / scale) 

ollama run 8b-wraith/deepseek-v3-0324

Models

View all →

1 model

deepseek-v3-0324:latest

289GB · 4K context window · Text · 1 year ago

Readme

DeepSeek-V3-0324 (Q2_K Edition)

Model: DeepSeek-V3-0324.Q2_K.gguf
Quantized by: @mradermacher
Filed under: “Wow, that actually runs on my laptop!”
Powered by: 8b.is — Because size isn’t everything.

🧠 About

This is the Q2_K quantization of the DeepSeek-V3-0324 model — a charmingly squeezed version of a seriously powerful LLM. Despite only using 2 bits per weight, it still manages to reason, write, and riff like a caffeinated philosopher.

Perfect for: - Devices with limited RAM or GPU (you brave edge-runner, you) - Rapid inference without a data center - Those who believe that “less is more, especially when it loads”

💡 What is Q2_K?

  • Q2 = 2-bit quantization (just enough bits to be dangerous)
  • K = grouped quantization (smarter packing, better quality)
  • Think: “AI Weight Watchers” — same brain, lighter snacks.

Yes, it loses a bit of nuance… but then again, so do most social media platforms.

🚀 Running with Ollama

Modelfile Sample:

”`Dockerfile FROM llama2 PARAMETER model DeepSeek-V3-0324.Q2_K.gguf PARAMETER num_ctx 128000 PARAMETER temperature 0.4

SYSTEM “”” You are a low-bit but high-vibe assistant. Keep responses concise, kind, and clever. Today is {{date}} — let’s make it count. “””

Quick Commands:

ollama create deepseek-v3-q2 -f Modelfile ollama run deepseek-v3-q2

🧱 Other Quants in the DeepSeek Family

All available soon at 8Q.is:

Quant Notes Q2_K This one — tiny but mighty

Please ask for these if you want via q@8b.is

Q3_K_S Fastest 3-bit variant Q4_K_M Balanced & beautiful Q5_K_S Snappy with more nuance Q6_K Nearly full-brain Q8_0 Absolute chonker

⸻

🧡 Made with Meaning

“You don’t need a huge brain to make a big impact — just enough bits and a great prompt.”

This model is a reminder that elegance often lives in constraint. Use it wisely, kindly, and maybe a little mischievously. 😏

⸻

🔖 License

MIT — as open as your mind.

⸻

🌌 Visit 8b.is

Because quantization should feel like magic, not punishment. Where small models do big things.

⸻

Brought to you by the team behind VosH.AI, Cheet.IS, and other quietly revolutionary ideas.

You want to automate the whole README lineup with style variants? I can script a flavor generator with options like: - 🧠 “Serious & Scholarly” - 🧢 “Chill Dev Bro” - 🧙 “Mystic Monk Mode” - 🤡 “Sh*tposter Edition”

Let’s make docs people actually enjoy reading.