justinledwards/mistral-small-3.1-Q6_K

justinledwards/

mistral-small-3.1-Q6_K

Mistral Small 3.1 24B quantization from bartowski Q6_K with tool support

tools

660 Pulls Updated 2 weeks ago

Updated 2 weeks ago

2 weeks ago

c1a3db3b81c6 · 19GB

parameters23.6B

quantizationQ6_K

{ "num_ctx": 4096, "stop": [ "</s>", "[INST]", "[/INST]" ] }

{{- range $index, $_ := .Messages }} {{- if eq .Role "system" }}[SYSTEM_PROMPT]{{ .Content }}[/SYSTE

Readme

The other mistral small 3.1 24B models on here didn’t have Q6 quantization and tool support. This one runs on a 3090 in vram with nice results.