Devstral Small 2 – 24B (Q4_K_S)

This is a quantized version (Q4_K_S) of the original:

Devstral Small 2 – 24B Instruct (FP16)

Overview

• Base model: Devstral Small 2 – 24B Instruct (FP16)
• Quantization: Q4_K_S
• License: Apache 2.0
• Format: Ollama model
• Type: Inference-optimized snapshot

Purpose

This model was created to:

• Reduce memory usage compared to FP16
• Improve inference speed on consumer GPUs
• Enable running a 24B model on more modest hardware
• Preserve a stable snapshot of the original model

Performance / Size

• Model footprint: ~14 GB (Q4_K_S)
• VRAM usage increases with context length due to KV cache
• Suitable for GPU inference on mid-to-high range consumer hardware

Hardware Notes (estimated)

• 16 GB VRAM GPU:

  • Stable at ~2K context length in typical setups

• ~20 GB VRAM GPU (not fully tested):

  • Likely supports ~4K–6K context depending on backend and KV cache settings

Trade-offs

Compared to FP16:

• ✔ Lower VRAM usage
• ✔ Faster inference
• ❌ Slight reduction in output quality and precision

Usage

Run locally:

ollama run LoPld/devstral-small-2-24b-instruct-2512-q4_K_S

Reproducibility

Created from:

FROM devstral-small-2:24b-instruct-2512-fp16

Quantized using:

ollama create <model> --quantize q4_K_S -f Modelfile

License

Original model: Apache 2.0 This quantized version inherits the same license terms.

Notes

• This is a frozen snapshot of the original model
• No upstream updates are expected for this version
• Intended for long-term local use and reproducibility