Devstral Small – 24B (Q4_K_S)

This is a quantized version (Q4_K_S) of the original:

Devstral Small – 24B (FP16)

Overview

• Base model: Devstral Small – 24B (FP16)
• Quantization: Q4_K_S
• License: Apache 2.0
• Format: Ollama model
• Type: Inference-optimized snapshot

Purpose

This model was created to:

• Reduce memory usage compared to FP16
• Improve inference speed on consumer GPUs
• Enable running a 24B model on more modest hardware
• Preserve a stable snapshot of the original model

Performance / Size

• Model footprint: ~13 GB (Q4_K_S)
• VRAM usage increases with context length due to KV cache
• Suitable for GPU inference on mid-to-high range consumer hardware

Hardware Notes (estimated)

• 16 GB VRAM GPU:

  • Stable at ~4K context length in typical setups

• ~20 GB VRAM GPU :

  • Supports ~6K–8K context depending on backend and KV cache settings

Trade-offs

Compared to FP16:

• ✔ Lower VRAM usage
• ✔ Faster inference
• ❌ Slight reduction in output quality and precision

Usage

Run locally:

ollama run LoPld/devstral-24b-small-2505-q4_K_S

Reproducibility

Created from:

FROM devstral:24b-small-2505-fp16

Quantized using:

ollama create <model> --quantize q4_K_S -f Modelfile

License

Original model: Apache 2.0 This quantized version inherits the same license terms.

Notes

• This is a frozen snapshot of the original model
• No upstream updates are expected for this version
• Intended for long-term local use and reproducibility