LoPld/ devstral-small-2-24b-instruct-2512-q4_K_S:latest

33 Downloads Updated 2 weeks ago

Quantized Q4_K_S version of Devstral small 2 24B FP16 model

vision tools

ollama run LoPld/devstral-small-2-24b-instruct-2512-q4_K_S

curl http://localhost:11434/api/chat \
  -d '{
    "model": "LoPld/devstral-small-2-24b-instruct-2512-q4_K_S",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from ollama import chat

response = chat(
    model='LoPld/devstral-small-2-24b-instruct-2512-q4_K_S',
    messages=[{'role': 'user', 'content': 'Hello!'}],
)
print(response.message.content)

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'LoPld/devstral-small-2-24b-instruct-2512-q4_K_S',
  messages: [{role: 'user', content: 'Hello!'}],
})
console.log(response.message.content)

Details

Updated 2 weeks ago

2 weeks ago

49d3870291c7 · 14GB ·

model

archmistral3

·

parameters24B

·

quantizationQ4_K_S

14GB

template

{{- $lastUserIndex := -1 }} {{- $hasSystemPrompt := false }} {{- range $index, $_ := .Messages }} {{

3.4kB

params

{ "temperature": 0.15 }

21B

Readme

Devstral Small 2 – 24B (Q4_K_S)

This is a quantized version (Q4_K_S) of the original:

Devstral Small 2 – 24B Instruct (FP16)

Overview

Base model: Devstral Small 2 – 24B Instruct (FP16)
Quantization: Q4_K_S
License: Apache 2.0
Format: Ollama model
Type: Inference-optimized snapshot

Purpose

This model was created to:

Reduce memory usage compared to FP16
Improve inference speed on consumer GPUs
Enable running a 24B model on more modest hardware
Preserve a stable snapshot of the original model

Performance / Size

Model footprint: ~14 GB (Q4_K_S)
VRAM usage increases with context length due to KV cache
Suitable for GPU inference on mid-to-high range consumer hardware

Hardware Notes (estimated)

16 GB VRAM GPU:
- Stable at ~2K context length in typical setups
~20 GB VRAM GPU (not fully tested):
- Likely supports ~4K–6K context depending on backend and KV cache settings

Trade-offs

Compared to FP16:

✔ Lower VRAM usage
✔ Faster inference
❌ Slight reduction in output quality and precision

Usage

Run locally:

ollama run LoPld/devstral-small-2-24b-instruct-2512-q4_K_S

Reproducibility

Created from:

FROM devstral-small-2:24b-instruct-2512-fp16

Quantized using:

ollama create <model> --quantize q4_K_S -f Modelfile

License

Original model: Apache 2.0 This quantized version inherits the same license terms.

Notes

This is a frozen snapshot of the original model
No upstream updates are expected for this version
Intended for long-term local use and reproducibility

# Devstral Small 2 – 24B (Q4_K_S)

This is a **quantized version (Q4_K_S)** of the original:

> Devstral Small 2 – 24B Instruct (FP16)

---

## Overview

* Base model: Devstral Small 2 – 24B Instruct (FP16)
* Quantization: Q4_K_S
* License: Apache 2.0
* Format: Ollama model
* Type: Inference-optimized snapshot

---

## Purpose

This model was created to:

* Reduce memory usage compared to FP16
* Improve inference speed on consumer GPUs
* Enable running a 24B model on more modest hardware
* Preserve a stable snapshot of the original model

---

## Performance / Size

* Model footprint: ~14 GB (Q4_K_S)
* VRAM usage increases with context length due to KV cache
* Suitable for GPU inference on mid-to-high range consumer hardware

---

## Hardware Notes (estimated)

* 16 GB VRAM GPU:

* Stable at ~2K context length in typical setups

* ~20 GB VRAM GPU (not fully tested):

* Likely supports ~4K–6K context depending on backend and KV cache settings

---

## Trade-offs

Compared to FP16:

* ✔ Lower VRAM usage
* ✔ Faster inference
* ❌ Slight reduction in output quality and precision

---

## Usage

Run locally:

```bash
ollama run LoPld/devstral-small-2-24b-instruct-2512-q4_K_S
```

---

## Reproducibility

Created from:

```text
FROM devstral-small-2:24b-instruct-2512-fp16
```

Quantized using:

```bash
ollama create <model> --quantize q4_K_S -f Modelfile
```

---

## License

Original model: Apache 2.0
This quantized version inherits the same license terms.

---

## Notes

* This is a frozen snapshot of the original model
* No upstream updates are expected for this version
* Intended for long-term local use and reproducibility

---

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)