LoPld/ devstral-24b-small-2505-q4_K_S:latest

39 Downloads Updated 1 month ago

Quantized Q4_K_S version of Devstral small 24B FP16 model

tools

ollama run LoPld/devstral-24b-small-2505-q4_K_S

curl http://localhost:11434/api/chat \
  -d '{
    "model": "LoPld/devstral-24b-small-2505-q4_K_S",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from ollama import chat

response = chat(
    model='LoPld/devstral-24b-small-2505-q4_K_S',
    messages=[{'role': 'user', 'content': 'Hello!'}],
)
print(response.message.content)

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'LoPld/devstral-24b-small-2505-q4_K_S',
  messages: [{role: 'user', content: 'Hello!'}],
})
console.log(response.message.content)

Details

Updated 1 month ago

1 month ago

576e80456f3a · 14GB ·

model

archllama

·

parameters23.6B

·

quantizationQ4_K_S

14GB

template

{{- $lastUserIndex := -1 }} {{- range $index, $_ := .Messages }} {{- if eq .Role "user" }}{{ $lastUs

823B

license

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR US

11kB

system

You are Devstral, a helpful agentic model trained by Mistral AI and using the OpenHands scaffold. Yo

5.7kB

Readme

Devstral Small – 24B (Q4_K_S)

This is a quantized version (Q4_K_S) of the original:

Devstral Small – 24B (FP16)

Overview

Base model: Devstral Small – 24B (FP16)
Quantization: Q4_K_S
License: Apache 2.0
Format: Ollama model
Type: Inference-optimized snapshot

Purpose

This model was created to:

Reduce memory usage compared to FP16
Improve inference speed on consumer GPUs
Enable running a 24B model on more modest hardware
Preserve a stable snapshot of the original model

Performance / Size

Model footprint: ~13 GB (Q4_K_S)
VRAM usage increases with context length due to KV cache
Suitable for GPU inference on mid-to-high range consumer hardware

Hardware Notes (estimated)

16 GB VRAM GPU:
- Stable at ~4K context length in typical setups
~20 GB VRAM GPU :
- Supports ~6K–8K context depending on backend and KV cache settings

Trade-offs

Compared to FP16:

✔ Lower VRAM usage
✔ Faster inference
❌ Slight reduction in output quality and precision

Usage

Run locally:

ollama run LoPld/devstral-24b-small-2505-q4_K_S

Reproducibility

Created from:

FROM devstral:24b-small-2505-fp16

Quantized using:

ollama create <model> --quantize q4_K_S -f Modelfile

License

Original model: Apache 2.0 This quantized version inherits the same license terms.

Notes

This is a frozen snapshot of the original model
No upstream updates are expected for this version
Intended for long-term local use and reproducibility

# Devstral Small – 24B (Q4_K_S)

This is a **quantized version (Q4_K_S)** of the original:

> Devstral Small – 24B (FP16)

---

## Overview

* Base model: Devstral Small – 24B (FP16)
* Quantization: Q4_K_S
* License: Apache 2.0
* Format: Ollama model
* Type: Inference-optimized snapshot

---

## Purpose

This model was created to:

* Reduce memory usage compared to FP16
* Improve inference speed on consumer GPUs
* Enable running a 24B model on more modest hardware
* Preserve a stable snapshot of the original model

---

## Performance / Size

* Model footprint: ~13 GB (Q4_K_S)
* VRAM usage increases with context length due to KV cache
* Suitable for GPU inference on mid-to-high range consumer hardware

---

## Hardware Notes (estimated)

* 16 GB VRAM GPU:

* Stable at ~4K context length in typical setups

* ~20 GB VRAM GPU :

* Supports ~6K–8K context depending on backend and KV cache settings

---

## Trade-offs

Compared to FP16:

* ✔ Lower VRAM usage
* ✔ Faster inference
* ❌ Slight reduction in output quality and precision

---

## Usage

Run locally:

```bash
ollama run LoPld/devstral-24b-small-2505-q4_K_S
```

---

## Reproducibility

Created from:

```text
FROM devstral:24b-small-2505-fp16
```

Quantized using:

```bash
ollama create <model> --quantize q4_K_S -f Modelfile
```

---

## License

Original model: Apache 2.0
This quantized version inherits the same license terms.

---

## Notes

* This is a frozen snapshot of the original model
* No upstream updates are expected for this version
* Intended for long-term local use and reproducibility

---

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)