siddhu19/ flint-unsloth:q4_xl

50 2 weeks ago

GGUF quantizations of Flint 39.8B by Recursive AI for coding agents, with Unsloth's Dynamic 3.0 quantization method 

ollama run siddhu19/flint-unsloth:q4_xl

Details

2 weeks ago

7a14cb0c2bd7 · 25GB

model
flint
·
39.8B
·
Q4_K_M
license
Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR US
params
{ "min_p": 0, "num_ctx": 262144, "repeat_penalty": 1, "temperature": 1, "top_k":

Readme

Recursive Flint 39.8B — Unsloth GGUF Builds

This Unsloth Dynamic 3.0 GGUF builds of Recursive Flint tested on a dual AMD 9060 XT system with 32GB combined VRAM. The focus is on practical coding-harness use, memory fit, and runtime behavior.

Model Overview

Flint is Recursive’s 39.8B-parameter mixture-of-experts language model for coding agents, with approximately 2.5B active parameters per token. It is designed to investigate codebases, fix bugs, implement features, and write tests.

Property Value
Developer Recursive
Architecture Decoder-only mixture-of-experts Transformer
Total parameters 39.8B
Active parameters per token Approximately 2.5B
Documented context configuration 262,144 tokens
License Apache 2.0

Number on 32 GB VRAM system

All builds were measured on the same local setup:

Setup Item Value
Hardware Dual AMD 9060 XT
Combined VRAM 32 GB
Runner Ollama with llama.cpp
Context length 262,144 tokens
FlashAttention Enabled
KV cache q8_0
Build Ollama Tag Size Reported Processor Prompt Rate Generation Rate Notes
Q2_XL siddhu19/flint-unsloth:q2_xl 16 GB 100% GPU 38.70 tok/s 45.33 tok/s Lowest memory footprint
Q3_XL siddhu19/flint-unsloth:q3_xl 21 GB 100% GPU 90.72 tok/s 40.82 tok/s Balanced size and prompt rate
Q4_XL siddhu19/flint-unsloth:q4_xl 27 GB 98% GPU / 2% CPU 97.61 tok/s 38.31 tok/s Highest prompt rate; slight system-RAM usage

Recommended Build by GPU VRAM

Available Combined VRAM Recommended Build Notes
16 GB Q2_XL only, if possible Q2_XL is the only tested build with a 16GB file size, but running at full 262k context may still be tight. Reducing context is likely required for reliable 16GB operation.
24 GB Q2_XL is safest; Q3_XL may work with headroom Q2_XL should fit most comfortably. Q3_XL may be usable depending on OS overhead, context usage, and runtime buffers, but I have only verified it on a 32GB system.
32 GB combined Q2_XL, Q3_XL, and Q4_XL All three builds were tested on my dual AMD 9060 XT setup. Q4_XL is the heaviest but shows only a small amount of CPU/system-RAM usage.
48 GB or more Q4_XL should be most comfortable I did not test this directly, but Q4_XL will have the most headroom at 262k context.

Notes on Q4_XL and System-RAM Usage

The Q4_XL build showed a small amount of non-GPU usage in Ollama’s processor report:

Q4_XL Reported Processor Meaning
98% GPU Most inference is still on GPU
2% CPU Small CPU/system-RAM overhead appears in the report

This is expected on AMD GPUs for two possible reasons:

  1. AMD/HIP runtime overhead
    AMD GPU runtimes can add overhead due to HIP and related runtime layers. In practice, this overhead can be up to around 2 GB in some setups.

  2. Long-context KV cache pressure
    With a 262,144-token context, KV-cache memory can still contribute to total memory pressure, even when:

    • FlashAttention is enabled
    • KV cache is set to q8_0

So the small system-RAM usage seen with Q4_XL is likely a combination of AMD/HIP overhead and long-context KV-cache allocation, not a broken model build.

If you need a stricter GPU-only memory profile, use Q3_XL or Q2_XL.