mahade315/ mamba-codestral:7b-q4_K_M

49 2 weeks ago

Mistral's Mamba-Codestral-7B (Mamba2 SSM code model), Q4_K_M GGUF. Apache 2.0. Requires Ollama ≥ 0.23.0. 

ollama run mahade315/mamba-codestral:7b-q4_K_M

Details

2 weeks ago

a20c7b61092b · 4.1GB

model
mamba2
·
7.29B
·
Q4_K_M
template
{{- range $index, $_ := .Messages }} {{- if eq .Role "user" }}[INST] {{ if and $.System (eq (len (sl
license
Apache License 2.0 Mamba-Codestral-7B-v0.1 by Mistral AI: https://huggingface.co/mistralai/Mamba-Cod
params
{ "num_ctx": 16384, "stop": [ "[INST]", "[/INST]", "</s>" ],

Readme

Mamba-Codestral 7B (Q4_K_M)

A Q4_K_M GGUF quantization of mistralai/Mamba-Codestral-7B-v0.1, a 7B code-generation model from Mistral AI built on the Mamba2 state-space architecture instead of a transformer.

Run

ollama run mahade315/mamba-codestral:7b-q4_K_M

Requires Ollama 0.23.0 or newer. This is a pure mamba2 model. Older versions may fail to load it.

Details
Base model mistralai/Mamba-Codestral-7B-v0.1
Architecture Mamba2 (64 layers, hidden size 4096, state size 128)
Quantization Q4_K_M (≈4.55 bits/weight), 4.1 GB
Kept in F32 ssm_a, ssm_d, ssm_dt.bias, ssm_conv1d, all norms
Output layer Q6_K
Default context 16384 (num_ctx)
Prompt format Mistral instruct: [INST] ... [/INST]
License Apache 2.0

Converted with llama.cpp’s convert_hf_to_gguf.py (bf16 → f16) and quantized with llama-quantize.

Default parameters

temperature 0.2
num_ctx     16384
stop        [INST], [/INST], </s>

Override per request, e.g. "options": {"temperature": 0.7, "num_ctx": 32768}.

Notes

  • Tested with Ollama 0.23.0 on an NVIDIA L40S: loads fully on GPU (~4.3 GB VRAM at 16k context), ~108 tokens/s generation.
  • System prompts are supported. Following Mistral’s convention, the system prompt is prepended to the last user message.
  • The training data predates recent library versions. For example, Terraform output may use deprecated AWS provider syntax (inline versioning {} on aws_s3_bucket).
  • Q4_K_M trades some quality for size. Evaluate against the original weights for quality-sensitive work.

License & attribution

Original model © Mistral AI, released under the Apache License 2.0. This repository only redistributes a quantized conversion. No weights were fine-tuned or modified beyond quantization.