library_name: gguf license: apache-2.0 pipeline_tag: feature-extraction tags: - qwen3 - embedding - nvfp4 - mteb - llama.cpp - ollama - blackwell

base_model: Qwen/Qwen3-Embedding-0.6B

Qwen3-Embedding-0.6B-Q8-NVFP4

Qwen3-Embedding-0.6B · NVFP4 quantized · 507 MB

A hand-built GGUF of the Qwen3-Embedding-0.6B text embedding model, quantized with a mixed NVFP4 recipe for Blackwell GPUs. 1024-dim embeddings, 32K context, last-token pooling.

Credits

The Qwen team (Qwen/Qwen3-Embedding-0.6B) — built the base model.
AIconjured — the NVFP4 quantization and packaging.

Files

File	Size	Description
`Qwen3-Embedding-0.6B-NVFP4.gguf`	507 MB	310 tensors: 84 NVFP4 (FFN GEMMs) + 113 Q8_0 (embeddings, attention, norms) + 113 F32 (norms/biases/`ssm`)

Size: 507 MB — down from 1198 MB (F16) and 639 MB (Q8_0).

Quantization recipe

Mixed-precision, built with llama.cpp’s --tensor-type-file plus an imatrix generated from ~144 KB of mixed calibration text:

Type	Tensors	What	Why
NVFP4	84	`ffn_gate/up/down` (all 28 layers)	The big GEMMs — where the bits are. Native Blackwell FP4 path.
Q8_0	113	`token_embd`, `attn_q/k/v/output`, all attention norms	Embedding geometry is sensitive; Q8_0 is near-lossless.
F32	113	All norm weights, biases	Tiny but quantizing them causes quality loss.

Recipe (llama.cpp --tensor-type-file format):

token_embd\.weight=q8_0
output_norm\.weight=q8_0
blk\.\d+\.ffn_gate\.weight=nvfp4
blk\.\d+\.ffn_up\.weight=nvfp4
blk\.\d+\.ffn_down\.weight=nvfp4

Measured quality

Embedding cosine similarity vs the F16 original, measured on 20 diverse texts through the actual llama.cpp runtime (last-token pooling, 1024-dim):

Quant	Mean cosine vs F16	Min (worst case)
Q8_0	0.99936	0.99765
NVFP4 (this model)	0.97472	0.95203

Short factual queries stay tight (~0.98); harder semantic queries drift up to ~18°. In retrieval, expect a small recall drop concentrated on nuanced queries. If you need near-lossless embeddings, use the Q8_0 version instead.

Usage

Ollama:

FROM ./Qwen3-Embedding-0.6B-NVFP4.gguf

llama.cpp:

llama-server -m Qwen3-Embedding-0.6B-NVFP4.gguf --embedding -c 32768
# POST /embeddings {"input": "your text"}

Hardware notes

Built and verified on 2× RTX 5060 Ti (Blackwell, sm_120a) with a CUDA 12.8 llama.cpp build.
NVFP4 kernels use the native Blackwell FP4 path; on non-Blackwell GPUs the tensors still load and run (dequantized), with less speed advantage.
The model is tiny (507 MB) — fits entirely in a single GPU with large context headroom.

Reproduction

llama.cpp: 378aa2ebc (2026-08-27), CUDA build
Quantize: llama-quantize --imatrix imatrix-emb.gguf --tensor-type-file recipe-emb-nvfp4.txt ...
Imatrix: 32 chunks × 2048 tokens, mixed wiki/prose calibration

License

Apache-2.0, matching the base Qwen/Qwen3-Embedding-0.6B model.

Embedding Model Qwen3-Embedding-0.6B-Q8-NVFP4

Models

Readme