EmbeddingGemma-300M NVFP4+Q8 Mixed Quantization

A high-fidelity mixed-precision requantization of Google’s EmbeddingGemma-300M embedding model, optimized for Blackwell GPU tensor-core inference while preserving near-8-bit embedding quality.

Model Overview

Quantization Strategy

This build uses a conservative mixed-precision recipe that prioritizes embedding quality while still leveraging Blackwell’s native NVFP4 tensor-core operations for the compute-heavy FFN layers.

Tensor Type Distribution

Recipe Details

# F32 protections (CUDA kernel requirements)
^blk\.\d+\.attn_norm\.weight$=f32
^blk\.\d+\.post_attention_norm\.weight$=f32
^blk\.\d+\.attn_q_norm\.weight$=f32
^blk\.\d+\.attn_k_norm\.weight$=f32
^blk\.\d+\.ffn_norm\.weight$=f32
^blk\.\d+\.post_ffw_norm\.weight$=f32
^output_norm\.weight$=f32

# Q8_0 protections (most sensitive to quantization)
^token_embd\.weight$=q8_0
^blk\.\d+\.attn_q\.weight$=q8_0
^blk\.\d+\.attn_k\.weight$=q8_0
^blk\.\d+\.attn_v\.weight$=q8_0
^blk\.\d+\.attn_output\.weight$=q8_0

# NVFP4 on FFN weights (compute-heavy, tolerant)
^blk\.\d+\.ffn_gate\.weight$=nvfp4
^blk\.\d+\.ffn_up\.weight$=nvfp4
^blk\.\d+\.ffn_down\.weight$=nvfp4
^dense_2\.weight$=nvfp4
^dense_3\.weight$=nvfp4

Why This Placement?

• Q8_0 on attention + token embedding: The token embedding layer (204 MB, 64% of the model) is the first transformation of input text into the model’s representation space. Attention weights directly shape how the model processes relationships between tokens. These are the most sensitive to quantization noise, so they stay at Q8_0.
• NVFP4 on FFN weights: The feed-forward layers are compute-heavy but more tolerant to lower precision. NVFP4 provides native Blackwell tensor-core operations with calibrated input scales, delivering faster inference with minimal quality impact.
• F32 on norms: Required by CUDA kernel constraints (llama.cpp’s embedding model loader asserts F32 for norm tensors).

Quality Verification

Embedding quality was measured by computing cosine similarity between embeddings generated by the Q8_0 original and this NVFP4+Q8 mixed version across 10 diverse prompts covering multiple semantic domains (animals, food, repairs, science, weather, entertainment, baking).

Results: - Average cosine similarity: 0.9703 - Minimum: 0.9653 - Maximum: 0.9756

All 10 prompts achieved ≥0.965 similarity, well above the 0.95 threshold generally considered acceptable for embedding models. This confirms that the conservative mixed-precision strategy preserves embedding quality effectively.

Comparison with More Aggressive Recipe

A more aggressive recipe (NVFP4 on all compute weights including attention) was also tested: - Average cosine similarity: 0.9517 - Minimum: 0.9401

While still acceptable, the mild version provides measurably better quality with only a 18 MB size difference, making it the preferred choice for most use cases.

Performance

NVFP4 FFN layers leverage Blackwell’s native FP4 tensor-core operations, delivering faster inference on RTX 50-series GPUs. Actual speedup varies by hardware and workload, but NVFP4 operations are significantly faster than Q8_0 on compatible hardware.

Hardware Requirements

At 286 MB, this model is extremely lightweight and will run on virtually any hardware.

Usage with Ollama

# Pull the model
ollama pull aiconjured/embeddinggemma-300M-NVFP4-Q8-GGUF

# Use as embedding model
ollama embed --model aiconjured/embeddinggemma-300M-NVFP4-Q8-GGUF "Your text here"

Usage with llama.cpp

# Embedding inference
./llama-embed -m embeddinggemma-300M-NVFP4-Q8-GGUF -f input.txt

# Batch embedding
./llama-embed -m embeddinggemma-300M-NVFP4-Q8-GGUF -f batch.txt -b 32

Files Included

• embeddinggemma-300M-NVFP4-Q8-GGUF.gguf — Main model (286 MB)

Reproduction

To reproduce this quantization:

# 1. Download Q8_0 source from ggml-org's repo
# 2. Quantize with mixed-precision recipe
./llama-quantize --allow-requantize \
  --tensor-type-file recipe-nvfp4-mild.txt \
  embeddinggemma-300M-Q8_0.gguf \
  embeddinggemma-300M-NVFP4-Q8-GGUF.gguf Q8_0 8

# 3. Patch file_type to 39 (MOSTLY_NVFP4)
# (Use a GGUF header patcher — the quantizer doesn't set this)

No imatrix is needed for embedding models — they don’t benefit from input-scale calibration the way generation models do.

License

Same as base model: Apache 2.0

Acknowledgments

• Base model: ggml-org/embeddinggemma-300M-GGUF — GGUF conversion of Google’s EmbeddingGemma-300M
• Original architecture: google/embedding-gemma-300m — state-of-the-art open embedding model by Google
• NVFP4 quantization: ggml-org/llama.cpp — native Blackwell tensor-core FP4 operations
• Quality benchmarks: Cosine similarity testing across diverse semantic domains

About the Author

Part of the AIconjured NVFP4 quantization series.