Main Model • Collection • GitHub (MNN) • Tech Report

English | 中文

MNN-Pocket-0.3B-GGUF

GGUF export of MNN-Pocket-0.3B for llama.cpp and compatible runtimes (LM Studio, ollama, …).

MNN-Pocket is a family of compact (~300M parameter) language models co-designed with the MNN inference stack for on-device deployment, targeting phones, tablets, laptops and smart glasses.

• Three quants to trade quality against footprint: F16 (726 MB), Q8_0 (387 MB), Q4_1 (192 MB)
• Q4_1 is quantization-aware trained, so the 4-bit file keeps near-BF16 accuracy (49.59 vs 50.00 average on the 11-benchmark suite)
• 32K context and ChatML-style chat template embedded in the GGUF metadata
• Best for on-device assistants, data extraction and lightweight agentic pipelines; like other sub-billion models it is not recommended for reasoning-heavy workloads (advanced math, long-horizon code)

Find more details in the original model card: https://huggingface.co/taobao-mnn/MNN-Pocket-0.3B

Demo

iPhone 18 Pro Demo

🏃 How to run

This repo ships three GGUF files, so always pass --hf-file to pick one explicitly. Example usage with llama.cpp (QAT Q4_1, the recommended on-device quant):

llama-cli -hf taobao-mnn/MNN-Pocket-0.3B-GGUF \
    --hf-file MNN-Pocket-0.3B-Instruct-Q4_1.gguf \
    --temp 0.8 --top-p 0.8 --top-k 20

Local server (OpenAI-compatible endpoint):

llama-server -hf taobao-mnn/MNN-Pocket-0.3B-GGUF --hf-file MNN-Pocket-0.3B-Instruct-Q4_1.gguf \
    -c 4096 --port 8080

The GGUF metadata carries the ChatML-style chat template, so conversation formatting is automatic in recent llama.cpp versions.

Which file is picked by default

Without --hf-file, llama.cpp searches the repo for Q4_K_M, then Q8_0, and otherwise falls back to the first GGUF it finds. In this repo that resolves to MNN-Pocket-0.3B-Instruct-Q8_0.gguf (387 MB) — note this is not the QAT Q4_1 build. To select a quant explicitly, either pass --hf-file as above, or use the shorthand tag form:

llama-cli -hf taobao-mnn/MNN-Pocket-0.3B-GGUF:Q4_1

Files

QAT Q4_1 GGUF

The quantization-aware-training checkpoint is available as MNN-Pocket-0.3B-Instruct-Q4_1.gguf.

This is distinct from the BF16-derived F16 / Q8_0 files: Q4_1 is converted from the QAT checkpoint, i.e. the model was trained with quantization in the loop rather than quantized after the fact, so it keeps near-BF16 accuracy at a 4-bit footprint.

llama-cli -hf taobao-mnn/MNN-Pocket-0.3B-GGUF \
  --hf-file MNN-Pocket-0.3B-Instruct-Q4_1.gguf \
  -p "What is MNN?"

Model Details

Chat Template

ChatML-style conversation format; the template is embedded in the GGUF metadata, so llama.cpp enters conversation mode automatically — no extra flag needed. Schematic:

[IM_START]system
You are a helpful assistant.[IM_END]
[IM_START]user
Hello![IM_END]
[IM_START]assistant

📊 Benchmarks

Scores of the underlying model (higher is better). F16 / Q8_0 track the BF16 column, Q4_1 tracks the INT4 (QAT) column.

Other Formats

• MNN-Pocket-0.3B — BF16 safetensors, full model card (723 MB)
• MNN-Pocket-0.3B-MNN — INT4 W4A16 QAT build for the MNN runtime (173 MB)
• MNN-Pocket-0.3B-Base / MNN-Pocket-0.3B-SFT — pre-training / SFT checkpoints
• MNN-Pocket-0.3B-Dec — non-autoregressive decision model (LoRA adapter + pointer head)

License

Apache License 2.0. See LICENSE.txt.