richardyoung/ kimi-vl-a3b-thinking:Q4_K_M

1,266 8 months ago

Kimi-VL-A3B-Thinking is a powerful vision-language model from Moonshot AI featuring extended thinking capabilities. Built on the DeepSeek2 architecture with Mixture of Experts (MoE), it excels at complex visual reasoning tasks, mathematical problem-s 

ollama run richardyoung/kimi-vl-a3b-thinking:Q4_K_M

Details

8 months ago

df2421324d74 · 11GB ·

model
deepseek2
·
16B
·
Q4_K_M
template
{{ if .System }}<|im_system|>system<|im_middle|>{{ .System }}<|im_end|>{{ else }}<|im_system|>system
system
You are Kimi, a helpful AI assistant with vision and reasoning capabilities from Moonshot AI. You ca
license
# Kimi-VL-A3B-Thinking: Advanced Vision-Language Model with Extended Reasoning ## 🚀 Overview Kimi
params
{ "num_ctx": 131072, "stop": [ "<|im_end|>", "<|endoftext|>" ] }

Readme

Kimi-VL-A3B-Thinking

Vision-language Mixture-of-Experts reasoning model from Moonshot AI, with extended “thinking” (chain-of-thought) for complex multimodal tasks.

🚀 Overview

Kimi-VL-A3B-Thinking is a compact yet powerful vision + text (multimodal) model that pairs a native-resolution visual encoder (MoonViT) and an MLP projector with a Mixture-of-Experts language decoder. Despite activating only ~3B parameters per token, it delivers strong long-horizon reasoning, working through problems step-by-step before answering, excelling at visual math, document and chart understanding, OCR-with-reasoning, and detailed image analysis. This repository provides GGUF quantizations for easy local inference with Ollama.

📊 Benchmarks

Reported by Moonshot AI for the base moonshotai/Kimi-VL-A3B-Thinking:

Benchmark Score
MMMU (val) 61.7
MathVista 71.3
MathVision 36.8
LongVideoBench 64.5
InfoVQA 83.2

🎯 Key Features

  • Multimodal: Accepts both images and text; reasons jointly over visual and textual context.
  • Extended Thinking: Produces explicit chain-of-thought before final answers for stronger reasoning.
  • Efficient MoE: ~16B total parameters with only ~3B active per token, high capability at low inference cost.
  • Native-Resolution Vision: MoonViT encoder handles high-resolution images and documents without aggressive downscaling.
  • Long Context: 128K-token context window for long documents, multi-image, and extended reasoning chains.

🏷️ Available Versions

Tag Size BPW Notes
iq4_xs 8.8 GB ~4.25 Smallest, good quality/size balance
latest / Q4_K_M 11 GB ~4.85 Recommended default
Q5_K_M 12 GB ~5.68 Higher quality
Q6_K 14 GB ~6.56 Near-lossless
q8_0 17 GB ~8.50 Highest-fidelity quant
f16 32 GB 16.0 Full precision (unquantized)

💻 Quick Start

# Pull and run the recommended version
ollama run richardyoung/kimi-vl-a3b-thinking

# Run a specific quantization
ollama run richardyoung/kimi-vl-a3b-thinking:Q6_K

# Ask about an image (provide the path inline)
ollama run richardyoung/kimi-vl-a3b-thinking "Solve the math problem in this image ./problem.png"

🛠️ Use Cases

  • Visual Math & Reasoning: Solve problems posed in images with step-by-step working.
  • Document & Chart Understanding: Interpret tables, charts, diagrams, and scanned documents.
  • OCR with Reasoning: Extract text and reason about its meaning, not just transcribe it.
  • Image Analysis: Detailed scene description and visual question answering.
  • Research: Explore efficient MoE vision-language reasoning locally.

📋 System Requirements

VRAM Performance
8 GB Runs iq4_xs; may offload to CPU
12 GB Good performance (Q4/Q5)
16 GB+ Excellent performance (up to Q8_0)
32 GB+ Full f16 precision

🔧 Technical Details

  • Base Model: moonshotai/Kimi-VL-A3B-Thinking
  • Architecture: Mixture-of-Experts language decoder + MoonViT visual encoder + MLP projector
  • Parameters: ~16B total / ~3B active per token (MoE)
  • Modality: Vision + Text (multimodal)
  • Context Length: 128K tokens
  • License: MIT
  • Quantization: GGUF (IQ4_XS through F16)

⚠️ Disclaimer

This is a quantized redistribution of Moonshot AI’s Kimi-VL-A3B-Thinking for local inference. Quantization can introduce minor quality degradation relative to the full-precision model; lower-bit quants trade accuracy for size and speed. Model outputs, including reasoning chains, may contain errors and should be verified for any consequential use. Use responsibly and in accordance with the upstream MIT license.

🙏 Acknowledgments

Built & maintained by Richard Young · DeepNeuro