NVIDIA Nemotron Nano 9B v2 — Ollama Model (Q4_K_M, single 16 GB GPU)

Model: oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU

DESCRIPTION

A Q4_K_M build of NVIDIA Nemotron Nano 9B v2, NVIDIA’s open 9B reasoning model, quantized locally from the BF16 source and tuned for a single 16 GB GPU card with maximum KV cache. The Q4_K_M weights (6.53 GB) fit entirely on a 16 GB GPU — verified 100% GPU / 0% CPU offload — leaving ~9 GB of VRAM for a large KV cache. Uses the hybrid Mamba-2 + attention architecture for efficient long-context inference.

The ideal single-16GB-GPU reasoning companion for: RTX 5060 Ti 16GB · RTX 4080 · RTX 5080 · RTX 4090 · any NVIDIA setup with 16 GB VRAM

Key Features

• Q4_K_M quantization: 6.53 GB weights — fits entirely on a single 16 GB GPU with ~9 GB left for KV cache
• Hybrid Mamba-2 + Attention: dense 9B model with efficient long-context inference
• 128K native context (131,072 tokens)
• Thinking mode: On by default — thinking… response reasoning blocks, returned by Ollama as message.thinking
• Native tool calling: <tool_call> XML format, parseable with the qwen3_coder parser used by agent harnesses (OpenCode, OpenClaw, Hermes Agent, Claude Code)
• /think & /no_think controls: force or skip extended reasoning per request
• NVIDIA Open Model License: free for commercial and personal use

Architecture

REQUIREMENTS

Platform support: - NVIDIA GPU (single): 16 GB VRAM (RTX 5060 Ti 16GB, RTX 4080, RTX 5080) — verified 100% GPU / 0% CPU - NVIDIA GPU (dual): 16 GB + 8 GB (RTX 5060 Ti + RTX 4060 Ti) - Apple Silicon: Mac with 16 GB+ unified memory

💡 Why Q4_K_M? This build quantizes the 17.8 GB BF16 source down to 6.53 GB Q4_K_M, so it fits entirely on a single 16 GB GPU with ~9 GB left for KV cache. For the full 128K context, use --no-kv-offload (KV cache in system RAM) via the llama.cpp config in the Hugging Face repo.

QUICK START

1. Install Ollama

# macOS (Homebrew)
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download from https://ollama.com/download

2. Pull & Run

# Pull the model (downloads ~6.5 GB)
ollama pull oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU

# Run interactively
ollama run oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU

# Single prompt
ollama run oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU \
  "Explain the difference between Mamba and Transformer architectures"

💡 latest tag: This model is also available as oamazonasgabriel/nemotron-nano-9b-v2 (defaults to latest, the BF16 weights).

USAGE

CLI

# Interactive chat (thinking on by default)
ollama run oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU

# Force extended reasoning
ollama run oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU "/think Solve this step by step: ..."

# Skip reasoning for a direct answer
ollama run oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU "/no_think What is 2+2?"

REST API

# Chat completion
curl -s http://127.0.0.1:11434/api/chat \
  -d '{
    "model": "oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# OpenAI-compatible endpoint
curl -s http://127.0.0.1:11434/v1/chat/completions \
  -d '{
    "model": "oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'

Python (ollama library)

pip install ollama

import ollama

# Chat (thinking returned as message.thinking)
response = ollama.chat(
    model='oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU',
    messages=[{'role': 'user', 'content': 'Write a Python function to merge two sorted lists'}],
)
print(response.message.content)

JavaScript (ollama.js)

npm install ollama

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU',
  messages: [{ role: 'user', content: 'Explain recursion with an example' }],
})
console.log(response.message.content)

SAMPLING PARAMETERS

These are baked into the model via its Modelfile:

MEMORY & PERFORMANCE

VRAM Budget (single 16 GB GPU)

Context Window Scaling (q4_0 KV)

On a single 16 GB GPU:

💡 For the full 128K context, use the llama.cpp config with --no-kv-offload (KV cache in system RAM) — see the Hugging Face repo.

Performance

• Text generation: ~25-40 tok/s (Q4_K_M, all layers on GPU)
• Model load time: ~20-30 seconds (~6.5 GB load)

Opencode Integration

Add to ~/.config/opencode/opencode.jsonc:

"oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU": {
  "name": "oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU",
  "options": {
    "supportsThinking": true,
    "contextWindow": 131072
  }
}

Use as agent:

"model": "ollama/oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU"

Or launch directly:

ollama launch opencode --model oamazonasgabriel/nemotron-nano-9b-v2:q4-km-16gbGPU

TROUBLESHOOTING

Model fails to load — “unable to allocate buffer”

Cause: Not enough free VRAM. This is a 6.53 GB Q4_K_M model. Fix: Free at least 8 GB before loading, or reduce num_ctx.

CPU offloading detected

Run ollama ps — if CPU% appears, reduce num_ctx or set OLLAMA_KV_CACHE_TYPE=q4_0.

Slow generation

On 16 GB you should get 25-40 tok/s. If slower: - Check for CPU offloading (ollama ps) - Verify GPU is on CUDA (not Vulkan) - Close competing GPU processes

Want the full 128K context?

Ollama keeps the KV cache in VRAM. For 128K context, use the llama.cpp config with --no-kv-offload (KV cache in system RAM) from the Hugging Face repo.

ALTERNATIVE MODELS

# Smaller / faster Nemotron (MoE, needs 24 GB)
ollama pull oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU

# Coding-optimized Qwen (needs 16 GB)
ollama pull oamazonasgabriel/qwen3.5-9b:q4-16gbGPU

# Heavy-lift Qwen (needs 32 GB)
ollama pull oamazonasgabriel/qwen3.8-27b:q4-km-32gbGPU

CREDITS

LINKS