NVIDIA Nemotron 3.5 Lightning (30B-A3B) — Ollama Model (24 GB) MoE

Model: oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU

---

DESCRIPTION

A memory-efficient Ollama build of NVIDIA Nemotron 3.5 Lightning 30B-A3B, the open 30B mixture-of-experts (MoE) model with ~3B active parameters built for the execution layer of always-on agents. Uses the official Ollama-library Q4_K_M quant (25 GB, digest e7a64ff15fb1) and a verified layer split that runs on a 24 GB total VRAM setup with only ~18% of weights in system RAM.

The ideal agentic/tool-calling companion for: RTX 5060 Ti 16GB + RTX 4060 Ti 8GB (dual) · RTX 4090 · RTX 5080 · any NVIDIA setup with 24 GB VRAM + 64 GB system RAM

Key Features

• Hybrid MoE architecture: 30B total params, ~3B active per token — quality of a 30B model at the speed of a 3B
• Mamba-2 + MoE + Attention hybrid: only ~6 attention layers, so the KV cache is tiny and long contexts are cheap
• Thinking mode: On by default — NVIDIA_Nemotron-3.5-Lightning-30B-A3B emits <think>…</think> reasoning blocks, returned by Ollama as message.thinking
• Native tool calling: <tool_call> XML format, parseable with the qwen3_coder parser used by agent harnesses (OpenCode, OpenClaw, Hermes Agent, Claude Code)
• Fast generation: verified ~28–38 tok/s (only ~3B active params fetched per token)
• Long context: 262K native (Ollama metadata reports 1M via rope scaling); 16K configured for the VRAM budget
• Multilingual: English, Spanish, French, German, Italian, Japanese
• OpenMDW-1.1 license: free for commercial and personal use

Architecture

Property	Value
Architecture	nemotron_h — hybrid Mamba-2 + MoE + Attention, Multi-Token Prediction (MTP)
Total Parameters	30B
Active Parameters	~3B (per token)
Experts	128 routed + 1 shared, 6 routed active per token
Layers	52 (interleaved; ~6 attention, rest Mamba-2/MoE)
Native Context	262,144 tokens
Extended Context	up to ~1,000,000 via rope scaling
Modalities	Text
Quantization	Q4_K_M (~25 GB, official Ollama library tag)
Model Size	~25 GB (weights)
License	OpenMDW-1.1
Upstream	NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

---

REQUIREMENTS

Resource	Minimum	Recommended (verified)
GPU Memory	24 GB total VRAM	24 GB+ total VRAM
System RAM	48 GB	64 GB
Disk Space	26 GB free	50 GB+ free
NVIDIA Driver	550+	580.173.02+ (CUDA 13)
Ollama Version	0.32.9+	Latest

⚠️ Ollama 0.32.9+ is required. This model’s GGUF metadata rejects older runtimes — ollama create returns HTTP 412 “requires a newer version of Ollama” on v0.30.6 (verified).

Platform support: - NVIDIA GPU (dual): 16 GB + 8 GB (RTX 5060 Ti + RTX 4060 Ti) — verified split 82% GPU / 18% CPU - NVIDIA GPU (single): 24 GB+ VRAM (RTX 4090, RTX 5080, A5000, etc.) — the 25 GB weights spill ~1 GB to CPU - Apple Silicon: 24 GB+ unified memory — use the official MLX tag instead (nemotron-3.5-lightning:30b-mlx, 23 GB) - AMD GPU: ROCm-compatible with 24 GB+ VRAM (not verified for this build)

💡 Why 24 GB? The Q4_K_M weights are ~25 GB — slightly over the 24 GB ceiling. This Modelfile offloads 44 of 54 layers to GPU (82%, ~20.7 GB on VRAM) and leaves ~4.7 GB in system RAM. Because the hybrid Mamba-2 architecture keeps the attention KV cache tiny (~90 MB at 16K with q4_0), total VRAM usage stays ~22–23 GB — comfortably inside the budget. MoE experts that aren’t activated stay memory-mapped, so only ~3B params are fetched per token.

---

QUICK START

1. Install Ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# macOS / Windows
# Download from https://ollama.com/download

Requires Ollama 0.32.9+ — check with ollama --version. On Debian/Ubuntu you may need to update the repo or install the newer .deb/binaries manually.

2. Set Environment Variables (Recommended)

Set these before starting Ollama to minimize memory usage:

# KV Cache at q4_0 — much smaller footprint than default f16
export OLLAMA_KV_CACHE_TYPE=q4_0

# IMPORTANT: do NOT set OLLAMA_SCHED_SPREAD=1 (leave default OFF).
#   Spread mode on a 16GB+8GB pair plans a ~9.8GB contiguous tensor block
#   for the 8GB card -> cudaMalloc OOM at load. With spread OFF, Ollama packs
#   the 16GB card first (verified working split: 14.3GB + 7.8GB GPU / 18% CPU).

# Limit to one request at a time (memory constraint)
export OLLAMA_NUM_PARALLEL=1
export OLLAMA_MAX_LOADED_MODELS=1

# Optional: force the CUDA backend explicitly
# export OLLAMA_LLM_LIBRARY=cuda_v13

# Optional: faster attention on compatible GPUs
# export OLLAMA_FLASH_ATTENTION=1

Permanent setup (Linux systemd):

sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_KV_CACHE_TYPE=q4_0"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MAX_LOADED_MODELS=1"

Then:

sudo systemctl daemon-reload
sudo systemctl restart ollama

3. Pull & Run

# Pull the model (downloads ~25 GB — fast, blob is the official-library digest)
ollama pull oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU

# Run interactively
ollama run oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU

# Single prompt
ollama run oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU \
  "Write a Python function to fetch and parse a JSON API"

---

USAGE

CLI

# Interactive chat
ollama run oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU

# With a system prompt
ollama run oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU \
  --system "You are a rigorous code reviewer"

REST API

# Chat completion (thinking + answer)
curl -s http://127.0.0.1:11434/api/chat \
  -d '{
    "model": "oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU",
    "messages": [{"role": "user", "content": "What is 17 * 23?"}],
    "stream": false
  }'
# Reasoning arrives in "message.thinking", the answer in "message.content"

# Generate (completion)
curl -s http://127.0.0.1:11434/api/generate \
  -d '{
    "model": "oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU",
    "prompt": "List 5 uses for a local LLM agent",
    "stream": false,
    "options": { "num_ctx": 8192, "num_predict": 512 }
  }'

# OpenAI-compatible endpoint
curl -s http://127.0.0.1:11434/v1/chat/completions \
  -d '{
    "model": "oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'

# Disable thinking for low-latency agents
curl -s http://127.0.0.1:11434/api/chat \
  -d '{
    "model": "oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU",
    "messages": [{"role": "user", "content": "Summarize this in one line."}],
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Python (ollama library)

pip install ollama

import ollama

# Chat — reasoning lands in response.message.thinking, answer in .content
response = ollama.chat(
    model='oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU',
    messages=[{'role': 'user', 'content': 'Debug this SQL query'}],
)
print(response.message.content)

# Generate
response = ollama.generate(
    model='oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU',
    prompt='Write a Python function to sort a list',
)
print(response.response)

# Disable thinking for low latency
response = ollama.chat(
    model='oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU',
    messages=[{'role': 'user', 'content': 'Say hello'}],
    chat_template_kwargs={'enable_thinking': False},
)

JavaScript (ollama.js)

npm install ollama

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU',
  messages: [{ role: 'user', content: 'Hello!' }],
})
console.log(response.message.content)

---

SAMPLING PARAMETERS

These are baked into the model via its Modelfile:

Parameter	Value	Rationale
num_ctx	16384	Safe with tiny q4_0 KV cache (hybrid Mamba-2 arch)
num_gpu	44	Verified max — 82% GPU / 18% CPU (14.3 GB + 7.8 GB). 45+ layers load but OOM at inference (compute buffers overflow the 8 GB card)
temperature	1.0	NVIDIA official recommendation (all tasks, incl. reasoning & tools)
top_p	0.95	NVIDIA official recommendation
top_k	20	Focused token selection
repeat_penalty	1.0	No repetition penalty
stop	<\|im_start\|>, <\|im_end\|>	ChatML delimiters

---

MEMORY & PERFORMANCE

VRAM Budget

Component	Size
Model weights (Q4_K_M, 25 GB) — GPU portion	~20.7 GB (44⁄54 layers)
Model weights — CPU portion	~4.7 GB (18%, system RAM)
KV cache (q4_0, 16K context)	~0.1 GB
Ollama process + compute buffers	~1-1.5 GB
Total (q4_0 + 16K)	~22-23 GB ✅
Total (f16 KV + 16K)	~23-24 GB ⚠️ tight

⚠️ On dual-GPU, account for ~1 GB of desktop baseline VRAM (X11/Wayland, browser).

Context Window Scaling (q4_0 KV — hybrid arch keeps it tiny)

Context	KV Cache	Fits 24 GB?	Notes
8,192	~0.05 GB	✅ Plenty of headroom	~22.2 GB total
16,384	~0.1 GB	✅ Recommended	~22.3 GB total
32,768	~0.2 GB	✅	~22.4 GB total
65,536	~0.4 GB	✅ (needs 64 GB RAM)	more CPU offload OK
131,072	~0.8 GB	⚠️	only with large system RAM
262,144 (native)	~1.6 GB	⚠️ tight	needs 64 GB+ RAM

Performance (verified: RTX 5060 Ti 16 GB + RTX 4060 Ti 8 GB, Ollama 0.32.9)

• Prompt processing: ~1540 tok/s (warm)
• Text generation: ~28–38 tok/s (only ~3B active params per token)
• GPU utilization: both GPUs active during inference (~19–21%)
• Model load time: ~1–2 min (25 GB cold load)

The MoE architecture activates only 6 of 129 experts per token, so generation stays surprisingly fast for a 30B-parameter model.

---

MONITORING

# Check model is GPU-resident (if CPU% is high, reduce context)
ollama ps

# Real-time GPU usage (NVIDIA)
watch -n 1 nvidia-smi

# GPU memory query
nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv

# Ollama server logs (Linux)
journalctl -u ollama -n 50 --no-pager

---

TROUBLESHOOTING

Symptom	Fix
OOM at load — cudaMalloc ... failed to allocate CUDA1 buffer of size 9843976320	OLLAMA_SCHED_SPREAD must be OFF. Spread mode plans a ~9.8 GB contiguous block for the 8 GB card. Unset it and restart the server.
Model loads but OOM when generating	num_gpu too high. Keep 44 (45–46 layers overflow the 8 GB card’s ~400 MB headroom at inference).
Empty content, answer stuck inside thinking	num_predict too low for reasoning prompts (raise to 256–512) or a <think> prefix was added to the template — this Modelfile already uses NVIDIA’s exact prompt (<\|im_start\|>assistant\n \n).
Falls back to Vulkan / no CUDA detected	Unset any mangled CUDA_VISIBLE_DEVICES (e.g. 0000:01:00.0,...), then force OLLAMA_LLM_LIBRARY=cuda_v13 and restart.
HTTP 412 “requires a newer version of Ollama”	Upgrade to Ollama 0.32.9+ (v0.30.6 rejects this model).
Second GPU not used / bad split	Update NVIDIA driver to 550+, ensure OLLAMA_SCHED_SPREAD is off, restart the server, re-check ollama ps.
Slow generation	Expected for MoE (~28–38 tok/s). If far slower, check for excessive CPU offloading (ollama ps CPU%) and reduce num_ctx.
Tool calling returns malformed XML	Keep Q4_K_M quality — lower quants (Q3_K_M/Q2_K) may degrade tool-call formatting.

---

Opencode Integration

Add to ~/.config/opencode/opencode.jsonc:

"oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU": {
  "name": "oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU",
  "options": {
    "supportsThinking": true,
    "contextWindow": 16384
  }
}

Use as agent:

"model": "ollama/oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU"

Tool calls use the qwen3_coder <tool_call> XML parser, so OpenCode/agent harnesses parse them natively. You can also launch it directly:

> ollama launch opencode --model nemotron-3.5-lightning
> ```

---

## ALTERNATIVE MODELS

```bash
# Fully-GPU variant for 24 GB single cards (Q3_K_M, ~19.9 GB — community GGUF)
ollama pull hf.co/ijohn07/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-Q3_K_M-GGUF:Q3_K_M

# Official higher-quality quants (need 48 GB+ VRAM)
ollama pull nemotron-3.5-lightning:q8_0    # 35 GB
ollama pull nemotron-3.5-lightning:bf16    # 66 GB

# Apple Silicon (23 GB, 256K context)
ollama pull nemotron-3.5-lightning:30b-mlx

# Similar-class MoE tuned for the same 24 GB setup
ollama pull oamazonasgabriel/qwen3.6-35b-a3b:q4-24gbGPU

---

CREDITS

Role	Entity
Base Model	NVIDIA
Original Model	NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
GGUF Quantization	NVIDIA (official Ollama library — Q4_K_M 25 GB, digest e7a64ff15fb1)
Ollama Packaging	impacte.tech
License	OpenMDW-1.1

---

LINKS

Resource	URL
This model on Ollama	https://ollama.com/oamazonasgabriel/nemotron-3.5-lightning
Upstream model (HuggingFace)	https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
Ollama library page	https://ollama.com/library/nemotron-3.5-lightning
OpenMDW-1.1 license	https://openmdw.ai/license/1-1/
Ollama documentation	https://docs.ollama.com
Project repository	https://github.com/oamazonasgabriel/ollama-training
Built by	impacte.tech