NVIDIA Nemotron 3.5 Lightning (30B-A3B) — Ollama Model (24 GB) MoE

Model: oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU

DESCRIPTION

A memory-efficient Ollama build of NVIDIA Nemotron 3.5 Lightning 30B-A3B, the open 30B mixture-of-experts (MoE) model with ~3B active parameters built for the execution layer of always-on agents. Uses the official Ollama-library Q4_K_M quant (25 GB, digest e7a64ff15fb1) and a verified layer split that runs on a 24 GB total VRAM setup with only ~18% of weights in system RAM.

The ideal agentic/tool-calling companion for: RTX 5060 Ti 16GB + RTX 4060 Ti 8GB (dual) · RTX 4090 · RTX 5080 · any NVIDIA setup with 24 GB VRAM + 64 GB system RAM

Key Features

• Hybrid MoE architecture: 30B total params, ~3B active per token — quality of a 30B model at the speed of a 3B
• Mamba-2 + MoE + Attention hybrid: only ~6 attention layers, so the KV cache is tiny and long contexts are cheap
• Thinking mode: On by default — NVIDIA_Nemotron-3.5-Lightning-30B-A3B emits <think>…</think> reasoning blocks, returned by Ollama as message.thinking
• Native tool calling: <tool_call> XML format, parseable with the qwen3_coder parser used by agent harnesses (OpenCode, OpenClaw, Hermes Agent, Claude Code)
• Fast generation: verified ~28–38 tok/s (only ~3B active params fetched per token)
• Long context: 262K native (Ollama metadata reports 1M via rope scaling); 16K configured for the VRAM budget
• Multilingual: English, Spanish, French, German, Italian, Japanese
• OpenMDW-1.1 license: free for commercial and personal use

Architecture

REQUIREMENTS

⚠️ Ollama 0.32.9+ is required. This model’s GGUF metadata rejects older runtimes — ollama create returns HTTP 412 “requires a newer version of Ollama” on v0.30.6 (verified).

Platform support: - NVIDIA GPU (dual): 16 GB + 8 GB (RTX 5060 Ti + RTX 4060 Ti) — verified split 82% GPU / 18% CPU - NVIDIA GPU (single): 24 GB+ VRAM (RTX 4090, RTX 5080, A5000, etc.) — the 25 GB weights spill ~1 GB to CPU - Apple Silicon: 24 GB+ unified memory — use the official MLX tag instead (nemotron-3.5-lightning:30b-mlx, 23 GB) - AMD GPU: ROCm-compatible with 24 GB+ VRAM (not verified for this build)

💡 Why 24 GB? The Q4_K_M weights are ~25 GB — slightly over the 24 GB ceiling. This Modelfile offloads 44 of 54 layers to GPU (82%, ~20.7 GB on VRAM) and leaves ~4.7 GB in system RAM. Because the hybrid Mamba-2 architecture keeps the attention KV cache tiny (~90 MB at 16K with q4_0), total VRAM usage stays ~22–23 GB — comfortably inside the budget. MoE experts that aren’t activated stay memory-mapped, so only ~3B params are fetched per token.

QUICK START

1. Install Ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# macOS / Windows
# Download from https://ollama.com/download

Requires Ollama 0.32.9+ — check with ollama --version. On Debian/Ubuntu you may need to update the repo or install the newer .deb/binaries manually.

2. Set Environment Variables (Recommended)

Set these before starting Ollama to minimize memory usage:

# KV Cache at q4_0 — much smaller footprint than default f16
export OLLAMA_KV_CACHE_TYPE=q4_0

# IMPORTANT: do NOT set OLLAMA_SCHED_SPREAD=1 (leave default OFF).
#   Spread mode on a 16GB+8GB pair plans a ~9.8GB contiguous tensor block
#   for the 8GB card -> cudaMalloc OOM at load. With spread OFF, Ollama packs
#   the 16GB card first (verified working split: 14.3GB + 7.8GB GPU / 18% CPU).

# Limit to one request at a time (memory constraint)
export OLLAMA_NUM_PARALLEL=1
export OLLAMA_MAX_LOADED_MODELS=1

# Optional: force the CUDA backend explicitly
# export OLLAMA_LLM_LIBRARY=cuda_v13

# Optional: faster attention on compatible GPUs
# export OLLAMA_FLASH_ATTENTION=1

Permanent setup (Linux systemd):

sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_KV_CACHE_TYPE=q4_0"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MAX_LOADED_MODELS=1"

Then:

sudo systemctl daemon-reload
sudo systemctl restart ollama

3. Pull & Run

# Pull the model (downloads ~25 GB — fast, blob is the official-library digest)
ollama pull oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU

# Run interactively
ollama run oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU

# Single prompt
ollama run oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU \
  "Write a Python function to fetch and parse a JSON API"

USAGE

CLI

# Interactive chat
ollama run oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU

# With a system prompt
ollama run oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU \
  --system "You are a rigorous code reviewer"

REST API

# Chat completion (thinking + answer)
curl -s http://127.0.0.1:11434/api/chat \
  -d '{
    "model": "oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU",
    "messages": [{"role": "user", "content": "What is 17 * 23?"}],
    "stream": false
  }'
# Reasoning arrives in "message.thinking", the answer in "message.content"

# Generate (completion)
curl -s http://127.0.0.1:11434/api/generate \
  -d '{
    "model": "oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU",
    "prompt": "List 5 uses for a local LLM agent",
    "stream": false,
    "options": { "num_ctx": 8192, "num_predict": 512 }
  }'

# OpenAI-compatible endpoint
curl -s http://127.0.0.1:11434/v1/chat/completions \
  -d '{
    "model": "oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'

# Disable thinking for low-latency agents
curl -s http://127.0.0.1:11434/api/chat \
  -d '{
    "model": "oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU",
    "messages": [{"role": "user", "content": "Summarize this in one line."}],
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Python (ollama library)

pip install ollama

import ollama

# Chat — reasoning lands in response.message.thinking, answer in .content
response = ollama.chat(
    model='oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU',
    messages=[{'role': 'user', 'content': 'Debug this SQL query'}],
)
print(response.message.content)

# Generate
response = ollama.generate(
    model='oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU',
    prompt='Write a Python function to sort a list',
)
print(response.response)

# Disable thinking for low latency
response = ollama.chat(
    model='oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU',
    messages=[{'role': 'user', 'content': 'Say hello'}],
    chat_template_kwargs={'enable_thinking': False},
)

JavaScript (ollama.js)

npm install ollama

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU',
  messages: [{ role: 'user', content: 'Hello!' }],
})
console.log(response.message.content)

SAMPLING PARAMETERS

These are baked into the model via its Modelfile:

MEMORY & PERFORMANCE

VRAM Budget

⚠️ On dual-GPU, account for ~1 GB of desktop baseline VRAM (X11/Wayland, browser).

Context Window Scaling (q4_0 KV — hybrid arch keeps it tiny)

Performance (verified: RTX 5060 Ti 16 GB + RTX 4060 Ti 8 GB, Ollama 0.32.9)

• Prompt processing: ~1540 tok/s (warm)
• Text generation: ~28–38 tok/s (only ~3B active params per token)
• GPU utilization: both GPUs active during inference (~19–21%)
• Model load time: ~1–2 min (25 GB cold load)

The MoE architecture activates only 6 of 129 experts per token, so generation stays surprisingly fast for a 30B-parameter model.

MONITORING

# Check model is GPU-resident (if CPU% is high, reduce context)
ollama ps

# Real-time GPU usage (NVIDIA)
watch -n 1 nvidia-smi

# GPU memory query
nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv

# Ollama server logs (Linux)
journalctl -u ollama -n 50 --no-pager

TROUBLESHOOTING

Opencode Integration

Add to ~/.config/opencode/opencode.jsonc:

"oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU": {
  "name": "oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU",
  "options": {
    "supportsThinking": true,
    "contextWindow": 16384
  }
}

Use as agent:

"model": "ollama/oamazonasgabriel/nemotron-3.5-lightning:q4-24gbGPU"

Tool calls use the qwen3_coder <tool_call> XML parser, so OpenCode/agent harnesses parse them natively. You can also launch it directly:

> ollama launch opencode --model nemotron-3.5-lightning
> ```

---

## ALTERNATIVE MODELS

```bash
# Fully-GPU variant for 24 GB single cards (Q3_K_M, ~19.9 GB — community GGUF)
ollama pull hf.co/ijohn07/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-Q3_K_M-GGUF:Q3_K_M

# Official higher-quality quants (need 48 GB+ VRAM)
ollama pull nemotron-3.5-lightning:q8_0    # 35 GB
ollama pull nemotron-3.5-lightning:bf16    # 66 GB

# Apple Silicon (23 GB, 256K context)
ollama pull nemotron-3.5-lightning:30b-mlx

# Similar-class MoE tuned for the same 24 GB setup
ollama pull oamazonasgabriel/qwen3.6-35b-a3b:q4-24gbGPU

CREDITS

LINKS