🧠 Nemotron-3-Nano-30B-A3B-AgentCoder-Q4-K-M

A Quantized GRPO/RLAIF Post-Trained Model for Multi-Turn Agent Trajectories and Tool Calling

• Author: Bruno Pistone
• License: NVIDIA Open Model License (inherited from the base model)
• Model: br1-pist/NVIDA-Nemotron-3-30B-Nano-AgentCoder

Model Description

Nemotron-3-Nano-30B-A3B-AgentCoder-Q4-K-M is a quantized post-trained version of NVIDIA’s Nemotron-3-Nano-30B-A3B, optimised for agentic behaviour in multi-turn, tool-calling conversations.

Unlike a supervised fine-tune, this model was trained with Group Relative Policy Optimization (GRPO) under Reinforcement Learning from AI Feedback (RLAIF): for every conversational decision point the policy samples a group of candidate continuations, an LLM judge scores each against a rubric, and the advantage is computed relative to the group mean — so no reward model and no value network are trained.

Highlights

• 🧰 Trained on decision points, not whole dialogues: each training row is a real conversation state plus the next assistant turn, so the reward targets the choice the agent actually has to make
• 📈 Mean judge reward improved +50.1 % over the run (0.412 → 0.618), with both single-sample quality and best-of-8 capability rising
• ✂️ Runaway generations largely eliminated: the share of over-long completions fell from 22.8 % to 4.4 %, and mean completion length dropped ~40 %
• ⚖️ DAPO loss with group-relative reward scaling, which removes the length bias of vanilla GRPO
• 🔬 Reward came from a rubric-scored LLM judge covering evidence gathering, tool selection, root-cause reasoning and verification
• 🪶 Parameter-efficient: 0.1435 % of weights trained (45.4 M of 31.6 B), so the base model’s general capability is preserved

🚀 Direct Use

• ✅ Multi-turn agent loops where the model must decide between answering, calling a tool, and reacting to tool output
• ✅ Tool / function calling in coding, data-management, and shell-execution contexts
• ✅ General reasoning and Q&A inherited from the base model

⚠️ Out-of-Scope Use

• ❌ Highly sensitive or confidential data
• ❌ Domains requiring expert-level specialisation or certification
• ❌ Tasks where full explainability is mandatory
• ❌ Autonomous execution of irreversible actions without human review

Tool calling

The chat template renders tools with <tool_call> / <function=…> / <parameter=…> tags. Pass the schemas through apply_chat_template(..., tools=[...]) rather than describing them in the system prompt, so the rendering matches what the model was trained on.

Serving notes

• Stop tokens: generation_config.json declares eos_token_id: [2, 11] — </s> and <|im_end|>. The tokenizer’s eos_token is <|im_end|> (id 11) and the chat template ends turns with it, while config.json still carries the base repo’s scalar eos_token_id: 2 (and pad_token_id: 0). Prefer the values in generation_config.json; dropping id 11 makes the model generate past the end of its answer.
• Reasoning traces: the model emits <think> blocks. nano_v3_reasoning_parser.py is included for servers that separate reasoning from the final answer.

🧠 Training Details

Phase 1 — Post-Training: RLAIF with Group Relative Policy Optimization (GRPO)

Reward model (RLAIF) - Judge: moonshotai/Kimi-K3 - Rubric-scored 0–10, normalised to 0–1; rewards evidence gathering, correct tool choice, root-cause reasoning and verification, and penalises speculation, wrong-tool calls and symptom patching - Unscorable completions (content filter, malformed verdict) are excluded from the group baseline rather than scored zero — 0.31 % of samples over the run - Judge critiques were logged per completion, giving a per-step audit trail of why a trajectory scored as it did

GRPO parameters - Generations per prompt (group size): 8 - Beta (KL coefficient): 0.01 - Loss type: dapo - Reward scaling: group (unit variance within each group) - Sampling temperature: 1.0 - Max completion length: 640 tokens (reference answers: p50 185, p95 456), truncated completions masked out of the loss

Training parameters - Learning rate: 4.0e-6, cosine schedule with a 40 % floor - Warmup steps: 46 - Per-device batch size: 2 × 16 GPUs × gradient accumulation 4 = 128 completions (16 prompts) per step - Epochs: 2 → 460 optimiser steps - Max grad norm: 0.5, weight decay 0.01 - LoRA: rank 64, alpha 128, dropout 0.1, applied to attention (q/k/v/o_proj), the Mamba input projection (in_proj) and the MoE shared expert (up_proj, down_proj) - Trainable parameters: 45,436,928 / 31,668,811,200 (0.1435 %) - Precision: bf16, FSDP full-shard with NemotronHBlock wrapping and activation checkpointing - Attention: kernels-community/flash-attn3 - Hardware: 2 × ml.p5.48xlarge (16 × H100 80 GB), 26 h 32 m of training (27 h billable end-to-end)

Note on LoRA targets. The 128 routed experts are 3-D stacked parameters and cannot be reached by LoRA, so only the shared expert’s up_proj/down_proj are adapted. out_proj and conv1d are rejected by PEFT ≥ 0.20.0 for Mamba architectures and are therefore excluded.

Note on the KL coefficient. An earlier run of this recipe used beta: 0.04 and a peak learning rate of 1.68e-6. Its policy KL settled at ~0.006 by step 4 and never moved again, the reward gain stalled halfway through, and the final improvement was only +6.1 %. Lowering beta to 0.01, raising the peak LR to 4.0e-6 and lifting the cosine floor from 10 % to 40 % is what produced the results below — the KL ceiling, not the data volume, was the binding constraint.

GRPO Data

• 3,686 training and 334 validation decision points, extracted from 800 source conversations (4,020 unique states before splitting)
• Every context/next-assistant pair in the winning trajectory becomes one row, with reasoning_content and tool_calls preserved alongside content so tool-calling turns are not flattened away
• Train/validation split is grouped by root request, giving 0 overlap between the two sets
• Domains: coding, general Q&A, data management, shell execution

Objective - Improve tool-selection accuracy and multi-turn trajectory quality - Reduce speculation and symptom patching in favour of evidence gathering

Outcome (measured on all 58,880 sampled completions) - Mean judge reward rose from 0.4121 (first 10 % of steps) to 0.6184 (final 10 %), +50.1 %, Welch t = 52.7, p < 10⁻³⁰⁰. A regression across every individual completion confirms the trend (slope +5.0 × 10⁻⁴ per step, r = 0.30) - Both capability and consistency improved. Best-of-8 sampling rose 0.648 → 0.741 (+14.5 %, p = 6 × 10⁻³⁵) and mean-of-8 rose 0.412 → 0.618 (+50.3 %, p = 2 × 10⁻¹²⁴). The larger share of the gain is consistency — the model became far more likely to produce a good trajectory on any single sample — but unlike the earlier low-LR run, peak achievable quality moved as well - The distribution shifted, not just the mean: median reward 0.350 → 0.650; the share of completions scoring below 0.25 collapsed from 26.3 % to 3.7 %, while the share above 0.70 grew from 15.5 % to 39.3 % - Runaway generations were trained away: over-long completions fell from 22.8 % to 4.4 % and mean completion length from ~1,430 to ~850 characters, with reward rising at the same time — concision, not truncation of useful content. Length remains the single strongest negative correlate of reward (r = −0.55) - Policy KL from the reference model rose from ~0.005 to ~0.047, roughly 7× the level the earlier over-anchored run reached, confirming the policy genuinely moved - Group signal stayed viable to the end: only 0.27 % of groups had zero within-group variance and 1.2 % of completions carried a negligible advantage. Within-group reward sd did narrow from 0.172 to 0.111 and token entropy from 0.83 to 0.66, which is the expected signature of a sharpening policy and the practical limit on adding further epochs at this group size

Scope of these numbers. All figures above are scored by the same LLM judge that produced the training reward, on the training prompts. They demonstrate that the optimisation worked; they are not a held-out benchmark. An independent evaluation on the 334 validation decision points is not included in this release.

🖥️ Technical Specifications

Model Architecture

Hybrid Mamba-2 / attention / MoE (NemotronHForCausalLM, model_type: nemotron_h).

🧭 Recommendations

• Serve with the stop-token list from generation_config.json; a single eos_token_id is not sufficient for this model
• Install mamba-ssm and causal-conv1d built against your PyTorch version, or expect a substantial throughput penalty
• Pass tool schemas through the chat template rather than the system prompt, matching the training-time rendering
• A gap remains between single-sample and best-of-8 quality (0.618 vs 0.741), so best-of-n sampling with a verifier still adds measurable headroom where latency allows
• Tool-use accuracy degrades as task complexity grows; keep a human in the loop for irreversible actions
• Avoid using this model for confidential or safety-critical applications