G2-nano-instruct

A 60.03M-parameter GPT trained from scratch on a single 8GB-RAM NVIDIA Jetson Orin Nano, then instruction-tuned on a 60k English mix.

Overview

G2-nano-instruct is the instruction-tuned version of G2-nano. It is a 60.03M-parameter decoder-only causal language model trained from scratch under an 8GB unified-memory budget (Jetson Orin Nano). Same architecture as G1-nano; pretraining used ~3.00B tokens (2× G1).

Lab record: more pretraining tokens, then an SFT mix that did not improve the canonical QCM suite. No 20-prompt qualitative set was scored.

Model variants

The raw pretrained version of the same model is available as azerdsq/g2-nano-base.

What this version adds

Compared with azerdsq/g2-nano-base, this checkpoint adds supervised instruction fine-tuning and chat tokens (<|user|> / <|assistant|> / <|end|> / <|system|>).

SFT was 3 epochs at sequence length 1024 on a 60k English mix. On lm-eval 0.4.11, mean_7 is 40.16% versus 42.13% for the base (−1.97 pt). Versus G1-nano-instruct, mean_6 is 42.28% vs 43.74% (−1.46 pt).

Architecture

Same as G1-nano / G2-nano-base:

• Parameters: 60.03M
• Layers: 14
• Hidden size: 576
• Attention: GQA, 9 query heads / 1 KV head, head_dim 64
• Position encoding: RoPE (θ=10000)
• Feed-forward network: SwiGLU, hidden 1664
• Normalization: RMSNorm
• Context length: 2048 tokens native; this SFT run used 1024
• Vocabulary: 16,388 (16,384 SentencePiece + 4 chat tokens)

Training

• Pretraining data: 3,001,842,523 training tokens
• Sources: FineWeb-Edu 66% / OpenWebText 15% / PG-19 7.5% / Wikipedia EN 5% / BookCorpus 5% / WikiHow 1.5% (capped)
• Instruction tuning: 60k English conversations (Claude 4.6, Qwen2.5-72B Magpie, SmolTalk2 Magpie, Llama-3.1-70B everyday, UltraChat, OpenHermes 2.5, Llama 3.3 70B Magpie)
• Conversation format: mixed single-turn and multi-turn
• Training hardware: NVIDIA Jetson Orin Nano (8GB unified memory)

Usage

ollama run azerdsq/g2-nano-instruct

The chat template is baked in. Prefer azerdsq/g1-nano-instruct for chat.

Weights are also available on Hugging Face.

Benchmarks

Limitations

• SFT recedes on the canonical suite; do not prefer this checkpoint over g1-nano-instruct for chat.
• 60M parameters cap factual retention.
• 2048-token native context; this SFT run used 1024.
• English only.
• Single-sequence generation only (no padded batched inference).

License

Apache 2.0.

Open weights: model weights, tokenizer and inference code (trust_remote_code). Training code, data pipelines and intermediate checkpoints are not included.