G2-nano-base

A 60.03M-parameter GPT trained from scratch on a single 8GB-RAM NVIDIA Jetson Orin Nano. Same architecture as G1-nano; new pretraining run (~3.00B tokens). This is the base checkpoint: it completes text.

Overview

G2-nano-base is a 60.03M-parameter decoder-only causal language model trained from scratch under an 8GB unified-memory budget (Jetson Orin Nano).

Lab note, not a product win: 2× tokens, weak lift, likely a 60M ceiling on this shape. The ~1B G2 model is not this release.

This is the base checkpoint. It predicts the next token and completes text; it is not a chat model and should not be expected to follow instructions.

Model variants

The instruction-tuned version of the same model is available as azerdsq/g2-nano-instruct.

What this version adds

About 3.00B training tokens (~50 tokens/parameter) versus ~1.50B for G1-nano. On the canonical 6-task mean this base scores 44.59% versus 43.03% for G1-nano-base (+1.56 pt). PIQA, WinoGrande and ARC-Challenge stay within noise.

Architecture

Llama-style decoder-only transformer (same as G1-nano):

• Parameters: 60.03M
• Layers: 14
• Hidden size: 576
• Attention: GQA, 9 query heads / 1 KV head, head_dim 64
• Position encoding: RoPE (θ=10000)
• Feed-forward network: SwiGLU, hidden 1664
• Normalization: RMSNorm
• Context length: 2048 tokens
• Vocabulary: 16,384 (SentencePiece BPE, shared)

Training

• Pretraining data: 3,001,842,523 training tokens
• Sources: FineWeb-Edu 66% / OpenWebText 15% / PG-19 7.5% / Wikipedia EN 5% / BookCorpus 5% / WikiHow 1.5% (capped)
• Training hardware: NVIDIA Jetson Orin Nano (8GB unified memory), 22,903 steps

Usage

ollama run azerdsq/g2-nano-base "The city of Paris is"

This is a base model: it completes text rather than answering questions.

Weights are also available on Hugging Face.

Benchmarks

Limitations

• 60M parameters cap factual retention. Completions can be fluent and wrong.
• Doubling the token budget did not produce a decisive quality jump on this architecture.
• 2048-token context.
• English only.
• Single-sequence generation only (no padded batched inference).
• No instruction tuning and no chat format.

License

Apache 2.0.

Open weights: model weights, tokenizer and inference code (trust_remote_code). Training code, data pipelines and intermediate checkpoints are not included.