G0-nano-base

A 62M-parameter GPT trained completely from scratch on a single 8GB-RAM NVIDIA Jetson device, without cloud infrastructure or multi-GPU setups. Base pretrained checkpoint for raw text completion, not instruction following.

Overview

G0-nano-base is a 62.1M, with embeddings shared with the language-model head-parameter decoder-only causal language model trained from scratch under a single 8GB-RAM NVIDIA Jetson.

The project focuses on making a complete model training workflow — tokenizer, pretraining, fine-tuning infrastructure and export — work on modest hardware.

This is the base checkpoint. It predicts the next token and completes text; it is not a chat model and should not be expected to follow instructions.

Model variants

The instruction-tuned version of the same model is available as azerdsq/g0-nano-instruct.

What this version adds

This is the pretrained foundation of the G0 Nano model line. It does not include supervised instruction fine-tuning or a chat format.

Architecture

Llama-style decoder-only Transformer:

• Parameters: 62.1M, with embeddings shared with the language-model head
• Layers: 12
• Hidden size: 640
• Attention: Grouped-Query Attention, 10 query heads / 2 key-value heads, head dimension 64
• Position encoding: RoPE, θ=10000
• Feed-forward network: SwiGLU, hidden dimension 1728
• Normalization: RMSNorm
• Context length: 1024 tokens
• Vocabulary: 16,384 SentencePiece tokens

Training

• Pretraining data: approximately 1.5B tokens of English web and book text
• Sources: FineWeb-Edu, BookCorpus, OpenWebText, PG-19 and WikiHow
• Training hardware: a single NVIDIA Jetson with 8GB of unified memory

Usage

ollama run azerdsq/g0-nano-base "The city of Paris is"

This is a base model: it completes text rather than answering questions.

Weights are also available on Hugging Face.

Limitations

• 62M parameters impose a hard limit on factual knowledge; expect fluent but frequently incorrect completions on knowledge-intensive prompts.
• Maximum context length is 1024 tokens.
• English-only training data.
• Single-sequence generation only; padded batched inference is not supported by the custom model code.
• No instruction tuning and no chat format.

License

Apache 2.0. This release contains model weights and the code required to load them; it does not include the training data or private training infrastructure.