G1-nano-instruct

A 60M-parameter GPT pretrained and instruction-tuned entirely on a single 8GB-RAM NVIDIA Jetson device, without cloud infrastructure or multi-GPU setups. Chat-oriented checkpoint with a native 2048-token context and real multi-turn conversation support.

Overview

G1-nano-instruct is the instruction-tuned version of G1 Nano. It is a 60.0M-parameter decoder-only causal language model trained from scratch under a single 8GB-RAM NVIDIA Jetson.

The project focuses on making a complete model training workflow — tokenizer, pretraining, fine-tuning infrastructure and export — work on modest hardware.

Model variants

The raw pretrained version of the same model is available as azerdsq/g1-nano-base.

What this version adds

Compared with azerdsq/g1-nano-base, this checkpoint adds supervised instruction fine-tuning and a chat format and real multi-turn conversation support.

Compared with the G0 Nano design, the main differences are the native 2048-token context and the multi-turn fine-tuning recipe, while keeping a similar model size and hardware budget.

Architecture

Llama-style decoder-only Transformer:

• Parameters: 60.0M
• Layers: 14
• Hidden size: 576
• Attention: Grouped-Query Attention, 9 query heads / 1 key-value head, head dimension 64
• Position encoding: RoPE, θ=10000
• Feed-forward network: SwiGLU, hidden dimension 1664
• Normalization: RMSNorm
• Context length: 2048 tokens, used natively in pretraining and fine-tuning
• Vocabulary: 16,388 tokens: 16,384 SentencePiece tokens plus 4 chat tokens

Training

• Pretraining data: approximately 1.5B tokens of English web and book text
• Sources: FineWeb-Edu, BookCorpus, OpenWebText, PG-19 and WikiHow
• Instruction tuning: approximately 39K conversations from UltraChat, OpenAssistant/oasst1 and cleaned Alpaca
• Conversation format: multi-turn conversations, packed without cross-conversation leakage
• Training hardware: a single NVIDIA Jetson with 8GB of unified memory

Usage

ollama run azerdsq/g1-nano-instruct

The chat format is built into the model. Conversation history can be carried across turns.

Weights are also available on Hugging Face.

Benchmarks

Limitations

• 60M parameters impose a hard limit on factual knowledge; expect fluent but frequently incorrect answers on knowledge-intensive prompts.
• Maximum context length is 2048 tokens, which remains short compared with modern language models.
• English-only training data.
• Multi-turn support does not imply long-term memory or reliable conversation consistency.
• Single-sequence generation only; padded batched inference is not supported by the custom model code.

License

Apache 2.0. This release contains model weights and the code required to load them; it does not include the training data or private training infrastructure.