341M-parameter English chat model (32 layers, GPT-2 tokenizer) trained end-to-end on a single consumer AMD GPU (RX 6750 XT, 12 GB) with a hand-written backward engine PyTorch autograd is unstable on this hardware for deep transformers.

Details

Updated 1 month ago

1 month ago

2a37387b9390 · 683MB ·

model

archllama

parameters341M

quantizationF16

683MB

template

{{- if .System }}{{ .System }} {{ end }} {{- if .Tools }}You can call these tools when useful: {{ ra

577B

system

You are AILO, a small 341M assistant. - Answer conversationally and stay on topic. - If a passage, s

249B

params

{ "num_predict": 200, "repeat_penalty": 1.25, "stop": [ "<|user|>", "<|a

180B

AILO-340M-v4 🪶

A 341M-parameter chat model trained end-to-end on one consumer AMD GPU (RX 6750 XT, 12 GB) — with a hand-written backward engine, because PyTorch autograd breaks on this hardware for deep transformers.

Distilled from GPT-J 6B (logit-level), then taught to converse by Gemma 3 4B.

At 215 MB in q4_k_m it fits a Raspberry Pi Zero 2 W (512 MB RAM) and runs at a few tokens/second.

Run

ollama run Alieno/ailo-340m-v4            # q8_0 — recommended
ollama run Alieno/ailo-340m-v4:q4_k_m     # smallest, for SBCs

Tag	Size	Notes
`latest` / `f16`	683 MB	full precision
`q8_0`	363 MB	recommended — near-lossless
`q4_k_m`	215 MB	fits 512 MB boards

What it does well

💬 Conversation — fluent, on-topic, natural register
📖 Reading a passage — give it a paragraph and it pulls out the right fact, numbers and dates included (“330 metres tall”, “7 November 1867”). Pair it with search or your own documents.
🎯 Output constraints — “summarise in exactly five words” actually gives five words
🐍 Basic Python shape — plausible functions, often with bugs

What it is NOT good at — please read

Arithmetic is wrong. Exact-match on unseen operands: 0%. It will tell you 25 + 17 = 51 with confidence.
Multi-step reasoning fails — word problems, number sequences.
Facts from memory are invented — wrong dates, wrong nationalities. Supply the facts in the prompt and it handles them correctly.

The gap to Qwen2.5-0.5B isn’t the architecture: that model saw ~18 trillion pre-training tokens, AILO’s base comes from distillation over a corpus smaller by many orders of magnitude. Fine-tuning shapes behaviour; it doesn’t replace pre-training scale.

Treat it as a small local reader and conversationalist — with retrieval for facts and a calculator for maths. Not a source of truth.

Chat format

<|user|>
{your question}
<|assistant|>

Suggested: temperature 0.3, top_k 20, top_p 0.9, repeat_penalty 1.25

License

CC BY-NC-SA 4.0 (non-commercial). Distilled from GPT-J (Apache-2.0) and Gemma 3 (Gemma Terms of Use) — downstream use must respect the teachers’ terms.