A small chat model written entirely in assembly, trained from scratch on one home graphics card. Outputs are not the best, but considering size and budget they're very good (also sometimes hilarious).

Details

Updated 1 week ago

1 week ago

86297f96948a · 85MB

model

archllama

parameters126M

quantizationQ4_0

85MB

template

{{- range .Messages }}<|{{ .Role }}|>{{ .Content }}<|end|>{{ end }}<|assistant|>

80B

params

{ "num_ctx": 1024, "stop": [ "<|end|>" ], "temperature": 0.7, "top_k": 6

87B

Mnemonic

A small chat model written entirely in assembly, trained from scratch on one home graphics card.

Everything that made this model is hand-written x86-64 assembly (NASM) and hand-written PTX for the GPU: downloading and parsing the data, training the tokenizer, pretraining, chat fine-tuning, quantization, and even the tool that wrote these GGUF files. There’s no Python, no PyTorch and no CUDA toolkit anywhere in the pipeline, only the Windows API and the NVIDIA driver.

ollama run supergoatscriptguy/mnemonic

Tag	Model	Format	Size	Notes
`latest`, `300m`	300M	Q8_0	324 MB	the default, about the same quality as full precision
`300m-q4_0`	300M	Q4_0	189 MB	smaller and faster, slightly worse
`300m-f16`	300M	F16	610 MB	16-bit, for reference
`126m`	126M	Q8_0	135 MB	the first version
`126m-q4_0`	126M	Q4_0	85 MB
`126m-f16`	126M	F16	253 MB

What to expect

Mnemonic is small: 304 million parameters for the current version, a few hundred times smaller than the models people usually chat with. It’s good at:

friendly conversation
simple explanations and how-tos
short lists, tips, poems and small code snippets

It isn’t reliable for facts, math or long answers. It will confidently make things up, so don’t trust its dates and numbers. It works best one question at a time; start a new chat when you change topic.

The 300M is a clear step up from the 126M: it gets more facts right, formats its answers better and rambles less.

The models

	300M	126M
Architecture	Llama-style decoder: RoPE, RMSNorm, SwiGLU, grouped-query attention, tied embeddings, no biases	same
Parameters	304M	126M
Layers	24	16
Width	1024, 16 query heads, 4 key/value heads	768, 12 query heads, 4 key/value heads
Feed-forward	2816	2048
Context	2048 tokens	1024 tokens
Vocabulary	32,768, byte-level BPE trained from scratch	same

How it was trained

Both models were trained on a single RTX 5070 Ti at home.

Pretraining on FineWeb-Edu:
- 300M: 10 billion tokens in about 90 hours, at about 32K tokens/s (68 TFLOPS). Final validation loss: 2.84.
- 126M: 5 billion tokens in 19 hours. Final validation loss: 3.10.
Chat fine-tuning on smol-smoltalk, with the loss on the assistant’s replies only. For the 300M it was one pass of 389M tokens at 2048 context, in 3.4 hours. Long conversations were cut only at whole turns, so every reply it learned from ends properly.
Identity: about 2,100 hand-written conversations mixed into the fine-tuning, so it knows its name, who made it and how it was built.

The GPU side uses tensor-core matmuls (mma.sync, bf16 with fp32 accumulation) and flash attention, all written in PTX. Every fast kernel is checked against a simple reference kernel, and the backward pass is checked with finite differences.

Chat format

<|bos|><|user|>Hi! Who are you?<|end|><|assistant|>Hi! I'm Mnemonic, a compact AI assistant built entirely in hand-written assembly. What can I help you with today?<|end|>

The template and stop token are already set up. The default sampling is temperature 0.7, top-p 0.9, top-k 64. For more predictable answers, lower the temperature:

/set parameter temperature 0.3

Good to know

Language: English only.
Numbers: Mnemonic’s own tokenizer splits digits into groups of one or two. llama.cpp has no exact equivalent, so numbers with 3 or more digits are split slightly differently here than in training. Everything else tokenizes exactly the same, and greedy output matches Mnemonic’s own engine word for word.