25  Downloads Updated  1 month ago

341M-parameter English chat model (32 layers, GPT-2 tokenizer) trained end-to-end on a single consumer AMD GPU (RX 6750 XT, 12 GB) with a hand-written backward engine PyTorch autograd is unstable on this hardware for deep transformers.