GPT-OSS Distills — 1B Experimental

An experimental 1.01 billion parameter language model trained on final answers generated by GPT-OSS 20B. The student uses a pruned TinyLlama architecture and is published on Ollama by treyleo16.

Model: treyleo16/gpt-oss-distills:1b-experimental

This release is a small response-distillation experiment. Testing found arithmetic contradictions, factual errors, repetition, and missed formatting instructions. Use it for experimentation and review its answers carefully.

Quick start

Install Ollama, then run:

ollama run treyleo16/gpt-oss-distills:1b-experimental

Ollama downloads the model on first use. To download it without starting a chat:

ollama pull treyleo16/gpt-oss-distills:1b-experimental

You can also provide a prompt directly:

ollama run treyleo16/gpt-oss-distills:1b-experimental "Explain what a Python list is."

Use the full :1b-experimental tag shown above.

Model details

Property	Value
Parameters	1,011,959,808
Student base	TinyLlama/TinyLlama-1.1B-Chat-v1.0
Architecture	Llama, 20 decoder layers
Teacher	`gpt-oss:20b`
Training method	Offline response distillation with LoRA
File format	GGUF
Quantization	Q4_K_M
GGUF size	619,208,800 bytes, approximately 619 MB
Configured context	2,048 tokens
License	Apache License 2.0, with attribution and notices

The student learns from GPT-OSS answers while retaining a TinyLlama-based architecture. Transfer of GPT-OSS reasoning quality, tool use, and broader capabilities has not been established. Runtime memory usage includes additional overhead beyond the model file size.

Local API

With Ollama running, send a chat request to its local API:

curl http://localhost:11434/api/chat \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "treyleo16/gpt-oss-distills:1b-experimental",
    "messages": [
      {"role": "user", "content": "Explain what a Python list is."}
    ],
    "stream": false
  }'

If the Ollama server is not already running, start ollama serve in another terminal.

Training

The experiment generated 168 synthetic teacher answers from original prompts, divided into 147 training examples and 21 held-out validation examples. Prompts covered writing, Python, mathematics, reasoning, science, web development, structured output, and practical questions. Teacher reasoning traces and private conversation history were excluded from the training data.

The training process:

Load TinyLlama-1.1B-Chat-v1.0 at revision fe8a4ea1ffedaf415f4da2f062534de366a451e6.
Remove original decoder layers 6 and 15, using zero-based indices, to produce the 20-layer student.
Train rank-16 LoRA adapters on teacher final answers for three epochs, with an initial learning rate of 0.00015.
Select epoch 2, which had the lowest validation loss.
Merge the selected adapters, export the weights, and quantize them to Q4_K_M for Ollama.

Evaluation

Validation measured assistant-token negative log likelihood against the 21 held-out teacher responses, covering 1,446 assistant tokens. Lower values indicate better prediction of those responses.

Checkpoint	Validation loss
Original TinyLlama 1.1B Chat	1.7375
Pruned 1.01B student before distillation	1.8366
Selected checkpoint: epoch 2	1.3719
Final training checkpoint: epoch 3	1.4073

The selected checkpoint reduced this loss by 25.3% relative to the pruned baseline. The validation set was also used for checkpoint selection. These results apply to a small synthetic dataset; broader accuracy and equivalence to GPT-OSS 20B have not been demonstrated.

Qualitative checks found:

Correct Python function logic and valid requested JSON in some samples.
A correct initial arithmetic answer followed by contradictory numbers.
Factual errors in an explanation of lightning and thunder.
Repetition and failures to follow requested sentence counts.
Incorrect example-output comments in otherwise valid generated Python.

The quantized model loaded and generated responses in Ollama. Its public registry model checksum matched the tested local GGUF.

Offline backups

The experiment also provides these backup files:

gpt-oss-distills-checkpoint.zip: training source, teacher data, metrics, evaluation notes, selected adapter, and resumable training state.
gpt-oss-distills-model-part1.zip: first half of the GGUF, with restoration instructions and metadata.
gpt-oss-distills-model-part2.zip: second half of the GGUF, with restoration instructions and metadata.

To restore the model, extract both model-part ZIPs into the same folder. From that folder, use Python 3.11 or newer to run:

python3 restore_model.py
ollama create treyleo16/gpt-oss-distills:1b-experimental -f Modelfile
ollama run treyleo16/gpt-oss-distills:1b-experimental

The restoration script joins the parts and verifies the model’s SHA-256 checksum. Allow approximately 1.3 GB for the extracted parts and assembled GGUF, plus space for the ZIP downloads and Ollama’s imported copy.

The training checkpoint’s adapter applies to the recorded 20-layer pruned student. Recreate that pruning before loading the adapter. The checkpoint includes instructions for restoring the pinned base weights and resuming training.

Model checksum

File: gpt-oss-distills-1b-experimental-Q4_K_M.gguf

c4ce7153ee8c1862fa3e9e3f90f6b2b7590303650a401617752b63e3cbb00595

License and attribution

This release includes the Apache License 2.0 text and attribution notices for its TinyLlama student base. Preserve the included license and applicable notices when redistributing it.

Student base: TinyLlama-1.1B-Chat-v1.0
Teacher: GPT-OSS 20B on Ollama
Distribution: GPT-OSS Distills on Ollama

GPT-OSS Distills is an independent experimental release. It is not an official OpenAI or TinyLlama release.

Distills of GPT Oss

Models

Readme