reecdev/tiny-oss

reecdev/ tiny-oss:latest

331 Downloads Updated 2 months ago

A variant of gpt-oss designed for consumer hardware

tools thinking e3b

ollama run reecdev/tiny-oss

curl http://localhost:11434/api/chat \
  -d '{
    "model": "reecdev/tiny-oss",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from ollama import chat

response = chat(
    model='reecdev/tiny-oss',
    messages=[{'role': 'user', 'content': 'Hello!'}],
)
print(response.message.content)

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'reecdev/tiny-oss',
  messages: [{role: 'user', content: 'Hello!'}],
})
console.log(response.message.content)

Details

Updated 2 months ago

2 months ago

ca82fcbfd714 · 3.6GB ·

model

archgpt-oss

parameters5.98B

quantizationQ3_K_S

3.6GB

license

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR US

11kB

params

{ "temperature": 1 }

18B

template

<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI. Knowledge cutof

7.2kB

Readme

tiny-oss

tiny-oss is a derivative of gpt-oss designed for consumer hardware. With 6 billion total parameters compressed, it is able to run on consumer GPUs with less than 4 GB of VRAM!

Why?

The original gpt-oss is too heavy for most consumer devices. tiny-oss is a derivative of gpt-oss that uses compression techniques such as expert pruning and optimized quantization to compress gpt-oss into only 6 billion parameters.

Quick Start

Run tiny-oss with ollama:

ollama run reecdev/tiny-oss

Compare models:

ollama run reecdev/tiny-oss --verbose Solve 1 + 1. && ollama run gpt-oss:20b --verbose Solve 1 + 1.