A bitnet-style LLM that runs on consumer hardware

21b

ollama run reecdev/bytenet:21b

curl http://localhost:11434/api/chat \
  -d '{
    "model": "reecdev/bytenet:21b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from ollama import chat

response = chat(
    model='reecdev/bytenet:21b',
    messages=[{'role': 'user', 'content': 'Hello!'}],
)
print(response.message.content)

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'reecdev/bytenet:21b',
  messages: [{role: 'user', content: 'Hello!'}],
})
console.log(response.message.content)

Details

Updated 3 weeks ago

3 weeks ago

1c4fbe057bc0 · 4.5GB ·

model

archernie4_5-moe

parameters21.8B

quantizationIQ1_S

4.5GB

template

{{ if .System }}<|begin_of_sentence|>{{ .System }}{{ end }} {{ if .Prompt }}User: {{ .Prompt }}{{ en

131B

system

You are ByteNet, a 1.6-bit 21 billion parameter Large Language Model.

69B

params

{ "num_ctx": 131072, "repeat_penalty": 1.1, "stop": [ "<|begin_of_sentence|>",

166B

Readme

ByteNet

ByteNet is a 21 billion parameter LLM based on ERNIE 4.5 that can run on consumer GPUs with less than 6 GB of VRAM using aggressive compression techniques.

Quick Start

Pull ByteNet and run on GPU (min vram: 5 GB)

ollama run reecdev/bytenet