second_constantine/
yandex-gpt-5-lite:8b

107 2 months ago

Instruct version of the large language model YandexGPT 5 Lite with 8B parameters with a context length of 32k tokens. (quantised version of Q5_K_M)

8b

2 months ago

903e8a36ef14 · 5.7GB ·

model
llama
·
8.04B
·
Q5_K_M
template
<s> Ассистент:[SEP]{{- range .Messages }}{{- if eq .Role "user" }}Response }} Пользо
params
{ "stop": [ "<s>", "[SEP]", "Response }}\n\n Пользователь:"

Readme

Based on https://huggingface.co/mradermacher/YandexGPT-5-Lite-8B-instruct-GGUF

Feature Value
vision false
thinking false
tools false
Device Speed, token/s Context VRAM, gb Versions
RTX 3090 24gb ~105 4096 6.9 Q5_K_M,0.12.2
RTX 3090 24gb ~105 15360 9.2 Q5_K_M,0.12.2
RTX 2080ti 11gb ~74 4096 6.9 Q5_K_M,0.12.2
RTX 2080ti 11gb ~75 15360 9.2 Q5_K_M,0.12.2
M1 Max 32gb ~41 4096 6.6 Q5_K_M,0.12.2
M1 Max 32gb ~41 15360 8.2 Q5_K_M,0.12.2
RTX 3070ti Mobile 8gb ~60 4096 6.9 Q5_K_M, 0.12.3
RTX 3070ti Mobile 8gb ~23 15360 9.2 (14%/86% CPU/GPU) Q5_K_M, 0.12.3