koesn/llama3-8b-instruct:Q4_K

koesn/

llama3-8b-instruct:Q4_K_M

1,527 Downloads Updated 1 year ago

Fixed num_ctx to 8192 and eos token. This Llama 3 8B Instruct model is ready to use for full model's 8k contexts window.

Updated 1 year ago

1 year ago

aae5b523ef30 · 4.9GB ·

archllama

parameters8.03B

quantizationQ4_K_M

4.9GB

You are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests t

129B

{ "num_ctx": 8192, "num_keep": 24, "stop": [ "<|start_header_id|>", "<|e

129B

{{ if .System }}<|start_header_id|>system<|end_header_id|> {{ .System }}<|eot_id|>{{ end }}{{ if .Pr

257B

Meta-Llama-3-8B-Instruct

Model Quants	Size	Bit	Perplexity
llama3-8b-instruct:Q4_0	4.7GB	4	+0.2166 ppl
llama3-8b-instruct:Q4_K_M	4.9GB	4	+0.0532 ppl
llama3-8b-instruct:Q5_K_M	5.7GB	5	+0.0122 ppl
llama3-8b-instruct:Q6_K	6.6GB	6	+0.0008 ppl

“max_position_embeddings” : 8192
“rope_theta” : 500000.0
“vocab_size” : 128256