koesn/mistral-7b-instruct:Q6

koesn/

mistral-7b-instruct:Q6_K

2,168 Downloads Updated 1 year ago

Fixed num_ctx to 32768. This Mistral 7B v0.2 Instruct model is ready to use for full model's 32k contexts window.

Updated 1 year ago

1 year ago

6e0226662051 · 5.9GB ·

archllama

parameters7.24B

quantizationQ6_K

5.9GB

Perform the task to the best of your ability.

47B

{ "num_ctx": 32768, "stop": [ "<|im_start|>", "<|im_end|>" ] }

75B

<|im_start|>system {{ .System }} <|im_start|>user {{ .Prompt }} <|im_start|>assistant

87B

Mistral-7B-Instruct-v0.2

Model Quants	Size	Bit	Perplexity
mistral-7b-instruct:Q4_0	4.1GB	4	+0.2166 ppl
mistral-7b-instruct:Q4_K_M	4.4GB	4	+0.0532 ppl
mistral-7b-instruct:Q5_K_M	5.1GB	5	+0.0122 ppl
mistral-7b-instruct:Q6_K	5.9GB	6	+0.0008 ppl

“max_position_embeddings” : 32768
“rope_theta” : 1000000.0
“vocab_size” : 32000