MrScratchcat22/IQuest-Coder-V1:Q8_0

MrScratchcat22/ IQuest-Coder-V1:Q8_0

941 Downloads Updated 1 month ago

https://huggingface.co/mradermacher/IQuest-Coder-V1-40B-Instruct-GGUF

tools

ollama run MrScratchcat22/IQuest-Coder-V1:Q8_0

curl http://localhost:11434/api/chat \
  -d '{
    "model": "MrScratchcat22/IQuest-Coder-V1:Q8_0",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from ollama import chat

response = chat(
    model='MrScratchcat22/IQuest-Coder-V1:Q8_0',
    messages=[{'role': 'user', 'content': 'Hello!'}],
)
print(response.message.content)

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'MrScratchcat22/IQuest-Coder-V1:Q8_0',
  messages: [{role: 'user', content: 'Hello!'}],
})
console.log(response.message.content)

Details

Updated 1 month ago

1 month ago

6d24244c2c24 · 42GB ·

model

archllama

·

parameters39.8B

·

quantizationQ8_0

42GB

template

{{- if .Tools }}<|im_start|>system {{- if .System }} {{ .System }} {{- else }} You are LoopCoder, a

1.4kB

params

{ "repeat_penalty": 1, "stop": [ "<|im_start|>", "<|im_end|>", "<|en

146B

Readme

base_model: IQuestLab/IQuest-Coder-V1-40B-Instruct language: - en library_name: transformers license: other license_link: https://huggingface.co/IQuestLab/IQuest-Coder-V1-40B-Instruct/blob/main/LICENSE license_name: iquestcoder mradermacher: readme_rev: 1

quantized_by: mradermacher

About

static quants of https://huggingface.co/IQuestLab/IQuest-Coder-V1-40B-Instruct

For a convenient overview and download list, visit our model page for this model.

weighted/imatrix quants are available at https://huggingface.co/mradermacher/IQuest-Coder-V1-40B-Instruct-i1-GGUF

Usage

If you are unsure how to use GGUF files, refer to one of TheBloke’s READMEs for more details, including on how to concatenate multi-part files.

Provided Quants

(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)

Link	Type	Size/GB	Notes
GGUF	Q2_K	14.9
GGUF	Q3_K_L	20.9
GGUF	Q4_K_M	24.1	fast, recommended
GGUF	Q5_K_M	28.3
GGUF	Q6_K	32.7	very good quality
GGUF	Q8_0	42.4	fast, best quality

Here is a handy graph by ikawrakow comparing some lower-quality quant types (lower is better):

And here are Artefact2’s thoughts on the matter: https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9

FAQ / Model Request

See https://huggingface.co/mradermacher/model_requests for some answers to questions you might have and/or if you want some other model quantized.

Thanks

I thank my company, nethype GmbH, for letting me use its servers and providing upgrades to my workstation to enable this work in my free time.

---
base_model: IQuestLab/IQuest-Coder-V1-40B-Instruct
language:
- en
library_name: transformers
license: other
license_link: https://huggingface.co/IQuestLab/IQuest-Coder-V1-40B-Instruct/blob/main/LICENSE
license_name: iquestcoder
mradermacher:
  readme_rev: 1
quantized_by: mradermacher
---
## About

static quants of https://huggingface.co/IQuestLab/IQuest-Coder-V1-40B-Instruct

***For a convenient overview and download list, visit our [model page for this model](https://hf.tst.eu/model#IQuest-Coder-V1-40B-Instruct-GGUF).***

weighted/imatrix quants are available at https://huggingface.co/mradermacher/IQuest-Coder-V1-40B-Instruct-i1-GGUF
## Usage

If you are unsure how to use GGUF files, refer to one of [TheBloke's
READMEs](https://huggingface.co/TheBloke/KafkaLM-70B-German-V0.1-GGUF) for
more details, including on how to concatenate multi-part files.

## Provided Quants

(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)

| Link | Type | Size/GB | Notes |
|:-----|:-----|--------:|:------|
| [GGUF](https://huggingface.co/mradermacher/IQuest-Coder-V1-40B-Instruct-GGUF/resolve/main/IQuest-Coder-V1-40B-Instruct.Q2_K.gguf) | Q2_K | 14.9 |  |
| [GGUF](https://huggingface.co/mradermacher/IQuest-Coder-V1-40B-Instruct-GGUF/resolve/main/IQuest-Coder-V1-40B-Instruct.Q3_K_L.gguf) | Q3_K_L | 20.9 |  |
| [GGUF](https://huggingface.co/mradermacher/IQuest-Coder-V1-40B-Instruct-GGUF/resolve/main/IQuest-Coder-V1-40B-Instruct.Q4_K_M.gguf) | Q4_K_M | 24.1 | fast, recommended |
| [GGUF](https://huggingface.co/mradermacher/IQuest-Coder-V1-40B-Instruct-GGUF/resolve/main/IQuest-Coder-V1-40B-Instruct.Q5_K_M.gguf) | Q5_K_M | 28.3 |  |
| [GGUF](https://huggingface.co/mradermacher/IQuest-Coder-V1-40B-Instruct-GGUF/resolve/main/IQuest-Coder-V1-40B-Instruct.Q6_K.gguf) | Q6_K | 32.7 | very good quality |
| [GGUF](https://huggingface.co/mradermacher/IQuest-Coder-V1-40B-Instruct-GGUF/resolve/main/IQuest-Coder-V1-40B-Instruct.Q8_0.gguf) | Q8_0 | 42.4 | fast, best quality |

Here is a handy graph by ikawrakow comparing some lower-quality quant
types (lower is better):

![image.png](https://www.nethype.de/huggingface_embed/quantpplgraph.png)

And here are Artefact2's thoughts on the matter:
https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9

## FAQ / Model Request

See https://huggingface.co/mradermacher/model_requests for some answers to
questions you might have and/or if you want some other model quantized.

## Thanks

I thank my company, [nethype GmbH](https://www.nethype.de/), for letting
me use its servers and providing upgrades to my workstation to enable
this work in my free time.

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)