Microsoft Phi4 pulled from HuggingFace and quantized to Q8

ollama run did100/phi4_Q8

curl http://localhost:11434/api/chat \
  -d '{
    "model": "did100/phi4_Q8",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from ollama import chat

response = chat(
    model='did100/phi4_Q8',
    messages=[{'role': 'user', 'content': 'Hello!'}],
)
print(response.message.content)

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'did100/phi4_Q8',
  messages: [{role: 'user', content: 'Hello!'}],
})
console.log(response.message.content)

Details

Updated 1 year ago

1 year ago

d04922f21d98 · 16GB ·

model

archphi3

parameters14.7B

quantizationQ8_0

16GB

license

Microsoft. Copyright (c) Microsoft Corporation. MIT License Permission is hereby granted, free of ch

1.1kB

params

{ "stop": [ "<|im_start|>", "<|im_end|>", "<|im_sep|>" ] }

82B

template

{{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1 -}} <|im_start|>{{ .R

283B

Readme

Phi4 with Q8 quantization

This is Phi4 from Microsoft with a Q8 quantization. The ollama version is Q4_K_M.

This model was built doing the following:

download of the official HuggingFace model
gguf conversion
quantization from BF16 to Q8
ollama import with the same model file than the official ollama Phi4

With a ctx-size of 3200, the official model is taking around 10GB of VRAM while this Q8 version takes 16GB.