Qwen3.5-4B — Ollama Model (8 GB) FIM-Optimized

Model: oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU

---

DESCRIPTION

A balanced configuration variant of Qwen3.5-4B, planned for FIM (Fill-In-the-Middle) and Tool Calling in restrict capacity environments. Using the Q3_K_S GGUF quantization from HuggingFace (unsloth). At only ~2 GB, it fits com fortably on any 8 GB single GPU with headroom for 8K context — perfect for real-time code completions, inline suggestions, and lightweight agentic coding tasks.

The ideal lightweight coding companion for: RTX 4060 · RTX 5060 · RTX 3060 · GTX 1660 · Intel Arc · Apple Silicon M-series · any GPU with 4 GB+ VRAM

Key Features

• FIM-optimized: temperature 0.5, top_p 0.8, repeat_penalty 1.15 — tuned for precise fill-in-the-middle code completions
• Efficient size: Q3_K_S quantization (~2 GB) — excellent quality-to-size ratio for the 4B parameter model
• Full capability: 4B dense model (all parameters active per token) — no MoE routing, maximum quality per response
• Blazing fast: 100-300+ tok/s on modern GPUs
• 8K context: ideal for function bodies, class definitions, and single-file completions
• FIM tokens: native <|fim_prefix|>, <|fim_middle|>, <|fim_suffix|> support
• ChatML format: compatible with standard instruct-tuning pipelines
• Thinking mode: toggleable via enable_thinking API parameter
• Tool calling: native support via qwen3_coder parser
• Apache 2.0 license: free for commercial and personal use

Architecture

Property	Value
Architecture	Gated DeltaNet + Gated Attention hybrid (dense)
Total Parameters	4B (all active — not MoE)
Native Context	262,144 tokens
Extended Context	up to ~1,000,000 via YaRN
Modalities	Text only
Chat Format	ChatML (`<
FIM Tokens	<|fim_prefix|>, `<
Quantization	Q3_K_S
Model Size	~2 GB
License	Apache 2.0
Upstream	unsloth/Qwen3.5-4B-GGUF
Original Model	Qwen/Qwen3.5-4B

---

REQUIREMENTS

Resource	Minimum	Recommended
GPU Memory	2 GB VRAM	4 GB+ VRAM
System RAM	4 GB	8 GB
Disk Space	4 GB free	8 GB+ free
Ollama Version	0.30.6+	Latest

Platform support: - Any GPU (NVIDIA, AMD, Intel Arc, Apple Silicon) with 2 GB+ VRAM - CPU-only — runs acceptably on modern CPUs - Integrated GPU — Intel UHD, AMD Radeon Graphics (iGPU) all work

💡 Why 8 GB GPU? The model is only ~2 GB at Q3_K_S. An 8 GB card has 6 GB of headroom — you can run this alongside your browser, IDE, and other GPU workloads w ithout breaking a sweat.

---

QUICK START

1. Install Ollama

# macOS (Homebrew)
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download from https://ollama.com/download

2. Pull & Run

# Pull the model (downloads ~2 GB)
ollama pull oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU

# Run interactively
ollama run oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU

# Single code prompt
ollama run oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU \
  "Write a Python function that merges two sorted lists"

---

USAGE

CLI

# Interactive chat
ollama run oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU

# With system prompt
ollama run oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU \
  --system "You are a helpful coding assistant. Provide concise code examples."

# FIM-style prompt (inline)
ollama run oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU \
  "Complete this function:\ndef fibonacci(n):\n    "

REST API

# Chat completion
curl -s http://127.0.0.1:11434/api/chat \
  -d '{
    "model": "oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU",
    "messages": [{"role": "user", "content": "Write a binary search in Python"}]
  }'

# Generate (completion)
curl -s http://127.0.0.1:11434/api/generate \
  -d '{
    "model": "oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU",
    "prompt": "Explain what FIM means in code completion",
    "stream": false
  }'

# FIM Completion (Fill-in-the-Middle)
curl -s http://127.0.0.1:11434/api/generate \
  -d '{
    "model": "oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU",
    "prompt": "<|fim_prefix|>def hello():\n    <|fim_suffix|>\n    return False<|fim_middle|>",
    "stream": false,
    "options": {
      "temperature": 0.5,
      "top_p": 0.8,
      "repeat_penalty": 1.15
    }
  }'

# With thinking mode
curl -s http://127.0.0.1:11434/api/chat \
  -d '{
    "model": "oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU",
    "messages": [{"role": "user", "content": "Solve this leetcode problem: ..."}],
    "options": {"enable_thinking": true}
  }'

# OpenAI-compatible endpoint
curl -s http://127.0.0.1:11434/v1/chat/completions \
  -d '{
    "model": "oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU",
    "messages": [{"role": "user", "content": "Write a React component"}],
    "stream": false
  }'

FIM Token Reference

Token	Purpose
<|fim_prefix|>	Marks the beginning of the code before the hole
<|fim_suffix|>	Marks the code after the hole
<|fim_middle|>	Marks where the model should fill in
<|repo_name|>	Optional: repository/file context
<|file_sep|>	Optional: separator between files

Python (ollama library)

pip install ollama

import ollama

# Chat
response = ollama.chat(
    model='oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU',
    messages=[{'role': 'user', 'content': 'Write a JavaScript debounce function'}],
)
print(response.message.content)

# FIM Completion
response = ollama.generate(
    model='oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU',
    prompt='<|fim_prefix|>def hello():\n    <|fim_suffix|>\n    return False<|fim_middle|>',
    options={
        'temperature': 0.5,
        'top_p': 0.8,
        'repeat_penalty': 1.15,
    },
)
print(response.response)

# With thinking mode
response = ollama.chat(
    model='oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU',
    messages=[{'role': 'user', 'content': 'Debug this SQL query'}],
    options={'enable_thinking': True},
)

JavaScript (ollama.js)

npm install ollama

import ollama from 'ollama'

// Chat
const response = await ollama.chat({
  model: 'oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU',
  messages: [{ role: 'user', content: 'Write a CSS animation' }],
})
console.log(response.message.content)

---

SAMPLING PARAMETERS

These are baked into the model via its Modelfile:

Parameter	Value	FIM Rationale
num_ctx	8192	Enough context for function bodies and class definitions
num_gpu	99	Offload all layers to GPU
temperature	0.5	Balances coherence vs. diversity in code completions
top_p	0.8	Focused nucleus sampling for correct code
top_k	40	Reasonable token variety for code generation
min_p	0.0	Disabled; top_p already controls the nucleus
repeat_penalty	1.15	Prevents FIM loops and repetitive code blocks
stop	<\|im_start\|>, <\|im_end\|>	Chat template tokens
stop	<\|fim_prefix\|>, <\|fim_middle\|>, <\|fim_suffix\|>	FIM boundary tokens

Equivalent llama.cpp Command

./llama-cli \
  -m Qwen3.5-4B-Q3_K_S.gguf \
  -ngl 99 \
  -c 8192 \
  --temp 0.5 \
  --top-p 0.8 \
  --repeat-penalty 1.15 \
  -cnv

---

MEMORY & PERFORMANCE

VRAM Budget

Component	Size
Model weights (Q3_K_S)	~2 GB
KV cache (fp16, 8K context)	~2-3 GB
Ollama process overhead	~0.5 GB
Total	~4.5-5.5 GB ✅ plenty of headroom

Context Window Scaling

On any GPU with 4 GB+ VRAM:

Context	KV Cache	Fits 8 GB?	Notes
8,192	~2-3 GB	✅ Lots of headroom	~4.5-5.5 GB total
16,384	~4-6 GB	✅ Fits	~6-8 GB total
32,768	~8-12 GB	⚠️ May be tight	Consider q4_0 KV cache
65,536	~16-24 GB	❌	CPU offload required

Performance

Hardware	Prompt Processing	Text Generation
RTX 4060 (8 GB)	~1,000-3,000 tok/s	~100-300 tok/s
RTX 3060 (12 GB)	~800-2,000 tok/s	~80-250 tok/s
Intel Arc A770	~500-1,500 tok/s	~60-200 tok/s
Apple Silicon M1	~300-1,000 tok/s	~50-150 tok/s
CPU-only (modern)	~30-100 tok/s	~10-50 tok/s

---

Opencode Integration

Add to ~/.config/opencode/opencode.jsonc:

"oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU": {
  "name": "oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU",
  "options": {
    "supportsThinking": false,
    "contextWindow": 8192
  }
}

Use as agent:

"model": "ollama/oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU"

Or launch directly:

ollama launch opencode --model oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU

---

TROUBLESHOOTING

Symptom	Fix
Model not found	Run ollama pull oamazonasgabriel/qwen3.5-4b:q3_k_s-8gbGPU first
Slow on CPU	Normal for CPU; enable any GPU with num_gpu 99
FIM not working	Use raw /api/generate (not chat) with FIM tokens
Repetitive output	Override repeat_penalty to 1.2 or 1.25 at runtime
Nonsensical code	Lower temperature to 0.3 or 0.4 at runtime
OOM error	Reduce num_ctx or switch to q4_0 KV cache

---

ALTERNATIVE MODELS

# Larger coding model (needs 16 GB)
ollama pull oamazonasgabriel/qwen3.5-9b:q4-16gbGPU

# Larger MoE model (needs 24 GB)
ollama pull oamazonasgabriel/qwen3.6-35b-a3b:q4-24gbGPU

# Even smaller FIM model (needs 2 GB)
ollama pull oamazonasgabriel/qwen2.5-coder-0.5b:fp16-8gbGPU

---

CREDITS

Role	Entity
Base Model	Qwen Team, Alibaba Group
Original Model	Qwen3.5-4B
GGUF Conversion	unsloth/Qwen3.5-4B-GGUF
Ollama Packaging	impacte.tech
License	Apache 2.0

---

LINKS

Resource	URL
This model on Ollama	https://ollama.com/oamazonasgabriel/qwen3.5-4b
Upstream model (HuggingFace)	https://huggingface.co/Qwen/Qwen3.5-4B
GGUF files (HuggingFace)	https://huggingface.co/unsloth/Qwen3.5-4B-GGUF
Ollama docs	https://docs.ollama.com
Built by	impacte.tech