aquiffoo/
aquif-moe-800m:latest

71 4 months ago

Our first Mixture of Experts model, with 800 million active parameters, reaches state-of-the-art performance in the sub-1b range.

tools

4 months ago

8a4705e9e938 · 2.0GB

model
granitemoe
·
3.3B
·
Q4_K_M
license
Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR US
system
You are aquif-moe, specifically the 800 million active parameter variant, an empathetic AI copilot d
params
{ "temperature": 0.7 }
template
<|start_of_role|>system<|end_of_role|> {{- if and (gt (len .Messages) 0) (eq (index .Messages 0).Rol

Readme

aquif-moe-800m

aquif-moe-800m is our first Mixture of Experts (MoE) model, with only 800 million active parameters. Despite its compact size, it delivers exceptional performance-per-VRAM efficiency compared to larger models.

Model Overview

  • Name: aquif-moe-800m
  • Parameters: 800 million active parameters (3.3 billion total)
  • Context Window: 128,000 tokens
  • Architecture: Mixture of Experts (MoE)
  • Type: General-purpose LLM
  • Hosted on: Ollama

Key Features

  • Extremely efficient VRAM utilization (57.8 performance points per GB)
  • Expansive 128K token context window for handling long documents
  • Competitive performance against models with more parameters
  • Optimized for local inference on consumer hardware
  • Ideal for resource-constrained environments
  • Supports high-throughput concurrent sessions

Performance Benchmarks

aquif-moe-800m demonstrates state-of-the-art performance across multiple benchmarks, especially when considering its parameter efficiency:

Benchmark aquif-moe (0.8b) Llama 3.2 (1b) Gemma 3 (4b)
MMLU 52.2 49.3 59.6
HumanEval 37.5 22.6 36.0
GSM8K 49.0 44.4 38.4
Average 46.2 38.8 44.7

VRAM Efficiency

One of aquif-moe-800m’s standout features is its exceptional VRAM efficiency:

Model Average Performance VRAM (GB) Performance per VRAM
aquif-moe 46.2 0.8 57.8
Llama 3.2 38.8 1.2 32.3
Gemma 3 44.7 4.3 10.4

Use Cases

  • Edge computing and resource-constrained environments
  • Mobile and embedded applications
  • Local development environments
  • Quick prototyping and testing
  • Personal assistants on consumer hardware
  • Enterprise deployment with multiple concurrent sessions
  • Long document analysis and summarization
  • High-throughput production environments

Limitations

  • No thinking mode capability
  • May show hallucinations in some areas
  • May struggle with more complex reasoning tasks
  • Not optimized for specialized domains

Getting Started

To run via Ollama:

ollama run aquiffoo/aquif-moe-800m

Technical Details

The aquif-moe-800m leverages a Mixture of Experts architecture to achieve high parameter efficiency. While the total parameter count is larger, only 800 million parameters are activated during inference, allowing for significantly reduced VRAM requirements while maintaining competitive performance.

Enterprise Deployment

The model’s exceptional VRAM efficiency makes it particularly valuable for enterprise deployments: - Concurrent Sessions: Run multiple model instances on a single GPU - High Throughput: Serve more users with the same hardware footprint - Cost Efficiency: Lower infrastructure costs for production deployments - Scalability: Easier horizontal scaling across available resources

The 128K context window enables comprehensive document analysis while maintaining the model’s efficient resource utilization, making it suitable for enterprises dealing with large documents or extended conversations.

*Note: All performance metrics are approximated estimates based on internal evaluations.