xuonglun123/ ui-voyager

6 11 hours ago

🚀 State-of-the-Art Qwen3-VL GUI & Mobile Agent (Q8_0) with intact CLIP vision projector for high-precision UI grounding, OCR, and automation.

vision 
ollama run xuonglun123/ui-voyager:q8-vision

Models

View all →

1 model

ui-voyager:q8-vision

4.7GB · 256K context window · Text, Image · 11 hours ago

Readme

🚀 UI-Voyager: Multimodal GUI & Mobile Navigation Agent (Q8_0 Vision)

UI-Voyager Overview

Base Architecture Quantization Vision Projector Grounding Pass Rate License

🌟 Overview

UI-Voyager (Q8_0) is a fine-tuned multimodal vision-language agent based on Qwen3-VL-4B, engineered specifically for Mobile GUI Navigation, Element Grounding, and Screen Automation (evaluated on AndroidWorld).

This distribution is packaged as a true multi-layer OCI artifact in Ollama: - LLM Weights: Q8_0 quantization (UI-Voyager.Q8_0.gguf, 4.28 GB) ensuring maximum reasoning capacity. - Vision Projector: Intact native clip projector with qwen3vl_merger layer (454 MB), allowing direct image understanding and pixel coordinate grounding without relying on external vision pipelines.

UI-Voyager Framework

⚡ Quick Start

1. Pull & Run from Ollama

ollama run xuonglun123/ui-voyager:q8-vision

2. Multi-modal Screenshot Grounding (CLI)

You can provide an image and an instruction directly:

ollama run xuonglun123/ui-voyager:q8-vision "Tap the Login button. Give its center coordinate. screenshot.png"

📊 Rigorous Quality & Benchmark Results

Evaluated across a comprehensive 40-case multi-resolution test battery:

Benchmark Test Metric Result Status
GUI Element Grounding Bounding Box Hit Rate (40 probes) 100.0% (40⁄40) PASS
Coordinate Precision Mean Distance to Element Center 11.85 pixels PASS
Target Absence Check Rejection of non-existent UI targets 100.0% PASS
OCR Accuracy English, Vietnamese, Numerics, Hex Codes 10 / 10 (CER: 0.00) PASS
Projector Integrity OCI Projector Layer Verification application/vnd.ollama.image.projector VERIFIED
Inference Stability 75+ Consecutive Multi-modal Calls 0 Crashes (1.56s / inference) PASS

🛠️ Action Output Format

UI-Voyager outputs official structured mobile_use actions in normalized 999x999 coordinate space:

Thought: The goal is to tap the "Login" button and enter the credentials.
Action: "Tap the Login button to initiate authentication."
<tool_call>
{"name": "mobile_use", "arguments": {"action": "click", "coordinate": [255, 190]}}
</tool_call>

🐍 Python SDK Integration

import base64
import requests

def analyze_ui(image_path, instruction):
    with open(image_path, "rb") as f:
        img_b64 = base64.b64encode(f.read()).decode("utf-8")

    response = requests.post(
        "http://localhost:11434/api/chat",
        json={
            "model": "xuonglun123/ui-voyager:q8-vision",
            "messages": [
                {
                    "role": "user",
                    "content": instruction,
                    "images": [img_b64]
                }
            ],
            "options": {"temperature": 0.0},
            "stream": False
        }
    )
    return response.json()["message"]["content"]

# Example:
result = analyze_ui("screen.png", "Tap the Settings icon. Give its center coordinate.")
print(result)

📜 Attribution & References