SmolVLM2-2.2B-Instruct-Agentic-GUI

A lightweight vision-language model fine-tuned for GUI automation and agentic tasks. This model can understand screenshots, locate UI elements, and execute multi-step interactions on desktop and mobile interfaces.

Available Variants

Model Details

Capabilities

• GUI Grounding: Locate and identify UI elements from screenshots
• Agentic Reasoning: Plan and execute multi-step GUI interactions
• Action Generation: Generate precise click, type, scroll, and drag actions
• Cross-Platform: Works with desktop, mobile, and web interfaces

Usage

# Default (Q4_K_M)
ollama run ahmadwaqar/smolvlm2-agentic-gui "Click on the search button" --images ./screenshot.png

# Q8_0 (higher precision)
ollama run ahmadwaqar/smolvlm2-agentic-gui:q8_0 "Click on the search button" --images ./screenshot.png

# FP16 (full precision)
ollama run ahmadwaqar/smolvlm2-agentic-gui:fp16 "Click on the search button" --images ./screenshot.png

API Usage

import ollama

response = ollama.chat(
    model='ahmadwaqar/smolvlm2-agentic-gui',  # or :q8_0 or :fp16
    messages=[{
        'role': 'user',
        'content': 'Click on the search button',
        'images': ['./screenshot.png']
    }]
)
print(response['message']['content'])

Training

This model was trained using a two-phase approach from Smol2Operator:

1. Phase 1 - Grounding: Trained on smolagents/aguvis-stage-1 to learn UI element localization
2. Phase 2 - Agentic Reasoning: Fine-tuned on smolagents/aguvis-stage-2 for multi-step task planning

Performance

Action Space

The model outputs actions in normalized [0,1] coordinates:

• click(x, y) - Click at normalized coordinates
• double_click(x, y) - Double-click at normalized coordinates
• long_press(x, y) - Long press at normalized coordinates
• type(text) - Type text input
• press(keys) - Press keyboard key(s) (e.g. “enter”, [“ctrl”, “c”])
• scroll(direction, amount) - Scroll up or down
• drag(from_coord, to_coord) - Drag from [x1, y1] to [x2, y2]
• navigate_back() - Go back to previous page
• wait(seconds) - Wait for specified duration

Example Prompts

"Click on the 'Submit' button"
"Type 'hello world' in the search field"
"Scroll down to see more content"
"Navigate to the settings menu"

Links

• GGUF Weights
• Original Model
• Training Code
• Blog Post

Citation

@misc{smol2operator2025,
  title={Smol2Operator: Post-Training GUI Agents for Computer Use},
  author={Hugging Face Team},
  year={2025},
  url={https://huggingface.co/blog/smol2operator}
}

License

Apache 2.0