Agent S: an open agentic framework that uses computers like a human
STEP-GUI: The top GUI agent solution in the galaxy. Developed by the StepFun-GELab team and powered by StepFun’s cutting-edge research capabilities.
💻 A curated list of papers and resources for multi-modal Graphical User Interface (GUI) agents.
[ICLR 2026 Oral] ScaleCUA is the open-sourced computer use agents that can operate on cross-platform environments (Windows, macOS, Ubuntu, Android).
An understudy watches. Then performs.
[ICLR'25 Oral] UGround: Universal GUI Visual Grounding for GUI Agents
DataClaw: Agentic Tailoring Multimodal Data from Raw Streams — coming soon (code, weights, dataset & DataClaw-val upon acceptance).
Official implementation of UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning
[AAAI 2026] Test-Time Reinforcement Learning for GUI Grounding via Region Consistency https://arxiv.org/abs/2508.05615
[CVPR 2025] Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents
Code repo for "Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding"
[ACL'25 (Findings)] Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents
Curated resources about automated GUI computer-use via LLMs. Highly opinionated, focus is on quality vs quantity.
PreAct: Computer-Using Agents that Get Faster on Repeated Tasks (arXiv:2606.17929). Compile successful runs into verified state-machine programs and replay them 8.5–13× faster with no per-step LLM calls.
A Gradio-based demonstration for the Microsoft Fara-7B model, designed as a computer use agent. Users upload UI screenshots (e.g., desktop or app interfaces), provide task instructions (e.g., "Click on the search bar"), and receive parsed actions with visualized indicators overlaid on the image.
A curated hub for GUI agent papers, benchmarks, datasets, projects, and computer-use research.
A Windows-only, agent-agnostic benchmark evaluating desktop agent reliability, safety, and recovery under safe, parameterizable, and reversible chaos injection.
Multi-agent system for automated mobile QA testing using LLMs, ADB, and vision-grounded execution with Simular Agent S3
Boost Codex and Claude Code workflows by doubling productivity and output with tools designed for efficient AI coding assistance.