simular-ai/Agent-S

An open-source computer use agent that operates a real GUI — mouse, keyboard, and screen — the way a person does.

  🌐 [Sai OSWorld 2.0 report]  🌐 [S3 blog]  📄 [S3 Paper (TMLR 2026)]  🎥 [S3 Video]

  🌐 [S2 blog]  📄 [S2 Paper (COLM 2025)]  🎥 [S2 Video]

  🌐 [S1 blog]  📄 [S1 Paper (ICLR 2025)]  🎥 [S1 Video]

• 2026/10/01: The Simular API is live. Build your always-on computer agents in a minute. Simular Platform is an autonomous computer API for building always-on agents that fit your needs. World-leading agent infrastructure, 90% token saving. Get an API key →
• 2026/08/28: Sai tops OSWorld 2.0, beating GPT and Opus with lower costs (73%) — our production computer use agent is now state of the art on the hardest desktop benchmark!
• 2026/07/30: The Agent S3 paper "Scaling Agents for Computer Use" is accepted to TMLR 2026!
• 2025/12/15: Agent S3 is the first to surpass human-level performance on OSWorld with an impressive score of 72.60%!
• 2025/10/02: Released Agent S3 and its technical paper, setting a new SOTA of 69.9% on OSWorld (approaching 72% human performance), with strong generalizability on WindowsAgentArena and AndroidWorld! It is also simpler, faster, and more flexible.
• 2025/08/01: Agent S2.5 is released (gui-agents v0.2.5): simpler, better, and faster! New SOTA on OSWorld-Verified!
• 2025/07/07: The Agent S2 paper is accepted to COLM 2025! See you in Montreal!
• 2025/04/27: The Agent S paper won the Best Paper Award 🏆 at ICLR 2025 Agentic AI for Science Workshop!
• 2025/04/01: Released the Agent S2 paper with new SOTA results on OSWorld, WindowsAgentArena, and AndroidWorld!
• 2025/03/12: Released Agent S2 along with v0.2.0 of gui-agents, the new state-of-the-art for computer use agents (CUA), outperforming OpenAI's CUA/Operator and Anthropic's Claude 3.7 Sonnet Computer-Use!
• 2025/01/22: The Agent S paper is accepted to ICLR 2025!
• 2025/01/21: Released v0.1.2 of gui-agents library, with support for Linux and Windows!
• 2024/12/05: Released v0.1.0 of gui-agents library, allowing you to use Agent-S for Mac, OSWorld, and WindowsAgentArena with ease!
• 2024/10/10: Released the Agent S paper and codebase!

1. 💡 Introduction
2. 🎯 OSWorld Benchmark Results
3. 🛠️ Installation & Setup
4. 🚀 Usage
5. 🎬 See Sai in Action
6. 🔌 Simular API
7. 🤝 Acknowledgements
8. 💬 Citation

Agent S is an open source computer use agent framework from Simular. It takes a natural-language task, looks at the screen, and completes the task by clicking, typing, and scrolling in ordinary desktop and web applications — no API integration and no per-app scripting required. Agent S3 is the third generation of the framework and was the first computer use agent to surpass human performance on the OSWorld benchmark, at 72.60%. It runs on macOS, Windows, and Linux, and works with models from OpenAI, Anthropic, and open-weight providers.

The research continues in production: in August 2026, Sai — Simular's hosted computer use agent, built on the ideas in this repo — reached a 73% success rate on OSWorld 2.0, ahead of GPT-5.6 Sol at 62.57% as reported by OpenAI, and at lower cost. If you are looking for a hosted, production version of this technology rather than a research framework, see Sai.

Whether you're researching OS agents, automating your own desktop, or contributing to open source computer use, we're excited to have you here.

Our flagship product Sai reaches a 73% success rate on OSWorld 2.0, a 108-task benchmark of long professional and everyday tasks that take skilled humans more than an hour to complete. That places Sai ahead of GPT-5.6 Sol at 62.57% as reported by OpenAI, and it does so at lower cost per task. Full write-up: Sai Tops OSWorld 2.0.

On OSWorld, Agent S3 alone reaches 66% in the 100-step setting, already exceeding the previous state of the art of 63.4% (GTA1 w/ GPT-5). With the addition of Behavior Best-of-N, performance climbs even higher to 72.6%, surpassing human-level performance on OSWorld (~72%)!

Agent S3 also demonstrates strong zero-shot generalization! On WindowsAgentArena, accuracy rises from 50.2% using only Agent S3 to 56.6% by selecting from 3 rollouts. Similarly on AndroidWorld, performance improves from 68.1% to 71.6%

• Single Monitor: Our agent is designed for single monitor screens
• Security: The agent runs Python code to control your computer - use with care
• Supported Platforms: Linux, Mac, and Windows

To install Agent S3 without cloning the repository, run

pip install gui-agents

If you would like to test Agent S3 while making changes, clone the repository and install using

pip install -e .

Don't forget to also brew install tesseract! Pytesseract requires this extra installation to work.

Add to your .bashrc (Linux) or .zshrc (MacOS):

export OPENAI_API_KEY=<YOUR_API_KEY>
export ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY>
export HF_TOKEN=<YOUR_HF_TOKEN>

import os
os.environ["OPENAI_API_KEY"] = "<YOUR_API_KEY>"

We support Azure OpenAI, Anthropic, Gemini, Open Router, and vLLM inference. See models.md for details.

For optimal performance, we recommend UI-TARS-1.5-7B hosted on Hugging Face Inference Endpoints or another provider. See Hugging Face Inference Endpoints for setup instructions.

⚡️ Recommended Setup:
For the best configuration, we recommend using OpenAI gpt-5-2025-08-07 as the main model, paired with UI-TARS-1.5-7B for grounding.

Note, this is running Agent S3, our improved agent, without bBoN.

Run Agent S3 with the required parameters:

agent_s \
    --provider openai \
    --model gpt-5-2025-08-07 \
    --ground_provider huggingface \
    --ground_url http://localhost:8080 \
    --ground_model ui-tars-1.5-7b \
    --grounding_width 1920 \
    --grounding_height 1080

For tasks that require code execution (e.g., data processing, file manipulation, system automation), you can enable the local coding environment:

agent_s \
    --provider openai \
    --model gpt-5-2025-08-07 \
    --ground_provider huggingface \
    --ground_url http://localhost:8080 \
    --ground_model ui-tars-1.5-7b \
    --grounding_width 1920 \
    --grounding_height 1080 \
    --enable_local_env

⚠️ WARNING: The local coding environment executes arbitrary Python and Bash code locally on your machine. Only use this feature in trusted environments and with trusted inputs.

• --provider: Main generation model provider (e.g., openai, anthropic, etc.) - Default: "openai"
• --model: Main generation model name (e.g., gpt-5-2025-08-07) - Default: "gpt-5-2025-08-07"
• --ground_provider: The provider for the grounding model - Required
• --ground_url: The URL of the grounding model - Required
• --ground_model: The model name for the grounding model - Required
• --grounding_width: Width of the output coordinate resolution from the grounding model - Required
• --grounding_height: Height of the output coordinate resolution from the grounding model - Required

• --model_temperature: The temperature to fix all model calls to (necessary to set to 1.0 for models like o3 but can be left blank for other models)

The grounding width and height should match the output coordinate resolution of your grounding model:

• UI-TARS-1.5-7B: Use --grounding_width 1920 --grounding_height 1080
• UI-TARS-72B: Use --grounding_width 1000 --grounding_height 1000

• --model_url: Custom API URL for main generation model - Default: ""
• --model_api_key: API key for main generation model - Default: ""
• --ground_api_key: API key for grounding model endpoint - Default: ""
• --max_trajectory_length: Maximum number of image turns to keep in trajectory - Default: 8
• --enable_reflection: Enable reflection agent to assist the worker agent - Default: True
• --enable_local_env: Enable local coding environment for code execution (WARNING: Executes arbitrary code locally) - Default: False

The local coding environment enables Agent S3 to execute Python and Bash code directly on your machine. This is particularly useful for:

• Data Processing: Manipulating spreadsheets, CSV files, or databases
• File Operations: Bulk file processing, content extraction, or file organization
• System Automation: Configuration changes, system setup, or automation scripts
• Code Development: Writing, editing, or executing code files
• Text Processing: Document manipulation, content editing, or formatting

When enabled, the agent can use the call_code_agent action to execute code blocks for tasks that can be completed through programming rather than GUI interaction.

Requirements:

• Python: The same Python interpreter used to run Agent S3 (automatically detected)
• Bash: Available at /bin/bash (standard on macOS and Linux)
• System Permissions: The agent runs with the same permissions as the user executing it

Security Considerations:

• The local environment executes arbitrary code with the same permissions as the user running the agent
• Only enable this feature in trusted environments
• Be cautious when the agent generates code for system-level operations
• Consider running in a sandboxed environment for untrusted tasks
• Bash scripts are executed with a 30-second timeout to prevent hanging processes

First, we import the necessary modules. AgentS3 is the main agent class for Agent S3. OSWorldACI is our grounding agent that translates agent actions into executable python code.

import pyautogui
import io
from gui_agents.s3.agents.agent_s import AgentS3
from gui_agents.s3.agents.grounding import OSWorldACI
from gui_agents.s3.utils.local_env import LocalEnv  # Optional: for local coding environment

# Load in your API keys.
from dotenv import load_dotenv
load_dotenv()

current_platform = "linux"  # "darwin", "windows"

Next, we define our engine parameters. engine_params is used for the main agent, and engine_params_for_grounding is for grounding. For engine_params_for_grounding, we support custom endpoints like HuggingFace TGI, vLLM, and Open Router.

engine_params = {
  "engine_type": provider,
  "model": model,
  "base_url": model_url,           # Optional
  "api_key": model_api_key,        # Optional
  "temperature": model_temperature # Optional
}

# Load the grounding engine from a custom endpoint
ground_provider = "<your_ground_provider>"
ground_url = "<your_ground_url>"
ground_model = "<your_ground_model>"
ground_api_key = "<your_ground_api_key>"

# Set grounding dimensions based on your model's output coordinate resolution
# UI-TARS-1.5-7B: grounding_width=1920, grounding_height=1080
# UI-TARS-72B: grounding_width=1000, grounding_height=1000
grounding_width = 1920  # Width of output coordinate resolution
grounding_height = 1080  # Height of output coordinate resolution

engine_params_for_grounding = {
  "engine_type": ground_provider,
  "model": ground_model,
  "base_url": ground_url,
  "api_key": ground_api_key,  # Optional
  "grounding_width": grounding_width,
  "grounding_height": grounding_height,
}

Then, we define our grounding agent and Agent S3.

# Optional: Enable local coding environment
enable_local_env = False  # Set to True to enable local code execution
local_env = LocalEnv() if enable_local_env else None

grounding_agent = OSWorldACI(
    env=local_env,  # Pass local_env for code execution capability
    platform=current_platform,
    engine_params_for_generation=engine_params,
    engine_params_for_grounding=engine_params_for_grounding,
    width=1920,  # Optional: screen width
    height=1080  # Optional: screen height
)

agent = AgentS3(
    engine_params,
    grounding_agent,
    platform=current_platform,
    max_trajectory_length=8,  # Optional: maximum image turns to keep
    enable_reflection=True     # Optional: enable reflection agent
)

Finally, let's query the agent!

# Get screenshot.
screenshot = pyautogui.screenshot()
buffered = io.BytesIO() 
screenshot.save(buffered, format="PNG")
screenshot_bytes = buffered.getvalue()

obs = {
  "screenshot": screenshot_bytes,
}

instruction = "Close VS Code"
info, action = agent.predict(instruction=instruction, observation=obs)

exec(action[0])

Refer to gui_agents/s3/cli_app.py for more details on how the inference loop works.

To deploy Agent S3 in OSWorld, follow the OSWorld Deployment instructions.

Most computer work is not one-off — it repeats. Sai turns those repeating jobs into recurring workflows: describe the routine once, set when it should run, and the agent does it on a cloud computer on your schedule. Browse ready-made routines in the Sai workflow template gallery, or start from the Simular API if you would rather trigger runs from your own code.

The Simular API gives you Sai, the computer use agent behind the OSWorld 2.0 result above, through one HTTP endpoint. Send a task in plain English; Sai opens apps, clicks and types on a cloud computer, then returns the result. There are no VMs, display drivers or agent harnesses to run yourself.

• For coding agents: add Sai to Claude Code, Codex or Cursor as an MCP server with one command, then ask your agent to hand Sai the desktop work.
• Free to start: the free plan includes a cloud computer. Sai asks for approval before sign-ins and other sensitive steps.
• Create an API key and follow the quick start at platform.simular.ai, or read the API docs.

If you find this codebase useful, please cite:

@misc{Agent-S3,
      title={The Unreasonable Effectiveness of Scaling Agents for Computer Use}, 
      author={Gonzalo Gonzalez-Pumariega and Vincent Tu and Chih-Lun Lee and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2025},
      eprint={2510.02250},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2510.02250}, 
}

@misc{Agent-S2,
      title={Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents}, 
      author={Saaket Agashe and Kyle Wong and Vincent Tu and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2025},
      eprint={2504.00906},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2504.00906}, 
}

@inproceedings{Agent-S,
    title={{Agent S: An Open Agentic Framework that Uses Computers Like a Human}},
    author={Saaket Agashe and Jiuzhou Han and Shuyu Gan and Jiachen Yang and Ang Li and Xin Eric Wang},
    booktitle={International Conference on Learning Representations (ICLR)},
    year={2025},
    url={https://arxiv.org/abs/2410.08164}
}