Ui Agents
A UI Agent is like a smart digital coworker that can use a computer just like a human — moving the mouse, clicking buttons, typing into forms, and switching between apps.
It uses a special kind of AI called a Large Action Model (LAM).
LAMs are trained to:
- Understand what you want (from text instructions).
- Figure out how to use software to get it done — just like a person.
Capabilities
- More resilient automations (When UIs get updated or elements move around dynamically, UI Agents can adapt without breaking)
- Lower development barrier (creating automation with less technical expertise)
- Cross-platform compatibility (works across operating systems)
Challenges
- slow compared to traditional RPA
- expensive to run
- can be unpredictable on longer, complex scenarios
|
Traditional RPA
|
UI Agents (LAMs Integrated)
|
|---|---|
|
Follows fixed, step-by-step instructions
|
Understands goals and figures out how to do them
|
|
Breaks if the screen layout changes
|
Adapts to new layouts and screen changes
|
|
Needs specific selectors (e.g., HTML tags, IDs)
|
Uses visual and contextual understanding
|
|
Needs to be programmed in detail
|
Accepts natural language goals
|
Example Use Case
- Locating and clicking a specific button that may move position based on dynamic content
- Filling out a form field with contextually appropriate information
|
Approach
|
How It Works
|
Problem
|
|---|---|---|
|
Traditional RPA
|
Uses fixed selectors like #row-1 > download-button
|
Breaks if the layout changes or if the newest entry isn't in the same position
|
|
UI Agent
|
Understands the instruction: "Click 'Download Report' next to the most recent entry"
|
Adapts even if the position or layout changes—no need to update code
|
