Linux-use – agent-first GUI control for Linux
Developer Javi Mosch launched linux-use, an agent-first tool allowing software agents to control Linux desktop widgets via AT-SPI without moving the mouse.
By Dillip Chowdary • Nov 07, 2026 • Source: javimosch.github.io

At a glance
- Product
- linux-use, open-source command-line tool
- Method
- AT-SPI accessibility tree, not screenshots
- Benchmark
- 28 elements walked in 48 ms
- Exit code
- 83 on a changed or missing element
- Display
- Full automation only on X11 today
- Untested
- Electron, Flutter, Java desktop apps
In this post
Developer Javi Mosch launched linux-use, an open-source command-line tool providing agent-first graphical interface control for Linux desktop systems. As outlined in javimosch.github.io's report, the software enables autonomous programs to manipulate application widgets directly through the Linux accessibility architecture rather than relying on computer vision or simulated mouse pointers.
This article examines the operational mechanics, framework compatibility, and engineering design choices within linux-use. Software engineers developing unattended workflows, desktop testing harnesses, and native Linux agent integrations can evaluate how the project eliminates fragile coordinate clicking while managing current display server limitations across modern Linux environments.
01What actually changed
Conventional desktop automation frameworks force artificial intelligence agents to analyze raw screenshots, estimate screen coordinates, and issue simulated mouse clicks. The linux-use utility replaces coordinate-based visual control with native Linux accessibility instrumentation using the Assistive Technology Service Provider Interface.
By communicating through accessibility channels, software agents discover running programs and extract structured interface trees containing interactive element roles, element names, bounding box centers, and verified actions. This design allows automated agents to actuate application components directly, completely bypassing visual rendering, screen capture latencies, and mouse pointer movements.
If an element shifts or disappears, an execution attempt exits with status code 83 and prints a fix suggestion rather than guessing an approximate match. Incomplete operations are also surfaced explicitly; when hierarchy traversals encounter depth cutoffs or truncation limits, warnings appear in both the JSON payload and standard error streams. This notification prevents silent lookup failures where uninspected subtrees mislead an agent into registering zero interactive widgets.
If an element shifts or disappears, an execution attempt exits with status code 83 and prints a fix suggestion rather than guessing an approximate match. Incomplete operations are also surfaced explicitly; when hierarchy traversals encounter depth cutoffs or truncation limits, warnings appear in both the JSON payload and standard error streams. This notification prevents silent lookup failures where uninspected subtrees mislead an agent into registering zero interactive widgets.
02How it works
The standard execution loop initiates with the linux-use apps command, which enumerates running desktop software exposing active accessibility interfaces. To read an application, an agent invokes linux-use state targeting an application flag such as gnome-calculator.
In author benchmark runs, this walk method traversed 28 interactive elements in 48 milliseconds, outputting a structured JSON array. Each element entry contains a unique reference path like gnome-calculator:/0/1/0/1/0/0/1/0/22#18124260, a role designation such as push button, an element name like 7, horizontal coordinate cx 114, vertical coordinate cy 376, and an action type specified as click.
Interaction occurs by passing the element reference string to linux-use act, which triggers the widget's internal accessible action directly. This invocation requires no window focus, avoids moving the mouse cursor, and succeeds even while the target window remains completely occluded behind another application.
When workflows need to pause for interface updates instead of polling continuously, the linux-use watch command streams AT-SPI events as newline-delimited JSON. By specifying an application target, a widget role such as editbar, and a duration limit like 5000 milliseconds, an agent can block execution until exact interface conditions are fulfilled.
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
03Why it matters
Eliminating visual screen scraping removes major points of failure in desktop automation. Because operations act on native widget hooks, actions remain fully functional when target windows are obscured, resized, or placed behind other desktop windows.
Background automation agents can run alongside active desktop users without stealing window focus or wresting control of the physical mouse pointer. Replacing repetitive screen polling with native AT-SPI callback streams substantially curbs compute and energy waste, allowing automated systems to idle smoothly while waiting for asynchronous operations like file downloads, remote network calls, or intensive background data calculations.
The software adheres to agent-first command-line conventions defined at cli-specs.intrane.fr, prioritizing reliable automated machine consumption. Structured data is delivered strictly over standard output in JSON format, while human-readable diagnostics, warnings, and contextual guidance are routed to standard error.
Explicit typed errors accompany semantic exit codes, providing concrete remediation suggestions whenever executions encounter unresolvable states. By refusing to guess or click unidentified interface elements, the runtime prevents automated pipelines from initiating incorrect actions or misinterpreting transient application states as successful outcomes during unattended enterprise workflows.
04Who is affected
In Qt 5 environments, menu actuation is verified through the ShowMenu call. Web content within Chromium and Microsoft Edge browsers functions when launched with the --force-renderer-accessibility flag. However, frameworks such as Electron, Flutter, and Java desktop applications remain completely untested, requiring engineering teams to perform independent validation before attempting automation across those runtimes.
In Qt 5 environments, menu actuation is verified through the ShowMenu call. Web content within Chromium and Microsoft Edge browsers functions when launched with the --force-renderer-accessibility flag. However, frameworks such as Electron, Flutter, and Java desktop applications remain completely untested, requiring engineering teams to perform independent validation before attempting automation across those runtimes.
System administrators and DevOps engineers must also navigate underlying display protocol constraints. While accessibility paths function across Wayland environments, synthetic mouse clicks and keyboard keystrokes remain broken, restricting full functional automation to X11 display servers today.
Installation on Linux requires compiling with development libraries including libatspi2.0-dev, libxtst-dev, libx11-dev, and libglib2.0-dev, followed by running the build script or downloading release binaries. Users must also configure their desktop interface by setting org.gnome.desktop.interface toolkit-accessibility to true via gsettings, and can verify host configuration using the built-in linux-use doctor command.
05What to watch
A critical consideration for engineering teams is the experimental status of the tool. The author conducted testing exclusively by hand on a single hardware and operating system profile: an x86-64 machine running Ubuntu 22.04.3 with GNOME 42.9 on X11.
The project currently lacks an automated test suite, meaning reported performance figures represent manual measurements that are not re-checked upon git commits. The developer emphasizes treating linux-use as a v0.x tool, signaling that early adopters should monitor future repository updates for automated integration test suites and cross-distribution verification.
Machin is designed to be concise enough for autonomous agents to author, compiling through C into a standalone native binary that bundles HTTP, SQLite, and cryptographic libraries without requiring external runtimes. The tool consists of a single source file supplemented by a small C shim for the AT-SPI foreign function interface boundary. Observers tracking projects in the awesome-machin ecosystem will watch for future Wayland input fixes.
Machin is designed to be concise enough for autonomous agents to author, compiling through C into a standalone native binary that bundles HTTP, SQLite, and cryptographic libraries without requiring external runtimes. The tool consists of a single source file supplemented by a small C shim for the AT-SPI foreign function interface boundary. Observers tracking projects in the awesome-machin ecosystem will watch for future Wayland input fixes.
Frequently asked questions
What is linux-use and how does it control Linux applications?
It is an agent-first command-line tool that controls Linux desktop software through AT-SPI accessibility interfaces rather than pixel coordinate simulation. It invokes accessible actions directly on individual widgets, allowing automation even when application windows are hidden or occluded.
Which Linux desktop environments and application toolkits are verified to work?
The tool has been tested on Ubuntu 22.04.3 with GNOME 42.9 under X11, verifying six GTK 3 and GTK 4 applications alongside Qt 5 menus. Chromium and Edge work with the accessibility renderer flag, while Electron, Java, and Flutter remain untested.
Why does linux-use fail on Wayland display servers?
While AT-SPI accessibility tree inspection works under Wayland, synthetic mouse clicks and keyboard input mechanisms are currently broken. Full actuation functionality currently requires running an X11 session.
What programming language is linux-use built with?
It is written in machin, or Machine-First Language (MFL), which compiles through C into a standalone native binary with built-in HTTP, SQLite, and crypto capabilities. The codebase consists of a single source file and a small C shim for the AT-SPI interface.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement