Enabling AI to Use Desktop Apps Like a Human

Enable AI to control macOS and Windows desktop apps using semantic UI understanding, not fragile screen coordinates. An open source automation game changer.

Today, let me introduce pi-computer-use — a project that genuinely blew my mind.

It can control macOS and Windows desktop applications, and unlike traditional tools that rely on brittle screenshot-based coordinate clicking, it first reads the app’s accessibility tree, understands the interface, and then acts. Vision is only used as a fallback.

Project Overview

Before diving into pi-computer-use, let’s briefly talk about Pi.

Pi is an open-source AI coding agent that lets developers collaborate with AI using natural language to complete coding tasks. Unlike traditional AI assistants, Pi emphasizes tool use and autonomous execution — it can run commands, manipulate files, call APIs, and behave like a real programmer.

pi-computer-use is a custom-built extension for Pi. Its core goal is simple: enable AI to operate desktop applications just like a human.