CUPComputer Use Protocol
Platform Guides

macOS

CUP on macOS using AXUIElement accessibility API.

Overview

On macOS, CUP uses the AXUIElement API via pyobjc (Python) or Swift/JXA interop (TypeScript).

Requirements

  • macOS 12 (Monterey) or later
  • Accessibility permission must be granted to your terminal or IDE

You must grant accessibility permissions in System Settings > Privacy & Security > Accessibility. Add your terminal app (Terminal, iTerm2, VS Code, etc.) to the allowed list.

Setup

See Installation for full setup instructions. Python installs pyobjc automatically; TypeScript bundles Swift/JXA interop.

How it works

The macOS adapter connects to the Accessibility API through AXUIElementCopyAttributeValue calls. It traverses the AX hierarchy, reading roles, subroles, titles, frames, and actions.

Role mapping

macOS AX RoleCUP Role
AXApplicationapplication
AXButtonbutton
AXCheckBoxcheckbox
AXComboBoxcombobox
AXGroupgroup / generic
AXHeadingheading
AXImageimg
AXLinklink
AXListlist
AXMenumenu
AXMenuItemmenuitem
AXProgressIndicatorprogressbar
AXRadioButtonradio
AXSliderslider
AXStaticTexttext
AXTabGrouptablist
AXTabletable
AXTextFieldtextbox
AXWindowwindow
AXWebAreadocument

Action mapping

CUP ActionmacOS Implementation
clickAXPress action, fallback to coordinate click
typeFocus element + CGEvent key dispatch
setvalueSet AXValue attribute
toggleAXPress action
expandAXPress action
collapseAXPress action
selectAXPick / AXPress action
scrollCGEvent scroll wheel at element center

Platform-specific properties

For the full platform-specific property schema, see Node Structure.

Troubleshooting

"Not authorized to send Apple events"

Grant accessibility permissions to your terminal in System Settings.

Empty tree for some apps

Some sandboxed apps restrict accessibility access. Try running from a terminal with full disk access.

On this page