# Computer use

Computer use is a revocable mode that combines observation, semantic actions,
portal input, and a visible ARC overlay. It is never implicitly enabled by a
normal chat turn.

```mermaid
sequenceDiagram
  participant User
  participant D as jarvisd
  participant H as Harness
  participant CU as CU session
  participant P as Portal/EIS
  participant A as AT-SPI
  User->>D: “take the wheel”
  D->>CU: grant(persist, monitors)
  CU->>P: ScreenCast + RemoteDesktop consent
  User-->>P: approve
  loop step budget
    CU->>P: capture frame
    CU->>A: tree snapshot
    CU->>H: observe bundle
    H-->>CU: one action
    CU->>CU: safety + target validation
    CU->>P: EIS input or
    CU->>A: named action
  end
  User->>D: “hands off” / Escape
  D->>CU: revoke
  CU->>P: close sessions immediately
```

## Actuator priority

1. Domain skill or app API.
2. App D-Bus/GIO action.
3. AT-SPI named action or stable reference.
4. GNOME Shell window/focus helper.
5. Vision coordinates for broken accessibility.
6. Opt-in ydotool/X11 fallback.

The primary Wayland input path is XDG RemoteDesktop plus EIS/libei. After
portal consent, Jarvis opens `ConnectToEIS` and injects pointer/keyboard
events through a built-in libei sender. Observation reads the live PipeWire
ScreenCast stream (the compositor frame buffer), not the Screenshot portal.
`JARVIS_LIBEI_BRIDGE` is an optional override. If EIS is unavailable, the helper
falls back to portal Notify methods instead of refusing readiness. Frames are
downscaled for vision,
kept in temporary storage, and removed on revoke unless trace retention is
enabled.

## Safety rules

- Grant is off by default and dies on lock screen.
- Password/PAM roles are refused.
- Dangerous keys and destructive targets need confirmation.
- Default budget is 20 actions; hard cap is 100.
- Each action gets a visible target highlight and audit record.
- Three state-changing steps with no observed change abort the run.

See [computer-use acceptance](cu-acceptance.md) for the live GNOME scenarios.