gnome-jarvis/docs/computer-use.md

Computer use

Computer use is a revocable mode that combines observation, semantic actions, portal input, and visible CU chrome (target highlight and agent cursor). It is never implicitly enabled by a normal chat turn.

sequenceDiagram
  participant User
  participant D as jarvisd
  participant H as Harness
  participant CU as CU session
  participant P as Portal/EIS
  participant A as AT-SPI
  User->>D: “take the wheel”
  D->>CU: grant(persist, monitors)
  CU->>P: ScreenCast + RemoteDesktop consent
  User-->>P: approve
  loop step budget
    CU->>P: capture frame
    CU->>A: tree snapshot
    CU->>H: observe bundle
    H-->>CU: one action
    CU->>CU: safety + target validation
    CU->>P: EIS input or
    CU->>A: named action
  end
  User->>D: “hands off” / Escape
  D->>CU: revoke
  CU->>P: close sessions immediately

Actuator priority

1. Domain skill or app API.
2. App D-Bus/GIO action.
3. AT-SPI named action or stable reference.
4. GNOME Shell window/focus helper.
5. Vision coordinates for broken accessibility.
6. Opt-in ydotool/X11 fallback.

The primary Wayland input path is XDG RemoteDesktop plus EIS/libei. After portal consent, Jarvis opens ConnectToEIS and injects pointer/keyboard events through a built-in libei sender. Observation reads the live PipeWire ScreenCast stream (the compositor frame buffer), not the Screenshot portal. JARVIS_LIBEI_BRIDGE is an optional override. If EIS is unavailable, the helper falls back to portal Notify methods instead of refusing readiness. Frames are downscaled for vision, kept in temporary storage, and removed on revoke unless trace retention is enabled.

Safety rules

• Grant is off by default and dies on lock screen.
• Password/PAM roles are refused.
• Dangerous keys and destructive targets need confirmation.
• Default budget is 20 actions; hard cap is 100.
• Each action gets a visible target highlight and audit record.
• Three state-changing steps with no observed change abort the run.

See computer-use acceptance for the live GNOME scenarios.