gnome-jarvis/docs/computer-use.md

Computer use

Computer use is a revocable mode that combines observation, semantic actions, portal input, and a visible ARC overlay. It is never implicitly enabled by a normal chat turn.

sequenceDiagram
  participant User
  participant D as jarvisd
  participant H as Harness
  participant CU as CU session
  participant P as Portal/EIS
  participant A as AT-SPI
  User->>D: “take the wheel”
  D->>CU: grant(persist, monitors)
  CU->>P: ScreenCast + RemoteDesktop consent
  User-->>P: approve
  loop step budget
    CU->>P: capture frame
    CU->>A: tree snapshot
    CU->>H: observe bundle
    H-->>CU: one action
    CU->>CU: safety + target validation
    CU->>P: EIS input or
    CU->>A: named action
  end
  User->>D: “hands off” / Escape
  D->>CU: revoke
  CU->>P: close sessions immediately

Actuator priority

1. Domain skill or app API.
2. App D-Bus/GIO action.
3. AT-SPI named action or stable reference.
4. GNOME Shell window/focus helper.
5. Vision coordinates for broken accessibility.
6. Opt-in ydotool/X11 fallback.

The primary Wayland input path is XDG RemoteDesktop plus EIS/libei. The portal helper refuses to report input readiness without a configured local EIS bridge (JARVIS_LIBEI_BRIDGE). Screenshot frames are downscaled for vision, kept in temporary storage, and removed on revoke unless trace retention is enabled.

Safety rules

• Grant is off by default and dies on lock screen.
• Password/PAM roles are refused.
• Dangerous keys and destructive targets need confirmation.
• Default budget is 20 actions; hard cap is 100.
• Each action gets a visible target highlight and audit record.
• Three state-changing steps with no observed change abort the run.

See computer-use acceptance for the live GNOME scenarios.