62 lines
2.0 KiB
Markdown
62 lines
2.0 KiB
Markdown
# Computer use
|
|
|
|
Computer use is a revocable mode that combines observation, semantic actions,
|
|
portal input, and visible CU chrome (target highlight and agent cursor). It is never implicitly enabled by a
|
|
normal chat turn.
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant User
|
|
participant D as jarvisd
|
|
participant H as Harness
|
|
participant CU as CU session
|
|
participant P as Portal/EIS
|
|
participant A as AT-SPI
|
|
User->>D: “take the wheel”
|
|
D->>CU: grant(persist, monitors)
|
|
CU->>P: ScreenCast + RemoteDesktop consent
|
|
User-->>P: approve
|
|
loop step budget
|
|
CU->>P: capture frame
|
|
CU->>A: tree snapshot
|
|
CU->>H: observe bundle
|
|
H-->>CU: one action
|
|
CU->>CU: safety + target validation
|
|
CU->>P: EIS input or
|
|
CU->>A: named action
|
|
end
|
|
User->>D: “hands off” / Escape
|
|
D->>CU: revoke
|
|
CU->>P: close sessions immediately
|
|
```
|
|
|
|
## Actuator priority
|
|
|
|
1. Domain skill or app API.
|
|
2. App D-Bus/GIO action.
|
|
3. AT-SPI named action or stable reference.
|
|
4. GNOME Shell window/focus helper.
|
|
5. Vision coordinates for broken accessibility.
|
|
6. Opt-in ydotool/X11 fallback.
|
|
|
|
The primary Wayland input path is XDG RemoteDesktop plus EIS/libei. After
|
|
portal consent, Jarvis opens `ConnectToEIS` and injects pointer/keyboard
|
|
events through a built-in libei sender. Observation reads the live PipeWire
|
|
ScreenCast stream (the compositor frame buffer), not the Screenshot portal.
|
|
`JARVIS_LIBEI_BRIDGE` is an optional override. If EIS is unavailable, the helper
|
|
falls back to portal Notify methods instead of refusing readiness. Frames are
|
|
downscaled for vision,
|
|
kept in temporary storage, and removed on revoke unless trace retention is
|
|
enabled.
|
|
|
|
## Safety rules
|
|
|
|
- Grant is off by default and dies on lock screen.
|
|
- Password/PAM roles are refused.
|
|
- Dangerous keys and destructive targets need confirmation.
|
|
- Default budget is 20 actions; hard cap is 100.
|
|
- Each action gets a visible target highlight and audit record.
|
|
- Three state-changing steps with no observed change abort the run.
|
|
|
|
See [computer-use acceptance](cu-acceptance.md) for the live GNOME scenarios.
|