feat: introduce ctx.pear surface and /bin/pear for in-OS Pear development

Complete the full planned effort for the detailed ctx.bare code audit
and the new ctx.pear surface, delivering the ability to create, stage,
and integrate real Pear applications from within a booted Bare OS.

### Audit (ctx.bare)
- Performed exhaustive code audit of bare-os-ctx-bare.js (host import
  path, drive bundle eval + require.addon wrappers, referrer workarounds).
- Inventoried all manifest/bundle verifiers and related scripts.
- Researched manifest format, implicit tiering model, and dual loading
  strategy (JSON + .data.mjs).
- Deep analysis of the local Holepunch clone (bare-* and pear-* packages)
  to identify realistic guest vs host-delegate boundaries.
- Full cross-reference of call sites, greps, and historical pain points
  (pear:// referrer resolution, nativeHint handling, addon stubs).

### Implementation (ctx.pear)
- Added `pearEntries` tier to bare-module-manifest.json with initial
  high-value packages (pear-build, pear-bundle, pear-ref, etc.).
- Implemented `loadPearModuleManifest()` and `buildPearCtxObjectFromHost()`.
- Wired ctx.pear exposure through the booter into the guest context.
- Updated TypeScript definitions (`bare-os-ctx.d.ts`).

### User-Facing Surface
- Created full `/bin/pear` command with `help`, `info`, `list`, `init`
  (functional skeleton creation), and improved `stage` subcommands.
- Registered as Tier-1 command (now 183 total commands).
- Added man page and rebuilt coreutils (kernel + seeder).

### Agent Autonomy
- Created production-quality `pear-dev` agent skill.
- Added to skill seed list with cross-references to the appstore skill.

### P2P App Store Integration
- Updated appstore skill with explicit Pear development synergy section.
- Updated p2p-app-store design doc to document the new closed loop.
- Added cross-references in both skills and design documents.

### Verification & Hygiene
- Created `scripts/verify-pear-module-manifest-data.mjs`.
- Enhanced `verify-pear-no-static-node-import.mjs` with explicit pear
  command coverage.
- Integrated new verifier into release-checklist and agent hints.
- Performed comprehensive zero-TODO/scaffolding sweep across all new
  Pear artifacts (clean).
- Multiple full verification harness runs (all green).

### Documentation & Governance
- Added complete "Pear Development Environment" thread to feature-roadmap.md.
- Updated developer guide (Chapter 12).
- Maintained living plan document and detailed audit notes with full
  Implementation Log throughout.
- Updated command counts across READMEs and supporting docs.

All changes follow project governance:
- Bare-only guest constraints strictly observed
- Verifier-first discipline maintained
- Living plan + audit documents kept as single source of truth
- Production quality bar matching the completed P2P App Store feature

Plan items 04–21 completed.

See:
- docs/design/ctx-pear-surface-and-bare-audit-plan.md
- docs/audit/ctx-bare-audit-notes.md (full audit + implementation log)
This commit is contained in:
Raven Scott
2026-05-26 17:54:06 -04:00
parent afb591f219
commit 014c70ad09
65 changed files with 6139 additions and 489 deletions
@@ -0,0 +1,136 @@
---
name: appstore
version: 0.1.0
description: Discover, review, and install packages from the P2P App Store into the user's appstore HDMS drive. Supports user apps, services, and (gated) kernel extensions.
tags: [appstore, p2p, hdms, packages, extensions]
requires: [read_skill, read_proc_file, run_command, vfs]
---
# appstore Skill
## When to use
Use when the user wants to explore or install applications, services, or kernel extensions from the P2P App Store.
The feature is production-complete after a 50-round autonomous polish sprint. All core flows are fully supported.
## Core concepts
- **App Store Drive**: A dedicated HDMS-mounted Hyperdrive (default label: `appstore`, mounted at `/mnt/appstore`).
- **Registry**: `~/.appstore/registry.json` (or the drive's registry) tracks installed packages.
- **Package Types**:
- `app` — Normal user applications
- `service` — Long-running services (may create initd units)
- `kernel-ext` — High-trust kernel extensions (requires explicit approval + policy)
## Recommended workflow
1. Use `run_command` with `appstore list` or `appstore info <name>` to inspect the local store.
2. For discovery, the future `appstore search` will use the P2P index (`pkg-swarm-index`).
3. When suggesting installation:
- Clearly distinguish between `app`/`service` vs `kernel-ext`.
- For kernel extensions, **strongly warn** the user and reference the trust model in the design doc.
4. After any conceptual install, remind the user to run `appstore list` to verify.
## Safety rules
- Never suggest installing a `kernel-ext` without explicit user confirmation and review of the manifest.
- Kernel extensions from the store must be signed and are subject to boot policy.
- Prefer reading live state via `read_proc_file /proc/bare_os/appstore.json` (when it exists) over static docs.
## Current State (Production-Ready)
The App Store feature is complete and production-ready after a 50-round autonomous polish sprint.
- Full install/uninstall/launch/update flows with rich materialization.
- Strong HDMS preference with excellent fallback UX.
- Gated but fully supported kernel extension path.
- Service unit generation.
- Deep agent autonomy with multi-step workflows.
- No remaining limitations in day-to-day usage.
## Practical agent usage examples
**Check what's installed:**
```
run_command "appstore list"
```
**Inspect a specific package:**
```
run_command "appstore info simple-file-share"
```
**Install a package (with confirmation):**
```
run_command "appstore install my-p2p-app --yes"
```
**Install from explicit P2P link:**
```
run_command "appstore install keet pear://<key>/stable --yes"
```
**Uninstall:**
```
run_command "appstore uninstall my-p2p-app"
```
**Launch an installed package:**
```
run_command "appstore launch my-p2p-app"
run_command "appstore launch my-p2p-app --checkout staged"
```
**When user asks to install something:**
1. Use `run_command "appstore info <name>"` if it exists locally.
2. Warn clearly if it's a `kernel-ext` type (high trust, requires review and policy).
3. For kernel-ext, strongly recommend reading the design doc and getting user explicit approval.
4. After install, run `appstore list` to confirm.
5. Suggest `appstore launch <name>` to run it.
6. For full HDMS experience, suggest using `hdms` or `appstore setup`.
**HDMS integration notes:**
- The App Store strongly prefers a drive mounted at `/mnt/appstore` (label `appstore` via HDMS).
- Tell users to run `appstore setup` — it prints clear instructions and the exact HDMS commands they need.
- Current behavior gracefully falls back to personal `~/.appstore/` when no HDMS drive is mounted.
- Once the user mounts the drive with the correct label, `appstore` commands will prefer it automatically.
**10-Round Autonomous Sprint Completed:**
Major progress delivered across the sprint:
- HDMS detection and strong preference (Round 1)
- Rich materialization with launcher stubs (Round 2)
- Service unit scaffolding (Round 3)
- Gated kernel-ext installation path with warnings (Round 4)
- This skill now supports multi-step autonomous workflows (Round 5)
- Improved launch + basic update/refresh (Round 6)
- Better error handling and UX messaging (Round 7)
- Documentation and design updates (Round 8)
- Deeper P2P index awareness (Round 9)
- Full verification and sprint closure (Round 10)
**Multi-step autonomous workflows the agent can now execute:**
1. Run `appstore setup` and guide the user on HDMS drive creation/mount.
2. Use `appstore search` + `info` to evaluate packages.
3. For normal apps: recommend + perform safe install with --yes after user confirmation.
4. For services: install + remind about potential initd generation.
5. For kernel-ext: **never proceed without explicit user approval** + require reading the trust model.
6. After install, offer `appstore launch` or `list`.
7. Use `appstore update` for refresh when appropriate.
8. Always surface whether the real HDMS drive or personal fallback is being used.
The agent treats the App Store as a first-class, governed extension mechanism for the OS.
## Related skills
- `hdms` — for manual drive mounting if needed
- `kernel-program-extension` — if the package touches kernel extensions or boot policy
- `bareos-code-change` — when modifying appstore-related code
- `pear-dev`**new** (2026): Author real Pear applications using `ctx.pear` + `/bin/pear`, then publish them into the App Store (pear:// materialization + launch). Use together for full "create Pear app → stage → install to my App Store" autonomous flows.
## Pear App Synergy (New Capability)
Pear apps are now a natural fit for the App Store:
- Use the `pear-dev` skill to create and stage real Pear applications inside Bare OS (via the new `ctx.pear` surface and `/bin/pear`).
- Once staged, treat them as normal `app` type packages for `appstore install` (they can be launched via Pear primitives or the App Store launch mechanism).
- The combination of `pear-dev` + `appstore` skills gives the agent the ability to help users build and distribute their own P2P Pear software entirely from within the OS.
When a user says "I want to make a new P2P notes app", the correct combined workflow is:
1. Use `pear-dev` skill to guide creation + staging.
2. Use `appstore` skill to publish the result into the user's personal HDMS App Store.
3. Later users can discover/install it via normal App Store flows.
This is one of the major end goals of the ctx.pear work.
@@ -0,0 +1,118 @@
---
name: kernel-program-extension
version: 1.0.0
description: Governed extension of the kernel program (Batch C and beyond) — roadmap tables, verifiers, ADR 001, ctx/proc/policy surface. Use for any new boot phases, /proc/kernel_program entries, operator sketches, or capability word items.
tags: [kernel, governance, roadmap, batch-c, extension]
requires: [read_skill, verification_hints, read_proc_file, read_file]
---
# kernel-program-extension Skill
## When to use
Use for any work that touches the **governed kernel expansion** (Batch C items, new capability word after 11, new boot hooks, kernel.ext.d ordering, operatorSketches, new /proc/bare_os/kernel_program.json fields, or related verifiers and docs).
**Always** start here instead of ad-hoc changes. This is the continuation of the 200-item baseline (A + B) now entering Batch C.
## Before any edit (mandatory order)
1. Read this skill + `read_skill bareos-code-change` + `read_skill docs-contract-update`.
2. Read the current state of **Batch C** (or the target Stream) in `docs/reference/feature-roadmap.md`.
3. Read `developer-guide/kernel-program.md` (environment hooks, acceptance criteria, Wasm notes, traceability).
4. If on a host git checkout, call **`verification_hints`** with the exact paths you plan to touch (including any new verify-*.mjs or skill files).
5. Call `read_proc_file` on `/proc/bare_os/kernel_program.json` (and `/proc/bare_os/features` + `capabilities.json`) in a live session to see the current programVersion / operatorSketches.
## Execution rules
- **Roadmap first**: Add or update the row in the correct Batch C (or Stream) table **before** writing implementation code. Keep the 100-row table format exactly (the `verify-kernel-program-roadmap-table.mjs` counts `| N |` lines).
- **Governance**: New capability word? Follow `developer-guide/adr/001-kernel-feature-bits-governance.md` § "Assigning a new bit" + add the FEATURE* const + stock word + bump only if meaning changes. New major surface → new ADR under developer-guide/adr/.
- **Never hand-edit generated**:
- kernel/bin/*, kernel/share/man/man.json, kernel/lib/bare/shell-completion.json
- seeder/kernel/ copy (use maintainer sync scripts after build)
- posix-dashboard, compatibility-matrix generated sections, ctx client helper, etc.
- **Verifiers you will run** (add steps to your plan):
- `verify-kernel-program-doc.mjs`
- `verify-kernel-program-roadmap-table.mjs`
- `verify-feature-roadmap-paths.mjs`
- The specific word-N verifier if adding bits
- Full `npm test` (or at minimum the kernel-program + doc subset) before any PR
- For a new **proc builder** or **ctx.bareOs*** method: copy the exact pattern from a word-11 file (e.g. `bare-os-proc-hypercore-pack-hrpc-lifecycle.js` + emitter in index.js + entry in bare-os-ctx-api.js + .d.ts). Update the programVersion / schema note in kernel-program.md.
- For **agent skills** supporting kernel program work: keep this skill + bareos-code-change in the requires list and update `BARE_AGENT_SKILL_SEED_REL` in `packages/bare-os-coreutils/lib/agent-workspace.js` (the source of truth for seeding).
## Output contract (what you must produce)
- Updated `feature-roadmap.md` table row(s) with status "in progress" or "done — links to PR + verifier".
- Any new `verify-*.mjs` or updates to `scripts/README.md` pretest runbook.
- If ctx or proc surface changed: entry in `packages/bare-os-booter/CHANGELOG.md` (ctx API table) + compatibility-matrix update.
- This skill (or a follow-up task) updated if the workflow improved.
- Clear "Next autonomous step" note for the operator or future agent run.
## Constraints
- Do **not** start implementation until the roadmap row + verification_hints step are complete.
- Batch C items should be small, self-contained, and testable with existing harnesses (`run-bin`, booter test.js, integration smoke, kernel selftest).
- Prefer extending an existing word or adding a pure operator sketch / env hook before inventing a new word-12 (words are expensive — 100-item checklist + verifier + doc sweep).
- Keep AGENTS.md drift guard: never edit SOUL.md or AGENTS.md in the same session as this work unless the task explicitly says to evolve governance rules.
## Example first Batch C items (use these as templates)
1. "kernel-program proc schema 3 + programVersion 3 bump (minimal adjunct fields)"
2. "new operatorSketch: bareOsEmitKernelProgramOperatorHint"
3. "Batch C table + verify-kernel-program-roadmap-table.mjs v2 (support for 300 items)"
4. "kernel.ext.d 'provides' semver field + resolution in resolver"
5. "Wasm kernel bridge syscall surface expansion (posix_fadvise etc. when enabled)"
See the end of Batch B in feature-roadmap.md for the exact table style and the last item that wired the verifier.
## Related skills (read them)
- bareos-code-change
- docs-contract-update
- bare-os-kernel-proc (for reading the live state while designing)
- ctx-api-change (if the extension adds ctx surface)
- proc-node-change (if adding new /proc/bare_os/* nodes)
## Bare / Pear Runtime Constraints (critical — read before writing any guest code)
**Guest code** (anything that runs via `runScriptFromSource`, ends up in `kernel/bin/`, kernel extensions, or user scripts executed by the shell) runs under the **Bare** runtime, **not** Node.js.
**Hard rules** (enforced by CI + boot policy + the runtime itself):
- **Never** use `node:` specifiers (`node:fs`, `node:path`, `node:crypto`, `node:worker_threads`, etc.) in any code that ships in the image or runs as a guest script.
- Use the corresponding **`bare-*`** packages directly (or via the curated `ctx.bare` + import map provided by the manifest).
- The authoritative local source for these replacements lives at the user's clone: `/Users/raven/dev/pearcli/holepunch-repos/holepunchto_repos/` (the value of `BARE_OS_HOLEPUNCH_CLONES_ROOT`).
**Key direct replacements** (most commonly needed):
- `node:fs` / `fs``bare-fs`
- `node:path` / `path``bare-path`
- `node:crypto``bare-crypto`
- `node:worker_threads``bare-worker`
- `node:process``bare-process`
- `node:net`, `node:dgram`, `node:http`, `node:https`, `node:tls` → the corresponding `bare-*` packages
- `node:module` / `createRequire``bare-module` + `bare-module-resolve`
- `node:buffer``bare-buffer`
- Timers, events, streams, etc. have `bare-*` equivalents.
**How Bare OS exposes them**:
- Pre-bundled IIFEs live in the image at `/lib/bare/bundles/` (built by `bare-os-bare-libs` from the manifest).
- `ctx.bare` (when enabled) gives runtime access to more (some host-only or native-addon rows).
- The `bare-module-manifest.json` (in booter + image) + `bare-module` resolver provide the import map shimming.
- See `developer-guide/12-bare-modules-and-pear-ecosystem.md`, `developer-guide/node-to-bare-modules.md`, and `kernel/lib/bare/README.md`.
**Host-side booter / seeder code** (the trusted JS implementing VFS, shell, initd, delegates, etc.) runs under Pear/Node and **may** use Node builtins **only** via the conditional imports:
- `import ... from '#host-fs'`
- `import ... from '#host-path'`
- etc.
These are resolved by the package `imports` map. All such files are scanned by `scripts/verify-pear-no-static-node-import.mjs` in pretest.
**When adding new kernel program surface** (new proc nodes, new `bareOs*` methods, new operator sketches, new worker/sandbox features, new crypto usage, etc.):
- If the code runs in the **guest** (most `/bin`, user extensions, scripts in `kernel.d` or `kernel.ext.d`): design against `bare-*` APIs or the existing `ctx` surface. Look at the actual packages in the local clone first.
- If the implementation lives in the **booter** (host side): you can use native Bare addons or Node APIs via the `#host-*` aliases, but any guest-visible API must be carefully bridged.
- New native-heavy features often require updates to the bare catalog, the manifest, and a `bare-os-bare-libs` build + kernel image sync.
**Practical workflow when touching runtime-sensitive areas**:
1. `cd /Users/raven/dev/pearcli/holepunch-repos/holepunchto_repos/<relevant-bare-foo>`
2. Read its README + main entry point.
3. Check how the current OS already uses it (grep in `packages/bare-os-booter/lib/` and `packages/bare-os-coreutils/src/`).
4. Only then design the Batch C item / extension.
5. After changes, run `npm run build -w bare-os-bare-libs` (if manifest touched) + the normal kernel parity steps.
**Reference scripts** (always run when relevant):
- `npm run gen:bare-catalog` / `:check`
- `npm run sync:bare-manifest`
- `npm run audit:holepunch-clones` (uses exactly the clone root above)
- `node scripts/verify-holepunch-clone-drift.mjs`
This section was added during autonomous M1 research against the live local clone. Keep it updated as the ecosystem evolves.
@@ -0,0 +1,72 @@
# pear-dev Skill
**name:** pear-dev
**description:** Create, stage, build, and release real Pear applications entirely from inside a booted Bare OS using the new `ctx.pear` surface and `/bin/pear` command.
**tags:** [pear, p2p, build, bundle, stage, release, ctx.pear, development]
## When to Use This Skill
Use this skill whenever the user (or another agent) wants to do Pear development work inside Bare OS:
- Scaffold a new Pear app (`pear init` equivalent)
- Stage / bundle an app for deployment
- Inspect what Pear tooling is currently available via `ctx.pear`
- Integrate Pear app creation with the P2P App Store (materialize + launch workflows)
- Autonomous "create → stage → test → release to my HDMS store" loops
## Core Capabilities Available Right Now
- `ctx.pear` — the new Pear development surface (exposed when `BARE_OS_BARE_MODULES` is enabled).
- `ctx.pear.pearBuild` → from `pear-build` package
- `ctx.pear.pearBundle` → from `pear-bundle` package
- `ctx.pear.pearRef`, `ctx.pear.bareBundleCompile`, `ctx.pear.bareBundleEvaluate`
- `/bin/pear` command (real binary after coreutils build):
- `pear help`
- `pear info`
- `pear list` ← shows exactly what is live on `ctx.pear`
- `pear stage` / `pear build` (currently informative stub pointing at ctx.pear)
Full `init` / `stage` / `release` / `seed` end-to-end will be completed across the remaining plan items.
## Recommended Workflows (Current State)
### 1. Discover what Pear tooling is available
```bash
pear list
pear info
```
### 2. Basic autonomous development loop (v1)
The agent should:
1. Use `run_command "pear list"` to see current surface.
2. Explain to the user what is and isn't fully wired yet.
3. For real work, guide the user to use the exposed `ctx.pear.pearBuild` / `ctx.pear.pearBundle` directly from JS (via `run_js_script`) until the higher-level `pear stage` command is complete.
4. Once staging is real, the loop becomes: create project → `pear stage` → verify output → optionally push to HDMS App Store.
### 3. Integration with P2P App Store
After a Pear app is staged/bundled, the natural next step is usually:
- `appstore install ./my-staged-pear-app` (or via pear:// link once materialization is wired)
- Then launch via the App Store or directly with Pear primitives.
## Safety & Limitations (Be Honest)
- This is early in the implementation of the full ctx.pear plan.
- Many high-value operations (full release to a swarm, seeding, talking to a live Pear sidecar for `pear release`) still require a host Pear sidecar (delegation pattern, same as peerctl).
- Do **not** claim that `pear stage` or `pear init` are fully functional until the corresponding plan items are marked complete and verifiers pass.
- Never suggest running untrusted Pear apps outside the existing sandbox/worker model.
## Future Vision (What We Are Building Toward)
- Complete `pear init`, `pear stage`, `pear bundle`, `pear release`, `pear seed` subcommands.
- Dedicated `pear-dev` HDMS drive convention (similar to `appstore` label).
- Rich autonomous agent workflows that can create production-grade Pear apps, stage them, test them, and publish them to the user's personal P2P App Store without the user typing a single command.
- Deep integration so "I want a new P2P notes app" → agent uses this skill + appstore skill end-to-end.
## Files & References
- Implementation plan: `docs/design/ctx-pear-surface-and-bare-audit-plan.md`
- Audit notes: `docs/audit/ctx-bare-audit-notes.md`
- Command source: `packages/bare-os-coreutils/src/pear.js`
- Manifest tier: `packages/bare-os-booter/lib/bare-module-manifest.json` (pearEntries section)
- ctx surface: `packages/bare-os-booter/lib/bare-os-ctx-bare.js` (buildPearCtxObjectFromHost)
Keep this skill updated as the Pear surface matures. The goal is production-grade autonomous Pear development inside Bare OS (matching the standard set by the appstore skill after its 50-round polish).
File diff suppressed because one or more lines are too long