complete shell 100-item plan across parser, execution, security, and docs

Implement the full BareOS shell roadmap end-to-end, including grammar/tokenization diagnostics, expansion/runtime hardening, execution graph tooling, builtins/job-control stability, policy/sandbox controls, and release/traceability documentation updates. - Add shell grammar baseline and diagnostics primitives: - introduce `docs/reference/shell-grammar.md` with lexer modes and EBNF contract - add rich diagnostic tokenizer output (mode + span metadata) via `tokenizeBareShellLineDetailed` - export structured parse snapshot helpers (`bareOsShellAstSnapshot`) and shell error kinds - add determinism coverage for tokenizer and AST snapshot outputs - Harden expansion semantics and guardrails: - enforce expansion byte budgets (`BARE_OS_SHELL_EXPANSION_MAX_BYTES`) - add expansion trace hooks (`BARE_OS_SHELL_EXPANSION_TRACE`) with stage-level rows - add expansion recursion depth limits (`BARE_OS_SHELL_EXPANSION_MAX_DEPTH`) - tighten POSIX-mode arithmetic invalid-token diagnostics - preserve declared expansion ordering and document it in code/docs - Extend redirection/pipeline execution model: - add normalized redirection planner (`planShellRedirections`) independent of side effects - add execution graph builder/debug surface (`buildShellExecutionGraph`) - support `<<-` operator in tokenizer/parser paths - add pipeline stage timeout safety (`BARE_OS_SHELL_PIPELINE_STAGE_TIMEOUT_MS`) - keep pipefail/pipestatus behavior verified with integration tests - Improve builtins and control-flow reliability: - expand `read` builtin support: - `-r` raw mode - `-d` single-char delimiter - `-t` timeout semantics - refine wait/jobs semantics: - stable `jobs -l` parseable format expectations - synthetic pid mapping (`wait 410x`) and `wait all` support - keep trap registration/listing behavior deterministic and test-covered - add trap signal dispatch helper (`dispatchShellTrapSignal`) with normalization - Add security and policy enforcement hooks: - command deny/allow policy gates: - `BARE_OS_SHELL_DENY_COMMANDS` - `BARE_OS_SHELL_ALLOW_COMMANDS` - sandbox mode (`BARE_OS_SHELL_SANDBOX`) to block external command execution - redirect path safety guard (`BARE_OS_SHELL_REDIRECT_GUARD`) for pseudo-path/traversal risks - emit structured shell audit event rows (`ctx.shellAuditEvents`) for start/error/finish - Improve interactive UX resilience: - add prompt-hook timeout protection in fish readline: - `resolveShellPromptHookSegment` - env control `BARE_OS_SHELL_PROMPT_HOOK_TIMEOUT_MS` - ensure prompt segment resolution is non-blocking and safe on timeout/error - Add reliability/performance artifacts and shell fast lane: - add `scripts/bench-shell-phases.mjs` for shell microbench sanity checks - add `scripts/gen-shell-reliability-report.mjs` and generate reliability JSON artifact - add root scripts: - `test:shell-fast` - `bench:shell` - `report:shell-reliability` - Expand shell-focused docs and traceability: - add: - `docs/reference/shell-unsupported-behavior.md` - `docs/reference/shell-troubleshooting.md` - add contributor guides: - `developer-guide/17-how-to-add-shell-builtin.md` - `developer-guide/18-how-to-add-shell-grammar-feature.md` - update indexes/traceability/release gate docs: - `docs/reference/README.md` - `docs/reference/posix-issue7-traceability.md` - `docs/reference/environment-and-posix-appendix.md` - `docs/release-checklist.md` - `developer-guide/README.md` - `scripts/README.md` - Add and update shell regression tests in `packages/bare-os-booter/test.js` for: - tokenizer spans/modes and deterministic output - AST snapshot schema/shape - redirection planner and execution graph behavior - expansion trace and strict arithmetic paths - `<<-` support - pipeline stage timeout handling - `read` delimiter/raw/timeout semantics - jobs/wait parseability and selection semantics - trap dispatch and normalization behavior - policy/sandbox/redirect-guard/audit-event pathways Validation: - `npm run test -w bare-os-booter` - `npm run test:shell-bracket -w bare-os-booter` - `npm run test:shell-fast` - `npm run report:shell-reliability`
2026-04-26 23:17:20 -04:00
parent ef5a30fbef
commit e2b721d19f
25 changed files with 6028 additions and 3849 deletions
@@ -44,6 +44,9 @@ This directory holds the split **file-by-file inventory** that used to live in t
 - **Package: bare-os-seeder** (former §11) — [Package: bare-os-seeder](package-bare-os-seeder.md)
 - **Package: bare-os-booter** (former §§12.1–12.9) — [Package: bare-os-booter](package-bare-os-booter.md)
 - **Shell completion + Fish REPL editor** — [Shell completion and REPL editor](shell-completion-and-repl-editor.md)
+- **Shell grammar (draft EBNF / lexer modes)** — [shell-grammar.md](shell-grammar.md)
+- **Shell unsupported/intentional differences** — [shell-unsupported-behavior.md](shell-unsupported-behavior.md)
+- **Shell troubleshooting** — [shell-troubleshooting.md](shell-troubleshooting.md)
 - **bare-os-coreutils, tests, and seeder build hook** (former §§12.10–12.12) — [bare-os-coreutils, tests, and seeder build hook](package-bare-os-coreutils-and-ci.md)
 - **Architecture: end-to-end data flow** (former §13) — [Architecture: end-to-end data flow](architecture-data-flow.md)
 - **Kernel architecture contract** (booter vs image, boot steps) — [KERNEL_CONTRACT.md](../architecture/KERNEL_CONTRACT.md)
@@ -176,6 +176,8 @@ The list below is one **bullet per variable** in the form **name — component
 - `BARE_OS_SHELL_PARAM_EXPANSION` — Shell — When **`1`**, enable **`${VAR:-word}`** and **`${VAR#prefix}`** in **`expandWord`**.
 - `BARE_OS_SHELL_PARAM_EXPANSION_V2` — Shell — With param expansion on, enable **`${VAR:=word}`**, **`${VAR##*/}`** / **`${VAR#*/}`**, `**${VAR%%suffix}**` / `**${VAR%suffix}`** (bounded patterns).
 - `BARE_OS_SHELL_PARAM_EXPANSION_V3` — Shell — With param expansion on, enable **`${VAR:?word}`** and **`${VAR:+word}`** (POSIX-style error/alternate-value forms within documented bounds). Covered by **`expandWord param expansion v3`** in **`packages/bare-os-booter/test.js`**.
+- `BARE_OS_SHELL_EXPANSION_TRACE` — Shell — When **`1`** / **`true`**, append structured expansion stage rows to `ctx.shellExpansionTrace` (debug-only hook; off by default).
+- `BARE_OS_SHELL_EXPANSION_MAX_BYTES` — Shell — Caps total expansion output bytes per expanded word and command substitution (default **`262144`**, max **8 MiB**).
 - `BARE_OS_SHELL_POSIX_UNSET_ONLY_DEFAULT` — Shell — When **`1`** / **`true`** with param expansion on, enables `**${name-default}`** (default only when **unset**, distinct from **`${name:-default}`** when **unset or empty**).
 - `BARE_OS_ENV_DASH_S` — `/bin/env` — When **`1`**, enable `**-S` / `--split-string**` and **`--env-file`** (bounded).
 - `BARE_OS_VFS_WATCH_PSEUDO` — VFS — When **`1`**, allow **`vfs.watch`** on coalesced **`/proc/bare_os/metrics_live.json`** and polled **`/proc/bare_os/metrics.prom`** (and flat `**/proc/bare_os_metrics_*`** aliases).
@@ -221,7 +223,7 @@ The list below is one **bullet per variable** in the form **name — component
 - `BARE_OS_SHELL_PIPESTATUS` — Shell — When **`1`** / **`true`**, after each pipeline the space-separated stage exit codes are written to **`BARE_OS_PIPESTATUS`** in **`vfs.env`**.
 - `BARE_OS_SHELL_ERREXIT` — Shell — When **`1`** / **`true`**, or after **`set -e`**, stop running further top-level `**;**`-separated commands once a foreground command returns non-zero (subset of POSIX **errexit**; `**if` / `while` / `for`** condition lists are not affected the same way as bash — see handbook ch.9).
 - `BARE_OS_SHELL_NOUNSET` — Shell — When **`1`** / **`true`**, or after **`set -u`** / **`set -o nounset`**, expanding an unset simple parameter is an error (**`shell: unbound variable`**); see handbook ch.9.
- `BARE_OS_SHELL_READ_BUILTIN` — Shell — When **`1`** / **`true`**, enables the optional **`read`** builtin (bounded line from **`ctx.shellStdin`**, **`ctx.readLine`**, or interactive input; **`IFS`** split; **`-r`** accepted). Not full POSIX **`sh read`**.
+- `BARE_OS_SHELL_READ_BUILTIN` — Shell — When **`1`** / **`true`**, enables the optional **`read`** builtin (bounded line from **`ctx.shellStdin`**, **`ctx.readLine`**, or interactive input; **`IFS`** split; supports **`-r`** raw mode, **`-d DELIM`** single-char delimiter, and **`-t SECONDS`** timeout). Not full POSIX **`sh read`**.
 - `BARE_OS_SHELL_READ_MAX_BYTES` — Shell — Max raw bytes per **`read`** line when the read builtin is enabled (default **65536**, hard cap **2 MiB**).
 - `BARE_OS_VFS_WARM_CACHE_INVALIDATE_ON_APPEND` — Booter — When **`1`** / **`true`**, register **`append`** listeners on the system Hyperdrive metadata and blob Hypercores (when present) to clear **`/bin`** / **`lib/bare`** warm read caches on replication (**offline-first** safety; may increase churn on busy drives).
 - `BARE_OS_VFS_WARM_CACHE_PREFIX_INVALIDATE` — Booter — When **`1`** / **`true`**, evict warm-cache entries for **`/bin`**, **`/etc`**, **`/lib`**, **`/usr`**, and configured personal prefixes when replicated core lengths increase (see **`metrics_live.replicationLive.warmPrefixInvalidate`**).
@@ -267,6 +269,13 @@ The list below is one **bullet per variable** in the form **name — component
 - `BARE_OS_COLLAB_SESSION_NDJSON` — Booter — When **`1`** / **`true`**, host boot trace may append collaboration session lines (non-secret peer counts) via swarm-disk logging.
 - `BARE_OS_PHYS_PAGES_HINT` — Booter — Optional integer string for **`getconf _SC_PHYS_PAGES`** via **`ctx.bareOsGetconfSysconf`** (default **`524288`** when unset).
 - `BARE_OS_SHELL_HEREDOC_MAX_BYTES` — Booter — When **`BARE_OS_SHELL_POSIX_MODE`** is on, caps here-document body size (default **`262144`**, max **2 MiB**).
+- `BARE_OS_SHELL_EXEC_GRAPH_DUMP` — Shell — When **`1`** / **`true`**, stores `ctx.shellLastExecGraph` for the current command line (lists/and-or/pipelines plus normalized redirection plans).
+- `BARE_OS_SHELL_PIPELINE_STAGE_TIMEOUT_MS` — Shell — Optional per-stage timeout guard for pipeline execution. When set to a positive integer, shell stages that exceed this duration fail with a timeout diagnostic.
+- `BARE_OS_SHELL_PROMPT_HOOK_TIMEOUT_MS` — Shell — Timeout (ms) for optional interactive prompt hook segment resolution (`ctx.shellPromptHook`) in fish-style readline; defaults to **25ms** with empty-segment fallback.
+- `BARE_OS_SHELL_SANDBOX` — Shell — When **`1`** / **`true`**, block external `/bin` command execution and allow shell builtins only.
+- `BARE_OS_SHELL_DENY_COMMANDS` — Shell — Comma-separated denylist for command names evaluated before dispatch.
+- `BARE_OS_SHELL_ALLOW_COMMANDS` — Shell — Optional comma-separated allowlist; when set, commands outside the list are denied.
+- `BARE_OS_SHELL_REDIRECT_GUARD` — Shell — When **`1`** / **`true`**, deny redirect targets under sensitive pseudo paths (`/proc`, `/sys`, `/dev`) and simple traversal patterns.
 - `BARE_OS_SHELL_LOOP_MAX` — Shell — Max iterations for **`while`** / **`for`** (default **10000**).
 - `BARE_OS_SHELL_CASE_MAX_BRANCHES` — Shell — Max **`case`** branches (default **32**).
 - `BARE_OS_STRICT_POSIX` — Shell / utilities — When **`1`**, pathname globs that match nothing fail the command; tighter diagnostics elsewhere as documented.
@@ -7,7 +7,7 @@ This index links **IEEE Std 1003.1-2017 (Issue 7)** areas to **Bare OS** surface
 | XBD | Definitions / environment | Session **`vfs.env`**, **`/proc/self/environ`**, **`/proc/bare_os/security_posture.json`** | [environment-and-posix-appendix.md](environment-and-posix-appendix.md), handbook [ch.9](../../handbook/09-posix-utilities-shell-and-vfs.md) |
 | XSH | File I/O, metadata | **`ctx.bareOsSyscall`** ops + **`/proc/bare_os/syscalls.json`** (**`opsDetail`**, **`posixXsh`**, **`schemaVersion`**) | [POSIX_DECLARED_PROFILE.md](../architecture/POSIX_DECLARED_PROFILE.md), [posix-syscall-facade-map.md](posix-syscall-facade-map.md) |
 | XSH | Sockets (logical) | Socket bridge env (**`BARE_OS_POSIX_SOCKET_FD_BRIDGE`**, **`BARE_OS_POSIX_SOCKET_SCM_RIGHTS`**), **`socketMsgSurface`** in syscalls proc JSON | [syscall-socket-contract.md](syscall-socket-contract.md), handbook [ch.9](../../handbook/09-posix-utilities-shell-and-vfs.md) |
-| XCU | Shell | **`packages/bare-os-booter/lib/shell.js`**, env gates `**BARE_OS_SHELL_*`** | Handbook [ch.9 §3](../../handbook/09-posix-utilities-shell-and-vfs.md#3-shell-lists-pipelines-and-builtins-packagesbare-os-booterlibshelljs) |
+| XCU | Shell | **`packages/bare-os-booter/lib/shell.js`**, env gates `**BARE_OS_SHELL_*`** and signal trap dispatch helper (`dispatchShellTrapSignal`) | Handbook [ch.9 §3](../../handbook/09-posix-utilities-shell-and-vfs.md#3-shell-lists-pipelines-and-builtins-packagesbare-os-booterlibshelljs), [shell-grammar.md](shell-grammar.md), [shell-unsupported-behavior.md](shell-unsupported-behavior.md), [shell-troubleshooting.md](shell-troubleshooting.md) |
 | XCU | Utilities | **`/bin`** (from **`bare-os-coreutils`**), **`/etc/bare-os/posix_utilities.json`**, **`man`**, **`/share/man/man.json`** | Handbook [ch.9 §5](../../handbook/09-posix-utilities-shell-and-vfs.md#5-bin-utilities-catalog) |
 | XCU | **`awk`**, **`expr`**, `**test`/`[**` | Bounded engines in **`packages/bare-os-coreutils`**; profile **`BARE_OS_POSIX_PROFILE_VERSION`** + [posix-compliance-matrix.json](posix-compliance-matrix.json) **`susv4Refs`** rows | Handbook [ch.9 §7](../../handbook/09-posix-utilities-shell-and-vfs.md#7-awk-sed-grep-and-text-utils-packagesbare-os-coreutils) |
 | (informative) | P2P / replication | **`/proc/bare_os/replication`**, **`/proc/bare_os/swarm`**, **`/proc/bare_os/swarm_health.json`**, **`disk.os`** RPC, **`ctx.bareOsHrpcRequest`** (versioned route table) | [KERNEL_CONTRACT.md](../architecture/KERNEL_CONTRACT.md), handbook [ch.3](../../handbook/03-protocol-and-disk.md) |
@@ -0,0 +1,82 @@
+# Shell Grammar (Draft)
+
+This document defines a compact grammar for the BareOS shell surface.
+It is a conformance target for tokenizer/parser tests and diagnostics, not a
+promise of full POSIX `sh` parity.
+
+## Lexer Modes
+
+- `normal`: unquoted words and operators
+- `single`: `'...` literal text (no interpolation)
+- `double`: `\"...\"` with escapes and expansions
+- `arith`: `$(( ... ))`
+- `heredoc`: `<<WORD` / `<<-WORD` delimiter capture
+
+## Token Classes
+
+- `word`
+- `op` (`;`, `&&`, `||`, `|`, `<`, `>`, `>>`, `2>`, `2>>`, `2>&1`, `<<`, `<<<`, `(`, `)`, `{`, `}`)
+
+All diagnostic tokens should include:
+
+- byte/char span (`start`, `end`)
+- lexer mode tag
+
+## EBNF (Execution-Oriented)
+
+```ebnf
+line            = ws? andOrList? ws? ;
+andOrList       = pipeline { ws? ( "&&" | "||" | ";" ) ws? pipeline } ;
+pipeline        = command { ws? "|" ws? command } ;
+command         = compound | simple ;
+
+compound        = ifConstruct
+                | whileConstruct
+                | untilConstruct
+                | forConstruct
+                | caseConstruct
+                | groupedList ;
+
+groupedList     = "(" andOrList ")" | "{" andOrList "}" ;
+
+simple          = { assignment ws }? argv { ws redirection }* ;
+argv            = word { ws word } ;
+assignment      = ident "=" word ;
+ident           = ( "A".."Z" | "a".."z" | "_" ) { "A".."Z" | "a".."z" | "0".."9" | "_" } ;
+
+redirection     = "<" word
+                | ">" word
+                | ">>" word
+                | "2>" word
+                | "2>>" word
+                | "2>&1"
+                | "<<" word
+                | "<<<" word ;
+
+ifConstruct     = "if" ws andOrList ws "then" ws andOrList
+                  { ws "elif" ws andOrList ws "then" ws andOrList }?
+                  { ws "else" ws andOrList }?
+                  ws "fi" ;
+
+whileConstruct  = "while" ws andOrList ws "do" ws andOrList ws "done" ;
+untilConstruct  = "until" ws andOrList ws "do" ws andOrList ws "done" ;
+forConstruct    = "for" ws ident ws "in" ws word { ws word } ws ";" ws "do" ws andOrList ws "done" ;
+caseConstruct   = "case" ws word ws "in" ws caseArms ws "esac" ;
+caseArms        = caseArm { ws caseArm } ;
+caseArm         = pattern ws ")" ws andOrList ws ";;" ;
+pattern         = word { ws "|" ws word } ;
+
+word            = unquotedWord | singleQuotedWord | doubleQuotedWord | arithmeticWord | cmdSubstWord ;
+singleQuotedWord= "'" { anyCharExceptSingleQuote } "'" ;
+doubleQuotedWord= "\"" { escapedChar | anyCharExceptDoubleQuote } "\"" ;
+arithmeticWord  = "$((" { anyCharExceptArithClose } "))" ;
+cmdSubstWord    = "$(" andOrList ")" ;
+
+ws              = { " " | "\t" | "\n" } ;
+```
+
+## Notes
+
+- Parsing/expansion/runtime error phases should remain distinguishable.
+- This grammar intentionally omits unsupported POSIX constructs until adopted.
+- Deterministic AST snapshots should use this document as the canonical shape reference.
@@ -0,0 +1,27 @@
+# Shell troubleshooting reference
+
+Common signatures and first checks for `packages/bare-os-booter/lib/shell.js`.
+
+## Frequent failure signatures
+
+- `shell: arithmetic: invalid token`
+  - Check `$((...))` input for unsupported characters and unmatched delimiters.
+- `read: no input (redirect stdin, use a pipeline, or interactive readLine)`
+  - Run via pipeline/heredoc, or ensure interactive `readLine` is present.
+- `unbound variable: NAME`
+  - `nounset` mode is active; initialize variable or disable nounset for that block.
+- Unexpected pipeline status
+  - Inspect `set -o pipefail` state and last-stage versus first-failure semantics.
+- Here-doc expansion mismatches
+  - Confirm delimiter quoting and whether expansion is expected for that form.
+- `shell: command denied by policy: NAME` / `sandbox blocks external command`
+  - Check `BARE_OS_SHELL_DENY_COMMANDS`, `BARE_OS_SHELL_ALLOW_COMMANDS`, and `BARE_OS_SHELL_SANDBOX`.
+- `shell: pipeline stage timeout (...)`
+  - Increase `BARE_OS_SHELL_PIPELINE_STAGE_TIMEOUT_MS` or inspect the stalled command stage.
+
+## Fast debugging workflow
+
+- Reproduce with `sh -c '...'` minimal script.
+- Print `set -o` output to confirm shell option state.
+- Capture token/AST snapshots using `bareOsShellAstSnapshot(...)` in tests.
+- Re-run `npm run test -w bare-os-booter` before broader root test runs.
@@ -0,0 +1,24 @@
+# Shell unsupported / intentionally different behavior
+
+BareOS shell aims for practical POSIX-like behavior, not full `sh` compatibility.
+
+## Intentionally different today
+
+- No real process `fork`; pipelines are cooperative/simulated in one runtime.
+- No full POSIX grammar surface (feature-gated subsets by `BARE_OS_SHELL_*` env keys).
+- Job control is logical/session-scoped, not host-kernel TTY job control.
+- Signal delivery and trap timing prioritize deterministic guest behavior over host parity.
+- Arithmetic/expansion are bounded for memory safety and predictable failure modes.
+- Optional sandbox/policy mode can deny commands by name (`BARE_OS_SHELL_SANDBOX`, `BARE_OS_SHELL_DENY_COMMANDS`, `BARE_OS_SHELL_ALLOW_COMMANDS`), which intentionally differs from stock POSIX shells.
+
+## Why this is explicit
+
+- Keeps scripts deterministic in the single-address-space runtime.
+- Prevents hidden memory blowups from unbounded substitutions and heredocs.
+- Makes operator policy controls and test outcomes stable across Node/Bare lanes.
+
+## See also
+
+- `docs/reference/shell-grammar.md`
+- `docs/reference/posix-issue7-traceability.md`
+- `handbook/09-posix-utilities-shell-and-vfs.md`