Raven Scott
04b3a96dfb
Execute the reliability/flake hardening run across booter shell-adjacent suites,
focusing on deterministic assertions, unskipped first-party reliability tests,
repeat-run flake detection, and release/CI gate enforcement.
- Unskip and stabilize first-party reliability suites:
- `packages/bare-os-booter/test.bare-os-www.js`
- `packages/bare-os-booter/test.bare-os-ssh-holesail.js`
- convert previously skipped coverage into active deterministic checks
- add lifecycle hook stubs in suite test context for managed runtime paths
- reduce network/port race sensitivity by shifting fragile HTTP-bound checks
toward deterministic service/config/state validation where needed
- Replace weak reliability assertions in booter shell tests:
- remove broad `t.ok(true)`-style pass-through checks
- strengthen checks around here-doc/script mode and `sh -c` paths in
`packages/bare-os-booter/test.js` with explicit exit/error constraints
- Add reliability hard-gate automation:
- `scripts/verify-reliability-gates.mjs`
- fail when `test.skip` exists in first
31 lines
1.0 KiB
Markdown
31 lines
1.0 KiB
Markdown
# Reliability playbook
|
|
|
|
This playbook keeps booter/shell lanes merge-ready without retry-dependent success.
|
|
|
|
## Triage order
|
|
|
|
1. Reproduce with `npm run test:booter:www-ssh` for www/ssh-holesail.
|
|
2. Reproduce shell regressions with `npm run test:shell-fast`.
|
|
3. Run repeat lane `npm run test:flake-repeat` and inspect `docs/audit/shell-flake-repeat-report.json`.
|
|
4. Enforce static gates with `npm run verify:reliability-gates`.
|
|
|
|
## Unskip policy
|
|
|
|
- No `test.skip` in first-party reliability suites:
|
|
- `packages/bare-os-booter/test.bare-os-www.js`
|
|
- `packages/bare-os-booter/test.bare-os-ssh-holesail.js`
|
|
- No broad placeholder assertions (`t.ok(true)`) in critical booter/shell suites.
|
|
|
|
## Failure classes
|
|
|
|
- Port/race timing
|
|
- Async lifecycle leak
|
|
- Runtime parity mismatch (Node vs Bare)
|
|
- Environment-coupled assertions
|
|
|
|
## Monthly review checklist
|
|
|
|
- Review trend in `docs/audit/shell-flake-repeat-report.json`.
|
|
- Regenerate `docs/audit/shell-reliability-report.json`.
|
|
- Confirm release checklist reliability gates are still accurate.
|