snxraven/bare-operating-system
1
1
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Updates to CURL and WGET

Browse Source
This commit is contained in:
Raven Scott
2026-04-03 17:36:57 -04:00
parent 77638c4138
commit 9d836e8072
16 changed files with 561 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -1,4 +1,5 @@
node_modules/
# Legacy repo-root Corestore; default host data is ~/.bare-os/corestore/
data/
coverage/
.DS_Store
DOCUMENTATION.md
+9 -8
View File
@@ -1,6 +1,6 @@
# bare-operating-system — full codebase reference
This document describes **every first-party file** in the repository: purpose, contents, dependencies, and how pieces connect. It does **not** enumerate third-party code under `node_modules/` or VCS metadata under `.git/`. Runtime directories `data/` and `packages/*/.test-data/` are **gitignored** and hold Corestore/RocksDB state when you run the apps or tests; they are not source files.
This document describes **every first-party file** in the repository: purpose, contents, dependencies, and how pieces connect. It does **not** enumerate third-party code under `node_modules/` or VCS metadata under `.git/`. **Host** Corestore defaults live under **`~/.bare-os/corestore/`** (seeder and booter). Legacy **`data/`** at the repo root and **`packages/*/.test-data/`** remain **gitignored** if present; brittle tests use `.test-data/` under the booter package. None of these are source files.
For a **narrative how-to** on writing `async function run(ctx, argv)` and `start(ctx)` scripts, extending `/bin`, and separating host Pear code from in-image `AsyncFunction` execution, see [developer-guide/README.md](developer-guide/README.md).
@@ -104,7 +104,7 @@ No runtime dependencies at the root; all stack deps live in workspace packages.
## 4. Root: [README.md](README.md)
User-oriented documentation: project goal, layout table, prerequisites (Node 20+, Pear/Bare), `npm ci`, `npm test`, how to run seeder and booter with `node index.js`, environment variables, Corestore path semantics (repo-root `data/`), placeholder `pear://` table, protocol summary (`bare-os-v1`, MBR layout), license pointer.
User-oriented documentation: project goal, layout table, prerequisites (Node 20+, Pear/Bare), `npm ci`, `npm test`, how to run seeder and booter with `node index.js`, environment variables, Corestore path semantics (`~/.bare-os` defaults), placeholder `pear://` table, protocol summary (`bare-os-v1`, MBR layout), license pointer.
Narrative handbook (architecture diagrams, chapter walkthrough): [handbook/README.md](handbook/README.md). Per-workspace overviews: [packages/bare-os-protocol/README.md](packages/bare-os-protocol/README.md), [packages/bare-os-coreutils/README.md](packages/bare-os-coreutils/README.md), [packages/bare-os-seeder/README.md](packages/bare-os-seeder/README.md), [packages/bare-os-booter/README.md](packages/bare-os-booter/README.md), [kernel/README.md](kernel/README.md), [scripts/README.md](scripts/README.md).
@@ -121,7 +121,7 @@ Short Apache License, Version 2.0 header: copyright year 2026, standard AS-IS di
Ignores:
- `node_modules/`
- `data/` (Corestore for seeder/booter at repo root)
- `data/` (legacy; optional local Corestore if you still keep trees here — defaults now use `~/.bare-os`)
- `coverage/`
- `.DS_Store`
- `*.log`
@@ -275,7 +275,7 @@ Pear-safe path resolution (same idea as Holepunch [pear-rti](https://github.com/
- **`packageRootDir(metaUrl)`** — If `import.meta.url` is `file:`, `path.dirname(fileURLToPath(...))` via **`node:url`** (so `file:` is only passed to `fileURLToPath`). If `pear:` / other, `global.Pear.constructor.RTI.mount`, else `Pear.config.swapDir`, else `process.cwd()`.
- **`defaultKernelRoot(pkgRoot, metaUrl)`** — `BARE_OS_KERNEL_ROOT` or `path.join(pkgRoot, 'kernel')` (vendored copy under the seeder package for Pear bundles).
- **`defaultSeedCorestorePath(pkgRoot, metaUrl)`** — `BARE_OS_SEED_STORE` or `../../data` from `pkgRoot` when `file:`, else `pkgRoot/data` for Pear checkout.
- **`defaultSeedCorestorePath(pkgRoot, metaUrl)`** — `BARE_OS_SEED_STORE` or `path.join(hostDataRoot(), 'corestore', 'seeder')` where **`hostDataRoot()`** is `BARE_OS_HOST_DATA` (resolved) or `~/.bare-os`. Signature keeps `pkgRoot` / `metaUrl` for callers; defaults do not use them.
### 11.3 [packages/bare-os-seeder/index.js](packages/bare-os-seeder/index.js)
@@ -325,7 +325,7 @@ Pear-safe path resolution (same idea as Holepunch [pear-rti](https://github.com/
### 12.2 [packages/bare-os-booter/lib/paths.js](packages/bare-os-booter/lib/paths.js)
- **`packageRootDir(metaUrl)`** — Same as seeder (Pear RTI / `swapDir` / `cwd`).
- **`defaultBootCorestorePath`** / **`defaultLocalSeedCorestorePath`** — `../../data` when `file:`, else `pkgRoot/data`.
- **`defaultBootCorestorePath`** / **`defaultLocalSeedCorestorePath`** — `BARE_OS_BOOT_STORE` / `BARE_OS_LOCAL_SEED`, or under **`hostDataRoot()`** (`BARE_OS_HOST_DATA` or `~/.bare-os`): `corestore/booter` and `corestore/seeder` respectively. Same signature stability as the seeder helper.
### 12.3 [packages/bare-os-booter/index.js](packages/bare-os-booter/index.js)
@@ -492,8 +492,9 @@ sequenceDiagram
| Variable | Used by | Meaning |
| ------------------------- | ---------- | ---------------------------------------------------------------------------------------------- |
| `BARE_OS_KERNEL_ROOT` | Seeder | Absolute path to kernel tree (default: `repo/kernel`) |
| `BARE_OS_SEED_STORE` | Seeder | Corestore directory (default: `repo/data/corestore-seeder`) |
| `BARE_OS_BOOT_STORE` | Booter | Corestore for boot side (default: `repo/data/corestore-booter`) |
| `BARE_OS_HOST_DATA` | paths | Base directory for host state (default `~/.bare-os`; Corestore dirs live under `corestore/`) |
| `BARE_OS_SEED_STORE` | Seeder | Corestore directory (default: `~/.bare-os/corestore/seeder`) |
| `BARE_OS_BOOT_STORE` | Booter | Corestore for boot side (default: `~/.bare-os/corestore/booter`) |
| `BARE_OS_BOOT_TIMEOUT_MS` | Booter | Wall-clock budget for peer wait + network boot (default `60000`) |
| `BARE_OS_NO_SPLASH` | Booter | If `1`, skip TTY splash (plain logs / non-TTY behavior unchanged) |
| `BARE_OS_LOCAL_SEED` | paths | Overrides local seed path helper (`defaultLocalSeedCorestorePath`); booter does not local-boot |
@@ -524,7 +525,7 @@ Reference: Open Group POSIX.1-2017 utilities index; GNU coreutils (external) for
- **No** `node_modules` documentation (upstream packages).
- **No** line-by-line `package-lock.json` (machine-generated).
- **No** committed `data/` or `.test-data/` binary stores (runtime artifacts).
- **No** committed host Corestore trees, legacy `data/`, or `.test-data/` binary stores (runtime artifacts).
---
README.md
+7 -2
View File
@@ -54,7 +54,8 @@ Runs workspace tests (`brittle-node` / `brittle-bare` where configured).
Optional:
- `BARE_OS_KERNEL_ROOT` — absolute path to a kernel tree (defaults to repo `kernel/`).
- `BARE_OS_SEED_STORE` — Corestore directory (default `./data/corestore-seeder`).
- `BARE_OS_HOST_DATA` — base directory for host state (default `~/.bare-os`).
- `BARE_OS_SEED_STORE` — Corestore directory (default `~/.bare-os/corestore/seeder`).
2. **Booter** (another terminal):
@@ -64,11 +65,15 @@ Runs workspace tests (`brittle-node` / `brittle-bare` where configured).
```
Optional:
- `BARE_OS_HOST_DATA` — base directory for host state (default `~/.bare-os`).
- `BARE_OS_BOOT_STORE` — Corestore directory (default `~/.bare-os/corestore/booter`).
- `BARE_OS_BOOT_TIMEOUT_MS` — max ms for peer discovery + loading the kernel from the network (default `60000`). The booter does not fall back to a local seed; run the seeder on the same topic.
- `BARE_OS_NO_SPLASH=1` — disable the TTY boot splash.
- `BARE_OS_SKIP_REPL=1` — non-interactive kernel (CI / automation).
Corestore defaults (repo root, not temp dirs): `data/corestore-seeder` and `data/corestore-booter` (gitignored). Paths resolve from the monorepo root so you can run `node index.js` from either app package.
Host Corestore defaults live under **`~/.bare-os/corestore/`** (`seeder` and `booter`), not in the repo. Override the base with `BARE_OS_HOST_DATA`, or set `BARE_OS_SEED_STORE` / `BARE_OS_BOOT_STORE` for individual stores.
If you previously used gitignored **`data/`** at the repo root, move or copy `data/corestore-seeder` → `~/.bare-os/corestore/seeder` and `data/corestore-booter` → `~/.bare-os/corestore/booter`, or point the env vars at the old paths.
## Guest session and identity
handbook/07-operations-and-development.md
+4 -2
View File
@@ -15,7 +15,7 @@ This chapter is the **operator’s desk**: how to install, test, run Pear apps,
| `packages/bare-os-seeder` | Publish OS drive |
| `packages/bare-os-booter` | Network boot + runtime |
| `scripts/` | Pear `node_modules` fixer, etc. |
| `data/` | Gitignored Corestore dirs (default for dev) |
| `data/` | Gitignored legacy Corestore dir (optional; defaults use `~/.bare-os`) |
Each workspace has its own **`README.md`** with package-specific commands.
@@ -49,7 +49,7 @@ From **`packages/bare-os-booter`**:
node index.js
```
Corestore paths resolve under repo **`data/`** when using `file:` URLs (see each package’s `lib/paths.js`).
Default Corestore paths are **`~/.bare-os/corestore/seeder`** and **`~/.bare-os/corestore/booter`** (override base with **`BARE_OS_HOST_DATA`**, or set **`BARE_OS_SEED_STORE`** / **`BARE_OS_BOOT_STORE`**). See each package’s `lib/paths.js`.
---
@@ -71,7 +71,9 @@ These run **`scripts/ensure-pear-node-modules.mjs`** first. **Do not** run `pear
| Variable | Component | Meaning |
| ------------------------- | ------------- | -------------------------------- |
| `BARE_OS_KERNEL_ROOT` | Seeder | Override kernel tree path |
| `BARE_OS_HOST_DATA` | paths | Host state base (default `~/.bare-os`) |
| `BARE_OS_SEED_STORE` | Seeder | Corestore directory |
| `BARE_OS_BOOT_STORE` | Booter | Booter Corestore directory |
| `BARE_OS_BOOT_TIMEOUT_MS` | Booter | Boot deadline (default 60000) |
| `BARE_OS_NO_SPLASH` | Booter | Disable TTY splash |
| `BARE_OS_SKIP_REPL` | Booter | Non-interactive kernel |
kernel/share/man/man.json
+1 -1
View File
File diff suppressed because one or more lines are too long
packages/bare-os-booter/README.md
+3
View File
@@ -41,6 +41,9 @@ node index.js
| Variable | Meaning |
| ------------------------- | ------------------------------------------------------------- |
| `BARE_OS_HOST_DATA` | Base directory for host state (default `~/.bare-os`) |
| `BARE_OS_BOOT_STORE` | Corestore directory (default `~/.bare-os/corestore/booter`) |
| `BARE_OS_LOCAL_SEED` | Overrides `defaultLocalSeedCorestorePath` (booter does not local-boot) |
| `BARE_OS_BOOT_TIMEOUT_MS` | Total time to find peers + load kernel (default `60000`) |
| `BARE_OS_NO_SPLASH` | Disable TTY splash |
| `BARE_OS_SKIP_REPL` | Non-interactive kernel (`readLine` → null) |
packages/bare-os-booter/lib/curl-cli.js
+134 -25
View File
@@ -3,6 +3,13 @@
* @see https://curl.se/docs/manpage.html for flag inspiration; behavior is a subset.
*/
import b4a from 'b4a'
import {
DEFAULT_CURL_USER_AGENT,
isSupportedFetchUrl,
normalizeFetchUrl
} from './http-fetch-url.js'
/** @type {boolean} */
let bareFetchTried = false
@@ -37,14 +44,6 @@ function basicAuthHeader(user, pass) {
throw new Error('curl: cannot encode Basic auth (no btoa/Buffer)')
}
function looksLikeUrl(s) {
return (
/^https?:\/\//i.test(s) ||
s.startsWith('data:') ||
s.startsWith('file://')
)
}
function usage() {
return (
'usage: curl [options] URL...\n' +
@@ -52,8 +51,26 @@ function usage() {
)
}
/**
* Split -H line into headers[] or record User-Agent for argv-order precedence with -A.
* @param {string} line
* @param {string[]} headers
* @param {string[]} userAgentSequence
* @returns {boolean} false if malformed
*/
function pushCurlHeaderLine(line, headers, userAgentSequence) {
const colon = line.indexOf(':')
if (colon < 1) return false
const name = line.slice(0, colon).trim()
const value = line.slice(colon + 1).trim()
if (name.toLowerCase() === 'user-agent') userAgentSequence.push(value)
else headers.push(line)
return true
}
function utf8Encode(str) {
return new TextEncoder().encode(str)
// Bare may not define TextEncoder; b4a works on Node and Bare.
return b4a.from(String(str), 'utf8')
}
/** @param {(string | Uint8Array)[]} parts */
@@ -80,6 +97,47 @@ function expandWriteOut(fmt, vars) {
})
}
const CURL_MAX_REDIRECTS = 50
/**
* curl -I -L: send HEAD first; after any 3xx, follow with GET (curl man: "GET is used after redirect").
* Using fetch(HEAD, redirect:'follow') keeps HEAD on each hop and breaks many CDNs; manual hops match curl.
*/
async function fetchHeadWithLocationFollow(fetchFn, startUrl, hdr, signal) {
let url = startUrl
let method = 'HEAD'
for (let hop = 0; hop < CURL_MAX_REDIRECTS; hop++) {
const res = await fetchFn(url, {
method,
headers: hdr,
redirect: 'manual',
signal,
body: undefined
})
if (res.status >= 300 && res.status < 400) {
const loc = res.headers.get('Location')
if (!loc) return res
let nextUrl
try {
nextUrl = new URL(loc, url).href
} catch {
return res
}
method = 'GET'
try {
if (res.body && typeof res.body.cancel === 'function')
await res.body.cancel()
} catch {
/* ignore */
}
url = nextUrl
continue
}
return res
}
return new Response('', { status: 310, statusText: 'Too many redirects' })
}
/**
* @param {Record<string, unknown>} ctx
* @param {string[]} argv
@@ -90,6 +148,8 @@ export async function runCurlCli(ctx, argv) {
let method = ''
const headers = []
/** @type {string[]} -H User-Agent and -A/--user-agent in argv order; last wins */
const userAgentSequence = []
/** @type {string[]} */
const dataChunks = []
let dataBinary = false
@@ -112,15 +172,23 @@ export async function runCurlCli(ctx, argv) {
/** @type {string | null} */
let uploadPath = null
/** @type {string[]} */
const urls = []
let i = 0
while (i < args.length) {
const a = args[i]
if (a === '--') {
i++
while (i < args.length) {
urls.push(normalizeFetchUrl(String(args[i++])))
}
break
}
if (!a.startsWith('-') || a === '-') {
break
urls.push(normalizeFetchUrl(String(a)))
i++
continue
}
if (a === '-h' || a === '--help') {
@@ -154,12 +222,43 @@ export async function runCurlCli(ctx, argv) {
ctx.exitCode = 2
return
}
headers.push(String(args[++i]))
const line = String(args[++i])
if (!pushCurlHeaderLine(line, headers, userAgentSequence)) {
ctx.console.error('curl: malformed header: ' + line)
ctx.exitCode = 2
return
}
i++
continue
}
if (a.startsWith('-H') && a.length > 2) {
headers.push(a.slice(2))
const line = a.slice(2)
if (!pushCurlHeaderLine(line, headers, userAgentSequence)) {
ctx.console.error('curl: malformed header: ' + line)
ctx.exitCode = 2
return
}
i++
continue
}
if (a === '-A' || a === '--user-agent') {
if (i + 1 >= args.length) {
ctx.console.error('curl: option requires an argument: ' + a)
ctx.exitCode = 2
return
}
userAgentSequence.push(String(args[++i]))
i++
continue
}
if (a.startsWith('--user-agent=')) {
userAgentSequence.push(a.slice('--user-agent='.length))
i++
continue
}
if (a.startsWith('-A') && a.length > 2) {
userAgentSequence.push(a.slice(2))
i++
continue
}
@@ -340,11 +439,6 @@ export async function runCurlCli(ctx, argv) {
i++
}
const urls = []
while (i < args.length) {
urls.push(String(args[i++]))
}
if (urls.length === 0) {
ctx.console.error(usage())
ctx.exitCode = 2
@@ -352,7 +446,7 @@ export async function runCurlCli(ctx, argv) {
}
for (const u of urls) {
if (!looksLikeUrl(u)) {
if (!isSupportedFetchUrl(u)) {
ctx.console.error(
'curl: URL rejected (need http(s)://, data:, or file://): ' + u
)
@@ -401,12 +495,17 @@ export async function runCurlCli(ctx, argv) {
const pass = colon >= 0 ? userColonPass.slice(colon + 1) : ''
hdr.set('Authorization', basicAuthHeader(user, pass))
}
{
const ua = userAgentSequence.length
? userAgentSequence[userAgentSequence.length - 1]
: DEFAULT_CURL_USER_AGENT
hdr.set('User-Agent', ua)
}
/** @type {string | Uint8Array | undefined} */
let body = undefined
if (jsonBody) {
if (!hdr.has('Content-Type'))
hdr.set('Content-Type', 'application/json')
if (!hdr.has('Content-Type')) hdr.set('Content-Type', 'application/json')
body = jsonBody
} else if (dataChunks.length > 0) {
const joined = dataChunks.join('&')
@@ -422,7 +521,8 @@ export async function runCurlCli(ctx, argv) {
return
}
body = new Uint8Array(buf)
if (!hdr.has('Content-Type')) hdr.set('Content-Type', 'application/octet-stream')
if (!hdr.has('Content-Type'))
hdr.set('Content-Type', 'application/octet-stream')
}
let lastSize = 0
@@ -430,7 +530,8 @@ export async function runCurlCli(ctx, argv) {
for (let ui = 0; ui < urls.length; ui++) {
const url = urls[ui]
const outForUrl = urls.length > 1 && outputPath ? `${outputPath}.${ui}` : outputPath
const outForUrl =
urls.length > 1 && outputPath ? `${outputPath}.${ui}` : outputPath
const ac = maxTimeMs > 0 ? new AbortController() : null
const t =
@@ -446,6 +547,14 @@ export async function runCurlCli(ctx, argv) {
let res
try {
if (headOnly && location) {
res = await fetchHeadWithLocationFollow(
fetchFn,
url,
hdr,
ac ? ac.signal : undefined
)
} else {
res = await fetchFn(url, {
method: m,
headers: hdr,
@@ -453,6 +562,7 @@ export async function runCurlCli(ctx, argv) {
redirect: location ? 'follow' : 'manual',
signal: ac ? ac.signal : undefined
})
}
} catch (e) {
if (t) clearTimeout(t)
const msg = e && e.message ? e.message : String(e)
@@ -466,7 +576,7 @@ export async function runCurlCli(ctx, argv) {
if (!location && res.status >= 300 && res.status < 400) {
const loc = res.headers.get('Location')
if (loc && m === 'GET') {
if (loc && (m === 'GET' || m === 'HEAD')) {
if (!silent || showError)
ctx.console.error(
'curl: redirect not followed (use -L): ' + res.status + ' -> ' + loc
@@ -494,8 +604,7 @@ export async function runCurlCli(ctx, argv) {
const chunks = []
if (m === 'HEAD' || (includeHeaders && m !== 'HEAD')) {
const statusLine =
'HTTP/1.1 ' + res.status + ' ' + (res.statusText || '')
const statusLine = 'HTTP/1.1 ' + res.status + ' ' + (res.statusText || '')
chunks.push(statusLine + '\r\n')
res.headers.forEach((v, k) => {
chunks.push(k + ': ' + v + '\r\n')
packages/bare-os-booter/lib/http-fetch-url.js
+31
View File
@@ -0,0 +1,31 @@
/**
* URL normalization for fetch-based curl/wget subsets (scheme guessing like CLI tools).
*/
/** curl-style default (see https://everything.curl.dev/http/modify/user-agent.html); not a real libcurl build version. */
export const DEFAULT_CURL_USER_AGENT = 'curl/8.14.1'
/** GNU wget-style default (many builds use Wget/VERSION (linux-gnu)). */
export const DEFAULT_WGET_USER_AGENT = 'Wget/1.25.0 (linux-gnu)'
/** True if the string is usable by our fetch stack without guessing a scheme. */
export function isSupportedFetchUrl(s) {
return (
/^https?:\/\//i.test(s) || s.startsWith('data:') || s.startsWith('file://')
)
}
/**
* If there is no http(s)/data/file scheme, prepend http:// for host-like URLs (wget default).
* Protocol-relative //host → https://host. Leaves other schemes (ftp:, mailto:, …) unchanged.
* Does not rewrite `-`, ./ ../, or absolute /paths (not treated as remote URLs).
*/
export function normalizeFetchUrl(s) {
const t = String(s)
if (isSupportedFetchUrl(t)) return t
if (t.startsWith('//')) return 'https:' + t
if (/^[a-zA-Z][a-zA-Z0-9+.-]*:/.test(t)) return t
if (t === '-') return t
if (t.startsWith('/') || t.startsWith('./') || t.startsWith('../')) return t
return 'http://' + t
}
packages/bare-os-booter/lib/paths.js
+9 -10
View File
@@ -13,6 +13,13 @@ function env(name) {
return globalThis.process?.env?.[name]
}
/** Host-side state root (Corestore, etc.): `BARE_OS_HOST_DATA` or `~/.bare-os`. */
function hostDataRoot() {
const override = env('BARE_OS_HOST_DATA')
if (override) return path.resolve(override)
return path.join(os.homedir(), '.bare-os')
}
/** Booter has no `kernel/`; use `package.json` to detect the staged app root. */
function stagedAppRootHeuristic(root) {
try {
@@ -47,19 +54,11 @@ export function packageRootDir(metaUrl) {
export function defaultBootCorestorePath(pkgRoot, metaUrl) {
const override = env('BARE_OS_BOOT_STORE')
if (override) return override
const href = String(metaUrl)
const dataRoot = href.startsWith('file:')
? path.join(pkgRoot, '..', '..', 'data')
: path.join(pkgRoot, 'data')
return path.join(dataRoot, 'corestore-booter')
return path.join(hostDataRoot(), 'corestore', 'booter')
}
export function defaultLocalSeedCorestorePath(pkgRoot, metaUrl) {
const override = env('BARE_OS_LOCAL_SEED')
if (override) return override
const href = String(metaUrl)
const dataRoot = href.startsWith('file:')
? path.join(pkgRoot, '..', '..', 'data')
: path.join(pkgRoot, 'data')
return path.join(dataRoot, 'corestore-seeder')
return path.join(hostDataRoot(), 'corestore', 'seeder')
}
packages/bare-os-booter/lib/wget-cli.js
+157 -45
View File
@@ -4,6 +4,11 @@
*/
import path from 'path'
import {
DEFAULT_WGET_USER_AGENT,
isSupportedFetchUrl,
normalizeFetchUrl
} from './http-fetch-url.js'
/** @type {boolean} */
let bareFetchTried = false
@@ -31,14 +36,6 @@ async function ensureBareFetchForWget() {
}
}
function looksLikeUrl(s) {
return (
/^https?:\/\//i.test(s) ||
s.startsWith('data:') ||
s.startsWith('file://')
)
}
function defaultLocalName(urlString) {
try {
const u = new URL(urlString)
@@ -59,6 +56,104 @@ function usage() {
)
}
/**
* One argv token of short options: -qO-, -T30, -O file (via -O then next arg).
* @returns {number} next index into args, or -1 on error, -2 on help/version done
*/
function wgetConsumeShortCluster(a, args, i, st, ctx) {
let j = 1
while (j < a.length) {
const c = a[j]
if (c === 'q') {
st.quiet = true
j++
continue
}
if (c === 'h') {
ctx.console.log(usage())
ctx.exitCode = 0
return -2
}
if (c === 'V') {
ctx.console.log('wget (Bare OS fetch subset) 0.1 — not GNU wget2')
ctx.exitCode = 0
return -2
}
if (c === 'O') {
const tail = a.slice(j + 1)
if (tail !== '') {
st.outputDocument = tail
return i + 1
}
if (i + 1 >= args.length) {
ctx.console.error('wget: option requires an argument: -O')
ctx.exitCode = 2
return -1
}
st.outputDocument = String(args[i + 1])
return i + 2
}
if (c === 'P') {
const tail = a.slice(j + 1)
if (tail !== '') {
st.directoryPrefix = tail
return i + 1
}
if (i + 1 >= args.length) {
ctx.console.error('wget: option requires an argument: -P')
ctx.exitCode = 2
return -1
}
st.directoryPrefix = String(args[i + 1])
return i + 2
}
if (c === 'U') {
const tail = a.slice(j + 1)
if (tail !== '') {
st.userAgent = tail
return i + 1
}
if (i + 1 >= args.length) {
ctx.console.error('wget: option requires an argument: -U')
ctx.exitCode = 2
return -1
}
st.userAgent = String(args[i + 1])
return i + 2
}
if (c === 'T') {
const tail = a.slice(j + 1)
if (tail !== '') {
const sec = Number(tail)
if (!Number.isFinite(sec) || sec < 0) {
ctx.console.error('wget: invalid timeout')
ctx.exitCode = 2
return -1
}
st.timeoutMs = Math.round(sec * 1000)
return i + 1
}
if (i + 1 >= args.length) {
ctx.console.error('wget: option requires an argument: -T')
ctx.exitCode = 2
return -1
}
const sec = Number(args[i + 1])
if (!Number.isFinite(sec) || sec < 0) {
ctx.console.error('wget: invalid timeout')
ctx.exitCode = 2
return -1
}
st.timeoutMs = Math.round(sec * 1000)
return i + 2
}
ctx.console.error('wget: invalid option: -' + c)
ctx.exitCode = 2
return -1
}
return i + 1
}
/**
* @param {Record<string, unknown>} ctx
* @param {string[]} argv
@@ -67,28 +162,35 @@ export async function runWgetCli(ctx, argv) {
const vfs = ctx.vfs
const args = argv.slice(1)
let quiet = false
/** @type {string | null} */
let outputDocument = null
/** @type {string | null} */
let directoryPrefix = null
/** @type {string | null} */
let userAgent = null
let timeoutMs = 0
const extraHeaders = []
/** @type {string | null} */
let postData = null
/** @type {string | null} */
let postFile = null
const st = {
quiet: false,
outputDocument: /** @type {string | null} */ (null),
directoryPrefix: /** @type {string | null} */ (null),
userAgent: /** @type {string | null} */ (null),
timeoutMs: 0,
extraHeaders: /** @type {string[]} */ ([]),
postData: /** @type {string | null} */ (null),
postFile: /** @type {string | null} */ (null)
}
/** @type {string[]} */
const urls = []
let i = 0
while (i < args.length) {
const a = args[i]
if (a === '--') {
i++
while (i < args.length) {
urls.push(normalizeFetchUrl(String(args[i++])))
}
break
}
if (!a.startsWith('-') || a === '-') break
if (!a.startsWith('-') || a === '-') {
urls.push(normalizeFetchUrl(String(a)))
i++
continue
}
if (a === '-h' || a === '--help') {
ctx.console.log(usage())
@@ -100,7 +202,7 @@ export async function runWgetCli(ctx, argv) {
}
if (a === '-q' || a === '--quiet') {
quiet = true
st.quiet = true
i++
continue
}
@@ -111,12 +213,12 @@ export async function runWgetCli(ctx, argv) {
ctx.exitCode = 2
return
}
outputDocument = String(args[++i])
st.outputDocument = String(args[++i])
i++
continue
}
if (a.startsWith('--output-document=')) {
outputDocument = a.slice('--output-document='.length)
st.outputDocument = a.slice('--output-document='.length)
i++
continue
}
@@ -127,12 +229,12 @@ export async function runWgetCli(ctx, argv) {
ctx.exitCode = 2
return
}
directoryPrefix = String(args[++i])
st.directoryPrefix = String(args[++i])
i++
continue
}
if (a.startsWith('--directory-prefix=')) {
directoryPrefix = a.slice('--directory-prefix='.length)
st.directoryPrefix = a.slice('--directory-prefix='.length)
i++
continue
}
@@ -143,12 +245,12 @@ export async function runWgetCli(ctx, argv) {
ctx.exitCode = 2
return
}
userAgent = String(args[++i])
st.userAgent = String(args[++i])
i++
continue
}
if (a.startsWith('--user-agent=')) {
userAgent = a.slice('--user-agent='.length)
st.userAgent = a.slice('--user-agent='.length)
i++
continue
}
@@ -165,7 +267,7 @@ export async function runWgetCli(ctx, argv) {
ctx.exitCode = 2
return
}
timeoutMs = Math.round(sec * 1000)
st.timeoutMs = Math.round(sec * 1000)
i++
continue
}
@@ -176,7 +278,7 @@ export async function runWgetCli(ctx, argv) {
ctx.exitCode = 2
return
}
timeoutMs = Math.round(sec * 1000)
st.timeoutMs = Math.round(sec * 1000)
i++
continue
}
@@ -187,12 +289,12 @@ export async function runWgetCli(ctx, argv) {
ctx.exitCode = 2
return
}
extraHeaders.push(String(args[++i]))
st.extraHeaders.push(String(args[++i]))
i++
continue
}
if (a.startsWith('--header=')) {
extraHeaders.push(a.slice('--header='.length))
st.extraHeaders.push(a.slice('--header='.length))
i++
continue
}
@@ -203,12 +305,12 @@ export async function runWgetCli(ctx, argv) {
ctx.exitCode = 2
return
}
postData = String(args[++i])
st.postData = String(args[++i])
i++
continue
}
if (a.startsWith('--post-data=')) {
postData = a.slice('--post-data='.length)
st.postData = a.slice('--post-data='.length)
i++
continue
}
@@ -219,7 +321,7 @@ export async function runWgetCli(ctx, argv) {
ctx.exitCode = 2
return
}
postFile = String(args[++i])
st.postFile = String(args[++i])
i++
continue
}
@@ -230,15 +332,22 @@ export async function runWgetCli(ctx, argv) {
return
}
ctx.console.error('wget: invalid option: ' + a)
ctx.exitCode = 2
return
const ni = wgetConsumeShortCluster(a, args, i, st, ctx)
if (ni === -1) return
if (ni === -2) return
i = ni
}
const urls = []
while (i < args.length) {
urls.push(String(args[i++]))
}
const {
quiet,
outputDocument,
directoryPrefix,
userAgent,
timeoutMs,
extraHeaders,
postData,
postFile
} = st
if (urls.length === 0) {
ctx.console.error(usage())
@@ -247,7 +356,7 @@ export async function runWgetCli(ctx, argv) {
}
for (const u of urls) {
if (!looksLikeUrl(u)) {
if (!isSupportedFetchUrl(u)) {
ctx.console.error(
'wget: URL rejected (need http(s)://, data:, or file://): ' + u
)
@@ -290,7 +399,8 @@ export async function runWgetCli(ctx, argv) {
if (postFile) {
const buf = await vfs.readFile(postFile)
if (!buf) {
if (!quiet) ctx.console.error('wget: cannot read --post-file: ' + postFile)
if (!quiet)
ctx.console.error('wget: cannot read --post-file: ' + postFile)
ctx.exitCode = 3
return
}
@@ -312,6 +422,8 @@ export async function runWgetCli(ctx, argv) {
hdr.set(line.slice(0, colon).trim(), line.slice(colon + 1).trim())
}
if (userAgent) hdr.set('User-Agent', userAgent)
else if (!hdr.has('User-Agent'))
hdr.set('User-Agent', DEFAULT_WGET_USER_AGENT)
if (body !== undefined && !hdr.has('Content-Type'))
hdr.set('Content-Type', 'application/x-www-form-urlencoded')
packages/bare-os-booter/test.js
+172 -9
View File
@@ -36,8 +36,19 @@ import {
jobMatchesDate
} from './lib/bare-cron.js'
import { registerBareInitdDisposer, stopBareInitd } from './lib/bare-initd.js'
import {
DEFAULT_CURL_USER_AGENT,
DEFAULT_WGET_USER_AGENT
} from './lib/http-fetch-url.js'
const __dirname = path.dirname(fileURLToPath(import.meta.url))
function headerFromInit(init, name) {
if (!init?.headers) return undefined
const h = init.headers
if (typeof h.get === 'function') return h.get(name)
return undefined
}
function testCorestoreDir(name) {
const dir = path.join(
__dirname,
@@ -446,11 +457,10 @@ test('expandArgvAliases expands first word and keeps trailing argv', async (t) =
})
test('expandArgvAliases leaves sed unchanged', async (t) => {
t.alike(expandArgvAliases(['sed', 's/a/b/', 'x.txt'], defaultShellAliases()), [
'sed',
's/a/b/',
'x.txt'
])
t.alike(
expandArgvAliases(['sed', 's/a/b/', 'x.txt'], defaultShellAliases()),
['sed', 's/a/b/', 'x.txt']
)
})
test('defaultShellAliases does not remap sed', async (t) => {
@@ -758,14 +768,19 @@ test('curl delegated from booter with stub fetch', async (t) => {
lines.push('e:' + String(s))
}
}
ctx.httpFetch = async () =>
new Response('hello', {
/** @type {RequestInit | undefined} */
let curlInitHello
ctx.httpFetch = async (_url, init) => {
curlInitHello = init
return new Response('hello', {
status: 200,
headers: { 'Content-Type': 'text/plain' }
})
}
await runBinCommand(ctx, ['curl', 'https://stub.example/x'])
t.is(ctx.exitCode, 0)
t.is(lines[0], 'hello')
t.is(headerFromInit(curlInitHello, 'User-Agent'), DEFAULT_CURL_USER_AGENT)
lines.length = 0
ctx.exitCode = 0
@@ -773,6 +788,112 @@ test('curl delegated from booter with stub fetch', async (t) => {
await runBinCommand(ctx, ['curl', '-f', '-s', 'https://stub.example/missing'])
t.is(ctx.exitCode, 22)
lines.length = 0
ctx.exitCode = 0
let curlSeenUrl = ''
/** @type {RequestInit | undefined} */
let curlSeenInit
ctx.httpFetch = async (url, init) => {
curlSeenUrl = String(url)
curlSeenInit = init
return new Response('', { status: 200 })
}
await runBinCommand(ctx, ['curl', 'example.com', '-IL'])
t.is(ctx.exitCode, 0)
t.is(curlSeenUrl, 'http://example.com')
t.is(curlSeenInit && curlSeenInit.method, 'HEAD')
lines.length = 0
ctx.exitCode = 0
/** @type {{ url: string, method: string }[]} */
const curlHopCalls = []
ctx.httpFetch = async (url, init) => {
const u = String(url)
const method = String(init?.method || 'GET')
curlHopCalls.push({ url: u, method })
if (method === 'HEAD' && u === 'http://hop.example/start') {
return new Response('', {
status: 302,
headers: { Location: 'https://hop.example/done' }
})
}
if (method === 'GET' && u === 'https://hop.example/done') {
return new Response('secret-body', {
status: 200,
headers: { 'Content-Type': 'text/plain' }
})
}
return new Response('unexpected', { status: 500 })
}
await runBinCommand(ctx, ['curl', '-IL', 'http://hop.example/start'])
t.is(ctx.exitCode, 0)
t.is(curlHopCalls.length, 2)
t.is(curlHopCalls[0].method, 'HEAD')
t.is(curlHopCalls[1].method, 'GET')
t.ok(lines[0].includes('200'))
t.ok(!lines[0].includes('secret-body'))
lines.length = 0
ctx.exitCode = 0
ctx.httpFetch = async (url) => {
curlSeenUrl = String(url)
return new Response('', { status: 200 })
}
await runBinCommand(ctx, ['curl', '-I', '//stub.example/p'])
t.is(ctx.exitCode, 0)
t.is(curlSeenUrl, 'https://stub.example/p')
lines.length = 0
ctx.exitCode = 0
/** @type {RequestInit | undefined} */
let curlInitUa
ctx.httpFetch = async (_url, init) => {
curlInitUa = init
return new Response('x', { status: 200 })
}
await runBinCommand(ctx, [
'curl',
'-A',
'CustomCurlUA/9',
'https://stub.example/ua'
])
t.is(ctx.exitCode, 0)
t.is(headerFromInit(curlInitUa, 'User-Agent'), 'CustomCurlUA/9')
lines.length = 0
ctx.exitCode = 0
ctx.httpFetch = async (_url, init) => {
curlInitUa = init
return new Response('x', { status: 200 })
}
await runBinCommand(ctx, [
'curl',
'-H',
'User-Agent: from-header',
'-A',
'from-flag',
'https://stub.example/ua2'
])
t.is(ctx.exitCode, 0)
t.is(headerFromInit(curlInitUa, 'User-Agent'), 'from-flag')
lines.length = 0
ctx.exitCode = 0
ctx.httpFetch = async (_url, init) => {
curlInitUa = init
return new Response('x', { status: 200 })
}
await runBinCommand(ctx, [
'curl',
'-A',
'from-flag',
'-H',
'User-Agent: from-header',
'https://stub.example/ua3'
])
t.is(ctx.exitCode, 0)
t.is(headerFromInit(curlInitUa, 'User-Agent'), 'from-header')
await store.close()
rmSync(dir, { recursive: true, force: true })
})
@@ -795,11 +916,15 @@ test('wget delegated from booter with stub fetch', async (t) => {
lines.push('e:' + String(s))
}
}
ctx.httpFetch = async () =>
new Response('payload', {
/** @type {RequestInit | undefined} */
let wgetInitPayload
ctx.httpFetch = async (_url, init) => {
wgetInitPayload = init
return new Response('payload', {
status: 200,
headers: { 'Content-Type': 'text/plain' }
})
}
await runBinCommand(ctx, [
'wget',
'-O',
@@ -808,6 +933,7 @@ test('wget delegated from booter with stub fetch', async (t) => {
'https://stub.example/a'
])
t.is(ctx.exitCode, 0)
t.is(headerFromInit(wgetInitPayload, 'User-Agent'), DEFAULT_WGET_USER_AGENT)
const out = await ctx.vfs.readFile('saved.txt')
t.ok(out)
t.is(ctx.b4a.toString(out), 'payload')
@@ -825,6 +951,43 @@ test('wget delegated from booter with stub fetch', async (t) => {
t.is(ctx.exitCode, 0)
t.is(lines[0], 'stdout-body')
lines.length = 0
ctx.exitCode = 0
let wgetSeenUrl = ''
ctx.httpFetch = async (url) => {
wgetSeenUrl = String(url)
return new Response('p2', {
status: 200,
headers: { 'Content-Type': 'text/plain' }
})
}
await runBinCommand(ctx, ['wget', 'stub.example/z', '-qO', 'w2.txt'])
t.is(ctx.exitCode, 0)
t.is(wgetSeenUrl, 'http://stub.example/z')
const w2 = await ctx.vfs.readFile('w2.txt')
t.ok(w2)
t.is(ctx.b4a.toString(w2), 'p2')
lines.length = 0
ctx.exitCode = 0
/** @type {RequestInit | undefined} */
let wgetInitU
ctx.httpFetch = async (_url, init) => {
wgetInitU = init
return new Response('u', { status: 200 })
}
await runBinCommand(ctx, [
'wget',
'-q',
'-U',
'WgetCustom/1',
'-O',
'u.txt',
'https://stub.example/u'
])
t.is(ctx.exitCode, 0)
t.is(headerFromInit(wgetInitU, 'User-Agent'), 'WgetCustom/1')
await store.close()
rmSync(dir, { recursive: true, force: true })
})
packages/bare-os-coreutils/man/pages/curl.json
+6 -8
View File
@@ -16,6 +16,10 @@
"flag": "-H, --header LINE",
"meaning": "Request header (repeatable)"
},
{
"flag": "-A, --user-agent STRING",
"meaning": "Set User-Agent (default curl/VERSION-style string; last -A or -H User-Agent on the command line wins)"
},
{
"flag": "-d, --data / --json",
"meaning": "Request body"
@@ -33,14 +37,8 @@
"meaning": "See full man curl.json"
}
],
"keywords": [
"curl",
"http",
"https",
"fetch",
"download"
],
"bareOsNotes": "Not https://curl.se libcurl; booter curl-cli.js. URLs: http(s), data:, file://.",
"keywords": ["curl", "http", "https", "fetch", "download"],
"bareOsNotes": "Not https://curl.se libcurl; booter curl-cli.js. Default User-Agent matches curl/VERSION form (see everything.curl.dev user-agent). Options and URLs may be interleaved (POSIX-style). Host-style URLs without a scheme get http://; protocol-relative //host gets https://. Fetch uses http(s), data:, or file:// after normalization.",
"seeAlso": [
{
"name": "git",
packages/bare-os-coreutils/man/pages/wget.json
+9 -16
View File
@@ -22,7 +22,7 @@
},
{
"flag": "-U, --user-agent STRING",
"meaning": "Set User-Agent request header"
"meaning": "Set User-Agent (default Wget/VERSION (linux-gnu)-style when -U and --header User-Agent are absent)"
},
{
"flag": "-T, --timeout SECONDS",
@@ -50,15 +50,8 @@
}
],
"environment": [],
"keywords": [
"wget",
"download",
"http",
"https",
"fetch",
"mirror"
],
"bareOsNotes": "Not GNU wget2 (https://gitlab.com/gnuwget/wget2). URLs must start with http://, https://, data:, or file://. No recursive retrieval, FTP, or WARC. Cannot combine -O and -P. On Pear/Bare, use bare-fetch for global fetch. Tests may set ctx.httpFetch.",
"keywords": ["wget", "download", "http", "https", "fetch", "mirror"],
"bareOsNotes": "Not GNU wget2 (https://gitlab.com/gnuwget/wget2); booter wget-cli.js. Default User-Agent matches GNU wget-style Wget/VERSION (linux-gnu). -U overrides; --header User-Agent is used if present unless -U is set. Options and URLs may be interleaved; short options can be clustered (e.g. -qO-, -T30). Host-style URLs without a scheme get http://; //host gets https://. No recursive retrieval, FTP, or WARC. Cannot combine -O and -P. Tests may set ctx.httpFetch.",
"seeAlso": [
{
"name": "curl",
@@ -88,11 +81,11 @@
}
],
"exitStatus": [
"0 — success",
"1 — generic error (reserved)",
"2 — bad usage or options",
"3 — file I/O error (e.g. --post-file unreadable)",
"4 — network failure or no fetch implementation",
"8 — HTTP 4xx/5xx response"
"0 \u2014 success",
"1 \u2014 generic error (reserved)",
"2 \u2014 bad usage or options",
"3 \u2014 file I/O error (e.g. --post-file unreadable)",
"4 \u2014 network failure or no fetch implementation",
"8 \u2014 HTTP 4xx/5xx response"
]
}
packages/bare-os-seeder/README.md
+2 -1
View File
@@ -27,7 +27,8 @@ npm run os:seeder
| Variable | Meaning |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `BARE_OS_KERNEL_ROOT` | Absolute path to kernel tree to stage (default: package `kernel/` vendored copy, or repo `kernel/` under Node `file:` URL) |
| `BARE_OS_SEED_STORE` | Corestore directory (default under repo `data/` when resolving from `file:`) |
| `BARE_OS_HOST_DATA` | Base directory for host state (default `~/.bare-os`) |
| `BARE_OS_SEED_STORE` | Corestore directory (default `~/.bare-os/corestore/seeder`) |
## Pear vs Node
packages/bare-os-seeder/kernel/share/man/man.json
+1 -1
View File
File diff suppressed because one or more lines are too long
packages/bare-os-seeder/lib/paths.js
+8 -5
View File
@@ -13,6 +13,13 @@ function env(name) {
return globalThis.process?.env?.[name]
}
/** Host-side state root (Corestore, etc.): `BARE_OS_HOST_DATA` or `~/.bare-os`. */
function hostDataRoot() {
const override = env('BARE_OS_HOST_DATA')
if (override) return path.resolve(override)
return path.join(os.homedir(), '.bare-os')
}
function kernelDirPresent(root) {
try {
return statSync(path.join(root, 'kernel')).isDirectory()
@@ -59,9 +66,5 @@ export function defaultKernelRoot(pkgRoot, metaUrl) {
export function defaultSeedCorestorePath(pkgRoot, metaUrl) {
const override = env('BARE_OS_SEED_STORE')
if (override) return override
const href = String(metaUrl)
const dataRoot = href.startsWith('file:')
? path.join(pkgRoot, '..', '..', 'data')
: path.join(pkgRoot, 'data')
return path.join(dataRoot, 'corestore-seeder')
return path.join(hostDataRoot(), 'corestore', 'seeder')
}