pearcord/pearcord-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Phase 636 guild sync pipeline and Phase 637 roadmap

Browse Source 
Document read receipt vectors, thread archive slices, member page debounce,
and mark Phase 636 completion; add 50-item Phase 637 sparse ACK and stage mesh plan.

Co-authored-by: Cursor <[email protected]>
This commit is contained in:
Raven Scott
2026-06-01 19:25:39 -04:00
co-authored by Cursor
parent fcfd1a7127
commit 430195ef21
3 changed files with 182 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
GUILD_REPLICATION.md
+41
View File
@@ -48,6 +48,47 @@ Owner / peer on GUILD_SYNC_REQUEST
- Optional **`PEARCORD_GUILD_SYNC_ZLIB=1`** gzip wrapper on large bundles (`_compressed` + `_payload`).
- Diagnostics: `exportGuildSyncDiagnostics()`, UI export + **guild-sync-errors** dev-log filter.
**v0.8.608 (Phase 636):**
- **Read receipt vector** — `readReceiptVector[]` on `GUILD_SYNC`; ingest LWW per `(userId, channelId)` clamped to message tail.
- **Thread archive delta** — `threadArchiveSlice[]` for thread `archived` + `updatedAt`; ingest via `guild.updateChannel`.
- **Forum tag palettes** — `forumTagPalettes[]` with `paletteVersion` on full bundles.
- **Voice occupancy ingest** — `voiceOccupancy[]` merged with LWW on `joinedAt` via `PearcordVoice#ingestGossip`.
- **Member page debounce** — `PEARCORD_GUILD_SYNC_MEMBER_PAGE_DEBOUNCE_MS` (default 600) coalesces `MEMBER_PAGE_REQUEST` storms; span **`guild.sync.member-page`**.
- **Push QoS** — RTT-scaled zlib threshold (4–8 KiB); `pushPeerSlot` round-robin metadata on `guildSyncHealth`.
| Variable | Default | Meaning |
|----------|---------|---------|
| `PEARCORD_GUILD_SYNC_MEMBER_PAGE_DEBOUNCE_MS` | `600` | Debounce auto member page chain |
| Script | Marker |
|--------|--------|
| `npm run test:guild-sync-read-receipt-vector` | `GUILD_SYNC_READ_RECEIPT_VECTOR_SMOKE_OK` |
| `npm run test:guild-sync-thread-archive` | `GUILD_SYNC_THREAD_ARCHIVE_SMOKE_OK` |
| `npm run test:guild-sync-member-page-debounce` | `GUILD_SYNC_MEMBER_PAGE_DEBOUNCE_SMOKE_OK` |
| `npm run test:phase636-guild-sync` | Phase 636 bundle (chains phase635) |
**v0.8.607 (Phase 635):**
- **Partial member roster** — `GUILD_SYNC` caps members at `PEARCORD_GUILD_SYNC_MEMBER_CAP` (default 100) with `membersPartial`, `memberPageNextOffset`, and paged fetch via **`MEMBER_PAGE_REQUEST` (RPC 92)** / **`MEMBER_PAGE` (RPC 93)**.
- **Attachment priority** — pull scheduler prefers `activeChannelId` attachments first; span **`guild.sync.attach`**; `attachmentPullQueueDepth` on health.
- **RTT-adaptive bursts** — `adaptiveBurstDelays` from `pearcord-swarm-manager`; peer join samples RTT for `_scheduleGuildSyncPushBurst`.
- **Delta slices** — `forumIndex[]`, `voiceOccupancy[]`, search index `version`, ingest metrics (`lastIngestMs`, `lastIngestMode`).
- **Thread ingest** — idempotent `rootMessageId` on duplicate `GUILD_SYNC` channel rows.
| Variable | Default | Meaning |
|----------|---------|---------|
| `PEARCORD_GUILD_SYNC_MEMBER_CAP` | `100` | Max members inline in one bundle |
| `PEARCORD_GUILD_SYNC_MEMBER_PAGE_SIZE` | `50` | Members per `MEMBER_PAGE` |
| `PEARCORD_GUILD_SYNC_RTT_MS` | off | Override RTT estimate for burst scaling |
| Script | Marker |
|--------|--------|
| `npm run test:guild-sync-partial-roster` | `GUILD_SYNC_PARTIAL_ROSTER_SMOKE_OK` |
| `npm run test:guild-sync-member-page` | `GUILD_SYNC_MEMBER_PAGE_SMOKE_OK` |
| `npm run test:guild-sync-attach-priority` | `GUILD_SYNC_ATTACH_PRIORITY_SMOKE_OK` |
| `npm run test:phase635-guild-sync` | Phase 635 bundle (chains phase634) |
**v0.8.606 (Phase 634):**
- **Attachment pull scheduler** — after `GUILD_SYNC` ingest, `_scheduleGuildSyncAttachmentPulls` prefetches up to `PEARCORD_GUILD_ATTACHMENT_PULL_CAP` (default 8) remote metadata rows via mesh/Hyperdrive.
GUILD_SYNC.md
+4
View File
@@ -99,6 +99,8 @@ Skipped when the local peer has no messages **and** no members to share.
**v0.8.606:** Attachment metadata pull scheduler after ingest (`PEARCORD_GUILD_ATTACHMENT_PULL_CAP`). Bandwidth QoS on push (`PEARCORD_GUILD_SYNC_BYTES_PER_SEC`). HyperDB-safe uint coercion on messages/members. Delta reaction tombstones, `pins[]` slice, contacts↔guild presence reconcile. `npm run test:phase634-guild-sync`.
**v0.8.607:** Partial member roster in `GUILD_SYNC` (`membersPartial`, `memberPageNextOffset`). Paged members via RPC **92** `MEMBER_PAGE_REQUEST` / **93** `MEMBER_PAGE`. Attachment pull prioritizes active channel. RTT-scaled sync push bursts. `forumIndex` + `voiceOccupancy` delta slices. `npm run test:phase635-guild-sync`.
**v0.8.30:** Bundles always include `members` from `listMembers()` so late joiners see the roster (including role changes) without relying on replayed `MEMBER_JOIN` gossip.
## Ingest
@@ -128,3 +130,5 @@ Skipped when the local peer has no messages **and** no members to share.
Host sends five messages before invite; guest joins, mesh connects, and receives full history without manual replay.
Live roster smoke: owner + mod on mesh → `MEMBER_ROLE_UPDATE` → third peer joins and ingests roster with moderator role. Presence smoke: host `setPresence` → guest `view.presence.peers`.
**v0.8.608 (Phase 636):** Delta/full bundles may include `readReceiptVector[]` (`userId`, `channelId`, `lastReadAt` — LWW merge clamped to channel message tail), `threadArchiveSlice[]` (`channelId`, `archived`, `updatedAt`), and `forumTagPalettes[]` (`channelId`, `tags`, `paletteVersion`). Joiners ingest `voiceOccupancy` via voice `ingestGossip` LWW. Member page auto-chain uses debounced `MEMBER_PAGE_REQUEST` (`PEARCORD_GUILD_SYNC_MEMBER_PAGE_DEBOUNCE_MS`, default 600). Env tuning: `PEARCORD_GUILD_SYNC_MEMBER_CAP`, `PEARCORD_GUILD_SYNC_MEMBER_PAGE_SIZE`. Bundle: `npm run test:phase636-guild-sync`.
PLATFORM_ROADMAP.md
+137 -17
View File
@@ -18953,19 +18953,19 @@ P2P-first phase: Hyperswarm guild mesh, `GUILD_SYNC` bundles, join/recovery pipe
---
### Phase 635 — Large guild partial mesh & replication scale (v0.8.607)
### Phase 635 — Large guild partial mesh & replication scale (v0.8.607) ✅
| ID | Item | Status |
|----|------|--------|
| P635-1 | Channel-scoped `GUILD_SYNC` without full member roster (>100 members) | [ ] |
| P635-2 | Lazy member page fetch over mesh `MEMBER_PAGE_REQUEST` gossip | [ ] |
| P635-3 | Hyperdrive attachment priority queue (visible channel first) | [ ] |
| P635-4 | Bandwidth QoS per-peer fairness (round-robin push slots) | [ ] |
| P635-5 | RTT-adaptive `GUILD_SYNC` burst intervals from mesh RTT samples | [ ] |
| P635-6 | Joiner sync duration metrics: gossip vs hypercore vs sparse | [ ] |
| P635-7 | Forum post index slice in delta bundles | [ ] |
| P635-8 | Thread parent idempotency on duplicate `GUILD_SYNC` ingest | [ ] |
| P635-9 | Voice occupancy low-bandwidth slice in delta bundles | [ ] |
| P635-1 | Channel-scoped `GUILD_SYNC` without full member roster (>100 members) | [x] |
| P635-2 | Lazy member page fetch over mesh `MEMBER_PAGE_REQUEST` gossip | [x] |
| P635-3 | Hyperdrive attachment priority queue (visible channel first) | [x] |
| P635-4 | Bandwidth QoS per-peer fairness (round-robin push slots) | [~] |
| P635-5 | RTT-adaptive `GUILD_SYNC` burst intervals from mesh RTT samples | [x] |
| P635-6 | Joiner sync duration metrics: gossip vs hypercore vs sparse | [x] |
| P635-7 | Forum post index slice in delta bundles | [x] |
| P635-8 | Thread parent idempotency on duplicate `GUILD_SYNC` ingest | [x] |
| P635-9 | Voice occupancy low-bandwidth slice in delta bundles | [x] |
| P635-10 | Discovery join: await guild sync + discovery listing refresh | [ ] |
| P635-11 | Explore join passes `hostPublicKey` + bounded sync wait UI | [ ] |
| P635-12 | Multi-device `guildSyncHealth` on settings mesh snapshot | [ ] |
@@ -19000,15 +19000,133 @@ P2P-first phase: Hyperswarm guild mesh, `GUILD_SYNC` bundles, join/recovery pipe
| P635-41 | Smoke: large guild 500-channel partial sync | [ ] |
| P635-42 | Smoke: attachment priority queue order | [ ] |
| P635-43 | Smoke: member page lazy fetch | [ ] |
| P635-44 | `test:phase635-guild-sync` bundle | [ ] |
| P635-45 | Span `guild.sync.attach` for pull scheduler | [ ] |
| P635-46 | Dev-log filter **guild-sync-attach** | [ ] |
| P635-47 | Dev-log filter **guild-sync-bandwidth** | [ ] |
| P635-48 | Module README org-wide replication audit | [ ] |
| P635-44 | `test:phase635-guild-sync` bundle | [x] |
| P635-45 | Span `guild.sync.attach` for pull scheduler | [x] |
| P635-46 | Dev-log filter **guild-sync-attach** | [x] |
| P635-47 | Dev-log filter **guild-sync-bandwidth** | [x] |
| P635-48 | Module README org-wide replication audit | [~] |
| P635-49 | E2E two-peer report refresh for v0.8.607 | [ ] |
| P635-50 | Phase 636 — CRDT read receipts & thread sync (v0.8.608) | [ ] |
| P635-50 | Phase 636 — CRDT read receipts & thread sync (v0.8.608) | [x] |
**Next:** Phase 635 — scale partial mesh for large guilds, Hyperdrive QoS, RTT-adaptive sync, lazy member pages.
**Next:** Phase 636 — CRDT read receipts, thread sync depth, mesh-cluster live DHT, large-guild channel paging.
---
### Phase 636 — CRDT read receipts & thread sync depth (v0.8.608)
| ID | Item | Status |
|----|------|--------|
| P636-1 | Per-channel read receipt vector in `GUILD_SYNC` delta | [x] |
| P636-2 | LWW merge for read cursor vs message tail on ingest | [x] |
| P636-3 | Thread archive state sync slice in delta bundles | [x] |
| P636-4 | Thread slowmode inherit from parent on sync ingest | [x] |
| P636-5 | Forum tag palette version field in delta bundles | [x] |
| P636-6 | Member page auto-chain debounce (avoid request storms) | [x] |
| P636-7 | Bandwidth QoS per-peer round-robin push slots | [x] |
| P636-8 | Zlib auto-threshold from estimated peer downlink | [x] |
| P636-9 | Sparse ACK-gated batch download windows | [ ] |
| P636-10 | Corestore replicate eager open on all guild members | [ ] |
| P636-11 | Hypercore compaction after channel archive gossip | [~] |
| P636-12 | Hyperbee legacy index migration CLI | [ ] |
| P636-13 | Guild search index incremental version bump | [~] |
| P636-14 | Audit export cursor persistence in sidecar JSON | [ ] |
| P636-15 | Partial reaction window scoped to visible messages | [ ] |
| P636-16 | UI dev panel: read receipt sync status per channel | [~] |
| P636-17 | UI dev panel: member page fetch progress | [~] |
| P636-18 | Agentctl `exportGuildSyncDiagnostics` journey step | [~] |
| P636-19 | Agentctl 3-peer live mesh-cluster DHT scenario | [ ] |
| P636-20 | Mesh `connection reset` recovery smoke | [ ] |
| P636-21 | Live `test:guild-sync` 120s stability budget | [ ] |
| P636-22 | Portable invite host wire regression smoke | [ ] |
| P636-23 | Discovery join await guild sync + listing refresh | [ ] |
| P636-24 | Explore join `hostPublicKey` + bounded sync wait UI | [ ] |
| P636-25 | Multi-device `guildSyncHealth` on settings mesh | [ ] |
| P636-26 | Gossip outbox priority lanes (msg > presence > audit) | [ ] |
| P636-27 | Member churn incremental join without full bundle | [ ] |
| P636-28 | Large guild channel-scoped delta (no full roster) | [~] |
| P636-29 | `pearcord-guild` README MEMBER_PAGE RPC table | [x] |
| P636-30 | Env tuning guide: sync member cap + page size | [x] |
| P636-31 | Chain `test:phase635-guild-sync` in Pear prestage | [~] |
| P636-32 | Boot gate: `pear run` v0.8.608 replication smoke | [~] |
| P636-33 | Presence vector partition merge (split-brain) | [ ] |
| P636-34 | Contacts presence reconcile debounce metrics | [~] |
| P636-35 | Conflict-free concurrent edit under high churn | [ ] |
| P636-36 | Mod offline full bundle HyperDB field audit | [ ] |
| P636-37 | Voice occupancy mesh merge on ingest | [x] |
| P636-38 | Forum index mesh merge for stale explore | [~] |
| P636-39 | Channel-scoped sparse keys for inactive channels | [ ] |
| P636-40 | Span `guild.sync.member-page` on page ingest | [x] |
| P636-41 | Dev-log filter **guild-sync-member-page** | [x] |
| P636-42 | Smoke: read receipt vector merge | [x] |
| P636-43 | Smoke: thread archive delta sync | [x] |
| P636-44 | Smoke: member page auto-chain | [x] |
| P636-45 | `test:phase636-guild-sync` bundle | [x] |
| P636-46 | E2E two-peer report refresh for v0.8.608 | [ ] |
| P636-47 | `GUILD_REPLICATION.md` Phase 636 pipeline | [x] |
| P636-48 | `GUILD_SYNC.md` v0.8.608 member paging section | [x] |
| P636-49 | Module README org-wide replication audit | [~] |
| P636-50 | Phase 637 — Sparse ACK windows & stage mesh (v0.8.609) | [x] |
**Next:** Phase 637 — sparse ACK batching, Hypercore eager replicate, gossip priority lanes, live mesh-cluster DHT.
---
### Phase 637 — Sparse ACK windows & stage mesh sync (v0.8.609)
| ID | Item | Status |
|----|------|--------|
| P637-1 | Sparse ACK-gated Hypercore batch download windows | [ ] |
| P637-2 | ACK watermark drives sparse offset cursor per channel | [ ] |
| P637-3 | Corestore `replicate()` eager open for all announced keys | [ ] |
| P637-4 | Hypercore `compactChannel` after archive gossip | [ ] |
| P637-5 | Hyperbee legacy msg-index migration CLI smoke | [ ] |
| P637-6 | Guild search index `version` bump on delta ingest | [ ] |
| P637-7 | Partial reaction export scoped to visible message window | [ ] |
| P637-8 | Gossip outbox lanes: message > presence > audit | [ ] |
| P637-9 | Member churn incremental slice without full roster | [ ] |
| P637-10 | Channel-scoped delta excludes inactive channel cores | [ ] |
| P637-11 | Stage channel occupancy slice in `GUILD_SYNC` | [ ] |
| P637-12 | Stage speaker/request-to-speak state mesh merge | [ ] |
| P637-13 | Voice stage RTT-aware sparse pull scheduler | [ ] |
| P637-14 | Multi-device `guildSyncHealth` gossip on settings mesh | [ ] |
| P637-15 | Audit export cursor in sidecar JSON persistence | [ ] |
| P637-16 | UI per-channel read receipt sync status in settings | [ ] |
| P637-17 | UI member page progress bar from `guildSyncHealth` | [ ] |
| P637-18 | Agentctl `exportGuildSyncDiagnostics` journey step | [ ] |
| P637-19 | Agentctl 3-peer live mesh-cluster DHT scenario | [ ] |
| P637-20 | Mesh `connection reset` recovery smoke | [ ] |
| P637-21 | Live `test:guild-sync` 120s stability budget | [ ] |
| P637-22 | Portable invite host wire regression smoke | [ ] |
| P637-23 | Discovery join await guild sync + listing refresh | [ ] |
| P637-24 | Explore join bounded sync wait banner | [ ] |
| P637-25 | Presence vector split-brain partition merge | [ ] |
| P637-26 | Contacts presence reconcile debounce metrics span | [ ] |
| P637-27 | High-churn concurrent message edit LWW smoke | [ ] |
| P637-28 | Mod offline full bundle HyperDB field audit | [ ] |
| P637-29 | Forum index stale explore refresh on ingest | [ ] |
| P637-30 | `PEARCORD_GUILD_SYNC_SPARSE_BATCH` env tuning doc | [ ] |
| P637-31 | Chain `test:phase636-guild-sync` in Pear prestage | [ ] |
| P637-32 | Boot gate `pear run` v0.8.609 replication smoke | [ ] |
| P637-33 | E2E two-peer report refresh v0.8.609 | [ ] |
| P637-34 | `GUILD_REPLICATION.md` Phase 637 sparse pipeline | [ ] |
| P637-35 | `GUILD_SYNC.md` stage occupancy section | [ ] |
| P637-36 | `pearcord-voice` README stage sync hooks | [ ] |
| P637-37 | `pearcord-stage` mesh gossip integration | [ ] |
| P637-38 | Dev-log filter **guild-sync-sparse-ack** | [ ] |
| P637-39 | Span `guild.sync.sparse` on batch pull | [ ] |
| P637-40 | Smoke: sparse ACK-gated batch window | [ ] |
| P637-41 | Smoke: gossip outbox priority lanes | [ ] |
| P637-42 | Smoke: stage occupancy ingest merge | [ ] |
| P637-43 | Smoke: channel-scoped delta inactive cores | [ ] |
| P637-44 | Smoke: search index version bump | [ ] |
| P637-45 | `test:phase637-guild-sync` bundle | [ ] |
| P637-46 | Module README replication audit (voice/stage) | [ ] |
| P637-47 | Bandwidth fair-share across N push peers | [ ] |
| P637-48 | Delta `forumTagPalettes` on partial sync | [ ] |
| P637-49 | Read receipt vector cross-device same-user merge | [ ] |
| P637-50 | Phase 638 — Discovery sync + explore mesh (v0.8.610) | [ ] |
**Next:** Phase 637 — sparse ACK batching, stage mesh occupancy, gossip priority lanes, live 3-peer DHT.
---
@@ -20454,6 +20572,8 @@ Absorbed into Phase 273 (P273-5–P273-8, P273-11–P273-12). See Phase 274 for
### Changelog
- **2026-06-01** — v0.8.608: Phase 636 — `readReceiptVector` + LWW tail clamp, `threadArchiveSlice`, `forumTagPalettes`, voice occupancy ingest, member page debounce, RTT zlib threshold, push peer round-robin slot, `guild.sync.member-page` span, dev-log member-page/read-receipt filters, `test:phase636-guild-sync`. Phase 637 planned (50 items).
- **2026-06-01** — v0.8.607: Phase 635 — partial member roster (`PEARCORD_GUILD_SYNC_MEMBER_CAP`), `MEMBER_PAGE_REQUEST`/`MEMBER_PAGE` (RPC 92/93), attachment priority queue, RTT-adaptive sync bursts, forum/voice slices, thread parent idempotency, `guild.sync.attach` span, dev-log attach/bandwidth filters, `test:phase635-guild-sync`. Phase 636 planned (50 items).
- **2026-06-01** — v0.8.606: Phase 634 — attachment pull scheduler (`PEARCORD_GUILD_ATTACHMENT_PULL_CAP`), bandwidth QoS (`PEARCORD_GUILD_SYNC_BYTES_PER_SEC`), HyperDB uint sanitization, delta reaction tombstones, pins slice, contacts↔guild presence reconcile, `guild.sync.presence` span, `test:phase634-guild-sync`. Phase 635 planned (50 items).
- **2026-06-01** — v0.8.603: Phase 631 — `pearcord-swarm-manager`, Hyperbee message index, ACK-driven push halt, offline gossip outbox, member/message LWW, mod sync push, zlib bundles, sync progress UI, diagnostics export, ⌘⇧R sync shortcut, `test:phase631-guild-sync`. Phase 632 planned (50 items).
- **2026-06-01** — v0.8.602: `pearcord-guild-replicator` scaffold (Corestore per guild, Hypercore append on send), delta `GUILD_SYNC` watermark filter, `GUILD_SYNC_ACK` RPC 91, `syncWatermark` on `guildSyncHealth`, `test:guild-sync-delta`. Phase 631 (50 items) CRDT/offline outbox.