Skip to content

Commit 10bfb5d

Browse files
feat(realtime): shared room spine + live Files/Tables collaboration + Yjs document editing (#5991)
* feat(realtime): add shared room identity + authorization spine (#5929) Introduces the foundation for a unified realtime "room" model spanning the Socket.IO presence server (apps/realtime), the durable SSE event log, and the ephemeral pub/sub fanout — all of which today reinvent their own room identity, naming, and authorization. - @sim/realtime-protocol/rooms: RoomRef { type, id }, ROOM_TYPES, and a roomName/parseRoomName codec. WORKFLOW deliberately maps to the bare id so the ~40 existing io.to(workflowId) callsites and presence state keys are unchanged; every other room type is namespaced so id spaces cannot collide. - @sim/platform-authz/rooms: authorizeRoom(userId, room, action) generalizing the exemplary authorizeWorkflowByWorkspacePermission — one resource->workspace resolver per room type, then the shared resolveEffectiveWorkspacePermission + permissionSatisfies gate. Pure foundation, no behavior change: nothing consumes these yet. Prune graph stays at 14/25 (platform-authz already depended transitively on realtime-protocol via apps/realtime). * refactor(realtime): generalize presence server to multi-room [2/N] (#5930) * refactor(realtime): generalize presence server to multi-room (RoomRef) Generalizes the Socket.IO presence layer from single-workflow-room-per-socket to a domain-neutral, multi-room-per-socket model keyed by RoomRef, so a second domain (workspace files, next PR) can reuse the same membership + presence engine. Behavior-preserving for workflow collaboration. IRoomManager is now domain-neutral (addUserToRoom/removeUserFromRoom/ getRoomForSocket/getRoomUsers/updateUserActivity/... all take a RoomRef). The workflow lifecycle broadcasts (deletion/revert/update/deploy) move out of the manager into WorkflowRoomService, composed over the generic manager. Backward-compat by design (no workflow migration, no regression): - Workflow Socket.IO room name stays the bare workflowId (roomName() maps workflow -> bare id), so the ~40 io.to(workflowId) callsites are untouched. - Workflow Redis presence keys stay workflow:{id}:users/:meta (the type prefix IS "workflow"). Multi-room correctness (from adversarial audit): - socket:{id}:workflow single-value key -> socket:{id}:rooms HASH (type->id). - The SHARED socket:{id}:session key is deleted only when the socket leaves its LAST room (refcount via HLEN) — a leave from one room no longer breaks the other room's handlers. - disconnect enumerates the socket's stored rooms and rebroadcasts presence per room, instead of picking an arbitrary socket.rooms entry. - presence broadcasts use a per-room-type event name (workflow keeps the bare presence-update; others are namespaced). Workflow handlers wrap manager calls with a shared workflowRoom(id) helper; UserPresence.workflowId -> room (the client never reads that field). Tests: existing 112 realtime tests pass unchanged (behavior gate) + 7 new multi-room tests (refcounted session, presence isolation, multi-room disconnect, per-type event names). tsc clean, boundaries + prune (14/25) green. * fix(realtime): harden multi-room disconnect + id-guard room removal Two fixes from an adversarial regression audit of the multi-room refactor: - Disconnect now handles `disconnecting` (where `socket.rooms` is still populated and authoritative) and falls back to the live Socket.IO room set for any room the manager's stored state no longer tracked. This restores reliable presence cleanup + departure broadcast even if the Redis `socket:{id}:rooms` key was evicted or TTL-expired — the one behavioral gap vs the pre-refactor disconnect. - REMOVE_ROOM_SCRIPT now only drops the socket's room mapping (and runs the last-room session cleanup) when the stored id matches the room being removed, matching the memory manager's existing id guard. Prevents a mismatched-room call from wiping a different room's mapping or the shared session. +1 test (id-guarded no-op removal). 120 realtime tests pass, tsc clean. * fix(realtime): only rebroadcast disconnect-fallback rooms whose removal succeeded Greptile 4/5 follow-up: the disconnecting-time fallback ignored removeUserFromRoom's boolean and rebroadcast presence even when the removal reported false. Now it only treats a room as removed (and rebroadcasts) when the manager confirms it — symmetric with removeSocketFromAllRooms, which already only returns rooms it actually removed. * fix(realtime): exclude the disconnecting socket from its farewell broadcast Greptile follow-up (transient-Redis-failure edge): if removeUserFromRoom fails on disconnect, the socket's presence entry can outlive it (room hashes have no TTL) and reappear as a ghost. Disconnect now broadcasts a correction to EVERY room the socket was in (union of the manager's removed rooms and the live Socket.IO membership) and passes the disconnecting socket id as excludeSocketId, so it is never shown as a collaborator regardless of whether the Redis delete succeeded. Any orphaned entry is still reclaimed by the next join's stale-presence sweep. broadcastPresenceUpdate gains an optional excludeSocketId; normal broadcasts are unchanged. +1 test. * fix(realtime): make presence broadcasts liveness-aware (root-cause ghost fix) Presence broadcasts now reconcile the stored list against the live Socket.IO membership (io.in(room).fetchSockets()) before emitting, via a shared filterVisiblePresence helper. This closes the residual behind the earlier disconnect fixes: an entry orphaned by a failed removal (room hashes have no TTL) could reappear in a LATER join's presence snapshot until the 75-min stale sweep. Now such an entry is never emitted, because a non-live socket is filtered out of every broadcast. Combined with excludeSocketId (which handles the disconnecting socket, still momentarily live). Fail-safe: on a fetchSockets throw or an empty result while entries remain, emit the unfiltered list rather than hide live collaborators. Also drops a dead guard in the disconnect union loop (rooms already removed are skipped by the wasInRooms check) and the now-unused isSameRoom import. +1 ghost-guard test. 122 realtime tests pass. * feat(files): live presence avatars + live file tree via realtime rooms (#5932) * refactor(tables): adopt shared durable event-log core (#5934) * fix(realtime): address post-merge review-comment findings (#5937) * fix(realtime): address post-merge review-comment findings A re-audit of every inline review comment on the merged stack surfaced real issues that the thread-resolutions and prior audits missed. Fixes: Presence server (#5930 comments): - connection.ts: snapshot `socket.rooms` SYNCHRONOUSLY before the first await. Socket.IO clears the room set once the synchronous part of a `disconnecting` handler returns, so reading it after `await removeSocketFromAllRooms` saw an empty set — the eviction fallback was dead. (Cursor: "Disconnect fallback misses live rooms".) - workflow-room-service: restore the original managers' final unconditional room-state wipe via a new `deleteRoom(room)` manager method, so a deleted workflow leaves no lingering presence/meta even if a per-socket removal failed or a socket joined mid-teardown. (Cursor: "Deletion skips final room wipe".) Files (#5932 comments): - workspace-file-manager.uploadWorkspaceFile now fans out the live-tree signal (all direct-upload paths: multipart fallback, copilot create, /api/files/upload, v1 files — the presigned path already notified). (Cursor: "Creates miss live tree fan-out".) - use-workspace-files-room: clear the pending retry timer on join success; and a module-scoped intended-room guard defers the unmount `leave` so a rapid remount re-claims the room and skips a stale leave — fixing presence flap + a leave-after-join race. (Cursor: "Retry timer survives join success" + "Remount churns files presence".) - workspace-files handler: roll back a partial join (leave room + remove presence) in the catch, mirroring the workflow join. (Cursor: "Join failure skips membership rollback".) +2 tests (deleteRoom). 127 realtime tests pass, both apps tsc clean, api-validation + boundaries green. * fix(files): scope workspace-files leave to a workspace (deferred-leave safety) Self-review of the deferred-leave guard found a real bug: leave-workspace-files was not workspace-scoped, so after a workspace switch (A->B) the deferred leave from A would evict the socket from its new room B. The leave now carries the workspaceId and the server no-ops if the socket's current files room differs. Also excludes the leaving socket from the leave broadcast (consistent with disconnect). * fix(realtime): close files-room presence leak + validate join payload Architecture-audit findings: - S1 (real Redis leak): the files room inherited the shared manager but not the workflow join's liveness sweep, so an UNGRACEFUL disconnect (pod crash — no `disconnecting` event) left its presence entry in the no-TTL room hash forever. Added a shared `sweepStalePresence(manager, room)` (fetchSockets liveness + remove not-live-AND-stale entries, matching the workflow 75min threshold) and run it on files join; also filter the join ack through `filterVisiblePresence` so a joiner never briefly sees an un-swept ghost. - S2: validate the client-supplied `workspaceId` on files join before it reaches the DB query (matches the /api/workspace-files-changed guard; fails closed). - N2: corrected the notify doc — it is awaited (guaranteed dispatch before a Node route returns) and hard-bounded to NOTIFY_TIMEOUT_MS, not "never block". +1 test (sweepStalePresence keeps live/fresh, reclaims not-live-stale). 128 realtime tests pass, both apps tsc clean, biome clean. * fix(realtime): workflow-deletion always notifies + cleans by socket.io membership Review-round findings on #5937: - Always emit `workflow-deleted` (was guarded by users.length>0), so a socket still in the Socket.IO room after a Redis presence eviction is told the workflow is gone before socketsLeave kicks it — the editor no longer keeps showing a deleted workflow. (Cursor: "Silent kick skips deletion event".) - Clean per-socket state for the UNION of live Socket.IO members and presence-tracked sockets, so an evicted/late-joined socket's room mapping + session are dropped too — not just presence-snapshot sockets. (Greptile: "Room deletion leaves reverse state".) - deleteRoom now logs AND rethrows on Redis failure (like addUserToRoom) so a failed wipe isn't reported as a clean deletion; the request surfaces it. (Greptile: "Room deletion failures are suppressed".) The two "deferred leave drops new membership" P1s were already fixed by the workspace-scoped leave in a prior commit (leave carries { workspaceId }; server no-ops on mismatch). 128 tests pass, tsc + biome clean. * refactor(files): drop module-scoped deferred-leave; rely on workspace-scoped leave Removes the one non-idiomatic construct (a module-level mutable `intendedFilesWorkspaceId` + queueMicrotask). It only guarded a same-workspace CONCURRENT remount, which doesn't occur in production (folder nav is shallow/no remount; list<->detail is sequential) — a dev-StrictMode-only case. The real cross-workspace race is already handled by the workspace-scoped leave: if B's join runs first (auto-leaving A), A's leave no-ops because the socket's current files room is B. Simpler, idiomatic, prod-correct. * feat(realtime): Yjs relay server for collaborative document editing [4/N] (#5941) Server-side Yjs relay for collaborative document editing (live carets + text selection) in the Files rich-markdown editor. Faithful y-websocket-style relay over the existing authenticated Socket.IO connection + shared room abstraction; in-memory Y.Doc + Awareness per file; awareness ownership binding, userId-keyed client-id uniqueness, seeder election with deadline re-election, concurrent-JOIN generation guard. 25 relay tests. Reviewed to Greptile 5/5 + Cursor pass across multiple rounds, plus an independent 4-lens audit (correctness/security/conventions/simplicity) and /simplify + /cleanup passes. * feat(files): collaborative document editing — client provider + editor (#5946) Client Yjs provider (FileDocProvider over the authenticated socket) + TipTap Collaboration/CollaborationCaret wiring for live carets + text-selection in the Files rich-markdown editor. Collaboration is a Files-page-only surface (explicit `collaborative` opt-in), disjoint from agent-streaming. Read-only + autosave-gated until synced+seeded. Merges into the realtime-rooms integration branch. * feat(tables): live collaboration — cell-selection presence + live mutation propagation (#5957) * feat(tables): live cell-selection presence — protocol + server + client hook The realtime spine for Google-Sheets-style table presence (mode A, socket): - @sim/realtime-protocol/table-presence: centralized wire protocol (events + TableCellSelection {anchor, focus, editing} + payloads) so server emits and client subscriptions can't drift. - ROOM_TYPES.TABLE + resolveTableWorkspace registered in ROOM_WORKSPACE_RESOLVERS (tableId -> workspace via userTableDefinitions, honoring archivedAt); roomName / presenceEventName / disconnect cleanup / authorizeRoom all derive automatically. - apps/realtime/src/handlers/tables.ts: join/leave (mirrors workspace-files) + a table-cell-selection relay (mirrors the workflow selection channel), broadcasting via roomName(room) since table rooms are namespaced. UserPresence gains a cell field threaded through the memory + Redis managers (Lua ARGV[7], null clears). - Extracted the duplicated resolveAvatarUrl into handlers/avatar.ts. - use-table-room.ts client hook: joins over the shared socket, tracks the roster (avatars) + patches per-socket cell deltas, exposes a throttled emitCellSelection. Grid UI (avatars + selection overlay) lands next; concurrent cell-value edits (last-write-wins via the durable log) are the follow-up PR. * feat(tables): render live cell-selection presence in the grid Wires the table presence room into the grid UI: - Page (table.tsx): useTableRoom (gated off in embedded/mothership mode) — renders <PresenceAvatars> in the header and passes remoteSelections + emitCellSelection down to the grid. - Grid emits its local selection: an effect resolves the index-based anchor/focus to stable (rowId, columnId) via refs and broadcasts it (with an editing flag for the active cell) through the throttled emitter. - RemoteSelectionOverlay: draws each remote viewer's selection in their color (getUserColor), a darker fill while editing, and name-on-hover — measured from live cell rects in the content wrapper's space (scrolls with the grid), hidden when rows are virtualized off-window, pointer-events-none so it never blocks cell clicks (hover via pointer hit-test). * test(tables): cover the table presence handler Mirrors workspace-files.test.ts: join auth/unavailable/denied/success, plus the cell-selection relay (asserts it persists via updateUserActivity and broadcasts on the namespaced roomName, not the bare id) and leave. * feat(tables): propagate manual cell edits live (last-write-wins) A manual row edit now appends a lightweight 'edit' event to the durable table stream; collaborators refetch the row (via the existing debounced rows-invalidate the job events use) so the winning value shows live. The event carries no value — peers refetch in their own wire format, so there's no auth-specific value translation on the wire, and last-write-wins falls out of the DB's committed order (the Google-Sheets model). Edits that also trigger a dispatch already emit dispatch/cell events; the debounce coalesces the two. * refactor(tables): apply /simplify findings - Drop the dead 'add unknown peer' upsert branch in use-table-room (Socket.IO ordering guarantees a peer is in the roster before their selection delta). - TableCellSelectionBroadcast = TablePresenceUser & { cell } (was a copy-paste). - Make TableGrid's presence props required + drop the unused empty-default/guard (only table.tsx mounts it, always passing both). - Drop the unused rowId from the 'edit' event (the handler invalidates all rows). - Overlay: subscribe scroll/resize/pointer listeners once per scroll element and cache the wrapper origin, so incoming deltas re-measure without re-subscribing and the pointer hit-test never forces a per-move layout read. - Server: cache the immutable socket session so a selection delta no longer reads it from Redis every time. * refactor(tables): apply /cleanup findings - Fix the remote-selection name label contrast: text-white is unreadable on the light-pastel user colors (same bug the Files caret fixed) → fixed dark #1a1a1a. - Re-measure via useLayoutEffect so a moving peer selection updates before paint (no one-frame position lag). - Drop 'mothership' from a comment (constitution copy rule). Six cleanup passes ran (effect, memo/callback, state, react-query, emcn, comment); the rest confirmed clean — all state/memos/callbacks/effects are load-bearing, presence correctly lives in useState (socket-pushed), and the edit→rows-invalidate granularity is right. * feat(tables): propagate every table mutation live (edit + schema signals) Comprehensive live collaboration for all user table mutations, via two value-less durable signals + named helpers (signalTableRowsChanged / signalTableSchemaChanged): - edit (rows refetch): single + batch row create, cell/row update, batch update, delete by id/filter, and upsert. - schema (definition + rows refetch): column add/update/delete, workflow-group add/update/delete, table rename, and CSV import (which can add columns). - Client handles 'schema' by invalidating the table detail (exact) + rows. Execution paths (column run, cancel-runs) and async jobs (delete/import-async, job-cancel) already propagate via cell/dispatch/job events — verified applyJob refetches on terminal. No reorder routes exist. Table archive (route DELETE) is a deliberate follow-up: it needs a table-deleted redirect event, not a refetch signal (which would 404). * refactor(tables): apply comprehensive /cleanup audit findings Holistic + react-query + comment audits over the whole PR: - Security/crash fix: a remote peer's rowId flowed unescaped into the overlay's querySelector — a hostile id ('x"]') threw SyntaxError inside a useLayoutEffect, crashing every other viewer's page. CSS.escape it, and validate + whitelist the untrusted cell payload server-side (shape + 200-char id bound) before it is stored/rebroadcast. - Simplify the CELL_SELECTION relay: the delta attached userId/userName/avatarUrl that the client discarded (identity comes from the roster). Drop them + the getUserSession lookup/cache entirely — the delta is now { socketId, cell }. - React Query: schema handler also invalidates lists() (parity with the local column-mutation set); document that the mutating client self-refetches by design. - Comment tightenings; biome fixed a stale import order in workspace-files.ts. * fix(tables): broadcast single-cell selections (focus falls back to anchor) Cursor High: a normal cell click leaves selectionFocus null (the grid treats it as a one-cell selection via focus ?? anchor), but the presence emit required BOTH anchor and focus to resolve — so the most common selection never broadcast and clicking even cleared a prior remote outline. Mirror the grid's focus ?? anchor semantics. * fix(tables): reviewer + regression + per-LOC audit findings Cursor review round (5 findings) + regression audit + per-LOC audit: - Presence roster snapshot now KEEPS the cell we already hold for a known socket, so a join/leave broadcast can't revert a fresher CELL_SELECTION delta. - Reset the selection throttle on table switch (was unmount-only), so a pending selection for table A can't flush into table B's room after a switch. - Metadata writes (column widths, display) use a new lightweight 'metadata' signal that refetches only the definition — a resize no longer forces peers to refetch rows. - Overlay re-measures on row add/remove/reorder via a tbody childList MutationObserver (a live refetch moves cells without a scroll/resize). - Document the actor self-refetch create caveat (scrolled multi-page insert) accurately. - isCellRef narrows to a partial instead of casting to the full type then re-checking; drop a redundant mount measure() (the layout effect covers it); text-[11px]→text-xs. * fix(tables): drop ineffective metadata propagation + re-measure overlay on column resize Cursor round on b8f28b04b: - Remove the 'metadata' signal entirely. The grid seeds columnWidths/pinnedColumns from metadata ONCE (metadataSeededRef) and deliberately never re-applies them (to avoid clobbering a local in-progress resize), so refetching the definition on a peer never surfaced their width/pin change — an ineffective path. Width/pin live-sync needs reconciliation that doesn't clobber a local resize; that's a deliberate follow-up, not a no-op refetch. Structural changes still propagate via 'schema'. - Overlay now also observes the content layer with the ResizeObserver, so a column resize (which grows the content, not the scroll container) re-measures remote outlines. - Presence-merge comment now states both sides of the trade-off. * fix(tables): re-broadcast local selection on (re)join Cursor Medium: a selection made before the room join completes (or held across a reconnect) was dropped server-side and never re-sent, so peers didn't see it until the local user moved it again. Track the current selection in a ref (set on every emit, cleared on table switch) and re-emit it from handleJoinSuccess once the room is joined. * fix(tables): re-broadcast selection when a peer's row change shifts it End-to-end lifecycle audit (Low-Med): the selection emit resolved the stable (rowId, columnId) only on selection/editing change, not when a live edit/schema refetch inserted/deleted/reordered rows. The index-based local selection then sat on a different logical row than the rowId peers held, so your outline showed on the old row until you moved. Re-run the emit on rows/displayColumns change and dedup an unchanged result (also drops the redundant null-on-open emit) so the broadcast stays consistent with the local highlight. * fix(tables): schema invalidates run-state/enrichment + guard stale join Cursor round on cdc8796b8 (2 Medium): - schema handler used detail exact:true, so it skipped the activeDispatches + enrichmentDetails sibling queries the local invalidateTableSchema refreshes via a prefix match. After a peer deletes/restructures a workflow group, peers could keep a stale running badge or enrichment panel. Now invalidates both siblings too (rows stay on the debounce). - Guard against a stale join stealing the room: a fast table A->B switch could let A's async authorize finish after B, leave B, and strand the socket in A. Added a per-socket monotonic join generation checked after authorize (mirrors the file-doc relay's guard) + a test. * feat(tables): live column width/pin/order sync Collaborators now see each other's column resizes, pins, and reorders live — the last piece of Google-Sheets-style layout parity. - New lightweight `metadata` durable event kind (distinct from `schema`): only the table definition carries UI metadata, so peers refetch the definition alone — no rows/run-state refetch. The metadata PUT route now signals it. - The grid reconciles server metadata against its in-progress gesture: the column being actively resized keeps its live local width, and an in-flight column drag blocks a reorder apply — so a peer's change never reverts the local action. Each field is reference-guarded (React Query structural sharing keeps unchanged sub-objects stable), so an unrelated peer change doesn't re-apply the others. * fix(tables): escalate to schema signal when a reorder scrubs group deps Independent audit of the metadata-sync commit found a stale-run-state hole: a columnOrder PUT that moves a column left of a workflow group's leftmost column makes updateTableMetadata scrub that group's dependencies and write a new schema — a real structural change. But the route only fired the lightweight 'metadata' signal (detail-only refetch), so peers' and the actor's activeDispatches / enrichmentDetails queries stayed stale (a lingering running badge / enrichment panel) — exactly what the 'schema' handler exists to prevent. updateTableMetadata now reports whether it scrubbed the schema; the route emits signalTableSchemaChanged in that case and the light signalTableMetadataChanged otherwise. Width/pin/plain-reorder stay on the cheap detail-only path. * feat(realtime): accurate in-file presence + collaborative-caret polish (#5965) Per-session file-doc presence (avatars count other sessions like the canvas), StrictMode-safe stable Y.Doc (fixes blank-doc on join), flush caret cap + restored hover hit-slop, and three join-lifecycle race fixes unifying file-doc + workspace-files on one intent-tracked monotonic generation model. All findings root-caused with regression tests. * feat(files): smarter bullet delete/indent and untitled-file title sync (#5971) * improvement(files): smarter bullet delete/indent, fix empty-nested-bullet heading corruption Backspace at the start of a list item now outdents a nested item or clears a top-level item to a paragraph in place instead of deleting the row and jumping the caret to the previous block; Enter on an empty nested item outdents. Empty non-trailing top-level items still collapse cleanly since they cannot round-trip as a lifted paragraph. Also strips nested empty list-item marker lines on serialize: a nested empty bullet re-parsed as a Setext heading underline, silently turning its parent line into an H2 and dropping the bullet. Top-level empty items are preserved. * feat(files): sync an untitled file's name with its leading heading While a file is still named untitled(.md), typing a leading heading auto-renames the file after it (debounced), and renaming the file first seeds a leading H1 from the new name. One-shot: coupling stops once the file has a real name, and the heading seed always prepends so existing content is never clobbered. * fix(files): count inline atoms in list-item emptiness, keep multi-block items on Backspace Addresses review findings on the list Backspace logic: - Emptiness now uses the caret block's content.size (counts inline images/mentions), not textContent, so a bullet holding only a non-text atom is no longer treated as empty and deleted. - An empty first block whose item has sibling blocks removes only that block instead of lifting the whole item out of the list. * fix(files): preserve the untitled to named heading seed across a rename during editor load The parent captures the file name at mount (before content/session finish loading) and passes it as the transition baseline, so a rename that lands in the loading window is still seen as an untitled to named transition and the leading heading seed is not skipped. * fix(files): drop the name-to-heading seed, keep title sync one-way Removes the effect that inserted a leading H1 when an untitled file was renamed. On the collaborative Files page every open client observed the untitled-to-named transition and inserted into the shared doc, producing duplicate headings; it could also re-insert a heading a user had just deleted while a rename was in flight. Seeding document content from an async rename transition is the wrong model on a shared editor. The primary direction — typing a leading heading renames a still-untitled file — is unaffected (it never mutates the doc). * fix(files): keep empty lines between paragraphs on reload The chunked markdown parser (parseMarkdownToDoc) parses each block stripped of the blank lines between them, so it dropped the empty paragraphs @tiptap/markdown builds from runs of blank lines — a saved visual blank line silently vanished on the next load (the settle/reopen re-seed goes through the chunker). The whole-document parser preserves them, but whether a gap yields an empty paragraph is a global, block-type- dependent decision (kept between two paragraphs, dropped after a heading), so it can't be reconstructed block-locally. Route documents with empty-paragraph blank-line spacing to the whole-document parser for exact fidelity — the same tradeoff NON_CHUNKABLE makes; ordinary single-blank-line separation still takes the fast chunked path. Adds a suite asserting chunked output matches the whole-document parser for leading/trailing/between gaps and around lists/headings. * fix(files): only auto-name an untitled file when the user can edit The debounced untitled→filename hook ran on every onUpdate — including the mount-time seed and for view-only viewers — without checking edit permission, so a read-only user could schedule a rename they have no permission to make (a spurious, server-rejected write). Gate the derive-title on editor.isEditable (canEdit + settled + collab-ready, the same signal the autosave path uses), at both schedule and fire time. * fix(files): normalize line endings before the empty-paragraph guard; Enter/Backspace symmetry - markdown-parse: EMPTY_PARAGRAPH_SPACING/NON_CHUNKABLE tested the raw body, but a classic \r-only file (blank lines are \r) would miss the \n-anchored guard and still be chunked, dropping empties. Normalize line endings once up front so the routing guards, the chunker, and the parser all see the same \n. +CRLF/CR test cases. - keymap: Enter on an empty first block of a multi-block item now removes only that block (removeEmptyWrappedBlock) instead of exiting the list, mirroring the Backspace hasSiblingBlocks case — the trailing check no longer swallows multi-block items. +test. * fix(files): editor audit follow-ups (trailing-blank read-only, collab rename, over-strip) A 4-agent independent audit (UX vs inkeep + SOTA, cleanliness, adversarial correctness) surfaced these: - HIGH regression: files ending in a blank line opened READ-ONLY. The empty-paragraph routing preserved a TRAILING empty paragraph, but postProcess collapses trailing newlines → serialize/parse non-idempotent → isRoundTripSafe flipped the file read-only. A trailing empty paragraph can't be serialized stably, so parseMarkdownToDoc now strips trailing empty paragraphs and the guard no longer routes on trailing blanks. Interior/leading empties are unaffected. +regression tests. - Medium: the debounced untitled→filename rename fired on remote Yjs edits too, so every peer renamed and could rename from a not-yet-synced heading. Gate on isChangeOrigin (local edits only; false for non-collab surfaces). - Medium: stripEmptyListItemLines over-stripped a nested empty item that follows a same-indent sibling (a real placeholder the parser keeps). Narrowed to the actual Setext hazard — an empty item DIRECTLY under a shallower parent line — matching the function's own docstring intent. Probe-verified. +test. - Low: corrected untitled-title.ts docstring that described a reverse name→heading coupling removed during review. * fix(files): a remote edit must not cancel the local rename debounce The isChangeOrigin gate cleared the debounce timer BEFORE bailing on a remote update, so a peer's edit arriving within the 600ms window cancelled the local user's pending rename. Bail on isChangeOrigin first, before touching the timer; only local edits clear/reschedule it. * docs(files): correct EMPTY_PARAGRAPH_SPACING rationale after trailing-strip The stacked trailing-empty-paragraph strip made the older comment overstate a correctness necessity it no longer owns, mislabel trailing runs of 2+ blanks, and advertise dead CRLF handling. Reword to match what the code actually does. --------- Co-authored-by: Waleed Latif <walif6@gmail.com> * fix(realtime): access-revalidation multi-room safety + cleanup pass Fix a blocker surfaced by a full cleanup/simplify audit of the branch: the access-revalidation sweep (staging's workflow-only #5917) treated every entry in socket.rooms as a workflow id, but the generalized multi-room model puts namespaced files/tables/file-doc rooms on the same io. It would resolve those as bogus workflows, get null, and evict files/tables collaborators every ~30s. collectScanTargets now decodes each room name with parseRoomName and sweeps only workflow rooms; added a regression test and fixed the now-false TSDoc. Other audit fixes (all behavior-preserving): - workflow.ts reuses resolveAvatarUrl (drops db/user/eq imports duplicated from avatar.ts) - PresenceAvatars: mr-1 was baked into the shared component, silently adding a margin to the workflow sidebar stack; moved to an optional layout className, re-applied on the tables/file-doc header surfaces only - table DELETE routes only signal collaborators when rows were actually removed (matches PUT) - events.ts definition kind: drop the never-emitted reason:'schema', fix its doc - event-log: rename buildMemory -> buildEntry (it builds the entry on the Redis success path too, not just the memory fallback) - remove dead resolveWorkspaceIdForRoom export; parallelize per-socket removals in handleWorkflowDeletion; gate the table columnIndexById map on remote selections; move file-doc module TSDoc off the FileDocOwner interface; fix a stale @returns * fix(realtime,tables): close table-presence race + v1/copilot live-collab gaps Validated each issue with subagents before implementing the cleanest fix. - tables LEAVE in-flight-join race (B8): the table handler tracked no current-table intent, so an unscoped/same-table leave during an in-flight authorize left the socket stranded in the room (present in the roster, broadcasting a ghost until disconnect). Mirror workspace-files: a closure-local currentTableId + a leave that advances joinGeneration to cancel the racing join. + 3 regression tests. - v1 + copilot live-collab signal gap (D1): tables edited via the v1 public API or Sim/copilot emitted no edit/schema signal, so open collaborators didn't live-update. Add the signals at those call sites (add-only, matching the existing route seam) — never in the service, so execution writes can't double-emit. Sync-only for copilot bulk ops, guarded on affected/deleted count; async job branches stay covered by their kind:'job' events; create/delete/get untouched. - table join read consolidation (B5): sweepStalePresence returns its roster so the same-tab dedup reuses it instead of a second getRoomUsers. - shared authorize slice (B6): extract only the guard-safe authorize->allowed branch into resolveRoomJoinAuth, shared by the three room handlers (the full preamble stays inline — file-doc's generation capture sits mid-ladder and must not move). - resize-revert flicker (E3): a peer's value-less metadata event forces a refetch that could momentarily revert a just-finished local resize; a pendingWidthWriteRef keeps local widths leading until the width PUT settles. - embedded-mode stray emit (E5): gate emitCellSelection on a bound table id so the embedded surface stops broadcasting cell selections the server drops. * fix(tables): close two copilot live-collab signal gaps + harden presence sweep Follow-ups from a comprehensive review of the branch: - copilot batch_update_rows and import_file's inline append branch wrote rows but emitted no live-collab signal, so collaborators didn't see those edits live (the append's sibling replace branch already signalled). Add the guarded signal to both, matching the internal route. - sweepStalePresence now reads the roster before the fetchSockets liveness probe and returns it on a probe failure, so same-tab dedup still runs during a transient fetchSockets outage instead of being skipped. - reword an internal comment off the retired "mothership" term. * fix(realtime): guard table join commit + rollback against supersession; drop no-op eviction cleanup Review round on #5991: - Table join re-checked the generation only once after authorize, then awaited leave/sweep/avatar before joining + registering presence. A table switch or leave in that window stranded the socket in the wrong room, and the failure catch could tear down a newer successful join. Resolve the avatar up-front, re-check generation immediately before the membership commit (matching the file-doc join), and skip the rollback/error for a superseded join. + a post-authorize-window regression test. - access-revalidation cleanup treated removeUserFromRoom's no-op false as a transport failure and re-enqueued a still-connected socket forever. Only retry when the socket is still mapped to the room (a healthy null mapping means the entry is already gone). Repurposed the expired-mapping test to lock it. * fix(realtime): guard table join leave-prior against superseding join Round 2 on #5991: a superseded join's leave-prior could still run — during its getRoomForSocket await a newer join commits to its room, so currentRoom is that newer room and the superseded join would leave/remove/broadcast it before the final guard aborts. Re-check the generation immediately after the lookup await, before the leave mutation. Extended the post-authorize-window test to assert the superseded join never tears down the newer join's room. * fix(realtime): roll back a table join superseded during addUserToRoom Round 3 on #5991: after the final generation guard, A could join + register presence while a newer join B commits to its room during addUserToRoom's await — B's leave-prior can't observe A's half-written entry, so A's late write wins and strands the socket. Re-check after addUserToRoom and roll back A's own Socket.IO join + presence (scoped to A's room, never touching B). + a regression test hanging addUserToRoom mid-commit. * refactor(realtime): DRY table-join supersession guards; fix stale comment Cleanliness pass after the review rounds (no behavior change): - Extract the four identical `joinGeneration !== joinAttempt || socket.disconnected` checks into a named `superseded()` helper (the catch keeps its intentionally narrower check). - Remove a stale guard comment that was left stranded above the avatar resolve. - Document the best-effort rollback catch. * fix(realtime): file-doc rebind must not drop the current doc or leave a writable ghost Two Cursor findings on the file-doc client-id ownership rebind: - On a document switch, the prior room was left BEFORE the ownership check, so a CLIENT_ID_IN_USE rejection dropped the socket from the old doc without joining the new one (contradicting its own comment). Run the ownership check first, and leave the previous doc only once the rebind is guaranteed to succeed. - Reclaiming a client id removed the stale prior socket from owners + awareness only; its socketToRoomName + Socket.IO membership remained, and handleMessage's SYNC path gates on socketToRoomName (not owners), so it stayed able to write document frames until disconnect. Fully evict the reclaimed socket. + 2 tests. * refactor(realtime): serialize table join/leave to fix map-corruption at the root Round 4 on #5991 surfaced a race the generation guards structurally cannot fix: two concurrent joins for one socket race on the single-valued socket→room map — a stalled addUserToRoom for table A lands late, clobbers a newer join's map entry to A, and the rollback then wipes it, stranding the socket (map empty while it holds table B). Guards protect JS suspension points; they can't stop an in-flight Redis write from landing late. Fix per architecture review: serialize this socket's JOIN + LEAVE on a per-socket promise chain so their multi-step async Redis commits can never interleave — restoring the atomic-commit property the synchronous sibling handlers get for free. This DELETES the leave-prior guard and the post-commit rollback (the code that caused the bug); four generation guards collapse to two identical superseded() checks (skip a superseded queued op + one pre-commit check). Reworked the interleaving-specific tests into a fast-switch-skips-superseded test; the leave- cancels-join tests are unchanged. Local to the tables handler — no shared-infra change. * fix(realtime): always roll back a failed table join; re-elect file-doc seeder on reclaim Two review findings: - Table join: the catch skipped rollback when superseded, but a socket.join that landed before addUserToRoom threw leaves the socket in the Socket.IO room with no matching socket->room map entry — unreclaimable by any later op (cleanup keys off the map). Under serialization the skip is unnecessary (the newer op hasn't committed), so always roll back. Simpler + fixes the strand. - File-doc reclaim: fully evicting the prior socket didn't release the seeder role if it held it, so electSeederIfNeeded (which no-ops while seederSocketId is set) never re-elected and an unseeded doc stayed empty until the deadline. Clear the role on eviction so the join's election picks a new seeder. + 2 regression tests. * fix(realtime): close revoke-race ghost presence in workflow join An access-revalidation revoke landing between socket.join and addUserToRoom socketsLeaves the socket while its presence mapping does not yet exist, so cleanupEvictedSocket finds nothing to remove and the join then writes presence for a socket already out of the room — a ghost collaborator until the stale sweep. Hoist resolveAvatarUrl (the only await in that gap) above the re-auth check so the whole re-auth -> socket.join -> addUserToRoom section is await-free, matching the invariant the handler already relies on for the pre-join re-auth. + ordering regression test. * refactor(realtime): serialize workflow join/leave; drop dead room-authz limb Comprehensive independent audit follow-ups: - workflow.ts join/leave now use the same opChain + joinGeneration serialization as the sibling handlers (tables, file-doc, workspace-files). It was the only async presence path left unserialized, so a rapid workflow switch A->B (or a leave racing an in-flight join) could strand presence in room A — a ghost collaborator still receiving A's operation broadcasts until disconnect. The join now aborts a superseded op at start and again right before the membership commit, and the catch always rolls back a partial join. - leave-workflow drops the '&& session' gate: an idle user whose 1h session key expired (while the 24h room mapping is still live) can now leave cleanly instead of being stranded until disconnect. The room ref alone suffices. - authorizeRoom: remove the dead ROOM_TYPES.WORKFLOW resolver + its getActiveWorkflowContext import. Workflow authorizes through its own path and never flows through authorizeRoom; the map now honestly covers only the workspace-scoped types (files, file-doc, table). - Remove unused isSameRoom (zero callers) and a needless useMemo in PresenceAvatars (plain derivation, single copy). - Tests: 4 workflow serialization/leave regressions. All gates green: tsc (sim/realtime/packages) 0, 204 realtime + 11 protocol tests, biome, api-validation, boundaries, prune 14/25. * fix(realtime): align session TTL, harden committed joins from post-success rollback Per-module comprehensive audit follow-ups: - redis-manager: SESSION_TTL now tracks SOCKET_ROOMS_TTL (was 1h vs 24h). The room set outlived the session, and since getRoomForSocket reads the room set while the workflow handlers gate edits/presence on `room && session`, an active-but-idle collaborator got wedged into 'session expired' after 1h — sticky until reload (activity only EXPIREs the already-gone session; only addUserToRoom re-HSETs it). Both keys refresh together, so they now expire together (restores the pre-refactor consistency, where both shared one TTL). - workflow.ts + tables.ts: a 'committed' flag stops the join catch from rolling back a genuinely-joined user when a trailing ack/broadcast/metric step fails on a Redis blip (a pure getUniqueUserCount log-metric failure could otherwise kick a live collaborator after success was already acked). - workflow.ts: leave-prior now guards `currentRoom.id !== workflowId` (a same-workflow re-join no longer leave→re-adds and flickers peers' presence), and the join ack is liveness-filtered via filterVisiblePresence — both for parity with the tables handler. - platform-authz: honest docstring + 400 message for the workspace-scoped-only authorizeRoom map (workflow authorizes via its own path). - caret-presence: corrected an over-stated batching comment. - +1 workflow regression test (post-success failure keeps the user joined). Gates: tsc (sim/realtime/packages) 0, 205 realtime + 11 protocol tests, biome, api-validation, boundaries, prune. * fix(realtime): narrow join commit-guard to post-success; skip empty presence broadcast Two Cursor findings on the prior audit-fix commit: - The 'committed' guard in join-workflow/join-table was too broad: a failure BETWEEN the membership commit and the success ack (e.g. getWorkflowState) hit 'if (committed) return' and emitted neither success nor error, hanging the client while it sat in the room. Replaced with the narrower shape: only the purely-decorative post-success steps (peer broadcast + log-metric) are wrapped best-effort; anything before the success ack still rolls back and surfaces a retryable error, so the client retries instead of hanging — while the original goal (a benign broadcast/metric blip never kicking a live, acked user) holds. - broadcastPresenceUpdate read the roster via getRoomUsers, which swallows a Redis transport error to []. On a disconnect broadcast that emitted an empty roster and cleared every remaining collaborator's presence until the next healthy update. Split out a throwing readRoomUsers; broadcastPresenceUpdate now skips the broadcast on a read failure (getRoomUsers keeps its swallow contract). - Tests: pre-success failure rolls back + retryable error (no hang); post-success failure keeps the user joined. Gates: realtime tsc 0, 206 realtime tests, biome, boundaries, prune. * fix(realtime): harden seeder recovery, join-generation, and misc robustness Final line-by-line audit follow-ups (all LOW/MED, no P0/P1): - file-doc: a sole client whose seed FETCH fails was added to triedSeeders, re-election found nobody, and the document stayed permanently empty until reload. Re-offer seeding a bounded number of rounds (MAX_SEED_ROUNDS) before giving up. Also bound clientId to a non-negative integer (it is an ownership key). - tables + workspace-files: validate the room id BEFORE advancing joinGeneration, so a malformed/rejected join can't cancel a legitimate in-flight join. - workflow + tables + workspace-files: suppress the client-facing join error when the op was already superseded (a retryable error naming the abandoned room could make a client re-join and cancel its newer join). The rollback still runs. - redis-manager: set isConnected=true only after scriptLoad succeeds (and reset it on failure) so isReady() can't report ready while the Lua SHAs are null. - connection: apply the presence-bearing filter to the manager-removed set too (symmetry with the fallback path). - http.ts: validate workflowId on the four workflow endpoints (matching the files one). - client: clear a pending join-retry timer before rescheduling (reconnect churn no longer orphans a stray extra join); clear caret fade timers on plugin destroy; seed-effect cleanup reports NOT-ready (safe direction). - Tests: bounded seeder recovery, cell-selection strip-junk, TABLE round-trip, presenceEventName. Gates: tsc (sim/realtime/packages) 0, 208 realtime + 12 protocol tests, biome, api-validation, boundaries, prune. * fix(realtime): gate isReady() on loaded script SHAs, not just connection Follow-up to the prior isConnected change, which was incomplete: the redis client's 'ready' event flips isConnected=true on connect — before initialize() loads the Lua scripts — so a bare isConnected check reports ready while removeUserFromRoom / updateUserActivity would silently no-op on a null SHA. Gate isReady() on the SHAs too, so the POST endpoints return a retryable 503 during that startup window instead of proceeding against unloaded scripts. Standard readiness-probe discipline. * fix(files): offline read-only fallback when the realtime doc never syncs When the realtime server is unreachable (offline, server down, socket never connects), the collaborative editor would sit blank and read-only forever — content only arrives via provider sync events that never fire. Add a bounded connect-deadline to the Yjs provider: if no first sync lands within CONNECT_DEADLINE_MS, latch fatal and emit a synthetic non-retryable join-error — the exact path a real fatal rejection already uses, which seeds the file's stored content read-only. Latching fatal also stops a late reconnect from syncing server state in and merge-duplicating the locally-seeded content (the documented Yjs 'non-empty doc ignores initial value' gotcha). Deliberately NOT adding durable Yjs snapshot persistence / server-side seeding: TipTap can't run the markdown->Yjs conversion server-side (Collaboration extension errors under jsdom), and a durable binary snapshot would create a dual source of truth with the markdown file (edited by copilot / PUT / download). The client-seeder + bounded re-election is the correct architecture for a markdown-is-truth model; this closes its one real user-facing gap without persistence, a migration, or dual-truth. Timer cleared on first sync, on a real fatal rejection, and on destroy. +2 tests. Gates: sim tsc 0, 496 editor tests, biome. Needs live offline->reconnect verification. * fix(realtime): validate workflow join id before generation bump; scope file-doc join rollback to its target Two Cursor findings: - join-workflow bumped joinGeneration before validating workflowId (unlike tables / workspace-files, which I'd already fixed). A malformed/empty join could advance the counter and cancel a legitimate in-flight workflow switch. Validate the id first. +test. - The file-doc join catch called cleanupFileDocForSocket unconditionally, which keys off socketToRoomName. During a document SWITCH that fails before rebinding (e.g. a throw in client-id reclaim), that binding still points at the socket's PRIOR, valid document — so the rollback tore down a document the socket was validly in. Only run that cleanup when the binding already points at THIS join's target; otherwise the socket never registered as an owner here and the only leftover is a freshly-created empty room, dropped by destroyRoomIfIdle. Gates: realtime tsc 0, 209 tests, biome, boundaries, prune. * fix(files): drop late sync frames once fatal; file-doc join error suppression + retry-budget reset Final safety-audit findings: - CRITICAL: FileDocProvider.handleMessage had no fatal guard. After the connect deadline latched fatal and the editor fell back to a read-only local seed, a late SyncStep2 (slow server / flaky network / deploy) was still applied — merging server state into the seeded doc (content duplication) and flipping synced=true, which un-gated autosave and would persist the duplicate to the real file. fatal guarded (re)join but not inbound sync. Now handleMessage returns early when fatal. +test (late SyncStep2 after the deadline is ignored, doc stays empty + gated). - file-doc join catch now suppresses the client-facing error when superseded (matches workflow/tables/workspace-files) so a retryable error for an abandoned file can't make a client re-join and cancel the newer one. - table/workspace-files room hooks reset the retry budget on (re)connect so a prior full exhaustion doesn't block retries after a reconnect. - presence-visibility: corrected a stale TTL comment. Gates: tsc (sim/realtime) 0, 209 realtime + collab/hooks suites, biome, boundaries, prune. * feat(collab-doc): server-authoritative Yjs seeding (#6008) Server-authoritative Yjs seeding for collaborative file documents: the realtime relay fetches a Yjs seed built from the file's markdown (via the shared TipTap engine) and applies it once per room, replacing the client-seeder election/handshake entirely. - DOM-free markdown<->Yjs conversion core (markdownToYDoc / yDocToMarkdown / applyMarkdownToYDoc) reusing the client markdown engine for parity by construction - Internal x-api-key seed endpoint + realtime fetch; single attempt bounded under the client readiness deadline, guard-release for join-driven retry, read-only fallback on persistent failure - Client readiness gate = synced && server seed flag; jsdom wired for the Next standalone build (serverExternalPackages + outputFileTracingIncludes) Foundation only — copilot-into-doc + markdown projection + durable persistence are Stage C. * feat(collab-doc): Sim merge endpoint for copilot-into-doc (Stage C foundation) buildFileDocMergeUpdate(docState, markdown) computes the minimal Yjs diff that turns a live document into target markdown, via applyMarkdownToYDoc (a real updateYFragment diff, not a replace) — so a copilot rewrite merges with concurrent user edits instead of clobbering them. Exposed over the internal x-api-key /api/internal/file-doc/merge endpoint the realtime relay will call: the relay owns the doc, the app owns the conversion engine, so the relay ships the current state and applies the returned diff. Tested incl. concurrent-edit no-clobber. * feat(collab-doc): realtime apply-edit — merge copilot markdown into a live doc The relay can now stream a copilot edit into open editors: applyMarkdownToLiveFileDoc finds the seeded live room, ships its state to the app's /merge endpoint for a minimal CRDT diff, applies it (relaying to every editor, reconciled with concurrent user edits), and reports 'no-live-room' so the caller falls back to a direct file write when nothing is open. - Generalize the realtime->app request module (file-doc-seed.ts -> file-doc-app.ts) with a shared POST helper + fetchFileDocSeed/fetchFileDocMerge - POST /api/file-doc/apply-edit on the internal x-api-key HTTP surface, returning { applied } - Tests for the seeded-room merge relay and the no-live-room fallback * feat(collab-doc): stream copilot edits into open editors (Stage C) edit_content now, after its durable file write, best-effort merges the same markdown into the file's live collaborative document (markdown files only). If a collaborator has it open, the edit streams into their editor as a CRDT merge — reconciled with their concurrent typing — instead of the file silently changing under them; the editor's existing autosave mirrors the merged doc back to the file. No-op when nothing is open. Never blocks or fails the edit. * fix(collab-doc): strip frontmatter on merge; gate live-merge to markdown - buildFileDocMergeUpdate now strips YAML frontmatter (splitFrontmatter().body) exactly as the seed does. Copilot passes full-file content, so without this the frontmatter merged into the doc as editor content and autosave wrote it back over the file (corruption). - Gate the live-doc merge on isMarkdownFileName (new server-safe helper) instead of the over-broad !isDoc, so code/text edits don't pay the realtime round-trip for a format the collaborative editor never renders. * fix(collab-doc): make frontmatter collaborative so a merge can't revert it The editor re-attaches its open-time frontmatter on every autosave, so the Stage C merge (which triggers an autosave with no user action) could write stale YAML back over a copilot frontmatter change — silently dropping it. Carry the file's frontmatter in the doc's config map instead of locking it at open: the seed stores it, the merge updates it (only when it actually changed, preserving the no-op diff), and the editor re-attaches THAT value on save — falling back to the locked copy before the seed lands and for non-collaborative docs. A server-side frontmatter change is now reflected rather than reverted. New FILE_DOC_SEED.frontmatterKey; seed/merge tests cover it. * fix(collab-doc): re-sync draft on a frontmatter-only merge A server edit that changes only the frontmatter updates the config map but not the body fragment, so TipTap's onUpdate never fires — the autosave draft kept the stale open-time frontmatter, and an explicit save could revert the live change. Observe the config map and, on a frontmatter-only change, re-attach the new frontmatter to the current body and push a fresh draft (guarded on a synced body so it never races the seed's own onUpdate). * fix(collab-doc): order the apply-edit/merge timeouts (outer > inner) The Sim->realtime apply-edit timeout (4s) was shorter than the nested realtime->Sim merge timeout (8s), so the outer call could abort while the relay was still merging — the relay then applied the merge after edit_content had returned, racing a follow-on edit. Split the shared realtime->app timeout: the seed keeps 8s (it reads a cold blob), the merge gets a tight 3s (it is a pure conversion, no I/O), and the outer apply-edit is raised to 6s so it always outlives the inner merge. Cross-referenced in comments to prevent drift. * fix(collab-doc): address deep-audit findings (jsdom trace, timeouts, gate, races) A 4-agent LOC audit + precedent research (TipTap/Hocuspocus/Yjs docs confirm the core patterns are idiomatic) surfaced these real issues: - The /merge route also lazy-requires jsdom but was missing its outputFileTracingIncludes entry — a Docker/standalone build would 500 with MODULE_NOT_FOUND. Add it. - The four conversion timeouts encoded an ordering invariant (merge<applyEdit, seed<readiness) living only in prose across two apps. Hoist to a shared FILE_DOC_TIMEOUTS in @sim/realtime-protocol with a test asserting the ordering; both apps import it. - The copilot live-merge gate used an extension-only check, but the editor treats any text/markdown-MIME file as markdown. Replace isMarkdownFileName with a MIME-aware isMarkdownFile mirroring the client, so those files stream too. - applyMarkdownToLiveFileDoc had no per-file serialization; overlapping merges could each diff the same stale snapshot and apply out of order. Serialize per file via a promise chain. - Editor hardcoded 'config'/'initialContentLoaded' instead of FILE_DOC_SEED constants (drift). - Add .max() bounds to the merge contract body; document the best-effort-merge failure window honestly (open editor + merge failure can drop a copilot edit until reload — closed by the deferred durable-doc work). * feat(collab-doc): multi-replica shared Yjs backend + server-side markdown persistence Make collaborative file-doc editing correct across multiple ECS tasks (the per-process Y.Doc previously assumed one replica per file). - Shared Yjs backend over Redis Streams (apps/realtime file-doc-store): each file's stream is the ordered, replayable log of updates; a multiplexed XREAD tailer converges every task's in-memory doc. Coordinated single-seeder election (SET NX + empty-stream recheck) fixes split-brain seeding. - Doc-sync fans out to local clients + the stream; awareness stays on the Socket.IO adapter. Snapshot+XTRIM compaction trims only integrated entries. - Server-side persistence: project the live doc back to markdown via a new /api/internal/file-doc/persist endpoint, debounced during editing and flushed on last-disconnect, from the authoritative stream state. Collaborative editors no longer client-autosave, closing the copilot clobber-window. - Copilot merges apply through the stream (reach the live doc on any task) and serialize cross-task via a Redis merge lock. - Degrades to the original single-replica behavior when REDIS_URL is unset. * fix(collab-doc): harden distributed locks and durability from review Address Greptile + Cursor review of the multi-replica backend: - Merge lock: retry LONGER than the lock TTL (guaranteed acquisition, never merges against a shared base while a peer holds the lock) and AWAIT the stream write before releasing, so the next task never diffs a stale base. - Distributed locks (seed/merge/compact) now use ownership tokens + a compare-and-delete release (Lua), so a lock that expired and was re-acquired by another task is never stolen; acquisition fails CLOSED on Redis error. - Seed: publish the seed to the stream AWAITED under the lock before releasing, so a later seeder's empty-stream fence always sees it (closes the fence's publish-after-release gap); TTL kept at the readiness deadline. - Persist: persist the AUTHORITATIVE stream state even when this task's local doc was never seeded, and capture the local fallback synchronously so a last-disconnect flush never encodes an already-destroyed doc. - Publish: retry a transient xAdd failure so a Redis blip can't silently drop an edit from the shared log. * fix(collab-doc): only persist a doc a user actually edited Cursor review: a copilot durable write landing while a doc is being seeded could have the stale seed projected back over it on last-disconnect, clobbering the copilot edit even with no user changes. Gate server-side persistence on a genuine user edit (socket-origin update): a seed-only or merge-only doc is never projected back to the file (copilot writes the file durably itself), so it can't clobber a concurrent external write. * fix(collab-doc): close review-round race/durability gaps Address Greptile P1 + Cursor findings: - Seed publishes to the shared stream AWAITED *before* seeding the local doc, so a publish failure leaves the doc unseeded and the stream empty for a clean retry rather than serving an unpublished local seed a peer would re-seed over (split-brain). - flushPersist falls back to the synchronously-captured local snapshot when getStreamState throws (not only when it returns null), so a transient Redis read no longer drops the final durable write as the room is torn down. - streamHasContent fails CLOSED (returns true on xLen error): a Redis blip can no longer let the seed fence pass and double-seed. - Client collabReady initializes from the collaborative prop, so a collaborative editor never has a mount-window where client autosave could clobber the server write. * fix(collab-doc): unstick seed retry and persist tailed edits Cursor review: - ensureServerSeed clears serverSeedStarted when it aborts at the streamHasContent fence, so a fail-closed Redis xLen (or a genuine peer-seed) no longer strands the room unseeded with no retry. - Persistence dirty-tracking now marks a doc edited on any post-seed update, including a peer's edit relayed via the tailer (REDIS_ORIGIN), tracked via a seededObserved flag. The last task to leave persists real edits even if it only tailed them; the seed transition itself is still never counted, so a seeded-but-unedited doc is never projected back over the file. * fix(collab-doc): match client markdown post-processing on server persist Cursor review: yDocToFileMarkdown serialized the body with yDocToMarkdown only, but the editor save path runs postProcessSerializedMarkdown before applyFrontmatter. Server persist could therefore write markdown differing from a client save (empty list markers, callout un-escaping) — spurious blob churn / round-trip drift despite the byte-identical claim. Apply postProcessSerializedMarkdown in yDocToFileMarkdown so a server persist is byte-identical to a client save and the client's dirty-check baseline. Add a regression test guarding the composition. * fix(collab-doc): close durability gaps at deploy boundaries + audit polish From a comprehensive from-scratch audit (correctness, SOTA, cleanliness, feature-completeness): - Persist max-wait: a continuous edit burst kept resetting the 5s debounce and never persisted; cap it so a burst flushes at least every 20s, bounding unpersisted edits. - Graceful-shutdown flush: flushAllFileDocRooms awaited in shutdown so a rolling deploy / scale-in secures open edited rooms to durable markdown before exit, instead of relying on the stream + a surviving task. - Compacted-snapshot catch-up now marks the doc edited (REDIS_SNAPSHOT_ORIGIN): a snapshot folds seed+edits into one frame, so a task catching up purely from it no longer treats real edits as an unedited seed and skips persisting. - Polish: delete dead __setFileDocStoreForTest; bounded retry loop; rename acquirePersistSlot -> tryClaimPersistWindow with accurate docs; tailer object-identity guard; fix stale comments (seed route, edit-content autosave). * improvement(realtime): post-review fixes for collab dirty-state + relay lifecycle - collab editor no longer latches a spurious 'Unsaved changes' prompt: report dirty only when the client owns durability (canAutosave), since in a collaborative session the relay persists the doc server-side - guard shutdown against a double SIGINT/SIGTERM running teardown twice - disconnect loc…
1 parent ecf1d7d commit 10bfb5d

164 files changed

Lines changed: 33786 additions & 2146 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

apps/realtime/package.json

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,9 +33,12 @@
3333
"@sim/workflow-types": "workspace:*",
3434
"@socket.io/redis-adapter": "8.3.0",
3535
"drizzle-orm": "^0.45.2",
36+
"lib0": "0.2.117",
3637
"postgres": "^3.4.5",
3738
"redis": "5.10.0",
3839
"socket.io": "^4.8.1",
40+
"y-protocols": "1.0.7",
41+
"yjs": "13.6.31",
3942
"zod": "4.3.6"
4043
},
4144
"devDependencies": {

apps/realtime/src/access-revalidation.test.ts

Lines changed: 50 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -50,16 +50,14 @@ function makeManager(sockets: FakeSocket[], presence: Partial<UserPresence>[] =
5050
const manager = {
5151
io: { sockets: { sockets: socketMap } },
5252
isReady: () => true,
53-
getWorkflowUsers: vi.fn().mockResolvedValue(presence),
54-
getWorkflowIdForSocket: vi.fn().mockResolvedValue(null),
55-
removeUserFromRoom: vi
56-
.fn()
57-
.mockImplementation(async (_socketId: string, workflowId?: string) => workflowId ?? null),
53+
getRoomUsers: vi.fn().mockResolvedValue(presence),
54+
getRoomForSocket: vi.fn().mockResolvedValue(null),
55+
removeUserFromRoom: vi.fn().mockResolvedValue(true),
5856
broadcastPresenceUpdate: vi.fn().mockResolvedValue(undefined),
5957
}
6058
return manager as unknown as IRoomManager & {
61-
getWorkflowUsers: ReturnType<typeof vi.fn>
62-
getWorkflowIdForSocket: ReturnType<typeof vi.fn>
59+
getRoomUsers: ReturnType<typeof vi.fn>
60+
getRoomForSocket: ReturnType<typeof vi.fn>
6361
removeUserFromRoom: ReturnType<typeof vi.fn>
6462
broadcastPresenceUpdate: ReturnType<typeof vi.fn>
6563
}
@@ -84,8 +82,11 @@ describe('access-revalidation sweep', () => {
8482
expect.objectContaining({ workflowId: 'wf-1' })
8583
)
8684
expect(socket.leave).toHaveBeenCalledWith('wf-1')
87-
expect(manager.removeUserFromRoom).toHaveBeenCalledWith('sock-1', 'wf-1')
88-
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith('wf-1')
85+
expect(manager.removeUserFromRoom).toHaveBeenCalledWith(
86+
{ type: 'workflow', id: 'wf-1' },
87+
'sock-1'
88+
)
89+
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith({ type: 'workflow', id: 'wf-1' })
8990
})
9091

9192
it('keeps a socket whose access is still valid', async () => {
@@ -141,7 +142,30 @@ describe('access-revalidation sweep', () => {
141142

142143
expect(mockResolveRole).toHaveBeenCalledWith('user-1', 'wf-1', 'read')
143144
// The security scan must stay Redis-free — presence is never consulted.
144-
expect(manager.getWorkflowUsers).not.toHaveBeenCalled()
145+
expect(manager.getRoomUsers).not.toHaveBeenCalled()
146+
})
147+
148+
it('never evicts a socket joined only to a non-workflow room (files/tables/file-doc)', async () => {
149+
// The sweep shares one io with the files/tables/file-doc handlers. Those rooms are
150+
// namespaced (`workspace-files:ws-1`, `table:t-1`), so treating every socket.rooms
151+
// entry as a workflow id would resolve a bogus permission → null → evict the socket
152+
// from its files/table room every pass. Non-workflow rooms must be filtered out.
153+
const filesSocket = makeSocket('sock-1', 'user-1', 'workspace-files:ws-1')
154+
const tableSocket = makeSocket('sock-2', 'user-2', 'table:t-1')
155+
const manager = makeManager([filesSocket, tableSocket])
156+
// Even if the role resolver would say "no access", these must never be swept.
157+
mockResolveRole.mockResolvedValue(null)
158+
159+
const sweep = startAccessRevalidationSweep(manager)
160+
await sweep.runOnce()
161+
sweep.stop()
162+
163+
expect(mockResolveRole).not.toHaveBeenCalled()
164+
expect(filesSocket.leave).not.toHaveBeenCalled()
165+
expect(filesSocket.emit).not.toHaveBeenCalled()
166+
expect(tableSocket.leave).not.toHaveBeenCalled()
167+
expect(tableSocket.emit).not.toHaveBeenCalled()
168+
expect(manager.removeUserFromRoom).not.toHaveBeenCalled()
145169
})
146170

147171
it('evicts only the revoked socket, not co-members of the room', async () => {
@@ -199,37 +223,37 @@ describe('access-revalidation sweep', () => {
199223
sweep.stop()
200224

201225
expect(manager.removeUserFromRoom).toHaveBeenCalledTimes(2)
202-
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith('wf-1')
226+
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith({ type: 'workflow', id: 'wf-1' })
203227
})
204228

205-
it('defers cleanup when removal fails with expired socket mappings', async () => {
229+
it('drops eviction cleanup when the socket is no longer mapped to the room (no infinite retry)', async () => {
206230
const socket = makeSocket('sock-1', 'user-1', 'wf-1')
207231
const manager = makeManager([socket], [{ socketId: 'sock-1', role: 'read' }])
208-
// Mapping keys already expired (lookup resolves null) AND the removal fails
209-
// (the Redis manager swallows the transport error into null) — the failed
210-
// removal must still defer instead of reading as success.
211-
manager.removeUserFromRoom.mockResolvedValueOnce(null)
232+
// A healthy lookup shows the socket is no longer mapped to any workflow room (its presence
233+
// is already gone), and removeUserFromRoom reports a no-op `false`. This is "already clean",
234+
// not a deferrable failure — the cleanup must drop it, never re-enqueue a still-connected
235+
// socket forever. (A genuine failure — still mapped + false — is covered by the next test.)
236+
manager.getRoomForSocket.mockResolvedValue(null)
237+
manager.removeUserFromRoom.mockResolvedValue(false)
212238
mockResolveRole.mockResolvedValue(null)
213239

214240
const sweep = startAccessRevalidationSweep(manager)
215241
await sweep.runOnce()
216-
217-
expect(manager.broadcastPresenceUpdate).not.toHaveBeenCalled()
218-
219242
await sweep.runOnce()
220243
sweep.stop()
221244

222-
expect(manager.removeUserFromRoom).toHaveBeenCalledTimes(2)
223-
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith('wf-1')
245+
// Attempted once, then dropped — not re-enqueued across passes, and no broadcast.
246+
expect(manager.removeUserFromRoom).toHaveBeenCalledTimes(1)
247+
expect(manager.broadcastPresenceUpdate).not.toHaveBeenCalled()
224248
})
225249

226250
it('defers cleanup when the manager swallows a removal failure into null', async () => {
227251
const socket = makeSocket('sock-1', 'user-1', 'wf-1')
228252
const manager = makeManager([socket], [{ socketId: 'sock-1', role: 'read' }])
229253
// Live mapping but the removal reports nothing removed — the Redis manager
230254
// swallows transport errors into null, so this is the only failure signal.
231-
manager.getWorkflowIdForSocket.mockResolvedValue('wf-1')
232-
manager.removeUserFromRoom.mockResolvedValueOnce(null)
255+
manager.getRoomForSocket.mockResolvedValue({ type: 'workflow', id: 'wf-1' })
256+
manager.removeUserFromRoom.mockResolvedValueOnce(false)
233257
mockResolveRole.mockResolvedValue(null)
234258

235259
const sweep = startAccessRevalidationSweep(manager)
@@ -243,15 +267,15 @@ describe('access-revalidation sweep', () => {
243267
sweep.stop()
244268

245269
expect(manager.removeUserFromRoom).toHaveBeenCalledTimes(2)
246-
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith('wf-1')
270+
expect(manager.broadcastPresenceUpdate).toHaveBeenCalledWith({ type: 'workflow', id: 'wf-1' })
247271
})
248272

249273
it('skips removal when the socket has since moved to a different workflow', async () => {
250274
const socket = makeSocket('sock-1', 'user-1', 'wf-1')
251275
const manager = makeManager([socket], [{ socketId: 'sock-1', role: 'read' }])
252276
// Between the membership snapshot and cleanup, the socket switched to a
253277
// workflow it can still access — removal must not touch its new presence.
254-
manager.getWorkflowIdForSocket.mockResolvedValue('wf-2')
278+
manager.getRoomForSocket.mockResolvedValue({ type: 'workflow', id: 'wf-2' })
255279
mockResolveRole.mockResolvedValue(null)
256280

257281
const sweep = startAccessRevalidationSweep(manager)
@@ -357,7 +381,7 @@ describe('access-revalidation sweep', () => {
357381
const manager = makeManager([socket], [{ socketId: 'sock-1', role: 'read' }])
358382
// A Redis outage where commands hang in the offline queue instead of
359383
// failing: the cleanup lane stalls, but scans must keep running.
360-
manager.getWorkflowIdForSocket.mockReturnValue(new Promise(() => {}))
384+
manager.getRoomForSocket.mockReturnValue(new Promise(() => {}))
361385
mockResolveRole.mockResolvedValue(null)
362386

363387
const sweep = startAccessRevalidationSweep(manager)

apps/realtime/src/access-revalidation.ts

Lines changed: 43 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
import { createLogger } from '@sim/logger'
22
import type { AccessRevokedBroadcast } from '@sim/realtime-protocol/events'
3+
import { parseRoomName, ROOM_TYPES } from '@sim/realtime-protocol/rooms'
34
import { sleep } from '@sim/utils/helpers'
45
import type { AuthenticatedSocket } from '@/middleware/auth'
56
import { ROLE_REVALIDATION_TTL_MS, resolveCurrentWorkflowRole } from '@/middleware/permissions'
6-
import type { IRoomManager } from '@/rooms'
7+
import { type IRoomManager, workflowRoom as wf } from '@/rooms'
78

89
const logger = createLogger('AccessRevalidation')
910

@@ -65,9 +66,14 @@ interface ScanTarget {
6566
* Collects this pod's authenticated sockets with the workflow room each has
6667
* joined, in stable socket order.
6768
*
68-
* The workflow room is derived from the socket's own `rooms` set (pod-local, no
69-
* Redis round-trips): a socket joins exactly one workflow room, so its rooms are
70-
* `{ ownSocketId, workflowId }`. Only local sockets are evaluated — sockets are
69+
* Rooms are derived from the socket's own `rooms` set (pod-local, no Redis
70+
* round-trips). A socket may occupy several rooms of different types at once
71+
* (workflow canvas, workspace-files browser, table, file-doc), all on the same
72+
* io — so each name is decoded with {@link parseRoomName} and only **workflow**
73+
* rooms are swept here. Non-workflow room names are namespaced (`type:id`) and
74+
* resolve to a non-workflow type; sweeping them as workflow ids would resolve a
75+
* bogus permission, come back `null`, and spuriously evict the socket from its
76+
* files/table room every pass. Only local sockets are evaluated — sockets are
7177
* sticky to a pod, so every socket is swept by exactly one pod using that pod's
7278
* warm role cache (mirroring the per-pod reasoning of the write-path cache).
7379
*/
@@ -78,7 +84,9 @@ function collectScanTargets(io: IRoomManager['io']): ScanTarget[] {
7884
if (!authed.userId) continue
7985
for (const room of socket.rooms) {
8086
if (room === socket.id) continue
81-
targets.push({ workflowId: room, socket: authed, userId: authed.userId })
87+
const ref = parseRoomName(room)
88+
if (ref?.type !== ROOM_TYPES.WORKFLOW) continue
89+
targets.push({ workflowId: ref.id, socket: authed, userId: authed.userId })
8290
}
8391
}
8492
return targets
@@ -129,9 +137,20 @@ export function startAccessRevalidationSweep(roomManager: IRoomManager): AccessR
129137
async function cleanupEvictedSocket(socketId: string, workflowId: string): Promise<void> {
130138
const key = `${socketId}:${workflowId}`
131139
try {
140+
// A fully-disconnected socket already had its presence removed by the
141+
// disconnect handler (removeSocketFromAllRooms), so there is nothing left to
142+
// clean. Dropping here also keeps the boolean removeUserFromRoom below from
143+
// reporting a false "not a member" for an already-gone entry and retrying it
144+
// forever (the pre-generalization manager returned the target on a no-op).
145+
if (!io.sockets.sockets.get(socketId)) {
146+
pendingCleanups.delete(key)
147+
return
148+
}
149+
132150
// Unlike removeUserFromRoom, this read does not swallow transport errors,
133151
// so a Redis outage lands in the catch below and defers the cleanup.
134-
const currentWorkflowId = await roomManager.getWorkflowIdForSocket(socketId)
152+
const currentRoom = await roomManager.getRoomForSocket(socketId, ROOM_TYPES.WORKFLOW)
153+
const currentWorkflowId = currentRoom?.id ?? null
135154
if (currentWorkflowId !== null && currentWorkflowId !== workflowId) {
136155
// The socket has since moved to a different workflow it can still
137156
// access; that join's room switch already removed this room's presence
@@ -148,16 +167,26 @@ export function startAccessRevalidationSweep(roomManager: IRoomManager): AccessR
148167
return
149168
}
150169

151-
const removed = await roomManager.removeUserFromRoom(socketId, workflowId)
152-
if (removed === null) {
153-
// The sweep always passes the target room, and both managers report a
154-
// performed removal by returning it — the Redis manager swallows
155-
// transport errors into null, so null means the removal did not happen
156-
// (even when the socket's mapping keys have already expired).
157-
throw new Error('room-state removal not confirmed')
170+
// A null mapping here is the normal case (the socket's mapping key may have
171+
// expired) and does NOT mean "skip" — the eviction still removes the presence
172+
// entry from the known target room via the explicit ref below.
173+
const removed = await roomManager.removeUserFromRoom(wf(workflowId), socketId)
174+
if (!removed) {
175+
// `false` conflates two outcomes: the entry was already gone (a no-op), or a
176+
// transport error the manager swallowed. Only retry when the socket is still mapped
177+
// to THIS room — then a false result is a genuine, deferrable failure. When a healthy
178+
// getRoomForSocket above returned no workflow mapping (`currentWorkflowId === null`),
179+
// the presence entry is already gone, so the cleanup is complete: dropping it avoids
180+
// re-enqueuing a still-connected socket forever. (A real Redis outage throws at
181+
// getRoomForSocket and is deferred by the outer catch, never reaching here.)
182+
if (currentWorkflowId === workflowId) {
183+
throw new Error('room-state removal not confirmed')
184+
}
185+
pendingCleanups.delete(key)
186+
return
158187
}
159188

160-
await roomManager.broadcastPresenceUpdate(workflowId)
189+
await roomManager.broadcastPresenceUpdate(wf(workflowId))
161190
pendingCleanups.delete(key)
162191
} catch (error) {
163192
pendingCleanups.set(key, { socketId, workflowId })
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
import { db, user } from '@sim/db'
2+
import { createLogger } from '@sim/logger'
3+
import { eq } from 'drizzle-orm'
4+
import type { AuthenticatedSocket } from '@/middleware/auth'
5+
6+
const logger = createLogger('PresenceAvatar')
7+
8+
/**
9+
* The avatar URL for a presence entry: the socket's authenticated image when
10+
* present, otherwise a single lookup of the user's stored image. Never throws —
11+
* presence must not fail on an avatar lookup, so a DB error resolves to `null`.
12+
*/
13+
export async function resolveAvatarUrl(
14+
socket: AuthenticatedSocket,
15+
userId: string
16+
): Promise<string | null> {
17+
if (socket.userImage) return socket.userImage
18+
try {
19+
const [record] = await db
20+
.select({ image: user.image })
21+
.from(user)
22+
.where(eq(user.id, userId))
23+
.limit(1)
24+
return record?.image ?? null
25+
} catch (error) {
26+
logger.warn('Failed to load user avatar for presence', { userId, error })
27+
return null
28+
}
29+
}

0 commit comments

Comments
 (0)