Commit 10bfb5d
feat(realtime): shared room spine + live Files/Tables collaboration + Yjs document editing (#5991)
* feat(realtime): add shared room identity + authorization spine (#5929)
Introduces the foundation for a unified realtime "room" model spanning the
Socket.IO presence server (apps/realtime), the durable SSE event log, and the
ephemeral pub/sub fanout — all of which today reinvent their own room identity,
naming, and authorization.
- @sim/realtime-protocol/rooms: RoomRef { type, id }, ROOM_TYPES, and a
roomName/parseRoomName codec. WORKFLOW deliberately maps to the bare id so the
~40 existing io.to(workflowId) callsites and presence state keys are unchanged;
every other room type is namespaced so id spaces cannot collide.
- @sim/platform-authz/rooms: authorizeRoom(userId, room, action) generalizing the
exemplary authorizeWorkflowByWorkspacePermission — one resource->workspace
resolver per room type, then the shared resolveEffectiveWorkspacePermission +
permissionSatisfies gate.
Pure foundation, no behavior change: nothing consumes these yet. Prune graph
stays at 14/25 (platform-authz already depended transitively on realtime-protocol
via apps/realtime).
* refactor(realtime): generalize presence server to multi-room [2/N] (#5930)
* refactor(realtime): generalize presence server to multi-room (RoomRef)
Generalizes the Socket.IO presence layer from single-workflow-room-per-socket
to a domain-neutral, multi-room-per-socket model keyed by RoomRef, so a second
domain (workspace files, next PR) can reuse the same membership + presence
engine. Behavior-preserving for workflow collaboration.
IRoomManager is now domain-neutral (addUserToRoom/removeUserFromRoom/
getRoomForSocket/getRoomUsers/updateUserActivity/... all take a RoomRef). The
workflow lifecycle broadcasts (deletion/revert/update/deploy) move out of the
manager into WorkflowRoomService, composed over the generic manager.
Backward-compat by design (no workflow migration, no regression):
- Workflow Socket.IO room name stays the bare workflowId (roomName() maps
workflow -> bare id), so the ~40 io.to(workflowId) callsites are untouched.
- Workflow Redis presence keys stay workflow:{id}:users/:meta (the type prefix
IS "workflow").
Multi-room correctness (from adversarial audit):
- socket:{id}:workflow single-value key -> socket:{id}:rooms HASH (type->id).
- The SHARED socket:{id}:session key is deleted only when the socket leaves its
LAST room (refcount via HLEN) — a leave from one room no longer breaks the
other room's handlers.
- disconnect enumerates the socket's stored rooms and rebroadcasts presence per
room, instead of picking an arbitrary socket.rooms entry.
- presence broadcasts use a per-room-type event name (workflow keeps the bare
presence-update; others are namespaced).
Workflow handlers wrap manager calls with a shared workflowRoom(id) helper;
UserPresence.workflowId -> room (the client never reads that field).
Tests: existing 112 realtime tests pass unchanged (behavior gate) + 7 new
multi-room tests (refcounted session, presence isolation, multi-room disconnect,
per-type event names). tsc clean, boundaries + prune (14/25) green.
* fix(realtime): harden multi-room disconnect + id-guard room removal
Two fixes from an adversarial regression audit of the multi-room refactor:
- Disconnect now handles `disconnecting` (where `socket.rooms` is still populated
and authoritative) and falls back to the live Socket.IO room set for any room
the manager's stored state no longer tracked. This restores reliable presence
cleanup + departure broadcast even if the Redis `socket:{id}:rooms` key was
evicted or TTL-expired — the one behavioral gap vs the pre-refactor disconnect.
- REMOVE_ROOM_SCRIPT now only drops the socket's room mapping (and runs the
last-room session cleanup) when the stored id matches the room being removed,
matching the memory manager's existing id guard. Prevents a mismatched-room
call from wiping a different room's mapping or the shared session.
+1 test (id-guarded no-op removal). 120 realtime tests pass, tsc clean.
* fix(realtime): only rebroadcast disconnect-fallback rooms whose removal succeeded
Greptile 4/5 follow-up: the disconnecting-time fallback ignored
removeUserFromRoom's boolean and rebroadcast presence even when the removal
reported false. Now it only treats a room as removed (and rebroadcasts) when the
manager confirms it — symmetric with removeSocketFromAllRooms, which already only
returns rooms it actually removed.
* fix(realtime): exclude the disconnecting socket from its farewell broadcast
Greptile follow-up (transient-Redis-failure edge): if removeUserFromRoom fails on
disconnect, the socket's presence entry can outlive it (room hashes have no TTL)
and reappear as a ghost. Disconnect now broadcasts a correction to EVERY room the
socket was in (union of the manager's removed rooms and the live Socket.IO
membership) and passes the disconnecting socket id as excludeSocketId, so it is
never shown as a collaborator regardless of whether the Redis delete succeeded.
Any orphaned entry is still reclaimed by the next join's stale-presence sweep.
broadcastPresenceUpdate gains an optional excludeSocketId; normal broadcasts are
unchanged. +1 test.
* fix(realtime): make presence broadcasts liveness-aware (root-cause ghost fix)
Presence broadcasts now reconcile the stored list against the live Socket.IO
membership (io.in(room).fetchSockets()) before emitting, via a shared
filterVisiblePresence helper. This closes the residual behind the earlier
disconnect fixes: an entry orphaned by a failed removal (room hashes have no TTL)
could reappear in a LATER join's presence snapshot until the 75-min stale sweep.
Now such an entry is never emitted, because a non-live socket is filtered out of
every broadcast. Combined with excludeSocketId (which handles the disconnecting
socket, still momentarily live). Fail-safe: on a fetchSockets throw or an empty
result while entries remain, emit the unfiltered list rather than hide live
collaborators.
Also drops a dead guard in the disconnect union loop (rooms already removed are
skipped by the wasInRooms check) and the now-unused isSameRoom import.
+1 ghost-guard test. 122 realtime tests pass.
* feat(files): live presence avatars + live file tree via realtime rooms (#5932)
* refactor(tables): adopt shared durable event-log core (#5934)
* fix(realtime): address post-merge review-comment findings (#5937)
* fix(realtime): address post-merge review-comment findings
A re-audit of every inline review comment on the merged stack surfaced real
issues that the thread-resolutions and prior audits missed. Fixes:
Presence server (#5930 comments):
- connection.ts: snapshot `socket.rooms` SYNCHRONOUSLY before the first await.
Socket.IO clears the room set once the synchronous part of a `disconnecting`
handler returns, so reading it after `await removeSocketFromAllRooms` saw an
empty set — the eviction fallback was dead. (Cursor: "Disconnect fallback
misses live rooms".)
- workflow-room-service: restore the original managers' final unconditional
room-state wipe via a new `deleteRoom(room)` manager method, so a deleted
workflow leaves no lingering presence/meta even if a per-socket removal failed
or a socket joined mid-teardown. (Cursor: "Deletion skips final room wipe".)
Files (#5932 comments):
- workspace-file-manager.uploadWorkspaceFile now fans out the live-tree signal
(all direct-upload paths: multipart fallback, copilot create, /api/files/upload,
v1 files — the presigned path already notified). (Cursor: "Creates miss live
tree fan-out".)
- use-workspace-files-room: clear the pending retry timer on join success; and a
module-scoped intended-room guard defers the unmount `leave` so a rapid remount
re-claims the room and skips a stale leave — fixing presence flap + a
leave-after-join race. (Cursor: "Retry timer survives join success" + "Remount
churns files presence".)
- workspace-files handler: roll back a partial join (leave room + remove presence)
in the catch, mirroring the workflow join. (Cursor: "Join failure skips
membership rollback".)
+2 tests (deleteRoom). 127 realtime tests pass, both apps tsc clean,
api-validation + boundaries green.
* fix(files): scope workspace-files leave to a workspace (deferred-leave safety)
Self-review of the deferred-leave guard found a real bug: leave-workspace-files
was not workspace-scoped, so after a workspace switch (A->B) the deferred leave
from A would evict the socket from its new room B. The leave now carries the
workspaceId and the server no-ops if the socket's current files room differs.
Also excludes the leaving socket from the leave broadcast (consistent with
disconnect).
* fix(realtime): close files-room presence leak + validate join payload
Architecture-audit findings:
- S1 (real Redis leak): the files room inherited the shared manager but not the
workflow join's liveness sweep, so an UNGRACEFUL disconnect (pod crash — no
`disconnecting` event) left its presence entry in the no-TTL room hash forever.
Added a shared `sweepStalePresence(manager, room)` (fetchSockets liveness +
remove not-live-AND-stale entries, matching the workflow 75min threshold) and
run it on files join; also filter the join ack through `filterVisiblePresence`
so a joiner never briefly sees an un-swept ghost.
- S2: validate the client-supplied `workspaceId` on files join before it reaches
the DB query (matches the /api/workspace-files-changed guard; fails closed).
- N2: corrected the notify doc — it is awaited (guaranteed dispatch before a Node
route returns) and hard-bounded to NOTIFY_TIMEOUT_MS, not "never block".
+1 test (sweepStalePresence keeps live/fresh, reclaims not-live-stale). 128
realtime tests pass, both apps tsc clean, biome clean.
* fix(realtime): workflow-deletion always notifies + cleans by socket.io membership
Review-round findings on #5937:
- Always emit `workflow-deleted` (was guarded by users.length>0), so a socket
still in the Socket.IO room after a Redis presence eviction is told the
workflow is gone before socketsLeave kicks it — the editor no longer keeps
showing a deleted workflow. (Cursor: "Silent kick skips deletion event".)
- Clean per-socket state for the UNION of live Socket.IO members and
presence-tracked sockets, so an evicted/late-joined socket's room mapping +
session are dropped too — not just presence-snapshot sockets. (Greptile: "Room
deletion leaves reverse state".)
- deleteRoom now logs AND rethrows on Redis failure (like addUserToRoom) so a
failed wipe isn't reported as a clean deletion; the request surfaces it.
(Greptile: "Room deletion failures are suppressed".)
The two "deferred leave drops new membership" P1s were already fixed by the
workspace-scoped leave in a prior commit (leave carries { workspaceId }; server
no-ops on mismatch). 128 tests pass, tsc + biome clean.
* refactor(files): drop module-scoped deferred-leave; rely on workspace-scoped leave
Removes the one non-idiomatic construct (a module-level mutable
`intendedFilesWorkspaceId` + queueMicrotask). It only guarded a same-workspace
CONCURRENT remount, which doesn't occur in production (folder nav is shallow/no
remount; list<->detail is sequential) — a dev-StrictMode-only case. The real
cross-workspace race is already handled by the workspace-scoped leave: if B's
join runs first (auto-leaving A), A's leave no-ops because the socket's current
files room is B. Simpler, idiomatic, prod-correct.
* feat(realtime): Yjs relay server for collaborative document editing [4/N] (#5941)
Server-side Yjs relay for collaborative document editing (live carets + text selection) in the Files rich-markdown editor. Faithful y-websocket-style relay over the existing authenticated Socket.IO connection + shared room abstraction; in-memory Y.Doc + Awareness per file; awareness ownership binding, userId-keyed client-id uniqueness, seeder election with deadline re-election, concurrent-JOIN generation guard. 25 relay tests. Reviewed to Greptile 5/5 + Cursor pass across multiple rounds, plus an independent 4-lens audit (correctness/security/conventions/simplicity) and /simplify + /cleanup passes.
* feat(files): collaborative document editing — client provider + editor (#5946)
Client Yjs provider (FileDocProvider over the authenticated socket) + TipTap Collaboration/CollaborationCaret wiring for live carets + text-selection in the Files rich-markdown editor. Collaboration is a Files-page-only surface (explicit `collaborative` opt-in), disjoint from agent-streaming. Read-only + autosave-gated until synced+seeded. Merges into the realtime-rooms integration branch.
* feat(tables): live collaboration — cell-selection presence + live mutation propagation (#5957)
* feat(tables): live cell-selection presence — protocol + server + client hook
The realtime spine for Google-Sheets-style table presence (mode A, socket):
- @sim/realtime-protocol/table-presence: centralized wire protocol (events +
TableCellSelection {anchor, focus, editing} + payloads) so server emits and
client subscriptions can't drift.
- ROOM_TYPES.TABLE + resolveTableWorkspace registered in ROOM_WORKSPACE_RESOLVERS
(tableId -> workspace via userTableDefinitions, honoring archivedAt); roomName /
presenceEventName / disconnect cleanup / authorizeRoom all derive automatically.
- apps/realtime/src/handlers/tables.ts: join/leave (mirrors workspace-files) + a
table-cell-selection relay (mirrors the workflow selection channel), broadcasting
via roomName(room) since table rooms are namespaced. UserPresence gains a cell
field threaded through the memory + Redis managers (Lua ARGV[7], null clears).
- Extracted the duplicated resolveAvatarUrl into handlers/avatar.ts.
- use-table-room.ts client hook: joins over the shared socket, tracks the roster
(avatars) + patches per-socket cell deltas, exposes a throttled emitCellSelection.
Grid UI (avatars + selection overlay) lands next; concurrent cell-value edits
(last-write-wins via the durable log) are the follow-up PR.
* feat(tables): render live cell-selection presence in the grid
Wires the table presence room into the grid UI:
- Page (table.tsx): useTableRoom (gated off in embedded/mothership mode) —
renders <PresenceAvatars> in the header and passes remoteSelections +
emitCellSelection down to the grid.
- Grid emits its local selection: an effect resolves the index-based
anchor/focus to stable (rowId, columnId) via refs and broadcasts it (with an
editing flag for the active cell) through the throttled emitter.
- RemoteSelectionOverlay: draws each remote viewer's selection in their color
(getUserColor), a darker fill while editing, and name-on-hover — measured from
live cell rects in the content wrapper's space (scrolls with the grid),
hidden when rows are virtualized off-window, pointer-events-none so it never
blocks cell clicks (hover via pointer hit-test).
* test(tables): cover the table presence handler
Mirrors workspace-files.test.ts: join auth/unavailable/denied/success, plus the
cell-selection relay (asserts it persists via updateUserActivity and broadcasts
on the namespaced roomName, not the bare id) and leave.
* feat(tables): propagate manual cell edits live (last-write-wins)
A manual row edit now appends a lightweight 'edit' event to the durable table
stream; collaborators refetch the row (via the existing debounced rows-invalidate
the job events use) so the winning value shows live. The event carries no value —
peers refetch in their own wire format, so there's no auth-specific value
translation on the wire, and last-write-wins falls out of the DB's committed order
(the Google-Sheets model). Edits that also trigger a dispatch already emit
dispatch/cell events; the debounce coalesces the two.
* refactor(tables): apply /simplify findings
- Drop the dead 'add unknown peer' upsert branch in use-table-room (Socket.IO
ordering guarantees a peer is in the roster before their selection delta).
- TableCellSelectionBroadcast = TablePresenceUser & { cell } (was a copy-paste).
- Make TableGrid's presence props required + drop the unused empty-default/guard
(only table.tsx mounts it, always passing both).
- Drop the unused rowId from the 'edit' event (the handler invalidates all rows).
- Overlay: subscribe scroll/resize/pointer listeners once per scroll element and
cache the wrapper origin, so incoming deltas re-measure without re-subscribing
and the pointer hit-test never forces a per-move layout read.
- Server: cache the immutable socket session so a selection delta no longer reads
it from Redis every time.
* refactor(tables): apply /cleanup findings
- Fix the remote-selection name label contrast: text-white is unreadable on the
light-pastel user colors (same bug the Files caret fixed) → fixed dark #1a1a1a.
- Re-measure via useLayoutEffect so a moving peer selection updates before paint
(no one-frame position lag).
- Drop 'mothership' from a comment (constitution copy rule).
Six cleanup passes ran (effect, memo/callback, state, react-query, emcn, comment);
the rest confirmed clean — all state/memos/callbacks/effects are load-bearing,
presence correctly lives in useState (socket-pushed), and the edit→rows-invalidate
granularity is right.
* feat(tables): propagate every table mutation live (edit + schema signals)
Comprehensive live collaboration for all user table mutations, via two value-less
durable signals + named helpers (signalTableRowsChanged / signalTableSchemaChanged):
- edit (rows refetch): single + batch row create, cell/row update, batch update,
delete by id/filter, and upsert.
- schema (definition + rows refetch): column add/update/delete, workflow-group
add/update/delete, table rename, and CSV import (which can add columns).
- Client handles 'schema' by invalidating the table detail (exact) + rows.
Execution paths (column run, cancel-runs) and async jobs (delete/import-async,
job-cancel) already propagate via cell/dispatch/job events — verified applyJob
refetches on terminal. No reorder routes exist. Table archive (route DELETE) is a
deliberate follow-up: it needs a table-deleted redirect event, not a refetch signal
(which would 404).
* refactor(tables): apply comprehensive /cleanup audit findings
Holistic + react-query + comment audits over the whole PR:
- Security/crash fix: a remote peer's rowId flowed unescaped into the overlay's
querySelector — a hostile id ('x"]') threw SyntaxError inside a useLayoutEffect,
crashing every other viewer's page. CSS.escape it, and validate + whitelist the
untrusted cell payload server-side (shape + 200-char id bound) before it is
stored/rebroadcast.
- Simplify the CELL_SELECTION relay: the delta attached userId/userName/avatarUrl
that the client discarded (identity comes from the roster). Drop them + the
getUserSession lookup/cache entirely — the delta is now { socketId, cell }.
- React Query: schema handler also invalidates lists() (parity with the local
column-mutation set); document that the mutating client self-refetches by design.
- Comment tightenings; biome fixed a stale import order in workspace-files.ts.
* fix(tables): broadcast single-cell selections (focus falls back to anchor)
Cursor High: a normal cell click leaves selectionFocus null (the grid treats it as
a one-cell selection via focus ?? anchor), but the presence emit required BOTH anchor
and focus to resolve — so the most common selection never broadcast and clicking even
cleared a prior remote outline. Mirror the grid's focus ?? anchor semantics.
* fix(tables): reviewer + regression + per-LOC audit findings
Cursor review round (5 findings) + regression audit + per-LOC audit:
- Presence roster snapshot now KEEPS the cell we already hold for a known socket, so
a join/leave broadcast can't revert a fresher CELL_SELECTION delta.
- Reset the selection throttle on table switch (was unmount-only), so a pending
selection for table A can't flush into table B's room after a switch.
- Metadata writes (column widths, display) use a new lightweight 'metadata' signal
that refetches only the definition — a resize no longer forces peers to refetch rows.
- Overlay re-measures on row add/remove/reorder via a tbody childList MutationObserver
(a live refetch moves cells without a scroll/resize).
- Document the actor self-refetch create caveat (scrolled multi-page insert) accurately.
- isCellRef narrows to a partial instead of casting to the full type then re-checking;
drop a redundant mount measure() (the layout effect covers it); text-[11px]→text-xs.
* fix(tables): drop ineffective metadata propagation + re-measure overlay on column resize
Cursor round on b8f28b04b:
- Remove the 'metadata' signal entirely. The grid seeds columnWidths/pinnedColumns
from metadata ONCE (metadataSeededRef) and deliberately never re-applies them (to
avoid clobbering a local in-progress resize), so refetching the definition on a peer
never surfaced their width/pin change — an ineffective path. Width/pin live-sync needs
reconciliation that doesn't clobber a local resize; that's a deliberate follow-up, not
a no-op refetch. Structural changes still propagate via 'schema'.
- Overlay now also observes the content layer with the ResizeObserver, so a column
resize (which grows the content, not the scroll container) re-measures remote outlines.
- Presence-merge comment now states both sides of the trade-off.
* fix(tables): re-broadcast local selection on (re)join
Cursor Medium: a selection made before the room join completes (or held across a
reconnect) was dropped server-side and never re-sent, so peers didn't see it until
the local user moved it again. Track the current selection in a ref (set on every
emit, cleared on table switch) and re-emit it from handleJoinSuccess once the room is
joined.
* fix(tables): re-broadcast selection when a peer's row change shifts it
End-to-end lifecycle audit (Low-Med): the selection emit resolved the stable
(rowId, columnId) only on selection/editing change, not when a live edit/schema
refetch inserted/deleted/reordered rows. The index-based local selection then sat
on a different logical row than the rowId peers held, so your outline showed on the
old row until you moved. Re-run the emit on rows/displayColumns change and dedup an
unchanged result (also drops the redundant null-on-open emit) so the broadcast stays
consistent with the local highlight.
* fix(tables): schema invalidates run-state/enrichment + guard stale join
Cursor round on cdc8796b8 (2 Medium):
- schema handler used detail exact:true, so it skipped the activeDispatches +
enrichmentDetails sibling queries the local invalidateTableSchema refreshes via a
prefix match. After a peer deletes/restructures a workflow group, peers could keep a
stale running badge or enrichment panel. Now invalidates both siblings too (rows stay
on the debounce).
- Guard against a stale join stealing the room: a fast table A->B switch could let A's
async authorize finish after B, leave B, and strand the socket in A. Added a
per-socket monotonic join generation checked after authorize (mirrors the file-doc
relay's guard) + a test.
* feat(tables): live column width/pin/order sync
Collaborators now see each other's column resizes, pins, and reorders live —
the last piece of Google-Sheets-style layout parity.
- New lightweight `metadata` durable event kind (distinct from `schema`): only the
table definition carries UI metadata, so peers refetch the definition alone — no
rows/run-state refetch. The metadata PUT route now signals it.
- The grid reconciles server metadata against its in-progress gesture: the column
being actively resized keeps its live local width, and an in-flight column drag
blocks a reorder apply — so a peer's change never reverts the local action. Each
field is reference-guarded (React Query structural sharing keeps unchanged
sub-objects stable), so an unrelated peer change doesn't re-apply the others.
* fix(tables): escalate to schema signal when a reorder scrubs group deps
Independent audit of the metadata-sync commit found a stale-run-state hole: a
columnOrder PUT that moves a column left of a workflow group's leftmost column
makes updateTableMetadata scrub that group's dependencies and write a new schema —
a real structural change. But the route only fired the lightweight 'metadata'
signal (detail-only refetch), so peers' and the actor's activeDispatches /
enrichmentDetails queries stayed stale (a lingering running badge / enrichment
panel) — exactly what the 'schema' handler exists to prevent.
updateTableMetadata now reports whether it scrubbed the schema; the route emits
signalTableSchemaChanged in that case and the light signalTableMetadataChanged
otherwise. Width/pin/plain-reorder stay on the cheap detail-only path.
* feat(realtime): accurate in-file presence + collaborative-caret polish (#5965)
Per-session file-doc presence (avatars count other sessions like the canvas), StrictMode-safe stable Y.Doc (fixes blank-doc on join), flush caret cap + restored hover hit-slop, and three join-lifecycle race fixes unifying file-doc + workspace-files on one intent-tracked monotonic generation model. All findings root-caused with regression tests.
* feat(files): smarter bullet delete/indent and untitled-file title sync (#5971)
* improvement(files): smarter bullet delete/indent, fix empty-nested-bullet heading corruption
Backspace at the start of a list item now outdents a nested item or clears a
top-level item to a paragraph in place instead of deleting the row and jumping
the caret to the previous block; Enter on an empty nested item outdents. Empty
non-trailing top-level items still collapse cleanly since they cannot round-trip
as a lifted paragraph.
Also strips nested empty list-item marker lines on serialize: a nested empty
bullet re-parsed as a Setext heading underline, silently turning its parent line
into an H2 and dropping the bullet. Top-level empty items are preserved.
* feat(files): sync an untitled file's name with its leading heading
While a file is still named untitled(.md), typing a leading heading auto-renames
the file after it (debounced), and renaming the file first seeds a leading H1
from the new name. One-shot: coupling stops once the file has a real name, and
the heading seed always prepends so existing content is never clobbered.
* fix(files): count inline atoms in list-item emptiness, keep multi-block items on Backspace
Addresses review findings on the list Backspace logic:
- Emptiness now uses the caret block's content.size (counts inline images/mentions),
not textContent, so a bullet holding only a non-text atom is no longer treated as
empty and deleted.
- An empty first block whose item has sibling blocks removes only that block instead
of lifting the whole item out of the list.
* fix(files): preserve the untitled to named heading seed across a rename during editor load
The parent captures the file name at mount (before content/session finish loading) and
passes it as the transition baseline, so a rename that lands in the loading window is still
seen as an untitled to named transition and the leading heading seed is not skipped.
* fix(files): drop the name-to-heading seed, keep title sync one-way
Removes the effect that inserted a leading H1 when an untitled file was renamed. On the
collaborative Files page every open client observed the untitled-to-named transition and
inserted into the shared doc, producing duplicate headings; it could also re-insert a heading
a user had just deleted while a rename was in flight. Seeding document content from an async
rename transition is the wrong model on a shared editor. The primary direction — typing a
leading heading renames a still-untitled file — is unaffected (it never mutates the doc).
* fix(files): keep empty lines between paragraphs on reload
The chunked markdown parser (parseMarkdownToDoc) parses each block stripped of the
blank lines between them, so it dropped the empty paragraphs @tiptap/markdown builds
from runs of blank lines — a saved visual blank line silently vanished on the next
load (the settle/reopen re-seed goes through the chunker). The whole-document parser
preserves them, but whether a gap yields an empty paragraph is a global, block-type-
dependent decision (kept between two paragraphs, dropped after a heading), so it can't
be reconstructed block-locally. Route documents with empty-paragraph blank-line spacing
to the whole-document parser for exact fidelity — the same tradeoff NON_CHUNKABLE makes;
ordinary single-blank-line separation still takes the fast chunked path. Adds a suite
asserting chunked output matches the whole-document parser for leading/trailing/between
gaps and around lists/headings.
* fix(files): only auto-name an untitled file when the user can edit
The debounced untitled→filename hook ran on every onUpdate — including the mount-time
seed and for view-only viewers — without checking edit permission, so a read-only user
could schedule a rename they have no permission to make (a spurious, server-rejected
write). Gate the derive-title on editor.isEditable (canEdit + settled + collab-ready,
the same signal the autosave path uses), at both schedule and fire time.
* fix(files): normalize line endings before the empty-paragraph guard; Enter/Backspace symmetry
- markdown-parse: EMPTY_PARAGRAPH_SPACING/NON_CHUNKABLE tested the raw body, but a classic
\r-only file (blank lines are \r) would miss the \n-anchored guard and still be chunked,
dropping empties. Normalize line endings once up front so the routing guards, the chunker,
and the parser all see the same \n. +CRLF/CR test cases.
- keymap: Enter on an empty first block of a multi-block item now removes only that block
(removeEmptyWrappedBlock) instead of exiting the list, mirroring the Backspace hasSiblingBlocks
case — the trailing check no longer swallows multi-block items. +test.
* fix(files): editor audit follow-ups (trailing-blank read-only, collab rename, over-strip)
A 4-agent independent audit (UX vs inkeep + SOTA, cleanliness, adversarial correctness)
surfaced these:
- HIGH regression: files ending in a blank line opened READ-ONLY. The empty-paragraph
routing preserved a TRAILING empty paragraph, but postProcess collapses trailing newlines
→ serialize/parse non-idempotent → isRoundTripSafe flipped the file read-only. A trailing
empty paragraph can't be serialized stably, so parseMarkdownToDoc now strips trailing empty
paragraphs and the guard no longer routes on trailing blanks. Interior/leading empties are
unaffected. +regression tests.
- Medium: the debounced untitled→filename rename fired on remote Yjs edits too, so every peer
renamed and could rename from a not-yet-synced heading. Gate on isChangeOrigin (local edits
only; false for non-collab surfaces).
- Medium: stripEmptyListItemLines over-stripped a nested empty item that follows a same-indent
sibling (a real placeholder the parser keeps). Narrowed to the actual Setext hazard — an empty
item DIRECTLY under a shallower parent line — matching the function's own docstring intent.
Probe-verified. +test.
- Low: corrected untitled-title.ts docstring that described a reverse name→heading coupling
removed during review.
* fix(files): a remote edit must not cancel the local rename debounce
The isChangeOrigin gate cleared the debounce timer BEFORE bailing on a remote update, so
a peer's edit arriving within the 600ms window cancelled the local user's pending rename.
Bail on isChangeOrigin first, before touching the timer; only local edits clear/reschedule it.
* docs(files): correct EMPTY_PARAGRAPH_SPACING rationale after trailing-strip
The stacked trailing-empty-paragraph strip made the older comment overstate a
correctness necessity it no longer owns, mislabel trailing runs of 2+ blanks,
and advertise dead CRLF handling. Reword to match what the code actually does.
---------
Co-authored-by: Waleed Latif <walif6@gmail.com>
* fix(realtime): access-revalidation multi-room safety + cleanup pass
Fix a blocker surfaced by a full cleanup/simplify audit of the branch: the
access-revalidation sweep (staging's workflow-only #5917) treated every entry
in socket.rooms as a workflow id, but the generalized multi-room model puts
namespaced files/tables/file-doc rooms on the same io. It would resolve those
as bogus workflows, get null, and evict files/tables collaborators every ~30s.
collectScanTargets now decodes each room name with parseRoomName and sweeps
only workflow rooms; added a regression test and fixed the now-false TSDoc.
Other audit fixes (all behavior-preserving):
- workflow.ts reuses resolveAvatarUrl (drops db/user/eq imports duplicated
from avatar.ts)
- PresenceAvatars: mr-1 was baked into the shared component, silently adding a
margin to the workflow sidebar stack; moved to an optional layout className,
re-applied on the tables/file-doc header surfaces only
- table DELETE routes only signal collaborators when rows were actually removed
(matches PUT)
- events.ts definition kind: drop the never-emitted reason:'schema', fix its doc
- event-log: rename buildMemory -> buildEntry (it builds the entry on the Redis
success path too, not just the memory fallback)
- remove dead resolveWorkspaceIdForRoom export; parallelize per-socket removals
in handleWorkflowDeletion; gate the table columnIndexById map on remote
selections; move file-doc module TSDoc off the FileDocOwner interface; fix a
stale @returns
* fix(realtime,tables): close table-presence race + v1/copilot live-collab gaps
Validated each issue with subagents before implementing the cleanest fix.
- tables LEAVE in-flight-join race (B8): the table handler tracked no current-table
intent, so an unscoped/same-table leave during an in-flight authorize left the
socket stranded in the room (present in the roster, broadcasting a ghost until
disconnect). Mirror workspace-files: a closure-local currentTableId + a leave that
advances joinGeneration to cancel the racing join. + 3 regression tests.
- v1 + copilot live-collab signal gap (D1): tables edited via the v1 public API or
Sim/copilot emitted no edit/schema signal, so open collaborators didn't live-update.
Add the signals at those call sites (add-only, matching the existing route seam) —
never in the service, so execution writes can't double-emit. Sync-only for copilot
bulk ops, guarded on affected/deleted count; async job branches stay covered by
their kind:'job' events; create/delete/get untouched.
- table join read consolidation (B5): sweepStalePresence returns its roster so the
same-tab dedup reuses it instead of a second getRoomUsers.
- shared authorize slice (B6): extract only the guard-safe authorize->allowed branch
into resolveRoomJoinAuth, shared by the three room handlers (the full preamble stays
inline — file-doc's generation capture sits mid-ladder and must not move).
- resize-revert flicker (E3): a peer's value-less metadata event forces a refetch that
could momentarily revert a just-finished local resize; a pendingWidthWriteRef keeps
local widths leading until the width PUT settles.
- embedded-mode stray emit (E5): gate emitCellSelection on a bound table id so the
embedded surface stops broadcasting cell selections the server drops.
* fix(tables): close two copilot live-collab signal gaps + harden presence sweep
Follow-ups from a comprehensive review of the branch:
- copilot batch_update_rows and import_file's inline append branch wrote rows
but emitted no live-collab signal, so collaborators didn't see those edits
live (the append's sibling replace branch already signalled). Add the guarded
signal to both, matching the internal route.
- sweepStalePresence now reads the roster before the fetchSockets liveness probe
and returns it on a probe failure, so same-tab dedup still runs during a
transient fetchSockets outage instead of being skipped.
- reword an internal comment off the retired "mothership" term.
* fix(realtime): guard table join commit + rollback against supersession; drop no-op eviction cleanup
Review round on #5991:
- Table join re-checked the generation only once after authorize, then awaited
leave/sweep/avatar before joining + registering presence. A table switch or
leave in that window stranded the socket in the wrong room, and the failure
catch could tear down a newer successful join. Resolve the avatar up-front,
re-check generation immediately before the membership commit (matching the
file-doc join), and skip the rollback/error for a superseded join. + a
post-authorize-window regression test.
- access-revalidation cleanup treated removeUserFromRoom's no-op false as a
transport failure and re-enqueued a still-connected socket forever. Only retry
when the socket is still mapped to the room (a healthy null mapping means the
entry is already gone). Repurposed the expired-mapping test to lock it.
* fix(realtime): guard table join leave-prior against superseding join
Round 2 on #5991: a superseded join's leave-prior could still run — during its
getRoomForSocket await a newer join commits to its room, so currentRoom is that
newer room and the superseded join would leave/remove/broadcast it before the
final guard aborts. Re-check the generation immediately after the lookup await,
before the leave mutation. Extended the post-authorize-window test to assert the
superseded join never tears down the newer join's room.
* fix(realtime): roll back a table join superseded during addUserToRoom
Round 3 on #5991: after the final generation guard, A could join + register
presence while a newer join B commits to its room during addUserToRoom's await
— B's leave-prior can't observe A's half-written entry, so A's late write wins
and strands the socket. Re-check after addUserToRoom and roll back A's own
Socket.IO join + presence (scoped to A's room, never touching B). + a regression
test hanging addUserToRoom mid-commit.
* refactor(realtime): DRY table-join supersession guards; fix stale comment
Cleanliness pass after the review rounds (no behavior change):
- Extract the four identical `joinGeneration !== joinAttempt || socket.disconnected`
checks into a named `superseded()` helper (the catch keeps its intentionally
narrower check).
- Remove a stale guard comment that was left stranded above the avatar resolve.
- Document the best-effort rollback catch.
* fix(realtime): file-doc rebind must not drop the current doc or leave a writable ghost
Two Cursor findings on the file-doc client-id ownership rebind:
- On a document switch, the prior room was left BEFORE the ownership check, so a
CLIENT_ID_IN_USE rejection dropped the socket from the old doc without joining
the new one (contradicting its own comment). Run the ownership check first, and
leave the previous doc only once the rebind is guaranteed to succeed.
- Reclaiming a client id removed the stale prior socket from owners + awareness
only; its socketToRoomName + Socket.IO membership remained, and handleMessage's
SYNC path gates on socketToRoomName (not owners), so it stayed able to write
document frames until disconnect. Fully evict the reclaimed socket. + 2 tests.
* refactor(realtime): serialize table join/leave to fix map-corruption at the root
Round 4 on #5991 surfaced a race the generation guards structurally cannot fix:
two concurrent joins for one socket race on the single-valued socket→room map —
a stalled addUserToRoom for table A lands late, clobbers a newer join's map entry
to A, and the rollback then wipes it, stranding the socket (map empty while it
holds table B). Guards protect JS suspension points; they can't stop an in-flight
Redis write from landing late.
Fix per architecture review: serialize this socket's JOIN + LEAVE on a per-socket
promise chain so their multi-step async Redis commits can never interleave —
restoring the atomic-commit property the synchronous sibling handlers get for free.
This DELETES the leave-prior guard and the post-commit rollback (the code that
caused the bug); four generation guards collapse to two identical superseded()
checks (skip a superseded queued op + one pre-commit check). Reworked the
interleaving-specific tests into a fast-switch-skips-superseded test; the leave-
cancels-join tests are unchanged. Local to the tables handler — no shared-infra change.
* fix(realtime): always roll back a failed table join; re-elect file-doc seeder on reclaim
Two review findings:
- Table join: the catch skipped rollback when superseded, but a socket.join that
landed before addUserToRoom threw leaves the socket in the Socket.IO room with no
matching socket->room map entry — unreclaimable by any later op (cleanup keys off
the map). Under serialization the skip is unnecessary (the newer op hasn't
committed), so always roll back. Simpler + fixes the strand.
- File-doc reclaim: fully evicting the prior socket didn't release the seeder role
if it held it, so electSeederIfNeeded (which no-ops while seederSocketId is set)
never re-elected and an unseeded doc stayed empty until the deadline. Clear the
role on eviction so the join's election picks a new seeder. + 2 regression tests.
* fix(realtime): close revoke-race ghost presence in workflow join
An access-revalidation revoke landing between socket.join and addUserToRoom
socketsLeaves the socket while its presence mapping does not yet exist, so
cleanupEvictedSocket finds nothing to remove and the join then writes presence
for a socket already out of the room — a ghost collaborator until the stale
sweep. Hoist resolveAvatarUrl (the only await in that gap) above the re-auth
check so the whole re-auth -> socket.join -> addUserToRoom section is await-free,
matching the invariant the handler already relies on for the pre-join re-auth.
+ ordering regression test.
* refactor(realtime): serialize workflow join/leave; drop dead room-authz limb
Comprehensive independent audit follow-ups:
- workflow.ts join/leave now use the same opChain + joinGeneration serialization
as the sibling handlers (tables, file-doc, workspace-files). It was the only
async presence path left unserialized, so a rapid workflow switch A->B (or a
leave racing an in-flight join) could strand presence in room A — a ghost
collaborator still receiving A's operation broadcasts until disconnect. The
join now aborts a superseded op at start and again right before the membership
commit, and the catch always rolls back a partial join.
- leave-workflow drops the '&& session' gate: an idle user whose 1h session key
expired (while the 24h room mapping is still live) can now leave cleanly
instead of being stranded until disconnect. The room ref alone suffices.
- authorizeRoom: remove the dead ROOM_TYPES.WORKFLOW resolver + its
getActiveWorkflowContext import. Workflow authorizes through its own path and
never flows through authorizeRoom; the map now honestly covers only the
workspace-scoped types (files, file-doc, table).
- Remove unused isSameRoom (zero callers) and a needless useMemo in
PresenceAvatars (plain derivation, single copy).
- Tests: 4 workflow serialization/leave regressions.
All gates green: tsc (sim/realtime/packages) 0, 204 realtime + 11 protocol
tests, biome, api-validation, boundaries, prune 14/25.
* fix(realtime): align session TTL, harden committed joins from post-success rollback
Per-module comprehensive audit follow-ups:
- redis-manager: SESSION_TTL now tracks SOCKET_ROOMS_TTL (was 1h vs 24h). The room
set outlived the session, and since getRoomForSocket reads the room set while the
workflow handlers gate edits/presence on `room && session`, an active-but-idle
collaborator got wedged into 'session expired' after 1h — sticky until reload
(activity only EXPIREs the already-gone session; only addUserToRoom re-HSETs it).
Both keys refresh together, so they now expire together (restores the pre-refactor
consistency, where both shared one TTL).
- workflow.ts + tables.ts: a 'committed' flag stops the join catch from rolling back
a genuinely-joined user when a trailing ack/broadcast/metric step fails on a Redis
blip (a pure getUniqueUserCount log-metric failure could otherwise kick a live
collaborator after success was already acked).
- workflow.ts: leave-prior now guards `currentRoom.id !== workflowId` (a same-workflow
re-join no longer leave→re-adds and flickers peers' presence), and the join ack is
liveness-filtered via filterVisiblePresence — both for parity with the tables handler.
- platform-authz: honest docstring + 400 message for the workspace-scoped-only
authorizeRoom map (workflow authorizes via its own path).
- caret-presence: corrected an over-stated batching comment.
- +1 workflow regression test (post-success failure keeps the user joined).
Gates: tsc (sim/realtime/packages) 0, 205 realtime + 11 protocol tests, biome,
api-validation, boundaries, prune.
* fix(realtime): narrow join commit-guard to post-success; skip empty presence broadcast
Two Cursor findings on the prior audit-fix commit:
- The 'committed' guard in join-workflow/join-table was too broad: a failure
BETWEEN the membership commit and the success ack (e.g. getWorkflowState) hit
'if (committed) return' and emitted neither success nor error, hanging the
client while it sat in the room. Replaced with the narrower shape: only the
purely-decorative post-success steps (peer broadcast + log-metric) are wrapped
best-effort; anything before the success ack still rolls back and surfaces a
retryable error, so the client retries instead of hanging — while the original
goal (a benign broadcast/metric blip never kicking a live, acked user) holds.
- broadcastPresenceUpdate read the roster via getRoomUsers, which swallows a Redis
transport error to []. On a disconnect broadcast that emitted an empty roster and
cleared every remaining collaborator's presence until the next healthy update.
Split out a throwing readRoomUsers; broadcastPresenceUpdate now skips the
broadcast on a read failure (getRoomUsers keeps its swallow contract).
- Tests: pre-success failure rolls back + retryable error (no hang); post-success
failure keeps the user joined.
Gates: realtime tsc 0, 206 realtime tests, biome, boundaries, prune.
* fix(realtime): harden seeder recovery, join-generation, and misc robustness
Final line-by-line audit follow-ups (all LOW/MED, no P0/P1):
- file-doc: a sole client whose seed FETCH fails was added to triedSeeders,
re-election found nobody, and the document stayed permanently empty until
reload. Re-offer seeding a bounded number of rounds (MAX_SEED_ROUNDS) before
giving up. Also bound clientId to a non-negative integer (it is an ownership key).
- tables + workspace-files: validate the room id BEFORE advancing joinGeneration,
so a malformed/rejected join can't cancel a legitimate in-flight join.
- workflow + tables + workspace-files: suppress the client-facing join error when
the op was already superseded (a retryable error naming the abandoned room could
make a client re-join and cancel its newer join). The rollback still runs.
- redis-manager: set isConnected=true only after scriptLoad succeeds (and reset it
on failure) so isReady() can't report ready while the Lua SHAs are null.
- connection: apply the presence-bearing filter to the manager-removed set too
(symmetry with the fallback path).
- http.ts: validate workflowId on the four workflow endpoints (matching the files one).
- client: clear a pending join-retry timer before rescheduling (reconnect churn no
longer orphans a stray extra join); clear caret fade timers on plugin destroy;
seed-effect cleanup reports NOT-ready (safe direction).
- Tests: bounded seeder recovery, cell-selection strip-junk, TABLE round-trip,
presenceEventName.
Gates: tsc (sim/realtime/packages) 0, 208 realtime + 12 protocol tests, biome,
api-validation, boundaries, prune.
* fix(realtime): gate isReady() on loaded script SHAs, not just connection
Follow-up to the prior isConnected change, which was incomplete: the redis client's
'ready' event flips isConnected=true on connect — before initialize() loads the Lua
scripts — so a bare isConnected check reports ready while removeUserFromRoom /
updateUserActivity would silently no-op on a null SHA. Gate isReady() on the SHAs
too, so the POST endpoints return a retryable 503 during that startup window instead
of proceeding against unloaded scripts. Standard readiness-probe discipline.
* fix(files): offline read-only fallback when the realtime doc never syncs
When the realtime server is unreachable (offline, server down, socket never connects),
the collaborative editor would sit blank and read-only forever — content only arrives
via provider sync events that never fire. Add a bounded connect-deadline to the Yjs
provider: if no first sync lands within CONNECT_DEADLINE_MS, latch fatal and emit a
synthetic non-retryable join-error — the exact path a real fatal rejection already uses,
which seeds the file's stored content read-only. Latching fatal also stops a late
reconnect from syncing server state in and merge-duplicating the locally-seeded content
(the documented Yjs 'non-empty doc ignores initial value' gotcha).
Deliberately NOT adding durable Yjs snapshot persistence / server-side seeding: TipTap
can't run the markdown->Yjs conversion server-side (Collaboration extension errors under
jsdom), and a durable binary snapshot would create a dual source of truth with the
markdown file (edited by copilot / PUT / download). The client-seeder + bounded
re-election is the correct architecture for a markdown-is-truth model; this closes its
one real user-facing gap without persistence, a migration, or dual-truth.
Timer cleared on first sync, on a real fatal rejection, and on destroy. +2 tests.
Gates: sim tsc 0, 496 editor tests, biome. Needs live offline->reconnect verification.
* fix(realtime): validate workflow join id before generation bump; scope file-doc join rollback to its target
Two Cursor findings:
- join-workflow bumped joinGeneration before validating workflowId (unlike tables /
workspace-files, which I'd already fixed). A malformed/empty join could advance the
counter and cancel a legitimate in-flight workflow switch. Validate the id first. +test.
- The file-doc join catch called cleanupFileDocForSocket unconditionally, which keys off
socketToRoomName. During a document SWITCH that fails before rebinding (e.g. a throw in
client-id reclaim), that binding still points at the socket's PRIOR, valid document —
so the rollback tore down a document the socket was validly in. Only run that cleanup
when the binding already points at THIS join's target; otherwise the socket never
registered as an owner here and the only leftover is a freshly-created empty room,
dropped by destroyRoomIfIdle.
Gates: realtime tsc 0, 209 tests, biome, boundaries, prune.
* fix(files): drop late sync frames once fatal; file-doc join error suppression + retry-budget reset
Final safety-audit findings:
- CRITICAL: FileDocProvider.handleMessage had no fatal guard. After the connect
deadline latched fatal and the editor fell back to a read-only local seed, a
late SyncStep2 (slow server / flaky network / deploy) was still applied — merging
server state into the seeded doc (content duplication) and flipping synced=true,
which un-gated autosave and would persist the duplicate to the real file. fatal
guarded (re)join but not inbound sync. Now handleMessage returns early when fatal.
+test (late SyncStep2 after the deadline is ignored, doc stays empty + gated).
- file-doc join catch now suppresses the client-facing error when superseded
(matches workflow/tables/workspace-files) so a retryable error for an abandoned
file can't make a client re-join and cancel the newer one.
- table/workspace-files room hooks reset the retry budget on (re)connect so a prior
full exhaustion doesn't block retries after a reconnect.
- presence-visibility: corrected a stale TTL comment.
Gates: tsc (sim/realtime) 0, 209 realtime + collab/hooks suites, biome, boundaries, prune.
* feat(collab-doc): server-authoritative Yjs seeding (#6008)
Server-authoritative Yjs seeding for collaborative file documents: the realtime relay
fetches a Yjs seed built from the file's markdown (via the shared TipTap engine) and
applies it once per room, replacing the client-seeder election/handshake entirely.
- DOM-free markdown<->Yjs conversion core (markdownToYDoc / yDocToMarkdown /
applyMarkdownToYDoc) reusing the client markdown engine for parity by construction
- Internal x-api-key seed endpoint + realtime fetch; single attempt bounded under the
client readiness deadline, guard-release for join-driven retry, read-only fallback on
persistent failure
- Client readiness gate = synced && server seed flag; jsdom wired for the Next standalone
build (serverExternalPackages + outputFileTracingIncludes)
Foundation only — copilot-into-doc + markdown projection + durable persistence are Stage C.
* feat(collab-doc): Sim merge endpoint for copilot-into-doc (Stage C foundation)
buildFileDocMergeUpdate(docState, markdown) computes the minimal Yjs diff that turns a live
document into target markdown, via applyMarkdownToYDoc (a real updateYFragment diff, not a
replace) — so a copilot rewrite merges with concurrent user edits instead of clobbering them.
Exposed over the internal x-api-key /api/internal/file-doc/merge endpoint the realtime relay
will call: the relay owns the doc, the app owns the conversion engine, so the relay ships the
current state and applies the returned diff. Tested incl. concurrent-edit no-clobber.
* feat(collab-doc): realtime apply-edit — merge copilot markdown into a live doc
The relay can now stream a copilot edit into open editors: applyMarkdownToLiveFileDoc finds
the seeded live room, ships its state to the app's /merge endpoint for a minimal CRDT diff,
applies it (relaying to every editor, reconciled with concurrent user edits), and reports
'no-live-room' so the caller falls back to a direct file write when nothing is open.
- Generalize the realtime->app request module (file-doc-seed.ts -> file-doc-app.ts) with a
shared POST helper + fetchFileDocSeed/fetchFileDocMerge
- POST /api/file-doc/apply-edit on the internal x-api-key HTTP surface, returning { applied }
- Tests for the seeded-room merge relay and the no-live-room fallback
* feat(collab-doc): stream copilot edits into open editors (Stage C)
edit_content now, after its durable file write, best-effort merges the same markdown into the
file's live collaborative document (markdown files only). If a collaborator has it open, the
edit streams into their editor as a CRDT merge — reconciled with their concurrent typing —
instead of the file silently changing under them; the editor's existing autosave mirrors the
merged doc back to the file. No-op when nothing is open. Never blocks or fails the edit.
* fix(collab-doc): strip frontmatter on merge; gate live-merge to markdown
- buildFileDocMergeUpdate now strips YAML frontmatter (splitFrontmatter().body) exactly as
the seed does. Copilot passes full-file content, so without this the frontmatter merged
into the doc as editor content and autosave wrote it back over the file (corruption).
- Gate the live-doc merge on isMarkdownFileName (new server-safe helper) instead of the
over-broad !isDoc, so code/text edits don't pay the realtime round-trip for a format the
collaborative editor never renders.
* fix(collab-doc): make frontmatter collaborative so a merge can't revert it
The editor re-attaches its open-time frontmatter on every autosave, so the Stage C merge
(which triggers an autosave with no user action) could write stale YAML back over a copilot
frontmatter change — silently dropping it.
Carry the file's frontmatter in the doc's config map instead of locking it at open: the seed
stores it, the merge updates it (only when it actually changed, preserving the no-op diff),
and the editor re-attaches THAT value on save — falling back to the locked copy before the
seed lands and for non-collaborative docs. A server-side frontmatter change is now reflected
rather than reverted. New FILE_DOC_SEED.frontmatterKey; seed/merge tests cover it.
* fix(collab-doc): re-sync draft on a frontmatter-only merge
A server edit that changes only the frontmatter updates the config map but not the body
fragment, so TipTap's onUpdate never fires — the autosave draft kept the stale open-time
frontmatter, and an explicit save could revert the live change. Observe the config map and,
on a frontmatter-only change, re-attach the new frontmatter to the current body and push a
fresh draft (guarded on a synced body so it never races the seed's own onUpdate).
* fix(collab-doc): order the apply-edit/merge timeouts (outer > inner)
The Sim->realtime apply-edit timeout (4s) was shorter than the nested realtime->Sim merge
timeout (8s), so the outer call could abort while the relay was still merging — the relay
then applied the merge after edit_content had returned, racing a follow-on edit.
Split the shared realtime->app timeout: the seed keeps 8s (it reads a cold blob), the merge
gets a tight 3s (it is a pure conversion, no I/O), and the outer apply-edit is raised to 6s
so it always outlives the inner merge. Cross-referenced in comments to prevent drift.
* fix(collab-doc): address deep-audit findings (jsdom trace, timeouts, gate, races)
A 4-agent LOC audit + precedent research (TipTap/Hocuspocus/Yjs docs confirm the core
patterns are idiomatic) surfaced these real issues:
- The /merge route also lazy-requires jsdom but was missing its outputFileTracingIncludes
entry — a Docker/standalone build would 500 with MODULE_NOT_FOUND. Add it.
- The four conversion timeouts encoded an ordering invariant (merge<applyEdit, seed<readiness)
living only in prose across two apps. Hoist to a shared FILE_DOC_TIMEOUTS in
@sim/realtime-protocol with a test asserting the ordering; both apps import it.
- The copilot live-merge gate used an extension-only check, but the editor treats any
text/markdown-MIME file as markdown. Replace isMarkdownFileName with a MIME-aware
isMarkdownFile mirroring the client, so those files stream too.
- applyMarkdownToLiveFileDoc had no per-file serialization; overlapping merges could each
diff the same stale snapshot and apply out of order. Serialize per file via a promise chain.
- Editor hardcoded 'config'/'initialContentLoaded' instead of FILE_DOC_SEED constants (drift).
- Add .max() bounds to the merge contract body; document the best-effort-merge failure window
honestly (open editor + merge failure can drop a copilot edit until reload — closed by the
deferred durable-doc work).
* feat(collab-doc): multi-replica shared Yjs backend + server-side markdown persistence
Make collaborative file-doc editing correct across multiple ECS tasks (the
per-process Y.Doc previously assumed one replica per file).
- Shared Yjs backend over Redis Streams (apps/realtime file-doc-store): each
file's stream is the ordered, replayable log of updates; a multiplexed XREAD
tailer converges every task's in-memory doc. Coordinated single-seeder
election (SET NX + empty-stream recheck) fixes split-brain seeding.
- Doc-sync fans out to local clients + the stream; awareness stays on the
Socket.IO adapter. Snapshot+XTRIM compaction trims only integrated entries.
- Server-side persistence: project the live doc back to markdown via a new
/api/internal/file-doc/persist endpoint, debounced during editing and flushed
on last-disconnect, from the authoritative stream state. Collaborative editors
no longer client-autosave, closing the copilot clobber-window.
- Copilot merges apply through the stream (reach the live doc on any task) and
serialize cross-task via a Redis merge lock.
- Degrades to the original single-replica behavior when REDIS_URL is unset.
* fix(collab-doc): harden distributed locks and durability from review
Address Greptile + Cursor review of the multi-replica backend:
- Merge lock: retry LONGER than the lock TTL (guaranteed acquisition, never
merges against a shared base while a peer holds the lock) and AWAIT the stream
write before releasing, so the next task never diffs a stale base.
- Distributed locks (seed/merge/compact) now use ownership tokens + a
compare-and-delete release (Lua), so a lock that expired and was re-acquired by
another task is never stolen; acquisition fails CLOSED on Redis error.
- Seed: publish the seed to the stream AWAITED under the lock before releasing,
so a later seeder's empty-stream fence always sees it (closes the fence's
publish-after-release gap); TTL kept at the readiness deadline.
- Persist: persist the AUTHORITATIVE stream state even when this task's local doc
was never seeded, and capture the local fallback synchronously so a
last-disconnect flush never encodes an already-destroyed doc.
- Publish: retry a transient xAdd failure so a Redis blip can't silently drop an
edit from the shared log.
* fix(collab-doc): only persist a doc a user actually edited
Cursor review: a copilot durable write landing while a doc is being seeded could
have the stale seed projected back over it on last-disconnect, clobbering the
copilot edit even with no user changes.
Gate server-side persistence on a genuine user edit (socket-origin update): a
seed-only or merge-only doc is never projected back to the file (copilot writes
the file durably itself), so it can't clobber a concurrent external write.
* fix(collab-doc): close review-round race/durability gaps
Address Greptile P1 + Cursor findings:
- Seed publishes to the shared stream AWAITED *before* seeding the local doc, so
a publish failure leaves the doc unseeded and the stream empty for a clean
retry rather than serving an unpublished local seed a peer would re-seed over
(split-brain).
- flushPersist falls back to the synchronously-captured local snapshot when
getStreamState throws (not only when it returns null), so a transient Redis
read no longer drops the final durable write as the room is torn down.
- streamHasContent fails CLOSED (returns true on xLen error): a Redis blip can no
longer let the seed fence pass and double-seed.
- Client collabReady initializes from the collaborative prop, so a collaborative
editor never has a mount-window where client autosave could clobber the server
write.
* fix(collab-doc): unstick seed retry and persist tailed edits
Cursor review:
- ensureServerSeed clears serverSeedStarted when it aborts at the streamHasContent
fence, so a fail-closed Redis xLen (or a genuine peer-seed) no longer strands the
room unseeded with no retry.
- Persistence dirty-tracking now marks a doc edited on any post-seed update,
including a peer's edit relayed via the tailer (REDIS_ORIGIN), tracked via a
seededObserved flag. The last task to leave persists real edits even if it only
tailed them; the seed transition itself is still never counted, so a
seeded-but-unedited doc is never projected back over the file.
* fix(collab-doc): match client markdown post-processing on server persist
Cursor review: yDocToFileMarkdown serialized the body with yDocToMarkdown only,
but the editor save path runs postProcessSerializedMarkdown before applyFrontmatter.
Server persist could therefore write markdown differing from a client save (empty
list markers, callout un-escaping) — spurious blob churn / round-trip drift despite
the byte-identical claim.
Apply postProcessSerializedMarkdown in yDocToFileMarkdown so a server persist is
byte-identical to a client save and the client's dirty-check baseline. Add a
regression test guarding the composition.
* fix(collab-doc): close durability gaps at deploy boundaries + audit polish
From a comprehensive from-scratch audit (correctness, SOTA, cleanliness,
feature-completeness):
- Persist max-wait: a continuous edit burst kept resetting the 5s debounce and
never persisted; cap it so a burst flushes at least every 20s, bounding
unpersisted edits.
- Graceful-shutdown flush: flushAllFileDocRooms awaited in shutdown so a rolling
deploy / scale-in secures open edited rooms to durable markdown before exit,
instead of relying on the stream + a surviving task.
- Compacted-snapshot catch-up now marks the doc edited (REDIS_SNAPSHOT_ORIGIN): a
snapshot folds seed+edits into one frame, so a task catching up purely from it
no longer treats real edits as an unedited seed and skips persisting.
- Polish: delete dead __setFileDocStoreForTest; bounded retry loop; rename
acquirePersistSlot -> tryClaimPersistWindow with accurate docs; tailer
object-identity guard; fix stale comments (seed route, edit-content autosave).
* improvement(realtime): post-review fixes for collab dirty-state + relay lifecycle
- collab editor no longer latches a spurious 'Unsaved changes' prompt: report
dirty only when the client owns durability (canAutosave), since in a
collaborative session the relay persists the doc server-side
- guard shutdown against a double SIGINT/SIGTERM running teardown twice
- disconnect loc…1 parent ecf1d7d commit 10bfb5d
164 files changed
Lines changed: 33786 additions & 2146 deletions
File tree
- apps
- realtime
- src
- handlers
- rooms
- routes
- sim
- app
- api
- internal/file-doc
- merge
- persist
- seed
- table/[tableId]
- cancel-runs
- columns
- run
- events/stream
- groups
- import
- metadata
- rows
- [rowId]
- upsert
- v1/tables/[tableId]
- columns
- rows
- [rowId]
- upsert
- workspaces/[id]/files/register
- workspace/[workspaceId]
- components/presence
- files
- components/file-viewer
- rich-markdown-editor
- collaboration
- hooks
- home
- components/mothership-view/components/resource-content
- hooks/preview
- hooks
- tables
- [tableId]
- components/table-grid
- hooks
- hooks
- w/components/sidebar/components/workflow-list/components/workflow-item/avatars
- hooks/queries
- lib
- api/contracts
- collab-doc
- copilot
- request/go
- tools/server
- files
- table
- vfs
- folders
- realtime
- table
- views
- uploads
- contexts/workspace
- utils
- workspace-files/orchestration
- workspaces
- docker
- packages
- db
- migrations
- meta
- platform-authz
- src
- realtime-protocol
- src
- testing/src/mocks
- scripts
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
| 40 | + | |
| 41 | + | |
39 | 42 | | |
40 | 43 | | |
41 | 44 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
58 | 56 | | |
59 | 57 | | |
60 | 58 | | |
61 | | - | |
62 | | - | |
| 59 | + | |
| 60 | + | |
63 | 61 | | |
64 | 62 | | |
65 | 63 | | |
| |||
84 | 82 | | |
85 | 83 | | |
86 | 84 | | |
87 | | - | |
88 | | - | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
89 | 90 | | |
90 | 91 | | |
91 | 92 | | |
| |||
141 | 142 | | |
142 | 143 | | |
143 | 144 | | |
144 | | - | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
145 | 169 | | |
146 | 170 | | |
147 | 171 | | |
| |||
199 | 223 | | |
200 | 224 | | |
201 | 225 | | |
202 | | - | |
| 226 | + | |
203 | 227 | | |
204 | 228 | | |
205 | | - | |
| 229 | + | |
206 | 230 | | |
207 | 231 | | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
212 | 238 | | |
213 | 239 | | |
214 | 240 | | |
215 | 241 | | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | 242 | | |
220 | 243 | | |
221 | 244 | | |
222 | | - | |
223 | | - | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
224 | 248 | | |
225 | 249 | | |
226 | 250 | | |
227 | 251 | | |
228 | 252 | | |
229 | 253 | | |
230 | 254 | | |
231 | | - | |
232 | | - | |
| 255 | + | |
| 256 | + | |
233 | 257 | | |
234 | 258 | | |
235 | 259 | | |
| |||
243 | 267 | | |
244 | 268 | | |
245 | 269 | | |
246 | | - | |
| 270 | + | |
247 | 271 | | |
248 | 272 | | |
249 | 273 | | |
250 | 274 | | |
251 | 275 | | |
252 | 276 | | |
253 | 277 | | |
254 | | - | |
| 278 | + | |
255 | 279 | | |
256 | 280 | | |
257 | 281 | | |
| |||
357 | 381 | | |
358 | 382 | | |
359 | 383 | | |
360 | | - | |
| 384 | + | |
361 | 385 | | |
362 | 386 | | |
363 | 387 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
6 | | - | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
65 | 66 | | |
66 | 67 | | |
67 | 68 | | |
68 | | - | |
69 | | - | |
70 | | - | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
71 | 77 | | |
72 | 78 | | |
73 | 79 | | |
| |||
78 | 84 | | |
79 | 85 | | |
80 | 86 | | |
81 | | - | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
82 | 90 | | |
83 | 91 | | |
84 | 92 | | |
| |||
129 | 137 | | |
130 | 138 | | |
131 | 139 | | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
132 | 150 | | |
133 | 151 | | |
134 | | - | |
| 152 | + | |
| 153 | + | |
135 | 154 | | |
136 | 155 | | |
137 | 156 | | |
| |||
148 | 167 | | |
149 | 168 | | |
150 | 169 | | |
151 | | - | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
156 | | - | |
157 | | - | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
158 | 187 | | |
159 | 188 | | |
160 | | - | |
| 189 | + | |
161 | 190 | | |
162 | 191 | | |
163 | 192 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
0 commit comments