Commit 630e7cb
authored
feat(copilot): stream file edits into the live collaborative Y.Doc (keep embedded view collaborative) (#6108)
* feat(copilot): stream file edits into the live collaborative Y.Doc
Copilot's file edits previously only reached the live doc once, at the final
edit_content write, so a collaborative editor watching the file saw nothing
until completion (streaming looked broken) and the client-side preview path
was suppressed in collab mode.
Make copilot a CRDT peer: as it streams append/update/patch content, merge the
growing markdown into the file's live Y.Doc via the existing apply-edit path
(a minimal updateYFragment diff, concurrent-edit-safe), throttled to ~250ms.
version is omitted for these intermediate merges — they advance the live doc
for viewers but are not durable checkpoints; the final edit_content write
carries the real contentUpdatedAt and reconciles the durable file. Per the
relay's persist gating, server-internal merges never schedule a persist, so a
copilot-only stream produces zero intermediate file writes.
- notify.ts: mergeEditIntoLiveFileDoc version is now optional (streaming omits it).
- file-preview-adapter.ts: throttled live-doc merge at the edit_content stream hook.
* fix(copilot): order + gate streaming live-doc merges; fast collab first render
Harden the streaming merge (adversarial review):
- Order + bound: dispatch through a per-file in-flight guard (drop-while-in-flight)
so a stale out-of-order snapshot can never land after a newer one and regress
the doc, and relay load is capped at one request per file regardless of rate.
- No wipe: gate append/patch on the base file content having loaded — a base-less
snapshot would diff to a delete-everything wipe of the seeded doc; update streams
a full rewrite from scratch and needs no base.
- Markdown-only gate: non-markdown files have no collaborative room, so skip the
wasted relay round-trip.
Fast collab first render (Issue 2): render the already-fetched markdown read-only
via generateHTML while the collaborative doc seeds, with the editor mounted-but-
hidden in the same layout box for a seamless swap on collabReady. Pure HTML — it
never touches the Y.Doc (client seeding duplicates the doc), and generateHTML
escapes text (raw-HTML snippets render escaped), so no XSS.
* test(copilot): cover streaming file edits into the live collaborative Y.Doc
Drives edit_content args_delta stream events through processFilePreviewStreamEvent
and asserts the live-doc merge: fires with the growing FULL previewText and no
version arg; is throttled (~250ms per file); is skipped for non-markdown files
and for a base-less append (the delete-everything wipe guard); and runs at most
one-in-flight per file. Verified to fail if any gate/guard is removed.
* fix(collab-doc): coordinate live-doc merge ordering in one place; close durable-clobber race
The second review found a residual: the durable edit_content write went through a
different path than the adapter's in-flight guard, so a late straggler streaming
merge could land after it and, via a persist, clobber the durable file's tail.
Move the per-file coordination into mergeEditIntoLiveFileDoc (the one place both the
streaming and durable paths call): a streaming (versionless) merge is dropped while
one is in flight for the file; a durable (versioned) write instead WAITS for the
in-flight streaming merge, so the final content is always the last merge applied and
can't be regressed by a straggler. Simplifies the adapter (drops its Set + helper).
Relocate the one-in-flight test to notify.test.ts (streaming-drops-while-busy +
durable-waits-then-applies-last); the adapter test keeps throttle/gates/previewText.
* fix(copilot): address review — order merges, exclude update, gate throttle, unhide stream
Review round on #6108:
- Greptile P1 (durable merges lose ordering): serialize ALL merges per file on one chain in
mergeEditIntoLiveFileDoc (each chains after the current tail), so concurrent durable writes
can't resume-and-fire out of order. notify now exposes isLiveDocMergeInFlight.
- Cursor High (update stream blanks the doc): only append/patch stream — they build on the loaded
base; update is a from-scratch rewrite whose partial snapshot would diff the full doc toward a
fragment, so it applies atomically at the durable write.
- Cursor Medium (throttle advances on a dropped merge): the adapter gates on !isLiveDocMergeInFlight,
so the send throttle advances only on an actual dispatch — no lag, no backlog behind a slow relay.
- Cursor Medium (placeholder hides a live stream): show the fast-render placeholder only when not
streaming, so a stream that starts before the doc seeds shows through the editor.
- Soften merge.ts/notify.ts comments per the lifecycle audit: only UNTOUCHED regions are preserved;
a region the merge rewrites reconciles toward copilot's content.
Tests updated: notify covers chain ordering + isLiveDocMergeInFlight; adapter covers append streaming,
throttle, non-markdown/base-less/update skips, and the in-flight skip.
* fix(collab-doc): reject stale durable merges at the relay (cross-process ordering)
The in-process merge chain only orders merges within one apps/sim process. Two durable
writes for the same file on DIFFERENT processes could reach the relay out of dispatch
order; the relay recorded the version monotonically but still APPLIED the older markdown,
regressing the live doc while the token stayed high (a later persist could then write the
stale content back over the durable file).
Enforce ordering at the relay — the single cross-process coordination point — using the
existing Redis primitives: under the per-file Redis merge lock, read the cluster-wide
synced version and SKIP a versioned merge that is not newer (a newer durable write already
landed). Make recordVersion await setSyncedVersion so it is durable before the lock
releases, so the next holder's staleness check reads a consistent value. Streaming
(versionless) merges are unaffected — they carry no durable version and are ordered
per-process by the caller.
Adds a relay test asserting a stale/idempotent versioned merge returns 'stale' and never
computes or publishes a diff.
* fix(copilot): match durable path — detect markdown by MIME type + name at the stream gate
The streaming gate checked isMarkdownFile with only the filename, while the durable merge
uses type + name — so a text/markdown file without a .md extension was skipped mid-stream
(it self-corrected at the durable write). Pass editIntent.contentType so streaming detects
the same set of markdown files as the durable path.
* test(copilot): assert throttle follow-through after an in-flight merge clears
* fix(collab-doc): order streaming merges by streamedAt so a late snapshot can't regress a newer durable write
* refactor(collab-doc): tidy merge-order docs + relay order object; cover multi-replica streaming stale-check
* fix(collab-doc): order streaming merges by causal base version, not wall-clock
A streaming snapshot now carries baseVersion (the durable contentUpdatedAt it was
built from) instead of a wall-clock streamedAt. The relay drops the snapshot when a
newer durable write landed since that base, so a concurrent human save can no longer
be clobbered in the live doc and then persisted over the durable file. Skew-immune:
both keys are DB-monotonic contentUpdatedAt values.
* fix(collab-doc): derive streaming baseVersion as contentUpdatedAt ?? updatedAt
Match the version line the seed/persist use so a legacy file with no content
version still ships an ordered streaming snapshot instead of an unordered one.
* fix(collab-doc): fail-closed on a streaming snapshot with no baseVersion
The live-merge gate now requires a numeric baseVersion, not just loaded base
content. A rare base with no file record (hence no version) would otherwise ship
an unordered snapshot the relay can't stale-check, risking a clobber of a
concurrent durable write. Skip the live merge instead; the durable write reconciles.
* docs(collab-doc): document the accepted concurrent-independent-streams limitation1 parent f07071c commit 630e7cb
14 files changed
Lines changed: 800 additions & 58 deletions
File tree
- apps
- realtime/src
- handlers
- routes
- sim
- app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor
- lib
- collab-doc
- copilot
- request
- go
- session
- tools/server/files
- realtime
- uploads/contexts/workspace
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
543 | 543 | | |
544 | 544 | | |
545 | 545 | | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
546 | 605 | | |
547 | 606 | | |
548 | 607 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
515 | 515 | | |
516 | 516 | | |
517 | 517 | | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
518 | 528 | | |
519 | 529 | | |
520 | 530 | | |
| |||
540 | 550 | | |
541 | 551 | | |
542 | 552 | | |
543 | | - | |
544 | | - | |
| 553 | + | |
| 554 | + | |
545 | 555 | | |
546 | 556 | | |
547 | 557 | | |
548 | | - | |
549 | | - | |
550 | | - | |
| 558 | + | |
551 | 559 | | |
552 | 560 | | |
553 | 561 | | |
| |||
561 | 569 | | |
562 | 570 | | |
563 | 571 | | |
564 | | - | |
565 | | - | |
| 572 | + | |
| 573 | + | |
566 | 574 | | |
567 | 575 | | |
568 | 576 | | |
569 | 577 | | |
570 | | - | |
571 | | - | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
572 | 583 | | |
573 | 584 | | |
574 | | - | |
575 | | - | |
576 | | - | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
577 | 588 | | |
578 | | - | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
579 | 617 | | |
580 | 618 | | |
581 | 619 | | |
| |||
595 | 633 | | |
596 | 634 | | |
597 | 635 | | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
598 | 641 | | |
599 | 642 | | |
600 | 643 | | |
| |||
604 | 647 | | |
605 | 648 | | |
606 | 649 | | |
607 | | - | |
| 650 | + | |
608 | 651 | | |
609 | 652 | | |
610 | 653 | | |
| |||
614 | 657 | | |
615 | 658 | | |
616 | 659 | | |
| 660 | + | |
617 | 661 | | |
618 | 662 | | |
619 | 663 | | |
620 | 664 | | |
621 | 665 | | |
622 | | - | |
| 666 | + | |
623 | 667 | | |
624 | 668 | | |
625 | 669 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
209 | 209 | | |
210 | 210 | | |
211 | 211 | | |
212 | | - | |
| 212 | + | |
213 | 213 | | |
214 | 214 | | |
215 | 215 | | |
216 | 216 | | |
217 | 217 | | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
223 | 223 | | |
224 | 224 | | |
225 | 225 | | |
| |||
Lines changed: 30 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| |||
297 | 297 | | |
298 | 298 | | |
299 | 299 | | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
300 | 312 | | |
301 | 313 | | |
302 | 314 | | |
| |||
923 | 935 | | |
924 | 936 | | |
925 | 937 | | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
926 | 943 | | |
927 | 944 | | |
928 | 945 | | |
| |||
947 | 964 | | |
948 | 965 | | |
949 | 966 | | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
950 | 975 | | |
951 | 976 | | |
952 | | - | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
953 | 981 | | |
954 | 982 | | |
955 | 983 | | |
| |||
0 commit comments