Decision: ADR-095. The full-text index is a disposable node-local derived index with exact source-cut/policy/schema manifests and the selected ZoneTree.FullTextSearch provider. Leased generations deliver publication and retirement; online maintenance delivers base-cut capture, delta catch-up and background build outside the canonical read gate, with canonical publication of the online generation. The capture/lifetime contract is RuntimeReadCuts under ADR-097. Files are owned by the node-local ZoneTree owner and never move with grain activations; distributed search merges results from several partitions through Orleans calls and does not share index files.
| Requirement | Acceptance and mapped tests |
|---|---|
| REQ-LEASE-001: publication retains borrowed old generations | AC-LEASE-001: an actual generation lease remains usable and its owned files remain present while another source-cut generation publishes; current acquisition selects only the fully verified replacement; the last old release retires only that generation. TextGenerationLifetimeTests |
| REQ-LEASE-002: finite lifetime and admission | AC-LEASE-002: independent readers of an already published generation share one read-only native index and retain separate budgets and iterators; releasing or cancelling a reader cannot close another reader's index. One builder, one retired generation and one current generation retain the existing three-generation physical ceiling; total disk/file/metadata/posting limits account every owned generation. Cold-build saturation, cancellation, retirement and healthy continuation remain explicit. |
| REQ-LEASE-003: cleanup and shutdown preserve truth | AC-LEASE-003: failed refresh/close/delete retains the unsettled owner/handle for joined retry, both primary and cleanup failures survive, repeated release/dispose cannot double-release or delete another generation; shutdown joins leases/build and closes every owned handle |
| REQ-LEASE-004: existing runtime behavior stays exact | AC-LEASE-004: actual SearchEngine/GraphSearch text/vector/corpus results, visibility/cut mismatch, restart and process interruption remain exact through the unchanged ITextProjection interface |
No raw borrowed storage view may escape its gate. leases operate on their owned generation; each query still performs current canonical candidate checks inside its authorized read cut. A source-cut mismatch builds a new generation and never borrows an old manifest as current truth. Publication closes and verifies the staged index and flushes its manifest before changing the manager's current pointer. A retired lease can finish its originally scoped work but cannot read new canonical state or make a current-authorization claim.
The lifetime-long single semaphore is replaced by a short synchronized manager transition plus independent bounded lease/build admission. No monitor or store gate is held across waits or user callbacks and no polling wait runs on a grain thread. The ITextProjection/lease public shape, faults and generated identity are unchanged. Published readers share the pinned provider's read-only index and create independent NoRefresh iterators. Every acquisition still verifies its complete scope, manifest, file inventory and canonical corpus. The generation owner retains the native handle between reads, refreshes and closes it during retirement or shutdown, and joins every reader before deleting files. An invalidation refuses new borrowing and waits for remaining readers before cleanup. Cold construction has one staged build slot. An asynchronous search that cannot acquire that slot while its owner is active releases its read view and awaits that owner's existing quiet signal with the original caller cancellation. It then recaptures current authorization and source data under the same cumulative budget and query admission. Synchronous search retains its typed busy refusal; permanent resource exhaustion and uncertain cleanup are never retried. TextGenerationAdmissionTests verifies a write during the wait, the fresh result, caller cancellation, unchanged storage on cancellation and healthy continuation.
The native owner records a bounded live file inventory after opening its published
index read-only and joining native background work. Subsequent leases verify that
inventory as well as the published manifest, scope and complete canonical corpus;
opening bookkeeping cannot silently replace persisted publication authority.
On retirement or shutdown, the owner joins the read-only native index and
validates the existing sealed manifest against the owned files and corpus.
Closing a reader does not stage or publish another manifest. If validation
fails after the native handle closes, shutdown retains the unsettled owner and
validates again on retry. TextReadOnlySettlementTests exercises a quota with
room for only the current manifest, unchanged manifest bytes, corruption on
repeated shutdown, and a healthy cold search after restoring the exact bytes.
TextProjectionLeaseTests covers overlapping reads without per-read manifest
publication, cancellation of one reader while another completes, and invalidation
with a held reader followed by repeated healthy rebuilds under a one-generation
physical bound. These are local operation checks, not competitor measurements.
Successful retirement removes the retired manager entry while still holding the existing physical gate that deleted its directory and settled its shared reservation. A current-generation read can overlap retirement, but its physical census never observes a deleted directory through a stale retired entry. The real RetiredGenerationDeleted boundary is observed after that joined transition, outside the gate. TextGenerationRetirementTests starts a current SearchEngine read at that boundary, checks the exact replacement result, joined old-reader cleanup, one retained current generation and healthy continuation. A failed close, delete or reservation settlement still retains its actual unsettled owner.
Capacity preflight must not reopen the published WAL while its index still owns that file. Inside the physical gate it captures only the manager's at most three actual generation references. An exact manager-owned, published generation with a still-open index receives a bounded live ownership/layout/size preflight without reopening payload files: root, leaf, source node and scope are verified against the retained generation and owner/manifest metadata, unknown, linked, nonregular or foreign paths are rejected, and all file, depth, generation and aggregate-byte ceilings are kept. Every tracked file is inspected without reopening, a current file over MaximumDiskBytes is rejected, and a live file length is not compared with its closed manifest length. Runtime generation references cannot be supplied by a caller, inferred from a sharing exception or replaced by a trusted-leaf list; every other generation keeps the complete checksum validation, and publication, reopen, retirement, deletion and restart keep exact lengths and checksums. The final aggregate census in RuntimeTextProjection.Release follows the same completed-operation rule as TextSettlement.Refresh: an incomplete or cancelled operation settles without consulting its exhausted caller budget while keeping all real file, byte and generation ceilings and distinct cleanup errors.
The overlap fixture owns a linked cancellation source for its admitted replacement task; an observation deadline cancels that source, resumes its actual posting barrier and joins the original task before the lease or projection is disposed. Expected generation leaves are sorted ordinally before the ordered comparison, every borrowed lease settles before its projection's shutdown even when an earlier operation fails, and the restart fixture preserves its real borrowed retired generation by the owned foreign-file failure before the failed projection shutdown closes both indexes. Frontend/new SDK/MCP syntax is N/A: unchanged public search surfaces consume this manager. Migration is N/A: no canonical record or persisted index format changes. Rollback stops the capable node and rebuilds disposable indexes.
flowchart LR
Build[One verified builder] --> Current[Publish current generation]
Current --> Retired[Previous generation retains old lease]
Retired --> Drain[Last reader settles]
Drain --> Delete[Verified owned retirement]
| Requirement | Complete operation acceptance |
|---|---|
| REQ-ONLINE-001 | AC-ONLINE-001: same-gate capture pins the actual snapshot and captures persisted principal/policy/resource, applied position, outbox upper and node/partition identity with an active canonical projection consumer retaining the necessary deltas; copied-byte seed traversal runs outside the canonical gate while genuine public update and delete commands complete |
| REQ-ONLINE-002 | AC-ONLINE-002: a distinct staged generation consumes the actual ordered retained deltas to a finite fresh upper; full independent literals prove updated terms replace original terms, deleted entities are excluded, and unchanged entities and complete metadata stay exact |
| REQ-ONLINE-003 | AC-ONLINE-003: final authority/cut/retention and complete corpus validation precede a fully flushed verified catalog publication; rejection, cancellation or exhausted bounds preserve the usable original active generation and joined cleanup |
| REQ-ONLINE-004 | AC-ONLINE-004: genuine same-root process interruption at the publication boundaries and same-owner restart select the last complete active generation, preserve exact canonical records and receipts and reject incomplete or corrupt staging; process kill is not power-loss durability |
| REQ-ONLINE-005 | AC-ONLINE-005: a real old lease keeps its exact original result and files through replacement; its final release retires only that generation, and an actual cleanup failure retains ownership and failures followed by a repaired healthy continuation |
Stages: internal capture composition and current-only derived catalog contract; capture metadata and pin; bounded off-gate seed build; ordered delta catch-up; final authority and complete corpus validation; durable selector publication; interrupted-publication recovery; leased retirement. Removing the existing RequireUpper equality is not delta catch-up, and the explicit TextIndexSelection, its single enrollment and the immutable Build/Restore upper U stay exact; implicit Search performs no canonical maintenance effect and no enrollment. Core/Search owns canonical capture/auth and consumer retention; the StorageRecovery provider owns the snapshot and physical-gate lifetime; Server/Search owns the staged build, derived catalog and retirement; Orleans/Search owns typed capabilities and the bounded parent/child calls; Query/Search consumes a validated selection without owning storage.
flowchart LR
Cut[Same-cut metadata and pin] --> Build[Off-gate staged build]
Build --> Delta[Ordered retained deltas]
Delta --> Validate[Fresh authority and complete corpus]
Validate --> Catalog[Durable derived catalog]
Catalog --> Retire[Leased retirement]
OperationKind.MaintainOnlineTextIndex=34 (appended; Batch 0 and the earlier kinds are unchanged). OnlineTextIndexMaintenanceRequest (alias keyload.search.online-text-index-maintenance-request): Id0 CommandId, Id1 Consumer (ProjectionConsumerRef), Id2 Collection, Id3 Field, Id4 ConsumerGeneration, Id5 NodeId, Id6 the partition identity; there is no caller-provided cut, trusted principal, capacity or deadline, and the persisted current administrator, consumer definition/generation/checkpoint, resource/field policy and partition identity are reloaded before every effect. OnlineTextIndexMaintenanceResult (alias keyload.search.online-text-index-maintenance-result): Id0 CommandId, Id1 Consumer, Id2 ConsumerGeneration, Id3 BaseCut and Id4 PublishedCut (TextIndexSourceCut), Id5 IndexSha256, Id6 TrackedRecords, Id7 Checkpoint (ProjectionBatchResult). BaseCut is the real initial snapshot and PublishedCut the final validated applied/outbox/owner authority; no later canonical head is claimed from an original completed result. A whole original stable-ID replay retains all fields and a changed same-ID request is rejected. OnlineTextPhase values: 0 Captured, 1 Seeded, 2 CaughtUp, 3 Validated, 4 Published, 5 RetirementMarked; GrainRequestProgress adds nullable OnlineTextPhase at Id4 (a closed operation phase, not a payload log or completion counter).
The SDK extension is MaintainOnlineTextIndexAsync returning Task<Result<OnlineTextIndexMaintenanceResult>> over the original write/commandId/cancellation path; the route is /search/text/online/maintain, the on-demand tool is keyload_search_text_online_maintain, and SQL SDK and official MCP use EXEC keyload_search_text_online_maintain @arguments through the existing SqlOperationRequest/catalog executor with no extra SQL grammar. Its advisory hints are readOnly=false, idempotent=true and destructive=true; guidance requires an already administrator-configured consumer, and a timeout or disconnect is an unknown write outcome. Initial discovery is not enlarged. The parent runs on the caller's connection grain with the ManagedCode CQRS stream (ICqrsStreamWriter<GrainRequestProgress,GrainOperationReply>): each producer write is awaited, existing stream shape admission/backpressure/cancellation and terminal failure mapping apply, and exactly one terminal result follows the actual publication and settlement. Producer, readers, child operations, pin and CTS are joined on success, cancellation and failure with primary plus cleanup errors and fatal semantics preserved; there is no one-way delivery, detached work, polling or deadline reset. The parent and its sequential children use the same connection identity, each with a fresh scoped typed envelope and current persisted administrator authority, run as ordinary connection operations with no per-connection operation count limit and use the exact original request expiry; each yielded child chunk is shape-admitted before the drain consumes it, and the stream has no frame cap. The ordered replay loop reads pages until the projection reports no more entries, under the request expiry; an elapsed expiry or lost retained history is a terminal refusal that preserves the original active index, followed by a healthy continuation.
The private operation OnlineTextPublicationPhase (purpose keyload-online-text-publication-verified) and the private capability read OnlineTextMaintenance (purpose keyload-online-text-capability) are issued as purpose-bound typed Orleans envelopes through GrainRequestCodec and identity restoration; generic public command and read purposes reject these identities and generic public command creation excludes the private publication kind. The internal Core factory alone issues it after validating the prepared candidate; the replicated payload is copied immutable data and the session and prepared bytes are node-local, bounded and owned.
OnlineTextPublicationPhaseCommand (alias keyload.core.search.online-text-publication-phase) has Id0 Request, Id1 Result, Id3 Leaf, Id4 ManifestSha256, Id5 GenerationId, Id6 ExpectedCurrentCommandId (nullable) and Id7 SessionId; its command ID is the original public CommandId. StoredOutcome.OnlineTextAuthority (Id8, nullable) is OnlineTextOutcomeAuthority (alias keyload.core.search.online-text-outcome-authority): Id0 ParentFingerprint, Id1 Collection, Id2 Field, Id4 Placement (PhysicalShardRecord), Id5 Leaf, Id6 ManifestSha256, Id7 GenerationId and Id8 ExpectedCurrentCommandId. The immutable public result stays StoredOutcome.Result. The canonical online-text-current family holds OnlineTextCurrentPublication (alias keyload.core.search.online-text-current-publication): Id1 PrincipalId, Id2 CommandId, Id3 ParentFingerprint, Id4 Consumer, Id5 ConsumerGeneration, Id6 Authority and Id7 PublishedCut. The current typed records have no application format discriminator. Physical placement, consumer generation, canonical source cut, manifest checksum and expected-current-command CAS keep their actual ownership and concurrency roles. The Orleans capability request carries SessionId, Request, Kind, Page, CheckpointIntent and Acknowledged; its closed kinds0–9 represent ResolveOriginal, Capture, Seed, PreparePage, ApplyIntent, SettleCheckpoint, Validate, IssuePublication, ReconcileCommitted and Abort. Capability results report actual phase state without granting caller authority.
Order: (1) the parent reloads the fresh persisted administrator, field/resource policy, the active already-configured consumer/generation/checkpoint and the partition identity, validates expiry/cancellation/buffers and resolves the original outcome through a separately scoped private capability before new work (a changed same-ID request conflicts, and a later generation retirement never erases an original completed result). (2) Capture of metadata plus the actual snapshot pin happens in the canonical gate through a callback-based disposable copied-byte pin; the charged bytes are traversed off gate while update/delete commands complete, within the bounded sessions, generations, files, bytes and leases. (3) A distinct generation is seeded, ordered retained projection pages are read to the observed finite upper, and intents are applied and the stable-ID projection acknowledgements committed through the connection grain; an empty final batch executes the empty-effects acknowledgement to obtain an actual receipt, and nothing is fabricated. (4) A fresh final validation cut validates complete records, revisions, token digest, inventory and the current policy/consumer/partition identity/retention; the Core factory rechecks the prepared witness and principal before issuing the private publication, and the expected applied/outbox/read generation/policy/schema/resource/consumer/identity/current-pointer values are verified again in the ordered AtomicCommandCommit transaction (an intervening canonical commit is a refusal). (5) The original Result, typed publication authority, canonical current pointer and apply watermark are committed together. A successful persisted receipt requires every configured copy to confirm durable apply and reports its actual PersistedCopies. PublishedCut is the validated document cut before this metadata publication. A failed CAS or cancellation preserves the prior selector, and an uncertain submission stays UnknownWriteOutcome until canonical resolution. (6) Only the committed pointer and outcome authorize the local catalog publication: encode the SHA256 envelope, CreateNew pending with WriteThrough/Flush(true), validate identity/manifest/owner/files, rename in the same directory and replace the runtime pointer; old files are retained for leased readers and deleted only after the last old lease joins.
The private publication belongs to the actual node-local storage owner; ordinary canonical data commands use Orleans calls for copies. Publication validates the prepared canonical expectations and source witness while the derived catalog retains its exact local owner and source cut. The checkpoint intent that produced the acknowledged ProjectionBatchResult is retained in that witness, and its stable command fingerprint is checked against the scoped stored outcome before the private phase is issued.
TextOnlineCatalog (alias keyload.server.execution-text.online-catalog, root text-online, catalog.bin / catalog.pending) has Id1 NodeId, Id2 Scope (TextProjectionScope), Id3 Placement (PhysicalShardRecord), Id4 Consumer, Id5 ConsumerGeneration, Id6 Leaf, Id7 AppliedPosition, Id8 ThroughSequence, Id9 ManifestSha256 and Id10 BuildCommandId (the original publication command identity). Before publication the staged index is closed and flushed, its complete manifest and file inventory are validated, the pending catalog is read back, and current canonical publication authority is checked. Failure before rename preserves the prior selector; an ambiguous failure after rename is neither success nor proof of no effect. Startup accepts only the exact committed pointer/outcome, manifest, owner and inventory. A committed publication with missing or pending catalog authorizes reconstruction only from its recorded generation after full validation; corrupt committed bytes refuse. Missing outcome, timeout or an unavailable owner cannot establish that no publication occurred. text-online/<generation>/staging.bin uses alias keyload.server.execution-text.online-staging, with Id1 NodeId, Id2 Leaf, Id3 Request, Id4 PrincipalId, Id5 ParentFingerprint, Id6 ExpectedCurrentCommandId and Id7 CapturedCut. Its checksummed CreateNew/WriteThrough/Flush(true)/readback protocol and exact owned-path validation remain required; expiry alone never authorizes deletion or promotion.
Cold reconciliation keeps the original canonical outcome, pointer, manifest and source-cut bytes. It verifies the committed authority, the original manifest checksum and the inventory, captures a fresh same-owner pin with matching incarnation/policy/schema/resource/partition identity and compares the complete corpus with the original manifest at the current outbox horizon; only legitimate read-generation advancement and nondecreasing applied positions may rebind the disposable catalog, and a changed corpus or authority refuses reuse until a genuine new authorized maintenance or exact-image repair. A superseded completed command returns its exact retained original result without reactivating retired files. Online checkpoint child IDs use the alias keyload.orleans.search.online-text-checkpoint-identity (Id0 original CommandId, Id1 page ThroughSequence; the Guid is the first 16 bytes of SHA256 over that serialization), and every page is read through the authorized projection API, committed with empty effects and verified against the stored outcome and the configured consumer checkpoint. The retained seed pin belongs to its session, which uses the original server-issued expiry, when the request has one, linked to shutdown, with one cumulative never-renewed ReadExecutionBudget; active capabilities use ReadExecutionBudget.EnterStageCancellation(callerToken), and parent failure joins Abort while the private publication completion/reconciliation ownership survives. The original coordinator completion is retained through caller cancellation: a cancelled wait may report UnknownWriteOutcome but neither settles the task nor permits catalog or staging release, and even a terminal UnknownWriteOutcome needs authenticated canonical reconciliation before retirement or promotion.
An implicit online read uses Query's internal ICurrentTextProjection with the actual WithQueryView view, persisted principal/resource, original SearchRequest and budget; explicit selection stays first. It validates the canonical online-text-current family (exact keys, full authority; only the matching local node, persisted principal, collection/field and active administrator-configured consumer is eligible, two eligible consumers are ambiguous and refuse, and corrupt eligible authority never falls back), the checksummed catalog, the original manifest and the full canonical corpus in the same view, and transfers ownership from the inspection pin to the reader lease. A missing or mismatched catalog refuses the read (the read never writes or promotes pending). Fresh ResolveOriginal uses a separate fresh budget, joins any retained original coordinator work and repairs the disposable catalog before returning the immutable original result; a retained pending file is removed only after checksum, shape and identity validation against a fully proven current publication.
One internal TextResourceOwnership, created at the text composition boundary of the node-local host, is passed to the bootstrap (text-projections), explicit (search-indexes) and online (text-online) owners. It uses only the centrally validated TextExecutionOptions ceilings, counts shared physical entries once, admits reservations before work and holds them until actual joined disposal or retirement; every root keeps its own node-owned receipt. Prospective disk admission reserves the canonical path and target file length of every admitted file mutation before the write begins and counts only growth beyond the observed length, metadata envelope publication admits its exact encoded length before CreateNew, and open file groups (linked handles of one canonical path) count once, detach on delete, move on replace and keep their length until the real stream disposal joins. Closing handles first join every write, then capture the final length, close the local stream and settle its ledger charge under the existing accounting gate, so DeleteOnClose cannot race a physical census. Whole-index eviction and index disposal run outside that gate. Seed traversal charges the same original grant for key/value examination, the retained decoded representation and the collection arrays: a VisitPrefix overload takes an admission callback invoked after each examined record and before owned copies, a refusal keeps the real attempted observation without admitting the allocation, and a capture failure whose cleanup settled completes the untouched grant while an aggregate (uncertain cleanup) failure retains it.
Every runtime text projection and native file provider requires its actual shared
TextResourceOwnership instance. Physical directory census and provider deletion
join through that owner's existing accounting gate, retaining each observed file
through its size inspection. TextPhysicalCensusTests holds the actual root
verification while native file deletion enters, verifies that deletion waits for
the census, then joins both operations and proves healthy owned-file creation,
readback and deletion plus the exact original canonical search result and store
position. This maps to REQ/AC-LEASE-002/003/004; broader performance and deployed
filesystem qualification remain separate.
REQ/AC-LEASE-001..004, REQ/AC-ONLINE-001..005 and REQ/AC-CUT-001..004 apply to a public search overlapping publication. Query SearchCapturedExecution validates the request inside DatabaseEngine.WithQueryView; Server TextCapturedReadFactory acquires the same selected text reader and canonical ZoneTree snapshot under that gate, carrying only observed scalar identity (SearchReadAuthority; no signing key, live view or role cache). StorageRecovery's IScopedReadCapture charges work before copies and captures the bounded target document prefix, exact resource metadata and, for hybrid search, the vector/lineage/effect/source dependency closure (an unadmitted lookup refuses without a live-store fallback). The live gate is released before off-gate closure work, the snapshot is joined and unused reservation settled, and the same independent FTS reader is retained through posting visitation, ranking and projection. Before exposing a result SearchTerminalPrivacy reacquires the persisted principal/resource/partition identity and validates the original owner/incarnation/read-generation/policy/schema/resource identity: committed position advancement is allowed, while changed authority, expiry, revocation or current row privacy refuses the entire result. A revoked publisher's repair advances the policy epoch and needs a genuine new authorized maintenance operation. The appended closed phase TextOriginalPostingRead reports a genuine posting Next while the lease is still owned; the reader, task and callback join before data is released. ConnectionReadCapabilities arms a call-local posting observation only around Query.ExecuteAsync.
Current functional source includes TextOnlineWholeOperationTests, TextOnlineSeedBudgetTests, TextOnlineFileOwnershipTests and TextCapturedPublicOperationTests. These exercise real capture/publication, literal bilingual reads, exact original result replay, bounded cancellation and file ownership, retained readers during update/delete replacement, persisted-policy repair and cold continuation. TextOnlineProcessRecoveryTests.ActualOnlinePublicationKillReopenRetainsFullOriginalOutcomeCorruptionRepairAndHealthyContinuation owns the actual process-kill cases at acknowledged canonical publication and pending-catalog flush. The complete .NET SDK, official MCP and SQL operation on the Aspire cluster remains a separate required qualification gate. Public maintenance and private publication have distinct current contracts; canonical documents, journals, outcomes and consumer records remain authoritative, with no older reader or format-conversion path.