Add large-payload blob auto-purge (opt-in singleton job, worker/SDK side)

[wangbill (YunchuWang)]

Summary

Large orchestration payloads are externalized to Azure Blob Storage by the AzureBlobPayloads extension, with a token persisted in SQL instead of the payload bytes. When the orchestration is purged, DTS removes the SQL state but cannot delete the backing blob — it has no customer storage credentials. Only the worker does.

This PR implements the worker/SDK side. The backend records a durable tombstone for each externalized payload whose orchestration state is gone; the worker fetches due tombstones, deletes the blobs, and reports the outcome of every row so the backend can resolve, reschedule, or quarantine it.

Companion changes: contract in microsoft/durabletask-protobuf#76, backend in AAPT-DTMB PR 16368738.

gRPC contract

Two unary RPCs on TaskHubSidecarService (worker is the client). The vendored src/Grpc/orchestrator_service.proto is byte-identical to protobuf#76 — verified mechanically by exact-substring comparison, not by eye.

rpc GetLargePayloadTombstones(GetLargePayloadTombstonesRequest) returns (GetLargePayloadTombstonesResponse);
rpc ReportLargePayloadPurgeResults(ReportLargePayloadPurgeResultsRequest) returns (ReportLargePayloadPurgeResultsResponse);

• LargePayloadTombstone { partitionId, instanceKey, payloadId, token, revision }
• LargePayloadPurgeResult { identity, revision, disposition }
• Opt-in is google.protobuf.BoolValue large_payload_auto_purge_enabled = 12 on the existing GetWorkItemsRequest — no new handshake, and null means "no opinion".

revision is echoed back unmodified as a compare-and-swap guard, so duplicate or stale reports are no-ops without a per-row lease.

Dispositions

Three dispositions, split on whether a failure can self-heal. There is no Discarded: a success is reported explicitly as Deleted, and every failure is carried by Retry or Quarantined.

Disposition	Meaning	Examples
Deleted	Tombstone resolved	blob deleted; already absent; blob not store-owned
Retry	Transient; backend reschedules	storage unreachable, 401/403, 408/429/5xx, unknown token prefix
Quarantined	Deterministic; retrying can never succeed	malformed v2 token, HTTP 400, legacy v1 token

The worker never computes a retry delay. It reports the failure and the backend owns scheduling and backoff. The orchestrator reports the whole batch unconditionally — including retryable rows — because the backend needs to hear about a failure in order to defer the row. If every row comes back Retry (a storage outage), the cycle applies ErrorBackoff so an outage cannot become a tight refetch loop.

disposition is the entire outcome. An earlier revision also carried reason and storageErrorCode; both were removed after verifying they were write-only end to end — the backend persists them and nothing reads either one. Failure detail is logged by the worker instead, at the classification site, which is strictly richer than the enum was: 401/403, 5xx, and an unreachable account are distinct in telemetry where the enum collapsed all three into one value.

Blob ownership marker

A recognized token proves only that the text looks like one this store emits — the column is customer-writable. So ownership is recorded on the object itself: UploadAsync writes fixed blob metadata managed_by=dts, and the worker re-reads the target's metadata immediately before deleting.

• Marker present → delete, using If-Match on the ETag from that same read, so a mid-flight overwrite fails the delete rather than destroying newer content.
• Marker absent → blob left untouched, and the row is still reported Deleted, so the tombstone is resolved rather than retried forever. This is an expected outcome, not a defect, and must not be quarantined. The worker logs it distinctly so a customer whose payloads are all self-authored is still visible in telemetry.

The metadata name uses an underscore because Azure requires blob metadata names to be valid C# identifiers; managed-by would be rejected at upload.

Only blob:v2: tokens are auto-purged. A v1 token reaching this path is an invariant violation (v1 is excluded at insertion) and is quarantined rather than deleted.

Testing

Verified on a clean (--no-incremental) build:

• dotnet build Microsoft.DurableTask.sln — 0 errors
• test/Extensions/AzureBlobPayloads.Tests — 94 passed
• test/Client/Grpc.Tests — 56 passed
• Blame-attributed warnings — 127 unique in-repo, exactly 1 introduced by this branch

LargePayloadPurgeEnumParityTests pins proto↔managed parity for LargePayloadPurgeDisposition by value and name in both directions, and asserts that no inbound type exposes an enum. Disposition is the one enum crossing the wire, and its numeric cast in ReportLargePayloadPurgeResultsActivity is what decides whether a row is deleted or quarantined; that cast is safe only because enums travel outbound-only, and the test fails the build if a future change breaks that invariant.

Notes / intentional deviations

• Ships one intentional warning: CA1873 at BlobPurgeJobOrchestrator.cs:88 (unguarded logging). Kept deliberately for consistency with 77 existing instances across the solution. Blame-based attribution against main confirms this is the only warning this branch introduces.
• PurgedCount counts every Deleted row, including blobs skipped for lacking the ownership marker. Excluding them would pin the counter at 0 for a customer whose payloads are all self-authored, making a healthy draining job read as wedged; the worker log supplies the precision instead.
• PayloadStore.DeleteAsync is virtual with a default that throws NotSupportedException, so existing external subclasses are unaffected.
• BlobPurgeJob.Create is a no-op when already Active so racing client processes don't disturb a running job.
• BlobPurgeJobStarter implements IDisposable rather than disposing its CTS in StopAsync: that method returns on the host shutdown token while the ensure task may still be live, so disposing there would fault it with an unobserved ObjectDisposedException.

Depends on protobuf#76 for the authoritative contract and on the DTS backend serving these RPCs.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:268

• The exception message here says the payload lives in a different storage account, but this branch is taken whenever the token’s container URI doesn’t exactly match the configured container URI (which can also be the same account with a different container). This can mislead operators when diagnosing why deletes are being rejected.

            throw new PayloadStorageException(
                $"The externalized payload lives in a different storage account ('{decoded.ContainerUri}') than the " +
                $"currently-configured payload store ('{this.containerClient.Uri}'). Cross-account payload deletes " +
                "require identity (AAD) authentication with access to both accounts; connection-string / " +
                "account-key credentials are account-specific and cannot delete in another account.");

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/Extensions/AzureBlobPayloads/AutoPurge/Client/BlobPurgeJobStarter.cs:163

• StopAsync waits for the background reconciliation task to finish but never observes its result. If the background task faults (e.g., unexpected exception escapes the loop), the exception remains unobserved and can surface later via TaskScheduler.UnobservedTaskException. It’s safer to observe/ignore the completion explicitly once the task has finished.

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        this.cts?.Cancel();

        Task? pending = this.backgroundTask;
        if (pending is not null)
        {
            // The background loop observes cancellation and returns promptly; swallow any faulted/cancelled
            // result.
            await Task.WhenAny(pending, Task.Delay(Timeout.Infinite, cancellationToken)).ConfigureAwait(false);
        }

src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:268

• This exception message claims the token points to a “different storage account”, but the comparison is against the full container URI (scheme/host/port/path). A token in the same account but a different container would also hit this branch, making the message/action guidance misleading. Consider wording this as “different storage account or container”, or splitting account-vs-container mismatch if you want to keep the stronger wording.

            throw new PayloadStorageException(
                $"The externalized payload lives in a different storage account ('{decoded.ContainerUri}') than the " +
                $"currently-configured payload store ('{this.containerClient.Uri}'). Cross-account payload deletes " +
                "require identity (AAD) authentication with access to both accounts; connection-string / " +
                "account-key credentials are account-specific and cannot delete in another account.");

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/Extensions/AzureBlobPayloads/AutoPurge/Orchestrations/ExecuteBlobPurgeJobOperationOrchestrator.cs:22

• RunAsync calls CallEntityAsync(...) and returns it as object. With the default JsonDataConverter, deserializing JSON into typeof(object) produces JsonElement/primitive wrappers rather than the entity operation's actual return type, so this orchestrator does not reliably “return the result” as its XML doc claims. Since the only current caller (BlobPurgeJobStarter) uses this orchestrator for side-effects only, prefer the non-generic CallEntityAsync overload and return null to avoid type ambiguity and unnecessary deserialization.

      public override async Task<object> RunAsync(
          TaskOrchestrationContext context, BlobPurgeJobOperationRequest input)
      {
          return await context.Entities.CallEntityAsync<object>(input.EntityId, input.OperationName, input.Input);
      }
  

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskClientBuilderExtensions.AzureBlobPayloads.cs:129

• RegisterBlobPurgeJobStarter adds a new IHostedService every time UseExternalizedPayloads() is called. Because PayloadStore is registered via TryAddSingleton (first builder wins), multiple named client builders in the same host can end up with multiple BlobPurgeJobStarter instances that each read different named options (and may try to start vs stop the same singleton job), while all sharing the same PayloadStore. This contradicts the per-host singleton behavior described in the remarks and can lead to conflicting reconciliation behavior.

    static void RegisterBlobPurgeJobStarter(IDurableTaskClientBuilder builder)
    {
        string builderName = builder.Name;
        builder.Services.AddSingleton<IHostedService>(sp => new BlobPurgeJobStarter(
            sp.GetRequiredService<IDurableTaskClientProvider>(),
            sp.GetRequiredService<PayloadStore>(),
            sp.GetRequiredService<IOptionsMonitor<LargePayloadStorageOptions>>(),
            builderName,
            sp.GetRequiredService<ILogger<BlobPurgeJobStarter>>()));
    }

src/Extensions/AzureBlobPayloads/AutoPurge/Client/BlobPurgeJobStarter.cs:164

• StopAsync claims it will "swallow any faulted/cancelled result", but it never observes backgroundTask exceptions (it only awaits Task.WhenAny). If the background task faults before/while StopAsync runs, the exception can remain unobserved and surface later as an UnobservedTaskException. Consider awaiting the task when it completes (and catching) so faults are actually observed.

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        this.cts?.Cancel();

        Task? pending = this.backgroundTask;
        if (pending is not null)
        {
            // The background loop observes cancellation and returns promptly; swallow any faulted/cancelled
            // result.
            await Task.WhenAny(pending, Task.Delay(Timeout.Infinite, cancellationToken)).ConfigureAwait(false);
        }

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/Extensions/AzureBlobPayloads/AutoPurge/Client/BlobPurgeJobStarter.cs:167

• StopAsync awaits Task.WhenAny(...) but never observes exceptions from the background task. If the reconcile loop faults, this can surface as an unobserved task exception (process-level event / potential crash depending on runtime settings). Swallowing faults is fine here, but the exception should still be observed when the task completes.

        if (pending is not null)
        {
            // The background loop observes cancellation and returns promptly; swallow any faulted/cancelled
            // result.
            await Task.WhenAny(pending, Task.Delay(Timeout.Infinite, cancellationToken)).ConfigureAwait(false);

src/Extensions/AzureBlobPayloads/AutoPurge/Activities/DeleteExternalBlobActivity.cs:182

• PayloadDeleteOutcome.NotStoreOwned is mapped to LargePayloadPurgeDisposition.Quarantined, but the contract semantics (proto comment for LARGE_PAYLOAD_PURGE_DISPOSITION_DELETED and managed LargePayloadPurgeDisposition.Deleted docs) treat "blob not store-owned" as a terminal success that should be reported as Deleted (tombstone resolved). Quarantining this case will unnecessarily move rows into the operator-only path and diverges from the documented wire meaning. (Any tests asserting Quarantined for NotStoreOwned should be updated accordingly.)

            if (outcome == PayloadDeleteOutcome.NotStoreOwned)
            {
                this.logger.BlobPurgeDeleteQuarantined("NotStoreOwned", null);
                return new BlobPurgeOutcome(LargePayloadPurgeDisposition.Quarantined);
            }

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/Extensions/AzureBlobPayloads/AutoPurge/Activities/DeleteExternalBlobActivity.cs:182

• This maps PayloadDeleteOutcome.NotStoreOwned (missing ownership marker) to Quarantined. However the PR description states that when the marker is absent the blob should be left untouched and the row should still be reported Deleted so the tombstone is resolved (i.e., “must not be quarantined”). Either the implementation or the PR description needs to be updated, because this changes backend behavior (quarantine vs resolve) and alerting semantics.

            // The blob exists but this store never wrote it - it carries no ownership marker - so it is
            // quarantined, not reported as deleted. The marker is new in this PR, but main has been emitting v2
            // tokens without writing it, so on upgrade every pre-existing v2 blob reaches this branch. Reporting
            // Deleted here would tell the backend to hard-delete the tombstone row, orphaning that blob
            // permanently AND destroying the only durable record that it exists. Quarantine keeps the row and its
            // token backend-side as evidence and stops the backend polling it - the same disposition, and for the
            // same reason, as the v1 and malformed-token branches: a delete that cannot be verified as this
            // store's own must not be discarded as a success.
            if (outcome == PayloadDeleteOutcome.NotStoreOwned)
            {
                this.logger.BlobPurgeDeleteQuarantined("NotStoreOwned", null);
                return new BlobPurgeOutcome(LargePayloadPurgeDisposition.Quarantined);
            }

src/Extensions/AzureBlobPayloads/AutoPurge/Client/BlobPurgeJobStarter.cs:168

• StopAsync uses Task.WhenAny without ever awaiting/observing the background task when it completes. If the background loop faults, the exception can go unobserved (and may surface via UnobservedTaskException) even though the comment says faulted results are swallowed. Consider explicitly awaiting the task when it wins the WhenAny (inside a try/catch) so faults are observed.

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        this.cts?.Cancel();

        Task? pending = this.backgroundTask;
        if (pending is not null)
        {
            // The background loop observes cancellation and returns promptly; swallow any faulted/cancelled
            // result.
            await Task.WhenAny(pending, Task.Delay(Timeout.Infinite, cancellationToken)).ConfigureAwait(false);
        }
    }

src/Extensions/AzureBlobPayloads/AutoPurge/Client/BlobPurgeJobStarter.cs:183

• Dispose unconditionally disposes the CancellationTokenSource even if StopAsync returned early due to the host shutdown token, which can still leave the background task running and potentially fault it with ObjectDisposedException (the same risk called out in the remarks for StopAsync). Consider only disposing the CTS once the background task has completed (or avoid disposing it entirely in the timeout case).

    /// Deliberately not disposed in <see cref="StopAsync"/>: that method stops waiting as soon as the host's
    /// shutdown token fires, so the background task may still hold this source's token. Disposing it there would
    /// fault that still-running task with an <see cref="ObjectDisposedException"/> when it next registers a
    /// callback. The container disposes singletons after every <see cref="StopAsync"/> has returned, which is
    /// the safe point.
    /// </remarks>
    public void Dispose()
    {
        this.cts?.Dispose();
    }

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/Extensions/AzureBlobPayloads/AutoPurge/Client/BlobPurgeJobStarter.cs:167

• StopAsync waits for the background task to complete but never observes its final status. If the background task faults, the exception can remain unobserved (potentially surfacing later via UnobservedTaskException). If the task completed before the shutdown token fires, explicitly await it in a try/catch to observe and swallow the exception as intended.

            // The background loop observes cancellation and returns promptly; swallow any faulted/cancelled
            // result.
            await Task.WhenAny(pending, Task.Delay(Timeout.Infinite, cancellationToken)).ConfigureAwait(false);
        }

src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskClientBuilderExtensions.AzureBlobPayloads.cs:128

• RegisterBlobPurgeJobStarter uses AddSingleton, so calling UseExternalizedPayloads on multiple client builders will register multiple BlobPurgeJobStarter instances. That can start duplicate reconciliation loops and contradicts the per-host singleton behavior described in the remarks. Use a TryAdd-style registration (e.g., TryAddEnumerable) so only one starter is registered per container.

        builder.Services.AddSingleton<IHostedService>(sp => new BlobPurgeJobStarter(

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/Grpc/durable-task-scheduler/large_payload_purge.proto:46

• The PR description says GetLargePayloadTombstones/ReportLargePayloadPurgeResults are unary RPCs on TaskHubSidecarService and that opt-in is carried via a google.protobuf.BoolValue field on GetWorkItemsRequest. In the code, these RPCs (plus opt-in) are instead defined on a separate LargePayloadPurge service (SetLargePayloadAutoPurge, GetLargePayloadTombstones, ReportLargePayloadPurgeResults).

Please update the PR description (or the contract) so reviewers and downstream implementers aren’t working from an incorrect service/opt-in shape.

service LargePayloadPurge {
  // Records whether the caller's task hub has opted into large-payload blob auto-purge.
  //
  // This is a setting, not a WorkerCapability: a worker being able to resolve externalized payloads
  // is not the same as the customer opting into deleting them, and a capability list is presence-only
  // so it cannot carry an explicit false.
  //
  // Send this once per worker connection, before requesting work items. The backend only writes
  // tombstones for a task hub that has opted in, and a worker about to start draining them should not
  // observe a stale setting. Not calling it at all leaves the stored value untouched, which is what
  // keeps a worker that does not know about auto-purge from silently disabling a task hub that opted
  // in.
  rpc SetLargePayloadAutoPurge(SetLargePayloadAutoPurgeRequest) returns (SetLargePayloadAutoPurgeResponse);

  // Returns a bounded, deterministically ordered batch of due large-payload tombstones whose
  // external blobs the worker must delete. Scoped to the caller's authenticated task hub.
  // Only rows that are pending and whose next attempt time has arrived are returned; a row stays
  // pending until its outcome is reported, so this is safe under retries and duplicate callers.
  rpc GetLargePayloadTombstones(GetLargePayloadTombstonesRequest) returns (GetLargePayloadTombstonesResponse);

  // Reports the outcome of each attempted blob deletion. The backend owns retry scheduling and
  // branches solely on `disposition`: it deletes rows reported as DELETED, reschedules RETRY on
  // its own backoff, and moves QUARANTINED rows out of the active fetch while preserving their
  // evidence. The worker never computes a retry delay.
  rpc ReportLargePayloadPurgeResults(ReportLargePayloadPurgeResultsRequest) returns (ReportLargePayloadPurgeResultsResponse);
}

[Copilot]

Pull request overview

Copilot reviewed 47 out of 47 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 47 out of 47 changed files in this pull request and generated no new comments.

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskWorkerBuilderExtensions.AzureBlobPayloads.cs:70

• The file remarks say externalized payloads are configured per-host and later named workers “silently share the first builder’s registration”, but this registration uses builder.Name to read LargePayloadStorageOptions in PostConfigure. If multiple named workers call UseExternalizedPayloads, later builders can apply different options (interceptor behavior + LargePayloadAutoPurgeEnabled) than the singleton PayloadStore was constructed with, leading to inconsistent runtime behavior that contradicts the documented constraint.

Consider capturing the first builder name used for the shared PayloadStore in a shared singleton and using that single name consistently for all option lookups in this extension (or explicitly throw when a second distinct builder name attempts to configure externalized payloads).

        // Reuse the shared payload store when one is already registered (e.g. via AddExternalizedPayloadStore or
        // the client builder in the same process); only register our own as a fallback so we never create a
        // second, redundant PayloadStore.
        builder.Services.TryAddSingleton<PayloadStore>(sp =>
        {
            LargePayloadStorageOptions opts = sp.GetRequiredService<IOptionsMonitor<LargePayloadStorageOptions>>().Get(builder.Name);
            return new BlobPayloadStore(opts);
        });

src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskClientBuilderExtensions.AzureBlobPayloads.cs:115

• RegisterBlobPurgeJobStarter unconditionally adds a new IHostedService each time UseExternalizedPayloads is called. In a host with multiple named DurableTask clients, this can register multiple BlobPurgeJobStarter instances even though the type-level remarks say externalized payloads are configured per-host and later builders share the first registration. Multiple starters will each run a reconciliation loop, causing redundant background work and potentially conflicting “enabled/disabled” decisions if different names are configured.

Consider ensuring only one BlobPurgeJobStarter is registered per process (e.g., gate on a shared singleton registration, or record the first builder name and ignore subsequent calls).

    static void RegisterBlobPurgeJobStarter(IDurableTaskClientBuilder builder)
    {
        string builderName = builder.Name;
        builder.Services.AddSingleton<IHostedService>(sp => new BlobPurgeJobStarter(
            sp.GetRequiredService<IDurableTaskClientProvider>(),
            sp.GetRequiredService<PayloadStore>(),
            sp.GetRequiredService<IOptionsMonitor<LargePayloadStorageOptions>>(),
            builderName,
            sp.GetRequiredService<ILogger<BlobPurgeJobStarter>>()));
    }

src/Grpc/durable-task-scheduler/large_payload_purge.proto:46

• The PR description states the auto-purge opt-in is carried via a BoolValue field on GetWorkItemsRequest and that the purge RPCs are on TaskHubSidecarService. However, this change introduces a separate LargePayloadPurge service with a SetLargePayloadAutoPurge RPC (in addition to the fetch/report RPCs) and the worker calls that service during connection setup.

Please update the PR description to match the implemented gRPC contract/service shape so reviewers and downstream implementers don’t follow stale guidance.

service LargePayloadPurge {
  // Records whether the caller's task hub has opted into large-payload blob auto-purge.
  //
  // This is a setting, not a WorkerCapability: a worker being able to resolve externalized payloads
  // is not the same as the customer opting into deleting them, and a capability list is presence-only
  // so it cannot carry an explicit false.
  //
  // Send this once per worker connection, before requesting work items. The backend only writes
  // tombstones for a task hub that has opted in, and a worker about to start draining them should not
  // observe a stale setting. Not calling it at all leaves the stored value untouched, which is what
  // keeps a worker that does not know about auto-purge from silently disabling a task hub that opted
  // in.
  rpc SetLargePayloadAutoPurge(SetLargePayloadAutoPurgeRequest) returns (SetLargePayloadAutoPurgeResponse);

  // Returns a bounded, deterministically ordered batch of due large-payload tombstones whose
  // external blobs the worker must delete. Scoped to the caller's authenticated task hub.
  // Only rows that are pending and whose next attempt time has arrived are returned; a row stays
  // pending until its outcome is reported, so this is safe under retries and duplicate callers.
  rpc GetLargePayloadTombstones(GetLargePayloadTombstonesRequest) returns (GetLargePayloadTombstonesResponse);

  // Reports the outcome of each attempted blob deletion. The backend owns retry scheduling and
  // branches solely on `disposition`: it deletes rows reported as DELETED, reschedules RETRY on
  // its own backoff, and moves QUARANTINED rows out of the active fetch while preserving their
  // evidence. The worker never computes a retry delay.
  rpc ReportLargePayloadPurgeResults(ReportLargePayloadPurgeResultsRequest) returns (ReportLargePayloadPurgeResultsResponse);
}