Environment
Microsoft.Azure.DurableTask.AzureStorage: 2.9.1
Microsoft.Azure.DurableTask.Core: 3.9.0
Runtime: .NET 8
Host: Kubernetes, self-hosted TaskHubWorker (not Azure Functions)
Partition manager: Table partition manager
PartitionCount: 16
Worker replicas: 25
UseAppLease: default (true)
Non-default settings: TaskHubName, PartitionCount, StorageAccountClientProvider, LoggerFactory only
Summary
A single control queue partition stopped being polled by any worker for hours/days. Messages continued to be enqueued to it, but nothing dequeued them until an unrelated worker eventually acquired the partition, at which point many stranded messages were drained at once (many were ExecutionStarted). Maximum observed message delay was 3 days for some cases.
No worker process crashed, no node was lost, and the previous owner never released the partition.
For example, sharing one request timeline for one partition:
Timeline (one partition, referred to below as control-NN)
T+00:07 Lease thrash: acquire → LeaseLost → DrainTablePartitionAsync → re-acquire → DropLostControlQueue. Occurs
T+00:21 twice.
T+02:26 Last message consumed from this partition. No lease release is logged.
T+02:26 16 messages enqueued. Zero dequeues.
T+17:23
T+17:23 A worker acquires the partition and drains 17 stranded messages.
Evidence the partition was unowned
No new messages were found backing off is emitted by the control queue reader loop, so its rate tracks fetch attempts ~1:1 whenever a partition is owned
Hour │ Fetches │ Backoff log lines │
01 │ 62 │ 61 │
02 │ 2 │ 2 │
04–15 │ 0 │ 0 │
18 │ 18 │ 18 │
Zero backoff lines during the outage rules out "owned but concurrency-starved" no worker was running a reader loop against this partition at all.
With LeaseAcquireInterval = 10s and LeaseInterval = 30s, expected reclaim after an owner stops renewing is ~10–30 seconds.
We were expecting to behave the partition normally, but it remained unowned for ~15.3 hours while the other workers continued polling the remaining partitions on the same task hub normally.
Impact
Orchestrations whose ExecutionStarted message landed on the affected partition remained in Pending indefinitely, then began executing 10–15 hours later. By that point the external resources they operate on had already been cleaned up by a separate lifecycle process, so the orchestrations failed with misleading downstream errors, which made the underlying cause difficult to attribute.
Activities were unaffected, since the work-item queue is unpartitioned and polled by all workers. Only orchestration-level messages (which ride the partitioned control queues) were impacted.
Recurrence
Occurring regularly for with about
Questions
- Is this a known defect in the table partition manager in 2.9.x, and is it addressed in a later release?
- Can DropLostControlQueue/DrainTablePartitionAsync following a LeaseLost leave a partition in a state where no worker re-acquires it, or where the owner record isn't cleaned up?
- Is running more workers (25) than partitions (16) known to aggravate lease contention in this path? We are considering to reduced replicas to match PartitionCount in our next change.
- Is UseAppLease = true appropriate for self-hosted (non-Functions) deployments?
- Any recommended mitigation or detection while a fix is pending?
Mitigation / detection we are considering in next change
• Restructured orchestrations to prefer activities over sub-orchestrations, so more work rides the unpartitioned work-item queue
• Reduced worker replicas to match PartitionCount
• Added a monitor that alerts when instances remain in Pending beyond a threshold, since a never-dispatched orchestration emits no telemetry of its own
I can provide additional detail if required.
Environment
Microsoft.Azure.DurableTask.AzureStorage: 2.9.1
Microsoft.Azure.DurableTask.Core: 3.9.0
Runtime: .NET 8
Host: Kubernetes, self-hosted TaskHubWorker (not Azure Functions)
Partition manager: Table partition manager
PartitionCount: 16
Worker replicas: 25
UseAppLease: default (true)
Non-default settings: TaskHubName, PartitionCount, StorageAccountClientProvider, LoggerFactory only
Summary
A single control queue partition stopped being polled by any worker for hours/days. Messages continued to be enqueued to it, but nothing dequeued them until an unrelated worker eventually acquired the partition, at which point many stranded messages were drained at once (many were ExecutionStarted). Maximum observed message delay was 3 days for some cases.
No worker process crashed, no node was lost, and the previous owner never released the partition.
For example, sharing one request timeline for one partition:
Timeline (one partition, referred to below as control-NN)
T+00:07 Lease thrash: acquire → LeaseLost → DrainTablePartitionAsync → re-acquire → DropLostControlQueue. Occurs
T+00:21 twice.
T+02:26 Last message consumed from this partition. No lease release is logged.
T+02:26 16 messages enqueued. Zero dequeues.
T+17:23
T+17:23 A worker acquires the partition and drains 17 stranded messages.
Evidence the partition was unowned
No new messages were found backing off is emitted by the control queue reader loop, so its rate tracks fetch attempts ~1:1 whenever a partition is owned
Hour │ Fetches │ Backoff log lines │
01 │ 62 │ 61 │
02 │ 2 │ 2 │
04–15 │ 0 │ 0 │
18 │ 18 │ 18 │
Zero backoff lines during the outage rules out "owned but concurrency-starved" no worker was running a reader loop against this partition at all.
With LeaseAcquireInterval = 10s and LeaseInterval = 30s, expected reclaim after an owner stops renewing is ~10–30 seconds.
We were expecting to behave the partition normally, but it remained unowned for ~15.3 hours while the other workers continued polling the remaining partitions on the same task hub normally.
Impact
Orchestrations whose ExecutionStarted message landed on the affected partition remained in Pending indefinitely, then began executing 10–15 hours later. By that point the external resources they operate on had already been cleaned up by a separate lifecycle process, so the orchestrations failed with misleading downstream errors, which made the underlying cause difficult to attribute.
Activities were unaffected, since the work-item queue is unpartitioned and polled by all workers. Only orchestration-level messages (which ride the partitioned control queues) were impacted.
Recurrence
Occurring regularly for with about
Questions
Mitigation / detection we are considering in next change
• Restructured orchestrations to prefer activities over sub-orchestrations, so more work rides the unpartitioned work-item queue
• Reduced worker replicas to match PartitionCount
• Added a monitor that alerts when instances remain in Pending beyond a threshold, since a never-dispatched orchestration emits no telemetry of its own
I can provide additional detail if required.