Deadline-aware cancellation (prototype)#10018
Conversation
CI systems (AzDO, GitHub Actions) hard-cancel a job at a fixed wall-clock time. When that happens the runner kills the test process, so we lose the TRX/HTML/AzDO reports and get no dump for a hanging test. This teaches MTP about that deadline through an environment variable and lets it react a bit early, while it still controls its own shutdown: - TESTINGPLATFORM_DEADLINE is an absolute instant (ISO 8601, parsed to UTC). - At deadline minus stop margin (default 60s) an in-process extension asks the framework to gracefully stop scheduling new tests, so the session ends normally and every reporter finalizes. - At deadline minus dump margin (default 30s) the out-of-process HangDump controller takes a dump of the process tree and kills the host, for the case where the host is wedged and never reaches the graceful stop. The deadline comes from the environment, so there is no hardcoded timeout in MTP. The margins are MTP side policy and are env overridable. Wiring the real deadline from the CI timeout is a small bit of YAML, left for a follow-up. It is opt-in: with no deadline set both timers stay unarmed and there is no behavior change. The graceful stop also degrades to a no-op when the framework does not expose IGracefulStopTestExecutionCapability. Verified: build.cmd -pack passes 0/0, and the new acceptance tests pass (AbortAtDeadlineTests 5/5, HangDumpTests 30/30 including the deadline dump). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a prototype “deadline-aware cancellation” mechanism to Microsoft.Testing.Platform (MTP) so CI can provide an absolute wall-clock deadline and MTP can proactively (a) request a graceful stop before the runner hard-kills the process, and (b) trigger HangDump as a fallback before the deadline.
Changes:
- Introduces
DeadlineHelper+ new env vars (TESTINGPLATFORM_DEADLINE,*_STOP_MARGIN,*_DUMP_MARGIN) for parsing an absolute UTC deadline and margins. - Registers a new in-proc
AbortAtDeadlineExtensionthat schedules a timer to requestIGracefulStopTestExecutionCapability. - Extends HangDump to arm an additional one-shot timer for the absolute deadline and adds acceptance coverage for both graceful stop and deadline-driven dump.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HangDumpTests.cs | Adds acceptance coverage ensuring HangDump can be triggered via absolute deadline env var. |
| test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/AbortAtDeadlineTests.cs | New acceptance suite validating the graceful-stop behavior around deadlines/margins and missing capability. |
| src/Platform/Microsoft.Testing.Platform/InternalAPI/InternalAPI.Unshipped.txt | Tracks newly added internal APIs/constants for PublicAPIAnalyzers. |
| src/Platform/Microsoft.Testing.Platform/Hosts/TestHostBuilder.Framework.cs | Registers AbortAtDeadlineExtension into the message bus when enabled. |
| src/Platform/Microsoft.Testing.Platform/Helpers/EnvironmentVariableConstants.cs | Adds constants for the new deadline-related environment variables. |
| src/Platform/Microsoft.Testing.Platform/Helpers/DeadlineHelper.cs | New helper to read/parse deadline + margins from environment. |
| src/Platform/Microsoft.Testing.Platform/Extensions/AbortAtDeadlineExtension.cs | New extension that arms a timer to request graceful stop before the deadline. |
| src/Platform/Microsoft.Testing.Extensions.TrxReport/InternalAPI/InternalAPI.Unshipped.txt | Tracks newly added env-var constants due to shared-source inclusion. |
| src/Platform/Microsoft.Testing.Extensions.Retry/InternalAPI/InternalAPI.Unshipped.txt | Tracks newly added env-var constants due to shared-source inclusion. |
| src/Platform/Microsoft.Testing.Extensions.MSBuild/InternalAPI.Unshipped.txt | Tracks newly added env-var constants due to shared-source inclusion. |
| src/Platform/Microsoft.Testing.Extensions.HotReload/InternalAPI/InternalAPI.Unshipped.txt | Tracks newly added env-var constants due to shared-source inclusion. |
| src/Platform/Microsoft.Testing.Extensions.HangDump/Microsoft.Testing.Extensions.HangDump.csproj | Links DeadlineHelper.cs into HangDump extension for shared deadline parsing. |
| src/Platform/Microsoft.Testing.Extensions.HangDump/InternalAPI/InternalAPI.Unshipped.txt | Tracks DeadlineHelper + env-var constants for this assembly. |
| src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs | Arms a new deadline-driven dump timer and prevents double dump via _dumpTaken. |
| @@ -191,6 +203,23 @@ | |||
| null, | |||
| _activityTimerValue!.Value, | |||
| TimeSpan.FromMilliseconds(-1)); | |||
There was a problem hiding this comment.
Fixed in 1ddfd5a. Both timer callbacks now go through a TriggerDumpOnce trampoline that owns the Interlocked one-shot gate and assigns _activityIndicatorTask only for the winning caller. The loser returns without touching it, so it can no longer overwrite the real dump task, and Dispose keeps awaiting the right one.
| _deadlineTimer = new Timer( | ||
| _ => _activityIndicatorTask = TakeDumpOfTreeAsync(cancellationToken), | ||
| null, | ||
| dueTime, | ||
| TimeSpan.FromMilliseconds(-1)); |
There was a problem hiding this comment.
Fixed in 1ddfd5a, same TriggerDumpOnce trampoline. The deadline timer no longer assigns _activityIndicatorTask directly; only the winner of the one-shot gate does.
| // The inactivity timer and the deadline timer can both fire; only dump once. | ||
| if (Interlocked.Exchange(ref _dumpTaken, 1) != 0) | ||
| { | ||
| return; | ||
| } |
There was a problem hiding this comment.
Fixed in 1ddfd5a. The deadline path now carries its own reason and its own output message (new HangDumpDeadlineReached resource, xlf regenerated) instead of reusing the inactivity "timeout expired" text.
| // Copyright (c) Microsoft Corporation. All rights reserved. | ||
| // Licensed under the MIT license. See LICENSE file in the project root for full license information. |
| // Copyright (c) Microsoft Corporation. All rights reserved. | ||
| // Licensed under the MIT license. See LICENSE file in the project root for full license information. |
| // Copyright (c) Microsoft Corporation. All rights reserved. | ||
| // Licensed under the MIT license. See LICENSE file in the project root for full license information. |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…-cancellation # Conflicts: # src/Platform/Microsoft.Testing.Extensions.TrxReport/InternalAPI/InternalAPI.Unshipped.txt
Fixes from the review of #10018: - HangDump: the deadline timer and the inactivity timer both wrote _activityIndicatorTask, so the losing one could overwrite the winner's real dump task with a completed no-op, and Dispose would stop waiting for the dump mid-flight. Move the one-shot guard into a TriggerDumpOnce trampoline so only the winning timer assigns _activityIndicatorTask. - HangDump: the deadline path logged "Hang dump timeout expired", which is misleading because no inactivity timeout expired. Give the deadline case its own reason and its own output message (new HangDumpDeadlineReached resource + regenerated xlf). - AbortAtDeadlineExtension: compute a local non-null stopAt instead of dereferencing _stopAt.Value, so there is no nullable deref. - Add the UTF-8 BOM to the three new source files to satisfy the charset=utf-8-bom editorconfig rule. Verified: build 0/0, AbortAtDeadline 5/5, full HangDump suite 30/30. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
| _deadlineTimer = new Timer( | ||
| _ => TriggerDumpOnce(cancellationToken, triggeredByDeadline: true), | ||
| null, | ||
| dueTime, | ||
| TimeSpan.FromMilliseconds(-1)); |
There was a problem hiding this comment.
Good catch. I moved the deadline timer so it is armed right after we have the test host process info, before the pipe handshake. Now if the host wedges during startup and never connects back over the pipe, the deadline dump and kill are still armed, which is the case the deadline is meant for. The in-progress-test list needs the consumer pipe, so I made that part best-effort and skip it when the pipe never connected.
🤖
| if (Interlocked.Exchange(ref _dumpTaken, 1) != 0) | ||
| { | ||
| return; | ||
| } | ||
|
|
||
| _activityIndicatorTask = TakeDumpOfTreeAsync(cancellationToken, triggeredByDeadline); |
There was a problem hiding this comment.
Right. The Interlocked.Exchange claimed the gate but did not publish _activityIndicatorTask atomically, so disposal could read it as null and tear the pipes down while a dump the timer just started was still running. I now take the dump-once gate and publish the task under one lock, and Dispose/DisposeAsync take the same lock, claim the gate so no new dump can start, and capture the in-flight task to wait on outside the lock.
🤖
|
|
||
| if (DeadlineHelper.TryGetDeadline(environment, out DateTimeOffset deadline) && capability is not null) | ||
| { | ||
| DateTimeOffset stopAt = deadline - DeadlineHelper.GetStopMargin(environment); |
There was a problem hiding this comment.
Agreed. deadline - margin can underflow DateTimeOffset for a very old (but valid) deadline or a large margin. I added DeadlineHelper.SubtractSaturating, which clamps at DateTimeOffset.MinValue, so underflow just means the stop instant is already in the past and we act immediately.
🤖
| // has a chance to complete before the CI runner hard-kills the process. | ||
| if (DeadlineHelper.TryGetDeadline(_environment, out DateTimeOffset deadline)) | ||
| { | ||
| _deadlineDumpAt = deadline - DeadlineHelper.GetDumpMargin(_environment); |
There was a problem hiding this comment.
Same underflow class here, so this path now uses DeadlineHelper.SubtractSaturating too.
🤖
…-cancellation # Conflicts: # src/Platform/Microsoft.Testing.Platform/InternalAPI/InternalAPI.Unshipped.txt
More fixes from the review of #10018: - HangDump: the absolute deadline timer was armed only after the pipe handshake in OnTestHostProcessStartedAsync. A test host that wedges during startup never connects back over the pipe, so those waits blocked past the deadline and the deadline dump/kill was never armed, which is exactly the case the deadline is for. Arm the deadline timer right after we have the test host process info (before the handshake). The dump path only needs the PID; the in-progress-test list needs the consumer pipe, so I make it best-effort and skip it when the pipe never connected. - HangDump: the winning timer published _activityIndicatorTask without any ordering against disposal. Disposal could read the field as null, release, and tear the pipes down while a dump the timer just started was still running. Guard the "take the dump once" gate and the task publish under one lock, and have Dispose/DisposeAsync take that lock, claim the gate so no new dump can start, and capture the in-flight task to wait on outside the lock. The lock is a System.Threading.Lock on net9.0 and an object below it, so it still compiles on netstandard2.0. - AbortAtDeadline and HangDump: deadline - margin on DateTimeOffset throws for a very old (but valid) deadline or a large margin. Add a shared DeadlineHelper.SubtractSaturating that clamps at DateTimeOffset.MinValue, so underflow means "already in the past" -> act immediately. Verified: build 0/0, AbortAtDeadline 5/5, full HangDump suite 30/30. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
| } | ||
| } | ||
|
|
||
| public Type[] DataTypesConsumed { get; } = [typeof(TestNodeUpdateMessage)]; |
There was a problem hiding this comment.
Good catch. DataTypesConsumed is now an empty array, so the bus keeps the reference for the run without routing every test result into a no-op ConsumeAsync.
🤖
| return; | ||
| } | ||
|
|
||
| _ = Task.Run(HandleDeadlineAsync); |
There was a problem hiding this comment.
Switched to calling HandleDeadlineAsync directly instead of Task.Run, so single-threaded runtimes (browser/WASI) don't queue work that never runs. The method is async and yields at the first await, so it doesn't block the timer thread.
🤖
| dueTime = TimeSpan.Zero; | ||
| } | ||
|
|
||
| _timer = new Timer(static state => ((AbortAtDeadlineExtension)state!).OnDeadlineReached(), this, dueTime, Timeout.InfiniteTimeSpan); |
There was a problem hiding this comment.
Added a clamp to the Timer maximum (~49.7 days) so the ctor can't throw for a far-future deadline. The run is disposed long before that, so the timer never fires early in practice.
🤖
| _deadlineTimer = new Timer( | ||
| _ => TriggerDumpOnce(cancellationToken, triggeredByDeadline: true), | ||
| null, | ||
| dueTime, | ||
| TimeSpan.FromMilliseconds(-1)); |
| // The consumer pipe is only present once the test host connected back over it. When the host | ||
| // wedged during startup and the deadline dump path fired, there is no pipe to ask for the | ||
| // in-progress tests, so we skip that best-effort list and still take the dump and kill the tree. | ||
| if (_namedPipeClient is not null) |
There was a problem hiding this comment.
Right, a non-null client isn't necessarily connected: it's created when the host sends its pipe name but connected later. I wrapped the in-progress query in try/catch so a deadline dump firing in that window can't block taking the dump and killing the tree.
🤖
| <value>Hang dump file</value> | ||
| </data> | ||
| <data name="HangDumpDeadlineReached" xml:space="preserve"> | ||
| <value>CI deadline reached: taking a hang dump before the run is hard-cancelled.</value> |
There was a problem hiding this comment.
Renamed the resource to HangDumpDeadlineApproaching and reworded the text and log reason. The dump fires at deadline minus the dump margin, so "approaching" is accurate. Regenerated the xlf files.
🤖
| await _logger.LogInformationAsync($"Deadline approaching (stop scheduled at {_stopAt:o}). Requesting graceful stop of test execution.").ConfigureAwait(false); | ||
|
|
||
| CancellationToken cancellationToken = _cancellationTokenSource.CancellationToken; | ||
| await _outputDevice.DisplayAsync( | ||
| this, | ||
| new FormattedTextOutputDeviceData("Deadline approaching: gracefully stopping the test run so reports can be finalized before the CI hard-cancel."), | ||
| cancellationToken).ConfigureAwait(false); | ||
|
|
||
| await capability.StopTestExecutionAsync(cancellationToken).ConfigureAwait(false); |
There was a problem hiding this comment.
Moved the graceful stop into its own try/catch, separate from the best-effort diagnostics, so a logging or output-device failure can't skip the stop. The diagnostics failure log is itself swallowed so it can't re-throw and skip the stop either.
🤖
The automated re-review flagged seven spots after the last push. All are in the two deadline timers and the hang dump resource text. AbortAtDeadlineExtension: - DataTypesConsumed is now empty. The extension only implements IDataConsumer so the message bus keeps a live reference to it (which keeps its timer alive). Returning [TestNodeUpdateMessage] made the bus route every test result to a no-op ConsumeAsync, which is O(test-count) for nothing. - The timer callback calls HandleDeadlineAsync directly instead of Task.Run. On single-threaded runtimes (browser/WASI) Task.Run can queue work that never runs; the method is already async and yields at the first await. - Arming the timer clamps a far-future due time to the Timer maximum (~49.7 days) instead of throwing. The run is disposed long before that, so the timer never fires early in practice. - The graceful stop now runs in its own try/catch, separate from the best-effort diagnostics. A logging or output-device failure can no longer skip the stop, and the diagnostics failure log is itself swallowed so it cannot re-throw and skip the stop either. HangDumpProcessLifetimeHandler: - Same far-future clamp on the deadline dump timer. - The in-progress-test query before dumping is wrapped in try/catch. A non-null pipe client is not necessarily connected (it is created when the host sends its pipe name but connected later), so a deadline dump firing in that window could hit an unconnected pipe. Any failure is logged and swallowed so it cannot block taking the dump and killing the tree. - Renamed the resource HangDumpDeadlineReached to HangDumpDeadlineApproaching and reworded the text and log reason to "approaching". The dump fires at deadline minus the dump margin, so the deadline has not been reached yet. Regenerated the xlf files. Local Debug pack is 0/0. AbortAtDeadline acceptance tests 5/5 and the full HangDump acceptance suite 30/30. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 28 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs:459
- This request is not actually best-effort when the host is wedged.
NamedPipeClient.RequestReplyAsyncwaits for a response until the supplied run token is canceled (NamedPipeClient.cs:103-105), so a connected host whose control-pipe callback no longer runs can block here indefinitely and prevent both dump creation and process-tree termination. Skip this query for deadline-triggered dumps or bound it to a short deadline-specific budget.
GetInProgressTestsResponse tests = await _namedPipeClient.RequestReplyAsync<GetInProgressTestsRequest, GetInProgressTestsResponse>(new GetInProgressTestsRequest(), cancellationToken).ConfigureAwait(false);
| } | ||
| } | ||
|
|
||
| public void Dispose() => _timer?.Dispose(); |
| await _outputDevice.DisplayAsync( | ||
| this, | ||
| new FormattedTextOutputDeviceData("Deadline approaching: gracefully stopping the test run so reports can be finalized before the CI hard-cancel."), | ||
| _cancellationTokenSource.CancellationToken).ConfigureAwait(false); |
| // Prototype defaults. stopMargin > dumpMargin so the graceful stop is attempted first and the | ||
| // hang dump is the fallback for a host that did not stop in time. | ||
| private static readonly TimeSpan DefaultStopMargin = TimeSpan.FromSeconds(60); | ||
| private static readonly TimeSpan DefaultDumpMargin = TimeSpan.FromSeconds(30); |
| catch | ||
| { | ||
| // Ignore: the graceful stop below is the only thing that must happen. | ||
| } |
CI hard-cancels a job at a fixed wall-clock time. When it fires the runner kills the test process, and we lose the TRX/HTML/AzDO reports and get no dump for a hanging test. This is a prototype that tells MTP when that deadline is, so it can react a little early while it still owns its shutdown.
What it does
The deadline arrives as an environment variable, an absolute instant:
TESTINGPLATFORM_DEADLINE: ISO 8601, parsed to UTC.TESTINGPLATFORM_DEADLINE_STOP_MARGIN: lead time for the graceful stop. Default 60s.TESTINGPLATFORM_DEADLINE_DUMP_MARGIN: lead time for the hang dump. Default 30s.Two reactions, armed off the same instant:
deadline - stopMarginan in-process extension asks the framework to gracefully stop scheduling new tests (IGracefulStopTestExecutionCapability). In-flight tests finish, the session ends normally, and every reporter gets to finalize.deadline - dumpMarginthe out-of-process HangDump controller dumps the process tree and kills the host. This is the fallback for a wedged host that never reaches the graceful stop. It reuses the controller that HangDump already runs, so it costs nothing extra when--hangdumpis on.stopMargin > dumpMarginon purpose: try the clean stop first, dump only if that did not happen in time.It is opt-in. No deadline set, both timers stay unarmed, no behavior change. The graceful stop also degrades to a no-op if the framework does not expose the capability.
Where the deadline comes from
Out of scope here, that is the "solve timing later" part. MTP has no hardcoded timeout, it only reacts to whatever instant the environment gives it. The CI side is a few lines of YAML that compute
job start + timeoutand export it. Rough sketches:GitHub Actions (job has
timeout-minutes: 60):Azure DevOps (job has
timeoutInMinutes: 60):Both approximate "now" as the job start, which is close enough. The nicer version is Arcade computing the real deadline once and exporting it for everyone, so nobody has to copy YAML around. Left for a follow-up.
Verified
build.cmd -packpasses 0/0. New acceptance tests:AbortAtDeadlineTests(5): past deadline stops immediately, future deadline stops when the timer fires, the stop margin is subtracted from the deadline, no deadline stays silent, missing capability is a no-op.HangDumpTests.HangDump_AbsoluteDeadline_CreateDump(3 tfms): 30 minute inactivity timeout so only the deadline path can fire, and it produces a dump. FullHangDumpTestsstill 30/30.This is a prototype, so LMK what you think about the shape before I polish it. Open questions: