Fix nested exception handler reachability in ILLink

[sbomer]

Fix exception-handler reachability analysis in the unreachable blocks optimizer by rescanning handlers until no newly reachable handler bodies expose additional nested protected regions.

Previously, handlers were scanned only once. A nested handler could therefore be classified as unreachable before its enclosing handler body was traversed, causing required cleanup code to be removed.

Add a regression test covering an async method with nested finally blocks.

Validation:

• Mono.Linker UnreachableBlock tests: 22 passed
• Full Mono.Linker test suite: 1,134 passed, 30 skipped
• Original standalone repro behaves identically with ipconstprop enabled and disabled

Fixes #131088

Note

This content was created with assistance from AI (GitHub Copilot).

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @dotnet/illink
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR fixes ILLink’s unreachable-block analysis for exception handlers by repeatedly discovering newly reachable handler bodies until reaching a fixed point, preventing nested handlers (e.g., nested finally blocks in async state machines) from being misclassified as unreachable and incorrectly removed.

Changes:

• Update UnreachableBlocksOptimizer reachability analysis to iteratively enqueue newly reachable exception handler (and filter) blocks until no additional handlers become reachable.
• Add a regression test case exercising an async method with nested finally blocks under ipconstprop.
• Add a dependency library used by the new test to reproduce the nested-handler shape.

Show a summary per file

File	Description
src/tools/illink/src/linker/Linker.Steps/UnreachableBlocksOptimizer.cs	Reworks exception-handler reachability to rescan handlers and traverse newly reachable handler bodies, enabling correct discovery of nested protected regions.
src/tools/illink/test/Mono.Linker.Tests.Cases/UnreachableBlock/NestedFinallyInAsyncMethod.cs	New test driver enabling ipconstprop and asserting nested-cleanup executes (via a counter).
src/tools/illink/test/Mono.Linker.Tests.Cases/UnreachableBlock/Dependencies/NestedFinallyInAsyncMethod_Lib.cs	New dependency library containing an async method with nested finally blocks to reproduce the scenario.

Copilot's findings

• Files reviewed: 3/3 changed files
• Comments generated: 0

[jtschuster]

Source changes look good to me, just wanted some clarifications on why tests are the way they are.

[Copilot]

Copilot's findings

Comments suppressed due to low confidence (1)

src/tools/illink/test/Mono.Linker.Tests.Cases/UnreachableBlock/NestedFinallyInFinallyHandler.cs:52

• After removing CleanupCount, Cleanup should no longer reference it. Keeping Cleanup as a no-op still allows the test to validate that the nested finally body remains reachable (via the existing [Kept] expectation on Cleanup).

        static void Cleanup() => CleanupCount++;

• Files reviewed: 3/3 changed files
• Comments generated: 2

[hydraxman]

Thanks for fixing this. The change looks good to me. Left 2 small comments. Also the PR text still says the test is async, but its sync now.

[Copilot]

Copilot's findings

• Files reviewed: 3/3 changed files
• Comments generated: 1