Skip to content

Dogfood dotnet test for Mono.Android device tests#11224

Open
jonathanpeppers wants to merge 64 commits into
mainfrom
jonathanpeppers/dogfood-dotnet-test
Open

Dogfood dotnet test for Mono.Android device tests#11224
jonathanpeppers wants to merge 64 commits into
mainfrom
jonathanpeppers/dogfood-dotnet-test

Conversation

@jonathanpeppers
Copy link
Copy Markdown
Member

@jonathanpeppers jonathanpeppers commented Apr 27, 2026

Summary

Migrate Mono.Android device test infrastructure from the legacy NUnitLite stack to dotnet test with the Microsoft Testing Platform (MTP), following the pattern established for the androidtest template in 031246d.

Callers now run device tests with stock dotnet test, no custom MSBuild targets required. NUnitLite is removed entirely from the workload — both as a framework assembly and as a vendored copy.

Changes

New: MTP-based TestInstrumentation base class

  • tests/TestRunner.Core/TestInstrumentation.cs is now an MTP-based on-device runner using stock NUnit (NUnitTestAssemblyRunner) plus Microsoft.Testing.Platform.
  • Mirrors the androidtest template: tests run synchronously on the instrumentation thread (NumberOfTestWorkers = 0, see comment in code), TRX is written to the app data dir, and the path is sent back through the instrumentation bundle so the host adapter can pick it up.

Updated test projects

  • Mono.Android.NET-Tests, Java.Interop-Tests, JcwGen-Tests: now reference stock NUnit 3.13.3 from NuGet (no NUnitLite / TestRunner.NUnit), set IsTestingPlatformApplication=true, and use the new TestInstrumentation base.

Deleted (~41K lines)

  • src/Xamarin.Android.NUnitLite/ — the Xamarin.Android.NUnitLite.dll framework assembly (NUnitLite wrapper, TestSuiteInstrumentation, GUI TestSuiteActivity / TestResultActivity / OptionsActivity, layouts).
  • src-ThirdParty/NUnitLite/ — the embedded NUnitLite source we used to compile into the above.
  • tests/TestRunner.NUnit/NUnitTestRunner / NUnitTestInstrumentation.
  • tests/TestRunner.xUnit/XUnitTestRunner / XUnitTestInstrumentation.
  • Xamarin.Android.NUnitLite.dll/.pdb/.xml removed from build-tools/installers/create-installers.targets (no longer shipped in the framework).
  • Various custom RunTestApp / device-test MSBuild targets.

The XA4313 deprecation message in Common.targets (which suggested customers migrate Xamarin.Android.NUnitLiteXamarin.Legacy.NUnitLite) is preserved as the existing escape hatch; the Xamarin.Legacy.NUnitLite NuGet is published from a separate repo and is unaffected by these changes.

CI updates (apk-instrumentation.yaml)

  • Build with -t:Install, run with dotnet test --no-build --report-trx.
  • Test results are published as VSTest (TRX) instead of NUnit XML.
  • Added adb logcat -d capture so device-test crashes can be diagnosed without re-running CI.

Test fixes / bookkeeping

  • A handful of NUnit-3-vs-NUnitLite differences had to be addressed:
    • NUnitCompatibility.cs shim: NUnit 3.13.3's netstandard2.0 assembly does not include [TestFixtureSetUp] / [TestFixtureTearDown], but the Java.Interop submodule still uses them. The shim aliases them to [OneTimeSetUp] / [OneTimeTearDown]. Will be removed once [tests] Replace legacy [TestFixtureSetUp]/[TestFixtureTearDown] with [OneTimeSetUp]/[OneTimeTearDown] java-interop#1437 lands and the submodule is bumped.
    • Centralized JavaSystem.LoadLibrary ("reuse-threads") in TestInstrumentation.OnCreate so it runs on the main Java thread instead of inside individual tests.
    • Synced ExcludedTestNames / ExcludedCategories to match main.

Out of scope

  • EmbeddedDSO and Locale-Tests — old-style csproj that still <Reference Include="Xamarin.Android.NUnitLite" />. They need an SDK migration before this PR can fully delete that assembly's identity from the build (the XA4313 deprecation will fire on these projects today). Tracked separately.

Related

@jonathanpeppers jonathanpeppers force-pushed the jonathanpeppers/dogfood-dotnet-test branch from 31e7d49 to 7b8a944 Compare April 27, 2026 21:45
@jonathanpeppers
Copy link
Copy Markdown
Member Author

jonathanpeppers commented Apr 27, 2026

With MTP, test filtering works via command-line args passed to TestApplication.CreateBuilderAsync(). NUnit's MTP adapter supports --treenode-filter which accepts NUnit filter expressions.

I've now added this support to TestInstrumentation. Subclasses can override:

  • ExcludedCategories — to exclude NUnit categories
  • ExcludedTestNames — to exclude tests by fully-qualified name (for Java.Interop submodule tests where adding attributes isn't practical)

The old am instrument extras still work too:

  • -e exclude "Cat1,Cat2" — exclude categories at runtime
  • -e include "Cat1" — include categories at runtime
  • -e noexclusions true — skip built-in exclusions

Everything gets translated to a --treenode-filter expression.

@jonathanpeppers jonathanpeppers force-pushed the jonathanpeppers/dogfood-dotnet-test branch from 04b483b to 8bc3984 Compare May 4, 2026 16:39
jonathanpeppers and others added 18 commits May 5, 2026 10:00
…form (MTP)

- Revamp `TestRunner.Core/TestInstrumentation.cs` to use MTP + NUnit
  (following the androidtest template pattern with `AddNUnit()`)
- Delete `TestRunner.NUnit/`, `TestRunner.xUnit/`,
  `Xamarin.Android.NUnitLite/`, `src-ThirdParty/NUnitLite/`
- Update `Mono.Android.NET-Tests` and `JcwGen-Tests` to use new
  `TestInstrumentation` base class with stock NUnit 4.3.2
- Update CI YAML to use `dotnet test --no-build` instead of
  `-t:RunTestApp`
- Update `NUnitReferences.projitems` to NUnit 4.3.2 + adapter 5.0.0
- Clean up `Directory.Build.props`, installer targets, and solution
  files

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NUnit 4.x upgrade broke all host-side test projects because NUnit 4
removed classic assert APIs (Assert.IsTrue, FileAssert, etc.).

Only NUnit3TestAdapter 5.0.0 is needed (for AddNUnit() MTP support) in
the device test projects. NUnit itself stays at 3.13.3, and the shared
NUnitReferences.projitems is restored to its original versions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The publish step was skipped because testResultsFiles was empty.
Add --logger trx and --results-directory to produce TRX files,
and use a deterministic file path for PublishTestResults@2.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Restore XA4313 AndroidError for Xamarin.Android.NUnitLite in Common.targets
- Restore NUnitLite in XA4313 test Values in BuildTest.cs
- Restore GenerateAssemblyInfo=false and AssemblyInfo.cs in TestRunner.Core
- Delete unused MainActivities and Main.axml layout
- Move MyFragment to its own file (still needed by FragmentFixup tests)
- Fix TRX publishing: copy device-pulled TRX to expected path instead
  of relying on --logger trx which doesn't work with Microsoft.Android.Run

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Root cause: dotnet test uses VSTest mode by default, which runs
vstest.console to discover tests from DLLs. Android device tests need
MTP mode so dotnet test uses RunCommand/RunArguments to invoke
Microsoft.Android.Run, which runs am instrument on the device.

Changes:
- Add global.json with MTP runner to each device test project directory
  (matches the androidtest template pattern)
- Add IsTestingPlatformApplication=true to device test csprojs so the
  MTP mode of dotnet test recognizes them
- Add GenerateTestingPlatformEntryPoint=false since Android projects
  have their own entry point
- Update apk-instrumentation.yaml to cd into the project directory
  before running dotnet test, so it finds the project-local global.json
- Fix NUnitInstrumentation.cs -> TestInstrumentation.cs in Compile items
- Remove stale TrimmerRootAssembly refs to deleted projects
- Add test exclusion support via ExcludedCategories/ExcludedTestNames

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ComputeRunArguments needs intermediate build outputs (AndroidManifest.xml)
to resolve the package name, and the app must be installed on the device
before Microsoft.Android.Run can invoke am instrument.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This project previously got NUnit transitively via Xamarin.Android.NUnitLite.
Now that NUnitLite is deleted, it needs a direct NUnit reference.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Define NUnitVersion in Configuration.props and reference it everywhere
instead of copying 3.13.3 into each csproj.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TestRunner.Core.NET.csproj did not import Configuration.props, so
$(NUnitVersion) was empty causing NU1015.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NUnit3TestAdapter targets netcoreapp3.1/net462 which are not compatible
with Android TFMs, so AddNUnit() and MTP types are unavailable on device.

Instead, use NUnitTestAssemblyRunner to run tests directly and generate
TRX output that Microsoft.Android.Run pulls via adb and reports through
MTP on the host side.

Also add NUnit compatibility shim for TestFixtureSetUpAttribute and
TestFixtureTearDownAttribute which were removed from stock NUnit but
are used by Java.Interop submodule test code.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previous edit accidentally removed the <ItemGroup> opening tag around
ProjectReferences when removing NUnit3TestAdapter, leaving orphaned
closing tags that caused Step_GenerateCGManifest XML parse failure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NUnit 3.13.3 filter classes (CategoryFilter, NotFilter, etc.) are
internal. Use TestFilter.FromXml() with NUnit XML filter syntax instead.

Remove [Obsolete] from TestFixtureSetUp/TearDown compatibility shims
since warnings-as-errors treats them as build errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Mono.Android.NET-Tests project does not have ImplicitUsings enabled,
so explicit using directives are needed for System, System.Collections.Generic,
and Android.OS.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NUnit 3.13.3 DefaultTestAssemblyBuilder.Build(Assembly, ...) calls
AssemblyHelper.GetAssemblyPath() which accesses Assembly.CodeBase.
On .NET Android, assemblies are in a single-file bundle so CodeBase
throws NotSupportedException.

Fix by using the string-based Load(string, settings) overload with
Assembly.FullName. NUnit resolves this via Assembly.Load(AssemblyName)
which finds the already-loaded assembly without needing CodeBase.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathanpeppers jonathanpeppers force-pushed the jonathanpeppers/dogfood-dotnet-test branch from 8bc3984 to cec4ac5 Compare May 5, 2026 15:01
jonathanpeppers and others added 4 commits May 5, 2026 14:00
The previous fix (using assembly.FullName with the string-based Load
overload) silently failed — NUnit sets AssemblyName.Name to the full
display name string, which does not resolve to the loaded assembly.

Replace DefaultTestAssemblyBuilder with a custom AndroidTestAssemblyBuilder
that redirects Build(Assembly, ...) through the string overload using
assembly.GetName().Name (simple name). This avoids Assembly.CodeBase
entirely while correctly resolving the already-loaded assembly via
Assembly.Load(AssemblyName).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add --report-trx and --results-directory to dotnet test so MTP
generates a TRX file on the host side. This replaces the fragile
find+copy step that searched $TMPDIR/AndroidTestResults for a
device-pulled TRX file (which was not being found).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…publishing

NUnit 3.13.3 (from NuGet) enforces that TestCaseSource must refer to
static fields. The old bundled NUnitLite allowed instance fields.
Make EncodingTestData, NinePatchDrawables, and AssetPngs static.

Restore ExcludedTestNames for TrimmableTypeMap mode that were in the
old NUnitInstrumentation but missing from the new TestInstrumentation
subclass.

Add --report-trx to dotnet test so MTP generates a TRX file on the
host side for PublishTestResults@2. Remove the broken find+copy step
that was searching $TMPDIR/AndroidTestResults and not finding anything.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The old TestApks.targets passed ExcludeCategories via am instrument -e
extras. AndroidTestAdapter does not support extras, so bake the category
exclusions into the TestInstrumentation subclass using RuntimeFeature
checks instead. This excludes CoreCLRIgnore/NTLM on CoreCLR and
NativeTypeMap/Export on trimmable typemap, matching the old behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathanpeppers and others added 2 commits May 6, 2026 12:36
…od-dotnet-test

# Conflicts:
#	THIRD-PARTY-NOTICES.TXT
#	tests/Mono.Android-Tests/Mono.Android-Tests/Xamarin.Android.RuntimeTests/NUnitInstrumentation.cs
The --report-trx flag is forwarded by MTP to the test host process
(Microsoft.Android.Run). Without registering the TrxReport extension,
the test host rejects the flag as unknown. Add the
Microsoft.Testing.Extensions.TrxReport package and call
AddTrxReportProvider() so MTP can generate TRX files on the host side.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

simonrozsival pushed a commit that referenced this pull request May 29, 2026
…#11523)

* Fix UrlEscaping_Bug43411 to actually run and assert correctly

Two issues with this test on main:

1. The method was declared `void UrlEscaping_Bug43411 ()` (non-public). NUnitLite silently skipped non-public `[Test]` methods, so the test has been effectively dead for years. Discovered while migrating to stock NUnit 3 via `dotnet test` in PR #11224, where NUnit 3 surfaces the method and it fails.

2. The assertion compared the request URL against `Uri.ToString ()`, which returns the human-readable form and unescapes safe characters like `%20` to a literal space. The canonical, encoded form is `Uri.AbsoluteUri` — switch to that so percent-encoding is preserved.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review: make test async and verify URL on the listener side

- Make `UrlEscaping_Bug43411` (and the `UrlEscaping_TestUrl` helper) `async Task` and `await` `SendAsync` instead of blocking with `.Wait ()`.

- The previous listener callback unconditionally set `failed = true`, so `Assert.IsNull (failed)` always failed (`Expected: null, But was: True`). Capture `HttpListenerContext.Request.RawUrl` on the server side instead and compare it against the expected `Uri.PathAndQuery` — this is the verification the test was clearly meant to do (confirm that percent-encoded characters survive the round trip to the server).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathanpeppers and others added 2 commits June 2, 2026 14:36
…od-dotnet-test

# Conflicts:
#	tests/Mono.Android-Tests/Mono.Android-Tests/Xamarin.Android.Net/AndroidMessageHandlerIntegrationTests.cs
Reverted the using directive reorder, the Console -> Android.Util.Log
conversion in ThreadReuse, and the try/catch in DoNotLeakWeakReferences.

Only keep what is strictly required for the migration: removing the two
`Java.Lang.JavaSystem.LoadLibrary ("reuse-threads")` calls from
RegisterTypeOnNewNativeThread and ThreadReuse, since the library is now
loaded once in TestInstrumentation.OnCreate on the main thread.

If Console.WriteLine on the native callback thread or the unhandled
exception in DoNotLeakWeakReferences resurface as "0 tests ran" in CI,
they will be addressed separately with logcat evidence in hand.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@jonathanpeppers jonathanpeppers changed the title [WIP] Dogfood dotnet test for device tests Dogfood dotnet test for Mono.Android device tests Jun 2, 2026
@jonathanpeppers
Copy link
Copy Markdown
Member Author

/review

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 2, 2026

Android PR Reviewer completed successfully!

Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Code Review Summary

Verdict: ⚠️ Needs Changes (1 issue)

Overview

Solid migration from NUnitLite to dotnet test + MTP. The new TestInstrumentation base class is well-structured with excellent comments (especially the NumberOfTestWorkers=0 rationale). The ~41K lines of deletion is a great cleanup. The TRX writer, filter builder, and AndroidTestAssemblyBuilder workaround are all clean.

Issues Found

Severity Category File Description
❌ error Nullable TestInstrumentation.cs:338 Null-forgiving ! operator used — banned per repo rules
⚠️ warning Patterns Mono.Android.NET-Tests.csproj NativeAOT/LLVM $(ExcludeCategories) may no longer reach instrumentation via dotnet test
💡 suggestion Formatting MyFragment.cs:17 #endif comment doesn't match #if __ANDROID_11__

Positive Callouts

  • 👍 The NumberOfTestWorkers=0 comment is exemplary — explains why, not what, and documents the exact failure mode avoided.
  • 👍 AndroidTestAssemblyBuilder cleanly works around NUnit's Assembly.CodeBase issue on Android single-file bundles.
  • 👍 CI yaml captures adb logcat -d — this will save debugging time for native crashes.
  • 👍 NUnitCompatibility.cs shim is pragmatic and well-documented with a clear removal plan.

Generated by Android PR Reviewer for issue #11224 · ● 42.4M

Comments that could not be inline-anchored

tests/TestRunner.Core/TestInstrumentation.cs:338

🤖 ❌ Nullable! (null-forgiving operator) is banned per repo conventions. Assembly.GetName().Name returns string?.

Replace with a null check:

=&gt; inner.Build (assembly.GetName ().Name ?? throw new InvalidOperationException (&quot;Assembly has no name&quot;), options);

Rule: Never use ! (null-forgiving operator)

tests/Mono.Android-Tests/Mono.Android-Tests/Xamarin.Android.RuntimeTests/MyFragment.cs:17

🤖 💡 Formatting — The #endif comment says ANDROID_11 but the #if guard is __ANDROID_11__. The comment should match the preprocessor symbol exactly for readability.

#endif  // __ANDROID_11__

Rule: #else/#endif comments must match the original expression (Postmortem #37)

tests/Mono.Android-Tests/Mono.Android-Tests/Mono.Android.NET-Tests.csproj:34

🤖 ⚠️ Patterns — The $(ExcludeCategories) MSBuild property defined here (for NativeAOTIgnore, SSL, LLVMIgnore, InetAccess, NetworkInterfaces) was previously wired through TestApks.targetsInstrumentationArgumentsam instrument -e exclude. With the migration to dotnet test via MTP, the RunTestApp target is no longer invoked, so these MSBuild-only exclusions may not reach the on-device instrumentation.

The C# TestInstrumentation.ExcludedCategories handles `CoreCLR…

- TestInstrumentation.cs: replace ! null-forgiving with explicit
  null check + InvalidOperationException, per repo nullable rules.
- MyFragment.cs: fix #endif comment to match #if __ANDROID_11__.
- Mono.Android.NET-Tests.csproj + RuntimeTests/TestInstrumentation.cs:
  the MSBuild \ property was previously flowed to
  the on-device runner via 'am instrument -e' from the custom
  RunTestApp target. With 'dotnet test' + MTP that target no longer
  runs, so PublishAot/EnableLLVM exclusions never reached the device.
  Flow them through [AssemblyMetadata] instead, read at runtime in
  ExcludedCategories.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathanpeppers added a commit to dotnet/java-interop that referenced this pull request Jun 2, 2026
…eSetUp]/[OneTimeTearDown] (#1437)

The legacy `[TestFixtureSetUp]` and `[TestFixtureTearDown]` attributes
were removed in NUnit 3 in favor of `[OneTimeSetUp]` and
`[OneTimeTearDown]`.  They have been kept around in NUnitLite for
back-compat, which is why the on-device Android test path used them
under `#if __ANDROID__`.  NUnitLite also supports the modern
`OneTime*` names, so the conditional is no longer needed.

Replacing the conditional with the unconditional modern attribute
unblocks dotnet/android#11224, which migrates Mono.Android device
tests from NUnitLite to stock NUnit 3.13.3 + MTP via `dotnet test`.
Stock NUnit 3.13.3's `netstandard2.0` assembly does not contain the
legacy attribute names, so consuming this code there currently
requires a small compatibility shim (NUnitCompatibility.cs).  After
this change that shim can be deleted.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dotnet/java-interop#1437 replaces the legacy [TestFixtureSetUp] /
[TestFixtureTearDown] attributes with the modern [OneTimeSetUp] /
[OneTimeTearDown] names that stock NUnit 3 ships out of the box, so the
local NUnitCompatibility.cs shim is no longer needed.

Submodule bump:
  e38211334 -> b982a1a3 (dotnet/java-interop):
    b982a1a [tests] Replace [TestFixtureSetUp]/[TestFixtureTearDown] with [OneTimeSetUp]/[OneTimeTearDown] (#1437)
    e68fa0a Bump to dotnet/android-tools@3c95718 (#1436)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathanpeppers
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

Under stock NUnit 3 (after the dotnet test migration), an unhandled
NUnit AssertionException thrown from inside a raw 'new Thread (...)'
body becomes an unhandled exception on the .NET runtime side, which
SIGABRTs the whole process. The instrumentation host then never gets
a Finish() call and 'dotnet test' reports 'Zero tests ran' for the
entire run -- masking the real failure (Assert #2 inside the worker)
and taking out every other test in the assembly with it.

Capture exceptions from the worker thread in a local, then rethrow on
the test thread after Join() so an Assert failure (or any other
exception) becomes a normal NUnit test failure instead of a process
abort.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathanpeppers
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

jonathanpeppers and others added 2 commits June 3, 2026 15:14
Build 14271469's NoAot lane reported 'Zero tests ran' with no
diagnostics in logcat or the dotnet test output: process simply
exited with code 1 mid-run. With no log of what threw, every flake
turns into another CI iteration just to gather information.

Hook AppDomain.UnhandledException and TaskScheduler.UnobservedTask-
Exception so the next time something escapes from an NUnit runner /
worker thread we at least learn which exception killed the process.
The process will still terminate, but logcat will name the culprit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build 14271469 / APKs 1 / Mono.Android.NET_Tests-Release reproduced
the exact NRE first diagnosed in 09acb6d:

  E DOTNET: Unhandled exception. System.NullReferenceException
    at NUnit.Framework.Internal.MethodWrapper.get_Name()
    at NUnit.Framework.Internal.TestExecutionContext.AdhocContext..ctor()
    at NUnit.Framework.Internal.Execution.TextCapture.WriteLine(String)
    at System.Console.WriteLine(...)
    at Java.InteropTests.JnienvTest.<>c.<ThreadReuse>b__7_0(IntPtr, IntPtr)
  F libc: Fatal signal 6 (SIGABRT) in tid 5229 (Thread-14)

The 'minimize the diff' commit 4d924c4 reverted both the Console-to-Log
conversion and the try/catch -- but only the try/catch needed to go.
NUnit 3's Console.Out redirection is unsafe on a raw native pthread
that has no TestExecutionContext, so route the diagnostics through
Android.Util.Log to bypass the redirect entirely.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathanpeppers
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

jonathanpeppers and others added 2 commits June 4, 2026 09:15
Build 14275720 had this test fail in 6 lanes simultaneously with two
failure modes:

  #2  Expected: 36/37  But was: 37/38   (+1 extra peer at allocation)
  #3  Expected: 36/37  But was: 34/35   (1-3 fewer peers after GC)

Build 14212378 on main (pre-PR) hit the same #2 failure, so this is a
pre-existing flake -- not a regression. Before ac81297, mode #2
escaped from new Thread() and SIGABRT'd the process (manifesting as
'Zero tests ran' for the whole lane); the marshal-exception fix now
surfaces it as a real test failure, which is why we suddenly see it.

Runtime.GetSurfacedObjects() is process-global: NUnit/MTP per-test
infrastructure (TestExecutionContext, listeners, Console capture)
allocates and releases Java.Lang.Object peers around every test, so
the count drifts by a few entries between snapshots. Observed drift
across recent builds is in the range -3..+1; allow +/-5 tolerance
which is plenty tight to still detect a real leak (which would dwarf
this) while removing the noise.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…od-dotnet-test

# Conflicts:
#	Directory.Build.props
@jonathanpeppers
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

Build 14283917 / Mono.Android.NET_Tests-IsAssignableFrom ran 262
tests and reported 3 JavaCast failures. That lane is configured in
build-tools/automation/yaml-templates/stage-package-tests.yaml as:

  -p:IncludeCategories=Intune -p:_AndroidIsAssignableFromCheck=false

so it is meant to run ONLY tests tagged [Category("Intune")] (there
is exactly one such test) -- the 3 JavaCast tests legitimately
require the IsAssignableFromCheck feature switch to be on and would
always fail when the lane disables it.

Before this PR, `IncludeCategories` was carried via build-tools/
scripts/TestApks.targets into `am instrument -e include`, which the
old NUnitLite runner read at runtime. The migration to `dotnet test`
+ MTP removed that custom RunTestApp target, so the MSBuild property
was being silently dropped. Logcat confirmed: no `Including category:`
log lines fired for IsAssignableFrom (vs. multiple `Excluding category:`
lines for lanes whose subclass populates ExcludedCategories).

Flow IncludeCategories through the same [AssemblyMetadata] pipe we
already use for PublishAot / EnableLLVM:

  * Mono.Android.NET-Tests.csproj emits
    [AssemblyMetadata("IncludeCategories", "$(IncludeCategories)")]
    when the property is non-empty.
  * TestRunner.Core.TestInstrumentation gains a virtual
    IncludedCategories property; BuildNUnitFilter merges it with any
    legacy `am instrument -e include` extras into the existing
    <or><cat>...</cat>...</or> include element.
  * Xamarin.Android.RuntimeTests.TestInstrumentation overrides
    IncludedCategories to read the attribute (refactored
    HasAssemblyMetadata into a small GetAssemblyMetadata helper to
    avoid two near-duplicate reflection loops).

Note: NativeAOT lane shows a related but distinct symptom -- the
Export filter is correctly emitted but Export_Method_* tests still
run, because NativeAOT trims [Category] attributes off the test
methods so NUnit can't match them. That is a separate, deeper issue
and not addressed here.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants