PHPUnit runner treats concrete helper TestCase classes as warning tests

[chubes4]

Problem

The generated plugin and core PHPUnit runners add every newly declared, non-abstract PHPUnit\Framework\TestCase subclass to the suite. Concrete helper/base classes that intentionally contain no runnable tests are therefore converted by PHPUnit 9 into warning tests:

Warning
No tests found in class "A8C\CDN\Tests\TestCase".

This breaks valid suites under TeamCity's --fail-on-warning contract. It also diverges from PHPUnit configuration discovery, where helper classes may coexist with executable test classes in discovered files.

The affected code is in packages/runtime-playground/src/phpunit-command-handlers.ts, in both the plugin and WordPress core class-loading loops.

Exact WPCOM comparator evidence at 7ae9ae21ca30f7fc9724aecfb54a16f674f82a23 reproduced warnings for:

• BkismetBaseTestCase
• A8C\CDN\Tests\TestCase
• A8C\GlotPressPlugins\Deliverables\DeliverablesTestCase
• GlotPressPluginsTestCase
• A8C\Test\Integration\WPCOM\Legacy\Upgrades\BaseAssignPayPalPaymentMethod

WPCOM's hierarchy adds a second edge case: base classes expose an inherited static suite() factory. Checking only hasMethod('suite') still admits every concrete helper. A suite factory should qualify the candidate only when the candidate class declares it; normal inherited executable test methods must still qualify.

Reproduction

Place a concrete helper beside a real test:

class ConcreteTestHelper extends WP_UnitTestCase {
    public function fixture(): string {
        return 'fixture';
    }
}

class ExampleTest extends WP_UnitTestCase {
    public function test_example(): void {
        $this->assertTrue(true);
    }
}

Run the generated wordpress.phpunit recipe with warning failures enabled. The real test passes, but the helper creates No tests found in class "ConcreteTestHelper" and fails the suite.

Expected

The generated runner should schedule only classes that PHPUnit recognizes as containing executable tests, plus candidate-owned static suite factories. Concrete helper classes should remain loadable without becoming warning tests.

Acceptance criteria

• Concrete TestCase subclasses with no runnable tests are not added as suites.
• Abstract test bases remain skipped.
• test* methods, @test methods, and inherited runnable test methods remain discoverable.
• A static suite() declared by the candidate class remains supported.
• An inherited generic suite() does not qualify an otherwise empty helper class.
• Plugin and WordPress core generated runners share the same behavior.
• Unit coverage exercises both public and private PHPUnit TestSuite constructors supported by the runner.
• A real PHPUnit 9.6 Playground integration proves a helper and executable class can coexist without warnings and without mutating readonly source.

Evidence

A local candidate on PR #1938's branch produced these exact-SHA results:

• isolated:cdn: failed before, passed after.
• isolated:glotpress-plugins: failed before, passed after.
• Nine-suite warning cluster: improved from 3/9 to 5/9, with all No tests found in class diagnostics removed and zero timeouts.

Related

• Execute bounded PHP commands through clean Playground processes #1937 owns clean-process bounded PHPUnit execution.
• PR Execute bounded PHP commands in clean Playground processes #1938 implements that path and is where this defect surfaced.

AI assistance

• Model: OpenAI GPT-5.6 Sol
• Tool: OpenCode
• Used for: Root-cause analysis, implementation and regression coverage, real PHPUnit integration verification, exact WPCOM comparator runs, and issue drafting under Chris Huber's direction.