Parallel test execution

[sebastianbergmann]

PHPUnit can now execute a test suite across several worker processes concurrently instead of one test after another. Parallel execution is opt-in via a new --parallel=<n> command-line option and changes nothing when it is not used: the sequential TextUI\TestRunner remains the default and parallel mode is selected only when <n> > 1.

Native parallelism here is a generalization of process isolation rather than a new subsystem: a worker reconstructs and runs a unit of work and ships its outcome home in the very same serialized envelope that process isolation already uses, and the parent replays that envelope through its normal event pipeline. As a result the parent process remains the single source of truth for all output, logging, results, and code coverage, which are produced exactly as in sequential mode.

How it works

• Distribution unit = one test class. All selected tests of a class are run together by a single worker, preserving #[BeforeClass]/#[AfterClass] and intra-class ordering.
• Worker pool + scheduler. Runner\Parallel\WorkerPool owns n persistent workers and multiplexes their control channels with stream_select(), pulling from a dynamic work-stealing queue so the load self-balances against stragglers.
• Ordered result aggregation. Workers finish out of order, but Runner\Parallel\ResultAggregator buffers each unit and releases it only once every preceding unit (in suite order) has been released, so the event stream, and therefore every output format, including the default progress output, is byte-for-byte what sequential mode produces.
• PHPT tests are parallelized too. A PHPT test is reconstructed in a worker from its file path alone and run there like it would be in the main process.

Running in the main process

Some tests cannot, or must not, run in a worker. These are run in the main process at their correct suite position (the aggregator runs them in place while releasing results, so global ordering is preserved):

• Tests attributed with #[DoNotRunInParallel] — a new attribute, valid on classes and methods, for tests that must not run alongside others (for instance because they share a process-global resource).
• Tests that require process isolation (#[RunInSeparateProcess], #[RunTestsInSeparateProcesses], or global --process-isolation) — a shared worker cannot provide isolation, but the main process spawns the isolated child as usual.
• Tests whose data cannot be serialized for transport to a worker — closures and other non-serializable values (which make serialize() throw) as well as resources (which serialize() silently degrades to 0).
• PHPT tests carrying a --CONFLICTS-- section — the PHPT equivalent of the attribute, since a PHPT file cannot carry PHP attributes.

New public API

• --parallel=<n> CLI option (also listed in --help).
• #[PHPUnit\Framework\Attributes\DoNotRunInParallel] (TARGET_CLASS | TARGET_METHOD), with full metadata-layer support.
• The --CONFLICTS-- section as supported by the PHP project's PHPT runner.
• Two worker-identity environment variables, exposed to the tests a worker runs so that fixtures can partition shared resources (a database, a port, a temporary directory, ...) per worker:
  • PHPUNIT_WORKER_ID — the small, stable ordinal (0, 1, 2, ...), ideal for indexing a fixed set of pre-provisioned resources.
  • PHPUNIT_WORKER_TOKEN — a value of the form <id>_<random> that is unique across workers and across runs, for resources that must not collide with those left behind by a previous run.

PHPUnit's own test suite

Running PHPUnit's own end-to-end suite in parallel surfaced a number of PHPT tests that rely on a fixed shared path in sys_get_temp_dir() (a shared phpunit.xml, the result cache, a test-results file) and therefore race when run concurrently. These 36 tests — in migration/, cli/validate-configuration/, and the result-cache/duration/defects-ordering tests in execution-order/ — have been marked with --DO_NOT_RUN_IN_PARALLEL-- so they run in the main process.

Results

• --testsuite unit --parallel 10 produces the same results as sequential mode (same test and assertion counts), stable across runs. Note that which worker runs which test class is not deterministic — the work-stealing scheduler assigns classes by timing — but the aggregated output is identical regardless.
• --testsuite end-to-end --parallel 10: 1105 tests, 0 failures, stable across runs; wall-clock dropped from ~3m18s (everything in the main process) to ~34s on a multi-core machine.

Notes and limitations (possible follow-ups)

• An in-process unit may temporally overlap workers still running later units; this is invisible in the output (worker events stay buffered) and harmless for process-global state (separate processes), but a test contending on an external shared resource should use #[DoNotRunInParallel].
• The result cache can seed a longest-processing-time-first ordering; the scheduler does not yet use it.
• Cross-class #[Depends] co-location, --stop-on-* early halt of in-flight workers, and retrying a unit on a fresh worker after a crash are not yet implemented.
• Parallel execution is configurable only via the CLI option; there is no XML configuration setting yet.

[github-actions]

API Surface Changes

If any of the additions below are not intended as public API, mark them with @internal in the docblock.

New API Surface

Classes

• PHPUnit\Framework\Attributes\DoNotRunInParallel — class PHPUnit\Framework\Attributes\DoNotRunInParallel
• PHPUnit\Metadata\DoNotRunInParallel — class PHPUnit\Metadata\DoNotRunInParallel extends PHPUnit\Metadata\Metadata

Methods

• PHPUnit\Metadata\Metadata::doNotRunInParallelOnClass — static public function doNotRunInParallelOnClass(): PHPUnit\Metadata\DoNotRunInParallel
• PHPUnit\Metadata\Metadata::doNotRunInParallelOnMethod — static public function doNotRunInParallelOnMethod(): PHPUnit\Metadata\DoNotRunInParallel
• PHPUnit\Metadata\Metadata::isDoNotRunInParallel — public function isDoNotRunInParallel(): bool
• PHPUnit\Metadata\MetadataCollection::isDoNotRunInParallel — public function isDoNotRunInParallel(): self
• PHPUnit\TextUI\Configuration\Configuration::numberOfParallelWorkers — public function numberOfParallelWorkers(): int

Modified API Surface

Methods

• PHPUnit\TextUI\Configuration\Configuration::__construct

  - public function __construct(array $cliArguments, ?string $testFilesFile, ?string $configurationFile, ?string $bootstrap, array $bootstrapForTestSuite, bool $cacheResult, ?string $cacheDirectory, ?string $coverageCacheDirectory, Source $source, string $testResultCacheFile, ?string $coverageClover, ?string $coverageCobertura, ?string $coverageCrap4j, int $coverageCrap4jThreshold, ?string $coverageHtml, int $coverageHtmlLowUpperBound, int $coverageHtmlHighLowerBound, string $coverageHtmlColorSuccessLow, string $coverageHtmlColorSuccessLowDark, string $coverageHtmlColorSuccessMedium, string $coverageHtmlColorSuccessMediumDark, string $coverageHtmlColorSuccessHigh, string $coverageHtmlColorSuccessHighDark, string $coverageHtmlColorSuccessBar, string $coverageHtmlColorSuccessBarDark, string $coverageHtmlColorWarning, string $coverageHtmlColorWarningDark, string $coverageHtmlColorWarningBar, string $coverageHtmlColorWarningBarDark, string $coverageHtmlColorDanger, string $coverageHtmlColorDangerDark, string $coverageHtmlColorDangerBar, string $coverageHtmlColorDangerBarDark, string $coverageHtmlColorBreadcrumbs, string $coverageHtmlColorBreadcrumbsDark, ?string $coverageHtmlCustomCssFile, ?string $coverageOpenClover, ?string $coveragePhp, ?string $coverageText, bool $coverageTextShowUncoveredFiles, bool $coverageTextShowOnlySummary, ?string $coverageXml, bool $coverageXmlIncludeSource, bool $pathCoverage, bool $branchCoverage, ?string $coverageDriver, bool $ignoreDeprecatedCodeUnitsFromCodeCoverage, bool $disableCodeCoverageIgnore, bool $disableCoverageTargeting, bool $failOnAllIssues, bool $failOnDeprecation, bool $failOnPhpunitDeprecation, bool $failOnPhpunitNotice, bool $failOnPhpunitWarning, bool $failOnEmptyTestSuite, bool $failOnIncomplete, bool $failOnNotice, bool $failOnRisky, bool $failOnSkipped, bool $failOnWarning, bool $doNotFailOnDeprecation, bool $doNotFailOnPhpunitDeprecation, bool $doNotFailOnPhpunitNotice, bool $doNotFailOnPhpunitWarning, bool $doNotFailOnEmptyTestSuite, bool $doNotFailOnIncomplete, bool $doNotFailOnNotice, bool $doNotFailOnRisky, bool $doNotFailOnSkipped, bool $doNotFailOnWarning, int $stopOnDefect, int $stopOnDeprecation, ?string $specificDeprecationToStopOn, int $stopOnError, int $stopOnFailure, int $stopOnIncomplete, int $stopOnNotice, int $stopOnRisky, int $stopOnSkipped, int $stopOnWarning, bool $outputToStandardErrorStream, int $columns, bool $noExtensions, ?string $pharExtensionDirectory, array $extensionBootstrappers, bool $backupGlobals, bool $backupStaticProperties, bool $beStrictAboutChangesToGlobalState, bool $colors, bool $processIsolation, bool $enforceTimeLimit, int $defaultTimeLimit, int $diffContext, int $timeoutForSmallTests, int $timeoutForMediumTests, int $timeoutForLargeTests, bool $reportUselessTests, bool $strictCoverage, bool $requireCoverageContribution, bool $disallowTestOutput, bool $displayDetailsOnAllIssues, bool $displayDetailsOnIncompleteTests, bool $displayDetailsOnSkippedTests, bool $displayDetailsOnTestsThatTriggerDeprecations, bool $displayDetailsOnPhpunitDeprecations, bool $displayDetailsOnPhpunitNotices, bool $displayDetailsOnTestsThatTriggerErrors, bool $displayDetailsOnTestsThatTriggerNotices, bool $displayDetailsOnTestsThatTriggerWarnings, bool $reverseDefectList, bool $requireCoverageMetadata, bool $requireSealedMockObjects, bool $noProgress, bool $noResults, bool $noOutput, int $executionOrder, int $executionOrderDefects, bool $resolveDependencies, ?string $logfileTeamcity, ?string $logfileJunit, ?string $logfileOtr, bool $includeGitInformation, bool $includeGitInformationInOtrLogfile, ?string $logfileTestdoxHtml, ?string $logfileTestdoxText, ?string $logEventsText, ?string $logEventsVerboseText, bool $compactOutput, bool $teamCityOutput, bool $testDoxOutput, bool $testDoxOutputSummary, ?array $testsCovering, ?array $testsUsing, ?array $testsRequiringPhpExtension, ?string $filter, ?string $excludeFilter, ?string $testIdFilterFile, ?string $testIdFilter, array $groups, array $excludeGroups, int $randomOrderSeed, int $repeat, int $retry, bool $includeUncoveredFiles, TestSuiteCollection $testSuite, string $includeTestSuite, string $excludeTestSuite, ?string $defaultTestSuite, bool $ignoreTestSelectionInXmlConfiguration, array $testSuffixes, Php $php, bool $controlGarbageCollector, int $numberOfTestsBeforeGarbageCollection, ?string $generateBaseline, bool $debug, bool $withTelemetry, int $shortenArraysForExportThreshold)
  + public function __construct(array $cliArguments, ?string $testFilesFile, ?string $configurationFile, ?string $bootstrap, array $bootstrapForTestSuite, bool $cacheResult, ?string $cacheDirectory, ?string $coverageCacheDirectory, Source $source, string $testResultCacheFile, ?string $coverageClover, ?string $coverageCobertura, ?string $coverageCrap4j, int $coverageCrap4jThreshold, ?string $coverageHtml, int $coverageHtmlLowUpperBound, int $coverageHtmlHighLowerBound, string $coverageHtmlColorSuccessLow, string $coverageHtmlColorSuccessLowDark, string $coverageHtmlColorSuccessMedium, string $coverageHtmlColorSuccessMediumDark, string $coverageHtmlColorSuccessHigh, string $coverageHtmlColorSuccessHighDark, string $coverageHtmlColorSuccessBar, string $coverageHtmlColorSuccessBarDark, string $coverageHtmlColorWarning, string $coverageHtmlColorWarningDark, string $coverageHtmlColorWarningBar, string $coverageHtmlColorWarningBarDark, string $coverageHtmlColorDanger, string $coverageHtmlColorDangerDark, string $coverageHtmlColorDangerBar, string $coverageHtmlColorDangerBarDark, string $coverageHtmlColorBreadcrumbs, string $coverageHtmlColorBreadcrumbsDark, ?string $coverageHtmlCustomCssFile, ?string $coverageOpenClover, ?string $coveragePhp, ?string $coverageText, bool $coverageTextShowUncoveredFiles, bool $coverageTextShowOnlySummary, ?string $coverageXml, bool $coverageXmlIncludeSource, bool $pathCoverage, bool $branchCoverage, ?string $coverageDriver, bool $ignoreDeprecatedCodeUnitsFromCodeCoverage, bool $disableCodeCoverageIgnore, bool $disableCoverageTargeting, bool $failOnAllIssues, bool $failOnDeprecation, bool $failOnPhpunitDeprecation, bool $failOnPhpunitNotice, bool $failOnPhpunitWarning, bool $failOnEmptyTestSuite, bool $failOnIncomplete, bool $failOnNotice, bool $failOnRisky, bool $failOnSkipped, bool $failOnWarning, bool $doNotFailOnDeprecation, bool $doNotFailOnPhpunitDeprecation, bool $doNotFailOnPhpunitNotice, bool $doNotFailOnPhpunitWarning, bool $doNotFailOnEmptyTestSuite, bool $doNotFailOnIncomplete, bool $doNotFailOnNotice, bool $doNotFailOnRisky, bool $doNotFailOnSkipped, bool $doNotFailOnWarning, int $stopOnDefect, int $stopOnDeprecation, ?string $specificDeprecationToStopOn, int $stopOnError, int $stopOnFailure, int $stopOnIncomplete, int $stopOnNotice, int $stopOnRisky, int $stopOnSkipped, int $stopOnWarning, bool $outputToStandardErrorStream, int $columns, bool $noExtensions, ?string $pharExtensionDirectory, array $extensionBootstrappers, bool $backupGlobals, bool $backupStaticProperties, bool $beStrictAboutChangesToGlobalState, bool $colors, bool $processIsolation, bool $enforceTimeLimit, int $defaultTimeLimit, int $diffContext, int $timeoutForSmallTests, int $timeoutForMediumTests, int $timeoutForLargeTests, bool $reportUselessTests, bool $strictCoverage, bool $requireCoverageContribution, bool $disallowTestOutput, bool $displayDetailsOnAllIssues, bool $displayDetailsOnIncompleteTests, bool $displayDetailsOnSkippedTests, bool $displayDetailsOnTestsThatTriggerDeprecations, bool $displayDetailsOnPhpunitDeprecations, bool $displayDetailsOnPhpunitNotices, bool $displayDetailsOnTestsThatTriggerErrors, bool $displayDetailsOnTestsThatTriggerNotices, bool $displayDetailsOnTestsThatTriggerWarnings, bool $reverseDefectList, bool $requireCoverageMetadata, bool $requireSealedMockObjects, bool $noProgress, bool $noResults, bool $noOutput, int $executionOrder, int $executionOrderDefects, bool $resolveDependencies, ?string $logfileTeamcity, ?string $logfileJunit, ?string $logfileOtr, bool $includeGitInformation, bool $includeGitInformationInOtrLogfile, ?string $logfileTestdoxHtml, ?string $logfileTestdoxText, ?string $logEventsText, ?string $logEventsVerboseText, bool $compactOutput, bool $teamCityOutput, bool $testDoxOutput, bool $testDoxOutputSummary, ?array $testsCovering, ?array $testsUsing, ?array $testsRequiringPhpExtension, ?string $filter, ?string $excludeFilter, ?string $testIdFilterFile, ?string $testIdFilter, array $groups, array $excludeGroups, int $randomOrderSeed, int $repeat, int $retry, bool $includeUncoveredFiles, TestSuiteCollection $testSuite, string $includeTestSuite, string $excludeTestSuite, ?string $defaultTestSuite, bool $ignoreTestSelectionInXmlConfiguration, array $testSuffixes, Php $php, bool $controlGarbageCollector, int $numberOfTestsBeforeGarbageCollection, ?string $generateBaseline, bool $debug, bool $withTelemetry, int $shortenArraysForExportThreshold, int $numberOfParallelWorkers)

[codecov]

Codecov Report

❌ Patch coverage is 98.03922% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.73%. Comparing base (e18530f) to head (5053a2d).
⚠️ Report is 11776 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/Event/Dispatcher/CollectingDispatcher.php	9.09%	10 Missing ⚠️
src/Event/Facade.php	76.47%	4 Missing ⚠️
...ramework/TestRunner/ChildProcessResultEnvelope.php	90.90%	1 Missing ⚠️
src/Runner/Phpt/TestCase.php	97.14%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #6784       +/-   ##
=============================================
+ Coverage     83.62%   97.73%   +14.11%     
- Complexity     4032     9286     +5254     
=============================================
  Files           153      903      +750     
  Lines         10068    28554    +18486     
=============================================
+ Hits           8419    27908    +19489     
+ Misses         1649      646     -1003     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[Slamdunk]

My 2 cents on the topic:

1. There is no logical possibility that Extensions support parallelization natively: I suggest to require extensions to explicitely provide support to it (in some way) and to stop the run if an extension without parallel support is detected

2. I've been requested to provide both simple worker ID like 1, 2, 3 as well as unique worker ID like uniqid('1_'), uniqid('2_'), uniqid('3_')

3. Coordinated bootstraping operations are crucial to the users, and many asked to be able to have bootstrap running both before the whole execution and within each worker, with a flag to distinguish between the two cases (see https://github.com/paratestphp/paratest#initial-setup-for-all-tests). For example users want to CREATE DATABASE in the parent process and DELETE FROM table in the workers to speed I/O up

4. I see that you chose Distribution unit = one test class which is sensible: be prepared to reply to a mass of users that wants/needs Distribution unit = one test method, what is ParaTest we provide with --functional

• --testsuite unit --parallel 10 matches sequential exactly (same test and assertion counts), stable across runs.

Parallel execution can't and will never be deterministic, a tiny difference between how much a unit lasts results in the following test being run in worker X instead of Y.
Many test suites are not well isolated, and if the developer is unaware of that a test run may fail if the aforementioned test runs in worker X instead of Y.
To be able to debug such situations, the developers needs an option to replay the previous run with exactly the same test order in the exact same worker ID, with no concern on the test speed since you (correctly) chose to pulling from a dynamic work-stealing queue so the load self-balances against stragglers

[sebastianbergmann]

Thank you, @Slamdunk, for taking the time to write all of this up. There's a lot of hard-won experience in here, and I appreciate it.

I want to set expectations honestly, though: for me, native parallel test execution in PHPUnit is still just an idea, and this pull request is a (remarkably well-working) proof of concept rather than a roadmap commitment.

If I ever decide to be serious about this and actually ship it, the feature will have limitations. And those limitations must (and will) be very well documented. Complaints about them will then be kindly refused. 🙂

Even in that case, I neither intend nor expect this to replace dedicated solutions such as ParaTest or Paraunit. The scope I have in mind right now is roughly:

PHPUnit supports parallel execution for well-architected test suites that properly deal with resource-usage conflicts like databases, etc.

So several of the needs you describe, coordinated bootstrapping, functional/method-level distribution, deterministic replay of a run, are exactly the kind of thing that lives outside that scope and is better served by the tools built specifically for it.

That said, your notes are genuinely useful for thinking about where the boundaries of that scope should sit, so thank you again.