Add support for running XCTest/XCUITest on-device#236
Open
Voten641 wants to merge 3 commits into
Open
Conversation
Implements a from-scratch DTX/testmanagerd client so xtool can build, install, and run a SwiftPM package's test targets on a connected iOS device, driving the same protocol Xcode uses under the hood. - xtool test: builds every .testTarget into SwiftPM's combined test bundle, packages it into a Runner.app, installs it, and runs it via testmanagerd - --test-target picks which target to run when a package has more than one (interactive prompt if unset); --only/--skip filter by class or method - --repeat, --parallel, --session-timeout, --report-directory - JUnit/JSON/HTML reporters, --screenshot-on-failure, --capture-syslog, --capture-crash-logs - Personalized Developer Disk Image mounting for iOS 17+ (TSS/manifest flow via libtatsu), auto-mounted on demand - iOS 17+ RSD/CoreDevice tunnel support (TUN device, HTTP/2 framing, RemoteXPC) for testmanagerd on newer devices - xtool tunnel: diagnostic command for exercising the tunnel/DTX handshake directly Core new subsystem lives under Sources/XKit/Testing/: a clean-room DTX wire protocol implementation (DTXMessage/DTXConnection/DTXTransport), an NSKeyedArchiver-compatible encoder (NSKeyedArchive), the XCTestConfiguration plist writer, and TestManagerdSession orchestrating the handshake.
- HTMLReporter.escape() didn't escape double quotes, so a device-reported test name or artifact path containing " could break out of src="..."/ href="..." and inject arbitrary attributes/markup into the generated report. Matches JUnitReporter's existing (correct) escaping. - Int(UInt64) conversions in DTXMessage.parseBody and NSKeyedArchive's UID decoding trap on values >= 2^63 instead of failing gracefully on a corrupted/desynced stream. Switched to Int(exactly:). - RemoteXPCCodec's array decoder reserved capacity for an unvalidated wire-provided element count before validating any of the backing bytes; clamped to the remaining buffer size. - SyslogRelayClient.LineBox's continuation was read/written outside its lock while buffer was correctly guarded; now both are lock-protected.
Plain `xtool test` picking a UI test target (no --target-bundle-id) was silently unable to launch the app under test -- XCUITest's XCUIApplication() had nothing to drive, so only the Runner opened. Xcode defaults a UI test target to the app in the same project, so do the same: when --target-bundle-id isn't given and the chosen target name ends in "UITests", resolve what this package's own app is actually installed as on the device (accounting for the team-ID prefix free-tier signing adds -- the plan only knows the raw, unprefixed bundle ID from xtool.yml/Package.swift) and use that. Falls back to a clear error if the app isn't installed at all. Verified against a real device: previously only opened the xctrunner app, now correctly launches and drives the actual app, with real UI tests passing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds
xtool testso you can build, install, and run a package's tests (XCTest + XCUITest) on a real device straight from Linux — no Xcode needed. Had to implement the DTX/testmanagerd protocol from scratch since nothing in the dependency tree covers it.Basic usage:
builds every
.testTarget, packages it into a Runner.app, installs it, and runs it over testmanagerd, same as Xcode does under the hood.Main pieces
--test-targetto pick which target runs if there's more than one (asks interactively otherwise)--only/--skipto filter down to a class or a single method--target-bundle-idfor XCUITest against an app you've already installed separately —xtool testnever touches the main app itself, install that withxtool install/xtool devlike normal--repeat,--parallel(runs on every connected device at once),--session-timeout--screenshot-on-failure,--capture-syslog,--capture-crash-logsUnder the hood there's a new DTX implementation (message/connection/transport), an NSKeyedArchiver-compatible encoder (testmanagerd needs real bplist archives with the right
$classname, and Foundation on Linux doesn't emit those correctly), andTestManagerdSessionwhich does the actual handshake — two testmanagerd sessions plus an instruments session to launch the runner.Also had to add:
xtool tunnelas a diagnostic command for poking at the tunnel/DTX handshake directly when something's not workingThings worth knowing
--test-target/multiple--onlyjust run one session per class and combine the results, which means big test targets take a while since each class needs its own app relaunch.@Test) support yet, only classicXCTestCase.Testing
Added 24 unit tests covering the DTX format, the archiver, RemoteXPC decoding, plist parsing, test-target detection, and the XCTestCase discovery logic (including the case where tests inherit from a shared base class instead of
XCTestCasedirectly, which was a real bug the first version had).Beyond that, ran it against a real device for actual test execution (both XCTest and XCUITest), multi-class filtering, repeat runs, crash log collection, syslog capture, and the install-cap error path.