Use SciMLTesting v1.2 (folder-based run_tests)#63
Merged
ChrisRackauckas merged 1 commit intoJun 14, 2026
Conversation
Convert the grouped-tests harness to the SciMLTesting v1.2 folder model. runtests.jl is now `using SciMLTesting; run_tests()`; group membership is discovered from the test/ folder layout + test_groups.toml. Core = top-level test/*.jl. The inline Core testsets that previously lived in runtests.jl (version bumping, project-file handling, repository processing, legacy deprecations) are extracted into a self-contained test/core_tests.jl; each existing Core file gains its own `using` lines so it runs under an isolated @safetestset module. QA = test/qa/ (qa.jl moved in from the top level so the Core glob does not pick it up; keeps its sub-env Project.toml). multiprocess_testing_tests.jl was commented out of the old dispatcher, so it moves to test/shared/ (not a group folder) to preserve the exact per-GROUP test set. Added SciMLTesting + SafeTestsets to the root [extras]/[targets].test and to test/qa/Project.toml (with [compat]). test_groups.toml unchanged. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ChrisRackauckas-Claude
force-pushed
the
sciml-testing-rollout
branch
from
c07c4b6 to
b0330b3
Compare
ChrisRackauckas-Claude
changed the title
ChrisRackauckas
added a commit
that referenced
this pull request
Jun 21, 2026
…re (#67) * Fix Core/Downgrade test failures: add `using Dates`, fix stdout capture Two latent bugs surfaced after the SciMLTesting folder-based `run_tests` migration (#63) isolated each test file in its own `@safetestset` module: - `test/import_timing_analysis_tests.jl` and `test/invalidation_analysis_tests.jl` use `now()` / `DateTime(...)` but only did `using OrgMaintenanceScripts` / `using Test`. Under the per-file module isolation those `Dates` names are no longer in scope, so they threw `UndefVarError: now/DateTime not defined`, erroring the Core (julia 1, lts) and Downgrade jobs. Add `using Dates` to both test files. - The import-timing subprocess script redirected stdout into an `IOBuffer`, which `redirect_stdout` does not support (`MethodError: no method matching (::Base.RedirectStdStream)(::IOBuffer)` on both 1.10 and 1.12). Capture into a temp file (an IOStream is supported) and read it back, so the timing analysis actually works instead of always falling into the error branch. Verified locally with `Pkg.test()` GROUP=Core on Julia 1.10 (lts) and 1.12 ("1"): all eight Core test files pass (import_timing 56/56, invalidation 36/36), "Testing OrgMaintenanceScripts tests passed", exit 0. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Raise JSON3 compat floor to 1.1 for Downgrade green The Downgrade job resolves JSON3 to v1.0.0, where `JSON3.pretty` does not exist (`UndefVarError: pretty not defined`). `write_import_timing_report` calls `JSON3.pretty`, so both its testset and `analyze_repo_import_timing` (which calls it) errored on Downgrade while passing on Core (newer JSON3). `JSON3.pretty` was introduced in JSON3 v1.1.0 (src/pretty.jl). Raising the compat floor from "1" to "1.1" makes the minimum resolved version one that has the function, fixing Downgrade without capping the upper bound (still allows the current v1.14.x). Verified locally on Julia 1.10 with JSON3 pinned to the floor v1.1.0: import_timing_analysis_tests.jl passes 56/56, exit 0, zero errors. Reproduced the original failure on v1.0.0 (UndefVarError) beforehand. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: ChrisRackauckas-Claude <accounts@chrisrackauckas.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Convert OrgMaintenanceScripts.jl to the SciMLTesting v1.2 folder model.
test/runtests.jlis now:Group membership is discovered from the
test/folder layout plus theexisting
test/test_groups.toml(unchanged):test/*.jl. The Core testsets that used to live inlinein
runtests.jlare extracted into a self-containedtest/core_tests.jl;the seven existing Core files each gained their own
usinglines so they rununder isolated
@safetestsetmodules.test/qa/(its sub-envProject.tomlis retained).qa.jlwasmoved from the top level into
test/qa/so the Core glob does not pick it up.multiprocess_testing_tests.jlwas commented out of the old dispatcher, so itis parked under
test/shared/(not a group folder) — the exact set of testsrun under each GROUP is unchanged.
Dep edits: added
SciMLTesting+SafeTestsetsto the root[extras]/[targets].testand totest/qa/Project.toml(with[compat]).Behavior-preserving: a folder-discovery dry-run confirms GROUP=Core/All fires
core_tests.jl+ the seven Core files (and not the dormant multiprocess file),and GROUP=QA fires
qa.jlin the QA sub-env — matching the previous dispatch.Supersedes the older v1.0.0 run_tests attempt and the separate
canonicalize-safetestsetPR.Ignore until reviewed by @ChrisRackauckas.