Skip to content

Canonicalize tests to @safetestset for module isolation#64

Closed
ChrisRackauckas-Claude wants to merge 1 commit into
SciML:mainfrom
ChrisRackauckas-Claude:canonicalize-safetestset
Closed

Canonicalize tests to @safetestset for module isolation#64
ChrisRackauckas-Claude wants to merge 1 commit into
SciML:mainfrom
ChrisRackauckas-Claude:canonicalize-safetestset

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor

What

Converts each independent test unit to run in its own module via @safetestset, matching the canonical SciML test structure (isolation between tests + world-age safety). Behavior-preserving: same tests run, same assertions, same GROUP dispatch.

Core (test/runtests.jl)

The Core group was a single monolithic @testset "OrgMaintenanceScripts.jl" wrapping 4 inline @testsets plus 7 include(...) calls, all relying on imports leaked into Main from the top of runtests.jl.

  • The 4 inline units (Version bumping, Project file handling, Repository processing, Basic functionality legacy) become inline @safetestset blocks, each carrying its own using lines (OrgMaintenanceScripts, Test, plus TOML where used).
  • Each per-file unit becomes @safetestset "Name" begin include("x.jl") end. The included files are made self-contained: the stale # Note: ... already loaded by runtests.jl comments are replaced with explicit using lines. Two files additionally need using Dates (they call now() / DateTime at top level), and one needs using TOML (calls TOML.print). These previously worked only via Main-leaked imports.
  • The already-commented-out multiprocess_testing include is preserved (still disabled), shown in @safetestset form for consistency.

QA (test/qa.jl)

The two plain @testset units (Aqua, JET) are extracted into qa_aqua.jl / qa_jet.jl and wrapped as @safetestset ... include(...). The include() form is required for the JET unit because @test_opt is a package macro that must resolve after using JET runs (top-level statements in the included file macroexpand one at a time, unlike an inline @safetestset body). @test_broken lines preserved.

Deps

SafeTestsets added to Project.toml [extras] / [targets].test and [compat] SafeTestsets = "0.1, 1"; same UUID + compat added to test/qa/Project.toml (QA runs in its own activated env). using SafeTestsets added at the top of runtests.jl.

Verification

Ran locally on Julia 1.11: GROUP=Core Pkg.test() passes. All 11 converted units pass:
Version bumping 5/5, Project file handling 6/6, Repository processing 8/8, Basic functionality (legacy) 2/2, Formatting Functions 9/9, Minimum Version Fixer 25/25, Version Check Finder 23/23, Invalidation Analysis 36/36, Import Timing Analysis 56/56, Explicit Imports Fixer 10/10, Documentation Cleanup 41/41.

(The "Analysis failed" log lines from the invalidation/import-timing tests are pre-existing expected behavior: those tests deliberately run analysis on a mock package and assert the failure path returns -1 — unrelated to this change.)

Ignore until reviewed by @ChrisRackauckas.

Convert each independent test unit to run in its own module via
@safetestset, matching the canonical SciML structure (isolation between
tests + world-age safety).

Core (runtests.jl):
- The 4 inline @testset units (Version bumping, Project file handling,
  Repository processing, Basic functionality legacy) become inline
  @safetestset blocks, each carrying the using lines it needs
  (OrgMaintenanceScripts, Test, plus TOML where used).
- Each per-file unit (formatting, min_version_fixer, version_check_finder,
  invalidation_analysis, import_timing_analysis, explicit_imports_fixer,
  documentation_cleanup) is now @safetestset "Name" begin include("x.jl") end.
  The included files are made self-contained: stale
  "already loaded by runtests.jl" comments replaced with explicit using
  lines (OrgMaintenanceScripts, Test, plus Dates/TOML where the body uses
  now()/DateTime or TOML.print). These previously relied on Main-leaked
  imports from runtests.jl.
- The commented-out multiprocess_testing include is preserved (still
  disabled), now shown in @safetestset form.

QA (qa.jl): the two plain @testset units (Aqua, JET) extracted into
qa_aqua.jl / qa_jet.jl and wrapped as @safetestset ... include(...). The
include() form is required for the JET unit because @test_opt is a package
macro that must be resolved after `using JET` runs (top-level statements in
the included file macroexpand one at a time). @test_broken lines preserved.

Deps: SafeTestsets added to Project.toml [extras]/[targets].test and
[compat] SafeTestsets = "0.1, 1"; same UUID + compat added to
test/qa/Project.toml (QA runs in its own activated env). using SafeTestsets
added at top of runtests.jl.

GROUP-dispatch ladder, run path, assertions, and test counts unchanged.
Verified locally: GROUP=Core Pkg.test passes (Version bumping 5, Project
file handling 6, Repository processing 8, Basic functionality 2, Formatting
9, Min Version Fixer 25, Version Check Finder 23, Invalidation 36, Import
Timing 56, Explicit Imports 10, Documentation Cleanup 41).

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor Author

Superseded by the v1.2 folder conversion on sciml-testing-rollout (#63).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants