test(setup): experimental end-to-end setup-flow matrix (all ecosystems, workspaces, monorepo)

[Mikola Lysenko (mikolalysenko)]

What

A non-blocking, experimental end-to-end test matrix that verifies the intended socket-patch setup flow for every supported ecosystem/package manager, including nested workspaces and a polyglot monorepo:

0. prepare a project with a dependency + a committed patch set → 1. socket-patch setup → 2. native install → 3. check whether the patch was applied.

No CLI/source behavior changes — purely test infrastructure (tests/, scripts/, two Dockerfiles, one CI job, a new setup-e2e feature).

Why aspirational + non-blocking

setup only configures npm-family install hooks today, so the non-npm *_with_setup cases are expected to fail. The suite encodes the ideal end state and records a per-case baseline of what works now, so each gap flips known_gap → pass automatically (no test edits) as setup grows support. The CI job is continue-on-error: true and must be left out of required status checks.

Coverage (114 cases)

Declarative in tests/setup_matrix/matrix.json (single source of truth for both the runner and the Rust wrappers):

• Package managers: npm, yarn, pnpm, bun · pip, uv, poetry, pdm, hatch · cargo · bundler · go · mvn · composer · dotnet · deno
• Layouts (SM_LAYOUT):
  • single — one project, one dependency (the 16-PM grid).
  • workspace — a nested workspace: root + several members (incl. a deeply-nested one and a member that doesn't use the patched package). Exercises setup's workspace handling (npm/yarn hook every member; pnpm only the root) + the cross-workspace apply on a root install. PMs: npm, pnpm, yarn (apply) and pip, uv (Python gaps).
  • monorepo — a polyglot all-ecosystem repo: an npm workspace alongside python/rust/go/php/ruby/nuget/deno manifests. Confirms setup works in a mixed environment — configures the npm hooks, doesn't choke on the foreign manifests; a root npm install patches the npm slice.
• Scenarios: baseline_with_setup, plus two ablation controls that confirm setup is correct (identical to the passing case except one removed factor, each must run UNPATCHED): no_setup_control (setup ablated) and patch_missing (committed patch ablated). Also empty_patchset, wrong_target_patchset, alt_content_patchset. Workspace/monorepo carry the matching ablation pair.

Results (verified in Docker, all 9 images on socket-patch 3.3.0)

Ecosystem	PMs	setup+install applies?
npm	npm/yarn/pnpm/bun (single and workspace)	✅ yes
monorepo	npm slice in a polyglot repo	✅ yes
pypi · cargo · gem · golang · maven · composer · nuget · deno	(the rest)	❌ known_gap

• 0 regressions, 0 errors, 0 leaks. Every native install genuinely ran (install_exit=0).
• All ablation/negative controls pass (run unpatched). Paired proof for single npm: baseline_with_setup → applied; no_setup_control & patch_missing → unpatched.
• Existing docker_e2e_npm suite still passes against the extended images; clippy clean on all setup-matrix test targets.

How it works

• tests/setup_matrix/run-case.sh — self-contained, layout-aware bash flow driver (scaffold → setup → install → verify → JSON). Generates npx/pnpm shims inline so the hook resolves to the binary under test (no registry fetch); apply runs offline+force against the committed .socket/ fixture (no Socket API).
• scripts/setup-matrix.sh — CLI/agent-friendly orchestrator: build / run / list / query / results. Classifies pass/known_gap/progress/regression; exits non-zero only on regression.
• crates/socket-patch-cli/tests/setup_matrix_*.rs (+ shared module) — thin Rust wrappers gated by a new setup-e2e feature; assert the aspirational ideal (red on gaps); Docker or SOCKET_PATCH_TEST_HOST=1.
• Dockerfiles: Dockerfile.npm (+pnpm/yarn via corepack) and Dockerfile.pypi (+uv/poetry/pdm/hatch), additively; existing tests unaffected.
• See tests/setup_matrix/README.md.

Try it

scripts/setup-matrix.sh build --ecosystem npm
scripts/setup-matrix.sh run   --ecosystem npm                 # npm family → pass
scripts/setup-matrix.sh run   --scenario workspace_with_setup # nested workspaces
scripts/setup-matrix.sh run   --scenario monorepo_with_setup  # polyglot monorepo
scripts/setup-matrix.sh run   --scenario patch_missing        # ablation → unpatched
scripts/setup-matrix.sh query --status known_gap              # what setup still can't do

What's still missing in setup (the gaps this maps out)

• composer — closest win: composer.json has post-install-cmd, directly analogous to npm; setup just needs to write it.
• pip/uv/poetry/pdm/hatch, cargo, gem, deno — no usable post-install hook (Python needs a .pth/wrapper; deno install ignores the root postinstall).
• golang/maven/nuget — need a hook and integrity-sidecar repair (go.sum, .jar.sha1, .nupkg.sha512).
• Cross-cutting: extend PackageManager beyond Npm/Pnpm, detect non-package.json manifests, stop hardcoding --ecosystems npm in the written hook.

Drive-by findings (not fixed here)

• apply --force is the one SOCKET_* bool flag without BoolishValueParser, so SOCKET_FORCE=1 errors; the harness uses SOCKET_FORCE=true.
• pnpm runs the root postinstall on pnpm install but not pnpm add (npm/yarn/bun run it on add).
• In a workspace the install hook's apply must use the package manager's per-script cwd (member dirs no-op, root applies); pinning SOCKET_CWD to the root breaks npm install mid-run.

🤖 Generated with Claude Code