fix(ci,test): stabilize JRuby and Rails matrix by hardening dependency resolution, test synchronization, and legacy image builds

[hegerchr]

Summary

This PR started as a dev workflow fix and was expanded to address several CI instabilities uncovered during matrix runs. The final result is a clean CI run by resolving framework compatibility drift, JRuby test races, and legacy Debian image build failures.

What changed

• BDD runner reliability:
  • Updated the BDD runner invocation to use Bundler-managed executables so cucumber is always available in CI environments.
• Matrix compatibility updates:
  • Excluded grape master on Ruby versions no longer supported upstream.
  • Pinned delayed_job for Rails 4.x and 5.x compatibility to avoid ActiveJob adapter NameError failures.
  • Added a targeted shoryuken compatibility constraint for the Ruby 3.4 + Rails 6.1 combination.
• Legacy Debian image resilience:
  • Added Debian Buster archive fallback handling for CI spec image builds.
  • Added required build tools for JRuby-based images, including curl and GPG tooling needed by gosu verification.
  • Added a safe default base image argument in the spec Docker build definition.
• JRuby test flake fixes:
  • Updated the Rails JSON schema integration test wait condition to wait for the full expected payload set.
  • Hardened MockIntake synchronization with mutex-protected writes and synchronized count/snapshot reads.
  • Fixed wait_for logic so each expected event type must independently satisfy its threshold, preventing cross-category cancellation.

Why this was needed

• Upstream framework changes introduced new resolver failures in the matrix.
• JRuby true concurrency exposed race and visibility issues in test intake polling.
• EOL Debian Buster repositories caused deterministic image build failures in legacy targets.
• Missing base-image utilities caused secondary CI build regressions in JRuby jobs.

Outcome

CI is now passing after these follow-up commits. Changes are focused on test, CI, and build stability, with no intended production runtime behavior change.

Notes for reviewers

The PR now includes both the original dev workflow fix and the required CI hardening discovered during validation.
Most changes are in test and CI support paths and are scoped to restore matrix reliability across legacy and JRuby targets.

[estolfo]

This is a lot of gymnastics and I don't know the motivation behind a lot of the changes, but maybe you can explain where I put specific comments?

[estolfo]

I can't say I've done deep research on the justification behind all the changes but they seem to make the tests pass so I think it's ok to merge.