feat: Adding e2e hook tests#990
Open
pen-pal wants to merge 8 commits into
Open
Conversation
pen-pal
requested review from
MaxymVlasov,
antonbabenko and
yermulnik
as code owners
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (18)
✅ Files skipped from review due to trivial changes (7)
🚧 Files skipped from review as they are similar to previous changes (10)
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdds end-to-end test infrastructure: a bash test runner, README, GitHub Actions workflow, and two terraform-focused e2e cases (terraform_fmt and terraform_wrapper_module_for_each) with inputs, expected outputs, and requires files. ChangesEnd-to-End Test Infrastructure
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/e2e-tests.yaml:
- Around line 31-33: The workflow uses env.IMAGE set to
ghcr.io/antonbabenko/pre-commit-terraform:latest which allows image drift;
change the IMAGE value to a digest-pinned reference
(ghcr.io/antonbabenko/pre-commit-terraform@sha256:<digest>) so the e2e runner
always pulls an immutable artifact, updating any references to IMAGE used in the
workflow (the env block and any docker pull/docker run invocations); obtain the
exact sha256 digest for the desired image tag via a docker registry query or
docker pull + docker inspect (or GHCR API) and replace :latest with
`@sha256`:<digest>.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 0f0b0b52-8698-431c-922f-5536c499b88a
📒 Files selected for processing (18)
.github/workflows/e2e-tests.yamltests/e2e/README.mdtests/e2e/cases/terraform_fmt/reformats-misaligned-hcl/.pre-commit-config.yamltests/e2e/cases/terraform_fmt/reformats-misaligned-hcl/expected/main.tftests/e2e/cases/terraform_fmt/reformats-misaligned-hcl/expected_returncodetests/e2e/cases/terraform_fmt/reformats-misaligned-hcl/input/main.tftests/e2e/cases/terraform_fmt/reformats-misaligned-hcl/requirestests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/.pre-commit-config.yamltests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/main.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/versions.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/wrappers/README.mdtests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/wrappers/main.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/wrappers/outputs.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/wrappers/variables.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/wrappers/versions.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/input/main.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/input/versions.tftests/e2e/run_e2e_tests.sh
yermulnik
reviewed
May 28, 2026
Collaborator
yermulnik
left a comment
yermulnik
left a comment
There was a problem hiding this comment.
Thank you for the contribution.
I'm out of context around tests and therefore I've added solely tech comments around Bash code.
pen-pal
added a commit
to pen-pal/pre-commit-terraform
that referenced
this pull request
May 29, 2026
Per review on antonbabenko#990: - replace the `RETURN` trap with explicit cleanup at the function's single exit (the `RETURN` trap is Bash-only and confused reviewers); - read the expected return code with `$(< file)` instead of `$(cat file)`; - quote the `true`/`false` flag values to avoid confusion with the builtins; - use `&>` instead of `> ... 2>&1`; - drop the redundant trailing slash on the copy destination; - print the summary with `printf` instead of a loop. No behavior change; the suite still passes natively and inside the image. Signed-off-by: penpal <unameme@proton.me>
pen-pal
added a commit
to pen-pal/pre-commit-terraform
that referenced
this pull request
May 29, 2026
Per review on antonbabenko#990, pin `ghcr.io/antonbabenko/pre-commit-terraform` by `sha256` digest instead of the floating `:latest` tag, so the bundled toolchain can't drift between runs. A Renovate annotation keeps the digest updated. Signed-off-by: penpal <unameme@proton.me>
yermulnik
reviewed
May 29, 2026
Collaborator
yermulnik
left a comment
yermulnik
left a comment
There was a problem hiding this comment.
@MaxymVlasov Shell code looks good to me, though I'm not really good with the rest of the tests logic — could you please have a look when you've got a moment? Thank you.
Introduce `tests/e2e/run_e2e_tests.sh`, a behavioral test runner that exercises each hook the way a user would -- through `pre-commit` against fixture files -- and compares the result against a committed golden tree. It auto-discovers `cases/<hook>/<case>/`, builds a temp git repo from `input/`, renders the case's `.pre-commit-config.yaml` (`__PCT_REPO__` -> checkout root), runs the hook via `pre-commit run --all-files`, then asserts the exit code matches `expected_returncode` (default 0) and the working tree is byte-identical to `expected/`. The tree compare uses `git diff --no-index` so it works with the BusyBox `diff` shipped in the project image. A `requires` file skips a case when a needed CLI tool is absent. Part of antonbabenko#823. Signed-off-by: penpal <unameme@proton.me>
Add `generates-wrapper-for-root-module`: a root module (with no
`provider_meta`) is wrapped, and the generated
`wrappers/{main,outputs,variables,versions}.tf` plus `README.md` are
checked against the golden tree. Exits 0 (only new files are created).
Part of antonbabenko#823.
Signed-off-by: penpal <unameme@proton.me>
Add `reformats-misaligned-hcl`: misaligned HCL is rewritten to canonical style. The hook modifies a tracked file, so `pre-commit` exits 1 (`expected_returncode=1`). The case requires `terraform`. Part of antonbabenko#823. Signed-off-by: penpal <unameme@proton.me>
Add `.github/workflows/e2e-tests.yaml`. Checkout runs on the host (the JS action needs Node), then the suite runs inside the published multi-tool image (`ghcr.io/antonbabenko/pre-commit-terraform:latest`), which supplies terraform/hcledit/pre-commit while the hook and test code come from the checkout -- so the image tag need not be in sync. Part of antonbabenko#823. Signed-off-by: penpal <unameme@proton.me>
Add `tests/e2e/README.md` describing the case layout, how to run the suite natively and inside the project image, and how to add a new case. Part of antonbabenko#823. Signed-off-by: penpal <unameme@proton.me>
Per review on antonbabenko#990: - replace the `RETURN` trap with explicit cleanup at the function's single exit (the `RETURN` trap is Bash-only and confused reviewers); - read the expected return code with `$(< file)` instead of `$(cat file)`; - quote the `true`/`false` flag values to avoid confusion with the builtins; - use `&>` instead of `> ... 2>&1`; - drop the redundant trailing slash on the copy destination; - print the summary with `printf` instead of a loop. No behavior change; the suite still passes natively and inside the image. Signed-off-by: penpal <unameme@proton.me>
Per review on antonbabenko#990, pin `ghcr.io/antonbabenko/pre-commit-terraform` by `sha256` digest instead of the floating `:latest` tag, so the bundled toolchain can't drift between runs. A Renovate annotation keeps the digest updated. Signed-off-by: penpal <unameme@proton.me>
Signed-off-by: penpal <unameme@proton.me>
pen-pal
force-pushed
the
feat/823-e2e-hook-tests
branch
from
235410e to
a6052ac
Compare
MaxymVlasov
reviewed
Jun 1, 2026
Collaborator
MaxymVlasov
left a comment
MaxymVlasov
left a comment
There was a problem hiding this comment.
Thank you for introducing this simple but powerful e2e test strategy!
I have a few requests for you before we proceed.
| env: | ||
| # renovate: datasource=docker depName=ghcr.io/antonbabenko/pre-commit-terraform | ||
| # yamllint disable-line rule:line-length | ||
| IMAGE: ghcr.io/antonbabenko/pre-commit-terraform:latest@sha256:4ef4b8323b27fc263535ad88c9d2f20488fcb3b520258e5e7f0553ed5f6692b5 |
Collaborator
There was a problem hiding this comment.
If we pin it to hash, then there's no sense to use the latest at all, which contradicts the main idea - test againt latest tool set & against the latest stable hook version.
In other words, we need 2 tests: against latest w/o pinning and against the latest available hook version, which can be bumped by renovate, but I'd like to have it as part of the release process.
|
|
||
| Behavioral tests that run the hooks the same way a user would — through | ||
| `pre-commit` against real fixture files — and compare the result against a | ||
| committed "golden" output. See [issue #823][issue-823]. |
Collaborator
There was a problem hiding this comment.
Never understand the idea of complicating things like this. It is used only in 1 place.
Suggested change
| committed "golden" output. See [issue #823][issue-823]. | |
| committed "golden" output. For details see [issue #823](https://github.com/antonbabenko/pre-commit-terraform/issues/823). |
| && tar -xzf hcledit.tar.gz hcledit && rm hcledit.tar.gz && sudo mv hcledit /usr/bin/ | ||
| ``` | ||
|
|
||
| See the repo [README "How to install"](../../README.md#how-to-install) for the |
Collaborator
There was a problem hiding this comment.
Suggested change
| See the repo [README "How to install"](../../README.md#how-to-install) for the | |
| See the repo [README "How to install"](/README.md#how-to-install) for the |
Comment on lines
+71
to
+72
| docker build -t pct:e2e --build-arg INSTALL_ALL=true . | ||
| docker run --rm -v "$PWD:/lint" -w /lint --entrypoint bash pct:e2e \ |
Collaborator
There was a problem hiding this comment.
And what's the reason to build a separate image?
Just use a exisitng one https://github.com/antonbabenko/pre-commit-terraform#1-install-dependencies
Comment on lines
+87
to
+88
|
|
||
| [issue-823]: https://github.com/antonbabenko/pre-commit-terraform/issues/823 |
Collaborator
There was a problem hiding this comment.
Suggested change
| [issue-823]: https://github.com/antonbabenko/pre-commit-terraform/issues/823 |
Collaborator
There was a problem hiding this comment.
Let's add a section about requesting the addition of e2e hooks to https://github.com/antonbabenko/pre-commit-terraform/blob/master/.github/CONTRIBUTING.md#add-new-hook
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
Increment toward #823: a behavioral end-to-end test harness for the hooks. It runs each hook the way a user would - through
pre-commitagainst fixture files — and compares the result against a committed "golden" output tree.Addresses #823 (lays the foundation and seeds it with two hooks). It does not close the issue, since most hooks still need cases.
Harness -
tests/e2e/run_e2e_tests.shtests/e2e/cases/<hook_id>/<case_name>/. Adding a hook is just adding a dir — no runner changes.input/, renders the case's.pre-commit-config.yaml(__PCT_REPO__→ checkout root), runs the hook viapre-commit run --all-files.expected_returncode(default0) and the working tree is byte-identical toexpected/.requiresfile skips a case when a needed CLI tool is absent.Seed cases
terraform_fmtreformats-misaligned-hclpre-commitexits1(hook modifies a tracked file)terraform_wrapper_module_for_eachgenerates-wrapper-for-root-modulewrappers/{main,outputs,variables,versions}.tf+README.mdgenerated for aCI -
.github/workflows/e2e-tests.yamlRuns the suite on every PR inside the published multi-tool image (
ghcr.io/antonbabenko/pre-commit-terraform:latest). The image supplies the tooling (terraform,hcledit, …); the hook and test code come from the checkout, so the image tag need not be in sync.How can we test changes
Locally (needs
pre-commit,git, and the hooks' tools —terraform,hcledit):bash tests/e2e/run_e2e_tests.sh # => 2 passed, 0 failed, 0 skippedInside the project image (matches CI):