validate.yml builds its matrix from the frameworks a PR touches:
changed=$(git diff --name-only origin/$BASE...HEAD | grep '^frameworks/' | cut -d'/' -f2 | sort -u)
So a PR that only changes scripts/validate.sh starts the workflow and validates nothing, and a PR that only changes the rules under site/content/docs/ does not start it at all. A new check never meets the entries that were merged before it.
That is not theoretical this week. #1209 on the 19th said static file bodies must be read from disk on every request. #1261 on the 22nd said a framework cache is fine when it is the framework's own and follows the disk, which is close to the opposite. #1267 added the staleness probe the same day, and validated no existing entry. Twelve commits touched the rules or the validator in the last ten days.
The effect is that an entry can be correct on Tuesday and wrong on Friday without anyone knowing, its author included, and the board can publish entries that no longer pass the rules it publishes next to them.
Two things would close it:
- When a PR changes
scripts/validate* or the implementation rules, validate every enabled entry rather than none. That is slow, so it does not have to block the merge: a scheduled run right after would be enough.
- When an entry fails and its
meta.json lists maintainers, tell them. notify-maintainers.yml already reads that field and pings them on PRs, so the machinery exists and would only need a failure to point at. An entry with no maintainer has nobody to tell, which is its own argument for filling the field in.
I am asking as someone with entries here: right now keeping them valid means following every commit to the rules, and I would rather be told by a bot than find out from a run.
validate.ymlbuilds its matrix from the frameworks a PR touches:changed=$(git diff --name-only origin/$BASE...HEAD | grep '^frameworks/' | cut -d'/' -f2 | sort -u)So a PR that only changes
scripts/validate.shstarts the workflow and validates nothing, and a PR that only changes the rules undersite/content/docs/does not start it at all. A new check never meets the entries that were merged before it.That is not theoretical this week. #1209 on the 19th said static file bodies must be read from disk on every request. #1261 on the 22nd said a framework cache is fine when it is the framework's own and follows the disk, which is close to the opposite. #1267 added the staleness probe the same day, and validated no existing entry. Twelve commits touched the rules or the validator in the last ten days.
The effect is that an entry can be correct on Tuesday and wrong on Friday without anyone knowing, its author included, and the board can publish entries that no longer pass the rules it publishes next to them.
Two things would close it:
scripts/validate*or the implementation rules, validate every enabled entry rather than none. That is slow, so it does not have to block the merge: a scheduled run right after would be enough.meta.jsonlistsmaintainers, tell them.notify-maintainers.ymlalready reads that field and pings them on PRs, so the machinery exists and would only need a failure to point at. An entry with no maintainer has nobody to tell, which is its own argument for filling the field in.I am asking as someone with entries here: right now keeping them valid means following every commit to the rules, and I would rather be told by a bot than find out from a run.