PEP 11: drop buildbot requirement when CI is being used

[brettcannon]

Right now, PEP 11 requires a buildbot for tier 2 and 3, but not for tier 1. The reason for the difference is tier 1 requires CI while the other tiers don't. But there's nothing precluding the lower tiers from being in CI as well (e.g. WASI is tier 2 but runs in CI).

Would it be okay if I changed the requirement in PEP 11 for tiers 2 and 3 to say, "Must have a reliable buildbot or be tested as part of CI"?

[itamaro]

As long as buildbot provides more coverage vs GitHub CI (fast/slow-ci flags, refleaks, sanitizers, PGO, installed runtime, sequential running, out-of-tree, ...), I'd argue the opposite - tier 1 should also require buildbots!

[brettcannon]

As long as buildbot provides more coverage vs GitHub CI (fast/slow-ci flags

The fast/slow CI flags only change whether the CPU resource is on and I don't see how that changes for e.g. WASI.

refleaks, sanitizers, PGO, installed runtime, sequential running, out-of-tree, ...)

The vast majority of that list doesn't need to be covered by every e.g. CPU type we support (does running out-of-tree differ between aarch and x64?). Note I'm not saying to shut down all buildbots, just not to blanket require a buildbot when a specific need isn't there. As well, that's a massive increase in resource requirements if we did add that sort of requirement for every platform in PEP 11, especially considering how much hardware costs these days.

[brettcannon]

I will also say that @hugovk as RM liked de-duplicating buildbots ("if it's already covered by GHA that's usually faster and more reliable than buildbots" and the GH UI is easier to work with than buildbot).

Basically I want to be trusted to make the call whether CI is enough to cover testing compared to requiring a buildbot that's (at least in my case) redundant because I messed up and didn't think about non-tier 1 platforms making it into CI when adding tiers to PEP 11.

[itamaro]

I'm ok with framing it around trusting the core team instead of strict requirements.

Maybe the wording can be something like "sufficient test coverage as appropriate for the platform" + tier 1 requires "pre-merge test coverage" (effectively GHA, but without being prescriptive about the technology to achieve the objective)