Filter `cuda` and `amd` out of the GitHub CI workflow

[kshyatt]

These run on Buildkite but I hate regex so much I will not get to it soon. Either way it is taking up space and churning through resources precompiling stuff.