Add periodic CI cluster-updates evaluation jobs for 4.22#82093
Draft
fao89 wants to merge 1 commit into
Draft
Conversation
Add CI configuration for cluster-updates-evals with periodic evaluation jobs for each LLM provider (OpenAI, WatsonX, Google Vertex, Google Vertex Anthropic) running daily against OCP 4.22 clusters. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
Contributor
|
Skipping CI for Draft Pull Request. |
Member
Author
|
/pj-rehearse |
Contributor
WalkthroughThe repository receives generated ownership and Prow configuration, alongside an OCP 4.22 CI evaluation definition with four scheduled provider-specific periodic tests using the cluster-updates test script. ChangesCluster updates evaluation onboarding
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant PeriodicScheduler
participant GenericClaimWorkflow
participant ProviderSecrets
participant LightspeedServiceAPI
participant ClusterUpdatesScript
PeriodicScheduler->>GenericClaimWorkflow: Start scheduled provider evaluation
GenericClaimWorkflow->>ProviderSecrets: Mount provider token secrets
GenericClaimWorkflow->>LightspeedServiceAPI: Provide OLS_IMAGE dependency
GenericClaimWorkflow->>ClusterUpdatesScript: Run test-cluster-updates.sh
ClusterUpdatesScript-->>GenericClaimWorkflow: Return evaluation result
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 1 warning)
✅ Passed checks (13 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
|
@fao89: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: fao89 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
Member
Author
|
/pj-rehearse periodic-ci-fao89-cluster-updates-evals-main-4.22-cu-eval-openai-periodics |
Contributor
|
@fao89: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/fao89/cluster-updates-evals/fao89-cluster-updates-evals-main__4.22.yaml`:
- Around line 25-180: Run make update to regenerate the downstream Prow job
definitions for the four new periodic tests—cu-eval-openai-periodics,
cu-eval-watsonx-periodics, cu-eval-vertex-periodics, and
cu-eval-vertex-anthropic-periodics—and include all generated artifacts in the
change.
- Around line 9-11: Update the Dockerfile fragment after the Git installation in
the main image build to restore the image’s standard non-root runtime user
instead of leaving USER 0 active. Add the appropriate non-root USER directive
immediately after the dnf cleanup, preserving the existing installation steps.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 48f09b52-3001-4666-874e-68ce37e75e35
⛔ Files ignored due to path filters (1)
ci-operator/jobs/fao89/cluster-updates-evals/fao89-cluster-updates-evals-main-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (5)
ci-operator/config/fao89/cluster-updates-evals/OWNERSci-operator/config/fao89/cluster-updates-evals/fao89-cluster-updates-evals-main__4.22.yamlcore-services/prow/02_config/fao89/cluster-updates-evals/OWNERScore-services/prow/02_config/fao89/cluster-updates-evals/_pluginconfig.yamlcore-services/prow/02_config/fao89/cluster-updates-evals/_prowconfig.yaml
Comment on lines
+9
to
+11
| FROM registry.ci.openshift.org/ocp/ubi-python-311:9 | ||
| USER 0 | ||
| RUN dnf install -y git && dnf clean all |
Contributor
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file='ci-operator/config/fao89/cluster-updates-evals/fao89-cluster-updates-evals-main__4.22.yaml'
echo "== File context =="
wc -l "$file"
sed -n '1,220p' "$file" | nl -ba | sed -n '1,120p'
echo
echo "== Search for related build-root Dockerfiles / USER instructions =="
rg -n --hidden --no-messages 'USER 0|USER 1001|build-root|dnf install -y git|FROM registry\.ci\.openshift\.org/ocp/ubi-python-311:9' ci-operator/config/fao89/cluster-updates-evals -S
echo
echo "== Nearby generated metadata / job config references =="
rg -n --hidden --no-messages 'zz_generated_metadata|periodic|presubmit|postsubmit|build_root|build-root' ci-operator/config/fao89/cluster-updates-evals -SRepository: openshift/release
Length of output: 309
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file='ci-operator/config/fao89/cluster-updates-evals/fao89-cluster-updates-evals-main__4.22.yaml'
echo "== Selected line ranges =="
sed -n '1,80p' "$file"
echo
sed -n '80,160p' "$file"
echo
sed -n '160,220p' "$file"
echo
echo "== Exact build-root / user references =="
rg -n '^( {2,})?(FROM|USER|RUN|build_root|build-root|dockerfile|dockerfile_literal):|USER 0|USER 1001|dnf install -y git' "$file" -n -S
echo
echo "== Other similar configs with USER restore patterns =="
rg -n 'USER 0|USER 1001|dnf install -y git' ci-operator/config -SRepository: openshift/release
Length of output: 41037
Restore a non-root runtime user after installing Git.
ci-operator/config/fao89/cluster-updates-evals/fao89-cluster-updates-evals-main__4.22.yaml:9-11 leaves the build-root image running as root after dnf install. Switch back to the image’s standard non-root UID before it is used in CI.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/config/fao89/cluster-updates-evals/fao89-cluster-updates-evals-main__4.22.yaml`
around lines 9 - 11, Update the Dockerfile fragment after the Git installation
in the main image build to restore the image’s standard non-root runtime user
instead of leaving USER 0 active. Add the appropriate non-root USER directive
immediately after the dnf cleanup, preserving the existing installation steps.
Source: Coding guidelines
Comment on lines
+25
to
+180
| tests: | ||
| - as: cu-eval-openai-periodics | ||
| cluster_claim: | ||
| architecture: amd64 | ||
| cloud: aws | ||
| labels: | ||
| region: us-east-1 | ||
| owner: obs | ||
| product: ocp | ||
| timeout: 2h0m0s | ||
| version: "4.22" | ||
| cron: 30 14 * * * | ||
| steps: | ||
| allow_skip_on_success: true | ||
| post: | ||
| - ref: gather-must-gather | ||
| test: | ||
| - as: e2e | ||
| cli: latest | ||
| commands: | | ||
| export OPENAI_PROVIDER_KEY_PATH=/var/run/openai/token | ||
| export OPENAI_API_KEY=$(cat /var/run/openai/token) | ||
|
|
||
| tests/scripts/test-cluster-updates.sh --artifact-dir "${ARTIFACT_DIR}" | ||
| credentials: | ||
| - mount_path: /var/run/openai | ||
| name: openai-apitoken | ||
| namespace: test-credentials | ||
| dependencies: | ||
| - env: OLS_IMAGE | ||
| name: lightspeed-service-api | ||
| from: src | ||
| resources: | ||
| requests: | ||
| cpu: 100m | ||
| workflow: generic-claim | ||
| - as: cu-eval-watsonx-periodics | ||
| cluster_claim: | ||
| architecture: amd64 | ||
| cloud: aws | ||
| labels: | ||
| region: us-east-1 | ||
| owner: obs | ||
| product: ocp | ||
| timeout: 2h0m0s | ||
| version: "4.22" | ||
| cron: 30 15 * * * | ||
| steps: | ||
| allow_skip_on_success: true | ||
| post: | ||
| - ref: gather-must-gather | ||
| test: | ||
| - as: e2e | ||
| cli: latest | ||
| commands: | | ||
| export OPENAI_PROVIDER_KEY_PATH=/var/run/openai/token | ||
| export OPENAI_API_KEY=$(cat /var/run/openai/token) | ||
| export WATSONX_PROVIDER_KEY_PATH=/var/run/watsonx/token | ||
|
|
||
| cp config/system_watsonx.yaml config/system.yaml | ||
| tests/scripts/test-cluster-updates.sh --artifact-dir "${ARTIFACT_DIR}" | ||
| credentials: | ||
| - mount_path: /var/run/openai | ||
| name: openai-apitoken | ||
| namespace: test-credentials | ||
| - mount_path: /var/run/watsonx | ||
| name: watsonx-apitoken | ||
| namespace: test-credentials | ||
| dependencies: | ||
| - env: OLS_IMAGE | ||
| name: lightspeed-service-api | ||
| from: src | ||
| resources: | ||
| requests: | ||
| cpu: 100m | ||
| workflow: generic-claim | ||
| - as: cu-eval-vertex-periodics | ||
| cluster_claim: | ||
| architecture: amd64 | ||
| cloud: aws | ||
| labels: | ||
| region: us-east-1 | ||
| owner: obs | ||
| product: ocp | ||
| timeout: 2h0m0s | ||
| version: "4.22" | ||
| cron: 30 16 * * * | ||
| steps: | ||
| allow_skip_on_success: true | ||
| post: | ||
| - ref: gather-must-gather | ||
| test: | ||
| - as: e2e | ||
| cli: latest | ||
| commands: | | ||
| export OPENAI_PROVIDER_KEY_PATH=/var/run/openai/token | ||
| export OPENAI_API_KEY=$(cat /var/run/openai/token) | ||
| export VERTEX_PROVIDER_KEY_PATH=/var/run/vertex/token | ||
|
|
||
| cp config/system_google_vertex.yaml config/system.yaml | ||
| tests/scripts/test-cluster-updates.sh --artifact-dir "${ARTIFACT_DIR}" | ||
| credentials: | ||
| - mount_path: /var/run/openai | ||
| name: openai-apitoken | ||
| namespace: test-credentials | ||
| - mount_path: /var/run/vertex | ||
| name: vertex-apitoken | ||
| namespace: test-credentials | ||
| dependencies: | ||
| - env: OLS_IMAGE | ||
| name: lightspeed-service-api | ||
| from: src | ||
| resources: | ||
| requests: | ||
| cpu: 100m | ||
| workflow: generic-claim | ||
| - as: cu-eval-vertex-anthropic-periodics | ||
| cluster_claim: | ||
| architecture: amd64 | ||
| cloud: aws | ||
| labels: | ||
| region: us-east-1 | ||
| owner: obs | ||
| product: ocp | ||
| timeout: 2h0m0s | ||
| version: "4.22" | ||
| cron: 30 17 * * * | ||
| steps: | ||
| allow_skip_on_success: true | ||
| post: | ||
| - ref: gather-must-gather | ||
| test: | ||
| - as: e2e | ||
| cli: latest | ||
| commands: | | ||
| export OPENAI_PROVIDER_KEY_PATH=/var/run/openai/token | ||
| export OPENAI_API_KEY=$(cat /var/run/openai/token) | ||
| export VERTEX_PROVIDER_KEY_PATH=/var/run/vertex/token | ||
|
|
||
| cp config/system_google_vertex_anthropic.yaml config/system.yaml | ||
| tests/scripts/test-cluster-updates.sh --artifact-dir "${ARTIFACT_DIR}" | ||
| credentials: | ||
| - mount_path: /var/run/openai | ||
| name: openai-apitoken | ||
| namespace: test-credentials | ||
| - mount_path: /var/run/vertex | ||
| name: vertex-apitoken | ||
| namespace: test-credentials | ||
| dependencies: | ||
| - env: OLS_IMAGE | ||
| name: lightspeed-service-api | ||
| from: src | ||
| resources: | ||
| requests: | ||
| cpu: 100m | ||
| workflow: generic-claim |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Regenerate and commit the Prow job definitions.
These four new periodic tests change the generated ProwJob structure, but no corresponding generated job artifacts are included. Run the repository’s required CI-config generation step and commit its output so Prow can schedule these evaluations.
As per coding guidelines, changes under ci-operator/config/ must run make update. Based on learnings, adding named periodic tests is a structural change that requires regenerated downstream Prow definitions.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/config/fao89/cluster-updates-evals/fao89-cluster-updates-evals-main__4.22.yaml`
around lines 25 - 180, Run make update to regenerate the downstream Prow job
definitions for the four new periodic tests—cu-eval-openai-periodics,
cu-eval-watsonx-periodics, cu-eval-vertex-periodics, and
cu-eval-vertex-anthropic-periodics—and include all generated artifacts in the
change.
Sources: Coding guidelines, Learnings
Member
Author
|
/pj-rehearse periodic-ci-fao89-cluster-updates-evals-main-4.22-cu-eval-openai-periodics |
Contributor
|
@fao89: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Contributor
|
@fao89: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add CI configuration for cluster-updates-evals with periodic evaluation jobs for each LLM provider (OpenAI, WatsonX, Google Vertex, Google Vertex Anthropic) running daily against OCP 4.22 clusters.
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED
Summary by CodeRabbit
fao89/cluster-updates-evalstargeting OCP 4.22.