docs(tutorials): add KAI Scheduler + HAMi-core lab on nvml-mock - #722
Open
maishivamhoo123 wants to merge 3 commits into
Open
docs(tutorials): add KAI Scheduler + HAMi-core lab on nvml-mock#722maishivamhoo123 wants to merge 3 commits into
maishivamhoo123 wants to merge 3 commits into
Conversation
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: maishivamhoo123 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
✅ Deploy Preview for project-hami ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
📝 WalkthroughWalkthroughAdds an advanced tutorial for installing and validating KAI Scheduler, HAMi-core, and kai-resource-isolator on an nvml-mock-backed kind cluster. It covers whole-GPU scheduling, shared-GPU injection, NVML limitations, cleanup, follow-up links, and sidebar registration. ChangesKAI Scheduler and HAMi lab
Estimated code review effort: 2 (Simple) | ~10 minutes Sequence Diagram(s)sequenceDiagram
participant kind
participant nvmlMock
participant devicePlugin
participant KAIScheduler
participant kaiResourceIsolator
participant TestPod
kind->>nvmlMock: Deploy simulated GPU support
nvmlMock->>devicePlugin: Provide GPU discovery
devicePlugin->>KAIScheduler: Publish allocatable GPU resources
TestPod->>KAIScheduler: Request whole GPU or gpu-memory
KAIScheduler->>TestPod: Bind Pod to a GPU allocation
kaiResourceIsolator->>TestPod: Inject HAMi-core configuration
TestPod-->>TestPod: Report Running or NVML library failure
Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
coderabbitai
Bot
added
the
kind/documentation
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (2)
tutorials/labs/kai-scheduler-hami.md (2)
49-60: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPin the Kubernetes version used by the lab.
The prerequisites constrain
kindandkubectl, butkind create clusterleaves the Kubernetes node image implicit. Different installations can produce different cluster versions and captured outputs. Add a testedkindest/nodeimage or document the exact Kubernetes version.
tutorials/labs/kai-scheduler-hami.md#L49-L60: pin the English lab cluster version.i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md#L50-L61: apply the same pin in the Chinese lab.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tutorials/labs/kai-scheduler-hami.md` around lines 49 - 60, Pin the kind Kubernetes node image in the cluster creation command to a tested kindest/node version, replacing the implicit default. Apply the same pin to the corresponding command in tutorials/labs/kai-scheduler-hami.md (lines 49-60) and i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md (lines 50-61); keep the existing NODE_NAME assignment unchanged.
128-149: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winDo not mutate or delete KAI's generated default queues.
KAI v0.17.0 creates
default-parent-queueanddefault-queueduring installation. This block creates another root, reparents the generated leaf, and later deletesdefault-queue. Use the generated queues, or create unique lab queues and clean up only those resources. (github.com)
tutorials/labs/kai-scheduler-hami.md#L128-L149: remove the manual default-queue mutation or use unique names.tutorials/labs/kai-scheduler-hami.md#L320-L320: do not delete KAI's generated queue in cleanup.i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md#L129-L150: apply the same queue change.i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md#L321-L321: do not delete KAI's generated queue in cleanup.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tutorials/labs/kai-scheduler-hami.md` around lines 128 - 149, Do not mutate or delete KAI-generated queues: in tutorials/labs/kai-scheduler-hami.md lines 128-149 and i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md lines 129-150, remove the manual default-queue setup or replace it with uniquely named lab queues; in both files’ cleanup lines 320 and 321, delete only resources created by the lab, not KAI’s generated default-queue.Source: MCP tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tutorials/labs/kai-scheduler-hami.md`:
- Around line 298-303: The reservation Pod lookup is arbitrary and may select no
Pod or the wrong workload. In tutorials/labs/kai-scheduler-hami.md:298-303, wait
for the reservation Pod and select the Pod associated with
gpu-sharing-with-isolation using its owner, label, or exact BindingError name
before running kubectl logs; apply the same correlated selection in
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md:299-304.
- Line 292: Correct the GPU fraction arithmetic in both tutorials: update
tutorials/labs/kai-scheduler-hami.md lines 292-292 and
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md
lines 293-293 so 20480 / 40960 is stated as 0.50; if retaining the reported 1.51
total, explain it through scheduler rounding or base-memory calculation rather
than the incorrect equation.
- Around line 97-100: Update the prerequisite lists in
tutorials/labs/kai-scheduler-hami.md at lines 49-52 and 97-100, and in
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md
at lines 50-53 and 98-101, to include NVCR/NGC access or document a mirrored
image for the NVIDIA device-plugin image used by the kubectl apply step. Keep
the English and Chinese prerequisite guidance consistent.
- Around line 121-153: Wait for KAI admission/webhook readiness after the Helm
install and before applying Queue resources in both
tutorials/labs/kai-scheduler-hami.md (lines 121-153) and
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md
(lines 122-154). Add a readiness wait or retry loop that handles certificate and
webhook startup delays, while preserving the existing Queue manifests and
subsequent kubectl checks.
---
Nitpick comments:
In `@tutorials/labs/kai-scheduler-hami.md`:
- Around line 49-60: Pin the kind Kubernetes node image in the cluster creation
command to a tested kindest/node version, replacing the implicit default. Apply
the same pin to the corresponding command in
tutorials/labs/kai-scheduler-hami.md (lines 49-60) and
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md
(lines 50-61); keep the existing NODE_NAME assignment unchanged.
- Around line 128-149: Do not mutate or delete KAI-generated queues: in
tutorials/labs/kai-scheduler-hami.md lines 128-149 and
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md
lines 129-150, remove the manual default-queue setup or replace it with uniquely
named lab queues; in both files’ cleanup lines 320 and 321, delete only
resources created by the lab, not KAI’s generated default-queue.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 2455f650-913e-4d91-afae-78f83f06b565
📒 Files selected for processing (3)
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.mdsidebars-tutorials.jstutorials/labs/kai-scheduler-hami.md
Signed-off-by: maishivamhoo123 <maishivamhoo@gmail.com>
maishivamhoo123
force-pushed
the
docs/kai-scheduler-hami-lab
branch
from
August 5, 2026 04:01
badda80 to
05fc746
Compare
Signed-off-by: maishivamhoo123 <maishivamhoo@gmail.com>
Contributor
|
Reproduced end-to-end on kind (KAI v0.17.0, isolator 1.1.0-chart) all 8 steps match. Two optional nits:
|
Signed-off-by: maishivamhoo123 <maishivamhoo@gmail.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
tutorials/labs/kai-scheduler-hami.md (1)
74-81: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winPin the external mock artifacts in both tutorials.
The verified output depends on a repository branch, an unversioned chart, and a mutable raw manifest. Pin every input used by the lab.
tutorials/labs/kai-scheduler-hami.md#L74-L81: pin thek8s-test-infracheckout andnvml-mockchart.tutorials/labs/kai-scheduler-hami.md#L99-L101: reference the manifest at the verified commit.i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md#L74-L81: apply the same checkout and chart pins.i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md#L99-L101: reference the same verified manifest commit.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tutorials/labs/kai-scheduler-hami.md` around lines 74 - 81, Pin all external mock artifacts used by the tutorials: in tutorials/labs/kai-scheduler-hami.md lines 74-81 and i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md lines 74-81, check out the verified k8s-test-infra commit and use the corresponding pinned nvml-mock chart version; in both files’ lines 99-101, replace the mutable manifest reference with the manifest at that same verified commit.Source: MCP tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tutorials/labs/kai-scheduler-hami.md`:
- Around line 133-154: Adopt one KAI queue ownership model across all affected
sites: in tutorials/labs/kai-scheduler-hami.md lines 133-154, remove the
manually defined default Queue (or disable chart creation via
defaultQueue.createDefaultQueue=false), then update lines 160-176 to show the
resulting queue hierarchy. Apply the same manifest and output changes in the
Chinese translation at lines 133-154 and 160-176, keeping the English and
Chinese descriptions consistent with the retained model.
- Line 180: Update the English explanation at
tutorials/labs/kai-scheduler-hami.md:180 and the Chinese explanation at
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md:180
to separate the KAI Scheduler admission configuration’s
admission.gpuPodRuntimeClassName responsibility from the isolator webhook.
Attribute HAMi-core mounts and /etc/ld.so.preload injection to the isolator, and
explain that runtimeClassName: nvidia is required only for shared/GPU resource
Pods via the shim RuntimeClass, not injected by the HAMi-core webhook.
---
Nitpick comments:
In `@tutorials/labs/kai-scheduler-hami.md`:
- Around line 74-81: Pin all external mock artifacts used by the tutorials: in
tutorials/labs/kai-scheduler-hami.md lines 74-81 and
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.md
lines 74-81, check out the verified k8s-test-infra commit and use the
corresponding pinned nvml-mock chart version; in both files’ lines 99-101,
replace the mutable manifest reference with the manifest at that same verified
commit.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: fb92386b-6102-48db-a75f-0ea68c42f16e
📒 Files selected for processing (2)
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami.mdtutorials/labs/kai-scheduler-hami.md
Member
Author
@rootsongjc i made all the suggested changes can you please look into this once. |
Contributor
|
@maishivamhoo123 This experiment is valid as a control plane and failure boundary experiment in an nvml-mock environment. However, the current title and the conclusions of steps 6 and 7 describe resource scheduling, queue accounting, and PodSpec injection as a complete GPU path and HAMi-core runtime verification. It is recommended to retain the expected failure design in step 8, while narrowing the experimental objective, modifying the verification statement, and adding checks for CUDA_DEVICE_MEMORY_LIMIT and the final Pod state. |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a new tutorial lab (Lab 11) that shows how to run NVIDIA KAI Scheduler with HAMi-core memory isolation on a local kind cluster using nvml-mock, so people can try the integration without a real GPU. It builds on the existing nvml-mock lab and walks through installing the NVIDIA device-plugin, KAI Scheduler with the hamicore plugin, queues, and kai-resource-isolator, then verifies whole-GPU scheduling, fractional GPU-memory accounting, and HAMi-core's ld.so.preload injection. It also contains where a fake GPU stops Every command was run and every output is a real capture (KAI v0.17.0, kai-resource-isolator 1.1.0-chart). Includes English and Chinese versions plus the sidebar and overview entries. This docs contain the whole logs which i got during my testing. https://docs.google.com/document/d/1IxAtURfCziGarAyD2avN6b5V3BqpdmwJUTUSBy2Tvxs/edit?usp=sharing
fix :- #721
Summary by CodeRabbit
Summary by CodeRabbit