blog/tutorials: Add KAI Scheduler + HAMi core blog and Lab 12 for KAI Scheduler and HAMi memory isolation on GKE - #768
Conversation
Bilingual (en/zh) post on KAI Scheduler v0.17.0 + kai-resource-isolator CUDA-level memory isolation. Signed-off-by: Jimmy Song <jimmy@dynamia.ai>
…on on GKE - Introduced Lab 12: Verify KAI Scheduler and HAMi Memory Isolation on GKE, detailing deployment and memory quota validation. - Updated overview.md to include Lab 12 in the tutorial summary. - Added new YAML files for queue configuration, runtime class, and GKE policies to support the lab. - Created shared Pods and memory limit CUDA example to demonstrate GPU memory isolation. - Enhanced tags.yml with new tags for GKE and GPU sharing. - Updated sidebars to include the new lab for navigation. Signed-off-by: Jimmy Song <jimmy@dynamia.ai>
|
@rootsongjc: The label(s) DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
✅ Deploy Preview for project-hami ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
rootsongjc
added
the
kind/documentation
|
Warning Review limit reached
Next review available in: 76 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughThis PR adds English and Chinese documentation for KAI Scheduler and HAMi-core GPU memory isolation on GKE. It adds Lab 12, Kubernetes resources, Kyverno policies, a CUDA validation program, blog content, author records, tags, and navigation entries. ChangesKAI Scheduler GKE GPU isolation
Estimated code review effort: 4 (Complex) | ~45 minutes Mergeability Score: 🟡 Moderate · up to The PR adds a GKE tutorial and examples for verifying scheduler and GPU memory-isolation behavior, but the documented procedure can run before required admission components are ready and may report success without confirming injection or the expected CUDA out-of-memory result. Merge should wait for these validation safeguards to be added or explicitly accepted by the owner. Sequence Diagram(s)sequenceDiagram
participant KAI Scheduler
participant kai-resource-isolator
participant CUDA Pod
participant HAMi-core
participant Monitor
KAI Scheduler->>kai-resource-isolator: Set GPU memory quota
kai-resource-isolator->>CUDA Pod: Inject preload library and device paths
CUDA Pod->>HAMi-core: Submit CUDA allocation
HAMi-core->>CUDA Pod: Allow or reject allocation
CUDA Pod->>Monitor: Expose memory limit and usage
Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md`:
- Line 17: Correct English hyphenation throughout the article, including the
referenced locations: use “so-called,” “open-source,” “GPU-related,”
“NUMA-aware,” “HAMi-compatible,” and “self-built” wherever applicable.
- Around line 105-118: The diagrams in
blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md:105-118 and
i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md:97-110
incorrectly assign responsibilities: update both diagrams so libsync
provides/distributes libvgpu.so, while the kai-resource-isolator mutating
webhook injects the hostPath and ld.so.preload into the Pod; keep the rest of
the flow unchanged.
- Around line 1-4: Rename the source directory
blog/kai-scheduler-hami-gpu-memory-hard-isolation to
blog/2026-08-11-kai-scheduler-hami-gpu-memory-hard-isolation, and rename the
Chinese mirror directory
i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation
to the same dated directory name so both posts share one dated path.
In
`@i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md`:
- Line 49: 在文档中更新对应表述：将 KAI Scheduler
段落中的“算好的显存配额”替换为“计算出的显存配额”，并将共享库段落中的“被列在里面的共享库”替换为“列在其中的共享库”。
In `@tutorials/labs/kai-scheduler-hami-gke.md`:
- Around line 147-174: Use KAI v0.17.0’s default queue hierarchy: in
tutorials/labs/kai-scheduler-hami-gke.md (147-174) and
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md
(147-174), remove the 01-queues.yaml application and update the documented queue
output to include default-parent-queue and default-queue; remove or otherwise
retire the custom hierarchy in
tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml (1-20).
Preserve --set-string for binder.plugins.gpusharing.arguments.cdiEnabled, while
keeping the separate top-level binder.cdiEnabled field boolean.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 478351b6-9146-4e9c-9b25-eb0a3290403d
📒 Files selected for processing (15)
blog/authors.ymlblog/kai-scheduler-hami-gpu-memory-hard-isolation/index.mdi18n/zh/docusaurus-plugin-content-blog/authors.ymli18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.mdi18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.mdi18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.mdi18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.ymlsidebars-tutorials.jstutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yamltutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yamltutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yamltutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yamltutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cututorials/labs/kai-scheduler-hami-gke.mdtutorials/overview.md
| --- | ||
| title: "GPU Memory Hard Isolation with KAI Scheduler and HAMi: How It Works and How to Verify It" | ||
| date: "2026-08-11" | ||
| description: "From the CUDA interception principle to a reproducible verification on GKE: use KAI Scheduler v0.17.0 for shared GPU scheduling and kai-resource-isolator 1.1.0-chart (built on HAMi-core) for CUDA level memory isolation, so two pods on a single NVIDIA T4 each see only their own memory quota. Explains the CUDA_DEVICE_MEMORY_LIMIT contract between the scheduling layer and the isolation layer, plus the node label, queue, and RuntimeClass pitfalls." |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use one dated directory for the source post and its Chinese mirror.
blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md#L1-L4: rename the source directory to2026-08-11-kai-scheduler-hami-gpu-memory-hard-isolation.i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md#L1-L4: mirror the same dated directory name in the Chinese translation.
📍 Affects 2 files
blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md#L1-L4(this comment)i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md#L1-L4
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md` around lines 1 -
4, Rename the source directory blog/kai-scheduler-hami-gpu-memory-hard-isolation
to blog/2026-08-11-kai-scheduler-hami-gpu-memory-hard-isolation, and rename the
Chinese mirror directory
i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation
to the same dated directory name so both posts share one dated path.
Source: Coding guidelines
…ation Signed-off-by: Jimmy Song <jimmy@dynamia.ai>
…ocumentation Signed-off-by: Jimmy Song <jimmy@dynamia.ai>
…ified outputs and notes - Added a note about the output blocks indicating the verification date and environment-specific details. - Updated GPU node reporting examples with actual node names from the verified run. - Included relevant `nvidia-smi` output for clarity on GPU memory reporting. - Enhanced steps with actual command outputs and statuses from the verified installation. - Clarified the requirements for KAI labels and the expected outputs for various components. - Documented the successful verification of Pods and their resource allocation results. Signed-off-by: Jimmy Song <jimmy@dynamia.ai>
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md`:
- Around line 161-166: Update both handoff-check commands in
blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md (lines 161-166) and
i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md
(lines 161-166) to assert that CUDA_DEVICE_MEMORY_LIMIT is set and
/usr/local/vgpu/libvgpu.so exists before invoking nvidia-smi; keep the existing
output checks intact.
- Line 227: Update
tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu so the second
cudaMalloc succeeds only when it returns cudaErrorMemoryAllocation, rejecting
all other errors. Keep the English result claim in
blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md:227 and the
corresponding Chinese claim in
i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md:227
aligned with this validation outcome; no direct documentation change is needed
unless the claims no longer match.
In
`@i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md`:
- Around line 245-258: Update the commands in
i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md
at lines 245-258 to wait for the kai-resource-isolator webhook Deployment to
become ready before creating validation Pods; after applying the Kyverno
policies, add readiness checks for both ClusterPolicy resources to reach
READY=true at lines 282-285.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: b6f9b3a6-ea90-4917-93a4-c9f3c0a1fda3
📒 Files selected for processing (4)
blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.mdi18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.mdi18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.mdtutorials/labs/kai-scheduler-hami-gke.md
🚧 Files skipped from review as they are similar to previous changes (1)
- tutorials/labs/kai-scheduler-hami-gke.md
…deployment status commands Signed-off-by: Jimmy Song <jimmy@dynamia.ai>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: archlitchi, rootsongjc The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
2 participants
/kind documentation
What this PR does / why we need it:
Add a bilingual blog post on KAI Scheduler and HAMi GPU memory hard isolation, along with a new lab tutorial for verifying these features on GKE. The changes include new YAML configurations, updated documentation, and example code to demonstrate GPU memory isolation. @archlitchi
Checklist:
npm run lintandnpm run format:checkpassnpm run buildsucceeds for bothenandzhgit commit -s)Summary by CodeRabbit