DAOS-19288 control: Add allow-numa-imbalance flag to config generate#18621
Draft
tanabarr wants to merge 1 commit into
Draft
DAOS-19288 control: Add allow-numa-imbalance flag to config generate#18621tanabarr wants to merge 1 commit into
tanabarr wants to merge 1 commit into
Conversation
Add --allow-numa-imbalance/-n flag to dmg and daos_server config generate commands. This flag distributes NVMe devices equally across engines regardless of NUMA affinity. Without this flag, config generation respects NUMA affinity and balances device assignment by limiting engines to the minimum SSD count across NUMA nodes. With this flag enabled: - All available SSDs are collected from all NUMA nodes - SSDs are distributed equally across all engines - Distribution ignores NUMA node boundaries - If total SSDs not evenly divisible, uses maximum divisible number - Remainder SSDs are not included in generated config - Notice logged when SSDs unused due to remainder - Generated config includes allow_numa_imbalance: true This maximizes device utilization in heterogeneous environments but may result in suboptimal performance due to cross-NUMA memory access. Example: 6 SSDs across 2 engines - Without flag: 2 per engine (4 used, 2 wasted) - With flag: 3 per engine (6 used, equally distributed) Example: 7 SSDs across 2 engines - With flag: 3 per engine (6 used, 1 unused due to remainder) - Notice: "using 6 SSDs (3 per engine), 1 SSDs will not be used" Required-githooks: true Signed-off-by: Tom Nabarro <tom.nabarro@intel.com>
Collaborator
|
Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18621/1/testReport/ |
|
Ticket title is 'Add config generate flag to ignore NVMe NUMA affinity' |
tanabarr
force-pushed
the
tanabarr/control-confgen-allow-numa-imbalance
branch
from
b5ad1ca to
9f076cd
Compare
Collaborator
|
Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18621/2/testReport/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add --allow-numa-imbalance/-n flag to dmg and daos_server config
generate commands. This flag distributes NVMe devices equally across
engines regardless of NUMA affinity.
Without this flag, config generation respects NUMA affinity and balances
device assignment by giving entrance the lowest common denominator
SSD count across NUMA nodes within the cluster.
With this flag enabled:
- All available SSDs are collected from all NUMA nodes
- SSDs are distributed equally across all engines
- Distribution ignores NUMA node boundaries
- If total SSDs not evenly divisible, uses maximum divisible number
- Remainder SSDs are not included in generated config
- Notice logged when SSDs unused due to remainder
- Generated config incl. allow_numa_imbalance: true to allow boot
This maximizes device utilization in heterogeneous environments but may
result in suboptimal performance due to cross-NUMA memory access.
Features: control
Signed-off-by: Tom Nabarro tom.nabarro@intel.com
Steps for the author:
After all prior steps are complete: