Add Cray/HPE pm_counters backend with cross-vendor GPU correlation hints - #9
Draft
vancraar wants to merge 7 commits into
Draft
Add Cray/HPE pm_counters backend with cross-vendor GPU correlation hints#9vancraar wants to merge 7 commits into
vancraar wants to merge 7 commits into
Conversation
Reads whole-node power/energy from Cray/HPE's /sys/cray/pm_counters sysfs interface, to validate hws's software energy measurements against measured hardware ground truth on HLRS's Hunter system. The counter set is discovered and classified at runtime instead of hardcoded, since the exposed files differ across PM counter versions and node generations; confirmed against a real Hunter mi300a node (23 files, PM counters v3). Includes CTest-based regression tests using fixtures from that real dump.
Generalizes the pm_counters accel[i] index guess from AMD-only to also cover NVIDIA, since pm_counters exposes the same accel[i] naming convention on Grace-Hopper (EX254n) nodes. Extracts the vendor-specific YAML formatting into a shared, vendor-agnostic hws::detail::accel_correlation_yaml_block() so AMD and NVIDIA share one code path in system_hardware_sampler's device_correlation_hints. Both gpu_amd_hardware_sampler and gpu_nvidia_hardware_sampler gained a pci_bus_id() that queries the same API/handle each sampler already uses for its actual measurements (ROCm SMI's rsmi_dev_pci_id_get(device_id_), NVML's nvmlDeviceGetPciInfo_v3() on the already-resolved handle) instead of a different API family (HIP/CUDA's own PCI bus ID calls). Those differ from ROCm SMI's/NVML's device enumeration under HIP_VISIBLE_DEVICES/CUDA_VISIBLE_DEVICES, which could otherwise silently attribute a correlation guess to the wrong physical device.
dump_yaml()/dump_yaml_global() wrote each sampler's own YAML document directly instead of going through as_yaml_string(), so device_correlation_hints_as_yaml_string() never made it into a dumped YAML file even though as_yaml_string() included it correctly. Found by running the actual example program on a real Hunter mi300a node - the correlation section was silently missing from track.yaml. dump_yaml(std::string)/dump_yaml(path) now forward to dump_yaml(const char*) instead of duplicating the sampler loop.
…vals The CPU turbostat backend could deadlock: run_subprocess() never combined the child's stdout/stderr, so an unread stderr pipe could fill up and block the child forever, wedging subprocess_join() (reproduced via gdb/wchan on two architectures). Fixed by passing subprocess_option_combined_stdout_stderr. That fix has a side effect: turbostat's own diagnostic messages (e.g. "Disabling Low Power Idle CPU output") now land in the same stream hws parses, which used to assume the header/value pair were always the first two lines. Parsing now takes the *last* two lines instead, since turbostat always prints diagnostics before its one-shot table, never after. Also fixed: turbostat's -i interval was hardcoded to 0.001s (1ms), which empirically (four real machines, full-core load) causes physically implausible readings (multi-kW power, multi-GHz clocks) because turbostat's own internal timing calculation degrades under load at short intervals. This is now the configurable HWS_TURBOSTAT_INTERVAL CMake option; the default stays at 0.001 for backwards compatibility with existing sudoers allowlists and paper-reproducibility setups, but a longer interval (e.g. 1s) is documented as the reliable choice. Related changes: - warn at runtime if sampling_interval is shorter than HWS_TURBOSTAT_INTERVAL - sleep_until() instead of sleep_for() so a slow turbostat call doesn't also get a full extra sampling_interval tacked on afterward - only invoke turbostat if an enabled sample_category actually needs it
gpu_amd_hardware_sampler used the HIP-relative device index directly as the ROCm SMI index for every rsmi_dev_* call. ROCm SMI's own device enumeration isn't filtered by HIP_VISIBLE_DEVICES/ROCR_VISIBLE_DEVICES the way HIP's is, so under a non-default visibility mask the sampler measured the wrong physical GPU. Fix: resolve the actual ROCm SMI index once at construction via a PCI-bus-ID match against every ROCm SMI device, throwing if no match is found instead of silently falling back to the HIP-relative index. The original HIP-relative index is kept in a new hip_device_id_ member for the one HIP call that still needs HIP-space, and for system_hardware_sampler's device_correlation_hints output. Also adds the missing find_dependency(MPI) to hwsConfig.cmake.in, alongside the existing ones for CUDA/HIP/ROCm-SMI/Level-Zero.
Mirrors the existing per-backend binding pattern: one .cpp file (bindings/cray_pm_counters_hardware_sampler.cpp) binding CrayPmCountersGeneralSamples, CrayPmCountersPowerSamples and CrayPmCountersHardwareSampler (constructors, general_samples, power_samples, discovered_accel_indices, samples_only_as_yaml_string, __repr__), gated behind HWS_FOR_CRAY_PM_COUNTERS_ENABLED like every other backend's bindings. device_identification() stays unbound, consistent with every other backend. Built and tested locally (non-MPI build) and on a real Hunter mi300a node via a PBS job: SystemHardwareSampler auto-discovery, direct CrayPmCountersHardwareSampler construction, and a dump_yaml roundtrip including the device_correlation_hints section all work correctly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cray/HPE pm_counters backend
New
cray_pm_countersbackend reading whole-node and per-acceleratorpower/energy from Cray/HPE's
/sys/cray/pm_counterssysfs interface, tovalidate hws's software-side energy measurements against measured
hardware ground truth on HLRS's Hunter system (HPE Cray EX, AMD MI300A
APU nodes). The counter set is discovered and classified at runtime
instead of hardcoded, since the exposed files differ across PM counter
versions and node generations. Confirmed against a real Hunter mi300a
node (23 files, PM counters v3). Includes CTest-based regression tests
using fixtures from that real dump.
Cross-vendor accel[i] <-> GPU correlation hints
pm_counters exposes per-accelerator counters as
accel[i]_energy/accel[i]_power, but doesn't say which physical GPU (as seen byHIP/CUDA) each index corresponds to. Adds a best-effort correlation
guess, generalized from AMD-only to also cover NVIDIA (pm_counters uses
the same accel[i] naming convention on Grace-Hopper/EX254n nodes):
gpu_amd_hardware_sampler/gpu_nvidia_hardware_samplereach gained apci_bus_id()querying the same handle already used for their actualmeasurements, and a shared, vendor-agnostic
accel_correlation_yaml_block()builds the resulting YAML hint blockthat
system_hardware_samplerappends todump_yaml/dump_yaml_global.Python bindings for cray_pm_counters
Mirrors the existing per-backend binding pattern (one
.cppfile,HWS_FOR_CRAY_PM_COUNTERS_ENABLED-gated). Built and tested locally andon a real Hunter mi300a node via a PBS job, including a
dump_yamlroundtrip with the correlation-hints section.
Known limitations
pci_bus_id(),enumerate_all_nvidia_gpu_pci_bus_ids()) currently sits on top of thesame NVML device-index-resolution bug fixed in Fix GPU device index resolution and init/shutdown lifecycle races #8
(fix/gpu-device-index-resolution) - this branch still has its own,
earlier, entangled copy of the AMD half of that fix (commit
a029bd0)predating the standalone, more thoroughly reviewed fix in Fix GPU device index resolution and init/shutdown lifecycle races #8. Once Fix GPU device index resolution and init/shutdown lifecycle races #8
merges, this branch needs a rebase onto the new
developto drop theduplicate and pick up the reviewed NVIDIA/lifecycle fixes cleanly.
disambiguated by the correlation guess (same limitation as Fix GPU device index resolution and init/shutdown lifecycle races #8).
Also included
A pre-existing, unrelated fix already merged onto this branch: turbostat
subprocess deadlock and unreliable short sampling intervals in the CPU
backend (commit
1884d03).