collector: add dmmultipath collector for DM-multipath sysfs metrics by sradco · Pull Request #3581 · prometheus/node_exporter

sradco · 2026-03-12T09:18:43Z

Summary

Add a new disabled-by-default dmmultipath collector that reads
/sys/block/dm-*/dm/ to expose Device Mapper multipath device
health metrics.

Exposed metrics

Metric	Type	Labels	Description
node_dmmultipath_info	Gauge (info)	device, name, uuid, vendor, revision	Device identity
node_dmmultipath_paths	Gauge	device	Total paths
node_dmmultipath_active_paths	Gauge	device	Active paths
node_dmmultipath_path_state	Gauge	device, path, state	Per-path state
node_dmmultipath_device_state	Gauge	device, state	Device state

Enable

--collector.dmmultipath

No special permissions required — reads only world-readable sysfs attributes.

Dependencies

Stacked on build(deps): bump prometheus/procfs to v0.21.1 #3715 (build(deps): bump prometheus/procfs to v0.21.1)
Depends on blockdevice: add DMMultipathDevices for DM-multipath sysfs parsing procfs#796

Test plan

Unit tests with testutil.GatherAndCompare for all metrics
Tests for no-devices case
End-to-end test passing

sradco · 2026-03-12T09:20:13Z

Hi @SuperQ , I created this PR for a new multipath collector.
I would appreciate your review.

jsafrane · 2026-03-13T12:23:30Z

+func normalizeDMPathState(raw string) string {
+	switch raw {
+	case "running", "offline", "blocked", "transport-offline":
+		return raw
+	case "created", "quiesce":
+		return raw
+	default:
+		return "unknown"
+	}
+}


The individual paths of multipath can be different devices with different states. For example,
I tested with NVMeoF in dm-multipath (to test #3579 in a single virtual machine) and NVMe states are quite different. A "live" NVMe device ends up as "unknown" and node_dmmultipath_device_paths_active is then zero.

If only SCSI is being considered for this collector, then it should be reflected in help texts of all metrics or the collector documentation, not only in node_dmmultipath_path_state. I understand that NVMe has its own multipath and typically does not use dm-multipath, still, explicit is better than implicit.

In addition, the Linux kernel can report more states for SCSI. I'd appreciate at least a comment why e.g. "deleted" is reported as "unknown".

Thanks for testing this!
I've updated the collector to pass the raw device state through, with no normalization or fixed enum.
The device_paths_active metric now counts both SCSI running and NVMe live as active paths, so your NVMe test case should report correctly.
The metric help texts have been updated to be device-type agnostic.

The raw value from /sys/block/<dev>/device/state is passed through as-is in path_state, so deleted, blocked, transport-offline, etc. will all appear as reported by the kernel.
I also opened the sysfs parsing as a separate PR in prometheus/procfs (prometheus/procfs#796) as requested by @simonpasquier.

simonpasquier · 2026-03-13T14:58:30Z

+
+// scanDMMultipathDevices discovers DM-multipath devices by scanning
+// /sys/block/dm-* and filtering on dm/uuid prefix "mpath-".
+func scanDMMultipathDevices(sysfsBase string) ([]dmMultipathDevice, error) {


this functionality should probably land into https://github.com/prometheus/procfs first.

Moved to prometheus/procfs#796

sradco · 2026-03-15T15:15:38Z

Results example:

node_dmmultipath_device_active{device="mpatha"} 1
node_dmmultipath_device_info{device="mpatha",sysfs_name="dm-0",uuid="mpath-uuid.b7874a5d-4632-44e5-a330-7045b65964a4"} 1
node_dmmultipath_device_paths_active{device="mpatha"} 2
node_dmmultipath_device_paths_failed{device="mpatha"} 0
node_dmmultipath_device_paths_total{device="mpatha"} 2
node_dmmultipath_device_size_bytes{device="mpatha"} 1.073741824e+09
node_dmmultipath_path_state{device="mpatha",path="nvme0n1",state="live"} 1
node_dmmultipath_path_state{device="mpatha",path="nvme1n1",state="live"} 1

simonpasquier · 2026-03-16T09:00:23Z

node_dmmultipath_device_paths_total => the _total suffix is reserved for counters while this is a gauge.

sradco · 2026-03-16T10:09:25Z

@simonpasquier Thank you for catching this!
I fixed the _total suffix and now its node_dmmultipath_device_paths.
I also fixed node_nvmesubsystem_paths_total in #3579. Updated it to node_nvmesubsystem_paths.

sradco · 2026-03-17T08:51:19Z

PR is failing since its based on prometheus/procfs#796 which is still not merged

sradco · 2026-06-09T11:31:50Z

Hi @jan--f, @SuperQ — the procfs dependency (prometheus/procfs#796) is now merged. This PR is ready for review. Thank you.

sradco · 2026-06-29T09:21:15Z

Hi @SuperQ , Please review this PR, Thank you in advance.

jan--f

The fixture fix might have been better as a separate commit or even a separate PR. Please at least reference Fixes: https://github.com/prometheus/node_exporter/issues/3691 in the PR description.

sradco · 2026-07-06T12:34:00Z

This PR depends on #3715 (build(deps): bump prometheus/procfs to v0.21.1).
CI will fail until #3715 is merged — the collector commit itself is ready for review.

sradco · 2026-07-06T12:47:25Z

Rebased.

SuperQ · 2026-07-06T14:14:01Z

+	deviceInfo        *prometheus.Desc
+	deviceActive      *prometheus.Desc
+	deviceSizeBytes   *prometheus.Desc
+	devicePaths       *prometheus.Desc
+	devicePathsActive *prometheus.Desc
+	devicePathsFailed *prometheus.Desc
+	pathState         *prometheus.Desc


One thing that would help with the auto-docs support is to make these package level vars, rather than create the prometheus.Desc every time NewDMMultipathCollector(). We've been slowly migrating to this pattern across collectors.

See #3575

SuperQ

Besides the package var thing, this looks good.

Add a new collector that reads /sys/block/dm-* to discover Device Mapper multipath devices and expose path health metrics. Enabled by default; the collector is a no-op on systems without multipath. Multipath devices are identified by checking that dm/uuid starts with "mpath-". The path state is reported as-is from /sys/block/<dev>/device/state, supporting both SCSI and NVMe devices without hardcoding a fixed set of states. Exposed metrics: - node_dmmultipath_device_info - node_dmmultipath_device_active - node_dmmultipath_device_size_bytes - node_dmmultipath_device_paths - node_dmmultipath_device_paths_active - node_dmmultipath_device_paths_failed - node_dmmultipath_path_state Signed-off-by: Shirly Radco <sradco@redhat.com> Co-authored-by: AI Assistant <noreply@cursor.com>

sradco · 2026-07-06T14:28:52Z

Besides the package var thing, this looks good.

Thank you @SuperQ , I updated the PR

SuperQ

Thanks!

Propose a new component that maps CSI volumes to their underlying node block devices, exposing the mapping as Prometheus metrics. This enables correlating storage path health (multipath, NVMe-oF) with impacted Kubernetes workloads via PromQL joins. The exporter is a standalone DaemonSet — it does not introduce new gRPC RPCs or CRDs. It reads kubelet metadata and host sysfs. Related upstream work: - prometheus/node_exporter#3581 (dmmultipath collector, merged) - prometheus/node_exporter#3579 (nvmesubsystem collector, merged) - openshift-virtualization/csi-volume-device-exporter#1 (reference impl) Signed-off-by: Shirly Radco <sradco@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

* [FEATURE] mountstats: Add NFS mountpoint info metric #3554 * [FEATURE] nvmesubsystem: Add new collector #3579 * [FEATURE] dmmultipath: Add new collector #3581 * [FEATURE] infiniband: Add device filtering option #3694 * [ENHANCEMENT] hwmon: disambiguate colliding chip labels #3646 * [ENHANCEMENT] cpufreq: Add support for cpuinfo_avg_freq #3712 * [ENHANCEMENT] filesystem: Support ext4 super block emergency_ro flag #3717 * [ENHANCEMENT] edac: Add per-channel error metrics with DIMM labels #3720 * [BUGFIX] diskstats: Only enforce mutual exclusivity for non-empty values #3617 * [BUGFIX] thermal_zone: Handle EINVAL errors gracefully #3657 * [BUGFIX] filesystem: Add comma separator in mount options #3659 * [BUGFIX] diskstats: Fix rotational queue stats scrape regression #3686 * [BUGFIX] filesystem: Prevent panic on mount points with non-UTF-8 names #3687 Signed-off-by: Ben Kochie <superq@gmail.com>

sradco force-pushed the add_collector_dmmultipath branch from 2ec5320 to 4666659 Compare March 12, 2026 09:55

sradco changed the title ~~collector: add dmmultipath collector for multipathd socket metrics~~ ollector: add dmmultipath collector for DM-multipath sysfs metrics Mar 12, 2026

sradco changed the title ~~ollector: add dmmultipath collector for DM-multipath sysfs metrics~~ collector: add dmmultipath collector for DM-multipath sysfs metrics Mar 12, 2026

jsafrane reviewed Mar 13, 2026

View reviewed changes

simonpasquier reviewed Mar 13, 2026

View reviewed changes

sradco force-pushed the add_collector_dmmultipath branch from 4666659 to 812baac Compare March 15, 2026 12:30

sradco mentioned this pull request Mar 15, 2026

blockdevice: add DMMultipathDevices for DM-multipath sysfs parsing prometheus/procfs#796

Merged

sradco force-pushed the add_collector_dmmultipath branch from 812baac to a8bde97 Compare March 16, 2026 10:05

sradco force-pushed the add_collector_dmmultipath branch 4 times, most recently from 66fc703 to 70e0645 Compare March 17, 2026 08:39

sradco force-pushed the add_collector_dmmultipath branch from 70e0645 to 618342b Compare March 23, 2026 13:23

This was referenced May 4, 2026

feat: add csi-volume-device-exporter openshift-virtualization/csi-volume-device-exporter#1

Open

collector: add nvmesubsystem collector for NVMe-oF path health #3579

Merged

sradco force-pushed the add_collector_dmmultipath branch 2 times, most recently from 04dad44 to cbb85f7 Compare June 29, 2026 09:20

sradco force-pushed the add_collector_dmmultipath branch 3 times, most recently from 313d3f5 to cfa742c Compare July 5, 2026 08:37

jan--f reviewed Jul 6, 2026

View reviewed changes

sradco force-pushed the add_collector_dmmultipath branch 2 times, most recently from 5b56619 to a694f19 Compare July 6, 2026 11:53

sradco mentioned this pull request Jul 6, 2026

NVMe namespace parsing broken in procfs 0.20.1 #3691

Closed

sradco force-pushed the add_collector_dmmultipath branch from a694f19 to 7d7d795 Compare July 6, 2026 12:33

sradco force-pushed the add_collector_dmmultipath branch from 7d7d795 to f23e2b3 Compare July 6, 2026 12:47

jan--f approved these changes Jul 6, 2026

View reviewed changes

SuperQ reviewed Jul 6, 2026

View reviewed changes

Comment thread README.md Outdated

SuperQ reviewed Jul 6, 2026

View reviewed changes

Comment thread collector/dmmultipath_linux.go Outdated

sradco force-pushed the add_collector_dmmultipath branch 3 times, most recently from cb3c4e3 to ded7535 Compare July 6, 2026 13:48

SuperQ reviewed Jul 6, 2026

View reviewed changes

sradco force-pushed the add_collector_dmmultipath branch from ded7535 to 32fbfd4 Compare July 6, 2026 14:28

SuperQ approved these changes Jul 6, 2026

View reviewed changes

SuperQ merged commit a498ed4 into prometheus:master Jul 6, 2026
17 checks passed

sradco mentioned this pull request Jul 8, 2026

feat: add csi-volume-device-exporter component csi-addons/kubernetes-csi-addons#1061

Open

5 tasks

SuperQ mentioned this pull request Jul 10, 2026

Release v1.12.0 #3726

Merged

Uh oh!

Conversation

sradco commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Exposed metrics

Enable

Dependencies

Test plan

Uh oh!

sradco commented Mar 12, 2026

Uh oh!

jsafrane Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

jsafrane Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

sradco Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simonpasquier Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

sradco Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sradco commented Mar 15, 2026

Uh oh!

simonpasquier commented Mar 16, 2026

Uh oh!

sradco commented Mar 16, 2026

Uh oh!

sradco commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sradco commented Jun 9, 2026

Uh oh!

sradco commented Jun 29, 2026

Uh oh!

jan--f left a comment

Choose a reason for hiding this comment

Uh oh!

sradco commented Jul 6, 2026

Uh oh!

sradco commented Jul 6, 2026

Uh oh!

Uh oh!

Uh oh!

SuperQ Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

SuperQ left a comment

Choose a reason for hiding this comment

Uh oh!

sradco commented Jul 6, 2026

Uh oh!

SuperQ left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sradco commented Mar 12, 2026 •

edited

Loading

sradco Mar 15, 2026 •

edited

Loading

sradco Mar 15, 2026 •

edited

Loading

sradco commented Mar 17, 2026 •

edited

Loading