Skip to content

collector: add nvmesubsystem collector for NVMe-oF path health#3579

Merged
SuperQ merged 1 commit into
prometheus:masterfrom
sradco:add_collector_multipath
Jul 6, 2026
Merged

collector: add nvmesubsystem collector for NVMe-oF path health#3579
SuperQ merged 1 commit into
prometheus:masterfrom
sradco:add_collector_multipath

Conversation

@sradco

@sradco sradco commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a new disabled-by-default nvmesubsystem collector that reads
/sys/class/nvme-subsystem/ to expose NVMe over Fabrics subsystem
path health metrics.

Exposed metrics

Metric Type Labels Description
node_nvmesubsystem_info Gauge (info) subsystem, nqn, model, serial, iopolicy Subsystem identity
node_nvmesubsystem_namespace_info Gauge (info) subsystem, device Maps namespace block device to subsystem
node_nvmesubsystem_paths Gauge subsystem Total controller paths
node_nvmesubsystem_paths_live Gauge subsystem Controller paths in live state
node_nvmesubsystem_path_state Gauge subsystem, controller, transport, state Per-controller state

The namespace_info metric enables precise correlation between NVMe
block devices (e.g. nvme0n1) and their parent subsystems, which is
needed for workload-aware storage path health alerting in Kubernetes
environments.

Enable

--collector.nvmesubsystem

No special permissions required — reads only world-readable sysfs attributes.

Dependencies

Test plan

  • Unit tests with testutil.GatherAndCompare for all metrics
  • Tests for no-devices case
  • State normalization tests (normalizeControllerState)
  • End-to-end test passing

@sradco sradco force-pushed the add_collector_multipath branch from 742c1b1 to a0a146e Compare March 11, 2026 19:18
@sradco

sradco commented Mar 11, 2026

Copy link
Copy Markdown
Contributor Author

Hi @SuperQ , I created this PR for a new multipath collector.
I would appreciate your review.

@sradco sradco force-pushed the add_collector_multipath branch from a0a146e to 1fa2099 Compare March 12, 2026 08:55
@sradco sradco changed the title Add multipath collector Add multipath collector for NVMe-oF subsystem path health Mar 12, 2026
@sradco sradco force-pushed the add_collector_multipath branch from 1fa2099 to 635b613 Compare March 12, 2026 09:13
@sradco sradco changed the title Add multipath collector for NVMe-oF subsystem path health collector: add nvmesubsystem collector for NVMe-oF path health Mar 12, 2026

@jsafrane jsafrane left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good to me. I tested it with 2 NVMe over TCP devices, I got:

node_nvmesubsystem_info{iopolicy="numa",model="Linux",nqn="tempdisk",serial="bab529a0f32e397e1319",subsystem="nvme-subsys0"} 1
node_nvmesubsystem_path_state{controller="nvme0",state="connecting",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme0",state="dead",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme0",state="live",subsystem="nvme-subsys0",transport="tcp"} 1
node_nvmesubsystem_path_state{controller="nvme0",state="resetting",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme0",state="unknown",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme1",state="connecting",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme1",state="dead",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme1",state="live",subsystem="nvme-subsys0",transport="tcp"} 1
node_nvmesubsystem_path_state{controller="nvme1",state="resetting",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_path_state{controller="nvme1",state="unknown",subsystem="nvme-subsys0",transport="tcp"} 0
node_nvmesubsystem_paths_live{subsystem="nvme-subsys0"} 2
node_nvmesubsystem_paths_total{subsystem="nvme-subsys0"} 2

Which looks reasonable.

Comment thread collector/nvmesubsystem_linux.go Outdated
@sradco sradco force-pushed the add_collector_multipath branch 3 times, most recently from 51fb644 to 8acec17 Compare March 16, 2026 10:15
@sradco sradco force-pushed the add_collector_multipath branch 2 times, most recently from 17875ea to d7d2b27 Compare March 16, 2026 17:33
@sradco sradco closed this Mar 17, 2026
@sradco sradco force-pushed the add_collector_multipath branch from d7d2b27 to 1a4cac6 Compare March 17, 2026 08:42
@sradco sradco reopened this Mar 17, 2026
@sradco

sradco commented Mar 17, 2026

Copy link
Copy Markdown
Contributor Author

PR is failing since its based on prometheus/procfs#797, which is still not merged

@sradco

sradco commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

Hi @SuperQ , Please review this PR, Thank you in advance.

@sradco sradco force-pushed the add_collector_multipath branch 4 times, most recently from 1177a63 to 6bb221b Compare July 2, 2026 08:57
Comment thread collector/fixtures/sys.ttar
Comment thread collector/mdadm_linux.go
Comment thread collector/nvmesubsystem_linux.go Outdated
@sradco sradco force-pushed the add_collector_multipath branch from 6bb221b to c7c93c6 Compare July 2, 2026 12:12
@sradco sradco force-pushed the add_collector_multipath branch 4 times, most recently from 1046825 to 1d29047 Compare July 5, 2026 08:30
@sradco sradco requested a review from jan--f July 5, 2026 08:49
Comment thread go.mod Outdated
@sradco sradco force-pushed the add_collector_multipath branch from 1d29047 to 26c3987 Compare July 6, 2026 11:53
@sradco sradco force-pushed the add_collector_multipath branch from 26c3987 to ebf4bd2 Compare July 6, 2026 12:33
@sradco

sradco commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

This PR depends on #3715 (build(deps): bump prometheus/procfs to v0.21.1).
CI will fail until #3715 is merged — the collector commit itself is ready for review.

@sradco sradco force-pushed the add_collector_multipath branch from ebf4bd2 to 97c3ee9 Compare July 6, 2026 12:47
@sradco

sradco commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Rebased

@sradco sradco force-pushed the add_collector_multipath branch 2 times, most recently from e0b6d5a to 30d6ff4 Compare July 6, 2026 13:48
Add a new opt-in collector that exposes NVMe-oF
subsystem path health via sysfs:

- node_nvmesubsystem_info (subsystem metadata)
- node_nvmesubsystem_namespace_info
- node_nvmesubsystem_path_state (per-state gauge)
- node_nvmesubsystem_paths (total controller count)
- node_nvmesubsystem_paths_live (live controllers)

Controller states are normalised through a single
map so adding a new kernel state requires one edit.

Signed-off-by: Shirly Radco <sradco@redhat.com>
Co-authored-by: AI Assistant <noreply@cursor.com>
@sradco sradco force-pushed the add_collector_multipath branch from 30d6ff4 to bba83bf Compare July 6, 2026 14:28

@SuperQ SuperQ left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@SuperQ SuperQ merged commit 1695060 into prometheus:master Jul 6, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants