feat(info): report command statistics per namespace by nhancdt2602 · Pull Request #3557 · apache/kvrocks

nhancdt2602 · 2026-07-13T04:18:11Z

Summary

This change tracks command calls, latency, and histograms per namespace. A namespaced connection sees its own numbers in INFO (Stats, CommandStats) and LATENCY HISTOGRAM; the admin/default namespace sees the aggregate across all namespaces.

Behavior:

In namespace ns1, call INFO commandstats -> returns stats scoped by ns1 only
In namespace admin, call INFO commandstats -> returns stats aggregated from all namespaces

Closes #755.

Changes

Per-namespace Stats kept in ns_stats_ field in the Server class (guarded by a shared_mutex), created lazily via GetOrCreateNamespaceStats.
Command hot path records into the connection's cached per-namespace Stats (no lock needed).
GetStatsInfo, GetCommandsStatsInfo, LATENCY HISTOGRAM resolve their source by caller namespace, or AggregateNamespaceStats() for the admin/default view.
Go integration test TestNamespaceStats.

Implementation notes

Stats such as command counts, ops-per-sec, commandstats, hist I/O bytes and keyspace hits/misses stay global for now. They're counted where the namespace isn't in scope (request parsing / storage layer). Open for follow-ups.

Test

Passed regression tests under unit/namespace module
Added and passed tests in unit/info module

AI disclosure

Human: outlined the behaviors, interfaces scaffolding and finalized approaches.
AI: implementation and writing tests

Track command calls, latency, and histograms per namespace and report them in INFO (stats, commandstats) and LATENCY HISTOGRAM scoped to the caller's namespace; the admin/default namespace sees the aggregate across all of them. Closes apache#755 Assisted-by: Claude Code Signed-off-by: nhancdt <nhancu2602@gmail.com>

nhancdt2602 · 2026-07-13T04:22:23Z

@PragmaTwice I'd love to hear your thought about this change. Especially, some stats still stay global in this PR for the sack of simplicity.

nhancdt2602 · 2026-07-14T03:06:12Z

@jihuayu Thanks for running the CI pipeline. The only failing test is:

--- FAIL: TestSlaveLostMaster (10.74s)
    client.go:99: forwarding tcp stream stopped
    client.go:147: forward tcp stream failed, err: read tcp 127.0.0.1:49515->127.0.0.1:49516: read: connection reset by peer
FAIL
exit status 1

This seems unrelated to the PR change. (as it was reported as flaky test in #3515).

Could you kindly re-run the pipeline again?

sonarqubecloud · 2026-07-14T07:12:57Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 50%)
B Maintainability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

💡 Need a hand with PR review? Try Gitar by Sonar!

Copilot

Pull request overview

This PR adds per-namespace command statistics so that INFO (Stats/CommandStats) and LATENCY HISTOGRAM report metrics scoped to the caller’s namespace, while the default/admin namespace view returns an aggregate across all namespaces.

Changes:

Introduces a per-namespace Stats map in Server, with lazy creation and aggregation for the default/admin namespace view.
Records command call/latency stats into a connection-cached per-namespace Stats on the command execution hot path.
Adds a Go integration test validating namespace scoping, aggregation, and namespace deletion behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/gocase/unit/info/info_test.go	Adds `TestNamespaceStats` to validate INFO/LATENCY stats scoping and aggregation semantics.
src/server/server.h	Extends Server APIs to accept namespace for Stats/CommandStats and adds per-namespace Stats storage/aggregation helpers.
src/server/server.cc	Implements per-namespace Stats creation, aggregation, and updates INFO sections to use namespace-aware sources.
src/server/redis_connection.h	Adds cached per-namespace `Stats` pointer on the connection object.
src/server/redis_connection.cc	Switches command stats recording to the connection’s cached per-namespace Stats.
src/commands/cmd_server.cc	Clears namespace stats on `NAMESPACE DEL` and makes `LATENCY HISTOGRAM` use namespace-aware histograms/aggregate.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

nhancdt2602 · 2026-07-14T07:57:56Z

+  std::unique_lock<std::shared_mutex> lock(ns_stats_mu_);
+  if (auto it = ns_stats_.find(ns); it != ns_stats_.end()) {
+    return it->second;
+  }
+  auto ns_stats = std::make_shared<Stats>(config_->histogram_bucket_boundaries);
+  initCommandStats(ns_stats.get());
+  ns_stats_[ns] = ns_stats;
+  return ns_stats;


histogram_bucket_boundaries is a read-only config, so it can't be changed at runtime.

nhancdt2602 · 2026-07-14T07:58:13Z

+    for (const auto &[cmd, hist] : ns_stats->commands_histogram) {
+      auto &agg_hist = agg->commands_histogram[cmd];
+      agg_hist.calls.fetch_add(hist.calls.load(), std::memory_order_relaxed);
+      agg_hist.sum.fetch_add(hist.sum.load(), std::memory_order_relaxed);
+      for (std::size_t i = 0; i < hist.buckets.size(); ++i) {
+        agg_hist.buckets[i]->fetch_add(hist.buckets[i]->load(), std::memory_order_relaxed);
+      }
+    }
+  }


histogram_bucket_boundaries is a read-only config, so it can't be changed at runtime.

nhancdt2602 · 2026-07-14T07:59:27Z

 void Server::recordInstantaneousMetrics() {
  auto rocksdb_stats = storage->GetDB()->GetDBOptions().statistics;
-  stats.TrackInstantaneousMetric(STATS_METRIC_COMMAND, stats.total_calls);
+  // Sample each namespace's command metric, and feed the sum into the global metric so the
+  // admin/default view reports aggregate ops/sec without keeping a global command counter on the hot path.
+  uint64_t total_calls = 0;
+  {
+    std::shared_lock<std::shared_mutex> lock(ns_stats_mu_);
+    for (const auto &[ns, ns_stats] : ns_stats_) {
+      auto calls = ns_stats->total_calls.load();
+      ns_stats->TrackInstantaneousMetric(STATS_METRIC_COMMAND, calls);
+      total_calls += calls;
+    }
+  }
+  stats.TrackInstantaneousMetric(STATS_METRIC_COMMAND, total_calls);


As this method is not called too frequently, I believe the overhead is acceptable.

nhancdt2602 · 2026-07-14T08:00:52Z

+Server::InfoEntries Server::GetStatsInfo(const std::string &ns) {
+  // Command stats are per namespace; the admin/default namespace sees the aggregate across all of them.
+  auto cmd_stats_ptr = ns == kDefaultNamespace ? AggregateNamespaceStats() : GetOrCreateNamespaceStats(ns);
+  const Stats &cmd_stats = *cmd_stats_ptr;
+
  Server::InfoEntries entries;
  entries.emplace_back("total_connections_received", total_clients_.load());
-  entries.emplace_back("total_commands_processed", stats.total_calls.load());
-  entries.emplace_back("instantaneous_ops_per_sec", stats.GetInstantaneousMetric(STATS_METRIC_COMMAND));
+  entries.emplace_back("total_commands_processed", cmd_stats.total_calls.load());
+  // Per-namespace ops/sec comes from the namespace's own sampled metric; the admin/default view uses
+  // the global metric, which the sampler feeds with the sum across all namespaces.
+  auto ops_per_sec = ns == kDefaultNamespace ? stats.GetInstantaneousMetric(STATS_METRIC_COMMAND)
+                                             : cmd_stats.GetInstantaneousMetric(STATS_METRIC_COMMAND);
+  entries.emplace_back("instantaneous_ops_per_sec", ops_per_sec);


As the aggregation logic is fast, I believe we can ignore the concern here.

jihuayu

Hello @nhancdt2602.Thank you for your contribution. I think this PR still has a few issues.

latency command is admin only. It will make the namespace user could not get it's latency.

kvrocks/src/commands/cmd_server.cc

Line 1827 in 634ed15

MakeCmdAttr<CommandLatency>("latency", -2, "read-only admin", NO_KEY), )
Deleting a namespace does not disconnect existing connections. This causes the old connections’ ns_stats_ references to diverge.

After authentication:
ns_stats_["ns1"] -> Stats A
Active connection -> Stats A

After deleting `ns1`:
ns_stats_ no longer contains ns1
Active connection -> Stats A  // kept alive by shared_ptr

After the old connection executes `INFO`:
ns_stats_["ns1"] -> newly created Stats B
Active connection -> Stats A

Deleting a namespace may cause an underflow here.

kvrocks/src/stats/stats.cc

Lines 107 to 117 in 8c26549

    
           void Stats::TrackInstantaneousMetric(int metric, uint64_t current_reading) { 
        
             uint64_t curr_time_ms = util::GetTimeStampMS(); 
        
             std::unique_lock<std::shared_mutex> lock(inst_metrics_mutex); 
        
             uint64_t t = curr_time_ms - inst_metrics[metric].last_sample_time_ms; 
        
             uint64_t ops = current_reading - inst_metrics[metric].last_sample_count; 
        
             uint64_t ops_sec = t > 0 ? (ops * 1000 / t) : 0; 
        
             inst_metrics[metric].samples[inst_metrics[metric].idx] = ops_sec; 
        
             inst_metrics[metric].idx++; 
        
             inst_metrics[metric].idx %= STATS_METRIC_SAMPLES; 
        
             inst_metrics[metric].last_sample_time_ms = curr_time_ms; 
        
             inst_metrics[metric].last_sample_count = current_reading;

nhancdt2602 · 2026-07-14T08:30:43Z

@jihuayu Thanks for your reply. To fix points 2 and 3, can I keep the stats entry in the namespace map even after the namespace is deleted? This would be consistent with the current behavior of db_scan_infos_

nhancdt2602 · 2026-07-14T09:08:14Z

@jihuayu Regarding point 1, I’m still not sure I understand your concern. Are you suggesting that we should not expose latency and histogram metrics for namespace-scoped connections?

jihuayu · 2026-07-16T00:34:12Z

@nhancdt2602 Sorry, I’m late.

@jihuayu Thanks for your reply. To fix points 2 and 3, can I keep the stats entry in the namespace map even after the namespace is deleted? This would be consistent with the current behavior of db_scan_infos_

I have no objection to that, since deleting a namespace is a low-frequency operation.

What I meant is that this command should not be restricted to ADMIN users. Namespace users are not ADMIN users, so they would be unable to retrieve latency information.

PragmaTwice reviewed Jul 13, 2026

View reviewed changes

Comment thread src/server/server.h

jihuayu requested a review from Copilot July 14, 2026 07:30

Copilot started reviewing on behalf of jihuayu July 14, 2026 07:30 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

jihuayu reviewed Jul 14, 2026

View reviewed changes

	void Stats::TrackInstantaneousMetric(int metric, uint64_t current_reading) {
	uint64_t curr_time_ms = util::GetTimeStampMS();
	std::unique_lock<std::shared_mutex> lock(inst_metrics_mutex);
	uint64_t t = curr_time_ms - inst_metrics[metric].last_sample_time_ms;
	uint64_t ops = current_reading - inst_metrics[metric].last_sample_count;
	uint64_t ops_sec = t > 0 ? (ops * 1000 / t) : 0;
	inst_metrics[metric].samples[inst_metrics[metric].idx] = ops_sec;
	inst_metrics[metric].idx++;
	inst_metrics[metric].idx %= STATS_METRIC_SAMPLES;
	inst_metrics[metric].last_sample_time_ms = curr_time_ms;
	inst_metrics[metric].last_sample_count = current_reading;

Uh oh!

Conversation

nhancdt2602 commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Implementation notes

Test

AI disclosure

Uh oh!

nhancdt2602 commented Jul 13, 2026

Uh oh!

Uh oh!

nhancdt2602 commented Jul 14, 2026

Uh oh!

sonarqubecloud Bot commented Jul 14, 2026

Quality Gate failed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

nhancdt2602 Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

nhancdt2602 Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

nhancdt2602 Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

nhancdt2602 Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

jihuayu left a comment

Choose a reason for hiding this comment

Uh oh!

nhancdt2602 commented Jul 14, 2026

Uh oh!

nhancdt2602 commented Jul 14, 2026

Uh oh!

jihuayu commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nhancdt2602 commented Jul 13, 2026 •

edited

Loading