[DOC-14129]: Magma Compaction rate limiter documentation

[RayOffiah]

Added page detailing REST call for Magma limit rate for compaction.

preview

Preview URL:
https://preview.docs-test.couchbase.com/docs-server-DOC-14129--8.1--Magma-Compaction-rate-limiter-documentation/server/current/rest-api/rest-magma-compaction-rate-limiting.html

You will need the Docs Team credentials on Confluence.

[Copilot]

Pull request overview

Adds REST API documentation for configuring Magma compaction rate limiting and exposes the page through the REST API navigation and memory/storage API index.

Changes:

• Added a new Magma Compaction Rate Limiting REST API page with settings, endpoints, and examples.
• Added the new page to the Memory and Storage API navigation.
• Added GET/POST endpoint entries to the memory and storage REST API table.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
modules/ROOT/nav.adoc	Adds the new REST API page to the Memory and Storage API nav.
modules/rest-api/partials/rest-memory-and-storage-table.adoc	Lists the new Magma compaction rate limiting endpoint entries.
modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc	Documents the Magma compaction rate limiting settings and examples.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[ggray-cb]

Have one question/suggestion on adding info about when these limits should be enabled. Otherwise, looks good.

[anuthan]

@t3rm1n4l Could you help with when the users might want to use this setting, and when they shouldn't?

[t3rm1n4l]

I would recommend removing the magma_enable_compaction_dataonly_ratelimiting setting from documentation. It's confusing to the customer and it's an advanced setting that we are still testing. We can also remove the detailed description about compaction types.

Some additional info and rephrase suggestions..

---

Description

Use these API calls to limit the disk I/O bandwidth (in bytes per second) that Magma Storage Engine Compaction may consume. The limit is global to the Data Service and shared across all buckets on the node

Compaction is a background maintenance activity that reclaims space from stale and deleted data and keeps data files efficient to read. It operates on the same data files that front-end reads and writes use, so large bursts of compaction can consume enough disk I/O to slow front-end operations. Rate limiting smooths those bursts into a steady background rate, keeping front-end performance predictable — at the cost of slower compaction.

When to use it

Rate limiting helps wherever compaction can surge and contend with front-end I/O, for example:

• Bursty write heavy workloads, where spikes in writes drive spikes in compaction.
• Large collection drops, which mark a large amount of data for removal at once.
• Sudden large number document expirations, such as a batch of items with closer TTL range expiring together.

The trade-off is that compaction runs more slowly, so disk usage may stay higher for longer. Set the limit low enough to protect front-end write bursts, but high enough that compaction keeps pace with the write rate.

Recommended value

A practical starting point is 100 MB/s (104857600 bytes/s), or about 10× your incoming write rate — estimated as writes per second per node × average document size. For example, 10,000 writes/s of 1 KB documents is roughly 10 MB/s incoming, giving a ~100 MB/s limit.

[t3rm1n4l]

@shivaniguptacb Please review

[anirudhRowjee]

If we don't have data-only rate limiting exposed, looking for this name instead of magma_compaction_rate_limit will confuse the customer - it would help if we used this key in both the set and get example (line 100)

[anirudhRowjee]

I'd prefer if we changed this to "Compaction Rate Limit Settings" (and on line 85)

[anirudhRowjee]

Can we say "compactions on all shards on one node consume"?

[anirudhRowjee]

Would it be worth adding that "though the setting is global, the application is on a per-node basis, allowing compactions on each node to consume a maximum of (compaction rate limit) bytes per second"