DOC-12561 Continuous backup PITR - #4138
Conversation
* Added minitor continuous backup. * Updated metrics. * Finished draft of manage continuous backup.
…he install section.
There was a problem hiding this comment.
Pull request overview
Adds Couchbase Server 8.1 documentation and reference updates for the new Continuous Backup + Point-in-Time Recovery (PITR) capabilities, including new Learn/Manage pages, navigation entries, and supporting operational reference updates (ports/processes/metrics metadata).
Changes:
- Introduces new Learn + Manage documentation for continuous backup and PITR, plus example command/log snippets.
- Updates site navigation and related “new features”/cross-reference pages to surface continuous backup.
- Extends metrics metadata and operational reference docs (install ports + server processes) to reflect new/updated functionality.
Reviewed changes
Copilot reviewed 19 out of 20 changed files in this pull request and generated 23 comments.
Show a summary per file
| File | Description |
|---|---|
| preview/DOC-12561_Continuous_backup_PITR.yml | Adds a preview build config enabling the backup content source for the DOC-12561 preview. |
| modules/ROOT/nav.adoc | Adds nav entries for the new continuous backup/PITR learn + manage pages. |
| modules/metrics-reference/attachments/n1ql_metrics_metadata.json | Adds/adjusts SQL++ (N1QL) metrics metadata entries. |
| modules/metrics-reference/attachments/kv_metrics_metadata.json | Adds/adjusts KV metrics metadata (including continuous backup/magma/fusion-related metrics). |
| modules/metrics-reference/attachments/index_metrics_metadata.json | Adds new index metrics metadata for vector-related index counts. |
| modules/metrics-reference/attachments/goxdcr_metrics_metadata.json | Updates XDCR metrics help/notes and adjusts a metric type/version entry. |
| modules/metrics-reference/attachments/fts_metrics_metadata.json | Adds new FTS metrics metadata for vector indexes in CPU/GPU memory. |
| modules/metrics-reference/attachments/cm_metrics_metadata.json | Adds a new CM telemetry metric metadata entry. |
| modules/manage/pages/manage-backup-and-restore/monitor-continuous-backup.adoc | New “monitor continuous backup” operational doc page. |
| modules/manage/pages/manage-backup-and-restore/manage-pitr.adoc | New PITR management doc page. |
| modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc | New “configure/manage continuous backup” doc page. |
| modules/manage/examples/continuous-backup.sh | Adds example cbcontbk CLI snippets referenced by the Manage docs. |
| modules/manage/examples/continuous-backup.log | Adds example output referenced by the Manage docs. |
| modules/learn/pages/services-and-indexes/services/continuous-backup.adoc | New Learn overview page describing continuous backup + PITR concepts. |
| modules/learn/pages/services-and-indexes/services/backup-service.adoc | Adds a section linking Backup Service to continuous backup/PITR. |
| modules/learn/pages/data/change-history.adoc | Adds guidance tying change history requirements to continuous backup usage. |
| modules/learn/pages/clusters-and-availability/hard-failover.adoc | Adds a caution about unsafe failover behavior when continuous backup is enabled. |
| modules/introduction/partials/new-features-81.adoc | Adds 8.1 release notes bullets for continuous backup + PITR. |
| modules/install/pages/server-processes.adoc | Documents the cbcontbk process and normalizes path formatting in the process table. |
| modules/install/pages/install-ports.adoc | Adds port 9125 and documents the cbcontbk port usage. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 20 changed files in this pull request and generated 14 comments.
Comments suppressed due to low confidence (1)
modules/metrics-reference/attachments/kv_metrics_metadata.json:993
- Spelling: "emphemeral" should be "ephemeral".
"help": "When true, DCP backfills will be checked for progress. Any backfill which makes no progress for dcp_backfill_idle_limit_seconds will be subject to further checks. If the directory referenced by dbname is over dcp_backfill_idle_disk_threshold percent used and cancelling the backfill will free disk space, the scan cancels and the associated DCP stream will end with reason slow. This is ignored for emphemeral buckets",
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 22 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (18)
modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc:170
- Step 1 under "Configure Continuous Backup Using the Web Console" has a typo ("Ic") and the second sentence is not formatted as part of the numbered list. This will likely render as a stray paragraph instead of an instruction step.
. Ic creating a new bucket, click menu:Buckets[Add Bucket].
If editing an existing bucket, click *Buckets*, expand an existing bucket, and click *Edit*.
modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc:208
- KMS prefix list contains a typo:
azurekeyvalueshould beazurekeyvault(consistent with the Web Console section and other backup docs).
This URI must start with `awskms`, `azurekeyvalue`, or `gcpkms`.
modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc:234
- Grammar: "then Backup Service" should be "the Backup Service", and adding commas improves readability.
* `--continuous-backup-interval`: How frequently in minutes then Backup Service copies data to the continuous backup location.
modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc:240
- KMS prefix list contains a typo:
azurekeyvalueshould beazurekeyvault.
This URI must start with `awskms`, `azurekeyvalue`, or `gcpkms`.
modules/rest-api/pages/rest-bucket-create.adoc:102
- Trailing whitespace after the parameter value placeholder is inconsistent with surrounding lines and can cause noisy diffs.
-d continuousBackupKmKeyUrl=<string>
modules/learn/pages/services-and-indexes/services/continuous-backup.adoc:121
- Grammar: "a higher volume mutations" is missing "of".
You can choose to have a shorter continuous backup interval on vital buckets that have a higher volume mutations.
modules/learn/pages/services-and-indexes/services/continuous-backup.adoc:202
- Typo: "to a point it time" should be "to a point in time".
For continuous backup to be able to reliably restore a bucket to a point it time, it must have a consistent history of the changes made to a bucket.
modules/metrics-reference/attachments/kv_metrics_metadata.json:993
- Spelling: "emphemeral" should be "ephemeral".
"help": "When true, DCP backfills will be checked for progress. Any backfill which makes no progress for dcp_backfill_idle_limit_seconds will be subject to further checks. If the directory referenced by dbname is over dcp_backfill_idle_disk_threshold percent used and cancelling the backfill will free disk space, the scan cancels and the associated DCP stream will end with reason slow. This is ignored for emphemeral buckets",
modules/manage/pages/manage-backup-and-restore/manage-pitr.adoc:91
- These bullet points are missing "is", which makes the list read ungrammatically.
* `--tmp-dir` a temporary directory the `cbcontbk` can use when restoring data.
* `-l` the continuous backup storage location.
modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc:206
- The REST API section says you must supply "1 of the following" encryption/KMS settings, but two items are listed and the CLI section below explicitly says "both of the following". This is internally inconsistent and unclear for readers.
* `continuousBackupRetentionPeriod`: the number of hours the Backup Service keeps the continuous backup data before purging it.
This value sets how far back in time you can use point-in-time recovery (PITR).
* If you have enabled encryption at rest for the bucket, you must supply 1 of the following:
modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc:209
- Parameter description should start with a capital letter ("the name" -> "The name") for consistency with other entries in the list.
** `continuousBackupKmCredId`: the name of a credential in the Couchbase Server credential store that grants the Backup Service access to a KMS to manage its encryption keys.
modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc:230
- Grammar: "This value must a string" is missing "be".
This value must a string containing either an absolute path to an NFS mount point mounted on every node or a cloud storage URI.
modules/rest-api/pages/rest-bucket-create.adoc:97
- Parameter syntax is missing a space before the closing bracket (consistent style elsewhere is
[ true | false ]).
This issue also appears on line 102 of the same file.
-d continuousBackupEnabled=[ true | false]
modules/rest-api/pages/rest-bucket-create.adoc:1668
- Sentence should start with a capital letter ("the number" -> "The number").
the number of hours the Backup Service keeps the continuous backup data before purging it.
modules/rest-api/pages/rest-bucket-create.adoc:1676
- KMS prefix list contains a typo:
azurekeyvalueshould beazurekeyvault(consistent with other docs).
This URI must start with `awskms`, `azurekeyvalue`, or `gcpkms`.
modules/learn/pages/services-and-indexes/services/continuous-backup.adoc:88
- Typo: "verses" should be "versus".
This issue also appears in the following locations of the same file:
- line 121
- line 202
When setting the interval, you must consider potential data loss verses the potential increased storage cost.
modules/install/pages/server-processes.adoc:32
- Path formatting is inconsistent: other Linux process entries use a trailing
/for directories, butcbcontbkdoes not. Keeping this consistent avoids confusion about whether the value is a directory vs file path.
| `/opt/couchbase/bin`
modules/manage/pages/manage-backup-and-restore/manage-pitr.adoc:84
- Grammar: missing article in "to point in time".
This issue also appears on line 90 of the same file.
You use the `cbcontbk` command line tool to restore a bucket backed up using continuous backup to point in time.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 22 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (18)
modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc:234
- Typo: “then Backup Service” should be “the Backup Service”.
* `--continuous-backup-interval`: How frequently in minutes then Backup Service copies data to the continuous backup location.
modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc:240
- The KMS scheme list contains
azurekeyvalue, which appears to be a typo (elsewhere it’sazurekeyvault).
This URI must start with `awskms`, `azurekeyvalue`, or `gcpkms`.
modules/metrics-reference/attachments/kv_metrics_metadata.json:993
- Spelling: “emphemeral” should be “ephemeral”.
"help": "When true, DCP backfills will be checked for progress. Any backfill which makes no progress for dcp_backfill_idle_limit_seconds will be subject to further checks. If the directory referenced by dbname is over dcp_backfill_idle_disk_threshold percent used and cancelling the backfill will free disk space, the scan cancels and the associated DCP stream will end with reason slow. This is ignored for emphemeral buckets",
modules/rest-api/pages/rest-bucket-create.adoc:103
- In the curl syntax block, the new continuous backup lines have inconsistent bracket spacing compared to other parameters and include trailing whitespace. Aligning the formatting improves readability and consistency.
-d continuousBackupEnabled=[ true | false]
-d continuousBackupLocation=<string>
-d continuousBackupCloudStorageCredId=<string>
-d continuousBackupInterval=<integer>
-d continuousBackupRetentionPeriod=<integer>
-d continuousBackupKmKeyUrl=<string>
-d continuousBackupKmCredId=<string>
modules/rest-api/pages/rest-bucket-create.adoc:1669
- The
continuousBackupRetentionPerioddefinition starts with a lowercase “the”. Capitalize it to match the surrounding parameter descriptions.
`continuousBackupRetentionPeriod`::
the number of hours the Backup Service keeps the continuous backup data before purging it.
This value sets how far back in time you can use point-in-time recovery (PITR).
modules/rest-api/pages/rest-bucket-create.adoc:1676
- The KMS scheme list contains
azurekeyvalue, which appears to be a typo (elsewhere in the same PR it’s written asazurekeyvault). This should be consistent and accurate because users will copy/paste it.
The URI for the KMS key the Backup Service uses to manage the encryption key it uses to encrypt data encryption keys.
This URI must start with `awskms`, `azurekeyvalue`, or `gcpkms`.
modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc:170
- Step text has typos (“Ic”), extra spaces, and a missing space after “click”. This reads like an instruction users will follow verbatim, so it should be corrected.
. Ic creating a new bucket, click menu:Buckets[Add Bucket].
If editing an existing bucket, click *Buckets*, expand an existing bucket, and click *Edit*.
modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc:209
- This section says “you must supply 1 of the following” but then lists a KMS key URL and a credential ID, which are not alternatives (and the CLI section below says “both”). Please make the requirement consistent and fix the
azurekeyvaluetypo.
* If you have enabled encryption at rest for the bucket, you must supply 1 of the following:
** `continuousBackupKmKeyUrl`: The URI for the KMS key the Backup Service uses to manage the encryption key it uses to encrypt data encryption keys.
This URI must start with `awskms`, `azurekeyvalue`, or `gcpkms`.
** `continuousBackupKmCredId`: the name of a credential in the Couchbase Server credential store that grants the Backup Service access to a KMS to manage its encryption keys.
modules/manage/pages/manage-backup-and-restore/manage-continuous-backup.adoc:230
- Grammar: “This value must a string …” is missing “be”.
This value must a string containing either an absolute path to an NFS mount point mounted on every node or a cloud storage URI.
modules/learn/pages/services-and-indexes/services/continuous-backup.adoc:88
- “verses” should be “versus” in this context.
When setting the interval, you must consider potential data loss verses the potential increased storage cost.
modules/learn/pages/services-and-indexes/services/continuous-backup.adoc:202
- Typo: “point it time” should be “point in time”.
For continuous backup to be able to reliably restore a bucket to a point it time, it must have a consistent history of the changes made to a bucket.
modules/learn/pages/services-and-indexes/services/continuous-backup.adoc:213
- Grammar: “decrypt older the data” should be “decrypt older data”.
Therefore, the current DEKs are usually not the same ones continuous backup needs to decrypt older the data in its backups.
modules/manage/pages/manage-backup-and-restore/manage-pitr.adoc:73
- Extra punctuation: the sentence ends with “information..”.
See <<#pitr_force_updates>> for more information..
modules/install/pages/install-ports.adoc:356
- Trailing whitespace in the port description (“Continuous backup ”).
| Continuous backup
modules/install/pages/server-processes.adoc:32
- The path formatting is inconsistent: most entries use
/opt/couchbase/bin/butcbcontbkis/opt/couchbase/bin. Standardizing avoids implying different paths.
| `/opt/couchbase/bin`
modules/manage/examples/continuous-backup.sh:7
- Trailing whitespace at the end of the curl line can cause copy/paste issues in some shells and fails some linters.
-d continuousBackupInterval=10
modules/manage/pages/manage-backup-and-restore/monitor-continuous-backup.adoc:32
- This xref targets
continuous-backup:cbcontbk.adoc, but there’s nocbcontbk.adocpage in this repository and no other existing references to acontinuous-backupcomponent. Unless that component/page is provided elsewhere at build time, this will render as a broken link.
See the xref:continuous-backup:cbcontbk.adoc[`cbcontbk`] documentation for details.
modules/manage/pages/manage-backup-and-restore/manage-pitr.adoc:105
- This xref targets
continuous-backup:cbcontbk.adoc, but there’s nocbcontbk.adocpage in this repository and no other existing references to acontinuous-backupcomponent. Unless that component/page is provided elsewhere at build time, this will render as a broken link.
See the documentation for xref:continuous-backup:cbcontbk.adoc[`cbcontbk`] for details.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
sarahlwelton
left a comment
There was a problem hiding this comment.
I'm leaving a lot of repetitive comments around some style nits... I haven't gotten to the last few files because this is already a lot of comments, but maybe call this good for a first pass, clean up some of the repetitive issues, and then I can take a look at the remaining few files? There's just some missing sections, word order, etc. that I keep bringing up that I think you can make a judgment call on your own about fixing :)
Just reach out if you want to discuss.
| This backup can limit the loss of data updates that occur between full and incremental backups. | ||
| In addition to backing up data, it also backs up the timestamps of mutations. | ||
| These timestamps let you perform point-in-time recovery (PITR) to restore the Magma bucket's data to the state it was in at a specific moment. | ||
| See xref:learn:services-and-indexes/services/continuous-backup.adoc[] for more information. |
There was a problem hiding this comment.
| See xref:learn:services-and-indexes/services/continuous-backup.adoc[] for more information. | |
| For more information, see xref:learn:services-and-indexes/services/continuous-backup.adoc[]. |
| CAUTION: If you have 1 or more buckets using continuous backup, you must stop the Couchbase Server process on the node before performing an unsafe failover on it. | ||
| Not stopping the Couchbase Server process could result in data inconsistencies that can cause issues with the continuous backup. | ||
| See xref:learn:services-and-indexes/services/continuous-backup.adoc[] for more information. |
There was a problem hiding this comment.
| CAUTION: If you have 1 or more buckets using continuous backup, you must stop the Couchbase Server process on the node before performing an unsafe failover on it. | |
| Not stopping the Couchbase Server process could result in data inconsistencies that can cause issues with the continuous backup. | |
| See xref:learn:services-and-indexes/services/continuous-backup.adoc[] for more information. | |
| CAUTION: If you have 1 or more buckets using continuous backup, you must stop the Couchbase Server process on the node before you can perform an unsafe failover on that node. | |
| Not stopping the Couchbase Server process could result in data inconsistencies that can cause issues with the continuous backup. | |
| For more information, see xref:learn:services-and-indexes/services/continuous-backup.adoc[]. |
| IMPORTANT: You must enable change history on any bucket you want to back up using continuous backup. | ||
| Once you enable continuous backup, you cannot turn off change history for the bucket unless you turn off continuous backup first. | ||
| Using collection-level overrides of change history in a bucket where you have enabled continuous backup is unsupported. | ||
| See xref:learn:services-and-indexes/services/continuous-backup.adoc[] for more information. |
There was a problem hiding this comment.
| See xref:learn:services-and-indexes/services/continuous-backup.adoc[] for more information. | |
| For more information, see xref:learn:services-and-indexes/services/continuous-backup.adoc[]. |
|
|
||
| In addition to the periodic backups provided by the Backup Service, you can also use the continuous backup agent to frequently back up changes to Magma buckets. | ||
| This backup can limit the loss of data updates that occur between full and incremental backups. | ||
| It also records the timestamps of when data was mutated. |
There was a problem hiding this comment.
| It also records the timestamps of when data was mutated. | |
| Continuous backup also records the timestamps of when data was mutated. |
| This backup can limit the loss of data updates that occur between full and incremental backups. | ||
| It also records the timestamps of when data was mutated. | ||
| These timestamps let you perform Point-In-Time Recovery (PITR) to restore the Magma bucket's data to the state it was in at a specific moment. | ||
| See xref:learn:services-and-indexes/services/continuous-backup.adoc[] for more information about using continuous backup. |
There was a problem hiding this comment.
| See xref:learn:services-and-indexes/services/continuous-backup.adoc[] for more information about using continuous backup. | |
| For more information about using continuous backup, see xref:learn:services-and-indexes/services/continuous-backup.adoc[]. |
| You can enable continuous backup when creating a bucket or editing it via the REST-API. | ||
|
|
||
| Whether you're creating a new bucket or editing an existing bucket, make a POST call to the `/pools/default/buckets/<bucket-name>` REST API endpoint. | ||
| Set the following: |
There was a problem hiding this comment.
I'm wondering if this part should be a table with "Setting", "Description", "Example" instead of a list?
|
|
||
| Use the `couchbase-cli` tool's `bucket create` or `bucket-edit` command to change the continuous backup settings for the bucket. | ||
|
|
||
| Use the following arguments to configure continuous backup: |
There was a problem hiding this comment.
Same thing, change this to a table instead of a bullet list?
| --continuous-backup-retention-period 270 | ||
| ``` | ||
|
|
||
| See the documentation for the `couchbase-cli` tool's xref:cli:cbcli/couchbase-cli-bucket-create.adoc[] and xref:cli:cbcli/couchbase-cli-bucket-edit.adoc[] commands for more information about creating and editing buckets using the command line. |
There was a problem hiding this comment.
| See the documentation for the `couchbase-cli` tool's xref:cli:cbcli/couchbase-cli-bucket-create.adoc[] and xref:cli:cbcli/couchbase-cli-bucket-edit.adoc[] commands for more information about creating and editing buckets using the command line. | |
| For more information about how to create and edit buckets using the command line, see `couchbase-cli` tool's xref:cli:cbcli/couchbase-cli-bucket-create.adoc[] and xref:cli:cbcli/couchbase-cli-bucket-edit.adoc[] commands. |
|
|
||
| You can turn off continuous backup using the Couchbase Server Web Console, REST API, or command line: | ||
|
|
||
| Using the Couchbase Server Web Console:: |
There was a problem hiding this comment.
I would personally set this to a tabbed block if you're going to go this route, and not just use headings like previously.
| --username $USERNAME --password $PASSWORD \ | ||
| --bucket travel-sample \ | ||
| --continuous-backup-enabled 0 | ||
| ``` |
sarahlwelton
left a comment
There was a problem hiding this comment.
I'm leaving a lot of repetitive comments around some style nits... I haven't gotten to the last few files because this is already a lot of comments, but maybe call this good for a first pass, clean up some of the repetitive issues, and then I can take a look at the remaining few files? There's just some missing sections, word order, etc. that I keep bringing up that I think you can make a judgment call on your own about fixing :)
Just reach out if you want to discuss.
…on of KMS to Key Management System.
Adds documentation for Continuous backup and point-in-time recovery,
The links in the following changes led to the preview site. You will need the Docs Team credentials on Confluence.
Also addresses DOC-14623 which asked for addinitional information about PITR performance.
New pages added:
Existing pages updated:
cbcontbkNOTE: Known issue with these documents is that there are places where it should link to the credential store documentation. However, those documents have not been written yet. I've added DOC-14531 to track this needed update.