DOC-13862-XDCR Key ID Filtering

modules/learn/pages/clusters-and-availability/xdcr-filtering.adoc

@@ -219,6 +219,50 @@ By default, any source-document deletion (or expiration) _is_ replicated to the
 Note that such replication is _not_ prevented by the specifying of a filter that is formed with regular and other filtering expressions: such expressions only determine which non-deleted documents are to be replicated.
 Therefore, to ensure that document-deletions (and expirations) are _not_ replicated, _deletion filters_ must specifically be configured.
 
+[#filtering-deletions-and-expirations-with-expressions]
+=== Filtering Deletions and Expirations with Key ID Expressions
+
+By default, XDCR allows you to filter out all document deletions (`filterDeletions`) or expirations (`filterExpirations`) binary-style across an entire replication stream. 
+Starting in Couchbase Server 8.5, you can apply granular, expression-based filtering to document deletions and expirations using Key ID filter expressions.
+
+When enabled, XDCR evaluates the document ID (`META().id`) of incoming deletion and expiration events against the replication's filter expression before sending them over the network.
+
+[#use-case-partitioning]
+=== Use Case: Regional and Key-Partitioned Topologies
+
+Consider a multi-cluster, geographically partitioned architecture such as one that routes documents with key prefix `eu:` to a European data center and `us:` to an American data center.
+In this topology, standard filter expressions route matching document insertions and mutations.`
+
+Prior to this feature:
+
+* If deletion filtering is disabled (`filterDeletions=false`), document deletions for all keys (including `us:*`) propagate to the European cluster, potentially deleting documents unnecessarily.
+
+* If deletion filtering is enabled (`filterDeletions=true`), no deletions are sent.
+
+With Key ID deletion/expiration filtering:
+
+* Setting `filterDeletionsWithExpression=true` ensures that only deletions for keys matching the filter expression (for example, `REGEXP_CONTAINS(META().id, "^eu:")`) replicate to the target cluster.
+
+[#expression-constraints]
+=== Expression Constraints and Requirements
+
+Because document deletions (tombstones) and expiration notices do not retain the original document body or extended attributes (xattrs), the following technical constraints apply:
+
+* Document ID References Only: The filter expression must evaluate only the document key ID (`META().id`). Expressions attempting to match JSON document fields (such as `type = "user"`) or extended attributes will fail validation or will evaluate to false during deletion/expiration processing.
+
+* Prerequisite Setting: `filterDeletionsWithExpression` requires `filterDeletions` to be enabled (`true`). Likewise, `filterExpirationsWithExpression` requires `filterExpirations` to be enabled (`true`).
+
+[#dynamic-updates]
+=== Dynamic Updates and Mutation Processing
+
+You can enable or disable `filterDeletionsWithExpression` and `filterExpirationsWithExpression` on active, running replications dynamically:
+
+* Changes take effect immediately for all mutations received after the configuration update timestamp.
+
+* Unlike modifying the primary `filterExpression` (which triggers a restream from sequence number 0), modifying deletion/expiration expression filtering does not restream existing mutations.
+
+IMPORTANT: In mixed-version clusters or during rolling upgrades to patch versions, make sure that all nodes in the source cluster have been upgraded before enabling `filterDeletionsWithExpression` or `filterExpirationsWithExpression`.
+
 === Tombstones, DCP Events, and Replication
 
 When a document is deleted or is expired, a tombstone is created.

modules/manage/pages/manage-xdcr/create-xdcr-replication.adoc

@@ -149,6 +149,7 @@ image::manage-xdcr/xdcr-advanced-settings.png[,400,align=left]
 The values displayed in the fields are defaults, which can be modified interactively, and saved: this may help in achieving optimal replication-performance.
 For details on the significance of each field, see the xref:xdcr-reference:xdcr-reference-intro.adoc[XDCR Reference].
 
+
 [#xdcr-ui-settings-for-conflict-logging]
 === Replication Settings for XDCR Conflict Logging
 
@@ -182,7 +183,8 @@ image::manage-xdcr/customise-conflict-logging-at-scope.png[,800,align=left]
 
 Customise Conflict Logging at collection level::
 Use this setting to specify a custom collection to store conflict logs.
-You can choose a custom conflict collection that overrides the default conflict collection. Choose one of the following collection types:
+You can choose a custom conflict collection that overrides the default conflict collection.
+Choose 1 of the following collection types:
 * *Default collection*: To set the collection to the default collection of the bucket.
 * *Parent collection*: To set the collection to that of the scope.
 * *Custom collection*: To set the collection to a custom collection that you specify.

modules/manage/pages/manage-xdcr/filter-xdcr-replication.adoc

@@ -178,6 +178,98 @@ For information on TTL and expiration, see xref:learn:data/expiration.adoc[Expir
 
 Note that the replication of deletions, expirations, and/or TTLs is _not_ prevented by the specifying of a filter that is formed with regular and other filtering expressions: to ensure that document-deletions, expirations, and/or TTLs are _not_ replicated, the appropriate deletion-filter checkboxes must be checked.
 
+
+
+== Filter Deletions and Expirations with Key ID Expressions
+
+By default, XDCR deletion filters operate across all documents in a replication stream.
+Starting in Couchbase Server 8.1.0 (and 7.6.10), you can apply granular expression-based filtering to document deletions and expirations using **Key ID filter expressions** (`META().id`).
+
+When enabled, XDCR evaluates the document key ID of incoming deletion and expiration events against the replication's filter expression before replicating them to the target cluster.
+
+[#ui-deletion-filtering-controls]
+=== Deletion and Expiration Filtering UI Controls
+
+In the Couchbase Web Console, under *Advanced replication settings*, the following controls manage deletion and expiration expression filtering:
+
+* *Do not replicate document expirations* (`filterExpirations`): When selected, prevents document expirations from replicating to the target.
+* *Apply Filter Expression to Expirations* (`filterExpirationsWithExpression`): Evaluates the replication's filter expression against expiring documents.
+Document expirations replicate to the target *only* if the document ID matches the filter expression.
+
+* *Do not replicate DELETE operations* (`filterDeletions`): When selected, prevents document deletions from replicating to the target.
+** *Apply Filter Expression to Deletions* (`filterDeletionsWithExpression`): Evaluates the replication's filter expression against deleted documents. Document deletions replicate to the target *only* if the document ID matches the filter expression.
+
+* *Remove TTL from replicated items* (`filterExpirationTTL`): Removes Time-To-Live (TTL) expiration timestamps from replicated documents.
+
+[#ui-prerequisites-and-validation]
+=== UI Prerequisites and Validation Rules
+
+The Web Console enforces the following validation dependencies:
+
+[cols="1,2,2"]
+|===
+| UI Checkbox | Prerequisite / Dependency | Constraint & Behavior
+
+| *Apply Filter Expression to Expirations*
+| Requires both:
+* A valid *Filter Expression* entered.
+* *Do not replicate document expirations* selected.
+| The checkbox remains disabled/greyed out until both prerequisites are satisfied. The filter expression must reference *only* `META().id`.
+
+| *Apply Filter Expression to Deletions*
+| Requires both:
+* A valid *Filter Expression* entered.
+* *Do not replicate DELETE operations* selected.
+| The checkbox remains disabled/greyed out until both prerequisites are satisfied. The filter expression must reference *only* `META().id`.
+|===
+
+NOTE: Because document deletions (tombstones) and expiration notices retain only the document key and metadata—not the document body or extended attributes (xattrs)—the filter expression must evaluate **only** the document ID (`META().id` or key-matching regular expressions). 
+Expressions attempting to match document JSON fields (such as `type = "hotel"`) or extended attributes will evaluate to `false` on deletions and expirations, preventing them from replicating.
+
+---
+
+[#enable-expression-filtering-create]
+=== Enable Key ID Deletion Filtering During Replication Creation
+
+To enable Key ID deletion and expiration filtering when creating a new replication:
+
+. In the Couchbase Web Console, navigate to menu:XDCR[Add Replication].
+. Specify the *Cluster Reference*, *Source Bucket*, and *Target Bucket*.
+. In the *Filter Expression* field, enter an expression that filters by document key ID.
+For example:
++
+[source,sql]
+----
+REGEXP_CONTAINS(META().id, "^eu:")
+----
+. Click *Advanced replication settings* to expand the configuration panel.
+. Under deletion filtering:
+.. Select *Do not replicate DELETE operations*.
+.. Select the nested checkbox *Apply Filter Expression to Deletions*.
+.. (Optional) Select *Do not replicate document expirations* and then select *Apply Filter Expression to Expirations*.
+. Configure any additional advanced replication settings as needed.
+. Click *Save* to start the replication.
+
+---
+
+[#enable-expression-filtering-edit]
+=== Modify Key ID Deletion Filtering on an Existing Replication
+
+You can dynamically enable or disable Key ID deletion and expiration filtering on an active replication without restarting or restreaming the replication:
+
+. In the Couchbase Web Console, navigate to the *XDCR* screen.
+. Locate the replication stream in the *Ongoing Replications* table.
+. Click the *Edit* (gear) icon in the row corresponding to the replication.
+. In the *Edit Replication* dialog, expand *Advanced replication settings*.
+. Modify the *Apply Filter Expression to Deletions* and/or *Apply Filter Expression to Expirations* checkboxes as required.
+. Click *Save*.
+
+NOTE: Unlike changing the primary *Filter Expression* (which triggers a restream of data from sequence number 0), updating `filterDeletionsWithExpression` or `filterExpirationsWithExpression` applies dynamically to mutations processed after the setting update without restreaming.
+
+
+
+
+
 [#filtering-binary-documents]
 === Filtering Binary Documents
 

modules/manage/pages/manage-xdcr/xdcr-management-overview.adoc

@@ -55,7 +55,8 @@ Information about establishing references is provided in xref:manage:manage-xdcr
 
 * Replication management: Once a reference has been registered on the local cluster, it can be specified as the target for a replication.
 This requires that an existing source and an existing target bucket also be specified.
-Optionally, filters can be established, so that only documents with matching ids, fields, values, or extended attributes are replicated.
+Optionally, filters can be established so that only documents with matching IDs, fields, values, or extended attributes are replicated.
+You can also configure deletion and expiration filters—including Key ID filter expressions—to selectively control whether document deletions and expirations are propagated to the target.
 Advanced settings can be configured to ensure optimal performance.
 Information about establishing replications is provided in xref:manage:manage-xdcr/create-xdcr-replication.adoc[Create a Replication] and information about deleting them in xref:manage:manage-xdcr/delete-xdcr-replication.adoc[Delete a Replication].
 +