From 4ecc02d68cfbccae13f3ea1a4664571e8cdf1b13 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Mon, 13 Jul 2026 17:17:18 -0600 Subject: [PATCH] docs: replicating the system database with a constrained topology (5.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Controlled-flow replication can now include the `system` database while keeping a constrained topology (directional self-record). Update the replication overview (Controlling Replication Flow) — the previous "avoid replicating system" note no longer applies for directional routes — and add a 5.2 release note. Documents per-database directional routes, the add_node database-scoping behavior change, and the central-visibility caveat. Companion to the harper-pro core change (branch kris/systemdb-routing-repro). Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/replication/overview.md | 31 ++++++++++++++++++++++++++++++- release-notes/v5-lincoln/5.2.md | 20 ++++++++++++++++++++ 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/reference/replication/overview.md b/reference/replication/overview.md index a9526c82..57ea163f 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -198,7 +198,36 @@ replication: In this example, the local node only receives from `node-two` (one-way inbound) and only sends to `node-three` (one-way outbound). -> **Note**: When using controlled flow replication, avoid replicating the `system` database. The `system` database contains node configurations, so replicating it would cause all nodes to have identical (and incorrect) route configurations. +You can also scope flow per database, so different databases flow in different directions between the same two nodes. Use `sendsTo` / `receivesFrom` entries with a `database`: + +```yaml +replication: + databases: + - cardata + - config + - system + routes: + - host: node-two + replicates: + sendsTo: + - database: config # push central config downstream + receivesFrom: + - database: cardata # aggregate telemetry upstream +``` + +#### Replicating the `system` database with controlled flow + + + +Before v5.2, replicating the `system` database under controlled flow was discouraged: because `hdb_nodes` (the node registry) lives in `system` and each node advertised itself as a full-mesh participant, replicating `system` caused every node to discover and directly connect to every other node — collapsing a constrained topology into a full mesh. + +As of v5.2 you can replicate `system` while keeping a constrained topology. When a node has directional routes, it advertises a **directional** registry record derived from those routes (which neighbors it sends to / receives from) instead of a blanket "connect to everyone." Discovered non-neighbor nodes therefore do not open direct connections. This lets central configuration — users, roles, and schemas — propagate transitively across the whole cluster while user-database connections stay on the routes you configured. For example, in a `roadside → middle → core` aggregation tree, a role created on a roadside node reaches the core through the middle tier, yet the core never opens a direct socket to a roadside node. + +Notes and current limitations: + +- This applies only when a node has **directional** routes (`replicates` with `sends`/`receives` or `sendsTo`/`receivesFrom`). A node with no directional routes keeps the legacy full-mesh advertisement. +- Central visibility of every node is not guaranteed: an aggregation node may not list every distant leaf in its `hdb_nodes` registry (the registry relay differs from data relay). This does not open a connection either way. +- Route changes to a node's own directionality take effect on restart. ### Explicit Subscriptions diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 365ddf38..2bb37f58 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -13,3 +13,23 @@ All patch release notes for 5.2.x are available on the [releases page](https://g ### Replicated `set_configuration` The `set_configuration` operation now accepts `"replicated": true` to apply a configuration change to all cluster nodes in a single Operations API call, with per-node outcomes reported in the response's `replicated` array. Only cluster-appropriate parameters should be replicated — see [Configuration Operations](/reference/v5/configuration/operations#set-configuration). + +## Replication + +### Replicating the System Database with a Constrained Topology + +Controlled-flow replication can now include the `system` database while keeping a constrained (non-mesh) topology. + +Previously, replicating `system` was discouraged when using [controlled replication flow](/reference/v5/replication/overview#controlling-replication-flow): because the node registry (`hdb_nodes`) lives in `system` and each node advertised itself as a full-mesh participant, replicating `system` caused every node to discover and directly connect to every other node — defeating the point of a constrained topology. + +As of 5.2, a node with directional routes advertises a **directional** registry record derived from those routes (the neighbors it sends to / receives from) instead of a blanket "connect to everyone." Discovered non-neighbor nodes therefore never open direct connections. This lets central configuration — users, roles, and schemas — propagate transitively across the entire cluster while user-database connections stay on the routes you configured. + +For example, in a `roadside → middle → core` aggregation tree, replicating `system` now lets a role created on a roadside node reach the core (through the middle tier) without the core ever opening a direct socket to a roadside node. + +Behavior notes: + +- This applies only to nodes that have **directional** routes. A node with no directional routes keeps the legacy full-mesh advertisement, so existing full-mesh clusters are unaffected. +- `add_node` / `set_node` with database-scoped `sendsTo` / `receivesFrom` entries now restrict replication to those databases (consistent with the config-route behavior). To replicate all databases while excluding specific tables, use a wildcard entry (`excludeTables` with no `database`). +- Central visibility of every node is not guaranteed: an aggregation node may not list every distant leaf in its `hdb_nodes` registry. This does not open a connection either way. + +See [Controlling Replication Flow](/reference/v5/replication/overview#controlling-replication-flow) for configuration details.