CBG-4244 cbgt: only recreate indexes if changed

[torcolvin]

Do not call cbgt.Manager.CreateIndex if an index in the bucket already exists.

In a multi node Sync Gateway cluster running import:

1. Create node 1
2. Node 1 is assigned all 16 import partitions
3. Node 2 joins
4. Partitions are split in two sets of 8 on each node

Prior to this code review, the partitions that stayed on node 1 would have their feeds closed and recreated. In this code change, the definitions are not updated if they match what is in the bucket.

Pre-review checklist

• Removed debug logging (fmt.Print, log.Print, ...)
• Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
• Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Dependencies (if applicable)

• Link upstream PRs
• Update Go module dependencies when merged

Integration Tests

• https://jenkins.sgwdev.com/job/SyncGatewayIntegration/972/

[Copilot]

Pull request overview

This PR improves cbgt sharded-DCP index creation behavior so that Sync Gateway nodes do not redundantly “update” (and thereby rotate UUIDs / restart PIndexes) when the persisted index definition is already identical—preventing unnecessary DCP feed teardown/recreate during cluster topology changes (e.g., nodes joining during import).

Changes:

• Add an “already up to date” check in cbgt index creation to skip no-op updates that would otherwise rotate the index UUID and restart PIndexes.
• Introduce semantic equality helpers for sharded-DCP feed source/index params (order-insensitive / canonicalized JSON comparisons) and unit tests for them.
• Add multi-node rebalance tests and new RestTesterCluster capabilities to add nodes and assert PIndex stability across rebalances.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
rest/utilities_testing_rest_tester_cluster.go	Adds locking around cluster node list and introduces AddNode to simulate nodes joining an existing cluster.
rest/cbgt_rebalance_test.go	New rebalance tests ensuring only moved PIndexes restart and unmoved PIndexes retain UUIDs.
db/util_testing.go	Adds test helper types/functions to snapshot import PIndex state (UUID + source partitions).
base/dcp_test.go	Adds tests ensuring cbgt index creation is idempotent and tolerant of transient metadata read errors; documents a known update/delete race.
base/dcp_sharded.go	Implements “skip redundant update” behavior by comparing persisted IndexDef to the intended one before calling CreateIndex.
base/dcp_sharded_test.go	Adds unit test coverage for cbgtIndexDefUnchanged behavior.
base/dcp_feed_type.go	Adds equality helpers for feed params (canonical JSON + order-insensitive collections) for robust comparisons.
base/dcp_feed_type_test.go	New unit tests covering SGFeedSourceParamsEqual / SGFeedIndexParamsEqual and collection order-insensitivity.

[adamcfraser]

Looks like a good approach, a couple of questions on the specifics.

Also, do we need to create some E2E tests to verify the functionality when not sharing cbgt/SGW globals?

[adamcfraser]

This works if cbgt will always persist SourceParams / IndexParams exactly as we send them (i.e. they don't do things like set default values, etc). Just to be sure, have you verified this is a safe assumption?

[adamcfraser]

Is this kick is for a race where the index isn't present when started the manager, but was created in between? I thought the manager would be listening for this in the usual way (i.e. the same way it detects indexes created on other nodes today). I'm fine if this is purely defensive ("it won't hurt to kick"), just want to understand the motivation.

[torcolvin]

At this point, it is defensive and I think I would rather not run an additional kick, reducing cbgt processing is part of the goal. Manager.Start has already run the appropriate Kick.