Skip to content

3.x: CCMBridge resolves server flavor from global state, not the cluster's own configuration #983

Description

@nikagra

Problem

CCMBridge (3.x test harness) decides which server flavor and version a CCM cluster
installs by reading global system properties (scylla.version, dse) rather than the
builder's own configuration. A cluster that configures its own version — via
CCMBridge.Builder.withVersion(...) or @CCMConfig(version = ...) — therefore gets
settings derived from the surrounding run's flavor instead of its own, in both directions.

Found while reviewing #982, which renamed driver-core's four never-executed *IT.java
CCM tests so Surefire discovers them. Fixes for items 1–3 below were written and verified
during that review, then deliberately pulled back out to keep #982 scoped to unblocking
the tests. This issue is that follow-up.

The 4.x driver has the same class of bug: #800 (CcmBridge reporting wrong database and
version).

Known defects

All line references are driver-core/src/test/java/com/datastax/driver/core/CCMBridge.java
on scylla-3.x.

1. buildCreateCommand ignores the Scylla flag entirely. It only ever receives the
resolved Cassandra and DSE versions, so its versionConfigured branch emits
-v <cassandraVersion> and never --scylla. For Scylla that Cassandra version is
hardcoded to 3.0.8 — what Scylla reports in system.local, not an install target.
So withVersion(...) under -Dscylla.version=... has ccm create build an Apache
Cassandra 3.0.8 cluster, while add() passes --scylla (line 743) and
getScyllaVersion() reports Scylla. The requested version is discarded outright.

2. SCYLLA_PRODUCT derives from the global scylla.version. ENVIRONMENT_MAP is
built once in the static initializer and, being immutable and shared by every cluster,
gives a builder that configures its own version the wrong repository in both directions:

  • global OSS + explicit Enterprise version → SCYLLA_PRODUCT unset → Enterprise version
    installed from the OSS repository.
  • global Enterprise + explicit OSS version → SCYLLA_PRODUCT=enterprise leaks in → OSS
    version looked up in the Enterprise repository.

3. Client encryption picks the wrong flavor's TLS yaml. withSSL()/withAuth()
choose between Cassandra's JKS keystore/truststore and Scylla's PEM
certificate/keyfile/truststore by reading the global scylla.version at
builder-configuration time — before the flavor is resolved and before withVersion()
has had any effect. Both directions are wrong. Unlike 1 and 2 this needs no
withScylla() call to reach: CCMTestsSupport.ccmBuilder calls withVersion() before
withSSL(), so @CCMConfig(version = ..., ssl = true) is enough.

Note for whoever fixes this: deriving the yaml at build time means ssl/auth stop
appearing in cassandraConfiguration at configuration time, so Builder.equals/
hashCode must compare them directly — CCMCache keys cached clusters on the builder,
and without it an encrypted cluster could be handed to a test that asked for a plaintext
one. (Builder.hashCode() also currently omits scylla while equals() includes it.)

4. ENVIRONMENT_MAP passes through an inherited SCYLLA_PRODUCT. The static
initializer seeds its env map from new ProcessBuilder().environment() and never strips
SCYLLA_PRODUCT, so an exported SCYLLA_PRODUCT=enterprise reaches every cluster. This
is longstanding behavior rather than a regression, and it interacts with item 2: any fix
for 2 must decide what the globally configured version does with an inherited value.
A non-numeric scylla.version (a branch spec) can't be recognised as Enterprise by the
year-prefix check, so exporting the variable is arguably the only way to select the
repository in that one case — but it should not apply to a numeric OSS version, a pure
Cassandra run, or no configured version.

Two things make this awkward to fix cleanly, and are the reason it wants its own change
rather than a one-liner:

  • ENVIRONMENT_MAP is assembled in a static initializer from the live process
    environment, so the behavior is not unit-testable without extracting the assembly into
    a package-private static method taking the base env map.
  • parseScyllaInputVersion calls getScyllaVersionThroughCcm, which runs ccm create
    through execute(File, String, Object...) before ENVIRONMENT_MAP is assigned
    (line 840). commons-exec then inherits this process's environment as-is, so stripping
    the variable from ENVIRONMENT_MAP does not seal the branch-spec probe path.

5. Builder.scylla defaults to the global property, and @CCMConfig cannot override
it.
private boolean scylla = GLOBAL_SCYLLA_VERSION_NUMBER != null; and
private boolean dse = isDse();withVersion() does not touch either. So under a
Scylla run, an explicit Cassandra version is classified as Scylla, which is the root
cause feeding items 1 and 2.

@CCMConfig has no scylla attribute at all: CCMTestsSupport forwards dse
(withDSE(dse)) but has no Scylla equivalent, and CCMConfig.version()'s javadoc still
describes it as "The C* or DSE version to use". A test therefore cannot express the
flavor even if it wanted to. RecommissionedNodeTest also calls withVersion(2.1.20)
with no flavor set.

Candidate fixes, cheapest first:

  1. Add a scylla() attribute to @CCMConfig and forward it in CCMTestsSupport.ccmBuilder()
    next to the existing withDSE call; give RecommissionedNodeTest an explicit
    .withScylla(false).
  2. Document in withVersion()'s javadoc that the version is interpreted per the current
    flavor flags, so callers must set the flavor first.
  3. Defensively, have the flavor resolution refuse a Scylla classification for a version
    that cannot be a Scylla release.

Why none of this is CI-reachable today

Worth stating so nobody treats this as urgent. The Scylla and Cassandra IT jobs run
mvn verify -Pshort, and no enabled short-group test configures a version while
creating a cluster:

  • ProtocolVersionRenegotiationTest pins version = "2.1.16" but is enabled = false,
    and its class resolves as PER_CLASS, where createCCMTestConfig runs with a null
    testMethod — so the method-level @CCMConfig is never read for CCM creation anyway.
  • DseCCMClusterTest sets dse = true (which takes an earlier branch in the flavor
    resolution) and is disabled.
  • RecommissionedNodeTest's withVersion call site is in a long-group,
    enabled = false test.
  • withScylla() has no callers.

SCYLLA_PRODUCT is also set nowhere in the Makefile or .github/workflows/, and every
Scylla IT leg resolves a year-prefixed version, so the inherited-value path in item 4 is
unreachable from CI.

This is therefore footgun removal that makes the harness behave as its API claims —
valuable the moment someone writes a versioned or SSL test, but not a live CI failure.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions