Skip to content

[APIE-1087] AssociatedNameStrategy support for the Odyssey initiative - CLI#3373

Open
Cynthia Qin (cqin-confluent) wants to merge 13 commits into
mainfrom
odyssey-cli
Open

[APIE-1087] AssociatedNameStrategy support for the Odyssey initiative - CLI#3373
Cynthia Qin (cqin-confluent) wants to merge 13 commits into
mainfrom
odyssey-cli

Conversation

@cqin-confluent

@cqin-confluent Cynthia Qin (cqin-confluent) commented May 29, 2026

Copy link
Copy Markdown
Member

Release Notes

Breaking Changes

  • PLACEHOLDER

New Features

  • Update confluent kafka topic produce and confluent kafka topic consume to resolve the Schema Registry subject via topic-to-subject associations when present(AssociatedNameStrategy), and otherwise fall back to the existing TopicNameStrategy (<topic>-key / <topic>-value).

Bug Fixes

  • PLACEHOLDER

Checklist

  • I have successfully built and used a custom CLI binary, without linter issues from this PR.
  • I have clearly specified in the What section below whether this PR applies to Confluent Cloud, Confluent Platform, or both.
  • I have verified this PR in Confluent Cloud pre-prod or production environment, if applicable.
  • I have verified this PR in Confluent Platform on-premises environment, if applicable.
  • I have attached manual CLI verification results or screenshots in the Test & Review section below.
  • I have added appropriate CLI integration or unit tests for any new or updated commands and functionality.
  • I confirm that this PR introduces no breaking changes or backward compatibility issues.
  • I have indicated the potential customer impact if something goes wrong in the Blast Radius section below.
  • I have put checkmarks below confirming that the feature associated with this PR is enabled in:
    • Confluent Cloud prod
    • Confluent Cloud stag
    • Confluent Cloud devel
    • Confluent Platform
    • Check this box if the feature is enabled for certain organizations only

What

Applies to Confluent Cloud only.

For the Odyssey initiative, Schema Registry now supports explicit topic-to-subject associations, which decouple the SR subject from the topic name. Previously confluent kafka topic produce / consume hard-coded TopicNameStrategy and the subject was always assumed to be <topic>-key / <topic>-value.

This PR makes produce/consume resolve the subject using a lookup-then-fallback strategy:

  1. Query the SR associations API by (kafkaClusterId, topic, mode).
  2. If an association exists, use its subject.
  3. Otherwise, fall back to TopicNameStrategy (the existing behavior).

Blast Radius

The new behavior adds an SR associations lookup to the produce/consume path on cloud. This change is designed to be backward compatible.

  • On-prem: unchanged. The path explicitly uses TopicNameStrategy and makes no associations call.
  • Cloud, no association: falls back to TopicNameStrategy, i.e. identical subject to today. The SR /associations endpoint is available in all regions and returns an empty result when no association exists, so the fallback is graceful.
  • Cloud, association exists: the subject resolves to the associated subject (the new behavior).

Existing and fall-back behavior has been manually tested and verified end-to-end (see Test & Review). However, if something goes wrong, Confluent Cloud customers who are using confluent kafka topic [consume | produce] may be blocked.

References

JIRA APIE-1087
INIT-11067 CLI/Terraform Workstream - Odyssey Explicit Topic Subject Association

Test & Review

Existing unit tests

❯ go test -count=1 ./pkg/serdes/... ./internal/kafka/...
... 
ok      github.com/confluentinc/cli/v4/pkg/serdes       3.060s
ok      github.com/confluentinc/cli/v4/internal/kafka   1.214s

New unit tests cover the helper and the SerDes wiring (associated subject vs. fallback)

❯ go test -count=1 -run 'TestResolveSubject|TestResolveProduceSubject|TestResolveAssociatedValueSubject|TestNewSchemaRegistryClient|TestConsumeMessage_ValueSubjectFallback|TestSubjectStrategy|TestPrimitiveSerdesInit|TestAvroSerdesUsesAssociatedSubject|TestJsonSerdesUsesAssociatedSubject|TestProtobufSerdesUsesAssociatedSubject' ./pkg/serdes/... ./internal/kafka/... -v
...
    --- PASS: TestResolveSubject/matching_association_returns_its_subject (0.00s)
    --- PASS: TestResolveSubject/association_for_other_mode_falls_back (0.00s)
    --- PASS: TestResolveSubject/association_under_different_cluster_id_falls_back (0.00s)
    --- PASS: TestResolveSubject/association_lookup_error_falls_back_to_TopicNameStrategy (0.00s)
PASS
ok      github.com/confluentinc/cli/v4/internal/kafka   1.090s

Manual tests (verified in DEVEL environment in Odyssey Testing org 722ddd57-b561-445c-88de-1a130e3783b8)
0-sanity-test.log
1-association-test-avro.log
2-association-lifecycle-test.log
3-association-test-protobuf.log
4-association-test-json.log
5-association-test-w-schema-flag.log
6-association-test-delete-key.log
7-association-test-schema-header.log
8-association-test-invalid-schema.log

@cqin-confluent Cynthia Qin (cqin-confluent) requested a review from a team as a code owner May 29, 2026 13:05
Copilot AI review requested due to automatic review settings May 29, 2026 13:05
@confluent-cla-assistant

Copy link
Copy Markdown

🎉 All Contributor License Agreements have been signed. Ready to merge.
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to add AssociatedNameStrategy support to the CLI’s Schema Registry serde flows by propagating a Kafka cluster ID into serializer/deserializer initialization and (for produce) resolving the SR subject via the associations API.

Changes:

  • Extend serde provider interfaces to accept kafkaClusterId and thread it through produce/consume paths.
  • Configure JSON/Protobuf serializers & deserializers to use serde.AssociatedNameStrategyType when a Kafka cluster ID is available.
  • Add helpers to create an SR client and resolve a subject via SR associations (with fallback).

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
pkg/serdes/serdes.go Updates provider interfaces; adds SR client helper and subject resolution via associations.
pkg/serdes/protobuf_serialization_provider.go Enables AssociatedNameStrategy for Protobuf serialization when Kafka cluster ID is provided.
pkg/serdes/protobuf_deserialization_provider.go Enables AssociatedNameStrategy for Protobuf deserialization when Kafka cluster ID is provided.
pkg/serdes/json_serialization_provider.go Enables AssociatedNameStrategy for JSON Schema serialization when Kafka cluster ID is provided.
pkg/serdes/json_deserialization_provider.go Enables AssociatedNameStrategy for JSON Schema deserialization when Kafka cluster ID is provided.
pkg/serdes/integer_serialization_provider.go Adjusts InitSerializer signature to match updated interface.
pkg/serdes/integer_deserialization_provider.go Adjusts InitDeserializer signature to match updated interface.
pkg/serdes/double_serialization_provider.go Adjusts InitSerializer signature to match updated interface.
pkg/serdes/double_deserialization_provider.go Adjusts InitDeserializer signature to match updated interface.
internal/kafka/confluent_kafka.go Threads Kafka cluster ID into deserializer initialization during consume.
internal/kafka/command_topic_produce.go Passes Kafka cluster ID to serializers and resolves subject via associations for registration/lookup.
internal/kafka/command_topic_consume.go Captures Kafka cluster ID in the consume handler for downstream deserializer initialization.
internal/asyncapi/command_export.go Updates deserializer init callsite for the new signature (passes empty Kafka cluster ID).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pkg/serdes/serdes.go
Comment on lines 70 to 72
type SerializationProvider interface {
InitSerializer(srClientUrl, srClusterId, mode string, schemaId int, srAuth SchemaRegistryAuth) error
InitSerializer(srClientUrl, srClusterId, kafkaClusterId, mode string, schemaId int, srAuth SchemaRegistryAuth) error
LoadSchema(string, map[string]string) error
Comment thread pkg/serdes/serdes.go
Comment on lines 79 to 81
type DeserializationProvider interface {
InitDeserializer(srClientUrl, srClusterId, mode string, srAuth SchemaRegistryAuth, existingClient schemaregistry.Client) error
InitDeserializer(srClientUrl, srClusterId, kafkaClusterId, mode string, srAuth SchemaRegistryAuth, existingClient schemaregistry.Client) error
LoadSchema(string, string, serde.Type, *kafka.Message) error
Comment thread pkg/serdes/serdes.go Outdated
Comment on lines +168 to +170
// returns the SR subject for (topic, mode) by querying the associations API with the Kafka cluster id
// as resource namespace. Falls backt o default TopicNameStrategy (<topic>-<mode>) if unmatched.
func ResolveSubject(client schemaregistry.Client, kafkaClusterId, topic, mode string) string {
Comment thread pkg/serdes/serdes.go Outdated
Comment on lines +175 to +177
associations, err := client.GetAssociationsByResourceName(topic, kafkaClusterId, "topic", []string{mode}, "", 0, -1)
if err != nil || len(associations) == 0 {
return fallback
Comment thread internal/kafka/command_topic_produce.go Outdated
Comment on lines +524 to +529
// Resolve subject via SR associations, fall back to TopicNameStrategy on miss.
subject := topicNameStrategy(topic, mode)
if kafkaClusterId != "" && srEndpoint != "" {
if client, err := serdes.NewSchemaRegistryClient(srEndpoint, srClusterId, srAuth); err == nil {
subject = serdes.ResolveSubject(client, kafkaClusterId, topic, mode)
}
Comment thread pkg/serdes/serdes.go
Comment on lines 79 to 81
type DeserializationProvider interface {
InitDeserializer(srClientUrl, srClusterId, mode string, srAuth SchemaRegistryAuth, existingClient schemaregistry.Client) error
InitDeserializer(srClientUrl, srClusterId, kafkaClusterId, mode string, srAuth SchemaRegistryAuth, existingClient schemaregistry.Client) error
LoadSchema(string, string, serde.Type, *kafka.Message) error
Comment thread pkg/serdes/serdes.go Outdated
Comment on lines +168 to +170
// returns the SR subject for (topic, mode) by querying the associations API with the Kafka cluster id
// as resource namespace. Falls backt o default TopicNameStrategy (<topic>-<mode>) if unmatched.
func ResolveSubject(client schemaregistry.Client, kafkaClusterId, topic, mode string) string {
Comment thread pkg/serdes/serdes.go Outdated
Comment on lines +175 to +177
associations, err := client.GetAssociationsByResourceName(topic, kafkaClusterId, "topic", []string{mode}, "", 0, -1)
if err != nil || len(associations) == 0 {
return fallback
Comment thread internal/kafka/command_topic_produce.go Outdated
Comment on lines +524 to +529
// Resolve subject via SR associations, fall back to TopicNameStrategy on miss.
subject := topicNameStrategy(topic, mode)
if kafkaClusterId != "" && srEndpoint != "" {
if client, err := serdes.NewSchemaRegistryClient(srEndpoint, srClusterId, srAuth); err == nil {
subject = serdes.ResolveSubject(client, kafkaClusterId, topic, mode)
}
Comment thread pkg/serdes/serdes.go Outdated
Comment on lines +170 to +176
func ResolveSubject(client schemaregistry.Client, kafkaClusterId, topic, mode string) string {
fallback := topic + "-" + mode
if kafkaClusterId == "" || client == nil {
return fallback
}
associations, err := client.GetAssociationsByResourceName(topic, kafkaClusterId, "topic", []string{mode}, "", 0, -1)
if err != nil || len(associations) == 0 {
@sonarqube-confluent

Copy link
Copy Markdown

Quality Gate failed Quality Gate failed

Failed conditions
50.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

@cqin-confluent Cynthia Qin (cqin-confluent) changed the title AssociatedNameStrategy support for the Odyssey initiative - CLI [APIE-1087] AssociatedNameStrategy support for the Odyssey initiative - CLI Jun 1, 2026
@sonarqube-confluent

Copy link
Copy Markdown

Quality Gate failed Quality Gate failed

Failed conditions
62.5% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

@sonarqube-confluent

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants