From e9cffb2eae2dc0f9a60964cba034be8d2e6010e3 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Fri, 10 Jul 2026 12:25:01 -0700 Subject: [PATCH 1/2] Doc schema context support with Iceberg --- .../pages/release-notes/redpanda.adoc | 236 +----------------- .../iceberg/iceberg-troubleshooting.adoc | 1 + .../pages/iceberg/specify-iceberg-schema.adoc | 37 +++ .../pages/schema-reg/schema-reg-contexts.adoc | 1 - 4 files changed, 43 insertions(+), 232 deletions(-) diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index 468dccf544..dbc3c13eb0 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -7,240 +7,14 @@ This topic includes new content added in version {page-component-version}. For a * xref:cloud-data-platform:get-started:whats-new-cloud.adoc[] * xref:cloud-data-platform:get-started:cloud-overview.adoc#redpanda-cloud-vs-self-managed-feature-compatibility[Redpanda Cloud vs Self-Managed feature compatibility] +== Iceberg translation with Schema Registry contexts -== Cloud Topics +Iceberg-enabled topics can now resolve schemas from a specific xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] instead of always using the default context. Set the new `redpanda.schema.registry.context` topic property to bind a topic's Iceberg translation to a context, useful for multi-tenant or per-environment schema isolation. -xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] are now available, making it possible to use durable cloud storage (S3, ADLS, GCS) as the primary backing store instead of local disk, eliminating over 90% of cross-AZ replication costs. This makes them ideal for latency-tolerant, high-throughput workloads such as observability streams, analytics pipelines, and AI/ML training data feeds, where cross-AZ networking charges are the dominant cost driver. - -You can use Cloud Topics exclusively in Redpanda Streaming clusters, or in combination with traditional Tiered Storage and local storage topics on a shared cluster supporting low latency workloads. - -Cloud Topics require Tiered Storage and an Enterprise license. For setup instructions and limitations, see xref:develop:manage-topics/cloud-topics.adoc[]. - -== Group-based access control (GBAC) - -Redpanda {page-component-version} introduces xref:manage:security/authorization/gbac.adoc[group-based access control (GBAC)], which extends OIDC authentication to support group-based permissions. In addition to assigning roles or ACLs to individual users, you can assign them to OIDC groups. Users inherit permissions from all groups reported by their identity provider (IdP) in the OIDC token claims. - -GBAC supports two authorization patterns: - -* Assign a group as a member of an RBAC role so that all users in the group inherit the role's ACLs. -* Create ACLs directly with a `Group:` principal. - -Group membership is managed entirely by your IdP. Redpanda reads group information from the OIDC token at authentication time and works across the Kafka API, Schema Registry, and HTTP Proxy. - -== FIPS 140-3 validation and FIPS Docker image - -Redpanda's cryptographic module has been upgraded from FIPS 140-2 to https://csrc.nist.gov/pubs/fips/140-3/final[FIPS 140-3^] validation. Additionally, Redpanda now provides a FIPS-specific Docker image (`docker.redpanda.com/redpandadata/redpanda:-fips`) for `amd64` and `arm64` architectures, with the required OpenSSL FIPS module pre-configured. - -NOTE: If you are upgrading with FIPS mode enabled, ensure all SASL/SCRAM user passwords are at least 14 characters before upgrading. FIPS 140-3 enforces stricter HMAC key size requirements. - -See xref:manage:security/fips-compliance.adoc[] for configuration details. - -== Iceberg: Expanded JSON Schema support - -Redpanda now supports additional JSON Schema patterns when translating to Iceberg tables: - -* `$ref` support: Internal references using `$ref` (for example, `"$ref": "#/definitions/myType"`) are resolved from schema resources declared in the same document. External references are not yet supported. -* Map type from `additionalProperties`: `additionalProperties` objects that contain subschemas now translate to Iceberg `map`. -* `oneOf` nullable pattern: The `oneOf` keyword is now supported for the standard nullable pattern if exactly one branch is `{"type":"null"}` and the other is a non-null schema. - -See xref:manage:iceberg/specify-iceberg-schema.adoc#how-iceberg-modes-translate-to-table-format[Specify Iceberg Schema] for JSON types mapping and updated requirements. - -== Ordered rack preference for Leader Pinning - -xref:develop:produce-data/leader-pinning.adoc[Leader Pinning] now supports the `ordered_racks` configuration value, which lets you specify preferred racks in priority order. Unlike `racks`, which distributes leaders uniformly across all listed racks, `ordered_racks` places leaders in the highest-priority available rack and fails over to subsequent racks only when higher-priority racks become unavailable. - -== User-based throughput quotas - -Redpanda now supports throughput quotas based on authenticated user principals. Unlike client-based quotas (which rely on self-declared `client-id` values), user-based quotas enforce limits using verified identities from SASL, mTLS, or OIDC authentication. - -You can set quotas for individual users, default users, or fine-grained user/client combinations. See xref:manage:cluster-maintenance/about-throughput-quotas.adoc[] for conceptual details, and xref:manage:cluster-maintenance/manage-throughput.adoc#set-user-based-quotas[Set user-based quotas] to get started. - -== Cross-region Remote Read Replicas - -Remote Read Replica topics on AWS can be deployed in a different region from the origin cluster's S3 bucket. This enables cross-region disaster recovery and data locality scenarios while maintaining the read-only replication model. - -To create cross-region Remote Read Replica topics, configure dynamic upstreams that point to the origin cluster's S3 bucket location. Redpanda manages the number of concurrent dynamic upstreams based on your `cloud_storage_url_style` setting (virtual_host or path style). - -See xref:manage:tiered-storage.adoc#remote-read-replicas[Remote Read Replicas] for setup instructions and configuration details. - -== Automatic broker decommissioning - -When continuous partition balancing is enabled, Redpanda can automatically decommission brokers that remain unavailable for a configured duration. The xref:reference:properties/cluster-properties.adoc#partition_autobalancing_node_autodecommission_timeout_sec[`partition_autobalancing_node_autodecommission_timeout_sec`] property triggers permanent broker removal, unlike xref:reference:properties/cluster-properties.adoc#partition_autobalancing_node_availability_timeout_sec[`partition_autobalancing_node_availability_timeout_sec`] which only moves partitions temporarily. - -Key characteristics: - -* Disabled by default -* Requires `partition_autobalancing_mode` set to `continuous` -* Permanently removes the node from the cluster (the node cannot rejoin automatically) -* Processes one decommission at a time to maintain cluster stability -* Manual intervention required if decommission stalls - -See xref:manage:cluster-maintenance/continuous-data-balancing.adoc[] for configuration details. - -== Schema Registry contexts - -xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry contexts] are namespaces that isolate schemas, subjects, and configurations within a single Schema Registry instance. Each context maintains its own schema ID counter, mode settings, and compatibility settings, so teams can share a Schema Registry without risking naming collisions or configuration drift. - -To enable contexts, set xref:reference:properties/cluster-properties.adoc#schema_registry_enable_qualified_subjects[`schema_registry_enable_qualified_subjects`] to `true` and restart your brokers. Once enabled, you reference schemas using qualified subject syntax, for example `:.staging:my-topic-value`. See xref:manage:schema-reg/schema-reg-contexts.adoc[] for configuration examples and limitations. - -== Schema Registry metadata properties - -xref:manage:schema-reg/schema-reg-overview.adoc#metadata-properties[Schema Registry metadata properties] let you store and retrieve arbitrary key-value pairs alongside schemas. Properties such as `owner`, `team`, or `application.version` travel with the schema through its lifecycle, making it easier to track ownership and lineage without modifying the schema itself. - -You can set metadata when registering a schema using the `POST /subjects/\{subject\}/versions` endpoint or with the xref:reference:rpk/rpk-registry/rpk-registry-schema-create.adoc[`--metadata-properties`] flag in `rpk registry schema create`. Metadata is returned in `GET /subjects/\{subject\}/versions/\{version\}` and `GET /schemas/ids/\{id\}` responses, and viewable with `rpk registry schema get --print-metadata`. New schema versions automatically inherit metadata from the most recent version of the subject unless you register with an explicit empty `metadata` object. - -== OIDC authentication with rpk - -Starting in v26.1.7, `rpk` supports the `OAUTHBEARER` SASL mechanism, so you can authenticate `rpk` to the Kafka API with an OIDC access token issued by your identity provider (IdP) instead of a SASL/SCRAM username and password. Pass the token through xref:reference:rpk/rpk-x-options.adoc#sasl-mechanism[`-X sasl.mechanism=OAUTHBEARER`] and xref:reference:rpk/rpk-x-options.adoc#pass[`-X pass="token:"`]. - -For end-to-end steps and prerequisites, see xref:manage:security/authentication.adoc#oidc-rpk[Connect to Redpanda with OIDC using rpk]. +See xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context] for configuration examples and considerations for changing a topic's context. == New configuration properties -**Storage mode:** - -* xref:reference:properties/cluster-properties.adoc#default_redpanda_storage_mode[`default_redpanda_storage_mode`]: Set the default storage mode for new topics (`local`, `tiered`, `cloud`, or `unset`) -* xref:reference:properties/topic-properties.adoc#redpanda-storage-mode[`redpanda.storage.mode`]: Set the storage mode for an individual topic, superseding the legacy `redpanda.remote.read` and `redpanda.remote.write` properties - -**Cloud Topics:** - -NOTE: Cloud Topics requires an Enterprise license. For more information, contact https://redpanda.com/try-redpanda?section=enterprise[Redpanda sales^]. - -* xref:reference:properties/cluster-properties.adoc#cloud_topics_allow_materialization_failure[`cloud_topics_allow_materialization_failure`]: Enable recovery from missing L0 extent objects -* xref:reference:properties/cluster-properties.adoc#cloud_topics_compaction_interval_ms[`cloud_topics_compaction_interval_ms`]: Interval for background compaction -* xref:reference:properties/cluster-properties.adoc#cloud_topics_compaction_key_map_memory[`cloud_topics_compaction_key_map_memory`]: Maximum memory per shard for compaction key-offset maps -* xref:reference:properties/cluster-properties.adoc#cloud_topics_compaction_max_object_size[`cloud_topics_compaction_max_object_size`]: Maximum size for L1 objects produced by compaction -* xref:reference:properties/cluster-properties.adoc#cloud_topics_epoch_service_max_same_epoch_duration[`cloud_topics_epoch_service_max_same_epoch_duration`]: Maximum duration a node can use the same epoch -* xref:reference:properties/cluster-properties.adoc#cloud_topics_fetch_debounce_enabled[`cloud_topics_fetch_debounce_enabled`]: Enable fetch debouncing -* xref:reference:properties/cluster-properties.adoc#cloud_topics_gc_health_check_interval[`cloud_topics_gc_health_check_interval`]: L0 garbage collector health check interval -* xref:reference:properties/cluster-properties.adoc#cloud_topics_l1_indexing_interval[`cloud_topics_l1_indexing_interval`]: Byte interval for index entries in long-term storage objects -* xref:reference:properties/cluster-properties.adoc#cloud_topics_long_term_file_deletion_delay[`cloud_topics_long_term_file_deletion_delay`]: Delay before deleting stale long-term files -* xref:reference:properties/cluster-properties.adoc#cloud_topics_long_term_flush_interval[`cloud_topics_long_term_flush_interval`]: Interval for flushing long-term storage metadata to object storage -* xref:reference:properties/cluster-properties.adoc#cloud_topics_metastore_lsm_apply_timeout_ms[`cloud_topics_metastore_lsm_apply_timeout_ms`]: Timeout for applying replicated writes to LSM database -* xref:reference:properties/cluster-properties.adoc#cloud_topics_metastore_replication_timeout_ms[`cloud_topics_metastore_replication_timeout_ms`]: Timeout for L1 metastore Raft replication -* xref:reference:properties/cluster-properties.adoc#cloud_topics_num_metastore_partitions[`cloud_topics_num_metastore_partitions`]: Number of partitions for the metastore topic -* xref:reference:properties/cluster-properties.adoc#cloud_topics_parallel_fetch_enabled[`cloud_topics_parallel_fetch_enabled`]: Enable parallel fetching -* xref:reference:properties/cluster-properties.adoc#cloud_topics_preregistered_object_ttl[`cloud_topics_preregistered_object_ttl`]: Time-to-live for pre-registered L1 objects -* xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_no_pid_concurrency[`cloud_topics_produce_no_pid_concurrency`]: Concurrent Raft requests for producers without a producer ID -* xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_write_inflight_limit[`cloud_topics_produce_write_inflight_limit`]: Maximum in-flight write requests per shard -* xref:reference:properties/cluster-properties.adoc#cloud_topics_reconciliation_max_interval[`cloud_topics_reconciliation_max_interval`]: Maximum reconciliation interval for adaptive scheduling -* xref:reference:properties/cluster-properties.adoc#cloud_topics_reconciliation_max_object_size[`cloud_topics_reconciliation_max_object_size`]: Maximum size for L1 objects produced by the reconciler -* xref:reference:properties/cluster-properties.adoc#cloud_topics_reconciliation_min_interval[`cloud_topics_reconciliation_min_interval`]: Minimum reconciliation interval for adaptive scheduling -* xref:reference:properties/cluster-properties.adoc#cloud_topics_reconciliation_parallelism[`cloud_topics_reconciliation_parallelism`]: Maximum concurrent objects built by reconciliation per shard -* xref:reference:properties/cluster-properties.adoc#cloud_topics_reconciliation_slowdown_blend[`cloud_topics_reconciliation_slowdown_blend`]: Blend factor for slowing down reconciliation -* xref:reference:properties/cluster-properties.adoc#cloud_topics_reconciliation_speedup_blend[`cloud_topics_reconciliation_speedup_blend`]: Blend factor for speeding up reconciliation -* xref:reference:properties/cluster-properties.adoc#cloud_topics_reconciliation_target_fill_ratio[`cloud_topics_reconciliation_target_fill_ratio`]: Target fill ratio for L1 objects -* xref:reference:properties/cluster-properties.adoc#cloud_topics_upload_part_size[`cloud_topics_upload_part_size`]: Part size for multipart uploads -* xref:reference:properties/cluster-properties.adoc#cloud_topics_epoch_service_epoch_increment_interval[`cloud_topics_epoch_service_epoch_increment_interval`]: Interval for cluster epoch incrementation -* xref:reference:properties/cluster-properties.adoc#cloud_topics_epoch_service_local_epoch_cache_duration[`cloud_topics_epoch_service_local_epoch_cache_duration`]: Cache duration for local epoch data -* xref:reference:properties/cluster-properties.adoc#cloud_topics_long_term_garbage_collection_interval[`cloud_topics_long_term_garbage_collection_interval`]: Interval for long-term storage garbage collection -* xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_batching_size_threshold[`cloud_topics_produce_batching_size_threshold`]: Object size threshold that triggers upload -* xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_cardinality_threshold[`cloud_topics_produce_cardinality_threshold`]: Partition cardinality threshold that triggers upload -* xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_upload_interval[`cloud_topics_produce_upload_interval`]: Time interval that triggers upload -* xref:reference:properties/cluster-properties.adoc#cloud_topics_reconciliation_interval[`cloud_topics_reconciliation_interval`]: Interval for moving data from short-term to long-term storage -* xref:reference:properties/cluster-properties.adoc#cloud_topics_short_term_gc_backoff_interval[`cloud_topics_short_term_gc_backoff_interval`]: Backoff interval for short-term storage garbage collection -* xref:reference:properties/cluster-properties.adoc#cloud_topics_short_term_gc_interval[`cloud_topics_short_term_gc_interval`]: Interval for short-term storage garbage collection -* xref:reference:properties/cluster-properties.adoc#cloud_topics_short_term_gc_minimum_object_age[`cloud_topics_short_term_gc_minimum_object_age`]: Minimum age for objects to be eligible for short-term garbage collection - -**Object storage:** - -* xref:reference:properties/object-storage-properties.adoc#cloud_storage_gc_max_segments_per_run[`cloud_storage_gc_max_segments_per_run`]: Maximum number of log segments to delete from object storage during each xref:manage:tiered-storage.adoc#object-storage-housekeeping[housekeeping run] -* xref:reference:properties/object-storage-properties.adoc#cloud_storage_prefetch_segments_max[`cloud_storage_prefetch_segments_max`]: Maximum number of small segments to prefetch during sequential reads - -**Authentication:** - -* xref:reference:properties/cluster-properties.adoc#nested_group_behavior[`nested_group_behavior`]: Control how Redpanda handles nested groups extracted from authentication tokens -* xref:reference:properties/cluster-properties.adoc#oidc_group_claim_path[`oidc_group_claim_path`]: JSON path to extract groups from the JWT payload -* xref:reference:properties/cluster-properties.adoc#schema_registry_enable_qualified_subjects[`schema_registry_enable_qualified_subjects`]: Enable parsing of qualified subject syntax in Schema Registry - -**Other:** - -* xref:reference:properties/cluster-properties.adoc#delete_topic_enable[`delete_topic_enable`]: Enable or disable topic deletion via the Kafka DeleteTopics API -* xref:reference:properties/cluster-properties.adoc#internal_rpc_request_timeout_ms[`internal_rpc_request_timeout_ms`]: Default timeout for internal RPC requests between nodes -* xref:reference:properties/cluster-properties.adoc#log_compaction_max_priority_wait_ms[`log_compaction_max_priority_wait_ms`]: Maximum time a priority partition (such as `__consumer_offsets`) waits before preempting regular compaction -* xref:reference:properties/cluster-properties.adoc#partition_autobalancing_node_autodecommission_timeout_sec[`partition_autobalancing_node_autodecommission_timeout_sec`]: Duration a node must be unavailable before Redpanda automatically decommissions it - -=== Changes to default values - -* xref:reference:properties/cluster-properties.adoc#log_compaction_tx_batch_removal_enabled[`log_compaction_tx_batch_removal_enabled`]: Changed from `false` to `true`. -* xref:reference:properties/cluster-properties.adoc#tls_v1_2_cipher_suites[`tls_v1_2_cipher_suites`]: Changed from OpenSSL cipher names to IANA cipher names. - -==== v26.1.4 - -* xref:reference:properties/cluster-properties.adoc#max_concurrent_producer_ids[`max_concurrent_producer_ids`]: Changed from unlimited to `100000` to prevent unbounded memory growth with heavy producer usage. - -* xref:reference:properties/cluster-properties.adoc#max_transactions_per_coordinator[`max_transactions_per_coordinator`]: Changed from unlimited to `10000` to prevent resource exhaustion from excessive transaction sessions. - -=== Removed properties - -The following deprecated configuration properties have been removed in v26.1.1. If you have any of these in your configuration files, update them according to the guidance below. - -**RPC timeout properties:** - -Replace with xref:reference:properties/cluster-properties.adoc#internal_rpc_request_timeout_ms[`internal_rpc_request_timeout_ms`]. - -* `alter_topic_cfg_timeout_ms` -* `create_topic_timeout_ms` -* `metadata_status_wait_timeout_ms` -* `node_management_operation_timeout_ms` -* `recovery_append_timeout_ms` -* `rm_sync_timeout_ms` -* `tm_sync_timeout_ms` -* `wait_for_leader_timeout_ms` - -**Client throughput quota properties:** - -Use xref:reference:rpk/rpk-cluster/rpk-cluster-quotas.adoc[`rpk cluster quotas`] to manage xref:manage:cluster-maintenance/manage-throughput.adoc#client-throughput-limits[client throughput limits]. - -* `kafka_admin_topic_api_rate` -* `kafka_client_group_byte_rate_quota` -* `kafka_client_group_fetch_byte_rate_quota` -* `target_fetch_quota_byte_rate` -* `target_quota_byte_rate` - -**Quota balancer properties:** - -Use xref:manage:cluster-maintenance/manage-throughput.adoc#broker-wide-throughput-limit-properties[broker-wide throughput limit properties]. - -* `kafka_quota_balancer_min_shard_throughput_bps` -* `kafka_quota_balancer_min_shard_throughput_ratio` -* `kafka_quota_balancer_node_period` -* `kafka_quota_balancer_window` -* `kafka_throughput_throttling_v2` - -**Timestamp alert properties:** - -* `log_message_timestamp_alert_after_ms`: Use xref:reference:properties/cluster-properties.adoc#log_message_timestamp_after_max_ms[`log_message_timestamp_after_max_ms`] -* `log_message_timestamp_alert_before_ms`: Use xref:reference:properties/cluster-properties.adoc#log_message_timestamp_before_max_ms[`log_message_timestamp_before_max_ms`] - -**Other removed properties:** - -No replacement needed. These properties were deprecated placeholders that have been silently ignored and will continue to be ignored even after removal. +**Iceberg:** -* `cloud_storage_disable_metadata_consistency_checks` -* `cloud_storage_reconciliation_ms` -* `coproc_max_batch_size` -* `coproc_max_inflight_bytes` -* `coproc_max_ingest_bytes` -* `coproc_offset_flush_interval_ms` -* `datalake_disk_space_monitor_interval` -* `enable_admin_api` -* `enable_coproc` -* `find_coordinator_timeout_ms` -* `full_raft_configuration_recovery_pattern` -* `id_allocator_replication` -* `kafka_memory_batch_size_estimate_for_fetch` -* `log_compaction_adjacent_merge_self_compaction_count` -* `max_version` -* `min_version` -* `raft_max_concurrent_append_requests_per_follower` -* `raft_recovery_default_read_size` -* `rm_violation_recovery_policy` -* `schema_registry_protobuf_renderer_v2` -* `seed_server_meta_topic_partitions` -* `seq_table_min_size` -* `tm_violation_recovery_policy` -* `transaction_coordinator_replication` -* `tx_registry_log_capacity` -* `tx_registry_sync_timeout_ms` -* `use_scheduling_groups` +* xref:reference:properties/topic-properties.adoc#redpanda-schema-registry-context[`redpanda.schema.registry.context`]: Bind a topic's Iceberg translation to a specific Schema Registry context diff --git a/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc b/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc index 6104223a1c..0139bcbc85 100644 --- a/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc +++ b/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc @@ -27,6 +27,7 @@ Use this page to: If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda by default writes the record to a separate DLQ Iceberg table named `~dlq`. The following can cause errors to occur when translating records in the `value_schema_id_prefix` and `value_schema_latest` modes to the Iceberg table format: - Redpanda cannot find the embedded schema ID in the Schema Registry. +- Redpanda cannot resolve the record's schema ID within the topic's configured Schema Registry context. For example, the schema exists only in a different context. See xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context]. - Redpanda fails to translate one or more schema data types to an Iceberg type. - In `value_schema_id_prefix` mode, you do not use the Schema Registry wire format with the magic byte. diff --git a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc index 672d12cb7e..2540a5f892 100644 --- a/modules/manage/pages/iceberg/specify-iceberg-schema.adoc +++ b/modules/manage/pages/iceberg/specify-iceberg-schema.adoc @@ -109,6 +109,43 @@ value_schema_latest:subject=,protobuf_name= --topic-config redpanda.schema.registry.context= +---- + +.Set the Schema Registry context on an existing topic +[,bash] +---- +rpk topic alter-config --set redpanda.schema.registry.context= +---- + +The context name must start with a period (`.`), for example `.staging`. If you don't set this property, Redpanda resolves schemas in the default context (`.`). + +If Redpanda cannot resolve a record's schema ID within the configured context, it doesn't translate the record and instead writes it to a dead-letter queue (DLQ) table. See xref:manage:iceberg/iceberg-troubleshooting.adoc[Troubleshoot Iceberg Topics]. + +[IMPORTANT] +==== +Redpanda resolves schema IDs using the topic's current `redpanda.schema.registry.context` value, not the context that was active when a record was ingested. Changing the context on a topic that has pending, uncommitted Iceberg translation entries can cause resolution errors or unexpected DLQ routing for records still being translated. + +To change the Schema Registry context on a topic that is actively translating to Iceberg: + +. Disable Iceberg translation for the topic. +. Wait for pending translation entries to commit. +. Change `redpanda.schema.registry.context`. +. Re-enable Iceberg translation. +==== + == How Iceberg modes translate to table format ifndef::env-cloud[] diff --git a/modules/manage/pages/schema-reg/schema-reg-contexts.adoc b/modules/manage/pages/schema-reg/schema-reg-contexts.adoc index 85ebdfa7e8..2e984d4614 100644 --- a/modules/manage/pages/schema-reg/schema-reg-contexts.adoc +++ b/modules/manage/pages/schema-reg/schema-reg-contexts.adoc @@ -79,7 +79,6 @@ The `schema_registry_enable_qualified_subjects` property defaults to `false`, so * *Non-Java SerDe clients*: Not supported in Schema Registry contexts. * *Server-side schema ID validation*: Schema ID validation using Kafka record headers does not support contexts. However, schema ID validation using magic byte and prefix are supported. -* *Iceberg topics*: You cannot use schemas within a context for Iceberg Topics. * *`referencedby` endpoint*: `GET /subjects/\{subject\}/versions/\{version\}/referencedby` returns a list of bare schema IDs with no context information. When references span contexts, it is not possible to determine which context each returned ID belongs to. * *Cross-context isolation*: Contexts provide organizational and ID-space isolation, but do not prevent cross-context schema references. There is no mechanism to block schemas in one context from referencing schemas in another context. * *Default context cannot be deleted*: You cannot delete the default context (`.`). From 0734ec933b698149992773549ade909face9ffe4 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Fri, 10 Jul 2026 12:25:14 -0700 Subject: [PATCH 2/2] Add topic property reference --- docs-data/property-overrides.json | 10 +++++ .../pages/properties/topic-properties.adoc | 8 +++- .../partials/properties/topic-properties.adoc | 38 +++++++++++++++++++ 3 files changed, 55 insertions(+), 1 deletion(-) diff --git a/docs-data/property-overrides.json b/docs-data/property-overrides.json index 9d34ab7769..017e6cca63 100644 --- a/docs-data/property-overrides.json +++ b/docs-data/property-overrides.json @@ -1813,6 +1813,16 @@ ], "config_scope": "topic" }, + "redpanda.schema.registry.context": { + "description": "The xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] Redpanda uses to resolve schema IDs for records in this topic, for example when translating the topic to an Iceberg table. Defaults to the Schema Registry default context (`.`).", + "related_topics": [ + "xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context]" + ], + "config_scope": "topic", + "category": "schema-registry", + "type": "string", + "default": "." + }, "redpanda.storage.mode": { "description": "The storage mode for a topic. Determines how topic data is stored and whether it is eligible for upload to object storage.\n\nAccepted values:\n\n* `local`: Topic data is stored only on the broker's local disk. Object storage upload is disabled for the topic, regardless of cluster-level Tiered Storage settings.\nifndef::env-cloud[]\n* `tiered`: Topic data is stored on local disk and also uploaded to object storage. Enables xref:manage:tiered-storage.adoc[Tiered Storage] for the topic.\n* `cloud`: Topic data is stored in object storage using the xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] architecture. Local storage is used only as a write buffer.\nendif::[]\nifdef::env-cloud[]\n* `tiered`: Topic data is stored on local disk and also uploaded to object storage. Enables Tiered Storage for the topic.\n* `cloud`: Topic data is stored in object storage using the Cloud Topics architecture. Local storage is used only as a write buffer.\nendif::[]\n* `unset`: Specifies that the topic's storage mode is unset, regardless of the cluster default. The topic may still have Tiered Storage enabled through the legacy properties `redpanda.remote.read` and `redpanda.remote.write`.\n\nThis property overrides the cluster-wide config_ref:default_redpanda_storage_mode,true,properties/cluster-properties[] setting for individual topics.", "related_topics": [ diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index e99bff7283..31d8d887b1 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -155,4 +155,10 @@ include::reference:partial$properties/topic-properties.adoc[tags=category-remote Integrate Redpanda topics as Iceberg tables. -include::reference:partial$properties/topic-properties.adoc[tags=category-iceberg-integration;!deprecated;!exclude-from-docs] \ No newline at end of file +include::reference:partial$properties/topic-properties.adoc[tags=category-iceberg-integration;!deprecated;!exclude-from-docs] + +== Schema Registry and Validation Properties + +The following properties control server-side schema ID validation and schema resolution for topics when using Schema Registry. + +include::reference:partial$properties/topic-properties.adoc[tags=category-schema-registry;!deprecated;!exclude-from-docs] \ No newline at end of file diff --git a/modules/reference/partials/properties/topic-properties.adoc b/modules/reference/partials/properties/topic-properties.adoc index 4f180a60fe..1f90330951 100644 --- a/modules/reference/partials/properties/topic-properties.adoc +++ b/modules/reference/partials/properties/topic-properties.adoc @@ -1190,6 +1190,44 @@ endif::[] // end::category-iceberg-integration[] +// tag::category-schema-registry[] +=== redpanda.schema.registry.context + +The xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] Redpanda uses to resolve schema IDs for records in this topic, for example when translating the topic to an Iceberg table. + +[cols="1s,2a"] +|=== +| Property | Value + +| Type +| `string` + + + +| Default +| +ifdef::env-cloud[] +Available in the Redpanda Cloud Console +endif::[] +ifndef::env-cloud[] +`.` +endif::[] + +| Nullable +| No + +ifndef::env-cloud[] +| Restored on xref:manage:whole-cluster-restore.adoc[Whole Cluster Restore] +| Yes +endif::[] + +| Related topics +|xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context] + +|=== + +// end::category-schema-registry[] + // tag::category-schema-registry[] === redpanda.key.schema.id.validation