Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
= Manage Secrets
:description: Learn how to manage secrets in Redpanda Connect using the Cloud UI or Data Plane API, and how to add them to your data pipelines.
:description: Learn how to manage secrets in Redpanda Connect using the Cloud UI, Data Plane API, or Terraform, and how to add them to your data pipelines.

Learn how to manage secrets in Redpanda Connect, and how to add them to your data pipelines without exposing them.

Secrets are stored in the secret management solution of your cloud provider and are retrieved when you run a pipeline configuration that references them.

== Manage secrets

You can manage secrets from the Cloud UI or the Data Plane API.
You can manage secrets from the Cloud UI or the Data Plane API. If you manage your Redpanda Cloud resources with Terraform, you can also create secrets with the `redpanda_secret` resource. See xref:manage:terraform-provider.adoc#manage-cluster-secrets[Manage cluster secrets].

=== Create a secret

Expand Down
6 changes: 5 additions & 1 deletion modules/get-started/pages/whats-new-cloud.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ You can now permanently delete a Serverless organization (free trial and pay-as-

The Redpanda Terraform provider (v2.1.0+) now supports enabling and managing Redpanda SQL on BYOC clusters on AWS. Use the `rpsql` block on a `redpanda_cluster` resource to enable SQL, configure compute replicas, and retrieve the SQL endpoint URL. See xref:manage:terraform-provider.adoc#enable-redpanda-sql-on-a-byoc-cluster[Enable Redpanda SQL on a BYOC cluster].

=== Terraform provider: Secrets management

The Redpanda Terraform provider (v2.0.0+) now supports managing cluster secrets with the new `redpanda_secret` resource. The secret value is a write-only attribute, so Terraform never stores it in your state file, and scopes control which features can use the secret, such as Redpanda Connect pipelines. See xref:manage:terraform-provider.adoc#manage-cluster-secrets[Manage cluster secrets].

=== Centralized egress for BYOC on GCP: beta

You can route all GCP BYOC cluster egress through your own GCP hub VPC and NAT VM instead of a per-cluster Cloud NAT, so outbound traffic exits through your centralized inspection point. This is useful for regulated environments that require a single, predictable public IP for outbound allowlisting or that prohibit per-cluster Cloud NAT. Centralized egress is in a glossterm:beta[] release and is enabled per organization. Contact your account team for access. See xref:networking:byoc/gcp/nat-free-egress.adoc[Configure Centralized Egress with GCP VPC Peering].
Expand Down Expand Up @@ -233,7 +237,7 @@ You can now enable xref:manage:schema-reg/schema-id-validation.adoc[schema ID va

=== Cross-region AWS PrivateLink

AWS PrivateLink now supports cross-region connectivity, allowing clients in different AWS regions to connect to your Redpanda cluster through PrivateLink. Configure supported regions in the xref:networking:configure-privatelink-in-cloud-ui.adoc#cross-region-privatelink[Cloud UI] or using the xref:networking:aws-privatelink.adoc#cross-region-privatelink[Cloud API] to specify which regions can establish PrivateLink connections. This feature requires multi-AZ cluster deployments.
AWS PrivateLink now supports cross-region connectivity, allowing clients in different AWS regions to connect to your Redpanda cluster through PrivateLink. Configure supported regions in the xref:networking:configure-privatelink-in-cloud-ui.adoc#cross-region-privatelink[Cloud UI], using the xref:networking:aws-privatelink.adoc#cross-region-privatelink[Cloud API], or with the Redpanda Terraform provider (v1.7.0+) using the `supported_regions` attribute of the `aws_private_link` block. See xref:manage:terraform-provider.adoc#configure-cross-region-aws-privatelink[Configure cross-region AWS PrivateLink]. This feature requires multi-AZ cluster deployments.

== January 2026

Expand Down
83 changes: 81 additions & 2 deletions modules/manage/pages/terraform-provider.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
:learning-objective-1: Configure and deploy Redpanda Cloud clusters using Terraform
:learning-objective-2: Manage sensitive credentials using Terraform write-only attributes
:learning-objective-3: Enable Redpanda SQL on a BYOC cluster using the rpsql block
:learning-objective-4: Configure cross-region AWS PrivateLink on a cluster

Use the https://registry.terraform.io/providers/redpanda-data/redpanda/latest[Redpanda Terraform provider^] to define, automate, and track changes to your Redpanda Cloud infrastructure as code.

Expand All @@ -19,6 +20,7 @@ With the Redpanda Terraform provider, you can manage:
* link:https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/role_assignments[Role assignments^]
* link:https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/schema[Schemas^]
* link:https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/schema_registry_acl[Schema Registry ACLs^]
* link:https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/secret[Secrets^]
* link:https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/serverless_cluster[Serverless clusters^]
* link:https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/serverless_private_link[Serverless private links^]
* link:https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/topic[Topics^]
Expand All @@ -29,6 +31,7 @@ After reading this page, you will be able to:
* [ ] {learning-objective-1}
* [ ] {learning-objective-2}
* [ ] {learning-objective-3}
* [ ] {learning-objective-4}

== Why use Terraform with Redpanda?

Expand Down Expand Up @@ -109,7 +112,6 @@ output "cluster_api_url" {
The following functionality is supported in the Cloud API but not in the Redpanda Terraform provider:

* Creating or deleting BYOVNet clusters on Azure
* Secrets
* Kafka Connect

[WARNING]
Expand Down Expand Up @@ -240,7 +242,7 @@ For each supported sensitive field, the provider exposes two new attributes alon
* `<field>_wo`: A write-only attribute. Terraform sends the value to the provider during `apply` but never persists it to state.
* `<field>_wo_version`: An integer version. Because Terraform cannot detect changes in a write-only value (there is nothing to compare against in state), you increment this number to signal that the value has changed and to trigger an update on the next apply.

NOTE: `redpanda_pipeline` is an exception to this naming convention. The existing `client_secret` attribute is the write-only attribute (no separate `client_secret_wo` field), and is paired with `secret_version` instead of `client_secret_wo_version`.
NOTE: `redpanda_pipeline` and `redpanda_secret` are exceptions to this naming convention. For `redpanda_pipeline`, the existing `client_secret` attribute is the write-only attribute (no separate `client_secret_wo` field), and is paired with `secret_version` instead of `client_secret_wo_version`. For `redpanda_secret`, the `secret_data` attribute is write-only by design (there is no plaintext variant), and is paired with `secret_data_version`.

The provider retains the original plaintext attributes for backward compatibility. You can migrate to the write-only variants on your own schedule. Avoid setting both the plaintext attribute and its write-only counterpart on the same resource. If both are set, the provider uses the write-only value.

Expand Down Expand Up @@ -268,6 +270,11 @@ The provider retains the original plaintext attributes for backward compatibilit
| `client_secret`
| `client_secret` (write-only)
| `secret_version`

| `redpanda_secret`
| None (write-only by design)
| `secret_data`
| `secret_data_version`
|===

=== Set a write-only attribute
Expand Down Expand Up @@ -319,6 +326,36 @@ resource "redpanda_user" "schema_user" {

After running `terraform apply`, the provider sends the new password to Redpanda Cloud. Neither the old nor the new value is written to state.

== Manage cluster secrets

The Redpanda Terraform provider (v2.0.0+) supports creating and managing secrets in a cluster's secret store with the `redpanda_secret` resource. Cluster secrets hold sensitive values, such as pipeline credentials, that other resources reference by name using `${secrets.<NAME>}`.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

The secret value (`secret_data`) is a write-only attribute, so Terraform never stores it in state. To rotate the value, change `secret_data` and increment `secret_data_version`, following the same pattern as <<manage-sensitive-attributes-with-write-only-fields,other write-only attributes>>.

[source,hcl]
----
variable "secret_value" {
description = "Value to store in the secret"
sensitive = true
}

resource "redpanda_secret" "example" {
name = "EXAMPLE_SECRET"
secret_data = var.secret_value
secret_data_version = 1
scopes = ["SCOPE_REDPANDA_CONNECT"]
cluster_api_url = data.redpanda_cluster.byoc.cluster_api_url
allow_deletion = true
Comment thread
coderabbitai[bot] marked this conversation as resolved.
}
----

* `name`: An uppercase identifier matching `^[A-Z][A-Z0-9_]*$`. You cannot rename a secret after you create it.
* `secret_data`: The secret value. Requires Terraform 1.11 or later.
* `scopes`: The features that can use the secret. For example, use `SCOPE_REDPANDA_CONNECT` for Redpanda Connect pipelines or `SCOPE_REDPANDA_CLUSTER` for Shadowing credentials. If a secret doesn't have a scope that matches its consumer, the consumer fails at runtime.
* `labels` (optional): Labels to organize your secrets. You can update labels on an existing secret (provider v2.1.0+).

For the full schema and import syntax, see the https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/secret[redpanda_secret resource documentation^]. To use secrets in Redpanda Connect pipelines, see xref:develop:connect/configuration/secret-management.adoc[].

== Configure cluster properties

To set xref:reference:properties/cluster-properties.adoc[cluster configuration properties] on a BYOC or Dedicated cluster, add a `cluster_configuration` block to the `redpanda_cluster` resource and pass the properties as a JSON string with `custom_properties_json`:
Expand All @@ -340,6 +377,46 @@ Only the properties listed in xref:reference:properties/cluster-properties.adoc[

NOTE: Cluster properties are not available on Serverless clusters or on BYOC and Dedicated clusters running on Azure.

== Configure cross-region AWS PrivateLink

By default, AWS PrivateLink only accepts connections from VPCs in the same region as your Redpanda cluster. The Redpanda Terraform provider (v1.7.0+) supports xref:networking:aws-privatelink.adoc#cross-region-privatelink[cross-region PrivateLink], which allows clients in other AWS regions to connect to your cluster through PrivateLink.

To enable cross-region PrivateLink, add the `supported_regions` attribute to the `aws_private_link` block of a `redpanda_cluster` resource. The attribute accepts a list of up to 50 unique AWS region identifiers from which PrivateLink endpoints can connect. If you omit it, only same-region connections are allowed.

Requirements:

* The cluster must be deployed across multiple availability zones (multi-AZ). This is an AWS limitation for cross-region PrivateLink.
* For BYOC clusters, the Redpanda agent IAM role must have the `vpce:AllowMultiRegion` and `elasticloadbalancing:DescribeListenerAttributes` permissions.

[source,hcl]
----
resource "redpanda_cluster" "example" {
# ... other cluster arguments ...

zones = ["use2-az1", "use2-az2", "use2-az3"] # Cross-region PrivateLink requires a multi-AZ cluster

aws_private_link = {
enabled = true
connect_console = true
allowed_principals = ["arn:aws:iam::123456789012:root"]
supported_regions = ["us-east-1", "us-west-2", "eu-west-1"]
}
}

output "privatelink_service_name" {
value = redpanda_cluster.example.aws_private_link.status.service_name
}
----

* `enabled`: Whether the Redpanda PrivateLink endpoint service is enabled.
* `connect_console`: Whether Redpanda Console is connected over PrivateLink.
* `allowed_principals`: The ARNs of the AWS principals allowed to access the endpoint service. To allow all principals, use an asterisk (`*`).
* `supported_regions` (optional): The AWS regions from which clients can create PrivateLink endpoints that connect to the cluster.

You can add or change the `aws_private_link` block on an existing cluster. Terraform updates the cluster in place without recreating it.

After you apply the configuration, clients in the listed regions can create VPC endpoints that connect to the cluster's endpoint service, identified by the `service_name` output in the example. See xref:networking:aws-privatelink.adoc#create-a-cross-region-vpc-endpoint[Create a cross-region VPC endpoint] for the AWS CLI workflow, and xref:networking:aws-privatelink.adoc[] for the full PrivateLink setup, including client VPC configuration and connection testing.

== Examples

The following examples use the Redpanda Terraform provider to create and manage clusters. For descriptions of resources and data sources, see the https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs[Redpanda Terraform Provider documentation^].
Expand Down Expand Up @@ -638,6 +715,8 @@ resource "redpanda_cluster" "test" {
}
----

For details on the `aws_private_link` block, including the multi-AZ requirement for cross-region PrivateLink, see <<configure-cross-region-aws-privatelink>>.

=== Create a Serverless cluster

A Serverless cluster is cost-effective and scales automatically based on usage. This example creates a cluster in the `us-east-1` region with minimal configuration.
Expand Down
2 changes: 1 addition & 1 deletion modules/networking/pages/aws-privatelink.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,7 @@ For more information about AWS cross-region PrivateLink support, see the https:/

=== Configure cross-region PrivateLink

To enable cross-region PrivateLink, add the `supported_regions` field to your `aws_private_link` configuration when <<create-new-cluster-with-privatelink-endpoint-service-enabled,creating a new cluster>> or <<enable-privatelink-endpoint-service-for-existing-clusters,enabling PrivateLink on an existing cluster>>.
To enable cross-region PrivateLink, add the `supported_regions` field to your `aws_private_link` configuration when <<create-new-cluster-with-privatelink-endpoint-service-enabled,creating a new cluster>> or <<enable-privatelink-endpoint-service-for-existing-clusters,enabling PrivateLink on an existing cluster>>. You can also configure it with the Redpanda Terraform provider. See xref:manage:terraform-provider.adoc#configure-cross-region-aws-privatelink[Configure cross-region AWS PrivateLink].

The `supported_regions` field accepts a list of AWS region identifiers where you want to allow PrivateLink connections from. For example:

Expand Down
2 changes: 2 additions & 0 deletions modules/security/pages/secrets.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,5 @@ https://cloud.google.com/secret-manager[GCP Secret Manager^] services. Static
secrets managed through Redpanda Console never leave their corresponding
data plane account or network. They stay securely stored in AWS Secrets Manager or
GCP Secret Manager.

To manage the static secrets in a cluster's secret store, use the Cloud UI, the link:/api/doc/cloud-dataplane/topic/topic-quickstart[Data Plane API], or the xref:manage:terraform-provider.adoc#manage-cluster-secrets[Redpanda Terraform provider].
Loading