Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@
*** xref:manage:kubernetes/k-manage-topics.adoc[Manage Topics]
*** xref:manage:kubernetes/k-cloud-topics.adoc[Cloud Topics]
*** xref:manage:kubernetes/k-manage-connectors.adoc[Manage Kafka Connect]
*** xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines]
*** xref:manage:kubernetes/storage/index.adoc[Storage]
**** xref:manage:kubernetes/storage/k-volume-types.adoc[Volume Types]
**** xref:manage:kubernetes/storage/k-configure-storage.adoc[Configure Storage]
Expand Down
30 changes: 30 additions & 0 deletions modules/manage/pages/kubernetes/k-connect-pipelines.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
= Redpanda Connect Pipelines in Kubernetes
:description: Run Redpanda Connect pipelines declaratively with the Redpanda Operator using the Pipeline resource.
:env-kubernetes: true
:page-categories: Management, Integration
:page-topic-type: concept

// DRAFT (DOC-2275): Hold until the Pipeline resource ships in a released operator
// and rp-connect-docs#457 is published. See that PR for the full guide this page routes to.

Run xref:connect:home:index.adoc[Redpanda Connect] pipelines as Kubernetes resources managed by the Redpanda Operator. You define each pipeline in a `Pipeline` custom resource, and the operator deploys it, validates its configuration before startup, rolls it on every change, and reports its health in the resource status. Pipelines follow the same declarative workflow as your Topic and User resources, so you can build and operate streaming data pipelines with the same GitOps tooling you already use for cluster resources. This page explains when to choose the Pipeline resource over other options and where to find the deployment guide.

This feature requires an xref:get-started:licensing/overview.adoc[Enterprise Edition license] and Redpanda Operator <operator-version> or later.
// TODO(DOC-2275): Set the real minimum operator version when the feature ships.

== When to use the Pipeline resource

Use the Pipeline resource when you want:

* *Declarative, GitOps-friendly management*: Pipelines are plain Kubernetes manifests that you can version, review, and deploy with tools such as Argo CD or Flux.
* *Validated rollouts*: The operator lints every configuration before the pipeline starts, and rolls the workload automatically when the configuration changes.
* *Cluster integration*: A pipeline can reference a Redpanda resource in the same namespace, and the operator injects the connection details, TLS material, and credentials for you.
* *Built-in connectors*: Redpanda Connect ships hundreds of connectors and processors, including AI integrations, without the JVM or plugin management that Kafka Connect requires.

For Kafka Connect, which is community-supported with Redpanda, see xref:manage:kubernetes/k-manage-connectors.adoc[]. To run Redpanda Connect on Kubernetes without the operator, see xref:connect:install:helm-chart.adoc[].

== Get started

For installation steps, pipeline examples, secrets handling, and lifecycle management, see xref:connect:install:kubernetes-operator.adoc[Run Pipelines with the Redpanda Operator] in the Redpanda Connect documentation.

For the Pipeline CRD source file, see the xref:reference:k-crd-index.adoc[CRD reference].
2 changes: 1 addition & 1 deletion modules/manage/pages/kubernetes/k-manage-connectors.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
:page-categories: Management, Integration
:env-kubernetes: true

When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API.
When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API. To build streaming data pipelines with enterprise support and without Kafka Connect infrastructure, consider running xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] with the Redpanda Operator instead.

include::shared:partial$community-supported-connectors.adoc[]

Expand Down
4 changes: 4 additions & 0 deletions modules/reference/pages/k-crd-index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ The CRD source definitions are maintained in the https://github.com/redpanda-dat
| https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_consoles.yaml[`cluster.redpanda.com_consoles.yaml`^]
| Defines Redpanda Console resources.

| Pipeline
| https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_pipelines.yaml[`cluster.redpanda.com_pipelines.yaml`^]
| Defines Redpanda Connect pipeline resources. See xref:connect:install:kubernetes-operator.adoc[Run Pipelines with the Redpanda Operator].

| Schema
| https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_schemas.yaml[`cluster.redpanda.com_schemas.yaml`^]
| Defines Schema Registry resources.
Expand Down
Loading