Skip to content

quickstart.mdx

Latest commit

 

History

History
176 lines (136 loc) · 4.41 KB

quickstart.mdx

File metadata and controls

176 lines (136 loc) · 4.41 KB
Raw
title Direct-mode Quickstart
description Build a direct-mode pipeline from an adopted ClickHouse stream table.

This guide creates one local ClickHouse table, adopts it as a replay source, and builds a live materialized-view target with stb build.

Prerequisites

1. Install StreamBuild

uv tool install streambuild
stb --help

2. Start ClickHouse

docker rm --force --volumes streambuild-clickhouse >/dev/null 2>&1 || true
docker run --detach --name streambuild-clickhouse \
  --env CLICKHOUSE_SKIP_USER_SETUP=1 \
  --publish 18123:8123 --publish 19000:9000 \
  clickhouse/clickhouse-server:24.8

until curl --fail --silent 'http://localhost:18123/ping' >/dev/null; do
  sleep 1
done

Create an input table with offset and timestamp columns. ClickHouse HTTP accepts one statement per request:

curl 'http://localhost:18123/' \
  --data-binary 'CREATE DATABASE IF NOT EXISTS quickstart'

curl 'http://localhost:18123/' --data-binary '
CREATE TABLE IF NOT EXISTS quickstart.orders_input (
  event_partition Int32,
  event_offset Int64,
  event_time DateTime64(3),
  order_id String,
  amount_cents UInt64
) ENGINE = MergeTree ORDER BY (event_partition, event_offset)
'

curl 'http://localhost:18123/' \
  --data-binary 'TRUNCATE TABLE quickstart.orders_input'

curl 'http://localhost:18123/' --data-binary "
INSERT INTO quickstart.orders_input VALUES
  (0, 1, now64(3), 'order-1', 2500),
  (0, 2, now64(3), 'order-2', 4100)
"

3. Create the project

quickstart/
  streambuild_project.toml
  sources/
    orders.yml
  pipelines/
    orders/
      order_totals.sql

quickstart/streambuild_project.toml:

name = "quickstart"
default_target = "local"

[connection]
host = "localhost"
port = 18123
username = "default"
password = ""

[targets.local]
database = "quickstart"

Direct mode is the default, so this file intentionally omits [defaults].pipeline_mode.

quickstart/sources/orders.yml:

sources:
  - name: orders
    kind: stream_table
    table_name: orders_input
    replay_boundary:
      mode: offsets
      columns:
        _replay_partition: event_partition
        _replay_offset: event_offset
        _replay_timestamp: event_time

quickstart/pipelines/orders/order_totals.sql:

MODEL (
  engine "MergeTree()",
  order_by ["order_id", "_replay_partition", "_replay_offset"],
);

SELECT
  CAST(order_id AS String) AS order_id,
  CAST(amount_cents AS UInt64) AS amount_cents,
  CAST(_replay_partition AS Int32) AS _replay_partition,
  CAST(_replay_offset AS Int64) AS _replay_offset,
  CAST(_replay_timestamp AS DateTime64(3)) AS _replay_timestamp
FROM __source("orders")

The direct child directory pipelines/orders/ defines the orders pipeline. StreamBuild infers its source by following the model's __source("orders") driving input to the source registry.

4. Discover and compile

stb discover --project-dir quickstart
stb compile --project-dir quickstart

Compilation is offline. Review quickstart/target/manifest.json, quickstart/target/streambuild_dag.json, and the generated SQL under quickstart/target/compiled/.

5. Plan and build

stb plan --project-dir quickstart
stb build --project-dir quickstart

Review the destructive plan and approve it with y. The build preserves orders_input, creates tbl__order_totals and mv__order_totals, then replays retained rows through the model.

curl 'http://localhost:18123/?query=SELECT%20*%20FROM%20quickstart.tbl__order_totals%20ORDER%20BY%20order_id'
Replay uses an inclusive cutoff and is intentionally at least once around that boundary. Choose model SQL and ClickHouse engines that tolerate duplicate delivery where required.

The startup command replaces any previous quickstart container, and the input setup truncates the source table, so you can repeat the guide without duplicating input rows.

Clean up

docker rm --force --volumes streambuild-clickhouse

This removes the quickstart container and its anonymous ClickHouse data volume.

Next steps