diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md new file mode 100644 index 000000000..1ebecb3b4 --- /dev/null +++ b/documentation/concepts/live-views.md @@ -0,0 +1,328 @@ +--- +title: Live views +sidebar_label: Live views +description: + Live views incrementally maintain window-function results over a base table so + that running totals, moving averages, and rankings can be read like a regular + table without recomputing on every query. +--- + +A live view is a QuestDB table that stores the incrementally maintained result +of a window-function query over a single base table. As new rows arrive in the +base table, the window functions run once per new row and the output is appended +to the view. Querying the live view then scans precomputed rows instead of +reprocessing the base table on every read. + +Live views target workloads where the same window aggregate is read frequently +against high-rate ingestion: rolling VWAP, cumulative volume, running ranks, or +day-over-day comparisons that would otherwise recompute a window over millions +of rows on each query. + +:::note + +Live views are a new feature. The supported SQL surface is deliberately narrow +in this first version. See [Limitations](#limitations) for the shapes that are +rejected at creation time. + +::: + +## Live views vs materialized views + +Both feature types pre-compute a query and refresh it incrementally, but they +serve different query shapes: + +| Aspect | Live view | [Materialized view](/docs/concepts/materialized-views/) | +| ------ | --------- | --------------------- | +| Query shape | Window functions (`OVER`) | `SAMPLE BY` / time-based `GROUP BY` | +| Output cardinality | One row per base row | One row per time bucket | +| Typical use | Running totals, moving averages, rankings | OHLC bars, downsampled summaries | +| Base tables | A single WAL-backed table | One or more tables (JOINs allowed) | +| Freshness / durability | `FLUSH EVERY`, `IN MEMORY` | `REFRESH` strategy | + +Use a materialized view when you want to aggregate rows into time buckets. Use a +live view when you want to keep a row-per-input result of a window computation. + +## Quick example + +Given a `trades` table of incoming trades: + +```questdb-sql title="Base table" +CREATE TABLE trades ( + symbol SYMBOL, + side SYMBOL, + price DOUBLE, + amount DOUBLE, + timestamp TIMESTAMP +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; +``` + +Create a live view that keeps a 300-row moving average of price per symbol: + +```questdb-sql title="Live view with a moving average" +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +IN MEMORY 5s +AS +SELECT + timestamp, + symbol, + price, + avg(price) OVER ( + PARTITION BY symbol + ORDER BY timestamp + ROWS 300 PRECEDING + ) AS moving_avg +FROM trades; +``` + +Query it like any table: + +```questdb-sql title="Query the live view" +SELECT * FROM trades_ma; +``` + +The view updates incrementally as new rows arrive in `trades`. Each new trade +produces one output row carrying its moving average. A direct `SELECT` of the +full output rows sees data as soon as it is refreshed. Filtering a read to a +timestamp interval (for example `WHERE timestamp IN '$today'`) is served from the +disk tier and can trail by up to one `FLUSH EVERY` interval; see +[Freshness](#freshness). + +## How live views work + +A live view is its own WAL-backed table maintained by a background refresh +worker. The worker reads new committed rows from the base table and runs the +view's window functions over them, appending the output. Two independent +cadences govern how that output becomes visible and durable: + +- **Refresh** runs continuously. As the worker computes output rows, it appends + them to an in-memory tier. This is what keeps the view fresh. +- **Flush** runs on the `FLUSH EVERY` cadence. It persists the in-memory rows to + the live view's own WAL-backed disk tier and advances a durability checkpoint. + +Reads combine both tiers. Recent rows are served from the in-memory tier and the +older prefix from disk, so a query sees the freshest computed rows without +waiting for a flush. + +```questdb-sql title="Show the live view definition" +SHOW CREATE LIVE VIEW trades_ma; +``` + +### Freshness + +Because refresh publishes to the in-memory tier ahead of flush, a direct +`SELECT` that reads the full output rows sees data as soon as it is refreshed. +This is independent of `FLUSH EVERY`, which is a durability and +write-amplification control, not a freshness control. + +Some read shapes are served from the disk tier only and therefore trail by up to +one `FLUSH EVERY` interval: + +- Reads that project or aggregate the view's columns rather than reading full + output rows +- Reads filtered to a timestamp interval +- A live view used as the right-hand side of an [`ASOF JOIN`](/docs/query/sql/asof-join/) + +Keep `FLUSH EVERY` small (for example `1s`) so this lag stays negligible. + +:::tip + +A live view falling behind sustained ingestion stays correct but grows stale. +There is no automatic throttle. Monitor `lag_seqtxn` and `lag_micros` in +[`live_views()`](/docs/query/functions/meta/#live_views) to detect a view that +cannot keep up. + +::: + +## Supported window functions + +Live views maintain the window functions whose result can be computed +incrementally in a single forward pass over a partitioned frame: + +- **Ranking**: `row_number`, `rank`, `dense_rank` +- **Cumulative and bounded aggregates**: `sum`, `avg`, `count`, `min`, `max`, + `ksum`, `first_value`, `last_value`, `nth_value` +- **Offset**: `lag` +- **Statistics**: `variance`, `stddev`, covariance, correlation, EMA, and VWEMA + +Every window function must have a `PARTITION BY` clause. Both bounded `ROWS` and +bounded `RANGE` frames are supported. + +String, `VARCHAR`, `BINARY`, `ARRAY`, and `SYMBOL` columns can appear as +pass-through output columns and as `count` arguments, but there are no +string- or array-valued window functions. + +The following shapes cannot be maintained by an append-only incremental refresh +and are rejected at creation time: + +- Multi-pass or look-ahead functions: `percent_rank`, `cume_dist`, `ntile`, + `lead` +- Window functions without `PARTITION BY` +- Unbounded frames on non-anchored windows + +## Anchored windows + +An anchored window resets its cumulative aggregate on a boundary, which is useful +for running totals that restart each day or on a period boundary. Declare it in a +named window with either the `ANCHOR DAILY` shorthand or an `ANCHOR EXPRESSION` +clause: + +```questdb-sql title="Cumulative daily volume per symbol" +CREATE LIVE VIEW trades_daily_volume +FLUSH EVERY 1s +AS +SELECT + timestamp, + symbol, + sum(amount) OVER w AS cumulative_volume +FROM trades +WINDOW w AS ( + PARTITION BY symbol + ORDER BY timestamp + ANCHOR DAILY +); +``` + +An anchored window must be partitioned, cannot use a bounded frame, and its +anchor expression must be deterministic. + +## Backfill + +By default a live view only reflects data that arrives after it is created. Rows +in the base table below the view's creation-time lower bound are not processed. + +Add the `BACKFILL` clause to materialize the base table's existing history before +the view starts live-tailing: + +```questdb-sql title="Backfill existing history" +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +BACKFILL +AS +SELECT + timestamp, + symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; +``` + +The backfill sweep is resumable: it checkpoints its progress and continues after +a restart. + +## Base table lifecycle + +A live view is tied to a single WAL-backed base table and tracks the exact set of +base columns its query references: + +- Changes to columns the view does not reference pass through transparently and + the view keeps refreshing. +- Dropping, renaming, or changing the type of a referenced column invalidates the + view. +- Renaming or dropping the base table invalidates the view. +- `DROP PARTITION`, `TRUNCATE`, and base TTL eviction freeze the already-emitted + rows and the view continues forward from where it was. + +An invalidated view keeps serving its existing data and reports the reason in +[`live_views()`](/docs/query/functions/meta/#live_views). It stops refreshing. + +Live views over [deduplicated](/docs/concepts/deduplication/) base tables are +supported. A keep-last `UPSERT` replacement at an earlier timestamp is reflected +in the view. A view over a deduplicated base is one `FLUSH EVERY` cycle behind +rather than sub-cycle fresh, because its refresh is coupled to base apply. + +## Monitoring + +The [`live_views()`](/docs/query/functions/meta/#live_views) function exposes the +state, refresh lag, in-memory footprint, and backfill progress of every live +view: + +```questdb-sql title="List all live views" +SELECT view_name, base_table_name, view_status, lag_seqtxn, lag_micros +FROM live_views(); +``` + +Live views also appear in [`tables()`](/docs/query/functions/meta/#tables) with +`table_type = 'L'`, and are recognized by `SHOW CREATE LIVE VIEW`, `EXPLAIN`, +`pg_class`, and `information_schema.tables`. + +## Limitations + +Live views have a deliberately narrow surface in this first version. Statements +outside it are rejected at creation time with a specific error: + +- **Single base table only.** No JOINs, subqueries, or CTEs in the view query. +- **No pre-aggregation.** `SAMPLE BY` and `GROUP BY` are not allowed between the + base table and the window functions. A view like "5-minute candles with a + rolling VWAP" must pre-aggregate upstream. +- **No live-view-on-live-view.** A live view cannot be the base of another live + view. +- **Deterministic queries only.** Non-deterministic functions such as `now()`, + `sysdate()`, `systimestamp()`, and `rnd_*()` are rejected in the projection, + the `WHERE` filter, and window-function arguments. +- **No TTL on the view.** Live-view disk growth is unbounded in this version. + Size retention on the base table instead. + +## Tradeoffs + +- **Storage grows with output.** The computed rows are stored on the live view's + disk tier in addition to the base table's rows. For wide projections or long + retention the view's footprint can exceed the base table. +- **No admission control.** A view that cannot keep up with ingestion stays + correct but stale, with no automatic throttle or drop. +- **Per-partition state for partitioned windows grows with distinct partition + cardinality.** A base table with high-cardinality partition keys (UUIDs, + session ids) holds one state entry per key seen, so native-memory use grows + over the life of the view. The `in_mem_bytes` column in + [`live_views()`](/docs/query/functions/meta/#live_views) reports this + footprint as a peak-sticky high-water mark. + +## Enterprise features + +QuestDB Enterprise adds access control, replication, and backup support for live +views. + +### Permissions + +Two dedicated permissions govern live-view DDL, modelled on the materialized-view +permissions: + +- `CREATE LIVE VIEW` is a database-level permission. +- `DROP LIVE VIEW` is checked against the target view. + +Querying a live view uses the standard table-level `SELECT` permission, since a +live view is a regular table token. See +[Role-based access control](/docs/security/rbac/) for the full permission model. + +### Replication + +A live view replicates physically like a materialized view. Its disk tier is a +regular WAL-backed table, so its rows transfer to replicas through the existing +object-store WAL path. A read-only replica never refreshes the view itself. It +reconstructs the primary's un-flushed in-memory rows in RAM so that reads on the +replica match the primary's freshness. Promoting a replica to primary resumes +refresh from the durable watermark. + +### Backup and restore + +A live view is captured by the object-store backup like a materialized view: its +table data rides the standard table path and its definition sidecars are carried +in the backup manifest. On restore, the un-flushed in-memory rows are re-derived +from the base table, which is the same bounded recompute a promote performs. + +## Related documentation + +- **SQL commands** + - [`CREATE LIVE VIEW`](/docs/query/sql/create-live-view/): Create a live view + - [`DROP LIVE VIEW`](/docs/query/sql/drop-live-view/): Remove a live view + +- **Related concepts** + - [Materialized views](/docs/concepts/materialized-views/): Incrementally + maintained `SAMPLE BY` aggregates + - [Views](/docs/concepts/views/): Virtual tables computed at query time + - [Window functions](/docs/query/functions/window-functions/overview/): The `OVER` functions a + live view maintains + +- **Configuration** + - [Live views configs](/docs/configuration/live-views/): Server configuration + options for live views diff --git a/documentation/concepts/views.md b/documentation/concepts/views.md index 0995c637c..fdb136ae0 100644 --- a/documentation/concepts/views.md +++ b/documentation/concepts/views.md @@ -313,6 +313,7 @@ SELECT table_name, table_type FROM tables() | `T` | Regular table | | `V` | View | | `M` | Materialized view | +| `L` | [Live view](/docs/concepts/live-views/) | ## Views vs materialized views diff --git a/documentation/configuration/live-views.md b/documentation/configuration/live-views.md new file mode 100644 index 000000000..bfa3a247a --- /dev/null +++ b/documentation/configuration/live-views.md @@ -0,0 +1,105 @@ +--- +title: Live views +description: Configuration settings for live views in QuestDB. +--- + +These settings control live view SQL support and the background refresh job that +maintains live views incrementally. For a conceptual overview, see +[Live views](/docs/concepts/live-views/). + +Live view refresh shares the materialized view refresh worker pool, so the +worker-pool settings under +[Materialized views](/docs/configuration/materialized-views/) +(`mat.view.refresh.worker.count`, `.affinity`, `.haltOnError`) also govern live +view refresh. There are no dedicated live-view worker-pool properties. + +## cairo.live.view.checkpoint.max.duration.micros + +- **Default**: `300000000` (5 minutes) +- **Reloadable**: no + +Time budget, in microseconds, for a single checkpoint write turn. Checkpoints let +a restart or out-of-order replay resume without rebuilding the whole view. + +## cairo.live.view.checkpoint.rows + +- **Default**: `1000000` +- **Reloadable**: no + +Number of newly flushed rows after which the refresh worker writes a head +checkpoint. Smaller values shorten restart replay at the cost of more checkpoint +writes. + +## cairo.live.view.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables or disables SQL support and the refresh job for live views. When +disabled, `CREATE LIVE VIEW` fails with `live views are disabled`. + +## cairo.live.view.flush.retry.max + +- **Default**: `5` +- **Reloadable**: no + +Maximum number of consecutive flush attempts before a view is marked invalid. A +flush persists the in-memory rows to the view's disk tier. + +## cairo.live.view.flush.retry.max.duration.micros + +- **Default**: `60000000` (60 seconds) +- **Reloadable**: no + +Maximum total time, in microseconds, spent retrying a stalled flush before the +view is marked invalid. + +## cairo.live.view.in.memory.buffer.growth.bytes + +- **Default**: `16777216` (16 MiB) +- **Reloadable**: no + +Increment by which the in-memory tier's buffer arena grows when it needs more +capacity. Accepts a size suffix such as `16M`. + +## cairo.live.view.in.memory.buffer.initial.bytes + +- **Default**: `65536` (64 KiB) +- **Reloadable**: no + +Initial size of a live view's in-memory tier buffer. Accepts a size suffix such +as `64K`. + +## cairo.live.view.in.memory.max + +- **Default**: `3600000000` (60 minutes) +- **Reloadable**: no + +Upper bound on the `IN MEMORY` retention window. A `CREATE LIVE VIEW` whose +`IN MEMORY` (or defaulted `FLUSH EVERY`) exceeds this value is rejected. + +## cairo.live.view.partition.compact.threshold + +- **Default**: `100000` +- **Reloadable**: no + +Row-count threshold at which an anchored live view compacts a partition's +per-function state. Compaction fires when a partition's anchor-map entry count +exceeds this value and the frontier has advanced. Applies only to anchored views +whose anchor is a monotone, fixed-duration-unit timestamp expression. + +## cairo.live.view.refresh.turn.max.commits + +- **Default**: `64` +- **Reloadable**: no + +Maximum number of base-table commits a refresh worker processes in a single turn +before yielding to other views. + +## cairo.live.view.refresh.turn.max.duration.micros + +- **Default**: `50000` (50 milliseconds) +- **Reloadable**: no + +Maximum wall-clock time, in microseconds, a refresh worker spends on one view per +turn before yielding. diff --git a/documentation/configuration/overview.md b/documentation/configuration/overview.md index 5d2aa018a..f9b9d0f93 100644 --- a/documentation/configuration/overview.md +++ b/documentation/configuration/overview.md @@ -531,6 +531,7 @@ http.net.connection.sndbuf=2m | [HTTP server](/docs/configuration/http-server/) | Web Console and REST API | | | [IAM](/docs/configuration/iam/) | Identity and Access Management | ✓ | | [Ingestion (ILP/HTTP)](/docs/configuration/ingestion/) | InfluxDB Line Protocol settings | | +| [Live views](/docs/configuration/live-views/) | Live view refresh settings | | | [Logging & Metrics](/docs/configuration/logging-metrics/) | Log levels and metrics | | | [Materialized views](/docs/configuration/materialized-views/) | Materialized view refresh settings | | | [Minimal HTTP server](/docs/configuration/http-min-server/) | Health check and metrics endpoint | | diff --git a/documentation/operations/backup.md b/documentation/operations/backup.md index 5af15cacc..e21d5b3f7 100644 --- a/documentation/operations/backup.md +++ b/documentation/operations/backup.md @@ -362,7 +362,7 @@ primary/replica backups below). - **Database-wide only**: Backup captures the entire database. You cannot exclude tables or backup selected tables individually. Every backup includes - all user tables, materialized views, and metadata. + all user tables, materialized views, live views, and metadata. - **One backup at a time**: Only one backup can run at any given time. Starting a new backup while one is running will return an error. - **Primary and replica backups are separate**: Each QuestDB instance has its diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index ada4004dd..869aa5263 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -134,6 +134,65 @@ If you want to re-read metadata for all user tables, simply use an asterisk: SELECT hydrate_table_metadata('*'); ``` +## live_views + +`live_views()` returns the list of all [live views](/docs/concepts/live-views/) +in the database, along with their status, refresh lag, in-memory footprint, and +backfill progress. + +**Arguments:** + +- `live_views()` does not require arguments. + +**Return value:** + +Returns a `table` with the following columns: + +| Column | Type | Description | +| ------ | ---- | ----------- | +| `view_name` | STRING | Live view name | +| `view_table_dir_name` | STRING | View directory name on disk | +| `base_table_name` | STRING | Base table name | +| `view_sql` | STRING | Query used to maintain the view | +| `view_status` | STRING | View status: `active`, `backfilling`, or `invalid` | +| `invalidation_reason` | STRING | Message explaining why the view was marked invalid | +| `flush_every_interval` | LONG | `FLUSH EVERY` interval value | +| `flush_every_interval_unit` | STRING | `FLUSH EVERY` unit: `MILLISECOND`, `SECOND`, `MINUTE`, `HOUR`, or `DAY` | +| `in_memory_interval` | LONG | `IN MEMORY` interval value | +| `in_memory_interval_unit` | STRING | `IN MEMORY` unit: `MILLISECOND`, `SECOND`, `MINUTE`, `HOUR`, or `DAY` | +| `in_mem_bytes` | LONG | Native footprint of the in-memory tier, a peak-sticky high-water mark | +| `in_mem_rows` | LONG | Live row count held in the in-memory tier | +| `o3_rejected_count` | LONG | Count of out-of-order base commits routed through replay | +| `below_lower_bound_count` | LONG | Count of base rows dropped below the view's lower bound | +| `lag_seqtxn` | LONG | Base transactions the view is behind (`base_table_txn - last_processed_seqtxn`) | +| `lag_micros` | LONG | Time the view is behind the base table, in microseconds | +| `last_processed_seqtxn` | LONG | Last base transaction processed by the refresh worker | +| `applied_watermark` | LONG | Last base transaction durably applied to the view's disk tier | +| `lv_consumed_seqtxn` | LONG | Base WAL purge floor held by this view | +| `view_lower_bound_timestamp` | TIMESTAMP | Lower timestamp bound below which base rows are ignored | +| `writer_stall_micros` | LONG | Time a flush has been stalled waiting to write, in microseconds | +| `backfill_target_seqtxn` | LONG | Target base transaction for an in-progress backfill | +| `head_checkpoint_lv_seqtxn` | LONG | View transaction of the latest head checkpoint | +| `head_checkpoint_max_ts` | TIMESTAMP | Maximum timestamp covered by the latest head checkpoint | +| `head_checkpoint_state_bytes` | LONG | Size of the latest head checkpoint's window state | + +The `in_mem_bytes` and `in_mem_rows` columns are complementary. `in_mem_bytes` is +the peak-sticky arena footprint that does not shrink after a burst, while +`in_mem_rows` is the live row count that drops as rows age out of the `IN MEMORY` +window. Together they distinguish a view actively buffering rows from one holding +capacity retained from a past burst. + +**Examples:** + +```questdb-sql title="List all live views" +SELECT view_name, base_table_name, view_status, lag_seqtxn, lag_micros +FROM live_views(); +``` + +| view_name | base_table_name | view_status | lag_seqtxn | lag_micros | +| --------- | --------------- | ----------- | ---------- | ---------- | +| trades_ma | trades | active | 0 | 0 | + ## materialized_views `materialized_views()` returns the list of all materialized views in the @@ -618,7 +677,7 @@ Returns a `table` with the following columns: | Column | Type | Description | |--------|------|-------------| | `table_suspended` | BOOLEAN | Whether a WAL table is suspended (`false` for non-WAL tables) | -| `table_type` | CHAR | Table type: `T` (table), `M` (materialized view), `V` (view) | +| `table_type` | CHAR | Table type: `T` (table), `M` (materialized view), `V` (view), `L` (live view) | | `table_row_count` | LONG | Approximate row count at last tracked write | | `table_min_timestamp` | TIMESTAMP | Minimum timestamp of data in the table (updated on WAL merge) | | `table_max_timestamp` | TIMESTAMP | Maximum timestamp of data in the table (updated on WAL merge) | diff --git a/documentation/query/sql/create-live-view.md b/documentation/query/sql/create-live-view.md new file mode 100644 index 000000000..a8b7bc121 --- /dev/null +++ b/documentation/query/sql/create-live-view.md @@ -0,0 +1,285 @@ +--- +title: CREATE LIVE VIEW +sidebar_label: CREATE LIVE VIEW +description: + Documentation for the CREATE LIVE VIEW SQL keyword in QuestDB. +--- + +Creates a live view that incrementally maintains the result of a window-function +query over a single base table and can be queried like a regular table. For a +conceptual overview, see [Live views](/docs/concepts/live-views/). + +## Syntax + +```questdb-sql title="CREATE LIVE VIEW" +CREATE LIVE VIEW [ IF NOT EXISTS ] viewName +FLUSH EVERY duration +[ IN MEMORY duration ] +[ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) ] +[ BACKFILL ] +AS [ ( ] query [ ) ] +[ OWNED BY ownerName ] +``` + +Where: + +- `duration`: a single token with a unit of `ms`, `s`, `m`, `h`, or `d`, for + example `100ms`, `5s`, or `30m`. +- `query`: a `SELECT` over one WAL-backed base table whose projection contains + [window functions](/docs/query/functions/window-functions/overview/). + +`FLUSH EVERY` is required and must come first. `IN MEMORY`, `PARTITION BY`, and +`BACKFILL` are optional and may appear in any order. These four clauses all +precede `AS`; the optional `OWNED BY` clause follows the query. + +## Parameters + +| Parameter | Description | +| --------- | ----------- | +| `viewName` | Name for the live view | +| `IF NOT EXISTS` | Create only if a view with this name does not already exist | +| `FLUSH EVERY` | How often computed rows are persisted to disk. Required | +| `IN MEMORY` | Window of recent rows kept in RAM for fresh reads. Defaults to `FLUSH EVERY` | +| `PARTITION BY` | Partitioning unit for the view's disk tier. Defaults to the base table's scheme | +| `BACKFILL` | Materialize the base table's existing history before live-tailing | +| `query` | A window-function `SELECT` over a single WAL-backed base table | +| `OWNED BY` | Assign ownership (Enterprise) | + +## Clauses + +### FLUSH EVERY + +`FLUSH EVERY` sets how often the view's computed rows are persisted from the +in-memory tier to the view's own WAL-backed disk tier. It controls durability and +write amplification, not read freshness: a direct `SELECT` reads the freshest +computed rows regardless of the flush cadence. + +A smaller interval persists more often, shortening crash recovery at the cost of +more write volume. A larger interval reduces write volume but lengthens recovery +and increases the staleness of the read shapes that are served from disk only +(see [Freshness](/docs/concepts/live-views/#freshness)). + +The minimum is `100ms`. The maximum is +[`cairo.live.view.in.memory.max`](/docs/configuration/live-views/#cairoliveviewinmemorymax) +(60 minutes by default), because `IN MEMORY` defaults to `FLUSH EVERY`. + +```questdb-sql +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; +``` + +### IN MEMORY + +`IN MEMORY` sets how long a window of recent output rows is retained in RAM to +serve fast, fresh reads. Reads of recent data are served from the in-memory tier +and older data from disk. It defaults to `FLUSH EVERY`. + +`IN MEMORY` must be at least `FLUSH EVERY` and at most +[`cairo.live.view.in.memory.max`](/docs/configuration/live-views/#cairoliveviewinmemorymax). + +```questdb-sql +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +IN MEMORY 5s +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; +``` + +### PARTITION BY + +`PARTITION BY` sets the partitioning of the view's disk tier. If omitted, the +view inherits the base table's partitioning scheme. + +```questdb-sql +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +PARTITION BY HOUR +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; +``` + +### BACKFILL + +By default a live view processes only the rows that arrive after it is created. +Add `BACKFILL` to materialize the base table's existing history first. The sweep +is resumable across restarts. + +```questdb-sql +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +BACKFILL +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; +``` + +## Anchored windows + +An anchored window resets its cumulative aggregate on a boundary. Declare it in a +named `WINDOW` with either the `ANCHOR DAILY` shorthand or an +`ANCHOR EXPRESSION` clause: + +```questdb-sql title="Cumulative daily volume, reset each day" +CREATE LIVE VIEW trades_daily_volume +FLUSH EVERY 1s +AS +SELECT timestamp, symbol, + sum(amount) OVER w AS cumulative_volume +FROM trades +WINDOW w AS (PARTITION BY symbol ORDER BY timestamp ANCHOR DAILY); +``` + +```questdb-sql title="Anchor on an arbitrary expression" +CREATE LIVE VIEW trades_hourly_volume +FLUSH EVERY 1s +AS +SELECT timestamp, symbol, + sum(amount) OVER w AS bucket_volume +FROM trades +WINDOW w AS ( + PARTITION BY symbol + ORDER BY timestamp + ANCHOR EXPRESSION timestamp_floor('1h', timestamp) +); +``` + +An anchored window must be partitioned, must `ORDER BY` the designated timestamp +ascending, cannot use a bounded frame, and its anchor expression must be +deterministic. + +## Query constraints + +The view query is validated at creation time and must: + +- Read a single WAL-backed base table that has a designated timestamp. No JOINs, + subqueries, or CTEs. +- Contain [window functions](/docs/query/functions/window-functions/overview/) that can be + maintained incrementally (see + [supported functions](/docs/concepts/live-views/#supported-window-functions)). +- Give every window function a `PARTITION BY` clause. +- Not use `SAMPLE BY`, `GROUP BY`, a top-level `ORDER BY`, or `LIMIT` in the view + query. The `ORDER BY` inside a window's `OVER (...)` is required and allowed. +- Not use non-deterministic functions such as `now()`, `sysdate()`, + `systimestamp()`, or `rnd_*()`. +- Not read another live view. + +## Complete example + +```questdb-sql title="Base table" +CREATE TABLE trades ( + symbol SYMBOL, + side SYMBOL, + price DOUBLE, + amount DOUBLE, + timestamp TIMESTAMP +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; +``` + +```questdb-sql title="Fully specified live view" +CREATE LIVE VIEW IF NOT EXISTS trades_ma +FLUSH EVERY 1s +IN MEMORY 5s +PARTITION BY HOUR +BACKFILL +AS +SELECT + timestamp, + symbol, + price, + avg(price) OVER ( + PARTITION BY symbol + ORDER BY timestamp + ROWS 300 PRECEDING + ) AS moving_avg +FROM trades; +``` + +This creates a view that: + +- Persists computed rows to disk every second (`FLUSH EVERY 1s`) +- Keeps 5 seconds of recent rows in RAM for fresh reads (`IN MEMORY 5s`) +- Partitions its disk tier by hour (`PARTITION BY HOUR`) +- Materializes the existing history in `trades` before live-tailing (`BACKFILL`) +- Keeps a 300-row moving average of price per symbol + +## Metadata + +Query view metadata with [`live_views()`](/docs/query/functions/meta/#live_views): + +```questdb-sql +SELECT view_name, base_table_name, view_status, lag_seqtxn +FROM live_views(); +``` + +## Permissions (Enterprise) + +Creating a live view requires the database-level `CREATE LIVE VIEW` permission +and `SELECT` on the base table: + +```questdb-sql title="Grant permission to create live views" +GRANT CREATE LIVE VIEW TO user1; +``` + +```questdb-sql title="Grant SELECT on the base table" +GRANT SELECT ON trades TO user1; +``` + +When you create a live view you automatically receive all permissions on it, +including `DROP LIVE VIEW`, with the `GRANT` option. + +### OWNED BY clause + +Assign ownership to a user, group, or service account: + +```questdb-sql +CREATE GROUP analysts; +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades +OWNED BY analysts; +``` + +## Errors + +| Error | Cause | +| ----- | ----- | +| `live views are disabled` | Live-view support is turned off (`cairo.live.view.enabled=false`) | +| `live view already exists` | A live view of this name exists and `IF NOT EXISTS` was not specified | +| `table or view with the requested name already exists` | The name is taken by a table, view, or materialized view | +| `live view FLUSH EVERY must be at least 100ms` | The `FLUSH EVERY` interval is below the minimum | +| `live view select must be a simple scan of a single WAL base table; joins, subqueries, GROUP BY, ORDER BY and LIMIT are not supported yet` | The view query is not a simple scan of one base table | +| `live view base table must have a designated timestamp` | The base table has no designated timestamp | +| `non-deterministic function cannot be used in materialized view` | The query uses `now()`, `rnd_*()`, or a similar non-deterministic function | +| `permission denied` | Missing required permission (Enterprise) | + +:::note + +The non-determinism check is the same guard materialized views use, so its error +message names "materialized view" even when it is raised for a live view. The +rule and its effect are identical for both view types. + +::: + +## See also + +- [Live views concept](/docs/concepts/live-views/) +- [DROP LIVE VIEW](/docs/query/sql/drop-live-view/) +- [Window functions](/docs/query/functions/window-functions/overview/) +- [live_views()](/docs/query/functions/meta/#live_views) diff --git a/documentation/query/sql/drop-live-view.md b/documentation/query/sql/drop-live-view.md new file mode 100644 index 000000000..dd684e04b --- /dev/null +++ b/documentation/query/sql/drop-live-view.md @@ -0,0 +1,72 @@ +--- +title: DROP LIVE VIEW +sidebar_label: DROP LIVE VIEW +description: + Documentation for the DROP LIVE VIEW SQL keyword in QuestDB. +--- + +Permanently deletes a live view and all of its data. For a conceptual overview, +see [Live views](/docs/concepts/live-views/). + +## Syntax + +```questdb-sql title="DROP LIVE VIEW" +DROP LIVE VIEW [ IF EXISTS ] viewName +``` + +## Parameters + +| Parameter | Description | +| --------- | ----------- | +| `viewName` | Name of the live view to drop | +| `IF EXISTS` | Suppress the error if the view does not exist | + +## Examples + +```questdb-sql title="Drop a live view" +DROP LIVE VIEW trades_ma; +``` + +```questdb-sql title="Drop only if it exists (no error if missing)" +DROP LIVE VIEW IF EXISTS trades_ma; +``` + +## Behavior + +| Aspect | Description | +| ------ | ----------- | +| Permanence | Deletion is permanent and not recoverable | +| Space reclamation | Disk space is reclaimed asynchronously | +| Active queries | Existing read queries may delay space reclamation | +| Permissions | On Enterprise, the view's access-control grants are removed with it | + +:::warning + +This operation cannot be undone. The view and all of its precomputed data are +permanently deleted. + +::: + +## Permissions (Enterprise) + +Dropping a live view requires the `DROP LIVE VIEW` permission on the specific +view: + +```questdb-sql title="Grant drop permission" +GRANT DROP LIVE VIEW ON trades_ma TO user1; +``` + +The view creator automatically receives this permission with the `GRANT` option. + +## Errors + +| Error | Cause | +| ----- | ----- | +| `live view name expected` | The name refers to a table or view that is not a live view | +| `live view does not exist` | The view does not exist and `IF EXISTS` was not specified | +| `permission denied` | Missing `DROP LIVE VIEW` permission (Enterprise) | + +## See also + +- [Live views concept](/docs/concepts/live-views/) +- [CREATE LIVE VIEW](/docs/query/sql/create-live-view/) diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 0d19bae2b..8f3fe6b86 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -18,6 +18,7 @@ SHOW { TABLES | PARTITIONS FROM tableName | CREATE TABLE tableName | CREATE VIEW viewName + | CREATE LIVE VIEW viewName | USER [userName] | USERS | GROUPS [userName] @@ -36,6 +37,8 @@ SHOW { TABLES - `SHOW PARTITIONS` returns the partition information for the selected table. - `SHOW CREATE TABLE` returns a DDL query that allows you to recreate the table. - `SHOW CREATE VIEW` returns a DDL query that allows you to recreate a view. +- `SHOW CREATE LIVE VIEW` returns a DDL query that allows you to recreate a live + view. - `SHOW USER` shows user secret (enterprise-only) - `SHOW GROUPS` shows all groups the user belongs or all groups in the system (enterprise-only) @@ -201,6 +204,21 @@ SHOW CREATE VIEW my_view; This returns the `CREATE VIEW` statement that would recreate the view, including any `DECLARE` parameters if the view is parameterized. +### SHOW CREATE LIVE VIEW + +```questdb-sql title="retrieving live view ddl" +SHOW CREATE LIVE VIEW trades_ma; +``` + +| ddl | +| --- | +| CREATE LIVE VIEW 'trades_ma' FLUSH EVERY 1s IN MEMORY 5s PARTITION BY DAY AS (
SELECT timestamp, symbol, avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) AS moving_avg FROM trades
); | + +This returns the `CREATE LIVE VIEW` statement that would recreate the +[live view](/docs/concepts/live-views/), including its `FLUSH EVERY`, +`IN MEMORY`, `PARTITION BY`, and `BACKFILL` clauses. On QuestDB Enterprise the +output also carries an `OWNED BY` clause identifying the view's owner. + ### SHOW PARTITIONS ```questdb-sql diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index 9bb9897a0..d92a6dd2e 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -580,6 +580,7 @@ SELECT * FROM all_permissions(); | CANCEL ANY COPY | Database | Cancel COPY operations | | CREATE TABLE | Database | Create tables | | CREATE MATERIALIZED VIEW | Database | Create materialized views | +| CREATE LIVE VIEW | Database | Create live views | | DEDUP ENABLE | Database | Table | Enable deduplication | | DEDUP DISABLE | Database | Table | Disable deduplication | | DETACH PARTITION | Database | Table | Detach partitions | @@ -589,6 +590,7 @@ SELECT * FROM all_permissions(); | DROP PARTITION | Database | Table | Drop partitions | | DROP TABLE | Database | Table | Drop tables | | DROP MATERIALIZED VIEW | Database | Table | Drop materialized views | +| DROP LIVE VIEW | Database | Table | Drop live views | | ENABLE STORAGE POLICY | Database | Table | Enable storage policies | | INSERT | Database | Table | Insert data | | REFRESH MATERIALIZED VIEW | Database | Table | Refresh materialized views | diff --git a/documentation/sidebars.js b/documentation/sidebars.js index f4734f2e9..e6bc0bfde 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -334,6 +334,7 @@ module.exports = { id: "query/sql/acl/create-group", type: "doc", }, + "query/sql/create-live-view", "query/sql/create-mat-view", { id: "query/sql/acl/create-service-account", @@ -355,6 +356,7 @@ module.exports = { id: "query/sql/acl/drop-group", type: "doc", }, + "query/sql/drop-live-view", "query/sql/drop-mat-view", { id: "query/sql/acl/drop-service-account", @@ -533,6 +535,11 @@ module.exports = { type: "doc", label: "Materialized Views", }, + { + id: "concepts/live-views", + type: "doc", + label: "Live Views", + }, "concepts/deduplication", "concepts/ttl", "concepts/storage-policy", @@ -589,6 +596,7 @@ module.exports = { "configuration/http-server", "configuration/iam", "configuration/ingestion", + "configuration/live-views", "configuration/logging-metrics", "configuration/materialized-views", "configuration/http-min-server",