diff --git a/documentation/concepts/deep-dive/indexes.md b/documentation/concepts/deep-dive/indexes.md index 68a865032..7f2b7d861 100644 --- a/documentation/concepts/deep-dive/indexes.md +++ b/documentation/concepts/deep-dive/indexes.md @@ -14,6 +14,16 @@ Indexing is available for [symbol](/docs/concepts/symbol/) columns in both table and [materialized views](/docs/concepts/materialized-views). Index support for other types will be added over time. +QuestDB supports two index types: + +| Index type | Syntax | Covering support | Best for | +|------------|--------|-----------------|----------| +| **Bitmap** (default) | `INDEX` or `INDEX TYPE BITMAP` | No | General-purpose, low write overhead | +| **Posting** | `INDEX TYPE POSTING` | Yes (via `INCLUDE`) | Read-heavy workloads, selective queries, wide tables | + +See [Posting index and covering index](/docs/concepts/deep-dive/posting-index/) +for the detailed guide on the posting index and its covering query capabilities. + ## Index creation and deletion The following are ways to index a `symbol` column: @@ -97,6 +107,9 @@ Consider the following query applied to the above table :::warning +Index capacity applies to **bitmap indexes only**. Posting indexes manage +their own storage layout and do not use this setting. + We strongly recommend to rely on the default index capacity. Misconfiguring this property might lead to worse performance and increased disk usage. @@ -114,8 +127,8 @@ When in doubt, reach out via the QuestDB support channels for advice. ::: -When a symbol column is indexed, an additional **index capacity** can be defined -to specify how many row IDs to store in a single storage block on disk: +When a symbol column has a bitmap index, an additional **index capacity** can be +defined to specify how many row IDs to store in a single storage block on disk: - Server-wide setting: `cairo.index.value.block.size` with a default of `256` - Column-wide setting: The diff --git a/documentation/concepts/deep-dive/posting-index.md b/documentation/concepts/deep-dive/posting-index.md new file mode 100644 index 000000000..e6d53d900 --- /dev/null +++ b/documentation/concepts/deep-dive/posting-index.md @@ -0,0 +1,439 @@ +--- +title: Posting index and covering index +sidebar_label: Posting index +description: + The posting index is a compact, high-performance index for symbol columns + that supports covering queries. Learn how it works, when to use it, and + how to optimize queries with INCLUDE columns. +--- + +The **posting index** is an advanced index type for +[symbol](/docs/concepts/symbol/) columns that provides better compression, +faster reads, and **covering index** support compared to the default bitmap +index. + +A **covering index** stores additional column values alongside the index +entries, so queries that only need those columns can be answered entirely from +the index without reading the main column files. + +## When to use the posting index + +Use the posting index when: + +- You frequently filter on a symbol column (`WHERE symbol = 'X'`) +- Your queries select a small set of columns alongside the symbol filter +- You want to reduce I/O by reading from compact sidecar files instead of + full column files +- You need efficient `DISTINCT` queries on a symbol column +- You need efficient `LATEST ON` queries partitioned by a symbol column + +The posting index is especially effective for high-cardinality symbol columns +(hundreds to thousands of distinct values) and wide tables where reading full +column files is expensive. + +## Creating a posting index + +### At table creation + +Inline syntax (index defined alongside the column): + +```questdb-sql +CREATE TABLE trades ( + timestamp TIMESTAMP, + symbol SYMBOL INDEX TYPE POSTING, + exchange SYMBOL, + price DOUBLE, + quantity DOUBLE +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; +``` + +Out-of-line syntax (index defined separately): + +```questdb-sql +CREATE TABLE trades ( + timestamp TIMESTAMP, + symbol SYMBOL, + exchange SYMBOL, + price DOUBLE, + quantity DOUBLE +), INDEX(symbol TYPE POSTING) +TIMESTAMP(timestamp) PARTITION BY DAY WAL; +``` + +### With covering columns (INCLUDE) + +```questdb-sql +CREATE TABLE trades ( + timestamp TIMESTAMP, + symbol SYMBOL INDEX TYPE POSTING INCLUDE (exchange, price), + exchange SYMBOL, + price DOUBLE, + quantity DOUBLE +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; +``` + +The `INCLUDE` clause specifies which columns are stored in the index sidecar +files. Queries that only read these columns plus the indexed symbol column +can be served entirely from the index. + +:::tip + +The designated timestamp column is automatically included in the covering +index when an `INCLUDE` clause is present — you do not need to list it +explicitly. This means timestamp-filtered covering queries work out of the +box. + +::: + +:::note + +The `INCLUDE` clause is only supported with inline column syntax and +`ALTER TABLE`. The out-of-line `INDEX(col TYPE POSTING)` syntax does not +support `INCLUDE`. + +::: + +### On an existing table + +```questdb-sql +ALTER TABLE trades + ALTER COLUMN symbol ADD INDEX TYPE POSTING INCLUDE (exchange, price); +``` + +### Encoding options + +The posting index supports three row ID encoding options with different +compression and query performance characteristics: + +| Syntax | Encoding | Best for | +|--------|----------|----------| +| `INDEX TYPE POSTING` | Adaptive (default) | General purpose — trial-encodes both EF and delta per stride, picks the smaller | +| `INDEX TYPE POSTING EF` | Elias-Fano | Irregular data distributions, point queries and selective lookups | +| `INDEX TYPE POSTING DELTA` | Delta | Regular, evenly-distributed data, large sequential scans | + +**Delta encoding** stores per-key deltas between consecutive row IDs with +Frame-of-Reference bitpacking. It compresses best when row IDs for each +symbol key are evenly spaced (e.g. round-robin or time-ordered ingestion +of a fixed set of symbols) and is faster for queries that scan large +ranges of matching rows. + +**Elias-Fano (EF) encoding** uses a stride-wide flat layout with +Frame-of-Reference bitpacking across all keys in a stride. It compresses +better for irregular data distributions (e.g. bursty or skewed symbol +frequencies) and is faster for point queries and selective lookups. + +The **adaptive (default)** encoding trial-encodes both EF and delta modes +per stride and picks whichever produces the smaller output. This is the +best choice when you are unsure about your data distribution or have a +mixed query workload. + +```questdb-sql +-- Default adaptive encoding (recommended for most workloads) +CREATE TABLE t1 (ts TIMESTAMP, s SYMBOL INDEX TYPE POSTING) + TIMESTAMP(ts) PARTITION BY DAY WAL; + +-- EF encoding (irregular data, point queries) +CREATE TABLE t2 (ts TIMESTAMP, s SYMBOL INDEX TYPE POSTING EF) + TIMESTAMP(ts) PARTITION BY DAY WAL; + +-- Delta-only encoding (regular data, large scans) +CREATE TABLE t3 (ts TIMESTAMP, s SYMBOL INDEX TYPE POSTING DELTA) + TIMESTAMP(ts) PARTITION BY DAY WAL; +``` + +:::note + +`CAPACITY` is only supported for bitmap indexes. Using `CAPACITY` with a +posting index will produce an error. + +::: + +## Covering index + +The covering index is the most powerful feature of the posting index. When all +columns in a query's `SELECT` list are either: + +- The indexed symbol column itself (from the `WHERE` clause) +- Listed in the `INCLUDE` clause + +...the query engine reads data directly from the index sidecar files, bypassing +the main column files entirely. This is significantly faster for selective +queries on wide tables. + +### Supported column types in INCLUDE + +All column types except the indexed symbol column itself can be included: + +| Type | Compression | Notes | +|------|-------------|-------| +| BOOLEAN, BYTE, GEOBYTE, DECIMAL8 | Raw copy | 1 byte per value | +| SHORT, CHAR, GEOSHORT, DECIMAL16 | Frame-of-Reference | 2 bytes uncompressed | +| INT, FLOAT, IPv4, GEOINT, DECIMAL32 | FoR (int) / ALP (float) | 4 bytes uncompressed | +| LONG, DOUBLE, TIMESTAMP, DATE, GEOLONG, DECIMAL64 | FoR / ALP / linear prediction | 8 bytes uncompressed | +| SYMBOL | Frame-of-Reference | Stored as integer key, resolved at query time | +| UUID, DECIMAL128 | Raw copy | 16 bytes per value | +| LONG256, DECIMAL256 | Raw copy | 32 bytes per value | +| VARCHAR, STRING | FSST compressed | Variable-width, typically 2-5x compression | +| BINARY | Variable-width sidecar | Stored in offset-based format | +| Arrays (DOUBLE[], INT[], etc.) | Variable-width sidecar | Stored in offset-based format | + +### How to choose INCLUDE columns + +Include columns that you frequently select together with the indexed symbol: + +```questdb-sql +-- If your typical queries look like this: +SELECT timestamp, price, quantity FROM trades WHERE symbol = 'AAPL'; + +-- Then include those columns (timestamp is auto-included as designated timestamp): +CREATE TABLE trades ( + timestamp TIMESTAMP, + symbol SYMBOL INDEX TYPE POSTING INCLUDE (price, quantity), + exchange SYMBOL, + price DOUBLE, + quantity DOUBLE, + -- other columns not needed in hot queries + raw_data VARCHAR, + metadata VARCHAR +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; +``` + +:::tip + +Only include columns that appear in your most frequent queries. Each included +column adds storage overhead and slows down writes slightly. Columns not in +the `INCLUDE` list can still be queried — they just won't benefit from the +covering optimization and will be read from column files. + +::: + +### Inspecting indexes with SHOW COLUMNS + +`SHOW COLUMNS` displays index metadata for each column, including the index +type and covered columns: + +```questdb-sql +SHOW COLUMNS FROM trades; +``` + +| column | type | indexed | indexBlockCapacity | indexType | indexInclude | symbolCached | symbolCapacity | designated | upsertKey | +|--------|------|---------|-------------------|-----------|-------------|-------------|----------------|------------|-----------| +| timestamp | TIMESTAMP | false | 0 | | | false | 0 | true | false | +| symbol | SYMBOL | true | 256 | POSTING | exchange,price | true | 128 | false | false | +| exchange | SYMBOL | false | 0 | | | true | 128 | false | false | +| price | DOUBLE | false | 0 | | | false | 0 | false | false | +| quantity | DOUBLE | false | 0 | | | false | 0 | false | false | + +The `indexType` column shows `POSTING`, `BITMAP`, or is empty for +non-indexed columns. The `indexInclude` column lists covered column names. + +### Verifying covering index usage + +Use `EXPLAIN` to verify that a query uses the covering index: + +```questdb-sql +EXPLAIN SELECT timestamp, price FROM trades WHERE symbol = 'AAPL'; +``` + +If the covering index is used, the plan shows `CoveringIndex`: + +``` +SelectedRecord + CoveringIndex on: symbol with: timestamp, price + filter: symbol='AAPL' +``` + +If you see `DeferredSingleSymbolFilterPageFrame` or `PageFrame` instead, the +query is reading from column files. This happens when the `SELECT` list +includes columns not in the `INCLUDE` list. + +## Comparison with bitmap index + +| Feature | Bitmap index | Posting index | +|---------|-------------|---------------| +| Storage size | 8-16 bytes/value | ~1 byte/value | +| Covering index (INCLUDE) | No | Yes | +| DISTINCT acceleration | No | Yes | +| Write overhead | Minimal | Minimal (without INCLUDE) | +| Write overhead with INCLUDE | N/A | Moderate (depends on INCLUDE columns) | +| LATEST ON optimization | Yes | Yes | +| Syntax | `INDEX` or `INDEX TYPE BITMAP` | `INDEX TYPE POSTING` | + +## Query patterns accelerated + +### Point queries (WHERE symbol = 'X') + +```questdb-sql +-- Reads from sidecar if price is in INCLUDE +SELECT price FROM trades WHERE symbol = 'AAPL'; +``` + +### Point queries with additional filters + +If the additional filter columns are also in INCLUDE, the covering index +is still used with a filter applied on top: + +```questdb-sql +-- Covering index + filter on covered column +SELECT price FROM trades WHERE symbol = 'AAPL' AND price > 100; +``` + +### IN-list queries + +```questdb-sql +-- Multiple keys, still uses covering index +SELECT price FROM trades WHERE symbol IN ('AAPL', 'GOOGL', 'MSFT'); +``` + +### LATEST ON queries + +```questdb-sql +-- Latest row per symbol, reads from sidecar +SELECT timestamp, symbol, price +FROM trades +WHERE symbol = 'AAPL' +LATEST ON timestamp PARTITION BY symbol; +``` + +### DISTINCT queries + +```questdb-sql +-- Enumerates keys from index metadata, O(keys x partitions) instead of full scan +SELECT DISTINCT symbol FROM trades; + +-- Also works with timestamp filters +SELECT DISTINCT symbol FROM trades WHERE timestamp > '2024-01-01'; +``` + +### COUNT queries + +```questdb-sql +-- Uses index to scan only matching rows instead of full table +SELECT COUNT(*) FROM trades WHERE symbol = 'AAPL'; +``` + +### Aggregate queries on covered columns + +```questdb-sql +-- Vectorized GROUP BY reads from sidecar page frames +SELECT count(*), min(price), max(price) +FROM trades +WHERE symbol = 'AAPL'; +``` + +## SQL optimizer hints + +Two hints control index usage: + +### no_covering + +Forces the query to read from column files instead of the covering index +sidecar. Useful for benchmarking or when the covering path has an issue. + +```questdb-sql +SELECT /*+ no_covering */ price FROM trades WHERE symbol = 'AAPL'; +``` + +### no_index + +Completely disables index usage, falling back to a full table scan with +filter. Also implies `no_covering`. + +```questdb-sql +SELECT /*+ no_index */ price FROM trades WHERE symbol = 'AAPL'; +``` + +## Trade-offs + +### Storage + +The posting index itself is very compact (~1 byte per indexed value). +The covering sidecar adds storage proportional to the included columns: + +- **Numeric columns** (DOUBLE, FLOAT): compressed with ALP (Adaptive + Lossless floating-Point) and Frame-of-Reference bitpacking +- **Integer columns** (INT, LONG, etc.): Frame-of-Reference bitpacking; + TIMESTAMP additionally uses linear-prediction encoding +- **Small fixed-width types** (BYTE, BOOLEAN, etc.): stored as raw copies +- **Wide fixed-width types** (UUID, LONG256, DECIMAL128/256): stored as + raw copies with a count header +- **Variable-width columns** (VARCHAR, STRING): FSST compressed in sealed + partitions, typically 2-5x smaller than raw column data +- **BINARY and arrays**: stored in an offset-based variable-width sidecar + +### Write performance + +Write overhead depends on the number and type of INCLUDE columns. Typical +ranges (measured with 100K row inserts, 50 symbol keys): + +- **Posting index without INCLUDE**: ~15-20% slower than no index +- **Posting index with fixed-width INCLUDE** (DOUBLE, INT): ~40-50% slower +- **Posting index with VARCHAR INCLUDE**: ~2x slower + +Actual overhead varies with row size, cardinality, and hardware. Query +performance improvements typically far outweigh the write cost for +read-heavy workloads. + +### Memory + +The posting index uses native memory for encoding/decoding buffers. +The covering index's FSST symbol tables use ~70KB of native memory per +compressed column per active reader. + +## Architecture + +The posting index stores data in three file types per partition: + +- **`.pk`** — Key file: double-buffered metadata pages with generation + directory (32 bytes per generation entry) +- **`.pv`** — Value file: delta + Frame-of-Reference bitpacked row IDs, + organized into stride-indexed generations +- **`.pci` + `.pc0`, `.pc1`, ...** — Sidecar files: covered column values + stored alongside the posting list, one file per INCLUDE column + +### Generations and sealing + +Data is written incrementally as **generations** (one per commit). Each +generation contains a sparse block of key→rowID mappings. Periodically, +generations are **sealed** into a single dense generation with stride-indexed +layout for optimal read performance. + +Sealing happens automatically when the generation count reaches the maximum +(125) or when the partition is closed. Sealed data uses two encoding modes +per stride (256 keys): + +- **Delta mode** (`POSTING DELTA`): per-key delta encoding with bitpacking — + compresses best for regular, evenly-distributed row IDs and is faster for + large sequential scans +- **Elias-Fano mode** (`POSTING EF`): stride-wide Frame-of-Reference with + contiguous bitpacking — compresses better for irregular distributions and + is faster for point queries + +With the default adaptive encoding (`POSTING`), the encoder trial-encodes +both modes per stride and picks the smaller one. + +### FSST compression for strings + +VARCHAR and STRING columns in the INCLUDE list are compressed using FSST +(Fast Static Symbol Table) compression during sealing. FSST replaces +frequently occurring 1-8 byte patterns with single-byte codes, typically +achieving 2-5x compression on string data with repetitive patterns. + +The FSST symbol table is trained per stride block and stored inline in the +sidecar file. Decompression is transparent to the query engine. + +## Limitations + +:::warning + +- `INCLUDE` is only supported for the posting index type (not bitmap) +- `INCLUDE` cannot list the indexed symbol column itself +- `INCLUDE` is not supported with out-of-line `INDEX(col ...)` syntax — + use inline column syntax or `ALTER TABLE` instead +- `CAPACITY` is not supported for posting indexes (bitmap only) +- `SAMPLE BY` queries do not currently use the covering index + (they fall back to the regular index path) +- `REINDEX` on WAL tables requires dropping and re-adding the index + (this applies to all index types, not just posting) + +::: diff --git a/documentation/concepts/deep-dive/sql-optimizer-hints.md b/documentation/concepts/deep-dive/sql-optimizer-hints.md index 93f207598..7a989a54b 100644 --- a/documentation/concepts/deep-dive/sql-optimizer-hints.md +++ b/documentation/concepts/deep-dive/sql-optimizer-hints.md @@ -358,3 +358,37 @@ your symbol set is high-cardinality. - superseded by `asof_index` - `asof_memoized_search` - superseded by `asof_memoized` + +----- + +## Index hints + +These hints control whether the query optimizer uses indexes (bitmap or posting) +for symbol column lookups. + +### no_covering + +Disables the [covering index](/docs/concepts/deep-dive/posting-index/) +optimization, forcing the query to read from column files instead of the +index sidecar. The index is still used for row ID lookup, but column values +are read from the main column files. + +```questdb-sql +SELECT /*+ no_covering */ price FROM trades WHERE symbol = 'AAPL'; +``` + +This is useful for benchmarking covering index performance or working around +a specific issue with the covering path. + +### no_index + +Completely disables all index usage for the query, including bitmap index, +posting index, and covering index. The query falls back to a full table scan +with a filter applied to every row. Also implies `no_covering`. + +```questdb-sql +SELECT /*+ no_index */ price FROM trades WHERE symbol = 'AAPL'; +``` + +This is useful for benchmarking index effectiveness or forcing a table scan +when you know the filter selectivity is low (many rows match). diff --git a/documentation/concepts/symbol.md b/documentation/concepts/symbol.md index 2e5bf740f..bfd7b0be5 100755 --- a/documentation/concepts/symbol.md +++ b/documentation/concepts/symbol.md @@ -117,6 +117,7 @@ ALTER TABLE trades ALTER COLUMN client_id CACHE; For columns frequently used in `WHERE` clauses, add an index: ```questdb-sql +-- Bitmap index (default) — low overhead, good for most cases CREATE TABLE trades ( timestamp TIMESTAMP, symbol SYMBOL INDEX, @@ -124,10 +125,23 @@ CREATE TABLE trades ( ) TIMESTAMP(timestamp) PARTITION BY DAY; ``` +For read-heavy workloads, a [posting index](/docs/concepts/deep-dive/posting-index/) +offers better compression and supports covering queries: + +```questdb-sql +-- Posting index with covering columns — reads from compact sidecar files +CREATE TABLE trades ( + timestamp TIMESTAMP, + symbol SYMBOL INDEX TYPE POSTING INCLUDE (price), + price DOUBLE +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; +``` + Or add an index later: ```questdb-sql ALTER TABLE trades ALTER COLUMN symbol ADD INDEX; +-- or: ALTER TABLE trades ALTER COLUMN symbol ADD INDEX TYPE POSTING; ``` See [Indexes](/docs/concepts/deep-dive/indexes/) for more information. diff --git a/documentation/configuration/configuration-utils/_cairo.config.json b/documentation/configuration/configuration-utils/_cairo.config.json index 9fd0f8689..64f928008 100644 --- a/documentation/configuration/configuration-utils/_cairo.config.json +++ b/documentation/configuration/configuration-utils/_cairo.config.json @@ -81,7 +81,15 @@ }, "cairo.index.value.block.size": { "default": "256", - "description": "Approximation of number of rows for a single index key, must be power of 2." + "description": "Approximation of number of rows for a single index key, must be power of 2. Applies to bitmap indexes only; posting indexes manage their own block layout." + }, + "cairo.posting.index.auto.include.timestamp": { + "default": "true", + "description": "When `true`, the designated timestamp column is automatically added to the covering index when a [posting index](/docs/concepts/deep-dive/posting-index/) is created with an `INCLUDE` clause." + }, + "cairo.posting.index.row.id.encoding": { + "default": "posting", + "description": "Default row ID encoding for posting indexes. Valid values: `posting` (adaptive delta/flat trial encoding) and `posting_delta` (delta-only encoding)." }, "cairo.max.swap.file.count": { "default": "30", @@ -105,7 +113,7 @@ }, "cairo.spin.lock.timeout": { "default": "1000", - "description": "Timeout when attempting to get BitmapIndexReaders in millisecond." + "description": "Timeout in milliseconds when attempting to acquire index readers (bitmap and posting)." }, "cairo.character.store.capacity": { "default": "1024", diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 832f56bb2..d7d450b9f 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -594,6 +594,10 @@ Returns a `table` with the following columns: - `indexed` - if indexing is applied to this column - `indexBlockCapacity` - how many row IDs to store in a single storage block on disk +- `indexType` - the [index type](/docs/concepts/deep-dive/indexes/) + (`POSTING`, `BITMAP`, or empty) +- `indexInclude` - comma-separated names of columns included in a + [posting index's](/docs/concepts/deep-dive/posting-index/) covering sidecar - `symbolCached` - whether this `symbol` column is cached - `symbolCapacity` - how many distinct values this column of `symbol` type is expected to have @@ -611,12 +615,12 @@ For more details on the meaning and use of these values, see the table_columns('my_table'); ``` -| column | type | indexed | indexBlockCapacity | symbolCached | symbolCapacity | designated | upsertKey | -| ------ | --------- | ------- | ------------------ | ------------ | -------------- | ---------- | --------- | -| symb | SYMBOL | true | 1048576 | false | 256 | false | false | -| price | DOUBLE | false | 0 | false | 0 | false | false | -| ts | TIMESTAMP | false | 0 | false | 0 | true | false | -| s | VARCHAR | false | 0 | false | 0 | false | false | +| column | type | indexed | indexBlockCapacity | indexType | indexInclude | symbolCached | symbolCapacity | designated | upsertKey | +| ------ | --------- | ------- | ------------------ | --------- | ------------ | ------------ | -------------- | ---------- | --------- | +| symb | SYMBOL | true | 1048576 | | | false | 256 | false | false | +| price | DOUBLE | false | 0 | | | false | 0 | false | false | +| ts | TIMESTAMP | false | 0 | | | false | 0 | true | false | +| s | VARCHAR | false | 0 | | | false | 0 | false | false | ```questdb-sql title="Get designated timestamp column" SELECT "column", type, designated FROM table_columns('my_table') WHERE designated = true; diff --git a/documentation/query/sql/alter-mat-view-alter-column-add-index.md b/documentation/query/sql/alter-mat-view-alter-column-add-index.md index d8b5d2b27..d866e788b 100644 --- a/documentation/query/sql/alter-mat-view-alter-column-add-index.md +++ b/documentation/query/sql/alter-mat-view-alter-column-add-index.md @@ -12,6 +12,7 @@ query performance for filtered lookups. ``` ALTER MATERIALIZED VIEW viewName ALTER COLUMN columnName ADD INDEX [ CAPACITY n ] +ALTER MATERIALIZED VIEW viewName ALTER COLUMN columnName ADD INDEX TYPE POSTING ``` ## Parameters @@ -20,7 +21,8 @@ ALTER MATERIALIZED VIEW viewName ALTER COLUMN columnName ADD INDEX [ CAPACITY n | --------- | ----------- | | `viewName` | Name of the materialized view | | `columnName` | Name of the `SYMBOL` column to index | -| `CAPACITY` | Optional index capacity (advanced; use default unless you understand implications) | +| `CAPACITY` | Optional index capacity for bitmap indexes (advanced; use default unless you understand implications) | +| `TYPE POSTING` | Use a [posting index](/docs/concepts/deep-dive/posting-index/) instead of the default bitmap index | ## When to use @@ -30,13 +32,29 @@ Add an index when: - The column has high cardinality (many distinct values) - Query performance on the materialized view needs improvement -## Example +## Examples -```questdb-sql title="Add index to symbol column" +### Adding a bitmap index (default) + +```questdb-sql title="Add bitmap index to symbol column" ALTER MATERIALIZED VIEW trades_hourly ALTER COLUMN symbol ADD INDEX; ``` +### Adding a posting index + +```questdb-sql title="Add posting index to symbol column" +ALTER MATERIALIZED VIEW trades_hourly + ALTER COLUMN symbol ADD INDEX TYPE POSTING; +``` + +:::note + +The `INCLUDE` clause for covering indexes is not supported on materialized +views. Use a posting index without `INCLUDE` for faster filtered lookups. + +::: + ## Behavior | Aspect | Description | diff --git a/documentation/query/sql/alter-table-alter-column-add-index.md b/documentation/query/sql/alter-table-alter-column-add-index.md index 3ddea73cb..12d77467a 100644 --- a/documentation/query/sql/alter-table-alter-column-add-index.md +++ b/documentation/query/sql/alter-table-alter-column-add-index.md @@ -10,13 +10,52 @@ Indexes an existing [`symbol`](/docs/concepts/symbol/) column. ![Flow chart showing the syntax of the ALTER TABLE ALTER COLUMN ADD INDEX keyword](/images/docs/diagrams/alterTableAddIndex.svg) - Adding an [index](/docs/concepts/deep-dive/indexes/) is an atomic, non-blocking, and non-waiting operation. Once complete, the SQL optimizer will start using the new index for SQL executions. -## Example +## Examples + +### Adding a bitmap index (default) -```questdb-sql title="Adding an index" +```questdb-sql ALTER TABLE trades ALTER COLUMN instrument ADD INDEX; ``` + +### Adding a posting index + +```questdb-sql +ALTER TABLE trades ALTER COLUMN instrument ADD INDEX TYPE POSTING; +``` + +An encoding variant can be specified: + +```questdb-sql +-- Force delta-only encoding +ALTER TABLE trades ALTER COLUMN instrument ADD INDEX TYPE POSTING DELTA; +``` + +### Adding a posting index with covering columns + +The `INCLUDE` clause stores additional column values in the index sidecar +files, enabling covering queries that bypass column file reads: + +```questdb-sql +ALTER TABLE trades + ALTER COLUMN symbol ADD INDEX TYPE POSTING INCLUDE (price, quantity); +``` + +The designated timestamp column is automatically included in the covering +index — you do not need to list it explicitly. + +After this, queries that only select columns from the `INCLUDE` list (plus the +indexed symbol column and designated timestamp) are served from the index +sidecar: + +```questdb-sql +-- This query reads from the index sidecar, not from column files +SELECT timestamp, price FROM trades WHERE symbol = 'AAPL'; +``` + +See [Posting index and covering index](/docs/concepts/deep-dive/posting-index/) +for supported column types and performance details. diff --git a/documentation/query/sql/create-table.md b/documentation/query/sql/create-table.md index ac77b4e20..6705f7753 100644 --- a/documentation/query/sql/create-table.md +++ b/documentation/query/sql/create-table.md @@ -475,6 +475,8 @@ must be of type [symbol](/docs/concepts/symbol/). ![Flow chart showing the syntax of the index function](/images/docs/diagrams/indexDef.svg) +### Bitmap index (default) + ```questdb-sql CREATE TABLE trades ( timestamp TIMESTAMP, @@ -484,13 +486,78 @@ CREATE TABLE trades ( ), INDEX(symbol) TIMESTAMP(timestamp); ``` +### Posting index + +The posting index offers better compression and read performance than the +default bitmap index. Use `INDEX TYPE POSTING` with either inline or +out-of-line syntax: + +```questdb-sql +-- Inline syntax +CREATE TABLE trades ( + timestamp TIMESTAMP, + symbol SYMBOL INDEX TYPE POSTING, + price DOUBLE, + amount DOUBLE +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; + +-- Out-of-line syntax +CREATE TABLE trades ( + timestamp TIMESTAMP, + symbol SYMBOL, + price DOUBLE, + amount DOUBLE +), INDEX(symbol TYPE POSTING) +TIMESTAMP(timestamp) PARTITION BY DAY WAL; +``` + +### Posting index with covering columns (INCLUDE) + +The `INCLUDE` clause stores additional column values in the index sidecar +files. Queries that only need these columns plus the indexed symbol can be +served entirely from the index, bypassing column files: + +```questdb-sql +CREATE TABLE trades ( + timestamp TIMESTAMP, + symbol SYMBOL INDEX TYPE POSTING INCLUDE (price, exchange), + exchange SYMBOL, + price DOUBLE, + amount DOUBLE +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; +``` + +The designated timestamp column is automatically included — you do not need +to list it in the `INCLUDE` clause. With this schema, the following query +reads only from the index sidecar: + +```questdb-sql +SELECT timestamp, price FROM trades WHERE symbol = 'AAPL'; +``` + +:::note + +`INCLUDE` is only supported with inline column syntax (not out-of-line +`INDEX(col ...)`). Use `ALTER TABLE` to add covering columns to an existing +table. + +::: + +See [Posting index and covering index](/docs/concepts/deep-dive/posting-index/) +for a comprehensive guide including supported column types, query patterns, +and performance characteristics. + :::warning - The **index capacity** and [**symbol capacity**](/docs/concepts/symbol/) are different settings. - The index capacity value should not be changed, unless a user is aware of all - the implications. ::: + the implications. +- `CAPACITY` is only supported for bitmap indexes — it cannot be used with + posting indexes. + +::: See the [Index concept](/docs/concepts/deep-dive/indexes/#how-indexes-work) for more information about indexes. diff --git a/documentation/query/sql/explain.md b/documentation/query/sql/explain.md index d3858d806..cabe99f72 100644 --- a/documentation/query/sql/explain.md +++ b/documentation/query/sql/explain.md @@ -76,6 +76,12 @@ The following list contains some plan node types: `INTERSECT`). - `Index forward/backward scan` - scans all row ids associated with a given `symbol` value from start to finish or vice versa. +- `CoveringIndex` - reads data from a + [posting index's](/docs/concepts/deep-dive/posting-index/) covering sidecar + files instead of main column files. Appears when all selected columns are + covered by the `INCLUDE` clause. +- `PostingIndex` - uses a posting index for accelerated operations such as + `DISTINCT` on a symbol column. - `Limit` - standalone node implementing the `LIMIT` keyword. Other nodes can implement `LIMIT` internally, e.g. the `Sort` node. - `Row forward/backward scan` - scans data frame (usually partitioned) records diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 5d14121fe..6bdcf9c12 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -57,13 +57,18 @@ SHOW TABLES; SHOW COLUMNS FROM trades; ``` -| column | type | indexed | indexBlockCapacity | symbolCached | symbolCapacity | symbolTableSize | designated | upsertKey | -| --------- | --------- | ------- | ------------------ | ------------ | -------------- | --------------- | ---------- | --------- | -| symbol | SYMBOL | false | 0 | true | 256 | 42 | false | false | -| side | SYMBOL | false | 0 | true | 256 | 2 | false | false | -| price | DOUBLE | false | 0 | false | 0 | 0 | false | false | -| amount | DOUBLE | false | 0 | false | 0 | 0 | false | false | -| timestamp | TIMESTAMP | false | 0 | false | 0 | 0 | true | false | +| column | type | indexed | indexBlockCapacity | indexType | indexInclude | symbolCached | symbolCapacity | symbolTableSize | designated | upsertKey | +| --------- | --------- | ------- | ------------------ | --------- | ------------ | ------------ | -------------- | --------------- | ---------- | --------- | +| symbol | SYMBOL | false | 0 | | | true | 256 | 42 | false | false | +| side | SYMBOL | false | 0 | | | true | 256 | 2 | false | false | +| price | DOUBLE | false | 0 | | | false | 0 | 0 | false | false | +| amount | DOUBLE | false | 0 | | | false | 0 | 0 | false | false | +| timestamp | TIMESTAMP | false | 0 | | | false | 0 | 0 | true | false | + +The `indexType` column shows the index type (`POSTING`, `BITMAP`, or empty for +non-indexed columns). The `indexInclude` column lists the names of columns +included in a [posting index's](/docs/concepts/deep-dive/posting-index/) +covering sidecar, as a comma-separated string. ### SHOW CREATE TABLE @@ -88,6 +93,22 @@ CREATE TABLE trades ( WITH maxUncommittedRows=500000, o3MaxLag=600000000us; ``` +#### Posting index with covering columns + +When a symbol column has a posting index with `INCLUDE`, the DDL reflects +the index type and covered columns: + +```questdb-sql +CREATE TABLE trades ( + symbol SYMBOL CAPACITY 128 CACHE INDEX TYPE POSTING INCLUDE (price, exchange), + exchange SYMBOL CAPACITY 128 CACHE, + price DOUBLE, + amount DOUBLE, + timestamp TIMESTAMP +) timestamp(timestamp) PARTITION BY DAY WAL +WITH maxUncommittedRows=500000, o3MaxLag=600000000us; +``` + #### Per-column Parquet encoding When columns have per-column Parquet encoding or compression overrides, they diff --git a/documentation/schema-design-essentials.md b/documentation/schema-design-essentials.md index 592e9d09f..b88a6c929 100644 --- a/documentation/schema-design-essentials.md +++ b/documentation/schema-design-essentials.md @@ -75,6 +75,47 @@ TIMESTAMP(ts) PARTITION BY MONTH; See [Partitions](/docs/concepts/partitions/) for details. +## Indexing + +Index your primary filter columns to speed up `WHERE` clause queries. QuestDB +supports two index types for SYMBOL columns: + +```questdb-sql +-- Default bitmap index — low overhead, good for most cases +CREATE TABLE trades ( + ts TIMESTAMP, + symbol SYMBOL INDEX, + price DOUBLE +) TIMESTAMP(ts) PARTITION BY DAY WAL; + +-- Posting index with covering columns — best for read-heavy, selective queries +CREATE TABLE trades ( + ts TIMESTAMP, + symbol SYMBOL INDEX TYPE POSTING INCLUDE (price), + price DOUBLE, + raw_data VARCHAR -- not in INCLUDE, read from column files +) TIMESTAMP(ts) PARTITION BY DAY WAL; +-- The designated timestamp (ts) is automatically included in the covering index. +``` + +**When to choose each:** + +| Scenario | Recommendation | +|----------|---------------| +| General purpose, write-heavy | Bitmap index (`INDEX`) | +| Read-heavy, filtering on symbol | Posting index (`INDEX TYPE POSTING`) | +| Frequent queries on a few columns | Posting with `INCLUDE` | +| Wide table, queries select subset | Posting with `INCLUDE` — biggest win | + +The covering index (`INCLUDE`) lets queries that only select covered columns +read from compact sidecar files instead of full column files. The designated +timestamp is automatically included, so timestamp-filtered queries benefit +without explicit listing. Use `EXPLAIN` to verify your queries use the +`CoveringIndex` plan. + +See [Indexes](/docs/concepts/deep-dive/indexes/) and +[Posting index](/docs/concepts/deep-dive/posting-index/) for details. + ## Data types ### SYMBOL vs VARCHAR diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 0a83522d1..9c3850465 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -538,6 +538,7 @@ module.exports = { collapsed: true, items: [ "concepts/deep-dive/indexes", + "concepts/deep-dive/posting-index", "concepts/deep-dive/interval-scan", "concepts/deep-dive/jit-compiler", "concepts/deep-dive/query-tracing",