diff --git a/docs/tables/schema.mdx b/docs/tables/schema.mdx index 807fece..6ff0577 100644 --- a/docs/tables/schema.mdx +++ b/docs/tables/schema.mdx @@ -196,6 +196,145 @@ Add timestamp columns that can contain NULL values: When adding columns that should contain NULL values, be sure to cast the NULL to the appropriate type, e.g., `cast(NULL as timestamp)`. +### Declare computed columns + +You can also declare a column whose values are defined by a SQL expression but +not evaluated at commit time. LanceDB stores the expression in the column's +field metadata, commits the column with no values, and fills the rows on a +later refresh. The column's type and its input columns are derived from the +expression, so you do not pass a data type. + +Use this form when you want to add a derived column to a large table without +paying the cost of computing every row up front. Declaring a computed column +costs the same on an empty table as on a large one, because no values are +written at declaration time. Regular `add_columns` transforms, in contrast, +evaluate the SQL expression against every existing row and write the results +in the same commit. + + +```python Python icon="python" +# Declare a computed column; values are filled by a later refresh. +table.add_columns(computed={"doubled": "x * 2"}) +``` + +```typescript TypeScript icon="square-js" +// Declare a computed column; values are filled by a later refresh. +await table.addColumns({ + computed: [{ name: "doubled", valueSql: "x * 2" }], +}); +``` + +```rust Rust icon="rust" +// Declare a computed column; values are filled by a later refresh. +table + .add_columns() + .computed("doubled", "x * 2") + .execute() + .await?; +``` + + +A declaration stays authoritative for the column's lifetime. While it is in +place, LanceDB rejects writes and schema changes that would give the column a +value or reshape its output: + +- `add`, `update`, `merge_insert`, and SQL `INSERT` are refused for the + declared column. +- The declared column cannot be renamed, retyped, or dropped. +- An input column named in the expression cannot be renamed, retyped, or + dropped while the declaration reads it. +- Volatile expressions (for example, expressions whose value can change + between calls) are refused at declaration time. + +A refresh fills every fragment that has no value for the declared column, +including fragments appended since the last refresh. A refresh does not +revisit a fragment it has already filled, so mutating an input row leaves +the previously computed value in place. To recompute values, drop the column +and declare it again. + + +Computed columns work on both local tables and LanceDB Enterprise. On +Enterprise the declaration is sent to the server, which plans the +expression against the published contract; refresh runs as a server-side +backfill job (see the next section). + + + +`add_columns` cannot mix a regular transform with a computed column in the +same call. Declare computed columns in a separate `add_columns` call from +any evaluated transforms. + + +### Refresh a computed column + +A declared computed column starts empty. Call `refresh_column` (Python and +Rust) or `refreshColumn` (TypeScript) to evaluate the expression and fill +every row that still has no value: + + +```python Python icon="python" +result = table.refresh_column("doubled") +print(result.rows_filled, result.version) +``` + +```typescript TypeScript icon="square-js" +const { rowsFilled, version } = await table.refreshColumn("doubled"); +``` + +```rust Rust icon="rust" +let result = table.refresh_column("doubled").await?; +println!("filled {} rows at version {}", result.rows_filled, result.version); +``` + + +The call returns the number of rows it filled and the new table version. Each +run picks up rows appended since the previous refresh; rows that already have +a value are left alone, so calling `refresh_column` when nothing new needs +filling is a no-op that costs one scan of the unfilled rows. Because refresh +never revisits a filled row, mutating an input after the fact does not change +the stored value — to recompute, drop the column and declare it again. + +The blocking form is refused when the table uses an LSM write specification, +and is refused on LanceDB Enterprise because a remote refresh runs as a +server job that does not report a fill count. On Enterprise, submit the +refresh with the async form below instead. + +#### Run the refresh in the background + +If you don't want to block on the refresh, call the async variant to get back +a job handle. On local tables the job runs as an in-process task; on LanceDB +Enterprise the call submits a server-side backfill job and returns a handle +that tracks it. Wait for it or poll its status when convenient. + + +```python Python icon="python" +job = table.refresh_column_async("doubled") +job.wait() +print(job.status()) # "finished" +``` + +```typescript TypeScript icon="square-js" +const job = await table.refreshColumnAsync("doubled"); +await job.wait(); +console.log(await job.status()); // "finished" +``` + +```rust Rust icon="rust" +let job = table.refresh_column_async("doubled").await?; +job.wait().await?; +``` + + +Invalid input — an unknown column, or one that is not a declared computed +column — is reported by the submitting call rather than by the job, so you +learn about mistakes before you start waiting. The returned job may already +be complete; treat the column as filled only after `wait` returns. + +On LanceDB Enterprise, a successful `wait` also advances the submitting +table handle's read-freshness baseline so subsequent reads see the refreshed +rows — unless a `checkout` has pinned the handle to a specific version by the +time the job completes. + ## Alter existing columns You can alter columns using the [`alter_columns`](https://lancedb.github.io/lancedb/python/python/#lancedb.table.Table.alter_columns)