Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 14 additions & 6 deletions docs/docs/multimodal-table/global-index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -311,8 +311,8 @@ catalog.alter_table(

Global index files cover row-id ranges. If more rows are appended after an index is built, those
new rows are not automatically covered by the existing index files. Build the global index again
to create index files for newly uncovered data. By default, queries use fast search and only read
indexed row ranges; rows in uncovered ranges are not returned for that indexed query.
to create index files for newly uncovered data. Scalar queries default to `full` search, while
vector and full-text queries default to `fast` and only search indexed row ranges.

To improve freshness for query types that support raw-data search, set:

Expand All @@ -321,7 +321,7 @@ To improve freshness for query types that support raw-data search, set:
<TabItem value="sql" label="SQL">

```sql
ALTER TABLE my_table SET ('global-index.search-mode' = 'full');
ALTER TABLE my_table SET ('vector-index.search-mode' = 'full');
```

</TabItem>
Expand All @@ -333,14 +333,14 @@ from pypaimon.schema.schema_change import SchemaChange

catalog.alter_table(
"db.my_table",
[SchemaChange.set_option("global-index.search-mode", "full")],
[SchemaChange.set_option("vector-index.search-mode", "full")],
)
```

For a read-only override on an existing `Table` instance:

```python
full_table = table.copy({"global-index.search-mode": "full"})
full_table = table.copy({"vector-index.search-mode": "full"})
```

</TabItem>
Expand All @@ -353,6 +353,11 @@ only when such a gap exists. Use `detail` search when data files may have been r
after index creation; it scans data file metadata to find the exact unindexed row ranges and can
handle index invalidation caused by updates or rewrites.

Use `scalar-index.search-mode`, `vector-index.search-mode`, or
`full-text-index.search-mode` for one index family. The legacy
`global-index.search-mode` has no default and is used as a fallback when the corresponding
family-specific option is not explicitly configured.

To temporarily disable global-index scan acceleration while keeping the index files, set:

<Tabs groupId="global-index-enabled">
Expand Down Expand Up @@ -389,7 +394,10 @@ These table options affect global index build and read behavior:
| Option | Default | Description |
|---|---|---|
| `global-index.enabled` | `true` | Whether scans can use global indexes. |
| `global-index.search-mode` | `fast` | Search mode for global-index queries. `fast` searches indexed data only. `full` checks snapshot `nextRowId` against global index row-id coverage and scans raw data only if a gap exists. `detail` scans data file metadata to find exact unindexed rows and can handle index invalidation caused by updates or rewrites. |
| `global-index.search-mode` | Not set | Legacy fallback search mode for global-index queries. Family-specific options take precedence. |
| `scalar-index.search-mode` | `full` | Search mode for BTree and Bitmap queries. |
| `vector-index.search-mode` | `fast` | Search mode for vector queries. |
| `full-text-index.search-mode` | `fast` | Search mode for full-text queries. |
| `global-index.external-path` | Not set | Root directory for global index files. If not set, files are stored under the table index directory. |
| `global-index.column-update-action` | `THROW_ERROR` | Action for updates to indexed columns. `THROW_ERROR` rejects the update, `DROP_PARTITION_INDEX` drops affected partition indexes, and `IGNORE` keeps existing index files unchanged. Use `IGNORE` only when a stale index is acceptable; for example, a vector updated from `NULL` to a value remains invisible until the index is rebuilt. |
| `sorted-index.records-per-range` | `10000000` | Expected number of records per sorted global index file for BTree and Bitmap builds. |
Expand Down
4 changes: 2 additions & 2 deletions docs/docs/multimodal-table/global-index/full-text.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -282,8 +282,8 @@ print(pa_table)

</Tabs>

By default, `global-index.search-mode=fast` searches indexed row ranges only.
When `global-index.search-mode` is `full` or `detail`, Paimon also covers
By default, `full-text-index.search-mode=fast` searches indexed row ranges only.
When `full-text-index.search-mode` is `full` or `detail`, Paimon also covers
unindexed row ranges by reading raw rows and building a temporary native
full-text index for the searched column.

Expand Down
14 changes: 8 additions & 6 deletions docs/docs/primary-key-table/global-index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,9 @@ schema validation.
| `fields.<column>.pk-btree.index.options` | Not set | JSON object containing BTree build options. Unqualified keys are scoped to `btree-index`. |
| `pk-bitmap.index.columns` | Not set | Comma-separated columns which own independent Bitmap indexes. |
| `fields.<column>.pk-bitmap.index.options` | Not set | JSON object containing Bitmap build options. Unqualified keys are scoped to `bitmap-index`. |
| `global-index.search-mode` | `fast` | Search mode for primary-key Vector and Full Text queries. `fast` searches indexed data only, so uncovered files are omitted. For Vector, `full` and `detail` search uncovered files exactly. Primary-key Full Text supports only `fast`. |
| `global-index.search-mode` | Not set | Legacy fallback search mode. Family-specific options take precedence. |
| `vector-index.search-mode` | `fast` | Search mode for primary-key Vector queries. `full` and `detail` search uncovered files exactly. |
| `full-text-index.search-mode` | `fast` | Search mode for primary-key Full Text queries. Only `fast` is currently supported. |

For algorithm-specific options, see the corresponding
[BTree](../multimodal-table/global-index/btree),
Expand Down Expand Up @@ -272,18 +274,18 @@ Index construction can execute asynchronously inside the writer. A writer which
compaction also waits for active index maintenance; a non-blocking writer can complete maintenance
in a later commit. Coverage can therefore be temporarily partial.

Coverage behavior depends on the index family and search mode. `global-index.search-mode` defaults
to `fast`. For primary-key Vector and Full Text queries, `fast` searches only data covered by an
active index group. Uncovered files are not searched, so partial coverage can make results
incomplete.
Coverage behavior depends on the index family and search mode. `vector-index.search-mode` and
`full-text-index.search-mode` default to `fast`. In `fast` mode, primary-key Vector and Full Text
queries search only data covered by an active index group. Uncovered files are not searched, so
partial coverage can make results incomplete.

- BTree and Bitmap scans always read uncovered files through the ordinary data path. Partial
coverage affects their acceleration, not result completeness. The original scalar predicate and
deletion vectors are applied after index pruning.
- Vector search in `full` or `detail` mode evaluates files without an active ANN group exactly and
merges those results with ANN candidates. In the default `fast` mode, those files are omitted.

Primary-key Full Text currently supports only `global-index.search-mode = fast`. It searches
Primary-key Full Text currently supports only `full-text-index.search-mode = fast`. It searches
persistent archives and ignores uncovered files; `full` and `detail` are rejected because a
merge-aware logical-row fallback is not implemented. Consequently, newly appended Level-0 rows
become full-text searchable only after compaction publishes an eligible data file and archive.
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pypaimon/data-evolution.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ You can use `update_by_predicate` for SQL-like `UPDATE ... SET ... WHERE ...`
operations. The `Predicate` identifies rows to update, and the assignment map
contains literal values for updated columns.
When global indexes are available, `update_by_predicate` discovers matching
`_ROW_ID` values with `global-index.search-mode=full` on the configured
`_ROW_ID` values with `scalar-index.search-mode=full` on the configured
point-in-time scan snapshot or, if none is configured, the latest snapshot.

```python
Expand Down
22 changes: 20 additions & 2 deletions docs/generated/core_configuration.html
Original file line number Diff line number Diff line change
Expand Up @@ -752,6 +752,12 @@
<td>Integer</td>
<td>For streaming write, full compaction will be constantly triggered after delta commits. For batch write, full compaction will be triggered with each commit as long as this value is greater than 0.</td>
</tr>
<tr>
<td><h5>full-text-index.search-mode</h5></td>
<td style="word-wrap: break-word;">fast</td>
<td><p>Enum</p></td>
<td>Search mode for full-text index queries.<br /><br />Possible values:<ul><li>"fast": Only search indexed data.</li><li>"full": Use snapshot next row id and global index coverage to detect missing row ids, and scan raw data only when a gap exists.</li><li>"detail": Scan data files to find exact unindexed rows. This can handle index invalidation caused by updates or rewrites.</li></ul></td>
</tr>
<tr>
<td><h5>global-index.build.max-parallelism</h5></td>
<td style="word-wrap: break-word;">4096</td>
Expand Down Expand Up @@ -790,9 +796,9 @@
</tr>
<tr>
<td><h5>global-index.search-mode</h5></td>
<td style="word-wrap: break-word;">fast</td>
<td style="word-wrap: break-word;">(none)</td>
<td><p>Enum</p></td>
<td>Search mode for global index queries. Supported values are 'fast', 'full', and 'detail'.<br /><br />Possible values:<ul><li>"fast": Only search indexed data.</li><li>"full": Use snapshot next row id and global index coverage to detect missing row ids, and scan raw data only when a gap exists.</li><li>"detail": Scan data files to find exact unindexed rows. This can handle index invalidation caused by updates or rewrites.</li></ul></td>
<td>Fallback search mode for global index queries.<br /><br />Possible values:<ul><li>"fast": Only search indexed data.</li><li>"full": Use snapshot next row id and global index coverage to detect missing row ids, and scan raw data only when a gap exists.</li><li>"detail": Scan data files to find exact unindexed rows. This can handle index invalidation caused by updates or rewrites.</li></ul></td>
</tr>
<tr>
<td><h5>global-index.thread-num</h5></td>
Expand Down Expand Up @@ -1320,6 +1326,12 @@
<td>String</td>
<td>The field that generates the row kind for primary key table, the row kind determines which data is '+I', '-U', '+U' or '-D'.</td>
</tr>
<tr>
<td><h5>scalar-index.search-mode</h5></td>
<td style="word-wrap: break-word;">full</td>
<td><p>Enum</p></td>
<td>Search mode for scalar index queries.<br /><br />Possible values:<ul><li>"fast": Only search indexed data.</li><li>"full": Use snapshot next row id and global index coverage to detect missing row ids, and scan raw data only when a gap exists.</li><li>"detail": Scan data files to find exact unindexed rows. This can handle index invalidation caused by updates or rewrites.</li></ul></td>
</tr>
<tr>
<td><h5>scan.bounded.watermark</h5></td>
<td style="word-wrap: break-word;">(none)</td>
Expand Down Expand Up @@ -1747,6 +1759,12 @@
<td>String</td>
<td>Specifies column names that should be stored as vector type. This is used when you want to treat a ARRAY column as a VECTOR.</td>
</tr>
<tr>
<td><h5>vector-index.search-mode</h5></td>
<td style="word-wrap: break-word;">fast</td>
<td><p>Enum</p></td>
<td>Search mode for vector index queries.<br /><br />Possible values:<ul><li>"fast": Only search indexed data.</li><li>"full": Use snapshot next row id and global index coverage to detect missing row ids, and scan raw data only when a gap exists.</li><li>"detail": Scan data files to find exact unindexed rows. This can handle index invalidation caused by updates or rewrites.</li></ul></td>
</tr>
<tr>
<td><h5>vector-search.distribute.enabled</h5></td>
<td style="word-wrap: break-word;">false</td>
Expand Down
44 changes: 41 additions & 3 deletions paimon-api/src/main/java/org/apache/paimon/CoreOptions.java
Original file line number Diff line number Diff line change
Expand Up @@ -2705,11 +2705,27 @@ public String toString() {

public static final ConfigOption<GlobalIndexSearchMode> GLOBAL_INDEX_SEARCH_MODE =
key("global-index.search-mode")
.enumType(GlobalIndexSearchMode.class)
.noDefaultValue()
.withDescription("Fallback search mode for global index queries.");

public static final ConfigOption<GlobalIndexSearchMode> SCALAR_INDEX_SEARCH_MODE =
key("scalar-index.search-mode")
.enumType(GlobalIndexSearchMode.class)
.defaultValue(GlobalIndexSearchMode.FULL)
.withDescription("Search mode for scalar index queries.");

public static final ConfigOption<GlobalIndexSearchMode> VECTOR_INDEX_SEARCH_MODE =
key("vector-index.search-mode")
.enumType(GlobalIndexSearchMode.class)
.defaultValue(GlobalIndexSearchMode.FAST)
.withDescription(
"Search mode for global index queries. "
+ "Supported values are 'fast', 'full', and 'detail'.");
.withDescription("Search mode for vector index queries.");

public static final ConfigOption<GlobalIndexSearchMode> FULL_TEXT_INDEX_SEARCH_MODE =
key("full-text-index.search-mode")
.enumType(GlobalIndexSearchMode.class)
.defaultValue(GlobalIndexSearchMode.FAST)
.withDescription("Search mode for full-text index queries.");

public static final ConfigOption<Integer> GLOBAL_INDEX_THREAD_NUM =
key("global-index.thread-num")
Expand Down Expand Up @@ -4312,10 +4328,32 @@ public boolean globalIndexEnabled() {
return options.get(GLOBAL_INDEX_ENABLED);
}

@Nullable
public GlobalIndexSearchMode globalIndexSearchMode() {
return options.get(GLOBAL_INDEX_SEARCH_MODE);
}

public GlobalIndexSearchMode scalarIndexSearchMode() {
return indexSearchMode(SCALAR_INDEX_SEARCH_MODE);
}

public GlobalIndexSearchMode vectorIndexSearchMode() {
return indexSearchMode(VECTOR_INDEX_SEARCH_MODE);
}

public GlobalIndexSearchMode fullTextIndexSearchMode() {
return indexSearchMode(FULL_TEXT_INDEX_SEARCH_MODE);
}

private GlobalIndexSearchMode indexSearchMode(
ConfigOption<GlobalIndexSearchMode> familySearchMode) {
if (options.contains(familySearchMode)) {
return options.get(familySearchMode);
}
return options.getOptional(GLOBAL_INDEX_SEARCH_MODE)
.orElseGet(() -> options.get(familySearchMode));
}

public Integer globalIndexThreadNum() {
return options.get(GLOBAL_INDEX_THREAD_NUM);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -315,7 +315,7 @@ private Optional<GlobalIndexResult> evalGlobalIndex() {
LOG.info(
"Scan table '{}' with global index. searchMode='{}', total={} ms, metadata={} ms, lookup={} ms, coverage={} ms.",
table.name(),
options.globalIndexSearchMode(),
options.scalarIndexSearchMode(),
totalDuration / 1_000_000,
metadataDuration / 1_000_000,
lookupDuration / 1_000_000,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,16 +51,32 @@ public class DataEvolutionGlobalIndexCoverage {
private final FileStoreTable table;
@Nullable private final Snapshot snapshot;
@Nullable private final PartitionPredicate partitionFilter;
private final GlobalIndexSearchMode searchMode;
private final Map<Integer, List<Range>> coverageByField;

public DataEvolutionGlobalIndexCoverage(
FileStoreTable table,
@Nullable Snapshot snapshot,
@Nullable PartitionPredicate partitionFilter,
Collection<IndexFileMeta> indexFiles) {
this(
table,
snapshot,
partitionFilter,
indexFiles,
table.coreOptions().scalarIndexSearchMode());
}

public DataEvolutionGlobalIndexCoverage(
FileStoreTable table,
@Nullable Snapshot snapshot,
@Nullable PartitionPredicate partitionFilter,
Collection<IndexFileMeta> indexFiles,
GlobalIndexSearchMode searchMode) {
this.table = table;
this.snapshot = snapshot;
this.partitionFilter = partitionFilter;
this.searchMode = checkNotNull(searchMode);
this.coverageByField = new HashMap<>();
for (IndexFileMeta indexFile : indexFiles) {
GlobalIndexMeta meta = checkNotNull(indexFile.globalIndexMeta());
Expand All @@ -83,7 +99,6 @@ public List<Range> unindexedRanges(int fieldId) {
}

public List<Range> unindexedRanges(Collection<Integer> fieldIds) {
GlobalIndexSearchMode searchMode = table.coreOptions().globalIndexSearchMode();
if (searchMode == GlobalIndexSearchMode.FAST) {
return Collections.emptyList();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,12 @@ private DataEvolutionGlobalIndexScanner(
GlobalIndexReadThreadPool.getExecutorService(options.get(GLOBAL_INDEX_THREAD_NUM));
this.indexPathFactory = indexPathFactory;
this.coverage =
new DataEvolutionGlobalIndexCoverage(table, snapshot, partitionFilter, indexFiles);
new DataEvolutionGlobalIndexCoverage(
table,
snapshot,
partitionFilter,
indexFiles,
table.coreOptions().scalarIndexSearchMode());
GlobalIndexFileReader indexFileReader = meta -> fileIO.newInputStream(meta.filePath());
Map<Integer, IndexMetaFileGroup> indexMetas = new HashMap<>();
Map<Integer, List<IndexMetaFileGroup>> extraIndexMetas = new HashMap<>();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,11 @@ public Plan scan() {
if (!allIndexFiles.isEmpty()) {
List<Range> rawRowRanges =
new DataEvolutionGlobalIndexCoverage(
table, snapshot, partitionFilter, allIndexFiles)
table,
snapshot,
partitionFilter,
allIndexFiles,
table.coreOptions().fullTextIndexSearchMode())
.unindexedRanges(textColumnIds);
if (!rawRowRanges.isEmpty()) {
splits.add(new RawFullTextSearchSplit(rawRowRanges));
Expand Down
Loading
Loading