[Bug] SELECT * FROM <view> throws "Index N out of bounds for length N" after ALTER TABLE ADD COLUMNS + REFRESH TABLE on the underlying Hive table

[zhaorongsheng]

Search before asking

I searched for "view schema drift index out of bounds" and "external catalog view schema change" in the issue tracker and did not find a duplicate.

Doris Version

master (confirmed in LogicalView.java as of commit aa91628)

What's Wrong

Querying a Doris view that was created with SELECT * over an external (Hive) catalog table fails with an IndexOutOfBoundsException after:

1. Adding a new column to the underlying Hive table (ALTER TABLE … ADD COLUMNS)
2. Refreshing the base table metadata in Doris (REFRESH TABLE <base_table>)

The view itself is not refreshed; only the base table is refreshed.

Error:

ERROR 1105 (HY000): errCode = 2, detailMessage = Index 3 out of bounds for length 3

Reproducer

-- Step 1: Create Hive table (3 columns + partition)
-- (executed in Hive)
CREATE TABLE test.test_view_schema_drift (
  id     bigint,
  name   string,
  age    string
)
PARTITIONED BY (dt string)
STORED AS PARQUET;

-- Step 2: In Doris (Hive catalog context)
SWITCH hive;
DESCRIBE test.test_view_schema_drift;   -- shows 3 non-partition columns

CREATE VIEW test.test_view AS
  SELECT * FROM test.test_view_schema_drift
  WHERE dt = date_sub(current_date(), 1);

SELECT * FROM test.test_view WHERE 1=0;
-- OK: returns 3 columns (empty result)

-- Step 3: Add a column in Hive
ALTER TABLE test.test_view_schema_drift ADD COLUMNS (score string COMMENT 'new col');

-- Step 4: Refresh base table in Doris
SWITCH hive;
REFRESH TABLE test.test_view_schema_drift;

SELECT * FROM test.test_view_schema_drift WHERE 1=0;
-- OK: returns 4 columns now

SELECT * FROM test.test_view WHERE 1=0;
-- FAIL: Index 3 out of bounds for length 3

Root Cause

LogicalView.computeOutput() iterates over childOutput (the output of the re-analyzed view body, which reflects the refreshed 4-column base table). For each slot it calls view.getFullSchema().get(i).

view.getFullSchema() is derived from the view's metadata in the Hive metastore, which was created when the base table had 3 columns. Since only REFRESH TABLE base_table was called (not REFRESH TABLE view), the view's stored schema still has 3 columns. When i = 3, get(3) throws IndexOutOfBoundsException.

// LogicalView.java – before fix
for (int i = 0; i < childOutput.size(); i++) {   // childOutput.size() = 4
    ...
    if (CollectionUtils.isEmpty(view.getFullSchema())) {
        qualified = originSlot.withQualifier(fullQualifiers);
    } else {
        // BUG: view.getFullSchema().size() == 3, crashes at i == 3
        qualified = originSlot.withOneLevelTableAndColumnAndQualifier(
            view, view.getFullSchema().get(i), fullQualifiers);
    }
}

The isEmpty() guard added in #40715 handles null/empty fullSchema but not the under-sized case introduced by schema drift.

Expected Behavior

Querying the view should not throw. The new column (added after view creation) should appear in the result set with its qualifier correctly applied (falling back to withQualifier() as the existing null-guard branch does).

Impact / Workaround

Impact: Any user who

1. Creates a Doris VIEW (using SELECT *) on an external catalog table, AND
2. Later adds columns to the base table and calls REFRESH TABLE <base_table>

will hit this crash when querying the view.

Workaround: Execute REFRESH TABLE <view> (or DROP VIEW + CREATE VIEW) after adding columns, so that the view's stored schema is also refreshed before querying.

Proposed Fix

Extend the guard condition to also cover i >= fullSchema.size():

List<Column> fullSchema = view.getFullSchema();
if (CollectionUtils.isEmpty(fullSchema) || i >= fullSchema.size()) {
    qualified = originSlot.withQualifier(fullQualifiers);
} else {
    qualified = originSlot
            .withOneLevelTableAndColumnAndQualifier(view, fullSchema.get(i), fullQualifiers);
}

PR: (to be linked after submission)

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_969e40840992

Initial triage: this looks like a real Nereids/external-view schema-drift bug on current master, not just a reporting artifact.

I re-read the issue and checked the referenced upstream tree (aa9162840f1). The reported failure path matches the code:

• For HMS external views, BindRelation reads HMSExternalTable.getViewText(), analyzes that SQL in the external catalog/database context, and wraps the analyzed plan in LogicalView(new ExternalView(table, ddlSql), ...).
• ExternalView.getFullSchema() delegates to the external view table's own getFullSchema(), i.e. the schema cached/loaded for the HMS view object.
• REFRESH TABLE <base_table> only invalidates the selected external table cache through RefreshManager.refreshTableInternal(...)->invalidateTableCache(table). It does not refresh the separate view table object.
• LogicalView.computeOutput() then iterates over child().getOutput() but indexes view.getFullSchema().get(i) for every child slot. If the view SQL is re-analyzed after the base table refresh and SELECT * now expands to 4 child slots while the view object's stored schema still has 3 columns, get(3) is exactly enough to produce Index 3 out of bounds for length 3.

So the issue's proposed diagnosis is consistent with the current code chain. The existing CollectionUtils.isEmpty(view.getFullSchema()) guard only protects null/empty schemas; it does not protect a non-empty but shorter stored view schema.

The suggested local fix is reasonable as a minimal crash fix:

List<Column> fullSchema = view.getFullSchema();
if (CollectionUtils.isEmpty(fullSchema) || i >= fullSchema.size()) {
    qualified = originSlot.withQualifier(fullQualifiers);
} else {
    qualified = originSlot.withOneLevelTableAndColumnAndQualifier(view, fullSchema.get(i), fullQualifiers);
}

One semantic point should be made explicit before merging: if Doris intends HMS external views to follow the re-analyzed view SQL, then falling back to withQualifier() for newly expanded slots is consistent with the current analyzer behavior and avoids losing the new column. If Doris intends the stored HMS view schema to be the authoritative output contract until REFRESH TABLE <view>, then the fix should instead reconcile/cap the child output to the stored view schema rather than only guarding the index. Either way, the current uncaught IndexOutOfBoundsException is a bug.

Recommended next steps:

1. Add a regression test for an HMS external view whose stored view schema has fewer columns than the re-analyzed view body output. The important assertion is that LogicalView.computeOutput() does not throw when childOutput.size() > view.getFullSchema().size().
2. Reuse view.getFullSchema() in a local variable inside computeOutput() so the code does not repeatedly fetch a potentially cache-backed schema during the loop.
3. Confirm the intended result-column contract for SELECT * external views after base-table schema drift: expose the newly added base column after base refresh, or keep the old view schema until the view itself is refreshed.
4. Keep the documented workaround for affected users: run REFRESH TABLE <view> or recreate the view after changing the underlying Hive table schema.

The issue currently has no labels; this should probably be routed to the Nereids + external catalog/HMS view owners.