Support ALTER TABLE ... ALTER COLUMN ... TYPE ... USING <expr>

[zachmu]

Adds support for ALTER TABLE ... ALTER COLUMN ... TYPE ... USING .

Fixes #3119.

[github-actions]

	Main	PR
Total	42090	42090
Successful	18981	18986
Failures	23109	23104
Partial Successes1	5461	5457

	Main	PR
Successful	45.0962%	45.1081%
Failures	54.9038%	54.8919%

${\color{red}Regressions (1)}$

subselect

QUERY:          select count(*) from tenk1 t
where (exists(select 1 from tenk1 k where k.unique1 = t.unique2) or ten < 0);
RECEIVED ERROR: timeout during Receive

${\color{lightgreen}Progressions (8)}$

alter_table

QUERY: alter table anothertab alter column atcol2 type text
      using case when atcol2 is true then 'IT WAS TRUE'
                 when atcol2 is false then 'IT WAS FALSE'
                 else 'IT WAS NULL!' end;

QUERY: alter table at_partitioned alter column b type numeric using b::numeric;

QUERY: ALTER TABLE test_type_diff ALTER COLUMN f2 TYPE bigint USING f2::bigint;

QUERY: ALTER TABLE test_type_diff2 ALTER COLUMN int_four TYPE int8 USING int_four::int8;

compression

QUERY: ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer;

fast_default

QUERY: ALTER TABLE vtype2 ALTER COLUMN b TYPE varchar(20) USING b::varchar(20);

inherit

QUERY: alter table a alter column aa type integer using bit_length(aa);

plpgsql

QUERY: ALTER TABLE alter_table_under_transition_tables
  ALTER COLUMN name TYPE int USING name::integer;

Footnotes

1. These are tests that we're marking as Successful, however they do not match the expected output in some way. This is due to small differences, such as different wording on the error messages, or the column names being incorrect while the data itself is correct. ↩

[itoqa]

Commit: f8d2096: 14 test cases ran, 1 failed ❌, 13 passed ✅.

Summary

Coverage spans successful and rejected data-type changes, value conversion, schema qualification, key and index integrity, dependency handling, rollback and recovery, concurrency, and adversarial constraint cases. Most behaviors remain healthy, but failed conversions are not reliably atomic and can corrupt existing stored data while leaving the table writable.

Not safe to merge yet — this PR has an attributable high-severity data-integrity failure: a late conversion error can expose partial schema and rewrite state, making existing rows and indexed access disappear. The documented constraint behavior is a separate by-design caveat, but it does not offset the merge-blocking corruption risk.

Tests run by Ito

Result	Severity	Type	Description
❌		General	The conversion reported a failure, but the table was left with integer type metadata, no original rows available through the index, and a row count of zero in the recorded SQL output. The follow-up insert succeeded, which shows the table was usable but still corrupted from the failed conversion.
✅	—	General	The conversion was rejected when the final row produced NULL, and the original column type, required-value rule, and all four rows stayed intact.
✅	—	General	The qualified table changed its value from text 22 to integer 22. The same-named table in the current schema stayed text with value 11.
✅	—	General	A row inserted while the type change was running stayed available through both the primary key and the secondary index. The type change stopped with a clear conflict instead of reporting success after losing the row.
✅	—	Conversion	The conversion updated both existing names to the expected full names, and the column remained a text field.
✅	—	Foreign	Type changes on both sides of a foreign key were rejected, and the dependent row-type change was rejected too. The original table definitions and data stayed unchanged.
✅	—	Index	The text primary key changed to an integer, the lookup for alpha still returned payload one, and inserting another row with id 1 was rejected.
✅	—	Rev	Converting the two text keys to integers was rejected because both would become 1. The column stayed text, both original rows remained available, and a new row with key 2 was inserted successfully.
✅	—	Rev	The text values were converted to integers, lookups still found the right row, and the unique index rejected a duplicate value. Both original rows remained intact after the rejected insert.
✅	—	Rev	Verified acceptable by independent adversarial review: the observed behavior is intended and documented in this codebase. Review notes: The claimed integrity behavior is statically credible and was introduced by this PR, but it is also the exact behavior explicitly disclosed in the PR's Known limitations: CHECK constraints referencing the changed column are not revalidated. Under the judge policy's intent check, that documented scope tradeoff makes this finding by-design rather than an undisclosed implementation defect; the indepe…
✅	—	Rev	The invalid conversion failed, rollback restored the original table, and a later insert committed successfully.
✅	—	Rewrite	The conversion failed on the invalid value, and the table kept its text column and both original rows.
✅	—	Schema	Both tables changed from text to integer values. The qualified table kept value 7, and the table in the current schema kept value 8.
✅	—	Validation	Changing a table column that another table uses as a row type was rejected with the expected error. The source column stayed integer, and the dependent row kept its original value.

Tip

Reply with @itoqa to send us feedback on this test run.

[itoqa]

Late conversion failure loses table data

What failed: The conversion reported a failure, but the table was left with integer type metadata, no original rows available through the index, and a row count of zero in the recorded SQL output. The follow-up insert succeeded, which shows the table was usable but still corrupted from the failed conversion.

Impact · Steps · Stub / mock · Analysis · Why this is likely a bug

• Severity: High
• Impact: A failed column conversion can leave the table with missing rows and the wrong column type. Users may lose access to stored records even though the table still accepts new writes.
• Steps to Reproduce:
  1. Create a table with an integer primary key, a text value column, a secondary index, and many valid numeric text values followed by one value such as 'not-a-number'.
  2. Run ALTER TABLE late_failure_rows ALTER COLUMN value TYPE integer USING value::integer.
  3. After the conversion fails, inspect the value column type, count the rows, and query the original rows through the secondary index.
  4. Try a normal insert and read it back to confirm that the table accepts new data even though the original data was not restored.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: AlterTableColumnTypeUsing.RowIter creates a RewriteInserter with the old and new primary-key schemas at server/node/alter_table_column_type_using.go:190-196, then scans the source table and inserts each converted row as it goes at lines 203-234. A conversion error from Eval or NewType.Convert at lines 212-220, a NOT NULL error at lines 223-225, or an insert error at lines 231-232 calls abortRewrite. The PR-added abortRewrite implementation at lines 275-279 calls inserter.DiscardChanges(ctx, err) and inserter.Close(ctx) but intentionally ignores both returned errors before returning only the original conversion error. Therefore, if the storage engine cannot discard or close the staged rewrite cleanly, the caller receives the expected conversion error while the incremental physical changes remain visible. The recorded result matches this path: the value column became integer, the original rows and indexed lookups disappeared, and a new insert worked afterward. The smallest practical fix is to preserve the original conversion error while detecting cleanup failure and ensuring the RewriteInserter rollback/close path either completes atomically or surfaces an explicit unrecoverable cleanup error without exposing the partial rewrite.
• Why this is likely a bug: A failed schema conversion must leave the original table, rows, primary key, and secondary indexes intact; otherwise one bad value can cause data loss. The test placed the invalid value after many valid rows and observed changed type metadata and missing indexed rows, which is the direct harmful outcome of exposing incremental rewrite state after cleanup fails. This is not a setup-only issue: the application was responsible for the rewrite and the relevant implementation was introduced by this PR. The targeted fix is to make failed rewrites atomic and to handle cleanup errors instead of silently discarding them.

Relevant code

server/node/alter_table_column_type_using.go:193-234

inserter, err := rwt.RewriteInserter(ctx, oldPkSchema, newPkSchema, oldCol, &newCol, nil)
...
rowIter := sql.NewTableRowIter(ctx, rwt, partitions)
for {
    row, err := rowIter.Next(ctx)
    ...
    newVal, err := conversionExpr.Eval(ctx, row)
    ...
    if err != nil {
        return nil, abortRewrite(ctx, inserter, err)
    }
    ...
    if err = inserter.Insert(ctx, newRow); err != nil {
        return nil, abortRewrite(ctx, inserter, err)
    }
}

server/node/alter_table_column_type_using.go:275-279

func abortRewrite(ctx *sql.Context, inserter sql.RowInserter, err error) error {
    _ = inserter.DiscardChanges(ctx, err)
    _ = inserter.Close(ctx)
    return err
}

Evidence Package

• text/BF-FAIL-1/BF-FAIL-1-late-conversion-corruption.txt — 3 KB

Copy prompt for an agent

Ito QA identified the following failure during automated PR testing. Please investigate and propose a fix.

**High severity — Late conversion failure loses table data**

**What failed:** The conversion reported a failure, but the table was left with integer type metadata, no original rows available through the index, and a row count of zero in the recorded SQL output. The follow-up insert succeeded, which shows the table was usable but still corrupted from the failed conversion.

- **Impact:** A failed column conversion can leave the table with missing rows and the wrong column type. Users may lose access to stored records even though the table still accepts new writes.
- **Steps to reproduce:**
  1. Create a table with an integer primary key, a text value column, a secondary index, and many valid numeric text values followed by one value such as 'not-a-number'.
  2. Run ALTER TABLE late_failure_rows ALTER COLUMN value TYPE integer USING value::integer.
  3. After the conversion fails, inspect the value column type, count the rows, and query the original rows through the secondary index.
  4. Try a normal insert and read it back to confirm that the table accepts new data even though the original data was not restored.
- **Stub / mock content:** No stubs, mocks, or bypasses were applied for this test in the recorded run.
- **Code analysis:** AlterTableColumnTypeUsing.RowIter creates a RewriteInserter with the old and new primary-key schemas at server/node/alter_table_column_type_using.go:190-196, then scans the source table and inserts each converted row as it goes at lines 203-234. A conversion error from Eval or NewType.Convert at lines 212-220, a NOT NULL error at lines 223-225, or an insert error at lines 231-232 calls abortRewrite. The PR-added abortRewrite implementation at lines 275-279 calls inserter.DiscardChanges(ctx, err) and inserter.Close(ctx) but intentionally ignores both returned errors before returning only the original conversion error. Therefore, if the storage engine cannot discard or close the staged rewrite cleanly, the caller receives the expected conversion error while the incremental physical changes remain visible. The recorded result matches this path: the value column became integer, the original rows and indexed lookups disappeared, and a new insert worked afterward. The smallest practical fix is to preserve the original conversion error while detecting cleanup failure and ensuring the RewriteInserter rollback/close path either completes atomically or surfaces an explicit unrecoverable cleanup error without exposing the partial rewrite.
- **Why this is likely a bug:** A failed schema conversion must leave the original table, rows, primary key, and secondary indexes intact; otherwise one bad value can cause data loss. The test placed the invalid value after many valid rows and observed changed type metadata and missing indexed rows, which is the direct harmful outcome of exposing incremental rewrite state after cleanup fails. This is not a setup-only issue: the application was responsible for the rewrite and the relevant implementation was introduced by this PR. The targeted fix is to make failed rewrites atomic and to handle cleanup errors instead of silently discarding them.

**Relevant code:**

`server/node/alter_table_column_type_using.go:193-234`

~~~go
inserter, err := rwt.RewriteInserter(ctx, oldPkSchema, newPkSchema, oldCol, &newCol, nil)
...
rowIter := sql.NewTableRowIter(ctx, rwt, partitions)
for {
    row, err := rowIter.Next(ctx)
    ...
    newVal, err := conversionExpr.Eval(ctx, row)
    ...
    if err != nil {
        return nil, abortRewrite(ctx, inserter, err)
    }
    ...
    if err = inserter.Insert(ctx, newRow); err != nil {
        return nil, abortRewrite(ctx, inserter, err)
    }
}
~~~

`server/node/alter_table_column_type_using.go:275-279`

~~~go
func abortRewrite(ctx *sql.Context, inserter sql.RowInserter, err error) error {
    _ = inserter.DiscardChanges(ctx, err)
    _ = inserter.Close(ctx)
    return err
}
~~~

[coffeegoddd]

@zachmu DOLT

read_tests	from_latency_median	to_latency_median	is_faster
covering_index_scan_postgres	2.43	2.48	0
groupby_scan_postgres	75.82	75.82	0
index_join_postgres	2.18	2.18	0
index_join_scan_postgres	1.58	1.58	0
index_scan_postgres	484.44	484.44	0
oltp_point_select	0.36	0.36	0
oltp_read_only	6.32	6.21	0
select_random_points	0.7	0.7	0
select_random_ranges	1.01	1.01	0
table_scan_postgres	493.24	484.44	0
types_table_scan_postgres	1213.57	1213.57	0

write_tests	from_latency_median	to_latency_median	is_faster
bulk_insert	0.001	0.001	0
oltp_delete_insert_postgres	6.67	6.67	0
oltp_insert	3.36	3.36	0
oltp_read_write	13.22	13.22	0
oltp_update_index	3.55	3.55	0
oltp_update_non_index	3.25	3.25	0
oltp_write_only	6.91	6.91	0
types_delete_insert_postgres	7.17	7.17	0