fix: handle shallow clones in dbt clone, table, and incremental rebuilds#1592
Open
saishreeeee wants to merge 6 commits into
Open
fix: handle shallow clones in dbt clone, table, and incremental rebuilds#1592saishreeeee wants to merge 6 commits into
saishreeeee wants to merge 6 commits into
Conversation
saishreeeee
changed the title
saishreeeee
marked this pull request as ready for review
July 17, 2026 04:42
sd-db
reviewed
Jul 17, 2026
| ## dbt-databricks 1.12.3 (TBD) | ||
|
|
||
| ### Fixes | ||
| - Fix rebuilding a table or incremental model over an existing shallow clone, and `dbt clone` over an existing managed table, failing with `INVALID_PARAMETER_VALUE.UPDATE_TABLE_TYPE`/`CLUSTER_BY_AUTO_UNSUPPORTED_TABLE_TYPE_ERROR`; the shallow clone is now dropped before it is recreated, and `CLUSTER BY AUTO` is skipped when altering a shallow clone in place ([#1592](https://github.com/databricks/dbt-databricks/pull/1592) resolves [#1165](https://github.com/databricks/dbt-databricks/issues/1165)) |
Collaborator
There was a problem hiding this comment.
nit: let's keep the changelog comment terse (~1 line) and not add implementation specifics
| shallow clone {{ defer_relation.render() }} | ||
| {% endmacro %} | ||
|
|
||
| {#-- Drop first unless already a shallow clone; `create or replace` can't change table_type in place. --#} |
Collaborator
There was a problem hiding this comment.
nit: we can remove the comment or make it more accurate here it is both table + shallow_clone
| {% else %} | ||
| {#-- Relation must be dropped & recreated --#} | ||
| {% if not is_replaceable %} {#-- If Delta, we will `create or replace` below, so no need to drop --#} | ||
| {% if not is_replaceable or existing_relation.is_shallow_clone %} {#-- Delta uses `create or replace` below, except a shallow clone must be dropped (table_type can't change in place) --#} |
Collaborator
There was a problem hiding this comment.
nit: drop the (table_type can't change in place), mostly implementation details. better not to have in comment.
| {%- elif existing_relation.is_view or existing_relation.is_materialized_view or existing_relation.is_streaming_table or should_full_refresh() -%} | ||
| {#-- Relation must be dropped & recreated --#} | ||
| {% if not is_replaceable_format %} {#-- If Delta or Iceberg, we will `create or replace` below, so no need to drop --#} | ||
| {% if not is_replaceable_format or existing_relation.is_shallow_clone %} {#-- Delta/Iceberg uses `create or replace` below, except a shallow clone must be dropped (table_type can't change in place) --#} |
Collaborator
There was a problem hiding this comment.
nit: drop (table_type can't change in place)
| @@ -3,6 +3,7 @@ | |||
| import pytest | |||
Collaborator
There was a problem hiding this comment.
Let's look to add functional tests
Collaborator
There was a problem hiding this comment.
(a) clone --full-refresh over an existing managed table
(b) table/incremental rebuild over a shallow clone with auto_liquid_cluster, with information_schema
/ history assertions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #1165
Description
Shallow clones have a distinct table_type that can't be changed in place, so
dbt cloneand table/incremental rebuilds over a clone failed (UPDATE_TABLE_TYPE,CLUSTER_BY_AUTO_UNSUPPORTED_TABLE_TYPE_ERROR).Adds
is_shallow_clone, drops the clone before recreating it (clone/table/incremental), and skipsCLUSTER BY AUTOwhen altering a clone in place.Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section./dbt-databricks-pr-ready(AI agent skill in.claude/skills/) and addressed its merge-readiness feedback