Skip to content

[core] Fix data evolution scan pruning files after a column type change#8843

Open
ArnavBalyan wants to merge 1 commit into
apache:masterfrom
ArnavBalyan:arnavb/fix-data-evolution-type-change-prune
Open

[core] Fix data evolution scan pruning files after a column type change#8843
ArnavBalyan wants to merge 1 commit into
apache:masterfrom
ArnavBalyan:arnavb/fix-data-evolution-type-change-prune

Conversation

@ArnavBalyan

Copy link
Copy Markdown
Member

Purpose

  • Data data evolution drops the rows when a column type/nullability changes
  • This is because evolutionStats leaves the -1 offset for a column whose stored type differs from the current type, which DataEvolutionArray encodes as all null.
  • LeafPredicate then prunes the pre alter version of the file and loses rows.
  • Correct such columns to the -2 so the file is kept

Tests

  • UT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant