Skip to content

HIVE-29641: Upgrade Calcite to 1.42.0 #6523

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rubenada wants to merge 40 commits into
base: master
Choose a base branch
from
Open

HIVE-29641: Upgrade Calcite to 1.42.0 #6523

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
114dffb
Bump Calcite version to 1.38, minor code adjustments:
rubenada Jun 3, 2026
139518d
Use non-deprecated version of RexBuilder#makeAbstractCast in HiveSqlS…
rubenada Jun 3, 2026
ad26672
HiveHepExtractRelNodeRule: do the conversion using a shuttle, instead…
rubenada Jun 3, 2026
92cd4db
In druid-handler pom-xml: add asm and asm-commons (version 9.7) depen…
rubenada Jun 4, 2026
ce10ab8
Update Calcite default charset system property (use UTF16 instead of …
rubenada Jun 4, 2026
4c08d3c
In MaterializationView rules, we cannot use HiveHepExtractRelNodeRule…
rubenada Jun 4, 2026
e45f4c3
Fix HiveRelFieldTrimmer#rewriteGBConstantKeys: the condition on its j…
rubenada Jun 4, 2026
e4378ca
Add asm dependency to parent pom, trying to fix the javadoc error see…
rubenada Jun 5, 2026
8e663c7
Adjust test plans due to
rubenada Jun 8, 2026
fcd2822
Temp: trying to force druid-handler rebuild on Jenkins...
rubenada Jun 8, 2026
9005061
Bump Calcite to 1.42 + required code adjustments:
rubenada Jun 8, 2026
f30fc05
Adjust test plans due to change in representation of floating point v…
rubenada Jun 8, 2026
0501cd1
In CalcitePlanner genFilterRelNode: simplify filter condition before …
rubenada Jun 10, 2026
e3754ba
Move fix from previous commit as earlier as possible (incorrect binar…
rubenada Jun 10, 2026
348a2eb
Avoid incorrect comparison ($snapshotIdInputRef <= NULL) in HiveAugme…
rubenada Jun 11, 2026
e1d74a7
In druid-handler pom add dependency to asm 9.6 to avoid error on java…
rubenada Jun 12, 2026
37692f1
minor: adjust test plan, use ORDER BY ... <ordinal> instead or repeat…
rubenada Jun 12, 2026
e188f5b
Identify NOT BETWEEN expression in SearchTransformer to avoid plan ch…
rubenada Jun 15, 2026
9ba25da
Previous change (identify NOT BETWEEN expression in SearchTransformer…
rubenada Jun 15, 2026
c115604
Test adjustments: rowcount changes in TestExplainCBOFormattedCliDrive…
rubenada Jun 17, 2026
df9725a
Bump maven.shade.plugin.version to 3.6.2 to resolve druid-handler jav…
rubenada Jun 18, 2026
92cd9fc
Override HiveTypeSystemImpl#deriveDecimalDivideType because CALCITE-6…
rubenada Jun 18, 2026
2beae20
Add ASM dependency 9.9.1 for druid handler shade plugin (still trying…
rubenada Jun 18, 2026
306f0ca
Adjust test output stats_histogram.q.out
rubenada Jun 19, 2026
fbf8ee7
Rever changes on Jenkinsfile + Fix druid-handler shade ASM error by r…
rubenada Jun 19, 2026
5df3003
Adjust test files: condition reordering, IS [NOT] NULL simplification…
rubenada Jun 19, 2026
ed76c18
Due to "[CALCITE-6044] RelMetadataQuery should regard single-row rela…
rubenada Jun 23, 2026
4277d01
Test adjustments (input8.q.out, lineage2.q.out, lineage3.q.out, ppd_c…
rubenada Jun 25, 2026
9d7a343
Adjust test file cross_prod_1.q.out: Equivalent plan; expression "A.v…
rubenada Jun 25, 2026
f12d57a
Adjust test file vector_aggregate_9.q.out: Decima cast is not removed.
rubenada Jun 29, 2026
56e7111
Test adjustment vector_coalesce.q.out: equivalent plan, with some IS_…
rubenada Jun 29, 2026
1b3c628
Test adjustment vector_interval_2.q.out: simplified predicates
rubenada Jun 29, 2026
0eebb14
Test adjustment join45.q.out, join47.q.out, mapjoin47.q, smb_mapjoin4…
rubenada Jun 29, 2026
283a073
Test adjustment join46.q.out, mapjoin46.q.out, vectorized_join46.q.ou…
rubenada Jun 30, 2026
779e484
Test adjustment iceberg_bucket_map_join_1.q.out:
rubenada Jul 1, 2026
851e103
Update tests files:
rubenada Jul 3, 2026
fdac744
Add org.jooq:joou-java-6 to ql/pom.xml to resolve init-metastore CI e…
rubenada Jul 6, 2026
ae05ff9
Add Calcite bug reference
rubenada Jul 14, 2026
7534912
Review:
rubenada Jul 17, 2026
5a4b5ea
Fix TestHivePointLookupOptimizerRule
rubenada Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
19 changes: 17 additions & 2 deletions druid-handler/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,10 +353,24 @@
<pattern>io.netty</pattern>
<shadedPattern>org.apache.hive.druid.io.netty</shadedPattern>
</relocation>
<!--
Calcite is intentionally NOT included or relocated here.
Calcite's calcite-core ships org/apache/calcite/runtime/SqlFunctions.class with a
malformed RuntimeVisibleTypeAnnotations attribute (LOCAL_VARIABLE start_pc/length
pointing inside instruction operands in the static initializer). The JVM tolerates it;
ASM's ClassRemapper does not, and shading + javadoc fails with
"Index 65536 out of bounds for length 334 [...] Error in ASM processing class".
Present in Calcite 1.35.0 through at least 1.42.0. Upstream: ASM gitlab issue
https://gitlab.ow2.org/asm/asm/-/issues/318008 (closed as invalid: root cause is on
Calcite's side); Calcite ticket: https://issues.apache.org/jira/browse/CALCITE-6393
(still unresolved). Druid 0.17.1 uses Calcite APIs compatible with Hive's Calcite 1.42+,
so skipping this relocation is safe. If Druid is upgraded to an incompatible Calcite,
restore the relocation (either after Calcite fixes their bytecode or by adding a shade
filter that excludes SqlFunctions.class from remapping).
Comment on lines +356 to +369

@zabetak zabetak Jul 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Relocation is one thing and inclusion in the shaded jar is another. I suppose they included calcite in the shaded jar because the latter is send to workers (running in different JVMs) with a different classpath. Not sure if the Druid module can still work if you remove calcite from shading.

Druid 0.17.1 is using calcite 1.21.0. I highly doubt that all the used APIs are compatible with 1.42.0. In fact, I am rather skeptical about the compatibility with previous upgrades as well (1.25.0 and 1.33.0).

Out of curiosity, I tried running TestMiniDruidCliDriver today and realized that the Druid module is broken since 2021. I send also an email to https://lists.apache.org/thread/qr0xxkvvg4wrlz4kg0bmktldw96yvpn2 explaining the current status and favoring the complete removal of the module.

All in all, the changes here do not really matter and we don't have to sweat too much about it since it's only a matter of days before getting rid of the entire module.

<relocation>
<pattern>org.apache.calcite</pattern>
<shadedPattern>org.apache.hive.druid.org.apache.calcite</shadedPattern>
</relocation>
</relocation> -->
<relocation>
<pattern>org.jboss.netty</pattern>
<shadedPattern>org.apache.hive.druid.org.jboss.netty</shadedPattern>
Expand All @@ -379,7 +393,8 @@
<include>org.apache.druid:*</include>
<include>org.apache.druid.extensions:*</include>
<include>io.netty:*</include>
<include>org.apache.calcite:*</include>
<!-- See comment above about Calcite not being relocated & included -->
<!-- <include>org.apache.calcite:*</include> -->
<include>com.fasterxml.jackson.core:*</include>
<include>com.fasterxml.jackson.datatype:*</include>
<include>com.fasterxml.jackson.dataformat:*</include>
Expand Down
4 changes: 2 additions & 2 deletions hbase-handler/src/test/results/positive/hbase_queries.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ STAGE PLANS:
alias: hbase_table_1
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: UDFToDouble(key) is not null (type: boolean)
predicate: key is not null (type: boolean)
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: UDFToDouble(key) (type: double)
Expand Down Expand Up @@ -604,7 +604,7 @@ STAGE PLANS:
alias: hbase_table_1
Statistics: Num rows: 1 Data size: 188 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: UDFToDouble(key) is not null (type: boolean)
predicate: key is not null (type: boolean)
Statistics: Num rows: 1 Data size: 188 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: int), value (type: string), UDFToDouble(key) (type: double)
Expand Down
103 changes: 56 additions & 47 deletions iceberg/iceberg-handler/src/test/results/positive/dynamic_partition_pruning.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1458,37 +1458,40 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcpart_double_hour_n0
filterExpr: ((UDFToDouble(hour) = 11.0D) and CAST( UDFToInteger((hr / 2.0D)) AS STRING) is not null) (type: boolean)
filterExpr: ((UDFToDouble(hour) = 11.0D) and hr is not null) (type: boolean)
Statistics: Num rows: 2 Data size: 188 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ((UDFToDouble(hour) = 11.0D) and CAST( UDFToInteger((hr / 2.0D)) AS STRING) is not null) (type: boolean)
predicate: ((UDFToDouble(hour) = 11.0D) and hr is not null) (type: boolean)

@zabetak zabetak Jul 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new plan is equivalent but it seems that the simplification didn't lead to a significant improvement. The simplification here seems to have introduced a new Filter[col0 is not null] higher up so at the end of the day the number of comparisons remains the same.

Statistics: Num rows: 1 Data size: 94 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: CAST( UDFToInteger((hr / 2.0D)) AS STRING) (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
Filter Operator
predicate: _col0 is not null (type: boolean)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
keys: _col0 (type: string)
minReductionHashAggr: 0.4
mode: hash
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Dynamic Partitioning Event Operator
Target column: hr (string)
Target Input: srcpart_iceberg
Partition key expr: hr
Group By Operator
keys: _col0 (type: string)
minReductionHashAggr: 0.4
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Target Vertex: Map 1
Dynamic Partitioning Event Operator
Target column: hr (string)
Target Input: srcpart_iceberg
Partition key expr: hr
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Target Vertex: Map 1
Execution mode: vectorized
Reducer 2
Reduce Operator Tree:
Expand Down Expand Up @@ -1587,21 +1590,24 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcpart_double_hour_n0
filterExpr: ((UDFToDouble(hour) = 11.0D) and CAST( UDFToInteger((hr / 2.0D)) AS STRING) is not null) (type: boolean)
filterExpr: ((UDFToDouble(hour) = 11.0D) and hr is not null) (type: boolean)
Statistics: Num rows: 2 Data size: 188 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ((UDFToDouble(hour) = 11.0D) and CAST( UDFToInteger((hr / 2.0D)) AS STRING) is not null) (type: boolean)
predicate: ((UDFToDouble(hour) = 11.0D) and hr is not null) (type: boolean)
Statistics: Num rows: 1 Data size: 94 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: CAST( UDFToInteger((hr / 2.0D)) AS STRING) (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
Filter Operator
predicate: _col0 is not null (type: boolean)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: vectorized
Reducer 2
Reduce Operator Tree:
Expand Down Expand Up @@ -3480,37 +3486,40 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: srcpart_double_hour_n0
filterExpr: ((UDFToDouble(hour) = 11.0D) and CAST( UDFToInteger((hr / 2.0D)) AS STRING) is not null) (type: boolean)
filterExpr: ((UDFToDouble(hour) = 11.0D) and hr is not null) (type: boolean)
Statistics: Num rows: 2 Data size: 188 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: ((UDFToDouble(hour) = 11.0D) and CAST( UDFToInteger((hr / 2.0D)) AS STRING) is not null) (type: boolean)
predicate: ((UDFToDouble(hour) = 11.0D) and hr is not null) (type: boolean)
Statistics: Num rows: 1 Data size: 94 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: CAST( UDFToInteger((hr / 2.0D)) AS STRING) (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
Filter Operator
predicate: _col0 is not null (type: boolean)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
keys: _col0 (type: string)
minReductionHashAggr: 0.4
mode: hash
Reduce Output Operator
key expressions: _col0 (type: string)
null sort order: z
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Dynamic Partitioning Event Operator
Target column: hr (string)
Target Input: srcpart_iceberg
Partition key expr: hr
Group By Operator
keys: _col0 (type: string)
minReductionHashAggr: 0.4
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Target Vertex: Map 1
Dynamic Partitioning Event Operator
Target column: hr (string)
Target Input: srcpart_iceberg
Partition key expr: hr
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: COMPLETE
Target Vertex: Map 1
Execution mode: vectorized
Reducer 2
Execution mode: vectorized
Expand Down
Loading
Loading