Skip to content

Fix pipe receiver type conversion load path#17849

Open
Caideyipi wants to merge 4 commits into
apache:masterfrom
Caideyipi:pipe-type-conversion-semantic-it
Open

Fix pipe receiver type conversion load path#17849
Caideyipi wants to merge 4 commits into
apache:masterfrom
Caideyipi:pipe-type-conversion-semantic-it

Conversation

@Caideyipi
Copy link
Copy Markdown
Collaborator

@Caideyipi Caideyipi commented Jun 5, 2026

Description

Fix Pipe receiver type conversion semantics for stream/default sink paths, especially table-model TsFile load.

The receiver can receive both realtime insert statements and flushed TsFiles from the default iotdb-thrift-sink. Insert statements now try the type-conversion wrapper before normal execution, so partial insert does not write nulls before conversion. Sync/async load also forces schema verification when convert-on-type-mismatch is enabled.

For table-model TsFile load, FIELD column type mismatch is now reported as LoadAnalyzeTypeMismatchException during verification. This lets LoadTsFileAnalyzer fall back to existing tablet conversion instead of native-loading a mismatched table TsFile and bypassing conversion.

The ITs cover the original semantic conversion matrix and the stream/default sink table scenario that previously reproduced null values after flush/load.

Tests

  • .\mvnw.cmd -pl iotdb-core/datanode "-Dtest=IoTDBDataNodeReceiverTest,LoadTsFileAnalyzerTest" test
  • .\mvnw.cmd -pl iotdb-core/datanode -DskipTests "-Drat.skip=true" install
  • .\mvnw.cmd -pl integration-test -P with-integration-tests -P MultiClusterIT2DualTableManualEnhanced -DskipUTs "-Dit.test=IoTDBPipeTypeConversionSemanticIT#testStreamPipeReceiverTypeConversionSemantics" "-DfailIfNoTests=false" "-Dfailsafe.failIfNoSpecifiedTests=false" "-Drat.skip=true" verify
  • .\mvnw.cmd -pl integration-test -P with-integration-tests -P MultiClusterIT2DualTableManualEnhanced -DskipUTs "-Dit.test=IoTDBPipeTypeConversionSemanticIT#testPipeReceiverTypeConversionSemantics" "-DfailIfNoTests=false" "-Dfailsafe.failIfNoSpecifiedTests=false" "-Drat.skip=true" verify
  • .\mvnw.cmd -pl integration-test -P with-integration-tests -P MultiClusterIT2DualTreeAutoEnhanced -DskipUTs "-Dit.test=IoTDBPipeTypeConversionSemanticIT#testAlignedStreamPipeReceiverTypeConversionSemantics" "-DfailIfNoTests=false" "-Dfailsafe.failIfNoSpecifiedTests=false" "-Drat.skip=true" verify

Copy link
Copy Markdown
Collaborator Author

@Caideyipi Caideyipi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this expected value is off because the source column is FLOAT, not DOUBLE. The literal 19700103.1 is stored as a 32-bit float before the receiver-side conversion, and it rounds to 19700104.0f. ValueConverter.convertFloatToDate then casts that float to int before parsing the date, so this row should convert to 1970-01-04, not 1970-01-03.

Could we update this expected value, or use a float value that is exactly representable enough for the intended 19700103 case?

@Caideyipi Caideyipi changed the title Add pipe type conversion semantic ITs Fix pipe receiver type conversion load path Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants