Skip to content

[SPARK-58193][SQL] Classify Databricks syntax errors by message#57333

Open
alekjarmov wants to merge 4 commits into
apache:masterfrom
alekjarmov:alekjarmov/classify-databricks-syntax-errors
Open

[SPARK-58193][SQL] Classify Databricks syntax errors by message#57333
alekjarmov wants to merge 4 commits into
apache:masterfrom
alekjarmov:alekjarmov/classify-databricks-syntax-errors

Conversation

@alekjarmov

@alekjarmov alekjarmov commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Recognize remote Databricks parse errors when the JDBC driver reports the SYNTAX_ERROR condition only in its message.
Optimise isSyntaxErrorBestEffort for Databricks to catch syntax error even if SQLState is missing.

Why are the changes needed?

To improve the error classification.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added a unit-test and checked a case on a Databricks workspace

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude 4.7

Recognize remote Databricks parse errors when the JDBC driver reports the
SYNTAX_ERROR condition only in its message.
Normalize JDBC error messages before detecting the remote SYNTAX_ERROR
condition while retaining the SQLState documentation.

@cloud-fan cloud-fan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 blocking, 2 non-blocking, 0 nits.
The change is correct and consistent with the MsSqlServerDialect precedent (message-substring classification for a vendor without a reliable syntax-error SQLState); the SQLState-42 path is retained, so the check only broadens. Two non-blocking test suggestions below.

Suggestions (2)

  • JDBCTableCatalogSuite.scala:232: pure dialect unit test placed here instead of JDBCSuite where the other DatabricksDialect unit tests live — see inline
  • JDBCTableCatalogSuite.scala:233: only the returns-true path is asserted; a negative case would guard the best-effort substring match — see inline

Move dialect coverage to JDBCSuite and verify that unrelated errors are not
classified as syntax errors.
@alekjarmov
alekjarmov requested a review from cloud-fan July 17, 2026 15:05

@cloud-fan cloud-fan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 addressed, 0 remaining, 0 new. (0 newly introduced, 0 late catches.)
0 blocking, 0 non-blocking, 0 nits.

Verification

Static trace: JDBCRDD consults this hook during schema resolution and query execution, while DatabricksDialect preserves the existing SQLState path and adds only the explicit vendor marker fallback. Tests were not run as part of this review.

PR description suggestions

  • Update the user-facing-change section from No: matching this driver error now produces Spark's JDBC_EXTERNAL_ENGINE_SYNTAX_ERROR classification during schema resolution or query execution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants