Quote the path in DataFrameReader.load when no format is given - #626
Open
Raaif-Yousuf wants to merge 1 commit into
Open
Raaif-Yousuf wants to merge 1 commit into
Raaif-Yousuf wants to merge 1 commit into
Conversation
load() without a format built its query as select * from {path},
splicing the raw path straight into the SQL text, so any path with a
directory separator in it fails to parse. That is every path except a
bare filename in the working directory, and on Windows the backslash
fails as well.
Quote it as a string literal instead, the same way the rest of the
reader and writer methods hand paths to DuckDB, and escape embedded
single quotes by doubling them.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DataFrameReader.load(path)fails to parse for essentially any path when noformat=is given.load()builds its query asf"select * from {path}"(duckdb/experimental/spark/sql/readwriter.py:119), splicing the raw path into the SQL text. Anything the parser treats as syntax ends the identifier, so a/, a space or a drive colon all break it, and on Windows the backslash does too. Only a bare filename in the working directory survives, because the replacement scan then picks it up. Every other reader and writer method in that file hands the path to DuckDB as an argument (read_csv(path),read_json(path),read_parquet(path)), which is why they are fine.This quotes the path as a SQL string literal and doubles any embedded single quote.
One behaviour change worth calling out:
spark.read.load("some_table")currently resolves as a table reference, and after this it is read as a path. That matches PySpark, whereload()takes a path andspark.read.table()is the way to read a table, but it is a change.Adds
tests/fast/spark/test_spark_read_load.pycovering a path with a space and a path with a single quote in it; nothing exercisedload()without a format before. The class is skipped underUSE_ACTUAL_SPARK, since real Spark defaultsload()to parquet rather than reading by extension.Verified on Windows with Python 3.12: both tests fail on
mainwith the parser error above and pass with the change, andpytest tests/fast/sparkis otherwise unchanged (289 passed, 5 skipped, 1 xfailed, plus one pre-existingtest_all_types_schemafailure that reproduces without this patch).