Skip to content

Quote the path in DataFrameReader.load when no format is given - #626

Open
Raaif-Yousuf wants to merge 1 commit into
duckdb:mainfrom
Raaif-Yousuf:fix/spark-read-load-unquoted-path
Open

Raaif-Yousuf wants to merge 1 commit into
duckdb:mainfrom
Raaif-Yousuf:fix/spark-read-load-unquoted-path

Conversation

@Raaif-Yousuf

Copy link
Copy Markdown

DataFrameReader.load(path) fails to parse for essentially any path when no format= is given.

from duckdb.experimental.spark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
spark.read.load("data/basic.csv")
duckdb.duckdb.ParserException: Parser Error: syntax error at or near "/"
LINE 1: select * from data/basic.csv
                          ^

load() builds its query as f"select * from {path}" (duckdb/experimental/spark/sql/readwriter.py:119), splicing the raw path into the SQL text. Anything the parser treats as syntax ends the identifier, so a /, a space or a drive colon all break it, and on Windows the backslash does too. Only a bare filename in the working directory survives, because the replacement scan then picks it up. Every other reader and writer method in that file hands the path to DuckDB as an argument (read_csv(path), read_json(path), read_parquet(path)), which is why they are fine.

This quotes the path as a SQL string literal and doubles any embedded single quote.

One behaviour change worth calling out: spark.read.load("some_table") currently resolves as a table reference, and after this it is read as a path. That matches PySpark, where load() takes a path and spark.read.table() is the way to read a table, but it is a change.

Adds tests/fast/spark/test_spark_read_load.py covering a path with a space and a path with a single quote in it; nothing exercised load() without a format before. The class is skipped under USE_ACTUAL_SPARK, since real Spark defaults load() to parquet rather than reading by extension.

Verified on Windows with Python 3.12: both tests fail on main with the parser error above and pass with the change, and pytest tests/fast/spark is otherwise unchanged (289 passed, 5 skipped, 1 xfailed, plus one pre-existing test_all_types_schema failure that reproduces without this patch).

load() without a format built its query as select * from {path},
splicing the raw path straight into the SQL text, so any path with a
directory separator in it fails to parse. That is every path except a
bare filename in the working directory, and on Windows the backslash
fails as well.

Quote it as a string literal instead, the same way the rest of the
reader and writer methods hand paths to DuckDB, and escape embedded
single quotes by doubling them.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant