docs: Rewrite operators page to show complete overview of what is and is not supported by Comet#4563
Draft
andygrove wants to merge 3 commits into
Draft
docs: Rewrite operators page to show complete overview of what is and is not supported by Comet#4563andygrove wants to merge 3 commits into
andygrove wants to merge 3 commits into
Conversation
…ference [skip ci]
- Rewrite datatypes.md to match the status-aware reference style introduced for operators and expressions, listing every Spark data type by status (✅ /⚠️ / 🔜 / 💤) with caveats and tracking issues. - In operators.md, link each scan row to the relevant compatibility doc (Parquet Scan Compatibility, Iceberg Guide). - Note that LocalTableScanExec is disabled by default because there is no acceleration advantage and it is typically only used in test code. [skip ci]
coderfender
reviewed
Jun 2, 2026
| | ✅ Supported | Native support; enabled by default. | | ||
| | ⚠️ Supported (caveats) | Works, but with limits: certain values, contexts, or configurations fall back to Spark. | | ||
| | 🔜 Planned | Intended; tracked by an open issue or pull request. | | ||
| | 💤 Not currently planned | Not on the current roadmap; queries referencing this type fall back to Spark and may be reconsidered. | |
Contributor
There was a problem hiding this comment.
Perhaps TBD instead of zzz ? Also, planned should probably be a different icon ? It is not very clear in dark mode
…-scan caveats Parquet-scan-specific caveats (ShortType INT16/UINT_8 disambiguation, Decimal binary encoding, datetime rebasing, TimestampNTZ read conversion) are documented in the Parquet scan compatibility guide, so they no longer downgrade the type status here. Supported types now show a green checkmark. [skip ci]
andygrove
force-pushed
the
docs-operator-support-reference
branch
from
a4b3024 to
024b4de
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
See rendered versions:
Rationale for this change
The user guide has a
Supported Spark Operatorspage, but it only lists the operators Comet replaces and does not tell users what is not supported or what is planned. This mirrors what #4550 did for expressions: turn the operators page into a complete, status-aware reference so users can see at a glance whether a given Spark physical operator is supported, supported with caveats, planned, or not currently planned.What changes are included in this PR?
Rewrites
docs/source/user-guide/latest/operators.mdinto a complete reference:Not currently plannedsection for operator families that fall back by design (Structured Streaming operators, Cartesian / cross joins, sampling and range generation).Support status is derived from the
createExecEnabledConfigdefaults inCometConf(for examplewindowandtakeOrderedAndProjectare enabled by default,localTableScanis disabled by default) plus the operator handling inCometExecRule. Notably this corrects a stale note:WindowExecis enabled by default now, not disabled.This is a draft because the status of several not-yet-supported operators is an initial proposal and would benefit from maintainer confirmation, in particular:
InMemoryTableScanExec(marked Planned, could be Not currently planned)SortAggregateExec(Planned vs Not currently planned)WindowGroupLimitExecand the Python / UDF operators (Planned references)Open issues / PRs referenced for Planned items: #4429 (nested loop join), #4234 (PyArrow UDFs), #4393 (LocalTableScan default), #2721 (window functions).
How are these changes tested?
Documentation only. Rendered locally and checked with
prettier.