Fix PySparkProcessor V3 ProcessingInput construction#5759
Open
Evan-W-ang wants to merge 1 commit intoaws:masterfrom
Open
Fix PySparkProcessor V3 ProcessingInput construction#5759Evan-W-ang wants to merge 1 commit intoaws:masterfrom
Evan-W-ang wants to merge 1 commit intoaws:masterfrom
Conversation
Author
SummaryThis PR updates ProblemIn V3,
and instead expects V3 fields such as However,
This can cause validation failures during pipeline definition / upsert. FixThis change:
TestsAdded regression tests covering:
Example failure before this change |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use V3-compatible ProcessingInput construction in PySparkProcessor.
PySparkProcessor still built internal ProcessingInput objects with the
legacy source/destination fields in _stage_configuration() and
_stage_submit_deps(). In V3, ProcessingInput now expects s3_input, so
those internal code paths can fail during pipeline definition or upsert
with validation errors.
This change updates both code paths to build ProcessingInput with
ProcessingS3Input while preserving the same staged S3 URIs and local
mount paths. It also adds regression tests covering configuration
staging and local dependency staging