fix(extraction): guard field transform against None in JSON extraction#2069
Open
Otis0408 wants to merge 1 commit into
Open
fix(extraction): guard field transform against None in JSON extraction#2069Otis0408 wants to merge 1 commit into
Otis0408 wants to merge 1 commit into
Conversation
_extract_single_field applied a field's `transform` (lowercase/uppercase/ strip) unconditionally, even when the type pipeline yielded None (a regex that did not match, or a missing attribute). That called None.lower()/ .strip() and raised AttributeError. Depending on the code path this either aborted the whole extraction (the baseFields loop in extract() has no try/except) or dropped an entire list of items (one non-matching sibling made _extract_field return the default for the whole list), silently losing valid siblings. Skip the transform when the value is None so it flows to the existing `default` handling, exactly as the following return line already assumes. Adds tests covering the baseFields crash and the list-drops-valid-siblings cases for both the CSS and XPath strategies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This targets the
developbranch, per CONTRIBUTING.Who hits this
Anyone using
JsonCssExtractionStrategy/JsonXPathExtractionStrategywith a schema field that pairs a fail-capable extraction (atype: "regex"that may not match, or atype: "attribute"whose attribute may be absent) with an optionaltransform(lowercase/uppercase/strip) — all documented, sanctioned schema keys. This is a common real-world shape, e.g. alistof anchors where one<a>has nohref.Root cause
In
JsonElementExtractionStrategy._extract_single_field, thetransformblock ran unconditionally after the type pipeline. When the pipeline yieldsNone(a regex miss, orelement.get()returningNonefor a missing attribute),_apply_transformcallsNone.lower()/None.strip()/None.upper()and raisesAttributeError: 'NoneType' object has no attribute 'lower'. The very next line already treatsNoneas a legitimate outcome (return value if value is not None else field.get("default")), soNoneis expected here and the unguarded transform was an oversight.Impact depends on the path:
extract()has no try/except, so theAttributeErroraborts the entire extraction._extract_field's try/except, which returns the field default for the whole list, silently dropping valid siblings.Fix
One line — only apply the transform to a non-None value, letting
Noneflow to the existing default handling:Before / after (public API)
Tests
Adds
tests/test_json_transform_none_guard.py(5 tests, CSS + XPath) covering the baseFields crash, the list-drops-valid-siblings regression, and thattransformstill applies to genuine values. All fail on the current code and pass with the fix.