Skip to content

fix(extraction): guard field transform against None in JSON extraction#2069

Open
Otis0408 wants to merge 1 commit into
unclecode:developfrom
Otis0408:fix/json-extraction-transform-none-guard
Open

fix(extraction): guard field transform against None in JSON extraction#2069
Otis0408 wants to merge 1 commit into
unclecode:developfrom
Otis0408:fix/json-extraction-transform-none-guard

Conversation

@Otis0408

Copy link
Copy Markdown

This targets the develop branch, per CONTRIBUTING.

Who hits this

Anyone using JsonCssExtractionStrategy / JsonXPathExtractionStrategy with a schema field that pairs a fail-capable extraction (a type: "regex" that may not match, or a type: "attribute" whose attribute may be absent) with an optional transform (lowercase / uppercase / strip) — all documented, sanctioned schema keys. This is a common real-world shape, e.g. a list of anchors where one <a> has no href.

Root cause

In JsonElementExtractionStrategy._extract_single_field, the transform block ran unconditionally after the type pipeline. When the pipeline yields None (a regex miss, or element.get() returning None for a missing attribute), _apply_transform calls None.lower() / None.strip() / None.upper() and raises AttributeError: 'NoneType' object has no attribute 'lower'. The very next line already treats None as a legitimate outcome (return value if value is not None else field.get("default")), so None is expected here and the unguarded transform was an oversight.

Impact depends on the path:

  • baseFields — the loop in extract() has no try/except, so the AttributeError aborts the entire extraction.
  • list of sub-fields — one non-matching sibling throws into _extract_field's try/except, which returns the field default for the whole list, silently dropping valid siblings.

Fix

One line — only apply the transform to a non-None value, letting None flow to the existing default handling:

-        if "transform" in field:
+        if "transform" in field and value is not None:
             value = self._apply_transform(value, field["transform"])
         return value if value is not None else field.get("default")

Before / after (public API)

from crawl4ai.extraction_strategy import JsonCssExtractionStrategy

html = """<div class='wrap'><ul>
  <li><a class='lnk' href='http://x'>a</a></li>
  <li><a class='lnk'>b</a></li>
  <li><a class='lnk' href='http://z'>c</a></li>
</ul></div>"""
schema = {
    "name": "links", "baseSelector": "div.wrap",
    "fields": [{
        "name": "links", "selector": "a.lnk", "type": "list",
        "fields": [{"name": "href", "type": "attribute",
                    "attribute": "href", "transform": "lowercase"}],
    }],
}
JsonCssExtractionStrategy(schema).extract(None, html)
# before: []  -> http://x and http://z silently lost
# after:  [{'links': [{'href': 'http://x'}, {}, {'href': 'http://z'}]}]

Tests

Adds tests/test_json_transform_none_guard.py (5 tests, CSS + XPath) covering the baseFields crash, the list-drops-valid-siblings regression, and that transform still applies to genuine values. All fail on the current code and pass with the fix.

_extract_single_field applied a field's `transform` (lowercase/uppercase/
strip) unconditionally, even when the type pipeline yielded None (a regex
that did not match, or a missing attribute). That called None.lower()/
.strip() and raised AttributeError.

Depending on the code path this either aborted the whole extraction (the
baseFields loop in extract() has no try/except) or dropped an entire list
of items (one non-matching sibling made _extract_field return the default
for the whole list), silently losing valid siblings.

Skip the transform when the value is None so it flows to the existing
`default` handling, exactly as the following return line already assumes.

Adds tests covering the baseFields crash and the list-drops-valid-siblings
cases for both the CSS and XPath strategies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant