Skip to content

fix(fetch): 3-stage SSR fallback in extract_content_from_html (#3878)#4511

Open
ZhangJing-gugugaga wants to merge 1 commit into
modelcontextprotocol:mainfrom
ZhangJing-gugugaga:fix/P1-fetch-ssr-fallback
Open

fix(fetch): 3-stage SSR fallback in extract_content_from_html (#3878)#4511
ZhangJing-gugugaga wants to merge 1 commit into
modelcontextprotocol:mainfrom
ZhangJing-gugugaga:fix/P1-fetch-ssr-fallback

Conversation

@ZhangJing-gugugaga

Copy link
Copy Markdown

Problem

Pages that use progressive SSR (Next.js streaming, Remix deferred, custom
Lambda SSR) deliver content in two phases: a small visible loading shell,
then the real content in a hidden container
(visibility:hidden; position:absolute; top:-9999px) that becomes
visible only after client-side hydration.

Mozilla Readability treats those hidden elements as non-content and
strips them, so extract_content_from_html silently returns just the
loading shell (e.g. "Unified Serverless Framework …") with no indication
that content was lost.

Upstream PR #3922 proposes a fix; this PR ports the same design.

Fix

src/fetch/src/mcp_server_fetch/server.py — 3-stage fallback in
extract_content_from_html:

  • Stage 1: readabilipy + Readability (existing behaviour, unchanged
    for normal readable pages).
  • Stage 2: readabilipy without Readability when Stage 1 produced
    < 1% of the HTML size as text — keeps visibility:hidden markup.
  • Stage 3: raw markdownify of the original HTML as a last resort.

Normal readable pages always hit Stage 1 only and are byte-for-byte
unchanged.

Verification

cd src/fetch && uv sync --frozen --all-extras --dev && uv run pytest tests/test_server.py -q

23 passed in 8.20s

The existing TestExtractContentFromHtml suite (3 tests, zero-behaviour-change
contract) still passes. Added TestExtractContentFromHtmlFallback (3 tests):
empty input errors, Stage 2/3 don't crash on an empty Stage 1, long readable
Stage 1 is unaffected.

Notes

  • No new runtime dependencies.
  • Adds tests/conftest.py to inject src/ onto PYTHONPATH so pytest
    collects mcp_server_fetch. The repo's other Python servers appear to
    configure this same way via [tool.uv] or equivalent; I used a
    conftest.py to stay minimally invasive.

Fixes #3878.

…extprotocol#3878)

Port of PR modelcontextprotocol#3922 design. When Readability extracts text that is short
relative to the input HTML size (a loading-shell signature common to
progressive-SSR pages like Next.js streaming), fall back through:
  Stage 1  readabilipy + Readability (unchanged behaviour)
  Stage 2  readabilipy without Readability (keeps hidden markup)
  Stage 3  raw markdownify (last resort)

Fallback only fires on the short-output path, so normal readable pages
see zero behaviour change (probed by the existing
TestExtractContentFromHtml suite, 3 tests, all green).

Co-Authored-By: Claude Opus 4.6 <<EMAIL>>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mcp-server-fetch drops SSR content from streaming/progressive rendering sites

1 participant