Skip to content

perf(http): lazy load DomCrawler instance on demand in Response - #294

Open
alex-kassel wants to merge 6 commits into
roach-php:mainfrom
alex-kassel:feature/lazy-dom-crawler
Open

perf(http): lazy load DomCrawler instance on demand in Response#294
alex-kassel wants to merge 6 commits into
roach-php:mainfrom
alex-kassel:feature/lazy-dom-crawler

Conversation

@alex-kassel

Copy link
Copy Markdown

Summary of Optimization

This PR refactors RoachPHP\Http\Response to lazy-load the Symfony\Component\DomCrawler\Crawler instance on demand rather than eagerly instantiating it inside Response::__construct().

The Problem

Previously, Response::__construct() eagerly instantiated a new Symfony\Component\DomCrawler\Crawler for every HTTP response. When crawling APIs or JSON endpoints returning large payloads (e.g. 2–10 MB JSON arrays), DomCrawler parsed raw JSON as HTML DOM nodes, causing memory spikes of 100 MB – 150 MB+ and memory limit exhaustion.

The Solution

Switch $crawler to a nullable property initialized lazily in getCrawler():

  • 0 MB Overhead on API/JSON Responses: Spiders parsing JSON directly never trigger Crawler parsing.
  • 100% Backward Compatibility: Any call to $response->filter() or mixin methods triggers getCrawler(), initializing the Crawler dynamically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant