[pull] master from apify:master - #258
Merged
Merged
Conversation
…3938) Fixes the cookie handling on redirects when using `ImpitHttpClient.stream()`. --------- Co-authored-by: Jindřich Bär <jindrichbar@gmail.com>
This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [lerna](https://lerna.js.org) ([source](https://redirect.github.com/lerna/lerna/tree/HEAD/packages/lerna)) | [`^9.0.7` → `^10.0.0`](https://renovatebot.com/diffs/npm/lerna/9.0.7/10.0.0) |  |  | --- ### Release Notes <details> <summary>lerna/lerna (lerna)</summary> ### [`v10.0.0`](https://redirect.github.com/lerna/lerna/blob/HEAD/packages/lerna/CHANGELOG.md#1000-2026-07-29) [Compare Source](https://redirect.github.com/lerna/lerna/compare/v9.0.7...v10.0.0) - feat!: throw error in case of stale remote for CI mode ([#​4369](https://redirect.github.com/lerna/lerna/issues/4369)) ([3e81682](https://redirect.github.com/lerna/lerna/commit/3e816822837edb0a1f70cd52c90207f520461907)), closes [#​4369](https://redirect.github.com/lerna/lerna/issues/4369) - feat!: support node ^22.13.0 || ^24.0.0 || ^26.0.0, ship lerna as ESM-only ([#​4390](https://redirect.github.com/lerna/lerna/issues/4390)) ([a148ba2](https://redirect.github.com/lerna/lerna/commit/a148ba23882e4e3d201b0198eefc37d6f18196d9)), closes [#​4390](https://redirect.github.com/lerna/lerna/issues/4390) - fix(core)!: replace deprecated conventional-changelog dependencies ([#​4332](https://redirect.github.com/lerna/lerna/issues/4332)) ([b1ff72f](https://redirect.github.com/lerna/lerna/commit/b1ff72f7053870eaf661b06af9759c9fe6c0c5d6)), closes [#​4332](https://redirect.github.com/lerna/lerna/issues/4332) ##### Bug Fixes - **core:** remove p-map-series, p-pipe, p-reduce, and p-waterfall ([#​4321](https://redirect.github.com/lerna/lerna/issues/4321)) ([fe066cb](https://redirect.github.com/lerna/lerna/commit/fe066cb52fa47a590022108a098a9ba5fb108af2)) - **core:** remove upath dependency ([#​4317](https://redirect.github.com/lerna/lerna/issues/4317)) ([aa65470](https://redirect.github.com/lerna/lerna/commit/aa654700fdfb3bfe79e79a38cabbe3ff078a5f5b)) ##### Features - **core:** add bun as supported package manager ([#​4264](https://redirect.github.com/lerna/lerna/issues/4264)) ([4ca7d2c](https://redirect.github.com/lerna/lerna/commit/4ca7d2cc46feef31641d6877cf32f420d2ebc897)) ##### BREAKING CHANGES - In CI, `EBEHIND` will now be thrown during versioning and publishing if the checkout is behind the latest on the remote. This previously only occurred outside of CI environments. If you wish to opt into the old behavior, you can do so by setting `--ci-behind-behavior` (error | skip, default error) or `command.version.ciBehindBehavior` in `lerna.json`. - Lerna is now shipped as ESM-only and the lowest supported node version has changed to 22.13.0, because on this version CommonJS consumers can still require in its entry points without any additional flags or warnings. - Lerna now uses the current conventional-changelog APIs instead of the deprecated conventional-changelog-core stack. Generated CHANGELOG.md output may differ, including normalized whitespace and URL-encoded tag names. Projects using custom changelog presets should verify their output; Lerna retains compatibility for legacy parser/writer option names and Handlebars string templates. CLI options and version-bump behavior remain unchanged. #### [9.0.7](https://redirect.github.com/lerna/lerna/compare/v9.0.6...v9.0.7) (2026-03-13) ##### Bug Fixes - **core:** remove multimatch dependency and legacy-core internals ([#​4314](https://redirect.github.com/lerna/lerna/issues/4314)) ([ec01462](https://redirect.github.com/lerna/lerna/commit/ec01462a9c6b6911f47d328e6662aa7afd7feea5)) #### [9.0.6](https://redirect.github.com/lerna/lerna/compare/v9.0.5...v9.0.6) (2026-03-11) ##### Bug Fixes - **deps:** add missing ci-info dependency ([#​4263](https://redirect.github.com/lerna/lerna/issues/4263)) ([b768187](https://redirect.github.com/lerna/lerna/commit/b76818783c92c582b11555c56c222947ecbd4791)) - **deps:** bump tar from 7.5.8 to 7.5.11 ([#​4296](https://redirect.github.com/lerna/lerna/issues/4296)) ([7a69a57](https://redirect.github.com/lerna/lerna/commit/7a69a5753324c70086c9ed760c61dd22d5c89b42)) #### [9.0.5](https://redirect.github.com/lerna/lerna/compare/v9.0.4...v9.0.5) (2026-02-28) ##### Bug Fixes - bump minimatch from 3.0.5 to 3.1.4 ([#​4285](https://redirect.github.com/lerna/lerna/issues/4285)) ([2e3f99e](https://redirect.github.com/lerna/lerna/commit/2e3f99ed2dfa5be614ba3f740ab0579bbd4c51c9)) - bump tar from 7.5.7 to 7.5.8 ([#​4273](https://redirect.github.com/lerna/lerna/issues/4273)) ([bdffd1d](https://redirect.github.com/lerna/lerna/commit/bdffd1d97c27cd2a43e853d73296936fbc0e4f27)) #### [9.0.4](https://redirect.github.com/lerna/lerna/compare/v9.0.3...v9.0.4) (2026-02-10) ##### Bug Fixes - bump tar to 7.5.7, rimraf to 6.1.2, [@​npmcli/run-script](https://redirect.github.com/npmcli/run-script) to 10.0.3 ([#​4267](https://redirect.github.com/lerna/lerna/issues/4267)) ([43e3d46](https://redirect.github.com/lerna/lerna/commit/43e3d46fd6d76f05e130e8f8f5a6299f0e93f2ae)) #### [9.0.3](https://redirect.github.com/lerna/lerna/compare/v9.0.2...v9.0.3) (2025-11-27) **Note:** Version bump only for package lerna #### [9.0.2](https://redirect.github.com/lerna/lerna/compare/v9.0.1...v9.0.2) (2025-11-27) **Note:** Version bump only for package lerna #### [9.0.1](https://redirect.github.com/lerna/lerna/compare/v9.0.0...v9.0.1) (2025-11-14) ##### Bug Fixes - expand version range to include nx v22.x ([#​4242](https://redirect.github.com/lerna/lerna/issues/4242)) ([0cca286](https://redirect.github.com/lerna/lerna/commit/0cca28612be720d39f55cc278a9a4a93e112d1e1)) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - "every weekday" - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/apify/crawlee). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4zLjIiLCJ1cGRhdGVkSW5WZXIiOiI0NC4zLjIiLCJ0YXJnZXRCcmFuY2giOiJtYXN0ZXIiLCJsYWJlbHMiOltdfQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
The scheduled E2E runs have been red since 21 July. Every failure is a `429` from `warehouse-theme-metal.myshopify.com` — `requestsFinished` was 0, so the `All requests finished` precondition failed before the behaviour under test ran. ## Why the proxy stopped helping Session rotation works correctly; the datacenter pool is the problem. Measured through `Actor.createProxyConfiguration()`: | | result | |---|---| | 8 Crawlee sessions → egress IPs | 8 distinct IPs, **7 got 429** | | datacenter (`auto`) vs `RESIDENTIAL`, 6 requests each | **0/6** vs **6/6** | | direct, no proxy | 200 every time | So routing the six store tests through `Actor.createProxyConfiguration()` in #3855 made them strictly worse once the pool's reputation degraded. ## What this does instead None of these tests are about scraping protection — `camoufox-cloudflare` covers that. Five of them use the store only as convenient HTML, so they now serve their own fixtures. `playwright-introduction-guide` keeps crawling the real store, since mirroring the published guide is the point of it. The store rejects the platform's egress IP whether the crawl goes direct or through the datacenter pool, so on the platform it goes through residential. That is gated on `Actor.isAtHome()`: GitHub runners reach the store directly and stay off the proxy. ## Three latent bugs this surfaced **The robots tests had stopped testing robots.txt.** The store's robots.txt no longer has a `Disallow: /cart` rule — only `/cart/` and `/cart.js` — so `/cart` is allowed, while all three tests assert `'/cart URL is not processed'` under a comment claiming it is disallowed. The fixture disallows it explicitly and links to it from every page, so the assertion means what it says again. **`adaptive-playwright-robots-file` was passing while crawling nothing.** It had no `requestsFinished` precondition, so its two "URL is not processed" assertions held vacuously — its Apify runs read `Total 1 requests: 0 succeeded, 1 failed`. It couldn't have had one: `getStats()` and the platform stats lookup hardcoded `SDK_CRAWLER_STATISTICS_0`, but `AdaptivePlaywrightCrawler` discards the `Statistics` its base constructor created and installs its own, which lands on the next id. Both lookups now match by prefix, and the test has the assertion. Worth deciding separately whether the library should change here — every other crawler persists to `_0`, so anything reading that key gets nothing from an adaptive crawler. **The localhost-fixture skip in #3674 rested on a false premise.** The comment on `cheerio-enqueue-links-base` and `playwright-enqueue-links-base` claimed a `127.0.0.1` fixture is unreachable from the Apify platform container. It isn't: the server is started inside `actor/main.js`, so it is plain loopback within the same container, and for the browser tests the browser is in that container too. Verified by running every fixture-based test against the platform with the skip removed — all seven pass, and the run logs show the crawls hitting the `127.0.0.1` URLs. Those two skips are removed here as well, so nothing loses PLATFORM coverage. Also: `validateDataset`'s URL regex rejects the fixture's `127.0.0.1` origin, so the two pagination tests check the `url` shape separately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )