refactor!: Add checkReadiness() and recordPacingSignal() to IRequestManager to fix leaky abstraction of ThrottlingRequestManager

[janbuchar]

• closes Capabilities of a ThrottlingRequestManager are lost when it is wrapped in another manager #3999
• closes Make the inner option of ThrottlingRequestManager optional #4027
• supersedes feat(core): make inner option of ThrottlingRequestManager optional #4037
• supersedes feat(core): unwrap capabilities of nested request managers (#3999) #4038
• contributes to A rate-limited domain is hammered when its requests come from the manager ThrottlingRequestManager wraps #4039

Worth reviewing in order — each change stands alone, with the reasoning in its own message.

1. checkReadiness() replaces isEmpty() / isFinished() on IRequestLoader and IRequestManager.
2. IRequestManager.recordPacingSignal() replaces the SupportsDomainThrottling capability probe.
3. The crawler builds the sameDomainDelaySecs throttler in its constructor, which is what puts it inside the tandem.
4. sameDomainDelaySecs goes in through recordPacingSignal() as a floor covering every domain, so a manager that already paces takes it instead of getting a second pacer wrapped around it.
5. ThrottlingRequestManager's inner is optional — omitted, it opens the default queue on first use. Same feature as feat(core): make inner option of ThrottlingRequestManager optional #4037, which is a much smaller patch on top of the lazy inner factory this branch already has.

On #3999 this takes the first option — wrappers forward — but the objection there was that it makes RequestManagerTandem know throttling exists. It doesn't: pacing signals are IRequestManager members, so the tandem forwards a PacingSignal it never looks inside, and learns nothing throttling-specific. That deletes the capability-discovery problem rather than solving it, which is why the delegation interface from #4038 isn't here.

One capability check did survive that — the guard refusing sameDomainDelaySecs over a ThrottlingRequestManager — and it missed a throttler behind a tandem (#discussion_r3851833671). The fourth change removes it rather than deepening it: the floor is a PacingSignal like any other, so whatever paces takes it wherever it sits in a composition, and a manager pacing only some of its domains throws rather than under-applying it. No instanceof left anywhere.

#4039 is the shape sameDomainDelaySecs was producing on its own: wrapping happened at first use, by which point the tandem for a requestList already existed, so the pacer went outside it and a list's requests reached the queue without passing a per-domain clock. The third commit builds the pacer in the constructor instead, so those requests are routed by domain like any other. The hand-built form in #4039's reproduction is untouched — fetchNextRequest still takes whatever inner offers without checking the clocks — so #4040 is still needed.

Migration notes in docs/upgrading/upgrading_v4.md.

[barjin]

Thanks @janbuchar !

Just one note regarding the wrapped managers, the rest is naming nits:

[barjin]

nit: Seeing the persistState method and RequestLoaderState interface next to each other, it seems that state is a rather overloaded term.

How about calling the latter ...Status or something similar?

[janbuchar]

Renamed: RequestSourceStatus / RequestLoaderStatus, and joinRequestSourceStatuses with them.

[barjin]

I'm not too keen on the readiness name. How about (get)?Status or sth more descriptive?

[janbuchar]

checkReadiness() now. Kept "readiness" over a bare getStatus(); status of what was the part that read badly IMO. What do you think?

[barjin]

What if the ThrottlingRequestManager is wrapped inside a Tandem? Then this check doesn't fire and we can still get double pacing, right?

[janbuchar]

Correct, it didn't fire — requestList.toTandem(throttler) wasn't picked up by the check and got a second pacer wrapped around it. Fixed by deleting the check:

• sameDomainDelaySecs now goes in as a pacing signal — { reason: 'minIntervalEverywhere', intervalMs, scope: 'registrableDomain' } — offered to whatever the crawler was going to read from.
• Anything that paces takes it as its minCrawlDelaySecs floor, through any number of wrappers, since they all forward. The crawler builds a ThrottlingRequestManager of its own only when nothing takes it, so a domain never ends up with two clocks.
• A manager pacing only some of its domains throws instead — same rule as a signal scoped wider than the grouping.
• Bonus: with no pacer of ours underneath a manager you own, the purgeRequestQueue ambiguity stops applying to that case too.

That leaves no instanceof anywhere, which is what actually closes #3999 — the delegation walk from #4038 would have kept BasicCrawler enumerating wrapper types instead, which sucked