Keep startup polling failures isolated

[dmulcahey]

Summary

Startup polling of recently seen mains-powered devices is intended to be best effort. A failure from one device should not terminate supervision of the remaining refreshes or prevent normal polling from being enabled afterward. Startup radio traffic should also be scheduled through zigpy's request-priority mechanism rather than a second whole-device concurrency limit in ZHA.

This change:

• starts every eligible device refresh and delegates radio-request scheduling to zigpy using PacketPriority.LOW;
• passes LOW priority explicitly to startup attribute reads so packet-priority consumers such as zigpy-ziggurat receive the intended priority;
• collects each device's startup-refresh outcome while retaining ownership of every initializer until the batch completes;
• records ordinary failures and per-device cancellations separately;
• restores normal polling whenever the startup-refresh operation exits; and
• preserves cancellation propagation when the overall operation is cancelled.

Problem

During startup, ZHA temporarily disables normal polling while it refreshes the current state of recently seen mains-powered devices.

Previously, those refreshes ran through a ZHA-specific limited-concurrency gather. Its default fail-fast behavior propagated the first unexpected device exception without cancelling or awaiting the remaining initializer tasks. Sibling refreshes could therefore continue issuing Zigbee requests after the supervising startup operation had ended.

The same exception skipped the later assignment that enables normal polling. Since availability and polling-based update paths consult this shared flag, one failing device could leave unrelated devices without normal periodic refreshes for the remainder of the gateway's lifetime.

The whole-device limit also duplicated scheduling policy now owned by zigpy's priority-aware request limiter and required ZHA to inspect zigpy's private concurrency state.

Root cause

The startup call site combined three independent concerns:

1. one device's best-effort refresh could fail the entire batch;
2. restoration of gateway-wide polling state depended on successful batch completion; and
3. ZHA limited complete device initialization pipelines instead of expressing startup traffic as LOW-priority radio requests.

There was also a packet-priority propagation gap relevant to zigpy-ziggurat. An ambient request-priority context controls backends that consult zigpy's context-aware limiter, but an unspecified request priority remains None on the outgoing packet. zigpy-ziggurat interprets that missing packet value as NORMAL priority. Startup attribute reads therefore need the LOW priority passed explicitly as well as established in the ambient context.

Resolution

Startup polling now establishes PacketPriority.LOW inside the operation itself and starts all eligible device initializers with asyncio.gather(..., return_exceptions=True). Zigpy and priority-aware radio backends control actual request scheduling.

Device.async_initialize() accepts an optional keyword-only request priority and forwards it through aggregated cluster initialization to each startup attribute read. Existing callers omit the argument and retain their current behavior. Startup polling supplies LOW explicitly, ensuring packet-priority consumers such as zigpy-ziggurat receive LOW rather than treating an unspecified packet priority as NORMAL.

Because gather results preserve input order, each failure or cancellation is associated with the correct device while every sibling initializer remains supervised and awaited.

The background startup operation restores allow_polling in a finally block. Unexpected orchestration failures are reported, while asyncio.CancelledError is explicitly re-raised so reload and shutdown cancellation continue to behave normally.

Potential real-world examples

• A custom quirk or entity handler raises unexpectedly while refreshing one mains-powered device. Other healthy devices still complete their startup refreshes, and periodic polling resumes instead of remaining globally disabled.
• A device is removed, reconfigured, or otherwise cancelled while startup polling is active. That device's cancellation is distinguished from an ordinary failure without preventing unrelated devices from completing.
• One mains-powered device is slow to answer while another initializer fails quickly. The slow initializer remains owned by the startup operation instead of continuing after its supervisor has exited.
• On a large Ziggurat-backed network, startup attribute reads carry explicit LOW priority. Normal interactive traffic can therefore outrank the startup queue instead of the server receiving those reads as NORMAL-priority work.
• A failure occurs around request-priority orchestration rather than inside a specific device. Normal polling is still restored, preventing polling-based light, sensor, and availability paths from remaining inactive for the lifetime of the gateway.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.27%. Comparing base (82a14cd) to head (28b6884).

Additional details and impacted files

@@           Coverage Diff           @@
##              dev     #841   +/-   ##
=======================================
  Coverage   97.27%   97.27%           
=======================================
  Files          55       55           
  Lines       10941    10949    +8     
=======================================
+ Hits        10643    10651    +8     
  Misses        298      298           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[puddly]

An alternative approach here would be to disable concurrency limiting entirely and make sure that device state polling is done with low priority:

async with self.request_priority(t.PacketPriority.LOW):
    # code to poll a device

Maybe we can add this as a kwarg to async_initialize or something?

Zigpy has a mechanism to control concurrency if a request is properly tagged with the above context manager.

[dmulcahey]

@puddly Thanks — I updated the PR to take this approach. ZHA no longer limits complete device-initialization pipelines during startup. It starts every eligible initializer as LOW-priority work and delegates actual radio-request scheduling to zigpy and the radio backend.

The revised call site is here. In abbreviated form, it is now:

startup_polling_priority = t.PacketPriority.LOW

async with self.request_priority(startup_polling_priority):
    initialization_results = await asyncio.gather(
        *(
            device.async_initialize(
                from_cache=False,
                request_priority=startup_polling_priority,
            )
            for device in online_devices
        ),
        return_exceptions=True,
    )

Why removing the ZHA-level limiter is the better boundary

The previous implementation inspected zigpy's private _concurrent_requests_semaphore.max_concurrency, subtracted four, and used that result to limit entire Device.async_initialize() calls. That mixed two different kinds of concurrency:

• device initialization also includes local discovery, cache processing, and entity work; and
• radio request concurrency is the scarce resource that actually needs scheduling.

Zigpy now owns the latter concern. Its request-priority context supplies the ambient priority, and its global request limiter resolves that context when a backend submits a request without an explicit priority. The limiter itself is a priority-aware cascading limiter, rather than a single undifferentiated semaphore.

With zigpy's default concurrency of 8 and 25% LOW / 50% NORMAL / 25% HIGH capacity fractions, the cumulative admission thresholds are 2 for LOW, 6 for LOW plus NORMAL, and 8 when HIGH is included. These are cascading priority thresholds, not a strict total-concurrency ceiling for every possible arrival ordering, but the important distinction is that startup traffic now enters the LOW tier instead of consuming an arbitrary number of whole-device slots.

For common configured values, the structural difference looks like this:

Configured concurrency	Old ZHA initializer cap (max(1, C - 4))	Zigpy LOW request tier (25%)
4	1 complete initializer	1 radio request
8	4 complete initializers	2 radio requests
12	8 complete initializers	3 radio requests

Those columns are intentionally not performance equivalents: the old limit covered the entire initialization pipeline, while the new tier governs individual radio requests. The latter is the abstraction we actually want. The old ZHA reservation heuristic came from ZHA #510; zigpy subsequently gained its priority-aware request limiting in zigpy #1635, so the scheduling responsibility can now live at the lower layer.

Important — zigpy-ziggurat needs explicit priority propagation, not only the context manager.

There is a subtle distinction between the priority used to acquire a local limiter and the priority stored on the outgoing packet:

1. The LOW ContextVar is inherited by the tasks created inside the async with block. Zigpy defines its default request context here.
2. If a cluster call leaves priority=None, zigpy.device.Device.request() resolves the ambient value for its per-device limiter, but still forwards the original None to ControllerApplication.request().
3. ControllerApplication.request() builds the packet with that value, so ambient LOW alone can still produce ZigbeePacket.priority is None.
4. The zigpy-ziggurat 1.0.1 send path does not re-resolve zigpy's ambient context. It serializes packet.priority, and deliberately converts None to 0 (NORMAL). The relevant SendAps priority field therefore receives NORMAL unless the packet was tagged explicitly.

That would make a context-only version look correct inside ZHA while Ziggurat still received the startup reads as normal interactive traffic. The revised implementation closes that gap:

• Device.async_initialize() now has an optional keyword-only request_priority.
• initialize_cluster_configs() adds the priority to both cache-permitted and fresh attribute reads when one is supplied. This matters because a cache-permitted read can still fall through to the radio when the value is absent.
• Existing callers pass no priority, and the helper omits the keyword entirely when it is None, preserving their current behavior.
• The ambient LOW context remains in place for nested request paths and backends that resolve it themselves.

PacketPriority.LOW is -1, while NORMAL is 0. On the server side, the current Ziggurat source carries the signed wire priority through its bridge conversion, maps -1 to UserLow and 0 to UserNormal in the Zigbee stack, and schedules transmissions through its priority heap when APS work is enqueued. I am describing the current server source separately here; the pinned zigpy-ziggurat client commit above does not imply that it pins this server revision.

Backend behavior

Radio library	How startup priority is handled after this change
bellows	Its send path uses zigpy's global priority-aware limiter. Ambient LOW is available, and the revised reads also carry explicit LOW.
zigpy-znp	Same global-limiter behavior.
zigpy-deconz	Same global-limiter behavior.
zigpy-ziggurat	The explicit argument is essential: it puts -1 on the packet so the server receives UserLow, rather than converting a missing value to NORMAL.
zigpy-zigate / zigpy-xbee	These send paths do not currently use zigpy's global priority limiter. We are intentionally not retaining a duplicate ZHA whole-device limiter solely for these backends.

The explicit propagation currently covers the radio-producing work in Device.async_initialize(): the aggregated attribute reads. The current entity-level async_initialize_cluster() override is local/cache-only. If a future entity hook adds network I/O, that new path should either accept the explicit priority as well or zigpy-ziggurat should resolve the ambient priority before serializing a packet. That is a future-proofing consideration, not a reason to broaden this focused change.

Concrete behavior on a larger network

Consider 50 recently seen mains-powered devices during startup:

• ZHA now starts all 50 initialization pipelines, so local discovery and cache work are not artificially serialized behind a radio-derived number.
• On bellows, ZNP, or deCONZ, each actual startup request enters zigpy's LOW tier. Normal and high-priority work retain their priority-aware capacity instead of waiting behind an undifferentiated ZHA batch.
• On Ziggurat, the startup attribute requests carry priority=-1; the server classifies them as UserLow, so normal interactive work can outrank queued startup transmissions.
• Starting all 50 initializers therefore does not mean that a priority-aware backend transmits 50 frames concurrently. Scheduling happens where the individual requests are visible.

F4 failure and cancellation semantics remain intact

Removing the outer concurrency helper does not undo the original corrective behavior:

• asyncio.gather(..., return_exceptions=True) retains one ordered outcome per device and waits for every sibling initializer, so one quirk or device failure cannot orphan the rest of the batch.
• A child that cancels independently is returned as that device's result and logged separately from an ordinary exception.
• Cancellation of the overall startup operation still propagates through gather, cancels unfinished child tasks, and is re-raised by the wrapper.
• allow_polling is restored in finally, so an unexpected device or orchestration failure cannot leave polling globally disabled for the rest of the gateway lifetime.

So I agree with the suggested direction and have changed the implementation accordingly: ZHA expresses startup polling as LOW-priority work, zigpy/backends own radio scheduling, and Ziggurat receives the priority explicitly on the packet rather than accidentally treating startup reads as NORMAL.

[zigpy-review-bot]

Approve. Nicely reasoned and thoroughly tested — verified the load-bearing pieces locally against zigpy 2.0.1 and CI is green across 3.12/3.13/3.14.

What I checked:

• request_priority is genuinely wired through, not dead code. Traced initialize_cluster_configs → cluster.read_attributes(priority=…) → read_attributes_raw → _read_attributes/general_command(**kwargs) → Cluster.request(priority=…) → the packet. When no priority is passed the packet's priority stays None (the ambient contextvar only feeds zigpy's per-request limiter fallback — it's never written onto the packet), and on-packet-priority backends like zigpy-ziggurat treat that None as NORMAL, so the explicit propagation is what actually puts LOW on the wire. test_initialize_request_priority locks this in.
• Dropping the radio_concurrency - 4 reservation is reasonable. For backends that route through zigpy's global request limiter (bellows/ZNP/deCONZ), RequestLimiter caps the LOW tier structurally — with the default 0.25/0.50/0.25 fractions and concurrency 8 the cumulative caps are LOW=2, LOW+NORMAL=6, +HIGH=8, so startup reads occupy at most 2 slots and interactive NORMAL/HIGH work retains reserved capacity (more than the old -4 left at default concurrency). Caveat: those fractions are user-configurable, and backends that don't call zigpy's limiter (zigpy-xbee/zigate) lose ZHA-level startup limiting entirely — which the PR calls out as an intentional trade-off rather than reinstating a duplicate whole-device limiter.
• Failure isolation and cancellation semantics are correct. gather(return_exceptions=True) awaits every sibling before returning, so one device's failure can no longer orphan the rest or skip the polling re-enable (the real pre-existing bug). zip(strict=True) over order-preserving results is sound, the isinstance(CancelledError)-before-Exception ordering is right (CancelledError is BaseException), and outer cancellation still propagates through the gather and re-raises — with finally: allow_polling = True now guaranteeing polling is restored on any exit. The three new gateway tests exercise exactly these paths.
• gather_with_limited_concurrency is still used in helpers.py; only the now-unused radio_concurrency property is removed, with no remaining references in zha (or ha-core's ZHA component).
• mypy clean on the changed files.