e2e nightly failing: grpcio 1.82.0 breaks flower-superlink startup

[faizollah]

Symptom

Since 2026-07-06, the nightly real-app e2e workflow fails. flwr run never produces a crate, and validate_crate.py reports:

FAIL: crate not found at /tmp/flcrate_out/ro-crate/ro-crate-metadata.json (did the run produce one?)

The actual failure is earlier in the run — Flower's server can't start:

Starting local SuperLink on 127.0.0.1:39093...
╭─ Error ─────────────────────────────────────────────╮
│ Failed to start local SuperLink: `flower-superlink`  │
│ exited with code 1. See logs at .../superlink.log.   │
╰──────────────────────────────────────────────────────╯

(The job then hangs ~7.5 min on the crate-poll loop before failing, because flwr run ... || true swallows the SuperLink error.)

Root cause: grpcio 1.82.0

Diffing installed packages between the last green run (07-05) and a red run (07-07), the only change is grpcio:

Package	Green 07-05	Red 07-07
grpcio	1.81.1	1.82.0
grpcio-health-checking	1.81.1	1.82.0
flwr	1.32.1	1.32.1
protobuf / ray / numpy / scikit-learn / …	identical	identical

grpcio 1.82.0 was published 2026-07-06 04:20 UTC — every e2e run after that timestamp fails, every run before it passed. Flower's SuperLink is a gRPC server, so a grpcio regression taking it down is consistent. The e2e installs grpcio unpinned (via flwr), so it upgraded automatically.

This is not a flwrCrate bug — it's an upstream Flower/grpcio incompatibility. The per-push tests workflow (recorded fixtures, no live Flower) is unaffected and green.

Workaround (this issue)

Pin grpcio<1.82 in the e2e workflow so it keeps validating flwrCrate against a working stack.

Follow-ups

• Remove the grpcio pin once Flower caps grpcio or grpcio ships a fix.
• Consider reporting upstream to Flower (flwr / flower-superlink fails to start under grpcio 1.82.0).
• Optionally make the e2e fail fast (surface the SuperLink error instead of hanging 7.5 min behind || true).

[faizollah]

Precise root cause (captured via direct flower-superlink invocation in CI)

flwr run only surfaces "flower-superlink exited with code 1" and swallows the traceback (the referenced superlink.log is never written — the process dies at import in ~0.4s). Running flower-superlink directly on a Linux runner with grpcio 1.82.0 gives the real error:

Traceback (most recent call last):
  File ".../bin/flower-superlink", line 3, in <module>
    from flwr.server.app import run_superlink
  ...
  File ".../flwr/supercore/grpc_health/health_server.py", line 22, in <module>
    from grpc_health.v1.health_pb2_grpc import add_HealthServicer_to_server
  File ".../grpc_health/v1/health_pb2.py", line 12, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  ...
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf
Gencode/Runtime versions when loading grpc_health/v1/health.proto:
gencode 7.35.0 runtime 6.33.6. Runtime version cannot be older than the
linked gencode version.

Why

Package	Declares	Effect
flwr 1.32.1	protobuf>=5.28,<7.0	pins runtime to protobuf 6.33.6
grpcio-health-checking 1.82.0	protobuf>=6.33.5,<8.0	resolves fine with 6.33.6, but its bundled health_pb2.py gencode requires protobuf ≥ 7.35.0
protobuf	latest 7.35.1	not installed — capped out by flwr

So grpcio-health-checking 1.82.0 ships protobuf 7.x gencode while under-declaring its floor as >=6.33.5. pip pairs it with protobuf 6.33.6 (allowed by flwr's <7.0), and it then raises VersionError at import because gencode 7.35.0 can't run on protobuf 6.33.6.

Root defect is in grpcio-health-checking 1.82.0 (wrong protobuf lower bound). Practical fix for us is upstream in Flower: cap grpcio-health-checking<1.82 (and grpcio<1.82) until Flower moves to the protobuf 7.x runtime.

Workaround status

The grpcio<1.82 + grpcio-health-checking<1.82 pin in realapps.yml (on the poetry-release branch) restores the e2e to green — confirmed. Remove it once Flower ships a fix.

[faizollah]

Already fixed upstream — Flower PR flwrlabs/flower#7541 (merged 2026-07-06)

Flower recognized this the same day and merged flwrlabs/flower#7541, which caps the dependency in framework/pyproject.toml:

-    "grpcio-health-checking>=1.70.0,<2.0.0",
+    "grpcio-health-checking>=1.70.0,<1.82.0",

That is the exact same cap as our workaround pin. Note it caps grpcio-health-checking only (not grpcio).

Status: merged but not yet in a PyPI release — flwr 1.32.1 (2026-07-01) predates it. Once flwr ships a version > 1.32.1 with this cap, the unpinned e2e on main will go green on its own, and the temporary pin in realapps.yml can be removed (and this issue closed). No upstream bug report needed — it's already handled.