fix(video): fall back to SDR instead of failing session when HDR isn't supported

[matix753]

Problem

When a client requests HDR (dynamicRange) for HEVC or AV1 but the active capture/encode
device combination can't actually deliver it, make_avcodec_encode_session() fails the
session outright (return nullptr) instead of falling back to SDR the way it already does
for H.264 (encoder.h264[DYNAMIC_RANGE] is hardcoded false, so H.264 never even attempts
HDR and always negotiates cleanly).

This isn't just a hardware-capability edge case. On Linux, NvFBC capture always uses the
cuda::cuda_t encode-device class, which only ever produces AV_PIX_FMT_NV12 /
AV_PIX_FMT_YUV444P frames (see src/platform/linux/cuda.cpp, cuda_t::set_frame()) — it
has no P010/10-bit path at all, regardless of what the encoder itself supports. The sibling
class gl_cuda_vram_t (used by the GL/dmabuf capture path) does support AV_PIX_FMT_P010LE,
so this is a structural gap specific to the NvFBC+CUDA path, not a hardware limitation.

Confirmed on an NVIDIA Quadro T1000 (Turing): ffmpeg -c:v hevc_nvenc -profile:v main10
encodes 10-bit HEVC perfectly outside of Sunshine. Inside Sunshine, with capture = nvfbc,
any client HDR-HEVC request fails with:

Error: cuda::cuda_t doesn't support any format other than AV_PIX_FMT_NV12 and AV_PIX_FMT_YUV444P
Error: Could not open codec [hevc_nvenc]: Function not implemented

Observed symptom on a real client (LG webOS Moonlight app, mariotaku/moonlight-tv):
the client doesn't give up after the failed session — it silently reconnects and requests
the exact same HDR-HEVC config again, forever. Sunshine correctly fails every single attempt
(no infinite loop inside Sunshine), but from the outside it looks like a permanent hang:
Creating encoder [hevc_nvenc] → fail → repeat, indefinitely, once per reconnect.

Fix

make_avcodec_encode_session() now downgrades to SDR in place instead of failing the
session when the requested dynamic range isn't supported for the current chroma sampling
mode — mirroring what H.264 already does by design:

• Introduces a local, possibly-downgraded dynamicRange bool (starts as config.dynamicRange).
• On the YUV444 and 4:2:0 unsupported-HDR checks, instead of return nullptr, logs a
  warning and sets dynamicRange = false, also correcting encode_device->colorspace
  (colorspace_e::rec709, bit_depth = 8) so the rest of the function (sw_fmt selection,
  ctx->profile, HDR option sets, mastering-display metadata gating via
  colorspace_is_hdr()) all agree on SDR for this session.
• All other uses of config.dynamicRange in the function now read the local dynamicRange
  instead, so the downgrade is consistent for profile selection and option handling.

This does not change behavior for the encoder capability probe at startup
(probe_encoders() / test_yuv420_hdr()), since that code path calls this same function
before encoder.hevc[DYNAMIC_RANGE] has been determined — the probe still runs exactly as
before and correctly records the capability. The fallback only engages for real sessions
after the probe has already determined dynamic range isn't supported, which is exactly the
case that previously looped forever.

Testing

Environment: server = Debian GNU/Linux 13 (trixie), NVIDIA Quadro T1000 (Turing), driver
580.173.02, capture = nvfbc, encoder = nvenc. Client = LG 75QNED84A6C TV, webOS TV 10.3.1
(build 3001, "ponytail-papikonda", kernel 5.4.268), streaming app com.limelight.webos
(community mariotaku/moonlight-tv client, not the official LG Moonlight app).

• Reproduced on a real Sunshine build (this same source, unpatched) with hevc_mode forced
  to request Main10, confirming the exact log signature above.
• Applied the fix, rebuilt, redeployed to the same host (NvFBC capture, Turing GPU).
• Verified against a real HDR-enabled LG webOS Moonlight client: session now negotiates
  HEVC, logs hevc_nvenc: dynamic range not supported, falling back to SDR, and streams
  successfully — no reconnect loop, stable 6-camera grid stream, zero perceived delay.
• As a side effect, this also fixes a real memory leak in the moonlight-tv webOS client
  (confirmed via /proc/<pid>/status, ~1MB/s VmRSS growth) that only manifests when the
  server keeps failing/re-negotiating HEVC — see
  mariotaku/moonlight-tv#479. With
  this fix, HDR can stay enabled globally on the client (e.g. for other, HDR-capable hosts)
  without breaking hosts that can't deliver HDR through NvFBC.

Scope

This fixes the hang/loop for any hardware+capture combination where dynamic range isn't
supported, not just NvFBC-on-Turing. The gl_cuda_vram_t P010 gap on the NvFBC path could
also be closed directly (giving true 10-bit HEVC on Turing+NvFBC), but that's a separate,
larger change to the CUDA conversion path — this PR only fixes the missing graceful fallback,
matching the existing H.264 behavior.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud