Intel QSV: 10-bit HEVC source → H264 output fails with "Current pixel format is unsupported"

[zarttak]

Intel QSV: 10-bit HEVC source → H264 output fails with "Current pixel format is unsupported"

Hardware: Intel Arc A770 (DG2), Unraid, jellyfin-ffmpeg 7.1.4
Profile: Default streaming test, "4K HEVC -> 1080p H264"

Summary

Every GPU run using the default "4K HEVC -> 1080p H264" streaming profile fails immediately with:

DG2 [Arc A770] pipeline did not start. [out#0/null @ ...]
Nothing was written into output file, because at least one of its streams received no packets.

Root cause confirmed via manual ffmpeg reproduction: the source clip (source_4k_hevc.mkv) is HEVC Main10 (yuv420p10le, 10-bit). The scale_qsv filter preserves the 10-bit format through the downscale to 1080p, but h264_qsv only accepts 8-bit NV12 input. The encoder then fails to open with:

[h264_qsv] Current pixel format is unsupported
[h264_qsv] some encoding parameters are not supported by the QSV runtime.
[vost#0:0/h264_qsv] Error while opening encoder - maybe incorrect parameters such as bit_rate, rate, width or height.

Reproduction (fails)

ffmpeg -v verbose -hwaccel qsv -hwaccel_output_format qsv \
  -i source_4k_hevc.mkv \
  -vf 'scale_qsv=w=1920:h=1080' \
  -c:v h264_qsv -f null -

Fix (works)

Adding an explicit format conversion to 8-bit NV12 in the same scale_qsv filter resolves it:

ffmpeg -v verbose -hwaccel qsv -hwaccel_output_format qsv \
  -i source_4k_hevc.mkv \
  -vf 'scale_qsv=w=1920:h=1080:format=nv12' \
  -c:v h264_qsv -f null -

This completes successfully — 1440/1440 frames encoded at ~11x realtime speed on the A770.

Suggested fix

When building the QSV filter chain for any HEVC 10-bit → H264 (8-bit only) conversion, append :format=nv12 (or equivalent 8-bit pixel format) to the scale_qsv filter. This likely affects any 10-bit source (HEVC Main10, HDR profiles) being converted to an 8-bit output codec on Intel QSV, not just the default streaming profile.

Environment

• GPU: Intel Arc A770 (DG2), detected correctly by the benchmark
• ffmpeg: 7.1.4-Jellyfin, iHD driver 25.4.6
• Confirmed device passthrough (/dev/dri), LIBVA_DRIVERS_PATH/LIBVA_DRIVER_NAME env vars, and file permissions are all correctly configured — this is purely a filter-chain issue in the pipeline the app generates for 10-bit sources.