RTX 5080 - Atomic commit modifying DRM Colorspace property always return EINVAL - Can't activate HDR

[clinche]

NVIDIA Open GPU Kernel Modules Version

610.57.04

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Arch Linux - CachyOS

Kernel Release

7.1.6-1-cachyos

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

RTX 5080

Describe the bug

The driver rejects any atomic commit that sets the DRM Colorspace connector property to BT2020_RGB (9) or BT2020_YCC (10), returning EINVAL. This makes HDR and wide colour gamut impossible to enable on this GPU.

Monitor: Acer XV322QK KV, 3840x2160@143.86 over HDMI, single output, nothing else connected to the card. The EDID advertises HDR10 (HDR Static Metadata Data Block with SMPTE ST2084) and BT2020RGB/BT2020YCC in the Colorimetry Data Block.

The connector exposes both properties normally:

    Colorspace: enum {Default=0, BT2020_RGB=9, BT2020_YCC=10} = 0
    HDR_OUTPUT_METADATA: blob

but any commit setting Colorspace to a BT2020 value fails. With drm.debug=0x116:

    drm_atomic_check_only.cold] atomic driver check for 00000000df4ddfa8 failed: -22

There is no nvidia or nvidia-modeset message at any log level during the failed commit the rejection is silent on the kernel side.

What I have ruled out

• Compositor. Reproduces with the session stopped and KWin entirely out of the picture, using plain modetest:

      sudo systemctl stop sddm
      sudo modetest -M nvidia-drm -s 131@65:3840x2160-60 -w 131:Colorspace:9

The modeset itself succeeds and the test pattern displays correctly at 60 Hz; only the property set fails, with "Invalid argument".

• Link bandwidth / bit depth. Identical failure at 4K60, 1440p60, 1080p60, all at 8 bpc (KWIN_DRM_PREFER_COLOR_DEPTH=24). Even at 4K60 8 bpc there should be a very large margin on the link, and the modeset succeeds anyway only the property is refused.

• HDR metadata. Enabling wide colour gamut alone, i.e. Colorspace=BT2020_RGB with no HDR_OUTPUT_METADATA, is rejected identically. The Colorspace property alone is the trigger.

• EDID. This panel's Colorimetry Data Block had non-zero reserved bits MD0-MD3 (flagged by di-edid-decode). I zeroed them, recomputed the CTA extension checksum and loaded the corrected EDID via drm.edid_firmware. Verified in place after reboot, di-edid-decode reports the CDB as conforming. Behaviour is unchanged: Colorspace=9 still returns EINVAL.

• CRTC assignment. KWin tries all four CRTCs in turn; all four are rejected with -22.

Prior working configuration

The same monitor over the same HDMI cable does HDR correctly on an RTX 3060 laptop, on the same distribution, same resolution and with the same driver version. It may be worth noting however the laptop can only manage 4k60Hz as it does not have HDMI 2.1.

To Reproduce

1. RTX 5080 GPU, HDR-capable display, driver 610.57.04
2. From a TTY with the display manager stopped: sudo modetest -M nvidia-drm -s <connector>@<crtc>:3840x2160-60 -w <connector>:Colorspace:9
3. The mode is set correctly, but the property set fails with EINVAL
4. Confirm the same display, driver version, and cable successfully sets the Colorspace with a laptop equipped with an RTX3060 mobile

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

Happy to provide modetest -c output for the connector, the full EDID, the patched EDID, and drm.debug traces. I can also provide the same info plus the nvidia-bug-report for the functioning RTX3060 laptop.

[clinche]

Root cause found, and it is a parser bug rather than anything to do with bandwidth, the compositor, or the monitor's capabilities. I have opened #1308 with a one-line fix.

The final loop in parseEdidHDMILLCTiming() (src/common/modeset/timing/nvt_edidext_861.c) consumes whatever bytes remain in the HDMI VSDB and interprets each one as a per-VIC 3D entry. Unlike the two 3D parsing blocks above it, it is not guarded by pHDMIVideo->ThreeD_Present. Since NVT_HDMI_3D_SUPPORTED_STRUCT_MASK(x) is 1 << x, any byte value produces a non-zero stereo mask, including 0x00 — so a single trailing padding byte leaves HDMI3DSupported set on a display whose VSDB advertises 3D_present = 0.

That then trips two gates: nvDpyIsHDRCapable() in nvkms-dpy.c, which makes any atomic commit setting Colorspace to a BT2020 value fail with EINVAL and no kernel log at any debug level, and SendHDRInfoFrame() in nvkms-hdmi.c, which returns early so the Dynamic Range and Mastering InfoFrame is never transmitted.

This accounts for both failure modes people report. If HDR cannot be enabled at all, it is the first gate. If HDR appears to enable but the picture is washed out and the display stays in SDR, as in #779, it is the second: the compositor composites in PQ/BT2020 while the sink is never told to switch.

On my setup (RTX 5080, 610.57.04, Acer XV322QK KV over HDMI) the VSDB is 13 bytes and ends with one padding byte after two HDMI VICs, with 3D_present = 0 and HDMI_3D_Len = 0. The parser walk gives DataSz = 5 and DataCnt = 4, so the loop runs once on that byte. Instrumenting nvDpyIsHDRCapable() shows every condition passing except hdmi3D.

Full reasoning, reproducer and verification are in the PR. Trailing reserved bytes in a VSDB are expected and are meant to be ignored for forward compatibility, so I would expect this to affect a range of displays rather than one vendor's EDID.