Major artifacts when using VBV limit in 12-bit

[raphael.lerbour]

Hello

Encoding noisy input in 12-bit, targeting a relatively high quality but with maximum bitrate enforced by VBV, we get a video full of artifacts. Also, the resulting file is much smaller (about half) than what I get with 8-bit and 10-bit encodings, that don’t have the issue.

Increasing (or disabling) VBV maxrate or bufsize fixes the artifacts, so I’m guessing the bug is triggered by hitting the VBV limit.

Tested with x265 4.0 but I also noticed the bug with earlier versions, I guess it’s always been there.

I have uploaded 1 second of video (y4m source + raw hevc and mp4 output) that reproduces the bug here: https://drive.google.com/drive/folders/1N3pBtyUQ-eFW8i0uxeFktE9mZc6XOUXC?usp=sharing

Options used: --crf 18 --vbv-maxrate 300000 --vbv-bufsize 50000