Color degradation when encoding in 10 bit or 12 bit

[Fabian Lau]

Versions tested

• x265 1.7+512, [Windows][MSVC 1600][64 bit] 8bit
• x265 1.7+512, [Windows][MSVC 1600][64 bit] 10bit
• x265 2.3+33, [Windows][GCC 6.3.0][64 bit] 10bit
• x265 2.5+6, [Windows][GCC 7.1.0][64 bit] 8bit+10bit+12bit
• x265 2.5+12, [Windows][GCC 6.2.1][64 bit] 10bit

Description

There seems to be some sort of pixel grid overlaying the picture when encoding to 10 or 12 bit regardless of the source bit depth, resulting in artificially increased picture complexity and an apparent if slight overall loss of color fidelity. It looks like a faint greenish veil is put over the picture (grid noticeable when zooming in), mostly noticeable with the skin tones of white people. When encoding to 8 bit the colors look identical to the source.

Steps to reproduce

Encode any video with --output-depth 10 or --output-depth 12 with --lossless.

Actual Result

The encoded video has a greenish looking tinge and when zoomed in exposes a faint pixel grid. I have attached screenshots of a small clip encoded with --lossless that exaggerate the difference between the original colors (8 bit) and the colors in 10 bit. After 8 encoding iterations I couldn't open the file in Staxrip anymore because of an index error.

• gs-grid-8bit.png with 5x magnification is bitwise identical to the source.
• gs-grid-10bit-1st-iteration.png with 5x magnification shows a faint grid overlay.
• gs-grid-10bit-8th-iteration.png completely exposes the grid.

Expected Result

Colors in 10 or 12 bit should look the same as the source, without a grid overlay. The output with --lossless in 10 or 12 bit should be bitwise identical to the source.

[Pradeep Ramachandran]

Thanks for the detailed description. Are you seeing this regardless of source bit-depth, or does this happen only when source is 8-bit and you try encoding with main10 or main12 profiles (trying to triage which portion of the encoder the issue is coming from).

[Fabian Lau]

Yes, I'm seeing this regardless of source bit-depth. Preset, mode and profile don't matter either (i.e. it also happens with profile set to unrestricted).

The only option that matters seems to be --output-depth 10 or 12. I used --lossless to observe the generational color loss without compression side effects.

• Source 8/10/12 bit -> Target 8 bit -- colors look fine
• Source 8/10/12 bit -> Target 10/12 bit -- faint color degradation

I only tested with YUV 4:2:0 sources though.

[Fabian Lau]

Added three pictures with 5x magnification that highlight the underlying issue: Some sort of pixel grid seems to overlay the picture, so it's not a uniform color loss.

• gs-grid-8bit.png (looks like source)
• gs-grid-10bit-1st.png (grid faintly visible)
• gs-grid-10bit-8th.png (grid after 8 encoding iterations)

[Fabian Lau]

Updated description to emphasize the underlying issue, not its perceived effect.

[Fabian Lau]

Also tested x265 1.7+512 8bit and 10bit. Same issue with 10bit.

[Ma0]

I can't reproduce this bug, but my CPU is only with AVX. Could you test with additional option: --asm=avx

[Fabian Lau]

Ma0: tried --asm=avx; same result. I also tried x264 10 bit by the way (with --qp = 0) and got the same grid overlay. Could this be caused by some encoder or decoder stuff in StaxRip or even something external like LAV filters or madVR, which I have installed?

I attached my StaxRip 1.6.0 log for the --asm=avx try.

[Ma0]

It looks like old bug in ffmpeg that is probably affecting LSMASHSource.dll when it reduces bit-depth from 10-bit to 8-bit -- see post http://forum.doom9.org/showthread.php?p=1801020#post1801020

[Fabian Lau]

Bit rate shouldn't be reduced though when the source is already 10 bit, should it? But yes, it does sound like it could be a similar issue.

I just tried direct command line encoding in Ubuntu to see if it behaves any differently from Windows.

> ffmpeg -i 8bit-source.mkv -c:v rawvideo -pix_fmt yuv420p 8bit.yuv
> x265-10bit --preset medium --input-res 1920x1080 --fps 24/1 --sar 1:1 --lossless --input "8bit.yuv" --output "output10bit.hevc"

Result: same grid visible. Same result when re-encoding a 10 bit source to 10 bit. Everything fine when encoding a 8 or 10 bit source to 8 bit (lossless).

However, with chroma subsampling 4:4:4 everything looks fine even in 10 bit. [edit] There do seem to be some very small pixel / color fluctuations when compared to 8 bit yuv444. I'll try to do some iterations tomorrow to see if it adds up to a grid structure ...

> ffmpeg -i 8bit-source.mkv -c:v rawvideo -pix_fmt yuv444p 8bit444.yuv
> x265-10bit --preset medium --input-res 1920x1080 --fps 24/1 --sar 1:1 --lossless --input "8bit444.hevc" --output "output10bit444.hevc"

Hope this helps. Not sure what else I could test.

I attached the 8bitsource.mkv I was using in my tests. Just 20 frames. I used frame 12 for screenshots via StaxRip (Tools -> Advanced -> Video comparison).

[Ma0]

The bug is not in x265, it is in ffmpeg (or its clones, like LSMASHSource.dll). If you want to properly decode high bit-depth source by ffmpeg (and workaround this bug), please use '-pix_fmt yuv420p16' (instead of yuv420p) and please add '--input-depth 16' to x265 command line (for rawvideo format). There is some activity in ffmpeg development team, so there is a hope for fix in ffmpeg for properly converting high bit-depth sources to 8-bit output.

[Fabian Lau]

Thanks, I'll try. I'm not much into command-line encoding and prefer using Staxrip, but I assume it should be possible to do make the workaround ... work somehow.

Is there any ffmpeg tracker for this bug somewhere or a ticket that I could follow?

[Ma0]

The patch currently tested is https://patchwork.ffmpeg.org/patch/4994/ You can build ffmpeg with this patch and test (or download compiled and patched ffmpeg for Win64 -- www.msystem.waw.pl/x265/patched.7z ) For test movie "KristenAndSara_1280x720_60.y4m" (you can download it from https://media.xiph.org/video/derf/ ) I've prepared batch file:

ffmpeg -i KristenAndSara_1280x720_60.y4m -v warning -strict -1 -pix_fmt yuv420p -f yuv4mpegpipe - | x265 -D10 --y4m - --lossless -o o2.hevc

for /L %%i in (1 1 100) do (
copy /B /Y o2.hevc o1.hevc
ffmpeg -i o1.hevc -v warning -strict -1 -pix_fmt yuv420p -f yuv4mpegpipe - | x265 -D10 --y4m - --lossless -o o2.hevc
)

which encodes 100 times this movie (lossless). You can compare result encode from original ffmpeg and from the patched one.

[Fabian Lau]

Nice. Getting there. I read the patch progress and it seems there are still some things to iron out. Nevertheless I did some testing with your patched ffmpeg binary vs. the current offical ffmpeg 3.3.3.

Thanks for the command lines and sample video -- I used them as a base for further tests.

1. I can confirm that your patched ffmpeg works for re-encoding in 10 bit.
2. It does not work when encoding from 8 to 10 bit, so the very first conversion from the y4m file to hevc before the loop still shows the grid / pattern defect.
3. When converting from 10 back to 8 bit, there's a pattern again but the image is identical to the original source. Without the patch, 10 bit and the encoded 8 bit image look identical.

Here's a list of the results:

ffmpeg 3.3.3:

First iteration:
    src   -> 8bit: OK
    src   -> 10bit: pattern
Second iteration:
    8bit  -> 8bit: OK
    8bit  -> 10bit: pattern
    10bit -> 10bit: pattern (cumulative)
    10bit -> 8bit: OK


ffmpeg patched:

First iteration:
    src   -> 8bit: OK
    src   -> 10bit: pattern
Second iteration:
    8bit  -> 8bit: OK
    8bit  -> 10bit: pattern
    10bit -> 10bit: OK
    10bit -> 8bit: pattern, but identical to source
                   (so 8bit -> 10bit -> 8bit looks lossless)

For simple visual comparison I used Photoshop to put all frames in different layers and used "difference" mode.

I attached an image of the "pattern" I'm talking about, which is the difference between two images, with the brightness increased a few times to make it really visible.

[Ma0]

Fixed in ffmpeg https://github.com/FFmpeg/FFmpeg/commit/f192f2f061d979e3d2716b9892ca276b5d5cd418

[Brian Fong]

I have the same green tinge problem with my skin tones when I encode with x265 crf 22 in 10-bit and 12-bit. 8-bit works great without greening my video.

Got a Intel i5 8600k and I am using Handbrake 1.2.2 (2019022300)