yuv444p lossless roundtrip for 1080p videos

[Peter Steinbach]

Hi,

I am testing the yuv444p roundtrip quality of the x265 lossless (ultrafast) mode on some 1080p imagery on todays "tip" (changeset: 11031:f4c267f28487). http://ultravideo.cs.tut.fi/video/Bosphorus_1920x1080_30fps_420_8bit_AVC_MP4.mp4

I am doing the following:

$ ffmpeg -i Bosphorus_1920x1080_30fps_420_8bit_AVC_MP4.mp4 -pix_fmt yuv444p Bosphorus_1920x1080_30fps_444_8bit_AVC_MP4.yuv
$ x265 --lossless --preset=ultrafast --pme --pmode --input-depth 8 --input-res 256x128 --fps 30 --input-csp i444 --input Bosphorus_1920x1080_30fps_444_8bit_AVC_MP4.yuv Bosphorus_1920x1080_30fps_444_8bit_AVC_MP4_lrt.hevc
$  ffmpeg -i Bosphorus_1920x1080_30fps_444_8bit_AVC_MP4_lrt.hevc -pix_fmt yuv444p Bosphorus_1920x1080_30fps_444_8bit_AVC_MP4_lrt.yuv
$ md5sum Bosphorus*yuv
a5218f68414d8956e6e69bdb48301228  Jockey_1920x1080_30fps_444_8bit_AVC_MP4.yuv
bc55e7201dc65f6cc542dabd4dc50ba9  Jockey_1920x1080_30fps_444_8bit_AVC_MP4_lrt.yuv

The funny thing, with smaller crops (made with ffmpeg) of the same video, lossless compression works perfectly and yields identical md5 hashes for the input and output *.yuv files.

I am unclear if this is a ffmpeg thing or just x265 misbehaving?

Best, P

[Deepthi Nandakumar]

Hi,

x265 is certainly expected to be fully lossless for all resolutions. Are you saying that for 1080p resolutions this does not work?

You've displayed the md5sums of Jockey**.yuv

[Peter Steinbach]

yes, this is my question. I guess I copy-and-pasted the wrong commands. Apologies for that. here is the stuff I'm doing with the x265 tip from Oct 7 from stable branch and using ffmpeg 2.8 linked to it.

$ ffmpeg -i Jockey_1920x1080_30fps_420_8bit_AVC_MP4.mp4 -pix_fmt yuv444p Jockey_1920x1080_30fps_444_8bit_AVC_MP4.yuv
$ x265 --lossless --preset=ultrafast --input-depth 8 --input-res 1920x1080 --fps 30 --input-csp i444 Jockey_1920x1080_30fps_444_8bit_AVC_MP4.yuv Jockey_1920x1080_30fps_444_8bit_AVC_MP4_lrt.hevc
$ ffmpeg -i Jockey_1920x1080_30fps_444_8bit_AVC_MP4_lrt.hevc -pix_fmt yuv444p Jockey_1920x1080_30fps_444_8bit_AVC_MP4_lrt.yuv 
$ md5sum Jockey_1920x1080_30fps_444*yuv
bc55e7201dc65f6cc542dabd4dc50ba9  Jockey_1920x1080_30fps_444_8bit_AVC_MP4_lrt.yuv
a5218f68414d8956e6e69bdb48301228  Jockey_1920x1080_30fps_444_8bit_AVC_MP4.yuv

Strange, isn't it?

PS. for completeness:

$ hg tip
changeset:   11037:a85011719ea1
branch:      stable
tag:         tip
parent:      11033:98ac099a766f
user:        Deepthi Nandakumar <[e-mail]>
date:        Wed Oct 07 09:22:44 2015 +0530
summary:     scenecut: fix an obvious bug in scenecut thresholds

[Peter Steinbach]

ping? this is a show breaker for our application of x265. we have gray16 progressive data and are looking into ways to compress it with x265. yuv444 was one option. I'll try the above again with the current tip.

$ hg tip
changeset:   11089:04575a459a16
tag:         tip
user:        Ramya Sriraman <[e-mail]>
date:        Wed Sep 30 11:22:16 2015 +0530
summary:     asm: Add sse_ss for [16x16],[32x32] & [64x64] for 8bpp avx2

and the result is:

$ md5sum Jockey_1920x1080_30fps_444*yuv
bc55e7201dc65f6cc542dabd4dc50ba9  Jockey_1920x1080_30fps_444_8bit_AVC_MP4_lrt.yuv
a5218f68414d8956e6e69bdb48301228  Jockey_1920x1080_30fps_444_8bit_AVC_MP4.yuv

So whatever it is, it's still there. Any help would be appreciated!

[Peter Steinbach]

one more piece of information, I just encoded the same file through ffmpeg/libx265, and the error prevails (so its unlikely due to some bogous i/o while writing the yuv "container"):

$ ffmpeg -pix_fmt yuv444p -s 1920x1080 -r 30 -i Jockey_1920x1080_30fps_444_8bit_AVC_MP4.yuv -c:v libx265 -preset ultrafast -x265-params lossless=1 -strict experimental Jockey_1920x1080_30fps_444_8bit_AVC_MP4_ffmpegencoded.hevc
$ ffmpeg -i Jockey_1920x1080_30fps_444_8bit_AVC_MP4_ffmpegencoded.hevc -pix_fmt yuv444p Jockey_1920x1080_30fps_444_8bit_AVC_MP4_ffmpegencoded.yuv
$ md5sum Jockey_1920x1080_30fps_4*yuv
bc55e7201dc65f6cc542dabd4dc50ba9  Jockey_1920x1080_30fps_444_8bit_AVC_MP4_ffmpegencoded.yuv
bc55e7201dc65f6cc542dabd4dc50ba9  Jockey_1920x1080_30fps_444_8bit_AVC_MP4_lrt.yuv
a5218f68414d8956e6e69bdb48301228  Jockey_1920x1080_30fps_444_8bit_AVC_MP4.yuv

[Deepthi Nandakumar]

Sorry for the delay, I tested this out on the x265 commandline with a couple of testcases (crowdrun, duckstakeoff - 8bit 444, 1080p) and it worked fine. Our automated regression test results have also not complained.

Can you use x265.exe and see if you're seeing the same results?

One more thing to check, are the csp parameters being passed from ffmpeg to x265? This was my standalone x265 commandline: D:\YUVClips\CrowdRun_1920x1080_50_8bit_444.yuv --input-res 1920x1080 --fps 25 --input-csp i444 -o crowdrun.hevc --psnr --ssim -p ultrafast --lossless --hash 1 -f 100

[Deepthi Nandakumar]

Never mind, I see that you used the x265 commandline earlier

[Peter Steinbach]

mmmh, my roundtrips with all frames (not just 100) still fail. I used the duckstakeoff 1080p50 sample from here. but I found something odd:

$ x265 ducks_take_off_1080p50_fps50_444.yuv --input-res 1920x1080 --fps 50 --input-csp i444 -o ducks.hevc --psnr --ssim -p ultrafast --lossless
...
$$ ffprobe ducks.hevc
ffprobe version n2.8 Copyright (c) 2007-2015 the FFmpeg developers
  built with gcc 4.8.3 (GCC) 20140911 (Red Hat 4.8.3-9)
  configuration: --prefix=/home/steinbac/software/ffmpeg/2.8-x265master --enable-libx265 --enable-libx264 --enable-gpl --enable-shared
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
[hevc @ 0xa5a600] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, hevc, from 'ducks.hevc':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: hevc (Rext), yuv444p(tv), 1920x1072, 50 tbr, 1200k tbn, 50 tbc

if I then decode this with ffmpeg to say y4m and look at the header I again encounter 1920x1072 not 1920x1080 as it should be! So is the ffmpeg 2.8 decoder wrong then?

$ ffmepg -i ducks.hevc ducks.y4m
$ strings ducks.y4m|head -n1
YUV4MPEG2 W1920 H1072 F50:1 Ip A0:0 C444 XYSCSS=444

[Peter Steinbach]

submitted a ffmpeg bug report for now: https://trac.ffmpeg.org/ticket/4980

It apparently is a problem with the ultrafast versus fast preset:

 $ ffmpeg -f lavfi -i testsrc=s=320x248 -vframes 10 -pix_fmt yuv444p
 out_10.yuv
 $ ffmpeg -f lavfi -i testsrc=s=320x248 -vframes 10 -strict experimental
 -pix_fmt yuv444p -vcodec libx265 -preset ultrafast -x265-params lossless=1
 out_10_lossless.hevc
 $ ffmpeg -i out_10_lossless.hevc -pix_fmt yuv444p out_10_lrt.yuv
 $ md5sum out_10*.yuv
 cd5e06d1217f093c4265a94b30e11e53  out_10_lrt.yuv
 bbc307efc496e074b177b51f555f4437  out_10.yuv
 $ ffmpeg -f lavfi -i testsrc=s=320x248 -vframes 10 -strict experimental
 -pix_fmt yuv444p -vcodec libx265 -preset fast -x265-params lossless=1
 out_10_lossless_fast.hevc
 $ ffmpeg -i out_10_lossless_fast.hevc -pix_fmt yuv444p out_10_lrt_fast.yuv
 $ md5sum out_10*.yuv
 bbc307efc496e074b177b51f555f4437  out_10_lrt_fast.yuv
 cd5e06d1217f093c4265a94b30e11e53  out_10_lrt.yuv
 bbc307efc496e074b177b51f555f4437  out_10.yuv

btw, the decoding without the lossless flag in ultrafast also produces the wrong resolution! so it seems not only limited to that ... I used x265 tip and ffmpeg 2.9-dev

[Peter Steinbach]

ping, any news on this? it seems to affect lossless and lossy compression!! (see https://trac.ffmpeg.org/ticket/4980)

[Ashok Kumar Mishra]

We are working on this issue with high priority, will let you know once we fix it.

[Peter Steinbach]

awesome.

[Ashok Kumar Mishra]

This issue is with ffmpeg decoder and that is for 444 colorspcae format only. I tested with HM decoder(HM 16.4) which is giving the correct output. ffmpeg decoder with 420 colorspace format is working fine.

[Ashok Kumar Mishra]

Make sure you are using the correct input color space format to test and check with HM decoder (latest 16.4) for conformance. As I found the video you pasted "Bosphorus_1920x1080_30fps_420_8bit_AVC_MP4" is 420 video. But you have given the command line option --input-csp i444

[Peter Steinbach]

Ok, I took the yuv444p ducks_take_off sample from here: https://media.xiph.org/video/derf/ (using the x265 tip and ffmpeg master pulled/checkedout on Nov 17, 1:40 CET)

$ x265 --lossless --preset=ultrafast --input-depth 8 --input-res 1200x720 --fps 50 --input-csp i444 ducks_take_off_444_1200x720p50.yuv ducks_take_off_444_1200x720p50_lossless_ultrafast.hevc
$ ffmpeg -i ducks_take_off_444_1200x720p50_lossless_ultrafast.hevc -pix_fmt yuv444p ducks_take_off_444_1200x720p50_lossless_ultrafast.yuv
$ ${HOME}/software/jctvc-hm/repo/bin/./TAppDecoderStatic -b ducks_take_off_444_1200x720p50_lossless_ultrafast.hevc -o ducks_take_off_444_1200x720p50_lossless_ultrafast_HM.yuv --OutputColourSpaceConvert=UNCHANGED
$ md5sum  ducks_take_off_444_1200x720p50.yuv ducks_take_off_444_1200x720p50_lossless_ultrafast.yuv ducks_take_off_444_1200x720p50_lossless_ultrafast_HM.yuv
c1d2b822af36d5bd84f3f1a516851b45  ducks_take_off_444_1200x720p50.yuv
022c44ad7e5623144e0e7082db243fd4  ducks_take_off_444_1200x720p50_lossless_ultrafast.yuv
2ea9da8efee359221599b502413e957b  ducks_take_off_444_1200x720p50_lossless_ultrafast_HM.yuv

I'll ping pong this back to the ffmpeg people then! thanks so far.