unstable SSIM with CRF

[Former user]

Hi, it seems SSIM became unstable since 1.2 with CRF. Is it expected?

x265 [debug]: POC:3001 I QP 23.73(28)     995232 bits RF:0.158 [SSIM: 17.699dB]
x265 [debug]: POC:3006 P QP 25.57(30)     658256 bits RF:0.158 [SSIM: 15.637dB] [L0 3001 ]
x265 [debug]: POC:3004 B QP 25.90(31)     384224 bits RF:0.158 [SSIM: 15.684dB] [L0 3001 ] [L1 3006 ]
x265 [debug]: POC:3002 b QP 31.09(32)     152400 bits RF:0.158 [SSIM: 13.741dB] [L0 3001 ] [L1 3004 3006 ]
x265 [debug]: POC:3003 b QP 31.18(32)     150816 bits RF:0.158 [SSIM: 12.867dB] [L0 3001 ] [L1 3004 3006 ]
x265 [debug]: POC:3005 b QP 31.06(32)      88712 bits RF:0.158 [SSIM: 14.673dB] [L0 3004 3001 ] [L1 3006 ]
x265 [debug]: POC:3010 P QP 24.96(30)     686032 bits RF:0.158 [SSIM: 16.418dB] [L0 3006 3004 3001 ]
x265 [debug]: POC:3008 B QP 28.42(31)     186344 bits RF:0.158 [SSIM: 14.984dB] [L0 3006 3004 3001 ] [L1 3010 ]
x265 [debug]: POC:3007 b QP 31.10(32)      81192 bits RF:0.158 [SSIM: 14.535dB] [L0 3006 3004 ] [L1 3008 3010 ]
x265 [debug]: POC:3009 b QP 31.15(32)      78080 bits RF:0.158 [SSIM: 13.676dB] [L0 3008 3006 3004 ] [L1 3010 ]

x265 [info]: frame I: 20     Avg QP:24.21  kb/s: 15594.81  SSIM Mean: 0.987350 (18.979dB)
x265 [info]: frame P: 826    Avg QP:26.31  kb/s: 8557.98   SSIM Mean: 0.980962 (17.204dB)
x265 [info]: frame B: 2303   Avg QP:29.74  kb/s: 3092.68   SSIM Mean: 0.973520 (15.771dB)
x265 [info]: global : 3149   Avg QP:28.80  kb/s: 4605.66   SSIM Mean: 0.975560 (16.119dB)

x265 [info]: HEVC encoder version 1.2
x265 [info]: build info [Linux][GCC 4.8.4][64 bit] 8bpp
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2
x265 [info]: WPP streams / pool / frames         : 17 / 2 / 1
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: CU size                             : 64
x265 [info]: Max RQT depth inter / intra         : 1 / 1
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut       : 29 / 600 / 20
x265 [info]: Lookahead / bframes / badapt        : 30 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb / refs: 1 / 1 / 0 / 3
x265 [info]: Rate Control / AQ-Strength / CUTree : CRF-24.0 / 1.0 / 1
x265 [info]: VBV/HRD buffer / max-rate / init    : 9000 / 9000 / 0.900
x265 [info]: tools: rd=3 lft sao-lcu signhide 

ffmpeg arguments:
-vf yadif,pp=ac -c:a copy -map 0:0 -map 0:1 -c:v libx265 -x265-params crf=24:ssim=1:keyint=600:rc-lookahead=30:scenecut=20:log=3:vbv-maxrate=9000:vbv-bufsize=9000 -preset medium

[Deepthi Nandakumar]

Okay, our typical use case for VBV is to set a target bitrate along with VBV parameters. We havent really tested CRF with VBV - the user would really need to know the average bitrate generated by a certain CRF and then tailor the remaining parameters accordingly.

Nevertheless, what makes you think the above output SSIM is unstable?

[Former user]

There is stable SSIM with 1.1
x265 [debug]: POC:3001 I QP 22.42(27)    1141056 bits RF:0.158 [SSIM: 16.550dB]
x265 [debug]: POC:3006 P QP 25.57(30)     659680 bits RF:0.158 [SSIM: 16.549dB] [L0 3001 ]
x265 [debug]: POC:3004 B QP 24.74(30)     479656 bits RF:0.158 [SSIM: 16.420dB] [L0 3001 ] [L1 3006 ]
x265 [debug]: POC:3002 b QP 28.83(30)     234624 bits RF:0.158 [SSIM: 16.426dB] [L0 3001 ] [L1 3004 3006 ]
x265 [debug]: POC:3003 b QP 28.94(30)     239448 bits RF:0.158 [SSIM: 16.351dB] [L0 3001 ] [L1 3004 3006 ]
x265 [debug]: POC:3005 b QP 28.80(30)     136984 bits RF:0.158 [SSIM: 16.553dB] [L0 3004 3001 ] [L1 3006 ]
x265 [debug]: POC:3010 P QP 24.96(30)     697952 bits RF:0.158 [SSIM: 16.492dB] [L0 3006 3004 3001 ]
x265 [debug]: POC:3008 B QP 27.30(30)     241696 bits RF:0.158 [SSIM: 16.608dB] [L0 3006 3004 3001 ] [L1 3010 ]
x265 [debug]: POC:3007 b QP 28.83(30)     127464 bits RF:0.158 [SSIM: 16.660dB] [L0 3006 3004 ] [L1 3008 3010 ]
x265 [debug]: POC:3009 b QP 28.88(30)     131576 bits RF:0.158 [SSIM: 16.637dB] [L0 3008 3006 3004 ] [L1 3010 ]

but unstable with 1.2
x265 [debug]: POC:3001 I QP 23.73(28)     994856 bits RF:0.158 [SSIM: 17.704dB]
x265 [debug]: POC:3006 P QP 25.57(30)     658048 bits RF:0.158 [SSIM: 15.601dB] [L0 3001 ]
x265 [debug]: POC:3004 B QP 25.90(31)     386848 bits RF:0.158 [SSIM: 15.686dB] [L0 3001 ] [L1 3006 ]
x265 [debug]: POC:3002 b QP 31.09(32)     149552 bits RF:0.158 [SSIM: 13.704dB] [L0 3001 ] [L1 3004 3006 ]
x265 [debug]: POC:3003 b QP 31.18(32)     150400 bits RF:0.158 [SSIM: 12.877dB] [L0 3001 ] [L1 3004 3006 ]
x265 [debug]: POC:3005 b QP 31.06(32)      88848 bits RF:0.158 [SSIM: 14.632dB] [L0 3004 3001 ] [L1 3006 ]
x265 [debug]: POC:3010 P QP 24.96(30)     686448 bits RF:0.158 [SSIM: 16.421dB] [L0 3006 3004 3001 ]
x265 [debug]: POC:3008 B QP 28.42(31)     182520 bits RF:0.158 [SSIM: 14.977dB] [L0 3006 3004 3001 ] [L1 3010 ]
x265 [debug]: POC:3007 b QP 31.10(32)      81400 bits RF:0.158 [SSIM: 14.537dB] [L0 3006 3004 ] [L1 3008 3010 ]
x265 [debug]: POC:3009 b QP 31.15(32)      77616 bits RF:0.158 [SSIM: 13.624dB] [L0 3008 3006 3004 ] [L1 3010 ]

With 1.1 I/P/B frames have similar SSIM Mean
x265 [info]: frame I: 20     Avg QP:23.03  kb/s: 17604.48  SSIM Mean: 0.977459 (16.470dB)
x265 [info]: frame P: 826    Avg QP:26.31  kb/s: 8455.28   SSIM Mean: 0.976481 (16.286dB)
x265 [info]: frame B: 2303   Avg QP:27.91  kb/s: 4181.57   SSIM Mean: 0.976276 (16.248dB)
x265 [info]: global : 3149   Avg QP:27.46  kb/s: 5387.84   SSIM Mean: 0.976337 (16.259dB)
x265 [info]: Weighted P-Frames: Y:1.6% UV:1.3%
x265 [info]: consecutive B-frames: 4.6% 5.4% 26.5% 40.1% 23.4%

It is different with 1.2 
x265 [info]: frame I: 20     Avg QP:24.21  kb/s: 15590.61  SSIM Mean: 0.987357 (18.982dB)
x265 [info]: frame P: 826    Avg QP:26.31  kb/s: 8556.66   SSIM Mean: 0.980979 (17.208dB)
x265 [info]: frame B: 2303   Avg QP:29.74  kb/s: 3093.23   SSIM Mean: 0.973534 (15.773dB)
x265 [info]: global : 3149   Avg QP:28.80  kb/s: 4605.69   SSIM Mean: 0.975575 (16.122dB)
x265 [info]: Weighted P-Frames: Y:1.6% UV:1.3%
x265 [info]: consecutive B-frames: 4.6% 5.4% 26.5% 40.1% 23.4%

I'm compiling latest ffmpeg trunk, which means numbers can be slightly different from above.

[Former user]

VBV+CRF seems to work fine with 1.1. Quality is decreased only when bitrate exceeds limit.

[Former user]

there is also unstable SSIM with CRF without VBV

[Deepthi Nandakumar]

Hi,

2 changesets are responsible for this. a) bf112abf087c pic: move SSIM, PSNR and hash to frame encoder This changeset fixed the SSIM computation. Previously, the SSIM metrics werent accurate as they werent being reset with each frame. This did not change the bitstream or recon though.

b) 0f68adb0e190 rc: set pbFactor to default with cutree This changeset changed some defaults that cutree was tampering with, that we felt cutree should not be tampering with. This changed the QPs assigned to B frames.

These are expected to change outputs.

[Former user]

1.2 is using higher QP for same frame which is expected and I should use lower CRF to get same QP. Right? You can close this issue if all changes are expected and there is no bug.

[Steve Borho]

VBV is non-deterministic, as a rule. It is much more non-deterministic if it needs to make mid-frame adjustments but even without them the outputs are not guaranteed to match

[Former user]

1.2 is using higher QP also without VBV

[Werner Robitza]

@sborho, since you seem to know most about this, could you please briefly explain (or point to some documentation) why VBV encoding with x265 is necessarily non-deterministic? Thanks!