x265 test-harness vbv-testcases failures

[Former user]

I believe the test-harness needs additional modifications to cope with vbv tests' variability. The following behavior is reproducible on both PowerPC and x86-64:

1. Clone x265.
2. Clone test-harness.
3. Create a sequences folder with only big_buck_bunny_360p24.y4m (used by smoke-tests.txt in x265 repository)
4. Edit conf.py in test harness to point to the new sequences folder.
5. Run smoke-test.py (This first run will pass all tests after creating sequences in goldens and using the HM decoder to verify them.)
6. Re-run smoke-test.py in test-harness.The vbv test will fail.

The compiler used on both PowerPC and x86-64 is GCC 4.9.2-10

The cause of the failures is that the tests allow only 5% differences in bitrates of vbv test runs; whereas the failures I observed had bitrate differences of 8-9%. With absolutely no modifications to the x265 source.

Even a note in the test-harness instructions anticipating the behavior would be helpful.

Thanks.

[Mahesh Pittala]

We compare the achieved bitrate to the setpoint(target bitrate in commandline) and reporting when the difference is 5% , in addition to comparing the achieved bitrate to the previously achieved bit-rate when VBV is set.

Here, the target bitrate is very low and number of encoding frames are 50 only in smoke test. so checked with rate control team and planning to increase the tolerance for bitrate comparsion with the setpoint to 10% and keep the tolerance for bitrate comparison with the previous bitrate to 5%.

Thanks for reporting.

[Deepthi Nandakumar]

Should be fixed in the latest test-harness now