"merge mode not ok" in checked build for rate-control

[Pradeep Ramachandran]

When x265 is built with checks and warnings checked, it reports "merge mode not ok" and dumps core.

Reproducible command line: ./x265.exe big_buck_bunny_360p24.y4m --preset medium --bitrate 1000 --pass 2 -F4 --no-info --hash=1 --psnr --ssim --log=full

[Deepthi Nandakumar]

Can you also share pass 1 commandline?

[Pradeep Ramachandran]

I think that the check fails even in the first pass - here's another command line: big_buck_bunny_360p24.y4m --preset medium --bitrate 400 --vbv-bufsize 600 --vbv-maxrate 600 --aud --hrd --tune fast-decode --no-info --hash=1 --psnr --ssim --asm=AVX

Pradeep.

[Deepthi Nandakumar]

There were overflows in distortion calculations at 10-bit and 12-bit. Fixed.