--rdpenalty 2 is unsafe with rdLevel 4 and below

[Steve Borho]

See valgrind trace of first bad reads, which eventually leads to a crash

http://privatepaste.com/c59793067e

If I comment out these two lines of xRecurIntraCodingQt(), the crashes go away:

if (m_param->rdPenalty == 2) bCheckFull = (log2TrSize <= (uint32_t)(X265_MIN(maxTuSize, 4));

For the 1.3 release, I will bypass this logic but we need a more proper long term solution (and the solution could be to remove this HM feature; it's not clear anyone uses it and psy-rd should do a better job of forcing splits for large intra TU)

[Steve Borho]

search: do not skip intra tu 32x32 unless TU intra depth is deep (refs #71)

This may not be the best workaround for the bug, and needs a proper long term fix.

→ <<cset 203c87c55bb3>>

[Selur]

and the solution could be to remove this HM feature; it's not clear anyone uses it

Probably it's not really clear what it's intension is :) Is it ment to 'trigger' x265 to use inter instead of intra coding or is it ment to promote the use of smaller CTUs. iirc: smaller ctu = faster encoding, better detail preservation, but higher bit rate requirements larger ctu = faster decoding, more detail loss, but less bit rate requirements

http://x265.readthedocs.org/en/latest/cli.html#cmdoption--rdpenalty only states 'Penalty for 32x32 intra TU in non-I slices. Default 0' which isn't really saying much :)

[Steve Borho]

true enough; I'll be digging around in this part of the code in the next week or so. I'll try to improve those docs.

but at a high level it tries to force TU splits (transform units) not CU splits (coding units). So it has no effect on the prediction size; only the residual transform size.

[Steve Borho]

search: fix --rdpenalty 2, make logic explicit (closes #71)

→ <<cset ca0090c8fc69>>

[Selur]

please do not forget to update the docs a bit :) Thanks!

[Steve Borho]

search: further --rd-penalty 2 fixes (refs #71)

→ <<cset b0eaff8d1d1a>>

[Steve Borho]

docs: update --tu-intra-depth / --tu-inter-depth docs (refs #71)

→ <<cset 654ea07f93de>>