--rdpenalty 2 is unsafe with rdLevel 4 and below
See valgrind trace of first bad reads, which eventually leads to a crash
http://privatepaste.com/c59793067e
If I comment out these two lines of xRecurIntraCodingQt(), the crashes go away:
if (m_param->rdPenalty == 2) bCheckFull = (log2TrSize <= (uint32_t)(X265_MIN(maxTuSize, 4));
For the 1.3 release, I will bypass this logic but we need a more proper long term solution (and the solution could be to remove this HM feature; it's not clear anyone uses it and psy-rd should do a better job of forcing splits for large intra TU)
Comments (7)
-
reporter -
and the solution could be to remove this HM feature; it's not clear anyone uses it
Probably it's not really clear what it's intension is :) Is it ment to 'trigger' x265 to use inter instead of intra coding or is it ment to promote the use of smaller CTUs. iirc: smaller ctu = faster encoding, better detail preservation, but higher bit rate requirements larger ctu = faster decoding, more detail loss, but less bit rate requirements
http://x265.readthedocs.org/en/latest/cli.html#cmdoption--rdpenalty only states 'Penalty for 32x32 intra TU in non-I slices. Default 0' which isn't really saying much :)
-
reporter true enough; I'll be digging around in this part of the code in the next week or so. I'll try to improve those docs.
but at a high level it tries to force TU splits (transform units) not CU splits (coding units). So it has no effect on the prediction size; only the residual transform size.
-
reporter - changed status to closed
search: fix --rdpenalty 2, make logic explicit (closes
#71)→ <<cset ca0090c8fc69>>
-
please do not forget to update the docs a bit :) Thanks!
-
reporter search: further --rd-penalty 2 fixes (refs
#71)→ <<cset b0eaff8d1d1a>>
-
reporter docs: update --tu-intra-depth / --tu-inter-depth docs (refs
#71)→ <<cset 654ea07f93de>>
- Log in to comment
search: do not skip intra tu 32x32 unless TU intra depth is deep (refs
#71)This may not be the best workaround for the bug, and needs a proper long term fix.
→ <<cset 203c87c55bb3>>