Abnormal distortion bug

[shinchiro]

There is abnormal distortion around walking people when I encode this clip with this parameter in x265 1.7+418-d56b2466c044

--bitrate 1000 --preset medium --tu-intra-depth 2 --tu-inter-depth 2 --rd 6 --psy-rd 0.5 --max-merge 5 --me umh --subme 4 --no-strong-intra-smoothing --rdpenalty 1 --rc-lookahead 90 --bframes 8 --aq-mode 2 --no-sao --deblock -3:-3 --output <output> <input>

clip: https://d.maxfile.ro/qbzhqlfipa.mkv

[fauxreaper]

Did you try disabling psy-rd? --psy-rd 0.0

[Deepthi Nandakumar]

You mean the ghosting around the characters? That's a pretty questionable set of parameters, why not just use --preset veryslow?

[shinchiro]

I'm not sure if its called ghosting effect..I'm just noticed strange/wild distortion on the ground around the character while they are walking

After more testing with --preset only, the problem become more noticeable starting from fast to veryslow.

When using preset veryfast, there is almost no distortion on the ground, just like the original clip

[fauxreaper]

Preset veryfast disables psy-rd.

[shinchiro]

veryfast still enable psy-rd to 0.30

After some experiment, I found that enabling cutree will produce this strange distortion problem.

--preset slower -> strange distortion can be noticed

--preset slower --no-cutree -> less noticeable distortion

--preset slower --no-cutree --ctu 32 -> almost no distortion at all ! :)

[Steve Borho]

Using --aq-mode 1 with --cutree might be safe

[shinchiro]

slower preset already used --aq-mode 1

[Mahesh Pittala]

can you turn off psy-rd, --psy-rdoq and test it again.

I have used same tip 1.7+418-d56b2466c044 and tested with this commandline, almost there is no distortion.

x265.exe --input qbzhqlfipa.yuv --input-res 1280x720 --fps 23.98 -p medium -o output.hevc --psy-rd 0 --psy-rdoq 0 --bitrate 1000 --tu-intra-depth 2 --tu-inter-depth 2 --max-merge 5 --me umh --subme 4 --no-strong-intra-smoothing --rdpenalty 1 --rc-lookahead 90 --bframes 8 --no-sao --deblock -3:-3

[shinchiro]

Sure. I'm testing with latest commit, 1.7+431-f63273fa3137 (just in case it might be accidentally fix the problem) I run tests with following parameter. I'm judging the quality with my eyes, not with psnr or ssim

slower.hevc

--bitrate 1000 --preset slower

slower-nopsy.hevc

--bitrate 1000 --preset slower --no-psy-rd --no-rdoq-level --no-psy-rdoq

slower-nocutree.hevc

--bitrate 1000 --preset slower --no-cutree

slower-nocutree-ctu32.hevc

--bitrate 1000 --preset slower --no-cutree --ctu 32

Here is the result: slower-nocutree-ctu32 > slower-nocutree > slower-nopsy > slower

slower-nocutree-ctu32 produce highest quality output (without distortion problem) compare to slower-nopsy

[Mahesh Pittala]

My observations( compared frame by frame using pro player),

1) I don't see any difference in quality between no-psyrd, -nopsy-rdoq and no-cutree, ctu 32 with your specific commandline, x265.exe --input qbzhqlfipa.yuv --input-res 1280x720 --fps 23.98 -p medium -o output.hevc --psy-rd 0 --psy-rdoq 0 --bitrate 1000 --tu-intra-depth 2 --tu-inter-depth 2 --max-merge 5 --me umh --subme 4 --no-strong-intra-smoothing --rdpenalty 1 --rc-lookahead 90 --bframes 8 --no-sao --deblock -3:-3

2) compared these two commandlines,

--bitrate 1000 --preset slower --no-cutree --ctu 32 : blurred images found in first few(8-10) frames and no distortion found.(we will check the blurring issue) --bitrate 1000 --preset slower --no-psy-rd --no-rdoq-level --no-psy-rdoq : small distortion found in first few(<15) frames and after that there is no distortion.

we will test further more and will update.

[shinchiro]

About blurring issue, I've done x265's encode test on more sample clips. What I found most of them have blurring issue on first few frames at beginning & after that its become normal (whether cu-tree is enabled or not). Is it how x265 works or bug instead?

When comparing to x264's encode under same bitrate, x264 doesn't have this bluring issue at first few frames.

[Mahesh Pittala]

we created an issue for blurring. looks like both 1) --bitrate 1000 --preset slower --no-cutree --ctu 32 2) --bitrate 1000 --preset slower --no-psy-rd --no-rdoq-level --no-psy-rdoq having issue in first few frames. fixing blurring images may solve both issues.

[Gerhard Bogner]

This effects me as well on other clips - tested with presets from medium to veryslow with 8 and 10 bit color depth using crf 22 (+vbv). Disabling psy-rd or lowering crf doesn't seem to help, however --no-cutree fixes/works around the problem (at the cost of increased file size).

For me the distortions are most noticable with moving foreground objects in front of a static background, especially when enabling early-skip in addition.

Additionally I believe I started noticing this before the changes for 1.8 were merged, possibly as far back as the release of 1.7.

[Deepthi Nandakumar]

Can we revisit this issue and see if the recent commits fixed something?

[shinchiro]

The issue still persist. With latest commit, the result is still same with https://bitbucket.org/multicoreware/x265/issues/169/abnormal-distortion-bug#comment-21057561

[shinchiro]

I tried doing this test again with commit 1471e4e433ccccf71f39a6eb93507c349fb8539e

Since someone said x265's mode decision is still bad, I tried to force x265 to use smallest CU/TU size as possible. The result quite promising as I dont notice that weird distortion

Here cli that I used:

--bitrate 1000 --preset slower --ctu 16 --max-tu-size 4 --no-psy-rdoq --no-rdoq-level --no-psy-rd

[Ben Waggoner]

You might try adding --tu-inter 4 --tu-intra 4 so that TUs can recurse down to the smallest size instead of lowering --ctu and --max-tu-size so aggressively. If that helps, it would suggest that having minimum tu sizes available is the issue, not the maximum size.

I’d also try --rskip 0, as sometimes recursion skip can cause issues.

If those work but are too slow, you can try --limit-tu 4 and --rskip 2 to get some of the speed back.