Bad psy-rd adoption (?) leading to banding artifacts in 10-bit encoding.

[joletb]

It seems like the current psychovisual rate-distortion implementation in x265 doesn't work quite well and often leads to banding artifacts in 10-bit (and probably 12-bit as well) encoding.

Input is 10-bit 4:4:4 1280x720p. (Encoder settings listed below)

Here are some of the sample comparisons: https://diff.pics/hRr1LIrNKJcS/1

x264 settings

Writing library                          : x264 core 148 r2762+56 7979737 t_mod_Custom [10-bit@all X86_64]
Encoding settings                        : cabac=1 / ref=16 / deblock=1:0:0 / analyse=0x3:0x133 / me=umh / subme=10 / psy=1 / fade_compensate=0.80 / psy_rd=0.80:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=4 / threads=12 / lookahead_threads=3 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / fgo=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=15.5000 / qcomp=0.65 / qpmin=0:0:0 / qpmax=81:81:81 / qpstep=4 / ip_ratio=1.40 / aq=3:0.80 / aq-sensitivity=10.00 / aq-factor=1.00:1.00:1.00 / aq2=0 / aq3=0

x265

Writing library                          : x265 2.4+28-f850cdbe381c:[Windows][GCC 5.3.1][64 bit] 10bit
Encoding settings                        : cpuid=1173503 / frame-threads=3 / numa-pools=8 / wpp / no-pmode / no-pme / no-psnr / ssim / log-level=2 / input-csp=3 / input-res=1280x720 / interlace=0 / total-frames=2157 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=5 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=23 / keyint=250 / bframes=16 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=40 / lookahead-slices=0 / scenecut=40 / no-intra-refresh / ctu=64 / min-cu-size=8 / rect / amp / max-tu-size=32 / tu-inter-depth=3 / tu-intra-depth=3 / limit-tu=4 / rdoq-level=2 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / no-strong-intra-smoothing / max-merge=4 / limit-refs=1 / limit-modes / me=3 / subme=4 / merange=57 / temporal-mvp / weightp / weightb / no-analyze-src-pics / deblock=0:0 / no-sao / no-sao-non-deblock / rd=6 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / b-intra / rdpenalty=0 / psy-rd=2.50 / psy-rdoq=1.00 / no-rd-refine / analysis-mode=0 / no-lossless / cbqpoffs=6 / crqpoffs=6 / rc=crf / crf=16.5 / qcomp=0.65 / qpstep=4 / stats-write=0 / stats-read=0 / ipratio=1.40 / pbratio=1.30 / aq-mode=3 / aq-strength=0.80 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / sar=0 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=1 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / opt-qp-pps / opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / refine-level=5 / no-limit-sao / ctu-info=0

x265 (psy disabled)

Writing library                          : x265 2.4+28-f850cdbe381c:[Windows][GCC 5.3.1][64 bit] 10bit
Encoding settings                        : cpuid=1173503 / frame-threads=3 / numa-pools=8 / wpp / no-pmode / no-pme / no-psnr / ssim / log-level=2 / input-csp=3 / input-res=1280x720 / interlace=0 / total-frames=2157 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=5 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=23 / keyint=250 / bframes=16 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=40 / lookahead-slices=0 / scenecut=40 / no-intra-refresh / ctu=64 / min-cu-size=8 / rect / amp / max-tu-size=32 / tu-inter-depth=3 / tu-intra-depth=3 / limit-tu=4 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / no-strong-intra-smoothing / max-merge=4 / limit-refs=1 / limit-modes / me=3 / subme=4 / merange=57 / temporal-mvp / weightp / weightb / no-analyze-src-pics / deblock=0:0 / no-sao / no-sao-non-deblock / rd=6 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / b-intra / rdpenalty=0 / psy-rd=0.00 / psy-rdoq=0.00 / no-rd-refine / analysis-mode=0 / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=16.5 / qcomp=0.65 / qpstep=4 / stats-write=0 / stats-read=0 / ipratio=1.40 / pbratio=1.30 / aq-mode=3 / aq-strength=0.80 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / sar=0 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=1 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / opt-qp-pps / opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / refine-level=5 / no-limit-sao / ctu-info=0

Encodes have been attached. Link to the lossless encode can be found here.

[Pradeep Ramachandran]

Thanks for the report and the clips. Anime content is a bit challenging owing to its sharp line and bold strokes. We will get around to looking into this issue, but given our current priorities, I'm not sure when we will get there. As always, patches to fix this problem are welcome in the meanwhile.

[Ilya87]

Banding artifacts appear even in 2 pass 10 bit encoding with crf 14 (--rd 5 -b 8 --b-adapt 2 --b-pyramid --ref 6 -I 240 --crf 14 --vbv-maxrate 20000 --vbv-bufsize 20000 --aq-mode 2 --weightp --weightb -m 2 --no-early-skip --me star --rect --rdoq-level 2 --no-psy-rd). Resulted bitrate 2880.45 for real blu-ray FullHD source. x264 deals much better with banding artifacts. Evangelion 1.11 - You Are (Not) Alone frame - https://diff.pics/PzThT-OrfrcT/1