Integer underflow in x265/source/common/x86/pixel-a.asm

[Wei Wu]

1.Summary

# A integer underflow vulnerability(CWE-191) exists in pixel-a.asm, the x86 assembly code for planeClipAndMax() in x265 release code up to 2.4. A small picture can cause an integer underflow which leads to a Denial of Service in the process of encoding. This vulnerability can be triggerd by calling x265_encoder_encode() via another library, e.g. libbpg.

2.vulnerability information

#The vulnerability is found while fuzzing a library which is built on x265: libbpg. x265 is used in libbpg to encode image, by calling x265_encoder_encode() in x265. planeClipAndMax in x265/source/common/x86/pixel-a.asm does not handle image of small size properly which may result in a integer underflow. Any library and applications built on x265 might be affected by this vulnerability.

3.vulnerability detail

#The vulnerability exists in the newest x265 release 2.4. The assembly code for planeClipAndMax_avx2 in x265/source/common/x86/pixel-a.asm(line 13732) does not properly handle a image with width less than 24 pixels which leads to a integer underflow and then leads to a out-of-bounds read. the related code listed below:

cglobal planeClipAndMax, 5,7,8
    movd            xm0, r5m
    vpbroadcastb    m0, xm0                 ; m0 = [min]
    vpbroadcastb    m1, r6m                 ; m1 = [max]
    pxor            m2, m2                  ; m2 = sumLuma
    pxor            m3, m3                  ; m3 = maxLumaLevel
    pxor            m4, m4                  ; m4 = zero

    ; get mask to partial register pixels
    mov             r5d, r2d
    and             r2d, ~(mmsize - 1)
    sub             r5d, r2d
    lea             r6, [pb_movemask_32 + mmsize]
    sub             r6, r5
    movu            m5, [r6]                ; m5 = mask for last couple column

.loopH:
    lea             r5d, [r2 - mmsize]      ;(vuls here, integer underflow if r2 < mmsize)

.loopW:
    movu            m6, [r0 + r5]           ;(vuls here, out of bounds read if integer underflow above happend)
    pmaxub          m6, m0
    pminub          m6, m1
    movu            [r0 + r5], m6           ; store back
    pmaxub          m3, m6                  ; update maxLumaLevel
    psadbw          m6, m4
    paddq           m2, m6

    sub             r5d, mmsize
    jge            .loopW
    ...

4.Test and crash reproduce

#To trigger this vulnerability, libbpg is used to call x265_encoder_encode()

The test is performed on Ubuntu(16.04 64bit),with gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609 and bpgenc in libbpg(0.9.7).

reproduce:

sudo apt-get install -y libsdl-image1.2-dev libsdl1.2-dev libjpeg8-dev emscripten
sudo apt-get install --force-yes -y yasm
wget https://bellard.org/bpg/libbpg-0.9.7.tar.gz
tar xvf libbpg-0.9.7.tar.gz
cd libbpg-0.9.7
echo 'modifying the Makefile according to https://askubuntu.com/questions/869442/cant-compile-libbpg-0-9-7-for-bpg-better-portable-graphics-on-ubuntu-14-04'
make
./bpgenc -o test.bpg /path/to/PoC

5.The crash: GDB information

#

[-------------------------------------regs-------------------------------------]
RAX: 0x10dc1b8 --> 0x0
RBX: 0x8
RCX: 0x8
RDX: 0x0
RSI: 0x100
RDI: 0x13f04c0 --> 0xbebebebebebebebe
RBP: 0x7fffffffddf0 --> 0x13e7aa0 --> 0x0
RSP: 0x7fffffffdd48 --> 0x574376 (<_ZN4x2656PicYuv15copyFromPictureERK12x265_pictureRK10x265_paramii+342>:    )
RIP: 0x5c91d8 (<x265_planeClipAndMax_avx2+56>:    vmovdqu ymm6,YMMWORD PTR [rdi+r9*1])
R8 : 0x7fffffffddb0 --> 0x3000 ('')
R9 : 0xffffffe0
R10: 0x9 ('\t')
R11: 0x7ffff6d562b0 (<__memcpy_avx_unaligned>:    mov    rax,rdi)
R12: 0x136ab80 --> 0x13eb460 --> 0x0
R13: 0x13f04c0 --> 0xbebebebebebebebe
R14: 0x0
R15: 0x0
EFLAGS: 0x10216 (carry PARITY ADJUST zero sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
   0x5c91cd <x265_planeClipAndMax_avx2+45>:    sub    rax,r9
   0x5c91d0 <x265_planeClipAndMax_avx2+48>:    vmovdqu ymm5,YMMWORD PTR [rax]
   0x5c91d4 <x265_planeClipAndMax_avx2+52>:    lea    r9d,[rdx-0x20]
=> 0x5c91d8 <x265_planeClipAndMax_avx2+56>:    vmovdqu ymm6,YMMWORD PTR [rdi+r9*1]
   0x5c91de <x265_planeClipAndMax_avx2+62>:    vpmaxub ymm6,ymm6,ymm0
   0x5c91e2 <x265_planeClipAndMax_avx2+66>:    vpminub ymm6,ymm6,ymm1
   0x5c91e6 <x265_planeClipAndMax_avx2+70>:    vmovdqu YMMWORD PTR [rdi+r9*1],ymm6
   0x5c91ec <x265_planeClipAndMax_avx2+76>:    vpmaxub ymm3,ymm3,ymm6
[------------------------------------stack-------------------------------------]
0000| 0x7fffffffdd48 --> 0x574376 (<_ZN4x2656PicYuv15copyFromPictureERK12x265_pictureRK10x265_paramii+342>:    )
0008| 0x7fffffffdd50 --> 0xff
0016| 0x7fffffffdd58 --> 0x574b56 (<_ZN4x2656PicYuv15copyFromPictureERK12x265_pictureRK10x265_paramii+2358>:    )
0024| 0x7fffffffdd60 --> 0x1
0032| 0x7fffffffdd68 --> 0x0
0040| 0x7fffffffdd70 --> 0x0
0048| 0x7fffffffdd78 --> 0x0
0056| 0x7fffffffdd80 --> 0x1363f40 --> 0x10011e7ff
[------------------------------------------------------------------------------]
Stopped reason: SIGSEGV
(gdb) bt
================================================================================
#0  0x00000000005c91d8 in x265_planeClipAndMax_avx2 ()
#1  0x0000000000574376 in x265::PicYuv::copyFromPicture(x265_picture const&, x265_param const&, int, int) ()
#2  0x000000000055685e in x265::Encoder::encode(x265_picture const*, x265_picture*) ()
#3  0x000000000040871e in x265_encoder_encode ()
#4  0x0000000000407dd4 in x265_encode (s=0x136a920, img=<optimized out>)
    at x265_glue.c:168
#5  0x0000000000407c01 in bpg_encoder_encode (s=s@entry=0x1363e90,
    img=img@entry=0x136a8c0, write_func=write_func@entry=0x40564f <my_write_func>,
    opaque=opaque@entry=0x1363c60) at bpgenc.c:2565
#6  0x0000000000403a9b in main (argc=argc@entry=0x4, argv=argv@entry=0x7fffffffe588)
    at bpgenc.c:2956
#7  0x00007ffff6c29830 in __libc_start_main (main=0x403320 <main>, argc=0x4,
    argv=0x7fffffffe588, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7fffffffe578) at ../csu/libc-start.c:291
#8  0x00000000004040e9 in _start ()
================================================================================

5.patch

#To patch this vulnerability, range checks on width should be added to planeClipAndMax_avx2 in ./source/common/x86/pixel-a.asm

6. PoC

#for the PoC, please contact [e-mail]

7. Credits

#This vulnerability was discoverd and researched by Wei WU(ww9210) @ VARAS of IIE.

[Pradeep Ramachandran]

Thank you very much for your detailed description of the issue, and a suggestion of how to fix it. Given that this is a security vulnerability, we will act immediately to address this asap!

[Min Chen]

Thank you report this issue, the read beyond is not a big problem.

Of course, this is hidden problem. Unfornately, x86 instruction sets does not support read part of memory with mask.

So I suggest a workaround: we can simple disable AVX2 planeClipAndMax() in small video/image since assembly can't get more performance in that case.

@Praveen could you please additional some code/function in x265_setup_primitives() to disable AVX2 on planeClipAndMax() in small video/image case?

[Praveen Tiwari]

We have already disabled 'planeClipAndMax' assembly primitives as it need to be fixed, this is what asm-primitive.cpp, line number - 2160 says -

    /* TODO: This kernel needs to be modified to work with HIGH_BIT_DEPTH only 
    p.planeClipAndMax = PFX(planeClipAndMax_avx2); */

Yes, for functional correctness it expects width to be minimum 32.

[Wei Wu]

Fixed

[Pradeep Ramachandran]

Vulnerability doesn't exist as relevant asm code is never enabled.

[Wei Wu]

Thank you for the explanation. I did not notice the assembly code is not used in current release, so the vulnerability can not be triggered in current x265 release, it is triggered in libbpg because libbpg uses an out-of-data version of x265.