Crash during shutdown due to NotificationService::current returning NULL

Issue #2316 wontfix
Tameem Ahmed created an issue

There's a class called "NotificationService" in libcef.dll. An instance of this can be accessed by "libcef!content::NotificationService::current", which goes to thread local storage (TLS) to retrieve an object that contains a reference to it. By setting a memory access breakpoint on the address retrieved from TLS, it can be seen that this referencing object is freed at the beginning of the un-initialization of libcef.dll (the address is 1ddb25f0):

Breakpoint 0 hit
eax=1ddb25f0 ebx=1ddb25e8 ecx=25b8cfc8 edx=25aed500 esi=25aed500 edi=1ddb0000
eip=773a2379 esp=0018cdf8 ebp=0018cef8 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00200246
ntdll!RtlpFreeHeap+0x5bb:
773a2379 894804          mov     dword ptr [eax+4],ecx ds:002b:1ddb25f4=00000000
*** WARNING: Unable to verify checksum for C:\Program Files (x86)\Vizrt\Viz Trio\libcef.dll
0:000> kv
ChildEBP RetAddr  Args to Child              
0018cef8 773a1d8a 1ddb25e8 1ddb25f0 0018d7a8 ntdll!RtlpFreeHeap+0x5bb (FPO: [Non-Fpo])
0018cf4c 1fc803ff 1ddb0000 00000000 1ddb25f0 ntdll!RtlFreeHeap+0x1b6 (FPO: [Non-Fpo])
0018cf60 1fc80393 1ddb25f0 0018cf7c 1fc8025a libcef!base::allocator::WinHeapFree+0x1a (FPO: [Non-Fpo]) (CONV: cdecl) [y:\work\cef3_git\chromium\src\base\allocator\winheap_stubs_win.cc @ 43]
0018cf6c 1fc8025a 22574228 1ddb25f0 0018e3b0 libcef!`anonymous namespace'::DefaultWinHeapFreeImpl+0xb (FPO: [Non-Fpo]) (CONV: cdecl) [y:\work\cef3_git\chromium\src\base\allocator\allocator_shim_default_dispatch_to_winheap.cc @ 47]
0018cf7c 1ef3247b 1ddb25f0 1ef8a810 0018e3cc libcef!ShimFree+0xf (FPO: [Non-Fpo]) (CONV: cdecl) [y:\work\cef3_git\chromium\src\base\allocator\allocator_shim.cc @ 241]
0018e3b0 1ef8a83b 0018e3d8 773a96de 1e7e0000 libcef!`anonymous namespace'::OnThreadExitInternal+0x4b (FPO: [Non-Fpo]) (CONV: cdecl) [y:\work\cef3_git\chromium\src\base\threading\thread_local_storage.cc @ 185]
0018e3b8 773a96de 1e7e0000 00000000 00000000 libcef!OnThreadExit+0x2b (FPO: [Non-Fpo]) (CONV: stdcall) [y:\work\cef3_git\chromium\src\base\threading\thread_local_storage_win.cc @ 71]
0018e3d8 773a9658 1ef8a810 1e7e0000 00000000 ntdll!LdrxCallInitRoutine+0x16
0018e428 773fd625 00000000 00000000 e212fdf5 ntdll!LdrpCallInitRoutine+0x43 (FPO: [SEH])
0018e468 7737fc8f e212fc9d 003e0e69 00000000 ntdll!LdrpCallTlsInitializers+0x7de35
0018e500 773cce71 004fce9c 00000000 727f53a0 ntdll!LdrShutdownProcess+0x14c (FPO: [SEH])
0018e5c8 76d99862 00000000 77e8f3b0 ffffffff ntdll!RtlExitUserProcess+0x81 (FPO: [Non-Fpo])
0018e5dc 0128fe0e 00000000 003e0e69 004fce9c KERNEL32!ExitProcessImplementation+0x12 (FPO: [1,0,0])

Soon afterwards the notification service is attempted used from some shutdown code called by DLLMain, resulting in an access violation:

(3828.4614): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000000 ebx=1de53d58 ecx=0018def0 edx=25a84178 esi=1de53cf8 edi=225d093c
eip=20c21df7 esp=0018dee8 ebp=0018def8 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00210246
libcef!Profile::MaybeSendDestroyedNotification+0x2c:
20c21df7 8b10            mov     edx,dword ptr [eax]  ds:002b:00000000=????????
0:000> kv
ChildEBP RetAddr  Args to Child              
0018def8 202c72e9 1de53cf8 1de5f808 1de53cf8 libcef!Profile::MaybeSendDestroyedNotification+0x2c (FPO: [Non-Fpo]) (CONV: thiscall) [y:\work\cef3_git\chromium\src\chrome\browser\profiles\profile.cc @ 232]
0018df14 202d1011 225a77e0 1de53cf8 00000006 libcef!CefBrowserContext::Shutdown+0xe (FPO: [Non-Fpo]) (CONV: thiscall) [y:\work\cef3_git\chromium\src\cef\libcef\browser\browser_context.cc @ 72]
0018df2c 202d1157 1ddf7538 0018e118 202e9ea1 libcef!CefBrowserContextImpl::~CefBrowserContextImpl+0x6c (FPO: [Non-Fpo]) (CONV: thiscall) [y:\work\cef3_git\chromium\src\cef\libcef\browser\browser_context_impl.cc @ 221]
0018df38 202e9ea1 00000001 1de05d70 1feebe99 libcef!CefBrowserContextImpl::`scalar deleting destructor'+0xb (FPO: [Non-Fpo]) (CONV: thiscall)
0018df44 1feebe99 1ddf7278 225a77e0 00000000 libcef!CefBrowserMainParts::PostMainMessageLoopRun+0x1a (FPO: [0,0,4]) (CONV: thiscall) [y:\work\cef3_git\chromium\src\cef\libcef\browser\browser_main.cc @ 212]
0018e118 1feecef1 1ddb4758 1ddb22b0 1ddb479c libcef!content::BrowserMainLoop::ShutdownThreadsAndCleanUp+0x14b (FPO: [Non-Fpo]) (CONV: thiscall) [y:\work\cef3_git\chromium\src\content\browser\browser_main_loop.cc @ 1216]
0018e1c4 202c0b75 1ddb2168 2028a13e 1ddb2168 libcef!content::BrowserMainRunnerImpl::Shutdown+0x1f2 (FPO: [Non-Fpo]) (CONV: thiscall) [y:\work\cef3_git\chromium\src\content\browser\browser_main_runner.cc @ 203]
0018e1cc 2028a13e 1ddb2168 2028a859 21b2f546 libcef!CefMainDelegate::ShutdownBrowser+0xf (FPO: [0,0,4]) (CONV: thiscall) [y:\work\cef3_git\chromium\src\cef\libcef\common\main_delegate.cc @ 619]
0018e1d4 2028a859 21b2f546 1fc8025a ffffffff libcef!CefContext::FinalizeShutdown+0x23 (FPO: [0,0,4]) (CONV: thiscall) [y:\work\cef3_git\chromium\src\cef\libcef\browser\context.cc @ 509]
0018e204 21b2f556 21b2f546 211ca94f b26fb33d libcef!CefContext::Shutdown+0xa6 (FPO: [Non-Fpo]) (CONV: thiscall) [y:\work\cef3_git\chromium\src\cef\libcef\browser\context.cc @ 426]
0018e20c 211ca94f b26fb33d 0018e33c 00000000 libcef!`anonymous namespace'::`dynamic atexit destructor for 'g_force_shutdown''+0x10 (FPO: [0,0,4]) (CONV: cdecl)
0018e23c 211ca6ba b277514d b26fb33d 0018e33c libcef!<lambda_f03950bc5685219e0bcd2087efbe011e>::operator()+0x89 (FPO: [Non-Fpo]) (CONV: thiscall) [d:\rs1\minkernel\crts\ucrt\src\appcrt\startup\onexit.cpp @ 206]
0018e270 211ca781 0018e284 0018e2a0 0018e288 libcef!__crt_seh_guarded_call<int>::operator()<<lambda_69a2805e680e0e292e8ba93315fe43a8>,<lambda_f03950bc5685219e0bcd2087efbe011e> &,<lambda_03fcd07e894ec930e3f35da366ca99d6> >+0x27 (FPO: [Non-Fpo]) (CONV: thiscall) [d:\rs1\minkernel\crts\ucrt\devdiv\vcruntime\inc\internal_shared.h @ 204]
0018e290 211caa11 00000002 0018e2a0 0018e2ac libcef!__acrt_lock_and_call<<lambda_f03950bc5685219e0bcd2087efbe011e> >+0x24 (FPO: [Non-Fpo]) (CONV: cdecl) [d:\rs1\minkernel\crts\ucrt\inc\corecrt_internal.h @ 912]
0018e2a4 211b7596 225e7a38 b27751e1 00000000 libcef!_execute_onexit_table+0x17 (FPO: [Non-Fpo]) (CONV: cdecl) [d:\rs1\minkernel\crts\ucrt\src\appcrt\startup\onexit.cpp @ 160]
0018e2dc 211b74d5 b2775031 00000000 00000001 libcef!<lambda_6e4b09c48022b2350581041d5f6b0c4c>::operator()+0x79 (FPO: [Non-Fpo]) (CONV: thiscall) [d:\rs1\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 215]
0018e30c 211b7519 0018e320 0018e33c 0018e324 libcef!__crt_seh_guarded_call<void>::operator()<<lambda_03b1d95aef87969028cfba75ccab2455>,<lambda_6e4b09c48022b2350581041d5f6b0c4c> &,<lambda_22bdf7517842c4b3e53723af5aa32b9e> >+0x23 (FPO: [Non-Fpo]) (CONV: thiscall) [d:\rs1\minkernel\crts\ucrt\devdiv\vcruntime\inc\internal_shared.h @ 225]
0018e32c 211b764d 00000002 0018e33c 0018e358 libcef!__acrt_lock_and_call<<lambda_6e4b09c48022b2350581041d5f6b0c4c> >+0x24 (FPO: [Non-Fpo]) (CONV: cdecl) [d:\rs1\minkernel\crts\ucrt\inc\corecrt_internal.h @ 912]
0018e34c 211b776c 00000000 00000000 00000001 libcef!common_exit+0x41 (FPO: [Non-Fpo]) (CONV: cdecl) [d:\rs1\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 263]
0018e35c 211a10c4 b27750ad 00000000 00000001 libcef!_cexit+0xb (FPO: [0,0,0]) (CONV: cdecl) [d:\rs1\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 298]
0018e390 211a0f6d 00000001 0018e3dc 211a11b0 libcef!dllmain_crt_process_detach+0x40 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp @ 109]
0018e39c 211a11b0 1e7e0000 00000000 00000001 libcef!dllmain_crt_dispatch+0x4e (FPO: [Non-Fpo]) (CONV: stdcall) [f:\dd\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp @ 134]
0018e3dc 211a1243 1e7e0000 00000000 00000001 libcef!dllmain_dispatch+0xaf (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp @ 207]
0018e3f0 773a96de 1e7e0000 00000000 00000001 libcef!_DllMainCRTStartup+0x1c (FPO: [Non-Fpo]) (CONV: stdcall) [f:\dd\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp @ 248]
0018e410 773a9658 211a1227 1e7e0000 00000000 ntdll!LdrxCallInitRoutine+0x16
0018e460 773ccdc8 00000000 00000001 e212fc9d ntdll!LdrpCallInitRoutine+0x43 (FPO: [SEH])
0018e500 773cce71 004fce9c 00000000 727f53a0 ntdll!LdrShutdownProcess+0x101 (FPO: [SEH])
0018e5c8 76d99862 00000000 77e8f3b0 ffffffff ntdll!RtlExitUserProcess+0x81 (FPO: [Non-Fpo])
0018e5dc 0128fe0e 00000000 003e0e69 004fce9c KERNEL32!ExitProcessImplementation+0x12 (FPO: [1,0,0])
0:000> uf libcef!Profile::MaybeSendDestroyedNotification
libcef!Profile::MaybeSendDestroyedNotification [y:\work\cef3_git\chromium\src\chrome\browser\profiles\profile.cc @ 227]:
  227 20c21dcb 55              push    ebp
  227 20c21dcc 8bec            mov     ebp,esp
  227 20c21dce 51              push    ecx
  227 20c21dcf 51              push    ecx
  227 20c21dd0 56              push    esi
  227 20c21dd1 8bf1            mov     esi,ecx
  228 20c21dd3 807e1100        cmp     byte ptr [esi+11h],0
  228 20c21dd7 7528            jne     libcef!Profile::MaybeSendDestroyedNotification+0x36 (20c21e01)

libcef!Profile::MaybeSendDestroyedNotification+0xe [y:\work\cef3_git\chromium\src\chrome\browser\profiles\profile.cc @ 231]:
  231 20c21dd9 56              push    esi
  231 20c21dda c6461101        mov     byte ptr [esi+11h],1
  231 20c21dde e8eb672cff      call    libcef!content::BrowserContext::NotifyWillBeDestroyed (1fee85ce)
  231 20c21de3 59              pop     ecx
  232 20c21de4 e888543dff      call    libcef!content::NotificationService::current (1fff7271)
  232 20c21de9 8365fc00        and     dword ptr [ebp-4],0
  232 20c21ded 8d4dfc          lea     ecx,[ebp-4]
  232 20c21df0 51              push    ecx
  232 20c21df1 8975f8          mov     dword ptr [ebp-8],esi
  232 20c21df4 8d4df8          lea     ecx,[ebp-8]
  232 20c21df7 8b10            mov     edx,dword ptr [eax]
  232 20c21df9 51              push    ecx
  232 20c21dfa 6a59            push    59h
  232 20c21dfc 8bc8            mov     ecx,eax
  232 20c21dfe ff5204          call    dword ptr [edx+4]

libcef!Profile::MaybeSendDestroyedNotification+0x36 [y:\work\cef3_git\chromium\src\chrome\browser\profiles\profile.cc @ 232]:
  232 20c21e01 5e              pop     esi
  237 20c21e02 8be5            mov     esp,ebp
  237 20c21e04 5d              pop     ebp
  237 20c21e05 c3              ret

This problem can be worked around by shutting down CEF explicitly before the process ends. However, in the context of an ActiveX, or any other kind of embedded plugin that can be used multiple times in a single process, this is not an option, as the plugin doesn't know if it is going to be needed again or not.

Version information:

0:000> lmvm libcef
start    end        module name
1e7e0000 228f3000   libcef   C (private pdb symbols)  ...\libcef.dll.pdb
    Loaded symbol image file: ...\libcef.dll
    Image path: ...\libcef.dll
    Image name: libcef.dll
    Timestamp:        Fri Apr 07 08:01:06 2017 (58E72B22)
    CheckSum:         00000000
    ImageSize:        04113000
    File version:     3.2987.1601.0
    Product version:  3.2987.1601.0
    File flags:       0 (Mask 17)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    ProductName:      Chromium Embedded Framework (CEF) Dynamic Link Library
    InternalName:     libcef
    OriginalFilename: libcef.dll
    ProductVersion:   3.2987.1601.gf035232
    FileVersion:      3.2987.1601.gf035232
    FileDescription:  Chromium Embedded Framework (CEF) Dynamic Link Library
    LegalCopyright:   Copyright (C) 2017 The Chromium Embedded Framework Authors

Operating System:

Windows 10.

Comments (8)

  1. Marshall Greenblatt
    • changed status to open

    This crash can also occur if some tasks that bind a CefBrowser are in-flight during CefShutdown, resulting in the CefBrowserContext being destroyed after the NotificationService. The call stack looks like this:

    Exception thrown at 0x105DA4FB (libcef.dll) in cefclient.exe: 0xC0000005: Access violation reading location 0x00000000.
    
    >   libcef.dll!content::RenderProcessHostImpl::Cleanup() Line 3464  C++
        libcef.dll!content::RenderProcessHostImpl::DisableKeepAliveRefCount() Line 2444 C++
        libcef.dll!content::BrowserContext::NotifyWillBeDestroyed(content::BrowserContext * browser_context) Line 560   C++
        libcef.dll!Profile::MaybeSendDestroyedNotification() Line 307   C++
        libcef.dll!CefBrowserContext::~CefBrowserContext() Line 222 C++
        libcef.dll!CefBrowserContext::~CefBrowserContext() Line 210 C++
        libcef.dll!CefBrowserContext::RemoveCefRequestContext(CefRequestContextImpl * context) Line 379 C++
        libcef.dll!CefRequestContextImpl::~CefRequestContextImpl() Line 205 C++
        [External Code] 
        libcef.dll!content::BrowserThread::DeleteOnThread<content::BrowserThread::UI>::Destruct<CefImageImpl>(const CefImageImpl * x) Line 148  C++
        [External Code] 
        libcef.dll!base::internal::BindState<void (CefBrowserHostImpl::*)(double) __attribute__((thiscall)),scoped_refptr<CefBrowserHostImpl>,double>::Destroy(const base::internal::BindStateBase * self) Line 857 C++
        libcef.dll!base::internal::CallbackBase::~CallbackBase() Line 84    C++
        libcef.dll!base::sequence_manager::internal::LazilyDeallocatedDeque<base::sequence_manager::Task,&base::subtle::TimeTicksNowIgnoringOverride>::~LazilyDeallocatedDeque() Line 55    C++
        libcef.dll!base::sequence_manager::internal::TaskQueueImpl::UnregisterTaskQueue() Line 196  C++
        libcef.dll!base::sequence_manager::internal::SequenceManagerImpl::~SequenceManagerImpl() Line 182   C++
        libcef.dll!base::sequence_manager::internal::SequenceManagerImpl::~SequenceManagerImpl() Line 169   C++
        libcef.dll!content::BrowserUIThreadScheduler::Shutdown() Line 51    C++
        libcef.dll!content::ContentMainRunnerImpl::Shutdown() Line 1001 C++
        libcef.dll!CefContext::FinalizeShutdown() Line 633  C++
        libcef.dll!CefShutdown() Line 272   C++
    
  2. Marshall Greenblatt

    @Teun Pronk The workaround for me was to not post tasks during shutdown that hold CefBrowser references.

  3. Log in to comment