Windows: chrome://sandbox crashes with M90 binary distribution build

Issue #3109 resolved
Marshall Greenblatt created an issue

What steps will reproduce the problem?

  1. Download and build cef_binary_90.4.3+gc48a797+chromium-90.0.4430.51_windows64_beta.tar.bz2 using VS2019 16.9.3.
  2. Run cefclient.exe --url=chrome://sandbox

What is the expected output? What do you see instead?

The page should load correctly. Instead, it crashes with the following stack trace:

Exception thrown at 0x000000592C4FEC08 in cefclient.exe: 0xC0000005: Access violation executing location 0x000000592C4FEC08
>   libcef.dll!sandbox::policy::`anonymous namespace'::ConvertToValuesAndRespond(std::__1::unique_ptr<sandbox::PolicyList,std::default_delete<sandbox::PolicyList>> policies, base::OnceCallback<void (base::Value)> response) Line 28  C++
    [Inline Frame] libcef.dll!base::internal::FunctorTraits<void (*)(std::unique_ptr<policy::CloudPolicyValidatorBase,std::default_delete<policy::CloudPolicyValidatorBase>>, base::OnceCallback<void ()>),void>::Invoke(void(*)(std::__1::unique_ptr<policy::CloudPolicyValidatorBase,std::default_delete<policy::CloudPolicyValidatorBase>>, base::OnceCallback<void ()>) && function, std::__1::unique_ptr<policy::CloudPolicyValidatorBase,std::default_delete<policy::CloudPolicyValidatorBase>> && args, base::OnceCallback<void ()> && args) Line 393    C++
    [Inline Frame] libcef.dll!base::internal::InvokeHelper<0,void>::MakeItSo(void(*)(std::__1::unique_ptr<policy::CloudPolicyValidatorBase,std::default_delete<policy::CloudPolicyValidatorBase>>, base::OnceCallback<void ()>) && functor, std::__1::unique_ptr<policy::CloudPolicyValidatorBase,std::default_delete<policy::CloudPolicyValidatorBase>> && args, base::OnceCallback<void ()> && args) Line 637 C++
    [Inline Frame] libcef.dll!base::internal::Invoker<base::internal::BindState<void (*)(std::unique_ptr<policy::CloudPolicyValidatorBase,std::default_delete<policy::CloudPolicyValidatorBase>>, base::OnceCallback<void ()>),std::unique_ptr<policy::CloudPolicyValidatorBase,std::default_delete<policy::CloudPolicyValidatorBase>>,base::OnceCallback<void ()>>,void ()>::RunImpl(void(*)(std::__1::unique_ptr<policy::CloudPolicyValidatorBase,std::default_delete<policy::CloudPolicyValidatorBase>>, base::OnceCallback<void ()>) && functor, std::__1::tuple<std::unique_ptr<policy::CloudPolicyValidatorBase,std::default_delete<policy::CloudPolicyValidatorBase>>,base::OnceCallback<void ()>> && bound, std::__1::integer_sequence<unsigned long long,0,1>) Line 710    C++
    libcef.dll!base::internal::Invoker<base::internal::BindState<void (*)(std::unique_ptr<policy::CloudPolicyValidatorBase,std::default_delete<policy::CloudPolicyValidatorBase>>, base::OnceCallback<void ()>),std::unique_ptr<policy::CloudPolicyValidatorBase,std::default_delete<policy::CloudPolicyValidatorBase>>,base::OnceCallback<void ()>>,void ()>::RunOnce(base::internal::BindStateBase * base) Line 679   C++
    [Inline Frame] libcef.dll!base::OnceCallback<void ()>::Run() Line 101   C++
    libcef.dll!base::TaskAnnotator::RunTask(const char * trace_event_name, base::PendingTask * pending_task) Line 163   C++
    libcef.dll!base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl(base::sequence_manager::LazyNow * continuation_lazy_now) Line 352  C++
    libcef.dll!base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() Line 266 C++
    libcef.dll!base::MessagePumpForUI::DoRunLoop() Line 226 C++
    libcef.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate) Line 83    C++
    libcef.dll!base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(bool application_tasks_allowed, base::TimeDelta timeout) Line 463 C++
    libcef.dll!base::RunLoop::Run(const base::Location & location) Line 135 C++
    libcef.dll!CefMainRunner::RunMessageLoop() Line 288 C++
    libcef.dll!CefContext::RunMessageLoop() Line 367    C++
    cefclient.exe!CefRunMessageLoop() Line 135  C++
    cefclient.exe!client::MainMessageLoopStd::Run() Line 15 C++
    cefclient.exe!client::`anonymous namespace'::RunMain(HINSTANCE__ * hInstance, int nCmdShow) Line 112    C++
    cefclient.exe!wWinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, wchar_t * lpCmdLine, int nCmdShow) Line 135    C++

What version of the product are you using? On what operating system?

M90 64-bit build on Windows 10.

Does the problem reproduce with the cefclient or cefsimple sample application at the same version? How about with a newer or older version?

Does not reproduce with cefclient built as part of the CEF/Chromium build (which uses clang). This suggests that it’s an issue with the MSVC build of either cefclient.exe or cef_sandbox.lib, or the combination of clang + MSVC build artifacts.

Official response

  • Marshall Greenblatt reporter

    Windows: Disable iterator debugging for cef_sandbox builds (fixes issue #3109)

    Iterator debugging has been disabled by default (_HAS_ITERATOR_DEBUGGING=0) for CEF/Chromium builds using clang/LLVM since 2018. Inversely, it is enabled by default (_ITERATOR_DEBUG_LEVEL=2) for the MSVC Debug build configuration. In order to minimize configuration-related headaches for MSVC-based clients we have been building the Debug cef_sandbox.lib with iterator debugging enabled. Recently, we have identified a number of crashes that may be due to bugs in current clang/LLVM or MSVC versions but in any case can be resolved by disabling iterator debugging:

    • Crash when loading chrome://sandbox.
    • Crash if an application lists libraries that utilize iterators after cef_sandbox.lib in the linker order.

    To resolve these crashes we will now disable iterator debugging by default for the cef_sandbox.lib builds. Client applications that link cef_sandbox.lib will now also need to build with iterator debugging disabled by setting _HAS_ITERATOR_DEBUGGING=0 or _ITERATOR_DEBUG_LEVEL=0 in their project configuration (this will be done for you if you use cef_variables.cmake from the binary distribution).

    In addition to the crash fixes mentioned above, this change also:

    • Reduces the size of the Debug cef_sandbox.lib by ~10MB.
    • May result in faster Debug executables (see https://crbug.com/539996).

    → <<cset 328de502ac3e>>

Comments (22)

  1. Lee Davis

    We also tried building with clang v11 and same issue occurs. I still don’t understand how the cefclient binary that gets build during the actual CEF build works fine, but doing this after the fact is an issue.

  2. Marshall Greenblatt reporter

    The cef_sandbox.lib included with the binary distribution is built with MSVC libc++ (currently VS2019 16.6.3) and clang as detailed here. It includes symbols from libc++ that may conflict with your local MSVC libc++ version, or the LLVM libc++ version used to build CEF/Chromium.

    Conflicting libc++ symbols is likely the problem in this case because code in libcef.dll is trying to iterate an stl type from cefclient.exe, as shown in symbols_conflict.png. The PolicyDiagnosticList class is compiled into both cefclient.exe (via cef_sandbox.lib) and libcef.dll. When the cefclient.exe version is passed into libcef.dll it crashes due to the conflicting libc++ implementations.

    It’s hard to fix the general problem with conflicting symbols for a few reasons:

    1. We can’t build cef_sandbox.lib with clang because the resulting binary fails to link with MSVC-built apps (results in fatal error LNK1136: invalid or corrupt file, might be related to this LLVM bug).
    2. We can’t mark the symbols as private in cef_sandbox.lib because Windows doesn’t support that.

  3. Marshall Greenblatt reporter

    Looks like BrokerServicesBase::TargetEventsThread (compiled into cefclient.exe via cef_sandbox.lib) is receiving C++ objects from libcef.dll, and calling into them:

    >   libcef.dll!sandbox::policy::ServiceManagerDiagnosticsReceiver::ReceiveDiagnostics(std::__1::unique_ptr<sandbox::PolicyList,std::default_delete<sandbox::PolicyList>> policies) Line 58  C++
        cefclient.exe!sandbox::BrokerServicesBase::TargetEventsThread(void * param) Line 373    C++
    

    That then triggers the async call to ConvertToValuesAndRespond which crashes in the above stack trace.

  4. Lee Davis

    @Marshall Greenblatt Thank you for the explanation and taking the time to walk through this! I’m still a little confused and this will likely reveal my ignorance on this topic.

    1. Why does the cefclient.exe binary produced when I build CEF binary distro (libcef.dll, etc…) work fine?

      1. I’m guessing because everything is built using the exact chromium bundled libc++, including cef_sandbox.lib?
      2. I’m confused by this, as we use the automate_git.py script which I believe builds cef_sandbox separately, not using the same chromium bundled libc++.
    2. When I build cefclient.exe with my local copy of clang or MSVC both fail?

      1. Either build could have conflicting symbols, local LLVM clang or MSVC with that of the CEF build (including cef_sandbox.lib)?
      2. Using the example of building the cef_sandbox.lib separately, detailed here, could we eliminate this problem, or do we still face issues with the disparity between the libcef compilation and our proposed new cef_sandbox build (going to just try this too)?
    3. Does this issue pose risk outside of this “sandbox” test chrome url?

      1. Could I be setting myself up for a similar crash elsewhere just by enabling the sandbox, or are there some specific scenarios I need to avoid?

  5. Marshall Greenblatt reporter

    @ Lee Davis:

    Why does the cefclient.exe binary produced when I build CEF binary distro (libcef.dll, etc…) work fine? I’m guessing because everything is built using the exact chromium bundled libc++, including cef_sandbox.lib?

    Correct.

    I’m confused by this, as we use the automate_git.py script which I believe builds cef_sandbox separately, not using the same chromium bundled libc++.

    The automate_git.py script will build a separate cef_sandbox.lib (with MSVC libc++) for use with the binary distribution. The cefclient.exe generated as part of the CEF/Chromium build uses a different version of cef_sandbox.lib (with LLVM libc++). We can’t include the LLVM version of cef_sandbox.lib with the binary distribution for the compatibility reasons mentioned above.

    When I build cefclient.exe with my local copy of clang or MSVC both fail?

    It will only succeed reliably if everything (libcef.dll, cef_sandbox.lib, cefclient.exe) is built with the same clang/LLVM version. Practically, that means building everything with the clang/LLVM version bundled by Chromium.

    Does this issue pose risk outside of this “sandbox” test chrome url?

    I don’t think so. Loading chrome://sandbox is the only sandbox-related problem that I’m aware currently, and I wouldn’t classify that as core/required sandbox functionality.

  6. Marshall Greenblatt reporter

    @Cristian Amarie It won’t crash if everything (exe, libcef, cef_sandbox) is built with the same compiler. Are you using an “official” cef_sandbox build as described here and linking to an MSVC exe?

  7. Cristian Amarie

    Everything is built from automate-git.py using a batch script (using VS2017), so no, my mistake; in my case everything is built with the same compiler.

  8. Marshall Greenblatt reporter

    This appears to only be an issue with Debug builds of cef_sandbox.lib. Disabling iterator debugging for both cef_sandbox.lib and the MSVC Debug configuration fixes the problem for me (tested on x86 and x64).

    Note that this change may also have performance benefits as documented in https://crbug.com/539996.

    It also shrinks the size of cef_sandbox.lib (Debug build) by about 10MB.

  9. Marshall Greenblatt reporter

    I’m not able to find any reported clang bugs about MSVC iterator debugging incompatibilities. However, I think that disabling iterator debugging is generally a positive change (see above perf and size benefits) so we’ll proceed with that starting in M90.

  10. Marshall Greenblatt reporter

    To clarify the above comment about “clang bugs”:

    The chrome://sandbox crash is due to cef_sandbox.lib building with iterator debugging enabled while libcef builds with iterator debugging disabled (default since 2018), resulting in incompatible stl containers being passed across the DLL boundary by BrokerServicesBase::TargetEventsThread. The reason why this only started crashing with M89 may be due to a specific change in the Chromium sandbox code (currently unidentified) and not a clang bug specifically.

    Unrelated to that particular issue, we’ve also seen problems where linker order has resulted in iterator-related crashes in other MSVC libraries built as part of client executables. For example, linking cef_sandbox.lib (with iterators enabled) may result in crashes in another linked MSVC-built library that uses stl iterators, if that other library is included after cef_sandbox.lib in the link order. Moving cef_sandbox.lib to the end of the link order resolved those particular crashes, but it was necessary to rebuild all other MSVC-built libs after making that change. This suggests that stl code from cef_sandbox.lib (xmemory, xstring, etc. built using clang/LLVM) was being incorporated into the MSVC-built libs, and that the clang/LLVM code was incompatible with the MSVC-built libraries (resulting in the aforementioned crashes during runtime). These types of crashes became substantially more frequent beginning with M90, so we suspect that there may be a recent change in clang/LLVM or MSVC contributing to the issue. Disabling iterator debugging appears to also resolve these crashes.

  11. Marshall Greenblatt reporter

    Windows: Disable iterator debugging for cef_sandbox builds (fixes issue #3109)

    Iterator debugging has been disabled by default (_HAS_ITERATOR_DEBUGGING=0) for CEF/Chromium builds using clang/LLVM since 2018. Inversely, it is enabled by default (_ITERATOR_DEBUG_LEVEL=2) for the MSVC Debug build configuration. In order to minimize configuration-related headaches for MSVC-based clients we have been building the Debug cef_sandbox.lib with iterator debugging enabled. Recently, we have identified a number of crashes that may be due to bugs in current clang/LLVM or MSVC versions but in any case can be resolved by disabling iterator debugging:

    • Crash when loading chrome://sandbox.
    • Crash if an application lists libraries that utilize iterators after cef_sandbox.lib in the linker order.

    To resolve these crashes we will now disable iterator debugging by default for the cef_sandbox.lib builds. Client applications that link cef_sandbox.lib will now also need to build with iterator debugging disabled by setting _HAS_ITERATOR_DEBUGGING=0 or _ITERATOR_DEBUG_LEVEL=0 in their project configuration (this will be done for you if you use cef_variables.cmake from the binary distribution).

    In addition to the crash fixes mentioned above, this change also:

    • Reduces the size of the Debug cef_sandbox.lib by ~10MB.
    • May result in faster Debug executables (see https://crbug.com/539996).

    → <<cset 328de502ac3e>>

  12. Marshall Greenblatt reporter

    Windows: Disable iterator debugging for cef_sandbox builds (fixes issue #3109)

    Iterator debugging has been disabled by default (_HAS_ITERATOR_DEBUGGING=0) for CEF/Chromium builds using clang/LLVM since 2018. Inversely, it is enabled by default (_ITERATOR_DEBUG_LEVEL=2) for the MSVC Debug build configuration. In order to minimize configuration-related headaches for MSVC-based clients we have been building the Debug cef_sandbox.lib with iterator debugging enabled. Recently, we have identified a number of crashes that may be due to bugs in current clang/LLVM or MSVC versions but in any case can be resolved by disabling iterator debugging:

    • Crash when loading chrome://sandbox.
    • Crash if an application lists libraries that utilize iterators after cef_sandbox.lib in the linker order.

    To resolve these crashes we will now disable iterator debugging by default for the cef_sandbox.lib builds. Client applications that link cef_sandbox.lib will now also need to build with iterator debugging disabled by setting _HAS_ITERATOR_DEBUGGING=0 or _ITERATOR_DEBUG_LEVEL=0 in their project configuration (this will be done for you if you use cef_variables.cmake from the binary distribution).

    In addition to the crash fixes mentioned above, this change also:

    • Reduces the size of the Debug cef_sandbox.lib by ~10MB.
    • May result in faster Debug executables (see https://crbug.com/539996).

    → <<cset f2f0bfbdb5a4>>

  13. Marshall Greenblatt reporter

    Windows: Disable iterator debugging for cef_sandbox builds (fixes issue #3109)

    Iterator debugging has been disabled by default (_HAS_ITERATOR_DEBUGGING=0) for CEF/Chromium builds using clang/LLVM since 2018. Inversely, it is enabled by default (_ITERATOR_DEBUG_LEVEL=2) for the MSVC Debug build configuration. In order to minimize configuration-related headaches for MSVC-based clients we have been building the Debug cef_sandbox.lib with iterator debugging enabled. Recently, we have identified a number of crashes that may be due to bugs in current clang/LLVM or MSVC versions but in any case can be resolved by disabling iterator debugging:

    • Crash when loading chrome://sandbox.
    • Crash if an application lists libraries that utilize iterators after cef_sandbox.lib in the linker order.

    To resolve these crashes we will now disable iterator debugging by default for the cef_sandbox.lib builds. Client applications that link cef_sandbox.lib will now also need to build with iterator debugging disabled by setting _HAS_ITERATOR_DEBUGGING=0 or _ITERATOR_DEBUG_LEVEL=0 in their project configuration (this will be done for you if you use cef_variables.cmake from the binary distribution).

    In addition to the crash fixes mentioned above, this change also:

    • Reduces the size of the Debug cef_sandbox.lib by ~10MB.
    • May result in faster Debug executables (see https://crbug.com/539996).

    → <<cset a149cf128f0b>>

  14. Lee Davis

    This occurs with our release builds as well. I didn’t think we were doing anything special here, but is there anything we should be looking for specifically with Release? I did try the default CefClient and sure enough Release was fine.

  15. Marshall Greenblatt reporter

    @Lee Davis Does your Release configuration match the CMake configuration from the CEF binary distribution?

  16. Lee Davis

    @Marshall Greenblatt

    I retested this with a more recent release build and it does NOT crash, sorry about the confusion. And, yes, our release configuration is very similar.

    However, now I need to address my other deps like boost that are running into a mismatch on iterator debug level.

  17. Lee Davis

    This change works great, thanks for getting to the bottom of this! The only thing I would ask is the use of _HAS_ITERATOR_DEBUGGING macro over _ITERATOR_DEBUG_LEVEL, it seems the former was deprecated.

  18. Log in to comment