Override paths: DIR_EXE = DIR_MODULE = <libcef.so.path> on Linux

Issue #1936 resolved
Czarek Tomczak created an issue

There is a problem with loading "icudtl.dat" and "natives_blob.bin" on Linux when executable is in a different directory than "libcef.so". In previous versions of CEF (branch 1650 for sure) these ICU/V8 files were looked for in the directory where libcef.so resided. In newer branches the code in Chromium changed and now it looks for these files in the dir where executable resides - using DIR_EXE path service key. This is a serious issue eg. in CEF Python because the python executable will almost always be in /usr/bin.

This issue was described in greater details on the CEF Forum [1] and in the CEF Python Issue Tracker [2]. I've already created a patch to expose CefOverridePath function to override arbitrary paths [3], but Marshall suggests it shouldn't be allowed to do this, as this is sensitive task and many things could go wrong if timing is wrong. Overriding DIR_EXE = DIR_MODULE = <libcef.so path> fixes the issue completely, so we should only do this.

Paths should be overriden during a call to CefExecuteProcess [libcef/context/browser.cc], as one of the first things this function does. In CEF Python I've overriden paths before calling CefExecuteProcess and only in the browser process, and that was enough to fix the issue. But now that I think of it again, paths should be overriden for all the CEF processes.

I will work on this patch soon, when I find time.

[1] Related topic on the CEF Forum: https://www.magpcss.org/ceforum/viewtopic.php?f=6&t=14271

[2] Issue explained in greater details: https://github.com/cztomczak/cefpython/issues/231#issuecomment-228781557

[3] Original CefOverridePath patch: https://github.com/cztomczak/cefpython/blob/0cc124e24a4852c4e74136d4413654dd2a7d7b44/patches/issue231.patch

Comments (39)

  1. Marshall Greenblatt

    We should only need to override DIR_EXE on Linux. DIR_MODULE should already be correct. On Windows Chromium looks for icudtl.dat in DIR_MODULE [1], and on OS X it looks in the app bundle.

    [1] https://cs.chromium.org/chromium/src/base/i18n/icu_util.cc?q=LazyInitIcuDataFile&sq=package:chromium&l=85&dr=CSs

    The override will need to be set in CefContext::Initialize before the call to |main_runner_->Initialize|. We may also need to set it from CefExecuteProcess for sub-processes, but I'm not sure if that's necessary.

  2. Czarek Tomczak reporter

    My testing environment is CEF Python which uses a separate subprocess executable which lies in the same directory as libcef.so. It makes me wonder whether overriding paths would affect other scenarios eg. when both the browser process and subprocess code paths are in the same executable. What if app.exe is placed in "/opt/app/" and lib/resources are placed in "/usr/local/lib/app/" and application expects that the debug.log file will be created in "/opt/app/"? If we override DIR_EXE and DIR_MODULE then this will affect placement of the log file. In main_delegate.cc there is function GetDefaultLogFile() which uses DIR_EXE. There may be more issues that I can't think of now. Maybe overriding these paths should be an option. Maybe add an additional bool argument to CefExecuteProcess that when set True will override these paths?

  3. Marshall Greenblatt

    The user can already override the log file location via CefSettings.log_file. I think it's OK to expect resource files to be in the same directory as libcef.so/dll on Linux and Windows. Temporary files like logs should probably be created in the current working directory or a known temp path (like /var/log) instead of the DIR_EXE/DIR_MODULE location, but that's a different issue.

  4. Czarek Tomczak reporter

    Marshall wrote:

    We should only need to override DIR_EXE on Linux. DIR_MODULE should already be correct.

    From my testing DIR_MODULE is also wrong, so we should override both:

    #!text
    
    [CEF Python] ---- PK_DIR_EXE: /usr/bin
    [CEF Python] ---- PK_DIR_MODULE: /usr/bin
    [CEF Python] ---- PK_DIR_CURRENT: /home/czarek/github/cefpython/src/linux/binaries_64bit
    [CEF Python] ---- PK_FILE_EXE: /usr/bin/python2.7
    [CEF Python] ---- PK_FILE_MODULE: /usr/bin/python2.7
    
  5. Czarek Tomczak reporter

    Marshall, Let's take again look at this issue. In my original patch that exposed CefOverridePath, you were worried that overriding paths is time-sensitive, so we shouldn't allow user to do that. You suggested that instead of making it configurable we should probably just set DIR_EXE = DIR_MODULE = <libcef.so path> on Linux. That is what I did in PR #66. However this caused issues with ceftests when running from the chromium/out/ directory. So you suggested that we expose a CefSettings option to set the path. I can make it that way, however I think exposing CefOverridePath was a simple and future-proof solution, because you don't know if upstream Chromium will make some changes again that will break things and another fix will be required. I don't see no advantage of having an another option like CefSettings.override_dir_exe_module_paths over making a call to CefOverridePath. In both cases you have to provide a path and need to know that this is a fix for a bug on Linux.

  6. Czarek Tomczak reporter

    In CEF Python I am calling CefOverridePath before calling CefInitialize and everything works great.

  7. Czarek Tomczak reporter

    I know that recently there were changes to unit tests, ceftests are now build the same as cefclient and other test apps. So is overriding DIR_EXE and DIR_MODULE still an issue for tests? If not then pull request #66 can be accepted.

  8. Marshall Greenblatt

    @Czarek: Do you have a PR for the CefOverridePath change that you reference? Is that an alternative to PR 66?

  9. Czarek Tomczak reporter

    @magreenblatt Have you took look at the new PR on GitHub? Please let me know if this solution is acceptable or whether I should create a new PR with this fix configurable with CefSettings.module_path option or similar.

  10. Marshall Greenblatt

    @Czarek See comments in the Github PR. I think CefOverridePath could be a good option with some safety modifications.

  11. Guillermo Zunino

    This issue is a critical for any integration of CEF with VM runtimes like java or python, to provide a seamless user experience that doesn't require manually copying files. I'm working on SWT java integration of CEF and facing this issue which force users to copy icudtl.dat and *.bin files to /usr/lib/jvm/java-8-oracle/jre/bin

  12. Joe Lauer

    I second @guilllermozunino and how this patch would be incredibly useful for integrations into Java. I am trying to get jcef packaged into a jar that gets expanded at runtime & loaded, but no matter what I do the icudtl.dat and *.bin files MUST reside where the java bin is. If I could provide my own path, I could then include everything in a single .jar, extract the contents at runtime, and then load up jcef.

  13. Marshall Greenblatt

    To explain the current behavior:

    We should probably load resources (locales/* and *.pak files) from DIR_ASSETS instead of DIR_MODULE in GetResourcesFilePath on Linux.

    The swiftshader binaries (swiftshader/*.so) are also currently loaded from DIR_MODULE in InitializeStaticEGLInternal.

  14. Riku Palomäki

    chrome-sandbox binary is not really used anymore, except maybe if you have some really ancient kernel. I wouldn't care about that.

  15. Marshall Greenblatt

    chrome-sandbox binary is not really used anymore, except maybe if you have some really ancient kernel. I wouldn't care about that.

    Thanks for pointing that out. The sandbox type is controlled in ZygoteHostImpl::Init. The chrome-sandbox binary will be used if the kernel doesn't support CLONE_NEWUSER or if it's not possible to immediately move to a new user namespace. CLONE_NEWUSER was added around kernel version 3.8 but I'm not sure about that second condition.

    It looks like Ubuntu 14.04 includes 3.13 and Debian Jessie includes 3.16, so maybe we can stop shipping the chrome-sandbox binary?

    I've filed this as issue #2630.

  16. Marshall Greenblatt

    Linux: Load additional binaries from DIR_ASSETS (fixes issue #1936)

    This adds .pak, locales/.pak , chrome-sandbox, libGLESv2.so, libEGL.so and swiftshader/*.so to the list of binaries that will be loaded from the libcef.so directory instead of the executable directory by default.

    → <<cset b3468451f545>>

  17. Marshall Greenblatt

    Linux: Load additional binaries from DIR_ASSETS (fixes issue #1936)

    This adds .pak, locales/.pak , chrome-sandbox, libGLESv2.so, libEGL.so and swiftshader/*.so to the list of binaries that will be loaded from the libcef.so directory instead of the executable directory by default.

    → <<cset 2e39487ad6d4>>

  18. Marshall Greenblatt

    Linux: Load additional binaries from DIR_ASSETS (fixes issue #1936)

    This adds .pak, locales/.pak , chrome-sandbox, libGLESv2.so, libEGL.so and swiftshader/*.so to the list of binaries that will be loaded from the libcef.so directory instead of the executable directory by default.

    → <<cset ee4b49f311d5>>

  19. Henri Beauchamp

    I just found out (with both CEF 74 and 75), that locales/*.pak still have to be located along the executable binay, and *not* along libcef.so…

  20. Marshall Greenblatt

    I just found out (with both CEF 74 and 75), that locales/*.dat still have to be located along the executable directory, and *not* along libcef.so…

    Do you mean locales/*.pak? Did you set CefSettings.locales_dir_path?

  21. Henri Beauchamp

    Yes, *pak (I was editing my post when you replied to it… You are damned fast ! 😃 )…

    And no, CefSettings.locales_dir_path is not set in Dullahan (the host executable using CEF, made by Second Life programmers). But even as the default path, should not it be the same path as for other .pak files (those not in the locales/ directory) ? After all, those locales/*.pak files are specific to libcef.so, not to the executable using it…

  22. Marshall Greenblatt

    From PR#265, without the above change loading will fail with the following errors when the application binary is not in the same directory as the .so:

    [1031/074208.386065:WARNING:child_process_launcher_helper_posix.cc(119)] Ignoring invalid file v8_context_snapshot.bin
    [1031/074208.386093:WARNING:child_process_launcher_helper_posix.cc(119)] Ignoring invalid file natives_blob.bin
    

  23. Log in to comment