Add support for component=shared_library builds

[Marshall Greenblatt]

Chromium supports a component (multi-DLL) build that can improve developer build times: http://www.chromium.org/developers/how-tos/component-build. It would be nice to support this build mode with CEF. Issue #1107 comment 13 has some related discussion. Note that this mode would be for development only and would not support the creation of binary distributions.

[Marshall Greenblatt]

Trying a component build of CEF on Windows at current master revision:

cd path\to\chromium\src\cef
set GYP_GENERATORS=ninja,msvs-ninja
set GYP_MSVS_VERSION=2013
set GYP_DEFINES=component=shared_library
call cef_create_projects.bat
cd ..
ninja -C out\Debug cefclient cefsimple cef_unittests

First build error:

[17/6256] CXX obj\third_party\WebKit\Source\core\html\webcore_html.HTMLProgressElement.obj
FAILED: ninja -t msvc -e environment.x86 -- "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64_x86\cl.exe
" /nologo /showIncludes /FC @obj\third_party\WebKit\Source\core\html\webcore_html.HTMLPlugInElement.obj.rsp /c ..\..\third_party\WebKit\Source\core\html\HTMLPlugInElement.cpp /Foobj\third_party\WebKit\Source\core\html\webcore_html.HTMLPlugInElement.obj /Fdobj\third_party\WebKit\Source\core\webcore_html.cc.pdb
d:\code\chromium_git\chromium\src\third_party\webkit\source\core\html\htmlpluginelement.cpp(63) : error C2220: warning treated as error - no 'object' file generated
d:\code\chromium_git\chromium\src\third_party\webkit\source\core\html\htmlpluginelement.cpp(63) : warning C4273: 'blink::HTMLPlugInElement::HTMLPlugInElement' : inconsistent dll linkage
        d:\code\chromium_git\chromium\src\third_party\webkit\source\core\html\htmlpluginelement.h(75) : see previous definition of '{ctor}'
d:\code\chromium_git\chromium\src\third_party\webkit\source\core\html\htmlpluginelement.cpp(76) : warning C4273: 'blink::HTMLPlugInElement::~HTMLPlugInElement' : inconsistent dll linkage
        d:\code\chromium_git\chromium\src\third_party\webkit\source\core\html\htmlpluginelement.h(47) : see previous definition of '{dtor}'
d:\code\chromium_git\chromium\src\third_party\webkit\source\core\html\htmlpluginelement.cpp(86) : warning C4273: 'blink::HTMLPlugInElement::trace' : inconsistent dll linkage
        d:\code\chromium_git\chromium\src\third_party\webkit\source\core\html\htmlpluginelement.h(48) : see previous definition of 'trace'
... etc

The CORE_EXPORT value is defined in third_party/WebKit/Source/core/CoreExport.h as follows:

#if LINK_CORE_MODULES_SEPARATELY && defined(COMPONENT_BUILD)
#if defined(WIN32)

#if defined(BLINK_CORE_IMPLEMENTATION) && BLINK_CORE_IMPLEMENTATION
#define CORE_EXPORT __declspec(dllexport)
#else
#define CORE_EXPORT __declspec(dllimport)
#endif // defined(BLINK_CORE_IMPLEMENTATION) && BLINK_CORE_IMPLEMENTATION

#else // defined(WIN32)
#if defined(BLINK_CORE_IMPLEMENTATION) && BLINK_CORE_IMPLEMENTATION
#define CORE_EXPORT __attribute__((visibility("default")))
#else
#define CORE_EXPORT
#endif
#endif

#else // defined(COMPONENT_BUILD)
#define CORE_EXPORT
#endif

Observations:

• link_core_modules_separately=1 by default in third_party/WebKit/Source/build/features.gypi for component=shared_library builds.
• All *.ninja files under out\Debug\obj\third_party\WebKit\Source\core define both LINK_CORE_MODULES_SEPARATELY and COMPONENT_BUILD. So far so good.
• BLINK_CORE_IMPLEMENTATION=1 (which is required for the blink DLL to export symbols correctly) is only defined in webcore_shared.ninja.
• Therefore, when creating a build with "component='shared_library' and link_core_modules_separately==1", the libcef_static target needs to depend on the webcore_shared target instead of the webcore target.
• Since CEF directly includes Blink headers in libcef\renderer\webkit_glue.cc we also need to define LINK_CORE_MODULES_SEPARATELY for the libcef_static target. This can be done using the feature_defines value from features.gypi.

[Marshall Greenblatt]

Next error:

[139/565] CXX obj\cef\libcef\common\libcef_static.upload_data.obj
FAILED: ninja -t msvc -e environment.x86 -- "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64_x86\cl.exe
" /nologo /showIncludes /FC @obj\cef\libcef\common\libcef_static.upload_data.obj.rsp /c ..\..\cef\libcef\common\upload_data.cc /Foobj\cef\libcef\common\libcef_static.upload_data.obj /Fdobj\cef\libcef_static.cc.pdb
d:\code\chromium_git\chromium\src\cef\libcef\common\upload_data.cc(12) : error C2220: warning treated as error - no 'object' file generated
d:\code\chromium_git\chromium\src\cef\libcef\common\upload_data.cc(12) : warning C4273: 'net::UploadData::UploadData' :inconsistent dll linkage
        d:\code\chromium_git\chromium\src\cef\libcef\common\upload_data.h(34) : see previous definition of '{ctor}'
d:\code\chromium_git\chromium\src\cef\libcef\common\upload_data.cc(17) : warning C4273: 'net::UploadData::AppendBytes' : inconsistent dll linkage
        d:\code\chromium_git\chromium\src\cef\libcef\common\upload_data.h(36) : see previous definition of 'AppendBytes'

d:\code\chromium_git\chromium\src\cef\libcef\common\upload_data.cc(27) : warning C4273: 'net::UploadData::AppendFileRange' : inconsistent dll linkage
        d:\code\chromium_git\chromium\src\cef\libcef\common\upload_data.h(38) : see previous definition of 'AppendFileRange'
d:\code\chromium_git\chromium\src\cef\libcef\common\upload_data.cc(34) : warning C4273: 'net::UploadData::~UploadData' : inconsistent dll linkage
        d:\code\chromium_git\chromium\src\cef\libcef\common\upload_data.h(72) : see previous definition of '{dtor}'
[139/565] CXX obj\cef\libcef\common\libcef_static.main_delegate.obj
ninja: build stopped: subcommand failed.

The UploadData and UploadElement classes are part of CEF (originally copied from Chromium) so they shouldn't specify NET_EXPORT.

[Marshall Greenblatt]

Now we get to all of the unresolved externals during the linking of libcef.dll (output attached).

[Marshall Greenblatt]

Here is the diff that fixes the build errors:

diff --git cef.gyp cef.gyp
index 9e7871c..32f6082 100644
--- cef.gyp
+++ cef.gyp
@@ -17,6 +17,8 @@
   'includes': [
     # Bring in the source file lists.
     'cef_paths2.gypi',
+    # Bring in feature defines for Blink.
+    '../third_party/WebKit/Source/build/features.gypi',
   ],
   'targets': [
     {
@@ -847,6 +849,9 @@
       'msvs_guid': 'FA39524D-3067-4141-888D-28A86C66F2B9',
       'defines': [
         'BUILDING_CEF_SHARED',
+        # Import features_defines from
+        # third_party/WebKit/Source/build/features.gypi.
+        '<@(feature_defines)',
       ],
       'include_dirs': [
         '.',
@@ -928,7 +933,6 @@
         '<(DEPTH)/third_party/hunspell/hunspell.gyp:hunspell',
         '<(DEPTH)/third_party/libxml/libxml.gyp:libxml',
         '<(DEPTH)/third_party/WebKit/public/blink.gyp:blink',
-        '<(DEPTH)/third_party/WebKit/Source/core/core.gyp:webcore',
         '<(DEPTH)/third_party/widevine/cdm/widevine_cdm.gyp:widevinecdmadapter',
         '<(DEPTH)/third_party/widevine/cdm/widevine_cdm.gyp:widevine_cdm_version_h',
         '<(DEPTH)/third_party/zlib/zlib.gyp:minizip',
@@ -1466,6 +1470,15 @@
             '<(DEPTH)/ui/views/test/test_views_delegate_aura.cc',
           ],
         }],
+        ['component!="shared_library" or link_core_modules_separately==0', {
+          'dependencies': [
+            '<(DEPTH)/third_party/WebKit/Source/core/core.gyp:webcore',
+          ],
+        }, {
+          'dependencies': [
+            '<(DEPTH)/third_party/WebKit/Source/core/core.gyp:webcore_shared',
+          ],
+        }]
       ],
     },
   ],
diff --git libcef/common/upload_data.h libcef/common/upload_data.h
index 2e3b89b..c2c2b46 100644
--- libcef/common/upload_data.h
+++ libcef/common/upload_data.h
@@ -27,7 +27,7 @@ namespace net {
 // Until there is a more abstract class for this, this one derives from
 // SupportsUserData to allow users to stash random data by
 // key and ensure its destruction when UploadData is finally deleted.
-class NET_EXPORT UploadData
+class UploadData
     : public base::RefCounted<UploadData>,
       public base::SupportsUserData {
  public:
diff --git libcef/common/upload_element.h libcef/common/upload_element.h
index 2b63a0d..10acaf3 100644
--- libcef/common/upload_element.h
+++ libcef/common/upload_element.h
@@ -15,7 +15,7 @@
 namespace net {

 // A class representing an element contained by UploadData.
-class NET_EXPORT UploadElement {
+class UploadElement {
  public:
   enum Type {
     TYPE_BYTES,

[Marshall Greenblatt]

Summary of unresolved externals:

• aura::client::ActivationChangeObserver
• aura::Env
• aura::WindowObserver
• aura::WindowTreeHost
• blink::V8RecursionScope
• blink::WebLocalFrameImpl
• content::InitializeSandboxInfo (can't be linked externally -- need to depend on content/content.gyp:sandbox_helper_win?)
• CefURLFetcherResponseWriter (bug in CEF code -- should not specify NET_EXPORT)
• content::ViewBlobInternalsJobFactory
• content::ViewHttpCacheJobFactory
• kWebuiResources (comes from generated code -- might need to fix the generator)
• kWebuiResourcesSize (comes from generated code -- might need to fix the generator)
• net::SQLitePersistentCookieStore
• ppapi::host::PpapiHost
• ppapi::host::ReplyMessageContext
• ppapi::host::ResourceMessageFilter
• ppapi::host::ResourceHost
• ppapi::host::MessageFilterHost
• ppapi::PpapiPermissions
• ppapi::proxy::PPBFlash_DrawGlyphs_Params
• ppapi::proxy::ResourceMessageReplyParams
• ppapi::proxy::SerializedFlashMenu
• ppapi::proxy::SerializedFontDescription
• PpapiHostMsg_Flash_GetLocalTimeZoneOffset, PpapiHostMsg_*, PpapiPluginMsg_*, etc... (might need to fix IPC macros)
• storage::CrackIsolatedFileSystemName
• storage::GetFileSystemPublicType
• storage::GetIsolatedFileSystemName
• storage::GetIsolatedFileSystemRootURIString
• storage::VirtualPath
• ui::Compositor
• ui::CompositorDelegate
• ui::CompositorLock
• ui::CompositorVSyncManager
• ui::Event
• ui::EventHandler
• ui::LatencyInfo
• ui::Layer
• wm::WMState
• WTFReportAssertionFailure
• WTF::AtomicString
• WTF::RefCountedBase
• WTF::String
• WTF::StringImpl

[Marshall Greenblatt]

Chromium tracking bug filed at https://crbug.com/632048.

[Marshall Greenblatt]

Component build support added in master revision b216f42.

The component build setup for GN is described at https://chromium.googlesource.com/chromium/src/+/master/docs/component_build.md.

Important considerations when using the component build:

• Component builds are for local Chromium/CEF development purposes only and cannot be used to create a CEF binary distribution.
• Component builds have only been tested on Windows where long link times with non-component builds make them beneficial.
• Execution of component builds is slower than non-component builds. Some unit tests may time out as a result.
• Testing on a Windows 10 64-bit system (i7-4980HQ@2.8GHz, 16GB RAM, fast SSD), creating a 32-bit Debug build of current CEF master with VS2015 after modifying one .cc file in libcef_static:
  • Building with GN_DEFINES=is_win_fastlink=true takes 507.80s total.
  • Building with GN_DEFINES=is_win_fastlink=true is_component_build=true takes 74.85s total.
  • This represents an ~6.8x improvement in link time with component builds.

Notes specific to the CEF component build integration:

1. To avoid "undefined externals" errors for symbols that are not already exported from Chromium the class or function should be modified to add the appropriate <component>_EXPORT definition. The "component_build_1617.patch" file or other appropriate patch file should then be updated to include these modifications.
2. To avoid "undefined externals" errors for symbols that are already exported from Chromium the deps section of the "libcef_static" target must explicitly list the target that exports the symbols.
3. Multiple targets can include source files like "<component>_switches.cc" if they contain const values only. In that case the 2nd+ target must be built as a static_library target and must disable exports if necessary by defining a COMPILE_<component>_STATICALLY value. See for example the "chrome_elf_set" target.
4. In cases where CEF sources depend on component internals (e.g. lots of non-exported symbols) those sources should be built as part of the component using a source_set target. See for example the "webkit_set" target.
   • To avoid "locally defined symbol 'symbol' imported in function 'function'" errors with these targets make sure to define the correct <component>_IMPLEMENTATION value.

[Marshall Greenblatt]

Might need to add some additional deps to make the build more deterministic. From http://magpcss.org/ceforum/viewtopic.php?f=6&t=15287 :

diff --git a/BUILD.gn b/BUILD.gn
index fe144b7..71e9bab 100644
--- a/BUILD.gn
+++ b/BUILD.gn
@@ -314,6 +314,9 @@ if (is_win) {
     deps = [
       "//components/crash/core/common",  # crash_keys
       "//gpu/config:crash_keys",
+      "//chrome/common:features",
+      "//ppapi/features:features",
+      "//printing/features:features",
     ]
   }

Fixed in master revision 85c945b.

[Marshall Greenblatt]

VS2015U3 crashes frequently while trying to debug the component build. VS2017 version 15.2 (26430.16) seems to give better results.

[PeterO]

On running the CefClient tests, all tests invoking CefJavaScriptDialogRunnerWin::Run crashes. (sourcefile: chromium\src\cef\libcef\browser\native\javascript_dialog_runner_win.cc)

The problem is this: To retrieve the full pathname to the dll where the dialog resource is to be found it uses PathService::Get(base::FILE_MODULE, ...). For non-compoment builds this returns libcef.dll but for non-component builds it returns base.dll (where PathService resides) which will fail. Solved it by taking the dir-path and appending libcef hardcoded.

diff --git a/libcef/browser/native/javascript_dialog_runner_win.cc b/libcef/browser/native/javascript_dialog_runner_win.cc
index a0cfd99..0bb0608 100644
--- a/libcef/browser/native/javascript_dialog_runner_win.cc
+++ b/libcef/browser/native/javascript_dialog_runner_win.cc
@@ -121,8 +121,10 @@ void CefJavaScriptDialogRunnerWin::Run(
   HMODULE hModule = NULL;

   // Try to load the dialog from the DLL.
-  if (PathService::Get(base::FILE_MODULE, &file_path))
+  if (PathService::Get(base::DIR_MODULE, &file_path)) {
+    file_path = file_path.Append(L"libcef.dll");
     hModule = ::GetModuleHandle(file_path.value().c_str());
+  }
   if (!hModule)
     hModule = ::GetModuleHandle(NULL);
   DCHECK(hModule);

[PeterO]

VS2015U3 has a hard time with loading and running the CEF solution. I've unloaded all projects and loaded only the projects I need to debug. This seems to go well.

[PeterO]

I had one unresolved external in class WebContentsView. Solved by adding CONTENT_EXPORT:

diff --git a/content/browser/web_contents/web_contents_view.h b/content/browser/web_contents/web_contents_view.h
index e4401f8..20b8560 100644
--- a/content/browser/web_contents/web_contents_view.h
+++ b/content/browser/web_contents/web_contents_view.h
@@ -24,7 +24,7 @@ struct ScreenInfo;
 // The WebContentsView is an interface that is implemented by the platform-
 // dependent web contents views. The WebContents uses this interface to talk to
 // them.
-class WebContentsView {
+class CONTENT_EXPORT WebContentsView {
  public:
   virtual ~WebContentsView() {}

[Marshall Greenblatt]

Thanks, dialog loading issue fixed in master revision 6da2bbf.