Cef3: Multi-touch support with offscreen rendering.

Issue #1059 resolved
Former user created an issue

Original issue 1059 created by jf.verdon@mgdesign.fr on 2013-08-29T12:13:04.000Z:

Functions to inject touchs events when rendering browser in offscreen mode would be very usefull.

Event data proposal:
typedef struct _cef_touch_event_t {
///
// X coordinate relative to the left side of the view.
///
int x;

///
// Y coordinate relative to the top side of the view.
///
int y;

///
// A touch point identifier that distinguishes a particular touch input.
// This value stays consistent in a touch contact sequence from the point a contact comes down until it comes back up.
// An id may be reused later for subsequent contacts.
///
int id;

///
// Bit flags describing any pressed modifier keys. See
// cef_event_flags_t for values.
///
uint32 modifiers;
} cef_touch_event_t;

Functions signatures proposal ('CefBrowser' class members):
virtual void SendTouchClickEvent(const CefTouchEvent& event, bool touchUp)=0;
virtual void SendTouchMoveEvent(const CefTouchEvent& event)=0;

Identifieds changes to support this functionality (non exhaustive):
* Add '_cef_touch_event_t' struct in 'cef_types.h'
* Add 'CefTouchEventTraits' class in 'cef_types_wrappers.h'
* Add 'CefTouchEvent' typedef in 'cef_types_wrappers.h'
* Add 'SendTouchClickEvent' and 'SendTouchMoveEvent' functions to 'cef_browser.h', 'browser_host_impl.h' and 'browser_host_impl.cc'
* Add 'PlatformTranslateTouchEvent' (and eventually 'PlatformTranslateTouchClickEvent' and 'PlatformTranslateTouchMoveEvent') in 'browser_host_impl'
* Provide implementation for 'PlatformTranslateTouchEvent' in 'browser_host_impl_win.cc', 'browser_host_impl_mac.mm' and 'browser_host_impl_gtk.cc'
* Add 'SendTouchEvent' in 'render_widget_host_view_osr.cc'

Comments (85)

  1. Marshall Greenblatt

    Comment 1. originally posted by magreenblatt on 2013-09-03T17:52:28.000Z:

    Sounds reasonable. Patches welcome. Please also add unit tests in os_rendering_unittest.cc and support in cefclient when running in OSR mode.

  2. Former user Account Deleted

    Comment 2. originally posted by johnjosephorr on 2013-10-04T19:18:24.000Z:

    I'm considering implementing this, but I am only windows literate when it comes to multitouch input handling. The suggestion here for the _cef_touch_event_t layout doesn't seems quite right to me for a multitouch system. The suggested structure layout looks more like a single "touch point" (TOUCHINPUT in Windows, WebTouchPoint in Webkit, see src/third_party/WebKit/public/web/WebInputEvent.h). In windows and apparently in webkit, a "touch event" contains one or more "touch points", so both Windows and Webkit appear to bunch individual touchpoints into a single TouchEvent.

    I don't know if the suggested structure layout and naming has some other justification that I don't know about, or if its just a design starting point.

  3. Former user Account Deleted

    Comment 3. originally posted by johnjosephorr on 2013-10-04T19:30:29.000Z:

    The windows specific code for handling and translating multitouch input lives in the chromium code base in src/content/browser/renderer_host/render_widget_host_view_win.cc. I'm guessing it makes sense to use that code as a starting point.

    I don't see how the mac or gtk code handles raw multitouch. Are there significant limitations on those platforms, or does the OS in those cases take care of "cooking" multitouch into ready-made gestures?

  4. Former user Account Deleted

    Comment 4. originally posted by jf.verdon@mgdesign.fr on 2013-12-11T14:24:27.000Z:

    @ johnjose: "I don't know if the suggested structure layout and naming has some other justification that I don't know about, or if its just a design starting point."
    It was starting point thoughts, but indeed, your proposal is better. From my small experience on mac, the system can provide raw touch information, the same way (or almost) Windows does.
    Did you start to implement something concerning this functionnality?

  5. Former user Account Deleted

    Comment 7. originally posted by jf.verdon@mgdesign.fr on 2014-01-08T16:07:08.000Z:

    Hi,

    Here commes the patch. Things that need to be done in order to validate it:
    - test compilation on other platforms (linux/osx). I added some #def for Windows related code but not sure they are all placed well.
    - test compilation on Windows XP. No problems for LibCef I think, but some function added in CefClient sample are Win7+ only. Dynamic linking these function could make the CefClient executable XP compliant.
    - Handle (or delete?) the small code part with "TODO JEFF" in it: I do not know if touch injection outside OSR mode makes sense, and nor how to integrate this easily in Cef...

    This patch applies on branch 1650.

  6. Former user Account Deleted

    Comment 8. originally posted by johnjosephorr on 2014-01-10T19:16:28.000Z:

    jf.ver...@ mgdesign.fr,

    I apologize, I didn't see that you had asked me a question until just now.

    To answer your question, no I haven't started to implement anything so far. I had to put my CEF work down for a little while, but I will need to pick it up again in a week or two. If expect I'll take your patch for a test drive if that's OK.

  7. Former user Account Deleted

    Comment 9. originally posted by johnjosephorr on 2014-01-17T23:12:24.000Z:

    I applied the patches to my local cef build and have been testing my app with "off screen rendering" for a few hours. This is strictly Windows 7 testing. I have two multitouch monitors for testing, one a two touch monitor, the other a 10 touch. No problems so far.

    I've used two publically available web site for multitouch smoke testing:

    http://blogs.msdn.com/b/ie/archive/2011/10/19/handling-multi-touch-and-mouse-input-in-all-browsers.aspx
    http://www.openstreetmap.org

  8. Former user Account Deleted

    Comment 10. originally posted by johnjosephorr on 2014-01-17T23:17:30.000Z:

    Of the changes, those made to these files below are straightforward and I don't see any issues there:

    include/capi/cef_browser_capi.h
    include/cef_browser.h
    libcef/browser/browser_host_impl.h
    libcef_dll/cpptoc/browser_host_cpptoc.cc
    libcef_dll/ctocpp/browser_host_ctocpp.cc
    libcef_dll/ctocpp/browser_host_ctocpp.h

    In the list below, the changes here are mostly to hold the newly defined "touch point" structure and a "touch event" which holds a set of "touch points". The changes here are mostly fine, but I'm not an expert on multitouch capabilities inside the browser or across OSes, so I can see someone tweaking these slightly. The struct _cef_touch_event_t uses a hard coded value of 12 to match the hard coded value of webkit's WebTouchEvent::touchesLengthCap defined in src/third_party/WebKit/public/web/WebInputEvent.h. Also, _cef_touch_event_t does not have the equivalent to changedTouches or targetTouches, but that's OK AFAIK.

    include/internal/cef_types.h
    include/internal/cef_types_wrappers.h

    The file libcef/browser/browser_host_impl.cc has an open question if there is a need for touch injection outside OSR. I don't believe you need to worry about the non-OSR case in Windows. AFAIK, that already worked fine before this patch (the WM_TOUCH events arrive through normal means when not using OSR). I can't speak to other platforms.

    The sample/test program code changes in tests/cefclient/cefclient_osr_widget_win.cpp are fairly straightforward. The only shortcoming I see is that the contain Windows 7 (or later) API calls, so they will cause cefclient.exe to fail to load in Windows OSes that predate RegisterTouchWindow, UnregisterTouchWindow, GetTouchInputInfo, and CloseTouchInputHandle (like Windows XP for example). Its possible to dynamically attach to these entry points in user32.dll with LoadLibrary and GetProcAddress, i.e.:

    typedef BOOL (WINAPI * PFNREGISTERTOUCHWINDOW)( HWND hWnd, ULONG ulFlags );

    g_hDUser32Dll = LoadLibraryW( L"user32.dll" );

    if ( g_hDUser32Dll )
    {
    g_FLASHLOADER_pfnRegisterTouchWindow = (PFNREGISTERTOUCHWINDOW)GetProcAddress( g_hDUser32Dll, "RegisterTouchWindow" );

    ...

    That's what I do in my app.

    The changes in these files below are the meat the work involved and will take a little more effort to get an understanding of the differences:

    libcef/browser/render_widget_host_view_osr.cc
    libcef/browser/render_widget_host_view_osr.h

  9. Marshall Greenblatt
    • changed status to open

    Comment 11. originally posted by magreenblatt on 2014-02-11T19:51:43.000Z:

    @ commentcomment 7.: Thanks for the patch. Can you update it to work with the current trunk? Some comments:

    1. Line 55:

    +typedef struct _cef_touch_point_t {
    + ///
    + //
    + ///
    + uint32 id;

    Improve the documentation. What is the id and how does the user generate it?

    1. Line 81:
    • ///
    • // Event timestamp, in seconds (needed for scrolling inertia for example).
    • ///
    • double timestamp;

    Improve the documentation. How should this timestamp be created or formatted?

    1. Line 183:
    • // TODO JEFF
    • /*if (widget)
    • widget->SendTouchEvent(event);*/

    Supporting usage when windowed rendering is enabled would be consistent with the other event types. Does it work?

    1. Line 436:
    • // Pinch gestures are disabled by default on windows desktop. See
    • // crbug.com/128477 and crbug.com/148816
    • if (gesture->type() == ui::ET_GESTURE_PINCH_BEGIN ||
    • gesture->type() == ui::ET_GESTURE_PINCH_UPDATE ||
    • gesture->type() == ui::ET_GESTURE_PINCH_END) {
    • return true;
    • }

    It seems that these bugs have been resolved with the "--enable-pinch" flag.

    1. Add unit tests in os_rendering_unittest.cc.

    Also, lots of style errors. You can use the check_style tool to catch most of these.

    1. Line 101:
    • ///
    • // List of touch points. WebKit supports a maximum of 12 simultaneous touch points, see WebInputEvent class in WebTouchEvent.h
    • ///

    Wrap lines to 80 characters. Same in other source files.

    1. Line 236:

    #include "webkit/common/cursors/webcursor.h"
    +#include "ui/base/sequential_id_generator.h"
    +#include "ui/events/event.h"
    +#include "ui/events/event_utils.h"
    +#include "ui/gfx/win/dpi.h"

    Include order should be alphabetical.

    1. Line 252:

    +class CefWebTouchState {
    + public:
    + explicit CefWebTouchState(const CefRenderWidgetHostViewOSR* osrWindow);

    Wrong indentation for this class. Some of the methods of this class should also be marked 'const'.

    1. Line 290:

    +#endif // defined(OS_WIN)

    Two spaces before //.

    1. Line 371:
    • // Copy event because we van modify it internally

    Typo and missing period.

    1. Line 377:
    • if(render_widget_host_->ShouldForwardTouchEvent()) {

    Need a space between if and (.

    1. Line 478:

    +size_t CefWebTouchState::UpdateTouchPoints(CefTouchEvent& event, size_t offset)
    +{

    Bracket in the wrong place.

    1. Line 598:
    • touch_event_.changedTouches[i].state =
    • WebKit::WebTouchPoint::StateReleased;

    Wrong indent. Should be 4 spaces on the continuation line.

  10. Former user Account Deleted

    Comment 12. originally posted by jf.verdon@mgdesign.fr on 2014-02-13T14:23:50.000Z:

    Thanks to both of you for your remarks. Unfortunately, I'm currently in "project mode" and do not have time to migrate to trunk (I try to launch automate to build a trunk version of Cef as a background task, but the automate build seem to be broken at this time...). Guess I will be able to do it soon...
    Alright for the documentation, I forgot to describe those 2 vars.
    Concerning the commented TODO, I don't know how to do to dispatch the message the same way as the mouse event are dispatched, because if I remember well, "widget" do not have an injection method for touch.
    If the "enable-pinch" bug is corrected in trunk, then obviously this could be deleted (although it seem necessary in branch 1650).
    And thanks for mentionning check_style tool, I'll know its existence for the next time I'll patch something :)

  11. Former user Account Deleted

    Comment 13. originally posted by jf.verdon@mgdesign.fr on 2014-02-18T13:50:49.000Z:

    Here comes the patch updated for the branch 1750.
    As Chromium sources still have the code concerning "ET_GESTURE_PINCH_x" (chromium\src\content\browser\renderer_host\render_widget_host_view_win.cc), I have choosen to keep this part.
    I made a pass concerning style, so it should be better.

  12. Former user Account Deleted

    Comment 14. originally posted by jf.verdon@mgdesign.fr on 2014-02-20T15:17:45.000Z:

    As CEF 1750 now use Aura, there's big chances that the proposed 1750 patch do not reflect exactly the comportement of non-offscreen mode... Unfortunaly, I was not aware about the switch to Aura, so I convert the existing 1650 patch directly. Porting the patch to 1750 will need more work than I though, as Aura input injection seems a little different than the old system.

    * As a side note, there's 2 missing lines in 1750 patch, in "CefRenderWidgetHostViewOSR.cpp":

    line 172 (inside CefRenderWidgetHostViewOSR ctor):
    + gesture_recognizer_->AddGestureEventHelper(this);
    line 182 (inside CefRenderWidgetHostViewOSR dtor):
    + gesture_recognizer_->RemoveGestureEventHelper(this);

    * Furthemore, there's another bug on 1650 patch (and so on 1750), still in "CefRenderWidgetHostViewOSR.cpp", "UpdateTouchPoints" function:

           if (!(point = AddTouchPoint(event.points[i + offset])))  
             continue;
    
    • last_type = TPT_RELEASED;
    • event.points[i + offset].type = TPT_RELEASED;
    • last_type = TPT_PRESSED;
    • event.points[i + offset].type = TPT_PRESSED;
      touch_event_.type = blink::WebInputEvent::TouchStart;
      }
      break;
  13. Former user Account Deleted

    Comment 15. originally posted by jf.verdon@mgdesign.fr on 2014-03-14T15:25:16.000Z:

    Is there's plans for converting "CefRenderWidgetHostViewOSR" inputs events injection functions to Aura style (make CefRenderWidgetHostViewOSR inherits from "aura::WindowDelegate")?

  14. Marshall Greenblatt

    Comment 16. originally posted by magreenblatt on 2014-03-14T15:37:47.000Z:

    #commentcomment 15.: Probably not, since Aura is only used on Windows and Linux.

  15. Former user Account Deleted

    Comment 17. originally posted by jf.verdon@mgdesign.fr on 2014-03-14T16:18:25.000Z:

    Ok, that's sad because there's a bug with win8 and cef non-aura (and as far as my tests goes, I can say more generally with chromium non-aura). Will compile cef 1750 branch without Aura to confirm my diagnostic...

  16. Marshall Greenblatt

    Comment 18. originally posted by magreenblatt on 2014-03-14T17:04:06.000Z:

    @ commentcomment 17.: 1750+ will never be non-Aura on Windows. The non-Aura code is being deleted from Chromium.

  17. David Whitlark

    If anyone cares I am going through the process of exposing touch event integration on branch 2272. I've gotten some moderate success and should have a patch file I can upload here when I am done.

  18. Stas Dunayev

    Hey David.

    did you manage to make any progress? on that. really interested to hear about that.

  19. John McCreton

    Hi David

    Just developed and application that uses Openlayers mapping the CefSharp WPF browser control. Unfortunately we've just discovered the absence of multi-touch support which is a bit of a shame as the application will be deployed on Windows touch screen devices in emergency service vehicles. Therefore, I'd be very interested in how you're getting on with the multi-touch integration you've been doing as mentioned here

    Cheers

  20. David Whitlark

    Hi all, this is my current progress on the patch for 2272.

    There is still an issue where if you navigate away from the initial page loaded with LoadURL, the gesture recognizer stops generating gesture events.

  21. David Whitlark

    Today I discovered my 2272 patch doesn't work on mac. I am going to fix that and post a new patch. Has anyone else had luck trying out the patch?

  22. Oldrich Esner

    Hi David,

    we have the same problem with touch support on Windows 7 (WPF WindowsFormsHost), the last working version is 2062. We've solved LoadUrl touch issue by using LoadRequest.

    Are you going to release patch also into 2357?

  23. David Whitlark

    this is my latest touch & gesture forwarding patch.
    refactored to use ui::FilteredGestureProvider gesture_provider_
    instead of the Aura specific code. I would love to get this integrated, it ended up being a lot less code than expected.

  24. David Whitlark

    Hi @BennyLava90,

    I wasn't able to. I am going to see about getting it to work on OSX in 2357. I think the gesture_provider_ exists in 2272 as well? I am not sure.
    But as far as I can tell it's the cross-platform way of forwarding touch & gesture events in.

  25. Benjamin Hamrick

    Thanks for the quick response! I am looking at it right now and it would be really cool for it to work. I guess I will spend some time upgrading my project to 2357.

    BTW will this let me send "touchBegin", "touchMoved", and "touchEnd" events instead of mouse clicks and scroll wheel events? I am especially interested in pinch to zoom!

  26. David Whitlark

    So internally it manages signaling touchBegin, touchMoved and touchEnd. You worry about telling CEF the state of each individual touch (pressed, released, moved, stationary, etc..)
    And based on the meta-data of each touch point it figures out the overall event type.

  27. Benjamin Hamrick

    I wonder if it would be possible to bypass actually creating a state (I am getting input from an actual touch screen) and just access the internal touchBegin, touchMoved and touchEnd. Are those properties we can access?

  28. Benjamin Hamrick

    I am trying to apply your patch but I seem to have a few files missing:

    Skipped missing target: 'libcef/browser/browser_host_impl.cc'
    Skipped missing target: 'libcef/browser/browser_host_impl.h'
    Skipped missing target: 'libcef/browser/browser_settings.cc'
    Skipped missing target: 'libcef/browser/render_widget_host_view_osr.cc'
    Skipped missing target: 'libcef/browser/render_widget_host_view_osr.h'
    Skipped missing target: 'libcef_dll/cpptoc/browser_host_cpptoc.cc'
    
  29. KarenA

    Looking forward to see this patch in main branches, to get it from CefBuilds is there any progress regards that? Thanks!

  30. Joachim Segala

    We successfully applied @goldbuick's patch on linux, and it works fine!

    Saddly, it seems that the --disable-pinch switch is ignored when using touch API (works fine with cefclient).

  31. Benjamin Hamrick

    Your patch has been working great! We got native touch input from our touch screen and can now play html5 touch games on it. We are however looking at upgrading to 2454 soon. Would your patch work for that version as well?

  32. David Whitlark

    @segalaj yeah I need to figure out how to get browser settings info at that layer. no idea how
    @BennyLava90 I have been actively working on a 2454 patch this morning.
    I need to bite the bullet and see about making a pull request on the master branch.

  33. Alexey Makarov

    @BennyLava90 How to test native touch input on cefclient.exe (win7 + dualtouch monitor). I apply patch, compile cef, run cefclient with options "cefclient --multi-threaded-message-loop --off-screen-rendering-enabled". But i see "mouse emulations".

  34. Jean-François Verdon

    Hi Alex,

    In the original patch I made a long time ago, there was changes for CefClient. This patch is probably out of date (because CefClient sources structure changed a lot), but it can give you insights about what to change in order to dispatch touch events to cef using the offscreen input events api.

    <edit>The original patch I was talking about is this one: osr_touchevent_1750.patch

    search for "tests/cefclient/cefclient_osr_widget_win.cpp" in it</edit>

  35. Alexey Makarov

    @Jean-François Verdon thanks! I tried to implemet path code to cefclient, but not succefully. the result is the same.

  36. Unknown Name

    hello,, i succefully compiled cef with your 2357 patch, i would like to know what we have to put in the touch event in cefclient, before dispatching it to the browser_host

    Thank!

  37. Alexandre Pia

    I quickly hacked an updated version of this patch for branch 2526 (attached with message)
    Unfortunately, I don't have time at the moment to make a proper patch with unit testing and integration in cefclient.

  38. Alexey Makarov

    Hello! I try implement touch in 2526 with path and cefclient in OSR.
    But in chromium\src\content\browser\renderer_host\input\motion_event_web.cc
    i do not understand
    const WebPointerProperties& pointer = event_.touches[pointer_index];
    and after pointer.pointerType always TOOL_TYPE_UNKNOWN

    I found this comment in WebPointerProperties class

    // This class encapsulates the properties that are common between mouse and
    // pointer events and touch points as we transition towards the unified pointer
    // event model.
    // TODO(e_hakkinen): Replace WebTouchEvent with WebPointerEvent, remove
    // WebTouchEvent and WebTouchPoint and merge this into WebPointerEvent.

  39. Benjamin Hamrick

    What is the status of this. It works really great for me, but I would like to upgrade the chrome version. Is this patch in trunk yet? If not could you make a patch for 2641 or 2623?

  40. Thibaut

    Looks like there is an issue with html drag and drop using the patch. It scrolls the web page instead of dragging the element. Other than that it works really well.

  41. Benjamin Hamrick

    Cool. I had a lot of luck with the older patch and am now updating CEF. Is there any plan to submit this to the chromium project so it will be easier to update? How do you guys go about integrating the patch in your build process?

  42. Pavlo Liahushyn

    @BennyLava90 I want to apply this patch for Visual Studio build which uses CEF library. How can I do this? Can anybody help me?

  43. Johnny Depp

    Why is the second most up-voted issue of the project, which clearly have functional patches attached, not on the roadmap?

  44. Benjamin Hamrick

    I would also like to have this in the repo. Does anybody have time for this? I would do it but I did not write any of the patches.

  45. Former user Account Deleted

    I won't have time to update the patch any time soon. I'm probably not gonna be able to test a new version with the patch soon but I'm gonna try.

  46. Former user Account Deleted

    I am in the process of negociating with my company for help either in development of those patches or testing of those patches, I'll post the updates (hopefully within a week)

  47. Bhavesh Savalia

    I am using branch 2623 patch and modified Xilium CefGlue WPF component for passing touch events to Cef library. However zooming functionality is not smooth when I Pinch and Spread, it delays zoom in/out more than a second.
    Has anybody faced this issue? Anybody used Cef patch with WPF application?
    Please suggest.

  48. Yolian Ignatov

    @bbsavalia Would you be kind enough to share the modifications done to Xillium.CefGlue WPF component for passing touch events to Cef library?

  49. Dmitry Azaraev

    @yolian please move discussions about unrelated things in other places. Private messages also works on bitbucket without any issues. As for Xilium.CefGlue, i'm as author say that WpfOsr had been crafted as minimal sample from contribution from some user. I'm personally doesn't understand why anyone need in this, instead of embedding control via winforms or other native window-placement methods. Native rendering will be much faster. Of course if you in game and/or should apply effects on top of rendered browser's content (like perspective effects) - then it is have sense. In ALL other cases - using OSR only because WPF doesn't has native way with embedding native windows - doesn't means that you should use this samples or this way. No, you should use native browser rendering whenever is possible.

  50. Bhavesh Savalia

    @Yolian, I agree with Dmitry Azaraev, if you are building new application go for native rendering. Off-screen rendering also has limitations with test automation framework and due to that in future we are planning to move from off screen rendering to native rendering. If you still need CefGlue modifications for multi touch support let me know through email, I will provide you.

  51. Yolian Ignatov

    @bbsavalia Thanks for getting back to me. I have a few questions I'd like to ask you, if possible. How could I reach you by email?

  52. Bhavesh Savalia

    @yolian I did not find how to use private message, please use email id bbsavalia3@gmail.com, I will monitor this mailbox for next few days.

  53. Christophe Aubert

    Hello Bhavesh,
    I am also interested in the multi touch support. Can you please provide me with some guidance on how to retrieve this version ? My email is christophe@cuddlup.fr. Thanks and regards.

  54. Vicente Teodoro

    Hi I am involved in a project where I am using the CEF3 to display pdf files. The customers want to have the same behavior as they have in the chrome web-browser. In the chrome they can pinch the pdf to activate zoom-in and zoom-out. On CEF3 I am not able to do that. I am using CEF3 as screen-off. Can somebody tell me if it is possible? Thanks.

  55. Denis Declara

    Hello all,
    this seems to be the second most voted CEF issue, and there also seems to be a patch!
    What is missing, in order for the fix to be merged?

  56. Jakub Głuszkiewicz

    It cant be megred cuz its not working with newest branch. Please follow discussion in pullrequest thread.

  57. Log in to comment