System.InvalidCastException raised in CfxTaskExecuteRemoteEventCall RemoteProcedure

Issue #185 resolved
David Desmaisons
created an issue

Hello, In some scenarios, I got this a InvalidCastException in CfrTaskRemoteCall.cs line 56

var self = (CfrTask)System.Runtime.InteropServices.GCHandle.FromIntPtr(gcHandlePtr).Target;

I suspect this may be due garbage collection or concurrence, since I was not abble to identify some 100% reproductible scenarios.

Any hints will be appreciated.

Comments (12)

  1. wborgsm

    A few more questions: is it always a System.Object[] array? If so, could you please inspect an instance of it in the immediate window and tell me what kind of objects the array contains? Any help would be appreciated...

  2. David Desmaisons reporter

    Hello, It is not always a System.Object[] array, so far I have seen CfrV8Value, Thread and System.Object[]. I am facing this problem using the last commit of ChromiumFx with Neutronium. Basically my scenario is heavy communication beetween javascript and C# using TaskRunner. I will tell you if/when I got a better scenario. Thanks.

  3. wborgsm

    Ok, so it looks like ChromiumFX is not submitting garbage pointers to the GC, because I think GC would complain about that. Being valid pointers, the only explanation I can remember of is that the CfrTask object has been collected without the framework being aware of that, and the GC reused the same slot for other handles. So the strong/weak handle switching I implemented to prevent this is not working properly under heavy load. I will fix this.

    In the meantime, if you need a quick fix, I think if you cache all CfrTask objects until execution (put them into a collection after the call to PostTask/PostDelayedTask so they stay rooted, and remove them from the collection once execute is called), the problem should go away. Otherwise, you can wait until I upload a fix later this day or tomorrow (more likely tomorrow).

  4. wborgsm

    I think I found the problem. In the task runner's PostTask and PostDelayedTask methods, the task object is last touched in the line call.task = CfrObject.Unwrap(task).ptr;. In the next call, the task object's gc handle is promoted to a strong handle - all task objects start with a weak handle and only if used with the task runner they are promoted to a strong handle. But the GC is allowed to collect the task object right after call.task = CfrObject.Unwrap(task).ptr; has finished, and under heavy load probably will do so sometimes. So this is a race condition and the KeepAlive calls added in this commit will solve this issue - IF and only if this issue was related to the described problem.

    Looking forward for your feedback - I hope the issue is gone.

  5. David Desmaisons

    @wborgsm Thanks for you quick answer. While investigating this issue, I discovered that some tasks are not executed after calling PostTask even if I retain the corresponding task reference. I am not sure of both issues are linked though. What do you think?

    I will try this fix Today and give you some feedback.

  6. wborgsm

    some tasks are not executed after calling PostTask even if I retain the corresponding task reference

    I added a CfrTask stress test in the test application, please refer to commit 1c6f182. It runs 10000 tasks in a row without pausing. It never missed a single task. I started 6 stress tests simultaneously and everything was working fine, processing 6 * 10000 tasks without failure.

    Could you please refer to the stress test and try to figure out what's different in your code?

  7. wborgsm

    PS: There is one scenario where the application can miss a task (besides there being a bug): if the framework swaps out the render process before the task gets executed. But there is nothing I can do about it ...

  8. David Desmaisons reporter

    @wborgsm Some good news:

    1) After updating with your correction, I was unable to reproduce the exception.

    2) Regarding the task not dispatched, it turned out that I was dispatching the same task many times and in somes cases it will only be executed once.

    It can be reproduced changing your example by creating one task and dispatching multiple times:

                     int taskExecutionCounter = 0;
                    var task = new CfrTask();
                    task.Execute += (s, e1) => {
                        ++taskExecutionCounter;
                        if (taskExecutionCounter % 1000 == 0)
                        {
                            LogWriteLine($"CfrTask stress test: {taskExecutionCounter}/10000 tasks executed.");
                        }
                    };
    
                    for (int i = 0; i < 10000; ++i) {
                          if(i % 2 == 0) {
                            CfrRuntime.PostTask(CfxThreadId.Renderer, task);
                        } else {
                            var tr = CfrTaskRunner.GetForThread(CfxThreadId.Renderer);
                            tr.PostTask(task);
                        }
                        if(i % 1000 == 0) {
                            LogWriteLine($"CfrTask stress test: {i}/10000 tasks posted.");
                        }
                    }
    

    It was not an issue in previous version though and I am not sure if this is a bug or an intended feature.

    In my case, I solve the issue by creating a new CfrTask each time and everything works fine. Thanks for your help!

  9. wborgsm

    Great news! So it looks like the task runner won't enqueue a task if the same task is already on queue. This might very well be a change in CEF since previous versions, I don't know. To be safe, it's better not to reuse task objects or make sure not to post them again before the previous post has executed.

    Thanks again for spotting the race condition and helping to make this program more stable!

  10. Log in to comment