Cop takes a long time to bootstrap (hasn't finished yet and not sure it will)

[Pavel Veller]

I am not sure why and if it's normal but I can't get Cop to bootstrap. The Cop.Intercept() no longer fails (see previous issues posted by me) but my web app wouldn't initialize past the Cop bootstrap. I can tell by what is (or rather isn't) in the log files.

It is a Web app and /bin is quite large (lots of assemblies) so I wonder what is normal for Cop to start and how would go about troubleshooting what takes it so long?

My Cop config looks like this:

{
  "Types": [
    {
      "TypeName": "Score.Custom.Pipelines.Editor.RunComponentDatasourceLocationRules, Score.Custom",
      "Methods": [
        {
          "MethodSignature": "*",
          "Interceptors": [ "ScoreMonitoringInterceptor" ]
        }
      ],
      "GenericArgumentTypes": [ ]
    },
    {
      "TypeName": "Score.Custom.Pipelines.Editor.RunGetLookupSourceItemRules, Score.Custom",
      "Methods": [
        {
          "MethodSignature": "*",
          "Interceptors": [ "ScoreMonitoringInterceptor" ]
        }
      ],
      "GenericArgumentTypes": [ ]
    },
    {
      "TypeName": "Score.Data.Extensions.ItemExtensions, Score.Data",
      "Methods": [
        {
          "MethodSignature": "*",
          "Interceptors": [ "ScoreMonitoringInterceptor" ]
        }
      ],
      "GenericArgumentTypes": [ ]
    }
  ],
  "GlobalInterceptors": [ ],
  "Key":  ""
}

And I bootstrap in Global.asax like this:

<%@Application Language='C#' Inherits="Sitecore.Web.Application" %>
<script runat="server">
    void Application_Start(object sender, EventArgs e)
    {
        try
        {
            CodeCop.Core.Cop.Intercept(new [] { new PG.CustomerOnline.Web.Areas.Customer.ScoreMonitoringInterceptor() });   
        }
        catch (Exception exc)
        {
            var reflection = exc.InnerException as System.Reflection.ReflectionTypeLoadException;
            if (reflection != null)
            {
                foreach (var ex in reflection.LoaderExceptions)
                {
                    Sitecore.Diagnostics.Log.Error("COP", ex, this);
                }
            }

            throw;
        }
    }
</script>

[Pavel Veller]

The /bin is about 120Mb across ~300 assemblies. Still spinning (it's been probably good 5-10 minutes) and I see IIS process at ~50% CPU doing something.

[Pavel Veller]

I guess I will give up right now and will wait if you guys can tell me how to diagnose / what to try. CPU is at 50%. No increase in memory footprint as it's "spinning".

[Ricardo Barbosa]

Hi Pavel,

Before we proceed with that large application, can you guarantee that CodeCop is working on a smaller demo app?

Let's filter that our first ok?

Waiting your feedback,

--Ricardo

[Pavel Veller]

I wish I had something smaller. All I do right now has to do with Sitecore CMS. A pretty large .NET web app. I am really after a few types that I want to instrument. Maybe there's a way to tell Cop to only load/inspect the types I need? Not sure how exactly "hooking into IL" works but maybe it's possible to ensure minimal bootstrapping time by limiting the "footprint"? Some of those assemblies are obfuscated, by the way. Similar to how Cop itself is obfuscated - controls flow mostly, nothing too fancy - but I wonder if those gotos get in the way of any offset calculations that you need to do to inject IL? Just thinking out loud...

[Ricardo Barbosa]

HI Pavel,

I've uploaded a small console solution that will show us if the CodeCop runtime on that machine is ok.

Please run it and tell me what result you got.

Here's the link: https://drive.google.com/file/d/0B0fAseiyUbw_NTE4U0d5aWk2bTA/view?usp=sharing

Your app is fantastic because it is huge and I will do my best to put CodeCop running inside of it, but first we have to filter this out ok?

I will tackle also the other issues one by one and see what can we do to implement what you so well suggested.

Waiting your feedback,

--Ricardo

[Ricardo Barbosa]

Hi Pavel,

After you test with the application I sent you , please try v.1.3.3 that I've just pushed to nuget where it is possible via an "AssemblyLoadConstraints" property in the JSON, specify by name the assemblies that will be inspected at runtime, which in your case, of a bin with 300 is priceless ;)

This is how you would use it to only load and inspect from bin the foo.dll and bar.dll assembly

{
  "Types": [

    {
      "TypeName": "ConsoleTests.Program, ConsoleTests",
      "Methods": [
        {
          "MethodSignature": "Test()",
          "Interceptors": [ "BeforeAfterInterceptor" ]
        }
      ],
      "GenericArgumentTypes": [ ],
      "AssemblyPath": ""
    }
  ],
  "GlobalInterceptors": [ ],
  "Key": "",
  "AssemblyLoadConstraints": ["foo", "bar"]
}

Please tell me how it went,

Thanks,

--Ricardo

[Pavel Veller]

Great! Thank you Ricardo. I will definitely try it later today, a bit busy at the moment. Thanks again and I will sure keep you posted!

[Pavel Veller]

Didn't bootstrap with 1.3.3. Same symptoms. You can see my current configuration in the last comment I posted to #10. No errors. Nothing in the event log either (the class resolution error I did see in there as well as in my own logs). I double checked that I am referencing 1.3.3. By the way, it would not be a bad idea if you were changing the assembly properties when running release builds to set FileVersion to the actual release number. You would have AssemblyVersion at 1.0.0.0 and FileVersion at 1.3.3.0. That's how we do with our product release :)

Anyway. I guess I will now try the utility you provided to see if it will tell you more about what's going on

[Pavel Veller]

@dotnetricardo, here's the results from the runtime tester:

C:\Bitbucket\playground\RuntimeTester\RuntimeTester\bin\Debug>RuntimeTester.exe
It Works!

Now, one thing I didn't tell you that may be relevant is that I am running my Windows in a VMWare VM. The host is Mac OS but everything else runs just fine so I don't think CodeCop would have an issue running on a VM.

[Ricardo Barbosa]

Hi Pavel,

Thanks for trying the basic app out. Now we are 100% sure that everything is ok with the environment.

Thanks for the version suggestion as well, that's not a miss that's intentional due to an internal constraint, although it can lead to confusion, I know. :)

The best way to track versioning is to see the .ver file you have on your %Temp%CodeCop folder. You should have a _133.ver file there.

A question, are you running the app on IIS or IIS Express?

The reason why I'm asking this is because if you are running on IIS, instead of IIS Express the deploy rules apply.

Basically, IIS runs on a non-iterative user mode, so the CodeCop directory under %Temp%/CodeCop must me manually deployed to /Windows/Temp. That may be one of the reasons for the hanging behavior.

No problem with VM environments as well.

Please tell me how it went.

P.S. - I've shared my skype contact directly with you via mail, if you want we cant try to see the problem with screen sharing.

[Pavel Veller]

I am looking into my %Temp% and I see a _132.ver. Let me see what happens if I clean it up and re-run my app with CodeCop "turned on"

[Pavel Veller]

Ok. Running my app doesn't leave anything in %Temp%. CodeCop can't finish the bootstrapping process so that's probably why. The artifacts in %Temp% were/are from the Runtime Tester utility you sent me. I deleted it, tried to access my app (after iisreset), it would hang like before with IIS worker process spinning at ~35% and nothing appears in Temp. I reset it again, ran the Runtime Tester and saw the _132.ver in there.

[Pavel Veller]

My IIS is not express. It's "full" IIS 8.5 running on Windows 8.

[Pavel Veller]

@dotnetricardo, I was playing with clr.dll and crljit.dll and I can't get them higher than 4.6.96.0. I see there's 4.6.100.0 but that update is not available for Windows 8 (https://support.microsoft.com/en-us/kb/3083186). I even went as far as uninstalling and installing .NET Framework 4.6 altogether. Downloaded the latest .exe and I still have 4.6.96.0. I see that CodeCop comes with symchk.exe that you're probably using to fetch symbols but I still can't see any traces of that in my Fiddler. Just wonder if you have any other ideas on how to troubleshoot CodeCop on my machine. The RuntimeTester reports "It Works!" but it doesn't leave the PDBs behind.

[Pavel Veller]

@dotnetricardo, also, I ran symchk.exe clr.dll and sumchk.exe clrjit.dll and I got both PDBs successfully for 4.6.96.0. I guess the 404 theory wasn't the right one. Maybe I can manually fetch the PDBs and stage the CodeCop cache? Is there a chance you can build a 1.3.x with detailed logging of what CodeCop is trying to do during bootstrapping and where exactly it gets stuck? I still have high hopes for the tool :) Let me know what you think

[Ricardo Barbosa]

Hi Pavel,

Yes, you're right. The PDB suspition is not applicable to your case. You can check here that this already happened, so I was just trying to filter if that wasn't repeating.

In fact the reason why it is running without the PDB's it because CodeCop already comes with a pre-built cache that works for your version of those dlls.

Let's do the following, I will do a special verbose debug build, just ping me today on skype for us to test it with screen sharing there ok?

Thanks!!

--Ricardo