Some Unit Tests are not running

[Andreas Wölfl]

Hey,

the test runner is not able to run all unit tests. 3 of 21 are starting correctly, the rest isn't. In the logs (attached) i see a warning about unexpected running nodes. Maybe this is the cause?

I also tried to invalidate caches etc., no success :(

best regards, Andy

[Scott Wells]

Hi, Andy. Let me take a look at this tomorrow and I'll let you know what I find. What I may need to do is add some debug logging in this area to help understand what's going on a little better. If that's the case, it'll likely be Monday before I get a build out with that as I'm currently wrapping up a pretty significant change to the deploy/retrieve dialog that I'm wanting to get out as well. I'll just bundle the additional logging with that.

[Andreas Wölfl]

Hi, Scott. Thanks for your quick reply. Please let me know if you need anything to reconstruct this issue, e.g. additional debug logs.

[Scott Wells]

Unless this is a blocker for you, I'll just add more debug logging to the next build (currently planning to release tomorrow morning) and then after you update have you provide the resulting logs. Please let me know if that doesn't work for you.

[Andreas Wölfl]

Nah, it's not a blocker. Running the tests via website works fine in the meantime.

[Scott Wells]

Okay. I will be posting the new build today (1.6.8.5) that contains extensive logging around unit test execution. Once you take the update, you can add the following under Help>Configure Debug Log Settings:

#com.illuminatedcloud.intellij.unittest.ApexUnitTestRunProcessHandler

Then after you execute unit tests, your idea.log should contain quite a bit of diagnostic information. Just send that my way and I'll let you know what I find.

[Scott Wells]

I've uploaded the update, Andy. Let me know if you need assistance getting the proper diagnostic info. Thanks for the assistance in debugging this!

[Andreas Wölfl]

Hi Scott. I updated the plugin, configured the log settings, and executed the unit tests. Logs are attached. As far as i can see, the ApexUnitTestRunProcessHandler recognizes all test classes and test methods, but posts the methods of just one test class to runTestsAsynchronous (see line 124 in the logs). Maybe an error in querying the test class IDs?

[Andreas Wölfl]

new logs

[Scott Wells]

Thanks, Andy. This is very helpful. Let me see if I can figure out why I'd say these classes/methods are for testing and then wouldn't include them in the posted request. I'll also fix the debug log where I call a method a class! Ugh...

Hopefully I'll have the root cause identified and fixed shortly based on this info.

[Scott Wells]

Andy, this is an updated build with some additional diagnostic logging in the area you also identified after querying class IDs and before creating the POST body. It should help me understand why the other test classes are being excluded. Would you mind running the process one more time and providing the debug log information?

To install this, just download the attached zip and use Settings>Plugins>Install Plugin from Disk, point at the downloaded archive, and let the IDE restart. This will not prevent you from getting further updates from the standard plugin repository.

Thanks much for the assistance!

[Andreas Wölfl]

sure, one moment

[Andreas Wölfl]

aaaaaah, the classes that are in a managed beta package are excluded

[Scott Wells]

Gotcha. So I either shouldn't let you add those to the run configuration to start or I should allow you to run those when added to a run configuration. Either way that's easy to fix and removes the confusion. My take is that you shouldn't be running packaged test classes since they have to pass with adequate coverage before the package can be uploaded successfully, but I'd appreciate your thoughts on the matter..

[Andreas Wölfl]

Basically i agree that managed tests are considered as final and there is technically no need to execute them again. Although i have two comments: First, managed tests might change in the beta phase. Second, it might be helpful to run all tests in a package to estimate the total coverage (e.g. after an extension).

[Scott Wells]

I think for now I'm going to make it consistent that tests from installed managed packages (beta or released) cannot be added to run configs. There's another enhancement request asking that IC be able to compute code coverage based on actual test code (annotated with @IsTest or testMethod) vs. Salesforce's own means of computing it which can be skewed. I'd rather revisit this at that point in the context of that enhancement and opt for consistency right now.

[Scott Wells]

Andy, I have a question about how things got into this state. Right now when you create/edit an Apex unit test run configuration, it lists only the local classes that are annotated as @IsTest. As a result, I wouldn't expect test classes from installed managed packages to be available for selection. Do you know how your run config ended up with packaged test classes? Were they being managed as local source when the run config was created and have since been moved into a beta package?

[Andreas Wölfl]

oh no, i did not install a managed package. I created the test classes by myself and added them to a beta package --> they were developed and executed in the same (dev) org.

[Scott Wells]

Ah, that helps. I was thinking that status meant the class was from an installed beta managed packaged, not added to a beta managed package in the development organization. Yes, that's local source. I'll make the correct change so that it can be included in local unit testing and have that out with this morning's build. Thanks for following up!

[Andreas Wölfl]

yeah, thanks! i appreciate your help and support

[Scott Wells]

Delivered in 1.6.8.6.