Code Coverage

[Alex Jung]

On latest version (2.0.2.5?) but I don't know if this is a bug or a misconfigured IDE.

I am not able to get good coverage data, even after successfully running All Tests (and not encountering errors)

I have the "explicitly collect coverage data" checkbox checked un the "Unit Tests & Coverage" setting. But symptoms that I currently see:

• When selecting "Analyze" menu > "Show Coverage Data", I only see an option to select data from 5/21. Previously, I seem to recall it updating rather frequently
• This "Project" explorer pane, many classes have "wonky" percentage covered data, e.g. 0%
• certain new files, created during Summer '18, despite having >0% of lines covered (as per "Project" explorer, when opening up the files to edit, I get a notice that "Coverage data outdated", and doesn't show blocks of code covered/uncovered.

I mention that I may have misconfigured something b/c during the Summer '18 beta period, i experienced runaway threads, where Intellij would consume all my system memory (i want to say after running tests, but of course have no confirmation of this). My only option was to force quit, but I also think i reset Log Levels on the "run/debug configurations" dialog box, so that most are info/debug

Anyways, let me know if you need logs and at what points you'd like them...

Thanks! -alex

[Scott Wells]

Alex, take a look at the release notes here:

http://www.illuminatedcloud.com/announcements/2025releasenotes

Now by default tests run without gathering coverage for speed. You can explicitly run with coverage or configure to behave as it did previously.

Let me know if that doesn't resolve it for you.

[Scott Wells]

Feel free to reopen if it doesn't work as described.

[Alex Jung]

So, i've had the checkbox checked since i manually installed the Summer '18 beta plugin from disk. It hasn't worked since, but I just thought it was b/c everything was still in beta.

[Scott Wells]

That's odd. Does it collect coverage properly if you run with coverage explicitly? I'll also try to reproduce what you're seeing but may need you to provide some logs so that I can see what's going on.

[Alex Jung]

Steps: 1. selected small test class created in Summer '18 (most were created pre-Summer '18) 2. right-clicked and successfully ran test w/o test failures. 3. clicked Analyze > Show Coverage Data 4. Project Explorer shows "27 of 32 lines covered (84.38%)" 5. Opened file to edit

Editor pane says: "! Coverage data outdated" with a "close" link on right side of the notice.

[Scott Wells]

Thanks, Alex. Do you mind re-reproducing with debug logging enabled for unit test execution and code coverage as documented here:

http://www.illuminatedcloud.com/support/debuglogging

Also, can you verify that if you right-click and click Run with Coverage instead of Run it does in fact gather coverage (or not in which case please provide that log as well)?

[Alex Jung]

So I can confirm "Run with Coverage" does gather coverage. Is that correct behavior?

If so, out of curiosity, what is the purpose of the configuration checkbox to always collect coverage?

[Scott Wells]

That's right. This is a behavioral change that actually makes testing and coverage for Salesforce projects more consistent with the behavior of other languages in IntelliJ IDEA/WebStorm where coverage is only collected when you use Run with Coverage. I added the new configuration option for Always collect code coverage when tests are run to allow users that prefer the original behavior to still have it, but that's no longer the default behavior.

For reference, here's what I put into the release notes regarding this change:

Integrated the new skipCodeCoverage unit test execution option.

• By default code coverage is now only collected when tests are run explicitly with coverage. Tests executed using standard run or debug actions specify this new option and will not change code coverage metrics. According to Salesforce this should result in a 10-20% decrease in the time required to execute unit tests.
• This behavior can be configured as desired using the Illuminated Cloud>Configure Application>Unit Tests and Code Coverage>Always Collect Code Coverage configuration option.

[Alex Jung]

I'm sorry, not trying to be difficult/obtuse. I have the Illuminated Cloud>Configure Application>Unit Tests and Code Coverage>Always Collect Code Coverage checkbox checked. From my reading, that should enable the "original behavior" of getting coverage data when i just simply Run.

So I guess I'm asking, if I have that checkbox checked, what should I expect when I hit Run?

Regardless, I guess I can always hit/select the appropriate button/option. But the reason for this ticket was that "the original behavior" was not what was happening after I checked the "always collect code coverage" option.

[Scott Wells]

Alex, you're not being difficult or obtuse. Based on what you've described, I think you've found a bug that I need to fix. I was just describing the intended behavior starting in this new release, and I was also trying to make sure that you had an effective workaround for the moment until I can reproduce and fix that bug. I'm currently on a family holiday so my ability to turn around new releases is limited, hence the desire to ensure that users can continue working properly. Sorry if it came off any other way. Assuming I can reproduce the issue successfully, I'll plan to issue a fix in a release next mid-to-late next week. Once I do so you'll be able to configure it to work as it did before where executing tests with any of the three actions causes coverage collection.

[Alex Jung]

Thanks Scott,

You're absolutely right. I have a workaround, so I don't want to disturb your holiday any further. Enjoy!

[Scott Wells]

Alex, I'm unable to reproduce the original behavior that you were seeing. Whenever I check Always Collect Code Coverage, I see it omit the skipCodeCoverage flag when tests are run both synchronously and asynchronously. When I uncheck that, I see the skipCodeCoverage flag supplied with a value of true when tests are run via Run and Debug and omitted when run via Run With Coverage, again both sync and async.

If you're still seeing this behavior, can you add the following under Help>Debug Log Settings:

#com.illuminatedcloud.intellij.unittest.ApexUnitTestRunProcessHandler

reproduce the issue, and send me the corresponding idea.log? If you're not seeing it any longer, let me know and we'll resolve the issue. Thanks!

[Alex Jung]

Hey Scott,

Welcome back! There is definitely something funky going on still. If IC is sending the appropriate flags when the tests are run, perhaps it's the behavior after it gets the coverage results back where the issue (whether it's my understanding/assumption or bug in IC) potentially lies? I didn't have debugging turned on (i know, silly me, but i'm trying to catch up on some work myself), but I just tried, with the "Always Collect Code Coverage" checkbox checked:

1) right-clicked a test class to simply run the test. all tests passed, but coverage did not increase (stayed at 98 out of 120 lines covered). Did this several times because i was pretty sure i added code to the test class that covered the code in question (it was for a trigger and it was originally missing coverage for the before/after update flows).

after my initial right-clicking the file, i subsequently ran the tests from the toolbar by clicking on the green play button. after each run, there was no change in coverage.

2) i then right-clicked to run with coverage. immediately after the first run, it incremented coverage to 120/120 lines covered, despite there being 0 changes in code.

---

my assumption is that with the checkbox checked, simply running tests should have also changed the coverage data (which i think was the original behavior). If this is correct, I'll spend some time to try tomorrow and hopefully recreate a reproducible issue and then turn on some debugging while i'm at it :).

If that assumption is incorrect, please go ahead and close this issue.

[Scott Wells]

Alex, when Always Collect Code Coverage is checked, testing and coverage should work exactly as it always has prior to Summer '18 where coverage is gathered no matter how you run tests. It sounds like you're not seeing that behavior, though. Do you mind producing a debug log as I described in my previous post as that will tell me more about the request it's making to the server when it runs tests. I should be able to see whether it's telling the server to skip tests or not.

[Alex Jung]

Hello again Scott,

Attached a log. Did back to back test "runs". One simply clicking "Run" (where coverage data was NOT updated after tests completed) and subsequently a "Run w/Coverage" (where coverage data was updated after tests completed)

[Scott Wells]

Thanks, Alex. Here's what I see from the first test run:

2018-07-09 21:18:37,744 [296500534]   INFO - .ApexUnitTestRunProcessHandler - Running unit tests asynchronously. 
2018-07-09 21:18:37,744 [296500534]  DEBUG - .ApexUnitTestRunProcessHandler - Queueing the test run collecting coverage. 
2018-07-09 21:18:37,745 [296500535]  DEBUG - .ApexUnitTestRunProcessHandler - Posting the following to runTestsAsynchronous: 
2018-07-09 21:18:37,745 [296500535]  DEBUG - .ApexUnitTestRunProcessHandler -   {"tests":[{"className":"MailingPreferencesController_Test","testMethods":["getContactRolesForContact_test","getPublications_test","getPublicationsForBranch_test","getPreferencesForContact_test"]}],"skipCodeCoverage":false} 

and here's the second test run:

2018-07-09 21:21:01,209 [296643999]   INFO - .ApexUnitTestRunProcessHandler - Running unit tests asynchronously. 
2018-07-09 21:21:01,209 [296643999]  DEBUG - .ApexUnitTestRunProcessHandler - Queueing the test run collecting coverage. 
2018-07-09 21:21:01,210 [296644000]  DEBUG - .ApexUnitTestRunProcessHandler - Posting the following to runTestsAsynchronous: 
2018-07-09 21:21:01,210 [296644000]  DEBUG - .ApexUnitTestRunProcessHandler -   {"tests":[{"className":"MailingPreferencesController_Test","testMethods":["getContactRolesForContact_test","getPublications_test","getPublicationsForBranch_test","getPreferencesForContact_test"]}],"skipCodeCoverage":false} 

You'll notice that both runs should be collecting coverage. Both requests to the server specify skipCodeCoverage: false. Ignoring the potentially confusing double-negative, basically the server should be collecting code coverage for all production classes/methods exercised by the requested tests.

The second test run immediately loads the coverage data upon completion. Unfortunately I didn't ask you to enable debug logging for coverage, so I'm not seeing much in the way of details there. Would you please do the following for one more log that should provide us the full story:

1. Go into your org and clear all existing coverage data to ensure we're starting with a clean slate.
2. Add debug logging for code coverage as documented here.
3. Make sure that Always Collect Code Coverage is enabled. It looks like it is since the first run above specifies that coverage should be collected.
4. Run a single test class for which you know the production classes which should then have coverage.
5. After test completion, show coverage in Illuminated Cloud.

Send me the resulting log as well as the names of the production classes which should have coverage. That should tell us very clearly whether the test run resulted in coverage or not.

If at the end we're still not seeing coverage, I'll need to consult with Salesforce because I'm invoking the testing API exactly as it's documented to work. Let's not jump too far ahead until I have that new log, though.

Thanks again!

[Alex Jung]

1) Cleared test data 2) Commented out test class code (left the 2 methods with "empty" bodies) 3) Ran test class, then displayed covered (displayed 0 lines covered; as did all other classes) 4) Made 1 method "live" by removing comments; 5) Ran test, displayed lines covered (0 lines) 6) Made both methods "live" 7) Ran test with coverage (lines covered automatically refreshed to 7 of 7 lines covered)

Observations:

1) running tests w coverage automatically hides coverage in Project Explorer pane until after completion of tests. When it reappears, coverage is updated. However, simply running does not hide/unhide, and simply displays what it previously displayed.

2) running tests w/coverage updates the date/timestamp in the "Choose Coverage Suite to Display" dialog box after selecting Analyze > Show Coverage Data...

simply running tests never updates the date/timestamp.

[Scott Wells]

Alex, unfortunately the provided log only shows the end of the last coverage display and doesn't show it in context with the unit test run.

I apologize for asking a seemingly odd question, but what exactly is it that we're debugging at this point? Originally we were trying to understand how to collect coverage during test runs with builds since Summer '18. You were definitely able to get it to work using Run with Coverage, and I thought we were trying to determine whether enabling Always Collect Code Coverage also did so just like the pre-Summer '18 behavior. Based on the logs from a few hours back, it certainly looks like it should be given the request that was sent, but it looks like from your step 3 in your most recent comment that perhaps it isn't. However, without the full log I can't see exactly what did happen in that step. I just want to make sure that I'm still looking at the correct issue(s).

As for your first observation, anytime you run tests while the Coverage view is displayed, that view should be closed. This is the default behavior of the JetBrains IDEs. And as for the second observation, anytime that tests are run such that coverage is collected--whether because of Always Collect Code Coverage or Run with Coverage--the coverage suite is updated.

In order to verify the correct (or incorrect) behavior, would you mind repeating the steps I provided in my previous comment and then sending all idea.log* files that were generated as a result of that test/coverage run? In particular I just want to see the logs from a Run (not Run with Coverage) with Always Collect Code Coverage enabled followed by a Show Coverage, all against an org that starts with no coverage data.

Thanks!

[Alex Jung]

I don't think anything has changed since my original submission in terms of my issue. I've always had the "Always Collect Code Coverage" checkbox checked since I originally installed the IC from disk when the plugin was in beta. I just was never seeing pre-Summer '18 behavior.

So, to perhaps be clearer: "Running" (w/o coverage) with the "Always Collect Code Coverage" checkbox checked doesn't appear to be behaving like pre-Summer '18 in terms of what it does after i run a test. It may send the request to Salesforce identically as "Running w/coverage" as per the logs, but if so, it does not do the same things with the coverage data it gets back, at least for me (on 2 separate machines).

E.g., it does NOT close the coverage view (which you said should be default behavior). It does NOT update the coverage suite (the date/timestamp never updates after a "Run"). And it doesn't update the lines of code covered. The class being covered does not correctly show coverage data.

Like I also said previously, I have a workaround so we can close this card if you'd like. I thought I was helping you to track down a bug and did not mean to keep this card open if you felt otherwise. I am certainly not trying to pester you, lol...

[Scott Wells]

Hi, Alex. Sorry if my previous comment came out wrong...I just want to make sure that I'm debugging the right issue. I've had a few instances in the past of miscommunication with users where I unknowingly wasted time in the diagnostic process by thinking I understood what was being requested when in fact I didn't. This appears to be another instance of that.

I was under the impression that Always Collect Code Coverage plus Run was not collecting coverage for you at all, but if I understand correctly now, it does collect coverage but does not provide the exact same behavior as it did previously in terms of how/when the coverage suite is updated. If that's the case then I've been debugging the wrong thing for the past bit. Can you confirm this new understanding to be correct?

Sorry again for any misunderstanding here. I just want to make sure that we're working toward the same goal.

[Alex Jung]

I think the new understanding is correct. TBH, I don't know if it's collecting coverage or not (since I'm not privy to what's happening behind the scenes). I'm assuming so because you pointed it out in the logs in an earlier comment :).

But Intellij and/or IC is certainly acting as if no coverage was returned (i.e. no corresponding updates in the IDE to the coverage related to the test i just ran, which wasn't the case pre-summer '18).

[Scott Wells]

Gotcha. Thanks for clarifying. Let me take this information and see if I can reproduce the same behavior now that I know what to focus on. I'll let you know what I find.

[Alex Jung]

Np, hope it's reproducible :).

[Scott Wells]

I just heard about it from another user, so I'll count that as a reproduction even if it's not my own. Hopefully I'll be able to get to the bottom of it with all of the info you've provided and, if necessary, perhaps some supplementary info from the other user.

[Ashwin Gadhave]

Thank you @RoseSilverSoftware

[Scott Wells]

Okay, I think I've finally found the issue that you guys have been seeing, and it does look like a regression in the builds since Summer '18. I should have a prospective fix for it in the next build. Thanks for your patience and all the info you've provided to help understand/debug this.

[Alex Jung]

Yay! Thanks!

[Scott Wells]

Delivered in 1.8.3.5 and 2.0.2.9. Please let me know if you still see the issue after updating to those releases or higher.

[Ashwin Gadhave]

Thank you @RoseSilverSoftware