1. Scott Wells Avatar Scott Wells
  2. Illuminated Cloud
  3. IlluminatedCloud
  4. Issues

IC fails to discover test classes

Issue #2645 resolved
Vic Ricker created an issue

I have a test configuration that includes all classes for which our team is responsible. (excludes third party stuff, etc.) I ran it today, and it only ran 122 test. We have several hundred. I went into the test configuration and found that many of the test classes were missing. Same thing for all apex tests. I cleared the cache and restarted. After that, all test classes were missing. I restarted again. This time it discovered one test class. I updated to 2.3.3.1. No change.

Any idea what might be wrong?

Official response

  •  Scott Wells repo owner

    Okay, let's try this. Anyone who has the time to help, please download and install the attached test build. Don't extract the attached archive, and it can be installed using Settings | Plugins | Install plugin from disk (under the gear drop-down menu). Restart the IDE and open a project (ideally small to keep debug logging volume low) that reproduces this behavior.

    Please add the following -- and only the following -- to Help | Diagnostic Tools | Debug Log Settings:

    com.illuminatedcloud.intellij.apex.psi.impl.ApexClassDeclarationMixin
com.illuminatedcloud.intellij.apex.psi.stub.ApexClassDeclarationElementType
com.illuminatedcloud.intellij.apex.unitTest.ApexUnitTestRunConfigurationEditorForm
com.illuminatedcloud.intellij.apex.util.ApexDeclarationUtil

    Then use Invalidate Caches | Restart to rebuild the caches. It should log how it's determining and indexing local Apex test classes/methods.

    Open the Apex unit test run configuration editor and reproduce the issue where legitimate test classes/methods are missing, then send me all idea*.log* files from when the IDE restarted after Invalidate Caches to just after you reproduced the behavior in the run configuration editor. That may include numerous files due to logs rolling from all of this verbose logging.

    Hopefully that will shine a nice bright light on what's happening here!

Comments (32)

  1. Scott Wells repo owner
    • changed status to open

    Hi. I've had two other users report the same thing, and it seems to be specific to the changes introduced in 2.3.3.0 last week. Unfortunately I've been unable to reproduce it locally, so I've asked those experiencing the issue to see if they can create a standalone project that reproduces the issue. I'd ask you -- and anyone else experiencing this issue since it's obviously real -- if you could do the same thing. The sooner I'm able to reproduce it, the sooner I can fix it and issue a hotfix build. Please let me know if you're able to create a shareable reproduction and I'll take care of it.

  2. Scott Wells repo owner

    Yeah, I just went through about 8-10 different projects and tried to reproduce this issue. In all cases, all test classes/methods were displayed in the Apex unit test run configuration editor. There must be something different in the projects of those of you seeing this issue from the ones I have. Doesn’t mean there’s anything wrong with those projects, but it would definitely be useful to have a clear and consistent reproduction so I can see it for myself.

  3. Chris

    I’m working on reproducing on a smaller project that I can share, it just takes a little bit of doing to get set up.

    Throwing this out there - could it be an issue where the All_Tests run config is removed completely? I have deleted and re-created the actual test config a couple of times in attempts to have this work - without luck.

  4. Chris

    I couldn't get it to happen at first, but finally I was able to reproduce with the following repository and steps listed in the README: https://github.com/arbokrad/ic-test-plan-issue/blob/main/ic-test-plan-issue/README.md

    Steps to reproduce:
1. Open the project, connect to and deploy classes to an empty Dev Sandbox
2. Run tests using the All Tests config
    1. Expect 3 classes with 2 tests each, all passing
3. Delete the All Tests config and re-create it
4. Restart your IDE
5. Open the All Tests config
   1. Config editor no longer finds the test classes in the project
6. Open a single test class in the IDE
7. Open the All Tests config
   1. Only the class you opened is found by the config editor

    Hopefully there's not a ton of noise in these steps - I wasn't completely sure how to reproduce since I think this issue started happening for a couple of days before I noticed it was a problem. This is on a fresh dev sandbox with no other code. I can provide the details for that org if you need it for some reason - but not on the public tracker :)

    ‌

  5. Mike Lockett

    I found the same thing, but when I use Intellij 2024.1.6, things work correctly. With Intellij 2024.2.3 the tests are not available. I’m using IC 2.3.3.2 with IJ 2024.2.3 and IC 2.3.3.1 with IJ 2020.1.6

  6. Scott Wells repo owner

    Let me work through the steps that Chris has provided and see if I can reproduce it. I’ll post back here shortly…

  7. Scott Wells repo owner

    Chris, I’ve followed your steps to reproduce and unfortunately in step 5.1, the run config is still showing all tests in the project when I uncheck All Tests. Same with step 7.1 where all tests are shown, not just the one that was opened explicitly in an editor.

    The attached project is a bit strange in that, when I clone the repo, I end up with ic-test-plan-issue that contains .git and ic-test-plan-issue, and when that IDE opens the project root directory, the resulting project structure is a bit strange.

    Is that properly representative of your project structure, or should the project root effectively be ic-test-plan-issue/ic-test-plan-issue?

    UPDATE: For what it’s worth, I just opened the IDE project at the nested directory so that it properly used the existing .iml file and IlluminatedCloud directory, and I’m still unable to see the described issue after following the provided steps.

  8. Chris

    Sorry I just threw the project structure into the repo and didnt look at it twice to catch that. Its intended to only be the one directory level, not two, and is so locally for me.

    Edit: what else can I provide here that could help troubleshoot?

  9. Scott Wells repo owner

    Okay. No issues. As I mentioned in the update above, I created a project treating it that way, and I still don’t see the issue.

    In one of the projects where you’re seeing this, can you please try using File | Invalidate Caches and see if the same behavior is still present after the IDE restarts and reindexes? I’m 99% sure that I already asked on of the other users who was seeing this to do the same, and the issue was still present after doing so, but let’s try just in case…

  10. Chris

    Unfortunately File | Invalidate Caches did not fix this issue

  11. Scott Wells repo owner

    Yeah, I didn’t expect that it would, but where I’m still unable to reproduce it locally, I’m trying to rule out the simple stuff. Thanks for confirming.

    What I do know is that this issue was introduced in 2.3.3.0 where I added caching/indexing of key modifiers, annotations, etc., so that they don’t have to be recalculated each and every time. In particular for this issue, my guess is that it’s due to some unexpected (and evidently inconsistent) behavior around the @IsTest annotation and/or testMethod modifier.

    I’ll instrument a test build with sufficient debug logging so that I can hopefully see what’s happening. I’ll attach it here shortly with instructions on how to use it.

  12. Chris

    If it helps, I am exclusively using the @IsTest annotation for tests.

    Thanks as always for quickly hopping on this stuff!

  13. Scott Wells repo owner

    Okay, let's try this. Anyone who has the time to help, please download and install the attached test build. Don't extract the attached archive, and it can be installed using Settings | Plugins | Install plugin from disk (under the gear drop-down menu). Restart the IDE and open a project (ideally small to keep debug logging volume low) that reproduces this behavior.

    Please add the following -- and only the following -- to Help | Diagnostic Tools | Debug Log Settings:

    com.illuminatedcloud.intellij.apex.psi.impl.ApexClassDeclarationMixin
com.illuminatedcloud.intellij.apex.psi.stub.ApexClassDeclarationElementType
com.illuminatedcloud.intellij.apex.unitTest.ApexUnitTestRunConfigurationEditorForm
com.illuminatedcloud.intellij.apex.util.ApexDeclarationUtil

    Then use Invalidate Caches | Restart to rebuild the caches. It should log how it's determining and indexing local Apex test classes/methods.

    Open the Apex unit test run configuration editor and reproduce the issue where legitimate test classes/methods are missing, then send me all idea*.log* files from when the IDE restarted after Invalidate Caches to just after you reproduced the behavior in the run configuration editor. That may include numerous files due to logs rolling from all of this verbose logging.

    Hopefully that will shine a nice bright light on what's happening here!

  14. Chris

    I have been trying for the last hour-ish to reproduce this on the new build and so far I have been unsuccessful in my attempts. Both the smaller project and the original where I was seeing this problem seem to be finding all the tests they should, each time, despite the same steps (and having left them in the broken state to begin with).

    This upgraded me from 2.3.3.0 (which I mentioned in the groups thread but not here) to 2.3.3.1, so possibly this was fixed already, even accidentally?

  16. Chris

    I swapped to the official build and I still cant reproduce. Frustrating - I spent the morning able to reliably make this happen and now I cant at all.

    What I was and am no longer seeing is the dialog for Finding Unit Test Classes was completely empty before, and now I’m seeing class names show up in there as its churning through them. And then of course after that, the test run dialog has those classes listed and all seems fine with the world.

    ‌

  17. Scott Wells repo owner

    Okay. Well, the good news is that there are a few others here who have been seeing this. Hopefully one of them can install the attached build and capture this as it’s happening. My guess is that it’s not resolved in 2.3.3.1 but rather whatever conditions were causing it in your project(s) originally are no longer present so, like me, you’re no longer seeing it.

    For now stay on the official build. If you see it recur, swap to the attached build and see if you can capture a diagnostic log. Either way, thanks for taking the time to help characterize it.

  18. Tony White

    Installed the latest update, definitely still seeing this issue on the official build after doing a rebuild of the OST, will try switching to the debug build…

  19. Atanas Vasilev

    I can confirm that the issue was not fixed by 2.3.3.1, as I’m using that version and it still manifested this morning. It was first manifested prior to me updating the plugin to 2.3.3.1, but unfortunately I cannot tell which version I had before - I just can’t remember when I had last updated the IC plugin before this morning.

  21. Chris

    RE: Tony’s comment - I was not able to use the workaround described to get different results. It seemed like IC couldnt find the tests either via All Tests or by unselecting that and checking off all the tests in the org. The major giveaway was when the total test number kept dropping from run to run.

  22. Tony White

    @chris - Is it Test Methods or Test Classes that are different?
    For me in 2.3.3.0 I would see who test classes gone, instead of seeing 800+ test classes, I would see about 10 or so.

  23. Tony White

    Note - my note was not a work around, but trying to get the list of missing classes as we have so many test classes it was difficult to identify what was missing

  24. Chris

    That's fair - I would see entire classes go, not individual tests, so I think that lines up with your experience.

  25. Scott Wells repo owner

    I know that some people are no longer seeing this issue, but I feel like that’s probably luck rather than a true “fix”. For those who are still seeing it, if you have a few minutes to provide diagnostic information, please install the attached build and follow the instructions in the pinned comment, then send over those logs for review.

  26. Jeroen Binnekamp

    I can confirm that after installing 2.3.3.1 and then a Full Regenerate of the Offline Symbol Table fixed it for me!

  27. Vic Ricker reporter

    Regenerating the Offline Symbol Table seems to have worked for me. I had to restart the IDE immediately after. The first time I tried, without doing that, it didn’t work.

    Thanks!

  28. Scott Wells repo owner

    Hmmmm…so it sounds to me like it was just a matter of rebuilding caches/indices since that happens automatically at the end of an OST generation. Restarting the IDE (or closing/opening the project) may also clean up some lingering retained references that were causing some issues to continue to occur.

    But if I understand correctly, it sounds like everyone who was seeing this issue is in good shape now on either the 2.3.3.1 official build or the attached version of the same build with addiitonal diagnostic logging. Is that correct, or are there still some folks still seeing this issue?

  29. Scott Wells repo owner

    Prospective fix/improvement delivered in 2.3.3.2. If you’re still seeing this issue in 2.3.3.2, please first use File | Invalidate Caches and see if that resolves it. If it doesn’t, please let me know and I’ll attach a new build with diagnostic logging based on 2.3.3.2 to see if we can capture this behavior as it happens.

  30. Scott Wells repo owner

    Is anyone here still having this issue? If not, I’m going to resolve it shortly, but if so, let’s get some diagnostic logs and figure out what’s going on.

  31. Chris

    Scott,

    I havent been able to reproduce this, even accidentally, since updating to 2.3.3.1 and it seems to still be ok on 2.3.3.2. Wish I could give you those logs!

    ‌

  32. Scott Wells repo owner

    Okay. I'm going to resolve this, but if anyone is still seeing it -- or starts to see it -- please reopen and we'll get diagnostics.

  33. Log in to comment
Assignee
–
Type
bug
Priority
minor
Status
resolved
Component
Apex Unit Testing
Version
2.3.3.0
Votes
1
Watchers
5
Jira: the preferred issue tracker for Bitbucket. Join the team!