IC fails to discover test classes

[Vic Ricker]

I have a test configuration that includes all classes for which our team is responsible. (excludes third party stuff, etc.) I ran it today, and it only ran 122 test. We have several hundred. I went into the test configuration and found that many of the test classes were missing. Same thing for all apex tests. I cleared the cache and restarted. After that, all test classes were missing. I restarted again. This time it discovered one test class. I updated to 2.3.3.1. No change.

Any idea what might be wrong?

[Scott Wells]

Hi. I've had two other users report the same thing, and it seems to be specific to the changes introduced in 2.3.3.0 last week. Unfortunately I've been unable to reproduce it locally, so I've asked those experiencing the issue to see if they can create a standalone project that reproduces the issue. I'd ask you -- and anyone else experiencing this issue since it's obviously real -- if you could do the same thing. The sooner I'm able to reproduce it, the sooner I can fix it and issue a hotfix build. Please let me know if you're able to create a shareable reproduction and I'll take care of it.

[Scott Wells]

Yeah, I just went through about 8-10 different projects and tried to reproduce this issue. In all cases, all test classes/methods were displayed in the Apex unit test run configuration editor. There must be something different in the projects of those of you seeing this issue from the ones I have. Doesn’t mean there’s anything wrong with those projects, but it would definitely be useful to have a clear and consistent reproduction so I can see it for myself.

[Chris]

I’m working on reproducing on a smaller project that I can share, it just takes a little bit of doing to get set up.

Throwing this out there - could it be an issue where the All_Tests run config is removed completely? I have deleted and re-created the actual test config a couple of times in attempts to have this work - without luck.

[Chris]

I couldn't get it to happen at first, but finally I was able to reproduce with the following repository and steps listed in the README: https://github.com/arbokrad/ic-test-plan-issue/blob/main/ic-test-plan-issue/README.md

Steps to reproduce:
1. Open the project, connect to and deploy classes to an empty Dev Sandbox
2. Run tests using the All Tests config
    1. Expect 3 classes with 2 tests each, all passing
3. Delete the All Tests config and re-create it
4. Restart your IDE
5. Open the All Tests config
   1. Config editor no longer finds the test classes in the project
6. Open a single test class in the IDE
7. Open the All Tests config
   1. Only the class you opened is found by the config editor

Hopefully there's not a ton of noise in these steps - I wasn't completely sure how to reproduce since I think this issue started happening for a couple of days before I noticed it was a problem. This is on a fresh dev sandbox with no other code. I can provide the details for that org if you need it for some reason - but not on the public tracker :)

‌

[Mike Lockett]

I found the same thing, but when I use Intellij 2024.1.6, things work correctly. With Intellij 2024.2.3 the tests are not available. I’m using IC 2.3.3.2 with IJ 2024.2.3 and IC 2.3.3.1 with IJ 2020.1.6

[Scott Wells]

Let me work through the steps that Chris has provided and see if I can reproduce it. I’ll post back here shortly…

[Scott Wells]

Chris, I’ve followed your steps to reproduce and unfortunately in step 5.1, the run config is still showing all tests in the project when I uncheck All Tests. Same with step 7.1 where all tests are shown, not just the one that was opened explicitly in an editor.

The attached project is a bit strange in that, when I clone the repo, I end up with ic-test-plan-issue that contains .git and ic-test-plan-issue, and when that IDE opens the project root directory, the resulting project structure is a bit strange.

Is that properly representative of your project structure, or should the project root effectively be ic-test-plan-issue/ic-test-plan-issue?

UPDATE: For what it’s worth, I just opened the IDE project at the nested directory so that it properly used the existing .iml file and IlluminatedCloud directory, and I’m still unable to see the described issue after following the provided steps.

[Chris]

Sorry I just threw the project structure into the repo and didnt look at it twice to catch that. Its intended to only be the one directory level, not two, and is so locally for me.

Edit: what else can I provide here that could help troubleshoot?

[Scott Wells]

Okay. No issues. As I mentioned in the update above, I created a project treating it that way, and I still don’t see the issue.

In one of the projects where you’re seeing this, can you please try using File | Invalidate Caches and see if the same behavior is still present after the IDE restarts and reindexes? I’m 99% sure that I already asked on of the other users who was seeing this to do the same, and the issue was still present after doing so, but let’s try just in case…

[Chris]

Unfortunately File | Invalidate Caches did not fix this issue

[Scott Wells]

Yeah, I didn’t expect that it would, but where I’m still unable to reproduce it locally, I’m trying to rule out the simple stuff. Thanks for confirming.

What I do know is that this issue was introduced in 2.3.3.0 where I added caching/indexing of key modifiers, annotations, etc., so that they don’t have to be recalculated each and every time. In particular for this issue, my guess is that it’s due to some unexpected (and evidently inconsistent) behavior around the @IsTest annotation and/or testMethod modifier.

I’ll instrument a test build with sufficient debug logging so that I can hopefully see what’s happening. I’ll attach it here shortly with instructions on how to use it.

[Chris]

If it helps, I am exclusively using the @IsTest annotation for tests.

Thanks as always for quickly hopping on this stuff!

[Scott Wells]

Okay, let's try this. Anyone who has the time to help, please download and install the attached test build. Don't extract the attached archive, and it can be installed using Settings | Plugins | Install plugin from disk (under the gear drop-down menu). Restart the IDE and open a project (ideally small to keep debug logging volume low) that reproduces this behavior.

Please add the following -- and only the following -- to Help | Diagnostic Tools | Debug Log Settings:

com.illuminatedcloud.intellij.apex.psi.impl.ApexClassDeclarationMixin
com.illuminatedcloud.intellij.apex.psi.stub.ApexClassDeclarationElementType
com.illuminatedcloud.intellij.apex.unitTest.ApexUnitTestRunConfigurationEditorForm
com.illuminatedcloud.intellij.apex.util.ApexDeclarationUtil

Then use Invalidate Caches | Restart to rebuild the caches. It should log how it's determining and indexing local Apex test classes/methods.

Open the Apex unit test run configuration editor and reproduce the issue where legitimate test classes/methods are missing, then send me all idea*.log* files from when the IDE restarted after Invalidate Caches to just after you reproduced the behavior in the run configuration editor. That may include numerous files due to logs rolling from all of this verbose logging.

Hopefully that will shine a nice bright light on what's happening here!