IC fails to discover test classes

[Vic Ricker]

I have a test configuration that includes all classes for which our team is responsible. (excludes third party stuff, etc.) I ran it today, and it only ran 122 test. We have several hundred. I went into the test configuration and found that many of the test classes were missing. Same thing for all apex tests. I cleared the cache and restarted. After that, all test classes were missing. I restarted again. This time it discovered one test class. I updated to 2.3.3.1. No change.

Any idea what might be wrong?

[Scott Wells]

Hi. I've had two other users report the same thing, and it seems to be specific to the changes introduced in 2.3.3.0 last week. Unfortunately I've been unable to reproduce it locally, so I've asked those experiencing the issue to see if they can create a standalone project that reproduces the issue. I'd ask you -- and anyone else experiencing this issue since it's obviously real -- if you could do the same thing. The sooner I'm able to reproduce it, the sooner I can fix it and issue a hotfix build. Please let me know if you're able to create a shareable reproduction and I'll take care of it.

[Scott Wells]

Yeah, I just went through about 8-10 different projects and tried to reproduce this issue. In all cases, all test classes/methods were displayed in the Apex unit test run configuration editor. There must be something different in the projects of those of you seeing this issue from the ones I have. Doesn’t mean there’s anything wrong with those projects, but it would definitely be useful to have a clear and consistent reproduction so I can see it for myself.

[Chris]

I’m working on reproducing on a smaller project that I can share, it just takes a little bit of doing to get set up.

Throwing this out there - could it be an issue where the All_Tests run config is removed completely? I have deleted and re-created the actual test config a couple of times in attempts to have this work - without luck.

[Chris]

I couldn't get it to happen at first, but finally I was able to reproduce with the following repository and steps listed in the README: https://github.com/arbokrad/ic-test-plan-issue/blob/main/ic-test-plan-issue/README.md

Steps to reproduce:
1. Open the project, connect to and deploy classes to an empty Dev Sandbox
2. Run tests using the All Tests config
    1. Expect 3 classes with 2 tests each, all passing
3. Delete the All Tests config and re-create it
4. Restart your IDE
5. Open the All Tests config
   1. Config editor no longer finds the test classes in the project
6. Open a single test class in the IDE
7. Open the All Tests config
   1. Only the class you opened is found by the config editor

Hopefully there's not a ton of noise in these steps - I wasn't completely sure how to reproduce since I think this issue started happening for a couple of days before I noticed it was a problem. This is on a fresh dev sandbox with no other code. I can provide the details for that org if you need it for some reason - but not on the public tracker :)

‌

[Mike Lockett]

I found the same thing, but when I use Intellij 2024.1.6, things work correctly. With Intellij 2024.2.3 the tests are not available. I’m using IC 2.3.3.2 with IJ 2024.2.3 and IC 2.3.3.1 with IJ 2020.1.6

[Scott Wells]

Let me work through the steps that Chris has provided and see if I can reproduce it. I’ll post back here shortly…

[Scott Wells]

Chris, I’ve followed your steps to reproduce and unfortunately in step 5.1, the run config is still showing all tests in the project when I uncheck All Tests. Same with step 7.1 where all tests are shown, not just the one that was opened explicitly in an editor.

The attached project is a bit strange in that, when I clone the repo, I end up with ic-test-plan-issue that contains .git and ic-test-plan-issue, and when that IDE opens the project root directory, the resulting project structure is a bit strange.

Is that properly representative of your project structure, or should the project root effectively be ic-test-plan-issue/ic-test-plan-issue?

UPDATE: For what it’s worth, I just opened the IDE project at the nested directory so that it properly used the existing .iml file and IlluminatedCloud directory, and I’m still unable to see the described issue after following the provided steps.

[Chris]

Sorry I just threw the project structure into the repo and didnt look at it twice to catch that. Its intended to only be the one directory level, not two, and is so locally for me.

Edit: what else can I provide here that could help troubleshoot?

[Scott Wells]

Okay. No issues. As I mentioned in the update above, I created a project treating it that way, and I still don’t see the issue.

In one of the projects where you’re seeing this, can you please try using File | Invalidate Caches and see if the same behavior is still present after the IDE restarts and reindexes? I’m 99% sure that I already asked on of the other users who was seeing this to do the same, and the issue was still present after doing so, but let’s try just in case…

[Chris]

Unfortunately File | Invalidate Caches did not fix this issue

[Scott Wells]

Yeah, I didn’t expect that it would, but where I’m still unable to reproduce it locally, I’m trying to rule out the simple stuff. Thanks for confirming.

What I do know is that this issue was introduced in 2.3.3.0 where I added caching/indexing of key modifiers, annotations, etc., so that they don’t have to be recalculated each and every time. In particular for this issue, my guess is that it’s due to some unexpected (and evidently inconsistent) behavior around the @IsTest annotation and/or testMethod modifier.

I’ll instrument a test build with sufficient debug logging so that I can hopefully see what’s happening. I’ll attach it here shortly with instructions on how to use it.

[Chris]

If it helps, I am exclusively using the @IsTest annotation for tests.

Thanks as always for quickly hopping on this stuff!

[Scott Wells]

Okay, let's try this. Anyone who has the time to help, please download and install the attached test build. Don't extract the attached archive, and it can be installed using Settings | Plugins | Install plugin from disk (under the gear drop-down menu). Restart the IDE and open a project (ideally small to keep debug logging volume low) that reproduces this behavior.

Please add the following -- and only the following -- to Help | Diagnostic Tools | Debug Log Settings:

com.illuminatedcloud.intellij.apex.psi.impl.ApexClassDeclarationMixin
com.illuminatedcloud.intellij.apex.psi.stub.ApexClassDeclarationElementType
com.illuminatedcloud.intellij.apex.unitTest.ApexUnitTestRunConfigurationEditorForm
com.illuminatedcloud.intellij.apex.util.ApexDeclarationUtil

Then use Invalidate Caches | Restart to rebuild the caches. It should log how it's determining and indexing local Apex test classes/methods.

Open the Apex unit test run configuration editor and reproduce the issue where legitimate test classes/methods are missing, then send me all idea*.log* files from when the IDE restarted after Invalidate Caches to just after you reproduced the behavior in the run configuration editor. That may include numerous files due to logs rolling from all of this verbose logging.

Hopefully that will shine a nice bright light on what's happening here!

[Chris]

I have been trying for the last hour-ish to reproduce this on the new build and so far I have been unsuccessful in my attempts. Both the smaller project and the original where I was seeing this problem seem to be finding all the tests they should, each time, despite the same steps (and having left them in the broken state to begin with).

This upgraded me from 2.3.3.0 (which I mentioned in the groups thread but not here) to 2.3.3.1, so possibly this was fixed already, even accidentally?

[Scott Wells]

I’d be very surprised if 2.3.3.1 fixed this as there were no changes in any area that should affect how IC2 determines whether a class/method is for testing. Are you not seeing this on the attached build? If not, can you switch to the official build:

https://plugins.jetbrains.com/plugin/10253-illuminated-cloud-2/edit/versions/stable/613386

and see whether the problem comes back?

[Chris]

I swapped to the official build and I still cant reproduce. Frustrating - I spent the morning able to reliably make this happen and now I cant at all.

What I was and am no longer seeing is the dialog for Finding Unit Test Classes was completely empty before, and now I’m seeing class names show up in there as its churning through them. And then of course after that, the test run dialog has those classes listed and all seems fine with the world.

‌

[Scott Wells]

Okay. Well, the good news is that there are a few others here who have been seeing this. Hopefully one of them can install the attached build and capture this as it’s happening. My guess is that it’s not resolved in 2.3.3.1 but rather whatever conditions were causing it in your project(s) originally are no longer present so, like me, you’re no longer seeing it.

For now stay on the official build. If you see it recur, swap to the attached build and see if you can capture a diagnostic log. Either way, thanks for taking the time to help characterize it.

[Tony White]

Installed the latest update, definitely still seeing this issue on the official build after doing a rebuild of the OST, will try switching to the debug build…

[Atanas Vasilev]

I can confirm that the issue was not fixed by 2.3.3.1, as I’m using that version and it still manifested this morning. It was first manifested prior to me updating the plugin to 2.3.3.1, but unfortunately I cannot tell which version I had before - I just can’t remember when I had last updated the IC plugin before this morning.

[Tony White]

FYI - for me it broke after 2.3.2.9 - https://plugins.jetbrains.com/plugin/10253-illuminated-cloud-2/edit/versions/stable/602262
So 2.3.2.9 worked ok.

Now what I have done is create a run configuration with all the test selected (just the ‘All’ not ‘All Tests’), Store that as a project file.
Then repeat that for the broken version and compare the files to get the differences…

[Chris]

RE: Tony’s comment - I was not able to use the workaround described to get different results. It seemed like IC couldnt find the tests either via All Tests or by unselecting that and checking off all the tests in the org. The major giveaway was when the total test number kept dropping from run to run.

[Tony White]

@chris - Is it Test Methods or Test Classes that are different?
For me in 2.3.3.0 I would see who test classes gone, instead of seeing 800+ test classes, I would see about 10 or so.

[Tony White]

Note - my note was not a work around, but trying to get the list of missing classes as we have so many test classes it was difficult to identify what was missing

[Chris]

That's fair - I would see entire classes go, not individual tests, so I think that lines up with your experience.

[Scott Wells]

I know that some people are no longer seeing this issue, but I feel like that’s probably luck rather than a true “fix”. For those who are still seeing it, if you have a few minutes to provide diagnostic information, please install the attached build and follow the instructions in the pinned comment, then send over those logs for review.

[Jeroen Binnekamp]

I can confirm that after installing 2.3.3.1 and then a Full Regenerate of the Offline Symbol Table fixed it for me!

[Vic Ricker]

Regenerating the Offline Symbol Table seems to have worked for me. I had to restart the IDE immediately after. The first time I tried, without doing that, it didn’t work.

Thanks!

[Scott Wells]

Hmmmm…so it sounds to me like it was just a matter of rebuilding caches/indices since that happens automatically at the end of an OST generation. Restarting the IDE (or closing/opening the project) may also clean up some lingering retained references that were causing some issues to continue to occur.

But if I understand correctly, it sounds like everyone who was seeing this issue is in good shape now on either the 2.3.3.1 official build or the attached version of the same build with addiitonal diagnostic logging. Is that correct, or are there still some folks still seeing this issue?

[Scott Wells]

Prospective fix/improvement delivered in 2.3.3.2. If you’re still seeing this issue in 2.3.3.2, please first use File | Invalidate Caches and see if that resolves it. If it doesn’t, please let me know and I’ll attach a new build with diagnostic logging based on 2.3.3.2 to see if we can capture this behavior as it happens.

[Scott Wells]

Is anyone here still having this issue? If not, I’m going to resolve it shortly, but if so, let’s get some diagnostic logs and figure out what’s going on.

[Chris]

Scott,

I havent been able to reproduce this, even accidentally, since updating to 2.3.3.1 and it seems to still be ok on 2.3.3.2. Wish I could give you those logs!

‌

[Scott Wells]

Okay. I'm going to resolve this, but if anyone is still seeing it -- or starts to see it -- please reopen and we'll get diagnostics.