New unit tests being ignored

[Steven]

Hi Scott,

An old issue started recurring again. I changed a test class by renaming 1 method and creating 1 new method, and deployed it. When trying to run the whole test class it doesn’t run the 2 new methods and mentions the old unit test, saying:

Test ignored.

Test method Batchable_ConvertLeads_Test.merge_noMatchesFound was never reported as completed.
Perhaps it doesn't exist on the server?

That is the one that has been renamed and does indeed not anymore exist on the server. Running the tests in the developer console finishes without problems.

Whatever I try, I don’t get the new unit tests to run. I’ve deleted the test configurations, deleted and re-retrieved the corresponding classes, checked if all code compiles (it does), retrieved everything again, rebuilt the offline symbol table, invalidated caches, checked the logs… I’m stuck!

Attached a log file from when I try to run a single unit test (1 of the new ones).

[Scott Wells]

Steven, what happens if you remove and recreate the Apex unit test run configuration? I think there's still an open issue with Apex class/method renames not updating references in unit test run configs.

[Steven]

Tried that, and in the run configuration it lets me select the new methods, but also the deleted ones. When running the configuration it does not run the new methods though… (And it does attempt to run the old ones)

[Scott Wells]

Okay. You mentioned that you rebuilt the OST. If you search for Batchable_ConvertLeads_Test in All Places, are there two copies of that class? It must be picking up the old method name from somewhere, and since you've (presumably) closed and re-opened the project (or even restarted the IDE) with this problem still occurring, it must be in some persistent location like the filesystem. My guess would be either a config file--likely a unit test run config--or a secondary/stale Apex class file--either in the local project or in the OST. Anything like that?

[Scott Wells]

Resolving due to lack of activity.

[Former user]

Hi Scott,

I see that I never responded here. I have run into the same issue again.
I have updated the IDE, IC2, and recreated the project in a completely different file location. Tried updating OST’s, reloading, etc.

What I just found out is that it will work when I add that method to the same class in another module.

My setup is as follows:

I have one project, ClientName, with 2 modules: Production and UAT.
In the file system they are \IdeaProjects\ClientName\Production\ and \IdeaProjects\ClientName\UAT\.

In order to get newly created tests in UAT to run, I have to add those test methods to the local Production module as well (no need to deploy), and it will run:

If I remove the test from the Production class, and run the test in UAT, I get:

The strange thing is that it usually works fine, and I use the same setup everywhere (one project with a module for each environment / stage. Am I not supposed to do it like that?)

The logs say (when running the test without having it in the Production module):

2020-12-04 11:40:58,554 [1039115] WARN - .ApexUnitTestRunProcessHandler - All test methods for TriggerHandler_DatlinqData_Test were filtered. No tests will be run.
2020-12-04 11:40:59,040 [1039601] INFO - .ApexUnitTestRunProcessHandler - Running unit tests synchronously.
2020-12-04 11:40:59,138 [1039699] WARN - .ApexUnitTestRunProcessHandler - Unexpected status code 403 returned.

[Scott Wells]

Steven, a 403 generally means that test classes/methods found locally and included by name in the unit test execution request do not exist on the server. This particular Salesforce API is an all-or-nothing proposition, so if you request that 20 test classes with 100 test methods execute, and exactly one of those isn't present on the server, you get this 403 response. Are you perhaps missing something from the server in this request?

[Steven]

Hi Scott, thanks for the swift response! And no, in fact I run a configuration that includes just this specific test method. In fact, If I run a configuration with all methods of that test class, it will return successfully with successes for all but the new test method.

I remember from a while ago (IIRC) that when I looked at the logs, it didn’t include the method at all in the request that was sent to the server. But I changed to IDEA 2020.3 and IC2 since and now the logs are less detailed somehow.

[Scott Wells]

My guess is that you perhaps lost the debug logging config when you upgraded:

http://www.illuminatedcloud.com/support/debuglogging

Add the entries for unit test execution, reproduce the issue, and feel free to attach or email the resulting logs for review.

[Steven]

That was it indeed! Full logs are attached.

So the logs say:

All test methods for TriggerHandler_DatlinqData_Test were filtered. No tests will be run. 

and then

Posting the following to runTestsSynchronous:
 {"tests":[],"skipCodeCoverage":true}
 Unexpected status code 403 returned.

‌

[Scott Wells]

Steven, the issue is that you're trying to run a test method named testCreateAccountFromDatlinqData_success but the test methods found on TriggerHandler_DatlinqData_Test are testSyncDatlinqDataToAccounts_success, testSyncDatlinqDataToAccounts_failure_incompatibleTypes, and testSyncDatlinqDataToAccounts_failure_errorOnSaving, none of which is a match for the configured unit test run configuration. Or does testCreateAccountFromDatlinqData_success actually exist on that class and IC isn't finding it properly?

[Steven]

Exactly, the latter! And once I add that method to its twin in the Production module (even just an empty method with that name, no deployment to production needed), it will work fine. So somehow it seems to scan the production version for methods, even though I either 1) right-click → Run test on the method in the editor, or 2) right-click in the Project bar on my class → Run 'TriggerHandler_…' or 3) run a test configuration from the IC toolbar in the top.

[Scott Wells]

Yeah, that's what I figured even as I was typing my previous response, hence the "Or does..." semi-afterthought question. Do you happen to have a version of that class in your OST that IC might be picking up?

I'm going to need to try to reproduce this a bit more. IC is definitely either caching something too aggressively or is picking up an older version of the test class. That won't likely happen until next week, but hopefully I can reproduce this and finally put it to rest!

[Steven]

Hi Scott, I can’t find any reference of that class in the OST, but I might be looking in the wrong place. (Under /Apex there is only EmptyStackException.cls and Stack.cls.)
I recreated this project in a different location with a fresh OST and retrieved the metadata which included the new test method, and even then it does not recognise the method.

A related issue might be that sometimes, if I navigate to failing tests for the UAT module by following the links in the log’s stack trace, it sends me to the class in the Production module.

[Scott Wells]

Steven, the stub Apex classes that are added to the OST aren't ever used for unit test execution. There is a lingering bug (that I think I may just attack this week) where if the same class exists both locally in the project and in the OST, the OST version can sometimes be linked instead of the real one, and that's likely what you're seeing in the latter.

I tried to reproduce this for a bit but wasn't successful. I obviously know/trust that it's happening to you, though. Can you please try to create a standalone reproducible scenario and document the steps? In other words, create a new test class with a (no-op?) test method, execute that class/method, add another test method, try to execute that one and it's not found. Something like that? It stinks not to be able to reproduce this...

[Caleb Weaver]

Hey there, not sure if I’m allowed to participate in someone else's bug but I’m seeing a similar issue. My apex classes are showing in OST, file references link me to the stubs rather than the local copy, and apex classes aren’t executing.

The only piece I wanted to add is that I haven’t seen this in an old metadata-style project structure, only when creating a new sfdx-style project. Hope that helps in reproduction

[Scott Wells]

Caleb, you can always join the party if it's the same or a related issue. In those situations I'll let you know if/when it's time to splinter off a separate issue.

For what you're seeing, please regenerate the OST and it will no longer include stubs for any classes that you have locally. That should fix the issue you're seeing--or at least provide a workaround for it--but I really, REALLY need to fix the underlying problem where it chooses the OST version when both are available ASAP. I'll look into that again this week.

[Caleb Weaver]

Hey Scott, thanks for the fast reply! I tried refreshing my OST and deleting the folder and recreating it but both still result in local class stubs being created in the OST.

[Scott Wells]

Caleb, that means to me that when the OST is generating, it's not finding the local versions of those classes properly to determine that it doesn't need to generate stubs for them. Can you verify that those classes are under a configured source root?

[Caleb Weaver]

Yep, I just double-checked and the classes are in a Content-Root noted in the Project Structure tab. “force-app/main/default” (where the classes are) is listed under Source folders and when I try to add the classes folder specifically it says the folder is located beneath an existing content-root

[Scott Wells]

Okay. Do you mind sending your .iml file to [e-mail] for review? It may be a few hours before I really get to look at it, by the way, and based on what I see I may request that you regenerate your OST one more time with debug logging enabled and provide those logs.

[Steven]

Hi Scott,

‌

Sorry for the lack of reply. I was assuming it was only happening at this project, given that my initial post and the update of december were for the same project.

However, it just happened to me in a unrelated project, and it seems related to removing the test class after creating it in a different module. At least that’s how I was able to reproduce it for this customer. I accidentally tried to write a new test method in the Production module, realised my mistake, cut and pasted the method in the same class in the UAT module. Steps:

1. Create test method in module A (no need to deploy it)
2. Cut test function (and save class)
3. Add method to class in different module B with the exact same class name
4. Save and deploy
5. Run test method in module B and find that it’s not running
6. Paste your function back into module A
7. Run test method in module B and find that this time it does run

Hope it helps!

If you need anything, let me know and I’ll try to respond in less then a month time next time

‌

Thanks, cheers,

Steven

[Scott Wells]

Thanks for the details, Steven. Hopefully that will allow me to get to the bottom of this once and for all. I won't get to take a close look until next week as I'm deep into Spring '21 updates this week, but I'll let you know what I find as soon as I do take a look.

[Steven]

Cool! (Will we be able to use SFDX with sandboxes? )

No worries, there’s no rush really now that I know how to prevent and to work around it.

[Scott Wells]

Support for source-tracked sandboxes has been in place since it was in early beta. Just check Use push/pull instead of deploy/retrieve/delete for the sandbox connection and IC2 will do exactly that. More information available here:

https://bitbucket.org/RoseSilverSoftware/illuminatedcloud/wiki/User_Guide/Configuring_Connections

[Steven]

Ah, great! Completely missed that. Thanks Scott!

[Scott Wells]

Steven, I just started working through your steps and want to clarify something. Are you saying that you have one project with two modules, and those modules have Apex classes with the exact same name? If so, I'm assuming that these are the same general project metadata but on different versions/branches and pointing at different orgs or something? Just trying to understand the use case.

While I agree that that project configuration should be 100% supported, realistically I can say that it's likely going to cause more issues than just test execution. There are times when an IDE action fires when there's only project context and not module context, and IC2 has to try to resolve names across the project. If that introduces an ambiguity (ignoring for a moment potential duplication in something like the OST that's managed by IC2 itself), it can result in incorrect behavior...either resolving to the wrong version of the metadata with a given name or deciding not to resolve to anything if it would be ambiguous.

So before I dive further into debugging this, let's chat through the use case. I'm not at all opposed to digging in on a multi-module project like this, but let's make sure that it's the only (and more importantly, best) way to meet the requirements.

[Steven]

Exactly, I set up my projects with a module Production and a module for each sandbox I work in. For deployments this gives me the advantage that Production is within my project connections, which is useful for deployments from sandbox to production, and I can easily diff between environments. (In fact I’ve set up git with multiple worktrees, which gives me merging and cherry-picking capabilities to push code to my production module, after which I deploy from there to production.)

I’m not sure if it is the optimal setup but in fact, I was not aware that this is not even the default way of doing things happy to hear what the IC best practice would be for non-SFDX development in sandboxes & deployments.

It’s true that sometimes there are related issues; for example I noticed that navigating to classes from test execution log stack traces sometimes leads you to the class in a different module.

[Scott Wells]

Okay, thanks for confirming. In general if you're working with multiple branches/orgs, it's best (IMO at least) to have a single branch open at any given time. Even ignoring the aforementioned IC2 shortcomings with having multiple copies of the same source in a single project, it helps to eliminate (or at least minimize) the change of confusion, ambiguity, and ultimately, mistakes when you deploy the something to the wrong org.

The primary workflow I see for this is that you have your master branch from which you release, whether that's into production, a packaging environment, or whatever, and when you're ready to do feature work, bug fixes, or whatever, you create a feature branch and a scratch org against which to work. That's all done from within the same project so that there's a single copy of the metadata present at any given time. You work against the scratch org until the feature/bug fix/whatever is complete, then begin the process of migrating that back into the master/sacred branch. Depending on your team dynamic, that's either through a pull request/peer review/CI validation process, or it's via direct merging/cherrypicking. Either way, the base JetBrains IDE's core Git functionality makes that pretty simple. When you switch branches, you switch connections as well. There are status bar widgets for selection of both, so it's generally quite quick and simple to switch environments.

If for some reason you need to perform one or more ad hoc deployments to or retrievals from another org than the one against which you're working, you can just choose that other connection in the bulk deployment/retrieval/removal dialog on a one-off basis. Similar with ad hoc anonymous Apex and SOQL query execution as well as view logs.

I'll stop there for the moment, but let me know if that doesn't make sense or leads to other questions.

[Scott Wells]

Issue tracker grooming. If this is still an issue, please feel free to reopen, ideally with a concrete reproduction scenario.