New unit tests aren't recoginzed by unit test runner

[Nathen Drees]

I have a test file which I ran against the org with only 3 tests in it:

After these tests ran and passed, I added a fourth unit test:

When I ran the same configuration (from the top right), it didn't correctly detect that I had added a new one:

This time it tried to run TestSetup instead of the 4th test, but usually when I see this happen is just doesn't recognize the new test at all.

This time it also terminated the tests, and gave back no indication why:

I see warnings about the test in the log (attached).

[Scott Wells]

Interesting. Not a fix, but can you open the run configuration (which refreshes the selection) as an effective workaround until I can get a proper fix out?

[Nathen Drees]

It doesn't look like that works either. I opened the run configuration, didn't change anything but clicked apply again anyway, still missing the new unit tests.

[Scott Wells]

Okay, thanks for checking. I'm going to be releasing SOQL query execution in the next day or so and will then return to a few of these bugs that have been opened for a patch. I'll make sure to include this one.

[Scott Wells]

Nathen, I've reproduced this now. Can you check one thing for me in that class? Can you rename your @TestSetup-annotated method to something other than TestSetup and see if the problem goes away? I'm pretty sure the underlying issue has to do with using the annotation name as an identifier name. I should be able to resolve that, but I'd like to see if your findings match my own.

Never mind...I was able to cause it to happen even without a setup method named TestSetup. Seems to be a cache in a weird state. More debugging to do...

[Scott Wells]

Okay, I have a prospective fix that will be in the next build.

Regarding the comment "This time it also terminated the tests, and gave back no indication why", it's just saying that the one non-test method TestSetup was never reported as finished - success or failure - before the parent test class was reported as finished. If you click the filter button on the toolbar you'll see that the other test methods passed successfully.

With the fix I'll be providing, the only reason you should see this is, for example, if you added a new test method locally and then ran tests without saving/deploying first. In that case, IC would start progress on the new test method, but Force.com would never tell us anything about it before the parent class completes.

I'm hoping to get this new build out tomorrow. I want to get in a little additional testing before doing so.

[Scott Wells]

Nathen, I just uploaded 1.4.1 with the prospective fix. I'll leave this particular bug unresolved until you can provide feedback on whether it seems to address the issue. Please let me know!

[Nathen Drees]

Thanks Scott, I'll take a look as soon as I get a chance.

Regarding the running without saving, is there any way to check to see if there's dirty files before kicking off the run, and prompting the user to save first? Is that too much effort for an edge case?

[Scott Wells]

No, not too much of an edge case at all. It's something I'd like to add, but I think I'd like to put it behind a related request to perform dirty checking against local file system time stamps vs. server time stamps (might even have been your request now that I think about it).

[Scott Wells]

I'm going to resolve this and you can feel free to reopen it if it's not fixed in 1.4.1+.

[Nathen Drees]

Scott -

Finally got a chance to test this out, looks like it's still an issue in 1.4.1. Added two new unit tests to a class I had already run, and it picked up my test setup method as a test, then terminates itself again.

[Scott Wells]

Well bummer. The good news is that I've definitely addressed a few variations on this one. The bad news is that I haven't addressed them all. I'll take another look! Really sorry that it's not fully resolved yet!

[Scott Wells]

Nathen, have you verified yet whether you've reproduced this under 1.4.1.1?

[Nathen Drees]

Scott -

I don't have any updates for the plugin, and it says i'm on 1.4.1. I'm not sure if it's really on 1.4.1 and just don't show the checkin version, or what.

[Scott Wells]

Nathen, it should look like this:

If it doesn't and you're not getting prompted to upgrade from Setup>Plugins, can you explicitly download from here:

https://plugins.jetbrains.com/plugin/7831

and install? And please let me know if that's the case so I can follow up with JetBrains to understand why you're not able to upgrade from within the IDE.

[Nathen Drees]

Scott -

Here's my plugin version:

I'll try explicitly upgrading later today and let you know if this is still an issue.

[Scott Wells]

Okay. I'll ping JetBrains and see why you're not seeing the upgrade from within the IDE.

[Scott Wells]

I've contacted JetBrains about you guys not seeing the latest build. Hopefully I'll hear back something from them soon. Let me know whether this one is reproducible on 1.4.1.1 or not. Thanks!

[Nathen Drees]

Scott -

Just checked, it seems to be working in 1.4.1.1.

[Scott Wells]

Okay, I'm going to resolve this one as well. I have an internal issue tracking the update problem. Thanks!

[Nathen Drees]

Scott -

It looks like this issue has re-appeared in 1.4.2.

[Nathen Drees]

Log on 8/21/15

[Scott Wells]

Thanks, Nathen. I've had another user mention issues with unit testing in 1.4.2.0 which shouldn't have included anything around unit tests. I'm going to see if perhaps I accidentally missed the bug fixes from 1.4.1.1 when branching for 1.4.2.0. Hopefully it's something as simple (and dumb on my part!) as that. More to come soon.

[Scott Wells]

I'm not seeing anything missing from that branch. I'll spend some time on it and see if I can reproduce the issue again and/or see what's going on.

[Scott Wells]

Nathen, just a quick update. The other user's trouble with unit tests ended up being a configuration issue, so unfortunately I'm not seeing a simple smoking gun here. My guess is that perhaps I didn't completely fix this issue last time around after all. I'll definitely continue to investigate the issue and hopefully will find the ultimate root cause very soon. Sorry for the inconvenience and thanks for the continued patience!

[Scott Wells]

Nathen, can you try something for me? In the class where this is happening, can you:

1. Open structure view (CTRL+F12) and verify that the method is also missing there.
2. Use ALT+UP/ALT+DOWN to navigate up and down the body declarations a few times.
3. Open structure view again and see if the method is back. You may have to do this a few times.
4. If the method does appear in structure view, run your test.

This may seem like insanity, so let me explain. The issue I fixed last time that seemed to resolve this was with something that was cached too aggressively. I seem to have another instance of this type of issue somewhere, and I've put code in place to detect the issue that executes when structure view and go to next/previous method actions are used (primarily because that's where the downstream effects of the issue manifest in a way that I can detect them). When I detect the issue, I can ask IDEA to flush a certain cache which resolves the issue. I'm definitely still trying to find the root cause of the issue, but I wanted you to see if this might a) help you run your missing test methods; b) help me confirm this as the same cause I've observed.

Let me know!

[Nathen Drees]

Hey Scott -

I actually just ran in to this again. My unit test runner terminated itself again, and in the logs i see:

2015-09-01 09:20:17,705 [ 506130]   WARN - asedToSMTRunnerEventsConvertor - [Apex] Trying to finish not existent node: TestFinishedEvent{name='Test_ProjectControllerExtension_SelectCandidateValidationError', id=206, duration=0} 
2015-09-01 09:20:17,711 [ 506136]   WARN - asedToSMTRunnerEventsConvertor - [Apex] Trying to finish not existent node: TestFinishedEvent{name='Test_ProjectControllerExtension_TestConstructoreStageWithPerformance', id=207, duration=1000} 
2015-09-01 09:20:17,753 [ 506178]   WARN - asedToSMTRunnerEventsConvertor - [Apex] Trying to finish not existent node: TestSuiteFinishedEvent{name='Test_WebServiceMocks', id=208} 
2015-09-01 09:20:17,755 [ 506180]   WARN - asedToSMTRunnerEventsConvertor - [Apex] Unexpected running nodes: [{id=98, parentId=84, name='Test_ProjectControllerExtension_SelectCandidate', isSuite=false, state=RUNNING}, {id=100, parentId=84, name='Test_ProjectControllerExtension_TestConstructorContactToValidate', isSuite=false, state=RUNNING}] 
2015-09-01 09:25:33,147 [ 821572]  ERROR - aemon.impl.PassExecutorService - null 

I can see the methods in the code, and the correct unit tests show up when I click Ctrl + F12

However, the unit test runner is still looking for old methods that no longer exist in this case, instead of ones I added:

I tried the ALT+UP/ALT+DOWN suggestions anyway, but it didn't have any affect.

[Scott Wells]

Okay. What you have confirmed is that it's picking up old cached data, though. I'll keep digging into it and find the root cause. Thanks for following up!

[Scott Wells]

Good news. Finally found the root cause of this. Will get a fix into the next patch.

[Scott Wells]

Nathen, sorry if this is a duplicate message for you, but I sent you a couple of direct messages with info about how to download a test build with what I hope is the final fix for this issue. I'd prefer to get some feedback from you on that build before throwing out a patch to all users if possible. Thanks!

[Scott Wells]

Nathen, I hope that this is actually resolved finally in 1.5.2.0. Let me know if you still see the issue after upgrading to that version.

[Nathen Drees]

Scott -

Seems to be working as expected now. Thanks!

[Scott Wells]

Thanks for confirming! Of course, we've been here before, so don't hesitate to let me know if you see it reappear. I don't think that will be the case, though. Last time I knew there was still one instance of the stale cache issue. This time I feel pretty confident (famous last words?!) that I found the root cause.

[Jake Juby]

I'm seeing this in 1.6.7.8 on Intelli J 15.0.2

I add another test method, and I cannot run it from within the IDE, wether I'm running the entire class or just that specific method it behaves exactly as nadrees described.

[Scott Wells]

Jake, just to clarify the behavior, you have an existing Apex test class with test methods, you create a Apex unit test run configuration from it and are able to run those tests successfully, you add a new test method to that class, and then running the same configuration doesn't include that new test method. Is that correct? Just want to make sure I have clear steps to reproduce...

[Jake Juby]

Yes. Also, I cannot run that unit test method individually from the right click menu and it doesn't run if I right click the class and run the tests from there.

[Scott Wells]

Okay. Let me see what I can find.

[Scott Wells]

Jake, I just did the following steps:

1. Right-clicked on an existing test class with three test methods and chose Run MyApexTest from the context menu. All test methods ran successfully.
2. Added a new method with signature static testMethod void testUsingTestMethod() to the class and saved it.
3. Reran the test config created in the first step. Now four test methods ran successfully including the new one.
4. Added a new method with signature @IsTest static void testUsingIsTestAnnotation() to the class and saved it.
5. Reran the test config created in the first step. Now five test methods ran successfully including the two new ones.
6. Right-clicked on each of the new test methods and chose Run MyApexTest.testMethodName from the context menu. The specific test method ran successfully.

So I'm not currently reproducing the problem that you're seeing. Can you look in idea.log and see if there's anything useful, e.g., a stack trace? You can get to it from Help>Show Log in Finder/Explorer.

[Jake Juby]

Scott,

I have a test class with 28 test methods in it, I have a unit test run configuration for that test class. I add a new test method to the class, save, and run. 28 methods run, and the following lines are added to the log:

2016-01-05 09:18:30,018 [60941743] WARN - inatedcloud.client.ApexLogUtil - Found old TraceFlags that cannot be updated. Deleting them and creating a new one. 2016-01-05 09:18:47,724 [60959449] INFO - openapi.editor.impl.EditorImpl - Cache hits:24576, total requests:66559,file:///src/classes/MyTestClass.cls

[Scott Wells]

Thanks, Jake. Those are normal messages that don't indicate any errors. So that's good (no errors) and bad (no helpful information).

Ignore that last question. I just re-read your comment and you addressed exactly that. Can you provide the signature of the new test method? Feel free to remove any revealing information. I'm not concerned with the method name as much as its total signature.

[Jake Juby]

That was the original issue. At this point I cannot find a way to run a new unit test in an existing class. I've tried right clicking on the class name and running, right clicking on the method name and running, creating a new run configuration, and running an existing configuration.

When I add the 29th test method save and run the following is added to the log:

2016-01-05 09:43:10,087 [62421812] INFO - agnostic.FrequentEventDetector - Too many events posted, #3 2016-01-05 09:43:10,139 [62421864] INFO - agnostic.FrequentEventDetector - Too many events posted, #3 2016-01-05 09:43:10,184 [62421909] INFO - agnostic.FrequentEventDetector - Too many events posted, #3 2016-01-05 09:44:20,311 [62492036] INFO - agnostic.FrequentEventDetector - Too many events posted, #3 2016-01-05 09:44:20,386 [62492111] INFO - agnostic.FrequentEventDetector - Too many events posted, #3 2016-01-05 09:44:29,292 [62501017] INFO - agnostic.FrequentEventDetector - Too many events posted, #3 2016-01-05 09:44:29,339 [62501064] INFO - agnostic.FrequentEventDetector - Too many events posted, #3 2016-01-05 09:44:29,385 [62501110] INFO - agnostic.FrequentEventDetector - Too many events posted, #3 2016-01-05 09:44:53,357 [62525082] INFO - openapi.editor.impl.EditorImpl - Cache hits:4096, total requests:10873,file:///src/classes/MyTestClass.cls 2016-01-05 09:45:05,915 [62537640] WARN - inatedcloud.client.ApexLogUtil - Found old TraceFlags that cannot be updated. Deleting them and creating a new one. 2016-01-05 09:49:55,555 [62827280] INFO - agnostic.FrequentEventDetector - Too many events posted, #3 2016-01-05 09:49:55,606 [62827331] INFO - agnostic.FrequentEventDetector - Too many events posted, #3 2016-01-05 09:49:55,653 [62827378] INFO - agnostic.FrequentEventDetector - Too many events posted, #3

[Jake Juby]

@isTest
private static void generate_deletePaymentFromAlreadyCancelledItem_fieldIsNotMissing() {

[Scott Wells]

Jake, let's try one thing to rule it out. Can you rebuild your caches & indices? You can do this for just Illuminated Cloud using Settings>Illuminated Cloud (yourConnectionName)>Rebuild or for everything using File>Invalidate Caches / Restart followed by Invalidate and Restart. After that try running the test again. If that works, wonderful. If not, what I might need to do is add some diagnostic logging in the area where I figure out which test methods to run and provide you an updated build so we can get to the bottom of this. Again, very sorry for the inconvenience!

[Jake Juby]

Scott, that did the trick. I can run the test method now.

[Scott Wells]

Glad to hear, Jake, though that means I still have something somewhere allowing those to get stale. I'll investigate and see if I can pinpoint it.