Running all unit tests in IC2 fails, if a specific unit test is part of the selection

[Roman Hentschke]

Hi

Run all Unit Tests from IC2

We’ve discovered some strange behaviour when running “all unit tests” in a bigger apex code base. This can be reproduced when we use a specific library (Nebula Logger - view on Github) and a specific unit test class of this library is also part of the “run all” unit test selection.

I am unsure, if this is a problem of the library itself - therefore a similar issue has been created here: https://github.com/jongpie/NebulaLogger/issues/433

Here is the configuration as a screenshot from a random project

(For the purpose of this screenshot, all unit tests have been selected manually)

When we now run all unit test, we get these error messages:

This is reproducible with the unit test class "Logger_Tests" selected. If we deselect it all tests run smoothly...

The corresponding class from the main branch of the repository (as a reference): https://github.com/jongpie/NebulaLogger/blob/d91e66522020759b0cae41ad27e2158007c79596/nebula-logger/core/tests/logger-engine/classes/Logger_Tests.cls

(It is kind of huuuuuge test class… maybe that’s an issue?)

As I can reproduce this in different projects, I am happy to help to figure this out and also provide further information, if needed.

Run all unit tests from Salesforce Setup

Interestingly, when you run all unit tests from Salesforce Setup it works just fine.

[Roman Hentschke]

fixed some typos.

[Scott Wells]

Hi. IC2 uses the Tooling API to execute unit tests, in particular using a payload where the test class/method names are specified in the request payload instead of class IDs. Unfortunately that API’s behavior is quite poor when a test class and/or method that doesn’t existing on the server is included in the request payload, failing the entire request instead of running everything that was found and reporting failures for things that are missing. I’ve raised this with Salesforce repeatedly but this behavior continues to be present.

I created an IDE project from that repo and of course saw the same behavior, and it seems that there are several root causes for this issue:

1. There are directories present in the repo which contain Apex test classes but which are not deployed to the server when running sfdx force:source:push against the project’s sfdx-project.json package directories, e.g., those under config/experience-cloud and, oddly, those under plugins/big-object-archiving. Any such directories should be marked in the IDE project as excluded to ensure that they are not included when building the payload for a bulk test run.
2. Even after making those changes, it appears that there’s a bug in the REST API used for unit test execution as trying to run just the tests under nebula-logger/core/tests/logger-engine/classes results in Error 500/Server Error - UNKNOWN_EXCEPTION: An unknown exception occurred. which is an internal server error on the Salesforce side. That certainly shouldn’t occur, even for an invalid request payload, so I’d need to follow up with Salesforce to see what exactly is going on under-the-hood.

I can see whether specifying classes by ID instead of by name in the API request payload helps at all, but unfortunately that’s not a perfect solution and there are several reasons that I switched IC2 from using IDs to names when name-based payload support was added. In my opinion the right solution is for Salesforce to address the two issues above and 1) make test failures not an all-or-nothing proposition; and 2) fix whatever is resulting in that internal server error. However, knowing that such fixes aren’t likely anytime soon, I’ll see if I can figure out how to work around the current behaviors and provide a better bulk test execution in IC2. I’ll report new findings here as I have them.

[Scott Wells]

As a slight update, I’ve managed to isolate this a bit to a specific test class in this repo that, when run in conjunction with pretty much any other test class, results in the same 500/Server Error failure. The specific test class is Logger_Tests. If I select all test class/methods in the repo except for that one, they execute together successfully. I haven’t yet isolated a specific method or set of methods that are the core culprit, but I’m going to spend some time trying to do that next so that I can see if I can further characterize the root cause of this issue.

UPDATE: I can reproduce this with only Logger_Tests if I tell IC2 to always run tests asynchronously. That same class passes just fine when run synchronously. That’s why I thought it had to be run in conjunction with another test class…because sync execution is only used for a single test class. So if you run Logger_Tests async with some large number of methods, it fails, and if you run any small(-ish) subset of methods from that same class async, they pass. That should be plenty to pass on to Salesforce.

[Scott Wells]

Okay, and as a further update, it seems like perhaps it has to do with a potential limit on the number of method names that can be specified together in a payload. The following REST request fails:

{
  "tests": [
    {
      "className": "Logger_Tests",
      "testMethods": [
          // 313 individual test methods listed
      ]
    },
    {
      "className": "Logger_Tests_InboundEmailHandler",
      "testMethods": [
        "it_should_save_logs_when_running_within_email_service"
      ]
    }
  ],
  "skipCodeCoverage": true
}

whereas the following payloads succeed:

{
  "tests": [
    {
      "className": "Logger_Tests",
      "testMethods": [
          // 150 individual test methods listed
      ]
    },
    {
      "className": "Logger_Tests_InboundEmailHandler",
      "testMethods": [
        "it_should_save_logs_when_running_within_email_service"
      ]
    }
  ],
  "skipCodeCoverage": true
}

and (which runs all tests methods in Logger_Tests):

{
  "tests": [
    {
      "className": "Logger_Tests"
    },
    {
      "className": "Logger_Tests_InboundEmailHandler",
      "testMethods": [
        "it_should_save_logs_when_running_within_email_service"
      ]
    }
  ],
  "skipCodeCoverage": true
}

I’ll follow up with Salesforce to see if this is a known bug and/or hard limit with a known maximum number of test methods (per-class?). That will help me know how to adjust accordingly to help IC2 users avoid running into this.

[Scott Wells]

Okay, I’ve implemented a workaround in IC2 where if it is starting an asynchronous test execution request that includes one or more classes with large numbers of individual test methods specified, it will execute those test classes in their entirety by omitting the individual test methods in its request. For the common “execute this test class” or “execute all tests” use cases, the end result is the same, but if you had a test run config that included 200 of the test class’s 300 test methods and that test was scheduled to be executed asynchronously, all 300 test methods would be executed.

With that workaround and IDE project exclusions properly configured for classes which aren’t ever deployed to the org, I’m able to run 100% of all of the Apex unit tests in Nebula_Logger:

‌

I’ll plan to include this workaround in the next official build, currently targeting Thursday of this week.

[Roman Hentschke]

Hi @Scott Wells

‌

awesome feedback in interesting detail! Thanks a lot! Looking forward to Thursday and will test it immediately. I will provide feedback.

‌

Thanks a lot for your great work!

[Scott Wells]

Okay, based on some additional characterization, the maximum number of tests per-class during async execution seems to be 149:

‌

vs.:

‌

I’m going to use that number in the initial implementation to see if it holds 100% true. If there are still failures of this nature for folks, I’ll get additional data points and adjust accordingly.

[Scott Wells]

Delivered in 2.2.5.4. Again, make sure that any test classes/methods which are going to be executed have been successfully deployed/pushed to the org. That may mean configuring directories containing those that aren't (and shouldn't be) deployed as excluded.

[Roman Hentschke]

Hi!

can confirm, that it now works like a charm! Thanks a lot!

[Scott Wells]

That’s great to hear! Thanks for confirming.