Running "All Tests" fails with "500/Server Error - UNKNOWN_EXCEPTION: An unknown exception occurred"

[Vladyslav Petrovych]

When I try to run all tests from WebStorm with Illuminated Cloud, it fails with unknown error.

Running just 1 specific test seems work fine.

[Scott Wells]

Hi. Can you please enable debug logging for both unit testing and Apex debug logs, reproduce the issue, and attach the new idea.log for review?

[Vladyslav Petrovych]

@Scott Wells Please let me know if this new one works for you.

[Scott Wells]

Thank you. That helped a bit in that I can now see the server returning a 500 in response to the REST request to run tests:

2022-04-19 16:38:11,809 [ 905568]  DEBUG - .ApexUnitTestRunProcessHandler - Posting the following to runTestsAsynchronous:
2022-04-19 16:38:11,809 [ 905568]  DEBUG - .ApexUnitTestRunProcessHandler -   {"tests":[{...}],"skipCodeCoverage":true}
2022-04-19 16:38:12,800 [ 906559]   WARN - .ApexUnitTestRunProcessHandler - Unexpected status code 500 returned.

However, since the message for this Salesforce internal server error is just "An unknown exception occurred", all I know is that the REST request to queue an asynchronous test run caused some type of unhandled exception in the Salesforce server.

My guess is that you have some test class and/or method locally that doesn't exist in the server, though that usually results in a more obvious 403/Forbidden HTTP response code. I'd recommend that you use IC2's Compare with Server (or Retrieve for Merge) action to reconcile your local Apex test classes/methods with those that have been deployed to the server. You likely need to deploy some of your test classes to the server to get things into a consistent state and then this internal error won't happen any longer.

Apologies that I can't provide a more precise answer, but when things start to fail inside the Salesforce servers in such an opaque manner, the best I can do is differential diagnosis to try to figure out what might be going on and how to work around it.

[Vladyslav Petrovych]

This is a new scratch org that I have deployed to and tried to run tests. Just checking Compare with server and there is no differences (except whitespace of end line)

From my trial and error so far, this could be an issue due to the amount of tests. When I run them one by one, it seems to work, except the biggest one. It contains around 400 tests. When I run, it tries to run, but then fails with “read time out”. Not sure if this is related to original issue. I’m going to attach another log to show this.

I can provide access to the org for you if that would help.

‌

[Vladyslav Petrovych]

  <div class="preview-container wiki-content"><!-- loaded via ajax --></div>
  <div class="mask"></div>
</div>

</div> </form>

[Scott Wells]

I can see the timeout you mentioned:

2022-04-19 20:45:50,864 [  90280]  DEBUG - .ApexUnitTestRunProcessHandler - Posting the following to runTestsSynchronous:
2022-04-19 20:45:50,864 [  90280]  DEBUG - .ApexUnitTestRunProcessHandler -   {"tests":[{"className":"ExampleNumbersTest","testMethods":["testFixedLine0_10","testFixedLine10_20","testFixedLine20_30","testFixedLine30_40","testFixedLine40_50","testFixedLine50_60","testFixedLine60_70","testFixedLine70_80","testFixedLine80_90","testFixedLine90_100","testFixedLine100_110","testFixedLine110_120","testFixedLine120_130","testFixedLine130_140","testFixedLine140_150","testFixedLine150_160","testFixedLine160_170","testFixedLine170_180","testFixedLine180_190","testFixedLine190_200","testFixedLine200_210","testFixedLine210_220","testFixedLine220_230","testFixedLine230_240","testFixedLine240_245","testMobile0_10","testMobile10_20","testMobile20_30","testMobile30_40","testMobile40_50","testMobile50_60","testMobile60_70","testMobile70_80","testMobile80_90","testMobile90_100","testMobile100_110","testMobile110_120","testMobile120_130","testMobile130_140","testMobile140_150","testMobile150_160","testMobile160_170","testMobile170_180","testMobile180_190","testMobile190_200","testMobile200_210","testMobile210_220","testMobile220_230","testMobile230_240","testMobile240_245","testTollFree0_10","testTollFree10_20","testTollFree20_30","testTollFree30_40","testTollFree40_50","testTollFree50_60","testTollFree60_70","testTollFree70_80","testTollFree80_90","testTollFree90_100","testTollFree100_110","testTollFree110_120","testTollFree120_130","testTollFree130_140","testTollFree140_150","testTollFree150_160","testTollFree160_170","testTollFree170_180","testTollFree180_190","testTollFree190_200","testTollFree200_210","testTollFree210_220","testTollFree220_230","testTollFree230_240","testTollFree240_245","testPremiumRate0_10","testPremiumRate10_20","testPremiumRate20_30","testPremiumRate30_40","testPremiumRate40_50","testPremiumRate50_60","testPremiumRate60_70","testPremiumRate70_80","testPremiumRate80_90","testPremiumRate90_100","testPremiumRate100_110","testPremiumRate110_120","testPremiumRate120_130","testPremiumRate130_140","testPremiumRate140_150","testPremiumRate150_160","testPremiumRate160_170","testPremiumRate170_180","testPremiumRate180_190","testPremiumRate190_200","testPremiumRate200_210","testPremiumRate210_220","testPremiumRate220_230","testPremiumRate230_240","testPremiumRate240_245","testVoip0_10","testVoip10_20","testVoip20_30","testVoip30_40","testVoip40_50","testVoip50_60","testVoip60_70","testVoip70_80","testVoip80_90","testVoip90_100","testVoip100_110","testVoip110_120","testVoip120_130","testVoip130_140","testVoip140_150","testVoip150_160","testVoip160_170","testVoip170_180","testVoip180_190","testVoip190_200","testVoip200_210","testVoip210_220","testVoip220_230","testVoip230_240","testVoip240_245","testPager0_10","testPager10_20","testPager20_30","testPager30_40","testPager40_50","testPager50_60","testPager60_70","testPager70_80","testPager80_90","testPager90_100","testPager100_110","testPager110_120","testPager120_130","testPager130_140","testPager140_150","testPager150_160","testPager160_170","testPager170_180","testPager180_190","testPager190_200","testPager200_210","testPager210_220","testPager220_230","testPager230_240","testPager240_245","testUan0_10","testUan10_20","testUan20_30","testUan30_40","testUan40_50","testUan50_60","testUan60_70","testUan70_80","testUan80_90","testUan90_100","testUan100_110","testUan110_120","testUan120_130","testUan130_140","testUan140_150","testUan150_160","testUan160_170","testUan170_180","testUan180_190","testUan190_200","testUan200_210","testUan210_220","testUan220_230","testUan230_240","testUan240_245","testVoicemail0_10","testVoicemail10_20","testVoicemail20_30","testVoicemail30_40","testVoicemail40_50","testVoicemail50_60","testVoicemail60_70","testVoicemail70_80","testVoicemail80_90","testVoicemail90_100","testVoicemail100_110","testVoicemail110_120","testVoicemail120_130","testVoicemail130_140","testVoicemail140_150","testVoicemail150_160","testVoicemail160_170","testVoicemail170_180","testVoicemail180_190","testVoicemail190_200","testVoicemail200_210","testVoicemail210_220","testVoicemail220_230","testVoicemail230_240","testVoicemail240_245","testSharedCost0_10","testSharedCost10_20","testSharedCost20_30","testSharedCost30_40","testSharedCost40_50","testSharedCost50_60","testSharedCost60_70","testSharedCost70_80","testSharedCost80_90","testSharedCost90_100","testSharedCost100_110","testSharedCost110_120","testSharedCost120_130","testSharedCost130_140","testSharedCost140_150","testSharedCost150_160","testSharedCost160_170","testSharedCost170_180","testSharedCost180_190","testSharedCost190_200","testSharedCost200_210","testSharedCost210_220","testSharedCost220_230","testSharedCost230_240","testSharedCost240_245","testCanBeInternationallyDialled0_10","testCanBeInternationallyDialled10_20","testCanBeInternationallyDialled20_30","testCanBeInternationallyDialled30_40","testCanBeInternationallyDialled40_50","testCanBeInternationallyDialled50_60","testCanBeInternationallyDialled60_70","testCanBeInternationallyDialled70_80","testCanBeInternationallyDialled80_90","testCanBeInternationallyDialled90_100","testCanBeInternationallyDialled100_110","testCanBeInternationallyDialled110_120","testCanBeInternationallyDialled120_130","testCanBeInternationallyDialled130_140","testCanBeInternationallyDialled140_150","testCanBeInternationallyDialled150_160","testCanBeInternationallyDialled160_170","testCanBeInternationallyDialled170_180","testCanBeInternationallyDialled180_190","testCanBeInternationallyDialled190_200","testCanBeInternationallyDialled200_210","testCanBeInternationallyDialled210_220","testCanBeInternationallyDialled220_230","testCanBeInternationallyDialled230_240","testCanBeInternationallyDialled240_245","testGlobalNetworkNumbers","testEveryRegionHasAnExampleNumber0_10","testEveryRegionHasAnExampleNumber10_20","testEveryRegionHasAnExampleNumber20_30","testEveryRegionHasAnExampleNumber30_40","testEveryRegionHasAnExampleNumber40_50","testEveryRegionHasAnExampleNumber50_60","testEveryRegionHasAnExampleNumber60_70","testEveryRegionHasAnExampleNumber70_80","testEveryRegionHasAnExampleNumber80_90","testEveryRegionHasAnExampleNumber90_100","testEveryRegionHasAnExampleNumber100_110","testEveryRegionHasAnExampleNumber110_120","testEveryRegionHasAnExampleNumber120_130","testEveryRegionHasAnExampleNumber130_140","testEveryRegionHasAnExampleNumber140_150","testEveryRegionHasAnExampleNumber150_160","testEveryRegionHasAnExampleNumber160_170","testEveryRegionHasAnExampleNumber170_180","testEveryRegionHasAnExampleNumber180_190","testEveryRegionHasAnExampleNumber190_200","testEveryRegionHasAnExampleNumber200_210","testEveryRegionHasAnExampleNumber210_220","testEveryRegionHasAnExampleNumber220_230","testEveryRegionHasAnExampleNumber230_240","testEveryRegionHasAnExampleNumber240_245","testEveryRegionHasAnInvalidExampleNumber0_10","testEveryRegionHasAnInvalidExampleNumber10_20","testEveryRegionHasAnInvalidExampleNumber20_30","testEveryRegionHasAnInvalidExampleNumber30_40","testEveryRegionHasAnInvalidExampleNumber40_50","testEveryRegionHasAnInvalidExampleNumber50_60","testEveryRegionHasAnInvalidExampleNumber60_70","testEveryRegionHasAnInvalidExampleNumber70_80","testEveryRegionHasAnInvalidExampleNumber80_90","testEveryRegionHasAnInvalidExampleNumber90_100","testEveryRegionHasAnInvalidExampleNumber100_110","testEveryRegionHasAnInvalidExampleNumber110_120","testEveryRegionHasAnInvalidExampleNumber120_130","testEveryRegionHasAnInvalidExampleNumber130_140","testEveryRegionHasAnInvalidExampleNumber140_150","testEveryRegionHasAnInvalidExampleNumber150_160","testEveryRegionHasAnInvalidExampleNumber160_170","testEveryRegionHasAnInvalidExampleNumber170_180","testEveryRegionHasAnInvalidExampleNumber180_190","testEveryRegionHasAnInvalidExampleNumber190_200","testEveryRegionHasAnInvalidExampleNumber200_210","testEveryRegionHasAnInvalidExampleNumber210_220","testEveryRegionHasAnInvalidExampleNumber220_230","testEveryRegionHasAnInvalidExampleNumber230_240","testEveryRegionHasAnInvalidExampleNumber240_245","testEveryTypeHasAnExampleNumber","testShortNumbersValidAndCorrectCost0_10","testShortNumbersValidAndCorrectCost10_20","testShortNumbersValidAndCorrectCost20_30","testShortNumbersValidAndCorrectCost30_40","testShortNumbersValidAndCorrectCost40_50","testShortNumbersValidAndCorrectCost50_60","testShortNumbersValidAndCorrectCost60_70","testShortNumbersValidAndCorrectCost70_80","testShortNumbersValidAndCorrectCost80_90","testShortNumbersValidAndCorrectCost90_100","testShortNumbersValidAndCorrectCost100_105","testShortNumbersValidAndCorrectCost105_110","testShortNumbersValidAndCorrectCost110_120","testShortNumbersValidAndCorrectCost120_130","testShortNumbersValidAndCorrectCost130_140","testShortNumbersValidAndCorrectCost140_150","testShortNumbersValidAndCorrectCost150_160","testShortNumbersValidAndCorrectCost160_170","testShortNumbersValidAndCorrectCost170_180","testShortNumbersValidAndCorrectCost180_190","testShortNumbersValidAndCorrectCost190_200","testShortNumbersValidAndCorrectCost200_210","testShortNumbersValidAndCorrectCost210_220","testShortNumbersValidAndCorrectCost220_230","testShortNumbersValidAndCorrectCost230_240","testEmergency0_10","testEmergency10_20","testEmergency20_30","testEmergency30_40","testEmergency40_50","testEmergency50_60","testEmergency60_70","testEmergency70_80","testEmergency80_90","testEmergency90_100","testEmergency100_110","testEmergency110_120","testEmergency120_130","testEmergency130_140","testEmergency140_150","testEmergency150_160","testEmergency160_170","testEmergency170_180","testEmergency180_190","testEmergency190_200","testEmergency200_210","testEmergency210_220","testEmergency220_230","testEmergency230_240","testCarrierSpecificShortNumbers0_10","testCarrierSpecificShortNumbers10_20","testCarrierSpecificShortNumbers20_30","testCarrierSpecificShortNumbers30_40","testCarrierSpecificShortNumbers40_50","testCarrierSpecificShortNumbers50_60","testCarrierSpecificShortNumbers60_70","testCarrierSpecificShortNumbers70_80","testCarrierSpecificShortNumbers80_90","testCarrierSpecificShortNumbers90_100","testCarrierSpecificShortNumbers100_110","testCarrierSpecificShortNumbers110_120","testCarrierSpecificShortNumbers120_130","testCarrierSpecificShortNumbers130_140","testCarrierSpecificShortNumbers140_150","testCarrierSpecificShortNumbers150_160","testCarrierSpecificShortNumbers160_170","testCarrierSpecificShortNumbers170_180","testCarrierSpecificShortNumbers180_190","testCarrierSpecificShortNumbers190_200","testCarrierSpecificShortNumbers200_210","testCarrierSpecificShortNumbers210_220","testCarrierSpecificShortNumbers220_230","testCarrierSpecificShortNumbers230_240","testSmsServiceShortNumbers0_10","testSmsServiceShortNumbers10_20","testSmsServiceShortNumbers20_30","testSmsServiceShortNumbers30_40","testSmsServiceShortNumbers40_50","testSmsServiceShortNumbers50_60","testSmsServiceShortNumbers60_70","testSmsServiceShortNumbers70_80","testSmsServiceShortNumbers80_90","testSmsServiceShortNumbers90_100","testSmsServiceShortNumbers100_110","testSmsServiceShortNumbers110_120","testSmsServiceShortNumbers120_130","testSmsServiceShortNumbers130_140","testSmsServiceShortNumbers140_150","testSmsServiceShortNumbers150_160","testSmsServiceShortNumbers160_170","testSmsServiceShortNumbers170_180","testSmsServiceShortNumbers180_190","testSmsServiceShortNumbers190_200","testSmsServiceShortNumbers200_210","testSmsServiceShortNumbers210_220","testSmsServiceShortNumbers220_230","testSmsServiceShortNumbers230_240","testPhoneNumbersV8_12_31"]}],"skipCodeCoverage":true}
2022-04-19 21:00:51,240 [ 990656]   WARN - .IlluminatedCloudExceptionUtil - Read timed out
java.net.SocketTimeoutException: Read timed out
    at java.base/java.net.SocketInputStream.socketRead0(Native Method)
    at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
    ...
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(SourceFile:545)
    at com.illuminatedcloud.intellij.unittest.ApexUnitTestRunProcessHandler.runTestsSynchronous(SourceFile:795)
    at com.illuminatedcloud.intellij.unittest.ApexUnitTestRunProcessHandler.lambda$startProcessing$7(SourceFile:648)

but that actually using a different Salesforce REST API which allows synchronous test execution. The timeout for any single Salesforce API call is 5 minutes, and evidently that particular test class takes longer than 5 minutes to execute, hence the API time since it's executed synchronously. The workaround for that would be to force it to run asynchronously by checking this option:

It might be interesting to do that and try to run just that single test class again in isolation to see if that produces the 500. If it doesn't, my best advice at this point is going to be to try to isolate which test classes are leading that the 500. Create an Apex unit test run config and select the first half of your classes for execution. If that works, change it to the other half. If one or the other fails, continue this type of binary search to try to isolate the offending test class(es).

That's the issue with a 500 response...it shouldn't happen. Such a response indicates an unhandled exception in the Salesforce server, so at this point you're just trying to figure out what part of the input is producing that specific error, and sometimes that ends up being a bit of an exhaustive process.

[Vladyslav Petrovych]

I’ve managed to single it out to this Test Class with 401 unit tests.

When I run it synchronously, it fails with “Read Timeout” error.

When I run it asynchronously, it fails wit the “Unknown Exception”.

‌

How can I get the actual API call behind it, so that I could open the case with Salesforce?

[Scott Wells]

In both cases it's using the Tooling API's REST resources for test execution:

https://developer.salesforce.com/docs/atlas.en-us.api_tooling.meta/api_tooling/intro_rest_resources.htm

For synchronous it uses runTestsSynchronous and for asynchronous it uses runTestsAsynchronous.

The posted payload specifies both class and method names and not class IDs.

You may even try to reproduce this behavior in Developer Console as I know it uses the Tooling API as well, though I don't know if it specifies its POST payload in the exact same fashion.

Let me know if you need any other info.

[Scott Wells]

Resolving as this is now isolated to the Tooling API behavior when a specific (and large) test class is specified for either synchronous or asynchronous test execution.

[Scott Wells]

Oh, and to be clear, I'd focus on the async behavior as the timeout when invoked synchronously is likely because IC2 is using a 5 minute max timeout, but it's possible that call would complete if left to run for (quite a bit?) longer. However, the fact that queueing the class for async execution causes a 500 is likely the real problem.