INVALID_CROSS_REFERENCE_KEY error when deploying to Spring '21 pre-release sandbox

[Mariyan Yankov Topalov]

This appears when trying to deploy my code to the org. As result of this error, the code is sometimes deployed, sometimes not.

[Mariyan Yankov Topalov]

This actually happens for 2.1.5.5 version.

[Scott Wells]

That's an error from the Salesforce deployment system itself, most often when you try to deploy a payload that's relationally incomplete. What exactly are you trying to deploy to the server when this happens? If you'd like, you can enable debug logging for metadata deployment, reproduce the behavior, and send over the resulting idea.log (Help>Show Log in Explorer/Finder/Files) and I can review it.

It also sounds like this is happening exclusively in the context of Spring '21 sandboxes? Is that correct?

[Scott Wells]

Issue #1802 was marked as a duplicate of this issue.

[Mariyan Yankov Topalov]

I’m simply trying to deploy any class, which I edited. Even if the class has 1 new blank line, the error appears.

And yes - that in the context of Spring '21 sandboxes as I reproduces the error in dev sandbox and partial copy sandbox - both have the same result.

[Scott Wells]

For those of you experiencing this issue, as a potential workaround can you try disabling deployment via the Tooling API since this seems to be occurring through that deployment path? You can disable it in Illuminated Cloud>Configure Application>Validation and Deployment under Deploy Tooling API for. Please disable Apex and try to deploy these classes again. If that fails, I'd like to see new debug logs. I'm also going to try to set up a Spring '21 pre-release org so I can reproduce this myself.

UPDATE: This appears to be a regression in the Tooling API in Spring '21. Disable deployment via the Tooling API as described above if you're encountering this. I will be consulting with Salesforce regarding this regression and will provide further status via this issue.

[Mariyan Yankov Topalov]

that's the latest debug log

[Scott Wells]

Just to clarify, did you try disabling the tooling API and deploying Apex classes with the metadata API? That latest log still shows deployment via the tooling API.

[Michael Richmond]

I’m also having this issue this morning after the Sandboxes that I’m working with were upgraded to Spring '21 over the weekend. Disabling the Tooling API for Apex as a workaround is working for me.

[Mariyan Yankov Topalov]

This log is through using Metadata API

[Mariyan Yankov Topalov]

Disabling Tooling API works! It’s a little bit slower, but get’s the job done.

[Alan Birchenough]

It worked after I switched to the metadata API.

[Scott Wells]

Okay, so it also deploys successfully via the Metadata API for Mariyan:

2021-01-11 18:11:55,798 [ 428111]   INFO - er.ForceComMetadataApiDeployer - Deployment status = SUCCEEDED 
2021-01-11 18:11:55,798 [ 428111]  DEBUG - er.ForceComMetadataApiDeployer - Processing 2 component successes. 
2021-01-11 18:11:55,799 [ 428112]  DEBUG - er.ForceComMetadataApiDeployer -   Successfully deployed classes/OpportunityLineItemTriggerHandler.cls 
2021-01-11 18:11:55,799 [ 428112]  DEBUG - er.ForceComMetadataApiDeployer -   Successfully deployed package.xml 

I've pinned that to the top of this issue as the known workaround, and I've emailed the appropriate folks at Salesforce regarding the regression. I'll provide updates here as I receive them, but unfortunately until I hear back from them--and likely until they provide a fix--the workaround is going to be the best/only option.

[Mariyan Yankov Topalov]

It’s better than nothing. Thank you very much for the fast reaction Scott.

[Scott Wells]

Oh yeah, you bet. Lovely way to start a Monday morning...for all of us! Hopefully Salesforce will be able to provide a quick resolution to the Tooling API regression.

[Scott Wells]

Lowering from blocker status since there is a workaround.

[Scott Wells]

Salesforce has asked that affected users file a support case. Can some/all of you please do that and provide the support case numbers here (or via email to [e-mail] if you'd prefer) so that I can convey them to the appropriate folks at Salesforce to help raise visibility?

[Michael Richmond]

Case Number: 40335220

[Scott Wells]

Thanks, Michael. I've passed the case number on to Salesforce.

[Scott Wells]

I'm continuing to work with Salesforce on this and they haven't yet been able to reproduce the issue. What I think would be the most useful from a diagnostic standpoint would be the raw SOAP and REST API interactions between IC2 and the server when this issue occurs. It's possible to get those by setting certain debug log levels, but that of course does mean that EVERY interaction with the server is logged verbatim. The resulting logs should NEVER be posted or shared without review and likely redaction of sensitive information such as credentials and intellectual property.

I'd like to request that at least one or two of you experiencing this issue help to create these types of logs. I can assist with enabling debug logging as needed. I would request that you carefully scrub the logs of sensitive information before providing them to me, and then finally I would extract the relevant subsets of the scrubbed logs for review by Salesforce to ensure a proper signal-to-noise ratio. Ultimately they will just need to see the API requests/responses that lead up to this issue.

In a perfect world we'd even get the same client/server dialog against an pre-Spring '21 org where it works and then against a Spring '21 org where it fails so we can look for any differences, likely in the responses. The requests shouldn't be changing since IC2 has not yet been updated for Spring '21 at all.

If you're willing to assist with this, please email me directly at [e-mail] and I'll walk you through what needs to happen. Thanks much!

[Jason Clark]

Just want to mention (in case it helps) that this is intermittent for me; if I simply try a second time after a failure it usually works. Observed on at least two separated Sp'21 sandboxes. Now that I’ve found this thread, I’ll just disable tooling API as per the workaround.

[Scott Wells]

I hadn't heard about the inconsistent behavior until now. That's actually surprising. The good news is that with Michael Richmond's help, I've provided the team at Salesforce VERY detailed logs against both Winter '21 and Spring '21 orgs showing the exact same operation working (former) and failing (latter). Hopefully with those logs they'll be able to figure out what happened and issue a patch quickly.

[Raghav Sharma]

I agree with Jason Clark. After Tuesday, I am also seeing the issue intermittently and also it works fine second time in case that happens. I would say after Tuesday, it’s not that frequent as it was on Monday and Tuesday when it was happening on every save operation.

[Scott Wells]

Very interesting. I wonder if Salesforce is already trying out a prospective fix. I'll pass this on to them as additional information. Thanks!

[Naveen Sakhamuri]

Same as Jason and Raghav, The issue is appearing for me since a few days and is occurring Intermittently. When I try again, the save is happening without any issue.

[Scott Wells]

Salesforce has provided some additional diagnostic information based on their investigation into this issue. It seems that it occurs when a client of the tooling API makes one request to start a tooling API deployment and then (almost) immediately makes another request to poll the status of that deployment. If that second request is slightly delayed, there's no issue. This explains the inconsistent behavior observed by some users.

This has never been an issue prior to Spring '21, so hopefully Salesforce will identify and fix the root cause, but in the meantime I can make IC2's tooling API deployer more resilient in this specific situation. I'll build in a small allowed/expected number of retries when this specific type of error is reported when initially polling the tooling API deployment request. If it keeps failing after that small number of attempts, the entire operation will fail, but if it switches over to a successful polling state, it will continue polling. My guess is that that will resolve/workaround the issue.

I'll try to implement that tomorrow and post a test build here with the change. At that point I'd appreciate it if as many folks who have been experiencing this issue as possible could install it, re-enable deployment via the tooling API, and provide feedback on whether it does indeed resolve the issue.

More details to come when the test build is available. As always, thanks for the patience and valuable feedback!

[Michael Richmond]

Thanks for the update Scott.

I can also provide some additional information after a call with Salesforce Support today. I believe they are chasing the possibility that the size of the Apex class being deployed is contributing to the inconsistent behaviour. I confirmed that I was unable to reproduce this issue on smaller Apex classes and it seemed to become a more consistent issue as sizes increased.

[Scott Wells]

I'm still trying to reproduce this myself so that when I instrument a workaround, I can have (more) confidence that it's implemented correctly. I've created a Spring '21 pre-release scratch org and am deploying a rather huge Apex class (MetadataService = 834 kb) to it via the Tooling API. I'm not getting any errors. Does anyone have any thoughts on how to raise the likelihood of this reproducing aside from deploying a large class?

Worst case scenario I'll just use the provided logs to simulate the error response, but hopefully needless to say, I'd prefer to be reproducing this locally as I work on the fix/workaround.

[Scott Wells]

Okay, this is a test build that's the same as the current official build (2.1.5.6) but with an attempt at a workaround for the INVALID_CROSS_REFERENCE_KEY issue during deployment via the Tooling API. Note that I have still not been able to reproduce the issue, so I'll need others who do experience the issue to help verify the change.

To install it download the attached archive (do not extract it) and use Settings/Preferences>Plugins>Install plugin from disk (under the gear drop-down menu). Allow the IDE to restart. Re-enable deployment via the Tooling API in Illuminated Cloud>Configure Application>Validation and Deployment by making sure that Prefer Tooling API for>Apex is checked. Finally, try to deploy Apex classes that were failing previously. Hopefully that will now work. Either way I'd like to see your idea.log from the deployment. If it fails, I'd like to see the error that caused the failure so I can adjust for it accordingly. If it succeeds, I'd like to see how this workaround was applied as it will log each initial INVALID_CROSS_REFERENCE_KEY error (up to five retries allowed before total failure) so I can get a better idea of the nature of the underlying issue.

Thanks in advance to all who continue to help characterize this issue! Hopefully this gets us a significant step closer to being able to use the Tooling API again for deployment in Spring '21.

[Scott Wells]

Attaching latest test build.

[Scott Wells]

New test build that can handle a nested exception properly. Please install this and try it. I'd like to see the logs either way, and debug logging can be completely disabled as the relevant info is logged at the warning level.

[Scott Wells]

I've received feedback from one user that this does in fact resolve (or rather work around) the issue with deployment via the Tooling API against Spring '21 orgs. I'd love more data points, though. After installing the attached build, please add the following to Help>Diagnostic Tools>Debug Log Settings exactly including the leading #s:

#com.illuminatedcloud.intellij.builder.IlluminatedCloudSaveAllAction
#com.illuminatedcloud.intellij.builder.ForceComBuilder
#com.illuminatedcloud.intellij.builder.ForceComBuilderUtil
#com.illuminatedcloud.intellij.builder.ForceComMetadataApiDeployer
#com.illuminatedcloud.intellij.builder.ForceComSfdxMetadataDeployer
#com.illuminatedcloud.intellij.builder.ForceComSfdxDeployer
#com.illuminatedcloud.intellij.builder.ForceComSfdxMetadataUtil
#com.illuminatedcloud.intellij.builder.ForceComToolingApiDeployer
#com.illuminatedcloud.intellij.builder.ForceComBuildFailureAnnotator
#com.illuminatedcloud.util.VariableLengthPollingInterval

then make sure that deployment via the Tooling API is enabled for Apex, deploy some Apex classes that were failing in this manner previously, and whether things work or not, please provide the resulting idea.log files using Help>Show Log in Explorer/Finder/Files. If things work, I'll be interested to see the entries like:

2021-01-26 18:04:15,060 [  53823]   WARN - der.ForceComToolingApiDeployer - Received error 'INVALID_CROSS_REFERENCE_KEY: invalid cross reference id' with invalidCrossReferenceKeyCount = 0. 
2021-01-26 18:04:15,060 [  53823]   WARN - der.ForceComToolingApiDeployer -   Retrying the status poll. 

to see how many times per-request that error occurred before a valid status was returned. Of course, if things fail, I'd like to see the logs for that as well just to see what happened given this increased tolerance for the error.

As always, thanks so much! If things continue to look good I'll just include this in the next official build, currently targeted for Thursday of this week.

[Scott Wells]

For those of your following progress here, I have a few updates. As mentioned previously, it appears that the attached build does in fact seem to work around the issue (good news). However, the queries that fail initially due to this regression seem to take quite a bit of time to return (upwards of 10 seconds each) leading to longer deployment times via the Tooling API when this problem occurs (bad news). While Salesforce support has been unable to reproduce the issue themselves (bad news), Salesforce engineering has requested that the case be moved to them for further investigation (good news).

So....I'm still hoping that there's a full resolution from Salesforce in advance of the official Spring '21 release that's coming shortly. If not, I may issue a build of Illuminated Cloud with this workaround but that also disables deployment via the Tooling API as the default behavior until the issue is properly resolved. Otherwise I fear that the deployment experience--at least for Apex--will be poor due to the expense of these error retries.

I'll keep you all posted as the case moves to Salesforce engineering and I hear more from them. Fingers crossed for a full and proper resolution.

[Scott Wells]

FYI, I was finally able to reproduce this myself. I had to load up a Spring '21 org with a significant number of Apex classes and deploy them together via the Tooling API. That resulted in the following behavior:

2021-01-27 11:06:52,049 [4324104]  DEBUG - der.ForceComToolingApiDeployer - Creating a container async request for metadata container 1dc1g000000Nkp5AAC with checkOnly = false. 
2021-01-27 11:06:52,222 [4324277]  DEBUG - der.ForceComToolingApiDeployer - Polling the status of the ContainerAsyncRequest. 
2021-01-27 11:07:02,017 [4334072]   WARN - der.ForceComToolingApiDeployer - Received error 'INVALID_CROSS_REFERENCE_KEY: invalid cross reference id' with invalidCrossReferenceKeyCount = 0. 
2021-01-27 11:07:02,017 [4334072]   WARN - der.ForceComToolingApiDeployer -   Retrying the status poll. 
2021-01-27 11:07:02,017 [4334072]  DEBUG - .VariableLengthPollingInterval - ForceComToolingApiDeployer.runContainerAsyncRequest: Using polling interval 1000 ms for polling iteration 1. 
2021-01-27 11:07:03,029 [4335084]  DEBUG - der.ForceComToolingApiDeployer - Polling the status of the ContainerAsyncRequest. 
2021-01-27 11:07:13,050 [4345105]   WARN - der.ForceComToolingApiDeployer - Received error 'INVALID_CROSS_REFERENCE_KEY: invalid cross reference id' with invalidCrossReferenceKeyCount = 1. 
2021-01-27 11:07:13,050 [4345105]   WARN - der.ForceComToolingApiDeployer -   Retrying the status poll. 
2021-01-27 11:07:13,050 [4345105]  DEBUG - .VariableLengthPollingInterval - ForceComToolingApiDeployer.runContainerAsyncRequest: Using polling interval 1000 ms for polling iteration 2. 
2021-01-27 11:07:14,055 [4346110]  DEBUG - der.ForceComToolingApiDeployer - Polling the status of the ContainerAsyncRequest. 
2021-01-27 11:07:20,860 [4352915]  DEBUG - der.ForceComToolingApiDeployer -   ContainerAsyncRequest status = QUEUED. 
2021-01-27 11:07:20,861 [4352916]  DEBUG - .VariableLengthPollingInterval - ForceComToolingApiDeployer.runContainerAsyncRequest: Using polling interval 1000 ms for polling iteration 3. 
2021-01-27 11:07:21,866 [4353921]  DEBUG - der.ForceComToolingApiDeployer - Polling the status of the ContainerAsyncRequest. 
2021-01-27 11:07:22,046 [4354101]  DEBUG - der.ForceComToolingApiDeployer -   ContainerAsyncRequest status = COMPLETED. 
2021-01-27 11:07:22,046 [4354101]  DEBUG - der.ForceComToolingApiDeployer -   ContainerAsyncRequest is complete. 
2021-01-27 11:07:22,046 [4354101]  DEBUG - der.ForceComToolingApiDeployer - Compiled all metadata container members in 29997 ms. 
2021-01-27 11:07:22,046 [4354101]  DEBUG - der.ForceComToolingApiDeployer - ContainerAsyncRequest state = COMPLETED 

Exactly as seen in the other logs, there's a pretty reliable 10 second delay on the requests that fail with INVALID_CROSS_REFERENCE_KEY. So the good news is that I'm able to reproduce this, and I'll offer to provide access to this org to Salesforce engineering/support if they want since I have a very specific environment in which it occurs. Let's see where this takes us...

[Scott Wells]

The workaround has been delivered in 2.1.5.7. I'm going to resolve this as the real fix must be delivered by Salesforce, and once it has been, the workaround will just go unused.

It's safe to re-enable Tooling API development with this build, but note that you may want to leave it disabled because, when this issue occurs, the initial failing queries can take ~10 seconds, thereby lengthening Tooling API-based deployment times.

Thanks so much to everyone to helped to characterize and debug this. Fingers crossed for a quick full fix from Salesforce.

[Scott Wells]

Quick update...Salesforce engineering is still actively trying to reproduce and resolve this issue. Given the larger-scale rollout of Spring '21 this weekend, I've improved the workaround behavior in 2.1.5.8 as follows:

Issue 1801 (redux) - Improved end user feedback when the workaround for the INVALID_CROSS_REFERENCE_KEY Tooling API deployment error is applied:

• If the error was encountered but the Tooling API deployer is able to complete the request, the status is changed to warning (if not already warning or error) and a message is added to the deployment details in the Messages tool window that the API error was encountered and may have lengthened the deployment time. The user is advised that Tooling API-based deployment of Apex and Visualforce should perhaps be disabled until Salesforce issues a fix.
• If the error was encountered and the allowed retry count (10) was exceeded, the user is notified about the nature of the deployment failure and is prompted to disable Tooling API-based deployment of Apex and Visualforce until Salesforce issues a fix.

That way users will know when and how this issue is affecting them and will have a better cue for when it might be appropriate to (temporarily) disable Tooling API-based deployment until a true fix is available.