New SFDX push/pull commands have changed in a breaking manner (issues after updating CLI)

[Christopher Matos]

Push Metadata to Scratch Org errors aren't displayed correctly in IntelliJ when using newest version of Salesforce CLI

Steps to reproduce:

1. Upgrade Salesforce CLI to most recent GA version (as of writing this is sfdx-cli/7.140.0),
2. Create an apex class with incorrect syntax (for example public class NewApexClass{{{{})
3. Push Metadata to scratch org

Expected Result:

• See an error in Illuminated Cloud tab with details about the error (incorrect syntax)

Actual Result:

• An error in the event log “Error running the Salesforce CLI: 1 -” with no further details

‌

This may be related to the changes that were made to force:source:push by Salesforce in the most recent release.

‌

Product: Intellij Ultimate Build #IU-213.6777.52

[Christopher Matos]

in addition it may be a nice feature to be able to configure whether IC uses beta, GA, or legacy commands via the CLI

[Scott Wells]

Hi. Thanks for filing. I'll take a look early next week, but my guess is that you're correct that the promotion of those beta commands to standard caused the issue.

[Scott Wells]

CLI ISSUE FILED: After a bit more investigation, these new CLI commands don't seem ready for prime-time just yet. I just filed a CLI bug with an example, but not only has the JSON interface changed completely, it no longer includes critical information for programmatic integrations, e.g., error messages and line/column information. I've also reached out to some folks on the team directly about this issue.

---

The CLI output of the force:source:push command after a CLI upgrade shows:

ERROR running force:source:push:  This project uses the old version of source tracking files.
Try this:
Use the old version of the command, 'force:source:legacy:push' with your existing tracking files.
Clear the old version of the tracking files by running 'sfdx force:source:legacy:tracking:clear'

I'm honestly a bit surprised that Salesforce doesn't automatically do this for you on first run of the new command as this is almost certainly going to break CI/CD systems and such as well as being a nuisance for interactive tooling integrations.

I'll reach out to them and see why they didn't just so I understand that decision, but I'll also have IC2 detect it and prompt the user to do it for them then retry the push.

UPDATE: Reviewing the CLI release notes, I'm guessing that it works this way because there's no in-place upgrade of old source tracking files to new source tracking files, and Salesforce didn't want to reset the source tracking state automatically, e.g., for source-tracked sandboxes. As a result, users seeing this issue with existing scratch orgs and source-tracked sandboxes will need to choose to reset the org's source tracking state or use the legacy commands until such a time as they're ready to do such a reset. That seems to be the rationale, at least.

I'll definitely do something in the coming week's IC2 build to make this more seamless for users encountering this issue. I may even release a build very early in the week with some tactical changes as I imagine this is going to impact quite a few folks. Perhaps the best short-term solution while some users are still on older CLI builds and some are on the latest is to detect this specific type of failure and use that to determine that the legacy commands should be used for now. The JSON output of the two sets of commands has changed such that a single implementation won't work (easily) against both sets of commands.

More to come here in the next day or so...

[Scott Wells]

Okay, in order to stay ahead of this a bit, I've done a few things:

1. As stated previously, I've filed a bug in the Salesforce CLI public issue tracker for all identified issues with the new commands.
2. I've also reached out directly to a few folks on the Salesforce CLI team to ensure that these issues and their impact, not just on IC2 but on any external tooling that relies on a consistent JSON interface.
3. I've released a new IC2 update, 2.2.1.1, that checks the CLI version on the first push or pull from a project and uses the respective legacy command if a CLI version of 7.140.0 or higher is found.

There are a few caveats on the new IC2 update:

• Because IC2 uses the legacy push/pull commands, if the CLI commands are also used in an IC2 project, the legacy commands must be used there as well, i.e., force:source:legacy:push and force:source:legacy:pull respectively. Otherwise you will receive errors about incompatible source tracking metadata.
• If you have projects which already use the new source tracking metadata and want to use them in IC2 for push/pull, you must first reset the source tracking metadata as follows:
  1. Run sfdx force:source:tracking:clear.
  2. Push your metadata either from IC2 saying you want to overwrite conflicts or from the CLI using sfdx force:source:legacy:push -f.
  3. I've noticed some situations where the source tracking metadata still isn't quite right and further pushes will show conflicts. If that happens, you'll need to pull metadata once via either IC2 or the CLI using sfdx force:source:legacy:pull. Note that this will overwrite local changes, though, so make sure you make a copy of your local project metadata before the pull (or have a version control baseline), then reconcile the pulled metadata with the original metadata to make sure that it's what you want, and finally push one more time. You shouldn't see any further false conflicts. Far from ideal, but that's the best way I've found to get a project in this state back into a good working state.

Once the issues with the new commands are resolved--or at least enough are resolved for a minimally-viable integration--I will update IC2 accordingly with support for the new commands.

Even with the new IC2 version that includes a workaround, I'm not going to resolve this issue until I know that things look okay for users working against source-tracked orgs. Please provide any feedback you have based on working with this new version here.

UPDATE 2022-03-07: I've spent much of the morning speaking with folks on the Salesforce CLI team about this release. I think they're going to push out a new build that rolls back the force:source:push and force:source:pull commands to their previous state, and I've also requested that they make the force:source:legacy:push and force:source:legacy:pull commands be aliases for those so that folks who have already started using them (such as IC2 against 7.140.0+) can do so without further disruption. I've also offered to work with them to ensure proper backward-compatibility for the new commands as they ready them again for promotion to GA status, and I've suggested that they also make the update less disruptive with regard to source-tracking metadata, ideally including an automatic and seamless migration to the new format instead of telling the user to clear the source-tracking status or move to the legacy commands. I'll post another update here when I have something new to share.

[Scott Wells]

Salesforce has released 7.141.1 which rolls back the promotion of the beta commands to GA while retaining the legacy commands as aliases to the previous commands to provide compatibility for those who have already started using those commands:

https://github.com/forcedotcom/cli/tree/main/releasenotes/sfdx

Specifically:

---

CHANGE: Due to unexpected issues, we reverted the GA of the force:source:beta:* commands that we announced on March 3, 2022. As a result, for example:

• force:source:status now uses the old pre-Beta functionality.
• force:source:legacy:status also uses the old functionality.
• We readded the force:source:beta:status command, which uses the new Beta functionality described in this release note.

The same change applies to all of these commands:

• force:source:push
• force:source:pull
• force:source:status
• force:source:tracking:clear
• force:source:tracking:reset

We apologize for the inconvenience and confusion. (GitHub issue #1431)

---

I will be removing the workaround to use the legacy commands in the next IC2 build, so please make sure that you're on 7.141.1 or higher once you update IC2 again.

I will also be working directly with the CLI team to test and verify beta commands used by IC2 in advance of their promotion to GA to help ensure that they provide a seamless upgrade experience once promoted.

[Scott Wells]

Version 2.2.1.2 removes the workaround to use legacy commands. Please make sure that your Salesforce CLI is at least 7.141.1 when updating to this version of IC2 for proper behavior.