Support for Granular Retrieval and Deploy of Components in an SFDX project

[Rohan Gore]

Current State: sfdx force:source:push and pull does not support specific metadata to be pushed and pulled and so does the IC's push and pull buttons in an SFDX project. Awesome State: Two more buttons, Deploy and Retrieve in SFDX project that on clicking will present the developers with the same screens to choose the metadata they want to deploy or retrieve respectively to/from the connected scratch org that they get in non SFDX project in IC. Also SFDX being supporting multiple folder project structure, after choosing the metadata, one more screen should be presented to choose/enter in which folder the selected components need to be retrieved. While on the other hand, the deployment selection screen for metadata should pick up all the components from the folders listed in sfdx-project.json.

PS: Please Scott, this is something many of the devs adopting to SFDX desperately need, at least 15-20 in my organization itself. sfdx cli clearly lacks this feature but I believe, with IC you already have the code and platform ready to deliver something so astonishing pretty quickly. I really want to avoid falling back to the route of developing sfdx cli plugins to accomplish this as I am sure, no cli plugin will be able to match the awesome state I was mentioning above that is possible with the IC's GUI. I can understand there are sfdx force:source:deploy and retrieve commands that currently do not support scratch orgs. I am behind Salesforce SFDX product team and they asked me to provide use cases etc. I gave them many and after they never came back, seems like never took me seriously ;) well they must be having their priorities. That said you would also be having a lot of priorities but if this happens in IC, it would be a game changer in SFDX adoption and true source control based development and yes even if deploy and retrieve commands starts supporting scratch orgs, they can not match a sleek UI which is possible with IC in IntelliJ and the piece of mind doing everything from the same tool in the same style. :)

Thank you!

[Scott Wells]

Unfortunately this is very challenging to provide because of the inherent manner in which SFDX push/pull works. It uses underlying source sync features where it maintains its own notion of local vs. server/org state. This is partially stored in the local filesystem and partially in the scratch org. For IC to provide more fine-grained deployment/retrieval capabilities against scratch orgs would either require it to use the Tooling/Metadata APIs directly, and this would bypass SFDX's state tracking. As a result, any subsequent CLI-based push/pull would be incorrect at best. Basically you would have two systems competing for the same information with no real ability to coordinate them.

I've discussed this at length with the SFDX team at Salesforce. I've heard repeatedly from IC users that one of the biggest obstacles with moving to SFDX is losing the level of control and visibility that IC provides in terms of bidirectional metadata management. There has been some interest on their side in enhancing the CLI to allow more fine-grained push/pull capabilities, but I don't know if/when that will ever happen.

Without explicit support from Salesforce, the best suggestion I can make is to provision an additional user in your scratch org and configure a connection for that user in IC, then treat the org as a standard org instead of an SFDX org. Unfortunately right now you would also need to convert your project metadata to MDAPI format and remove/hide the sfdx-project.json file so that IC doesn't treat it as an SFDX project. Obviously far from ideal... There are perhaps things I could do to automate this, but before investing heavily in it--because it is really a workaround--I'd prefer to chat with the SFDX team and see what might be coming from them to improve this situation. Let me do that and I'll keep you posted here, at least as much as I'm able based on the confidentiality of what I hear from them.

[Rohan Gore]

Hi Scott,

I definitely understand the challenges because of the source tracking managed by SFDX. My proposal here is to certainly bypass that and use the Metadata/Tooling API the way IC does in a non sfdx project. The difference and probably the extra work for you here is to pull in the more granular xmls like fields, listviews etc. unlike the traditional huge object file and retrieve them in the right folders. A word of caution here for devs which you can clearly mention in the specs: "With this you will get the benefits of granular and more control to retrieve/deploy to your scratch orgs and the benefits of source driven development and the logical structuring of your source. But with great power comes great responsibility, SFDX push-pull might not work and will break or behave strangely. You would be responsible for source tracking." Frankly speaking, being accustomed to source driven development with traditional Salesforce way, I do not mind losing the so-called useless benefits of half-baked source tracking and would love to just leverage the scratch orgs, multi-folder support and the CI/CD tools leveraging the CLI and sfdx source format to deploy to sandboxes or other scratch orgs and creation of unlocked packages. I know numerous situations in which pull does not retrieve the changed components and the devs end up wasting time there in some nasty work arounds. I am really opinionated here based on my couple of months experience with SFDX. I hate Salesforce for marketing such half-baked things so well!!

Thank you!

[Scott Wells]

This was delivered quite a bit back.