Support for SFDX projects that don't use Dev Hubs or Scratch Orgs

[Gustavo Tandeciarz]

I saw some references that spoke to supporting this functionality, but I did not see a specific ticket for tracking this feature.

If I create an SFDX project --manifest and auth into a Sandbox (i.e. no scratch or dev hub orgs configured in the project), the SFDX CLI works - I can deploy and retrieve metadata from my sandbox and it puts it into the SFDX folder structure as expected. If I try to open that project folder in IntelliJ, the project can't be configured without a dev hub org and scratch org config.

It would be sweet if this was supported and I could use IntelliJ + IC instead of VS Code since IC has much nicer support for SF dev...all while using the nice new folder/file structure from SFDX.

I referenced this article when I setup my SFDX project in VS Code: https://github.com/forcedotcom/salesforcedx-vscode/wiki/Develop-Against-Any-Org-in-Visual-Studio-Code#source-format

[Gustavo Tandeciarz]

...Tracking using my personal bitbucket account.

[Scott Wells]

Yep, this will be coming. There's another issue for this somewhere, but I couldn't find it at a glance. When I do I'll tie them together for easier tracking.

[George Doenlen]

Scott any update on this? I would really like to get back to using IC instead of vscode.

[Scott Wells]

George, no concrete timeframe other than to say that it's going to be part of the next set of work I do after I implement a few additional LWC-related things. In other words, hopefully quite soon as I know that people are really wanting it, but unlikely in the next week or two.

[George Doenlen]

Thanks! Glad to know its on the agenda at least!

[Scott Wells]

Both on the agenda and in the quite near-term alongside a whole host of enhancements, fixes, improvements, etc., in the way that metadata handling occurs. More details to come on that soon, actually...

[Scott Wells]

FYI, I have gotten started on this in earnest and am making very solid progress. I imagine that I'll have a new build out with full support next week (barring unforeseen obstacles).

[Oleg Malitsky]

Scott, It's nice to hear that you've done progress on this feature. I will be glad to test your new build with this feature if U'll have it before regular release.

[Scott Wells]

I'm happy to share an early build here once it's ready for consumption. It's always helpful to get additional burn-in and feedback from users before releasing widely. My guess is that I'll have something ready to share either tonight or tomorrow night. Once I do, I'll post a test build here with instructions on how to use it.

[Scott Wells]

FYI, this is pretty close to complete now. I have a few things I want to do before putting it in other folks' hands, but minimally I'll post a test build here no later than Monday to start getting feedback on it. I'll also plan to record a short video walking through everything I've done to support this scenario.

Oh, and the good news is that I was able to retain almost all of the standard IC deployment/retrieval features with this model including incremental deployment, collision detection, selective deployment/retrieval/delete, etc. Pretty much only Retrieve for Merge and Refresh had to be removed for these types of projects because they don't make sense when the CLI is already retrieving directly into the project source directories. And of course integrated OAuth support is SO nice!

Stay tuned...I'll be posting a build in the next couple of days...

[Scott Wells]

Okay, this is very close to being ready for beta-level access..."beta" being the key word as the changes to support this properly have been quite extensive.

Before I post a build, though, let me explain what will be there and what won't, and I have a few questions I'd like to ask about the projects that this will(/should) support. First, a few terms that I'll be using:

• OAuth connection - A connection to a non-scratch org established and managed using the SFDX CLI via web login flow, JWT flow, etc.
• SFDX manifest project - A project that was created using sfdx force:project:create -x and that is managed using sfdx force:source:deploy/retrieve, generally based on the contents of a package.xml manifest file. The project's contents are stored in SFDX's source form instead of the more traditional metadata form.

What's there/not there

• Support for creation and use of OAuth connections in IC. You create them using a new toolbar button in IC's connection manager. IC will prompt you for a connection alias and require you to login via your Web browser.
  • OAuth connections are available for use in the Anonymous Apex, SOQL Query, and Log Viewer tool windows, and if you're working in a module associated with an OAuth connection, you can choose any other OAuth connection for deployment, retrieval, and delete.
  • Currently only web login flow is directly integrated into the IDE, but all authorized SFDX orgs are available as IC connections.
  • Currently OAuth connections can only be assigned to and used by SFDX manifest projects.
    • NOTE: Very shortly I will allow you to use OAuth connections anywhere you can currently use native IC username/password connections (will resolve #159).
  • OAuth connections do not yet support extended connection configuration/options, in particular conflict detection. Again, to resolve #159 I will need to allow OAuth connections to support the same options as native IC connections.
  • OAuth connections are currently hard-configured not to auto-authorize custom fields for the authenticated user. Until it's possible to configure this explicitly, you may find that the offline symbol table is missing custom fields that are inaccessible to your user.
• First-class support for SFDX manifest projects (module is actually the more appropriate term, but I'll stick with project for now) in IC.
  • The Illuminated Cloud SFDX new project wizard can now create SFDX manifest projects by checking Generate Manifest in the first step.
  • The import project wizard automatically recognizes SFDX manifest projects and configures IC for them as appropriate (though I have a question below about how it's currently recognizing them to ensure it's sufficient).
  • The metadata subscription for an SFDX manifest project is automatically set to Package.xml and configured to use the package.xml file created during SFDX CLI project creation.
    • Subscription management is based mostly on manual package.xml authoring, but IC will prompt to add/remove package.xml entries as metadata files are added/removed in the IDE.
  • With very few exceptions, SFDX manifest projects work exactly like username/password projects.
    • Deploy modified/all metadata, deploy-on-save, force save, retrieve/refresh metadata, and delete metadata are all supported. You can choose exactly which subset of metadata to deploy/retrieve/delete through context menu-based selection and/or filtering in the deploy/retrieve/delete dialog.
    • Retrieve-for-merge is not supported because there's no opportunity to redirect retrieved metadata into a temporary location for comparison.
    • Compare with server does work, though for decomposed metadata types such as custom objects with fields, validation rules, etc., the comparison isn't correct because the server version isn't decomposed.
    • Unit testing, code coverage, offline debugger, etc., all work exactly as expected as well.
  • Tooling API-based deployment works for Apex classes, Visualforce pages, individual simple static resources (not bundles), and individual Aura bundle files.
    • This can often result in much faster deployment times for these metadata types than sending them through the CLI/Metadata API.
    • Static resource bundles in SFDX projects are managed by the SFDX CLI.
    • Deployment of multiple Aura bundle files/full Aura bundles and LWC files/bundles must be handled by the SFDX CLI.
• Previously an IC SFDX project hosted by IntelliJ IDEA could have only one module, and it had to be an SFDX module. Now IC's SFDX support is module-oriented instead of project-oriented. This means that you may now have an IntelliJ IDEA project with multiple SFDX scratch org modules, SFDX manifest modules, and/or traditional username/password modules.
  • The logic to discover IC project/modules is now much more flexible and can find one or more SFDX modules nested below the project root, potentially as peers to username/password modules, and set up the project accordingly.
  • WebStorm does not support multi-module projects, so unfortunately this change only applies when IC is hosted by IntelliJ IDEA Community Edition or Ultimate Edition.

Questions

For those of you working against SFDX manifest projects now, I have a few questions:

• Where is your project's package.xml file? The SFDX CLI's force:project:create -x command places it under <projectRoot>/manifest/package.xml, but technically you can provide a path to any package.xml file when deploying/retrieving source. Is anyone using a package.xml file that's not <projectRoot>/manifest/package.xml? If so, can you provide details?
• How is your project/repo structured? Is it a single SFDX project, or do you have multiple projects nested under a common root directory? If the latter, what is the relationship between SFDX manifest projects and orgs/connections? Do all projects converge on the same org, does each project deploy to its own org, or is it a mix?
• What types of orgs are you using with these projects? DE orgs? Sandboxes? Packaging orgs? Production orgs?

Okay, let's start with that. Based on the responses to this info and these questions, I may need to make a few tweaks before a build will be ready for you to use, or I may just put what I have now out there and use you all as a sounding board to prep these changes for wide release.

Thanks!

[George Doenlen]

Hi Scott,

To answer your questions:

1. package.xml is always under <projectRoot>/manifest/package.xml
2. For our orgs we structure the projects using the default package. force-app/main/default/{classes, triggers, etc}. However I was going to start experimenting with adding additional packages for dependencies. Something like force-app/main/lib/{dependencies} (if that's even possible).
3. All sandbox development.

[Scott Wells]

Okay, here's a build with beta-level access to this feature set. You can install it by downloading the attached archive (do not extract it) and, from IntelliJ IDEA/WebStorm, go into Preferences/Settings>Plugins and use Install plugin from disk (under the gear drop-down in recent base IDE builds).

You should then be able to open/import or create an SFDX manifest project and work with it from within IC as described in my previous comment on this issue. I'll also try to put together a short video showing this new functionality today/tonight to clarify the behavior.

I've already tried to put some serious test cycles on this, but I feel confident that there are still issues to be addressed. For those of you who want to help with additional testing/burn-in, please let me know of anything that doesn't work properly and/or as-expected and I'll take care of it. Oh, and you can add the following to Help>Debug Log Settings to get pretty extensive debug logging for everything that IC is doing here:

#com.illuminatedcloud.intellij.builder.ForceComSfdxMetadataDeployer
#com.illuminatedcloud.intellij.builder.ForceComSfdxMetadataRetriever
#com.illuminatedcloud.intellij.sfdx.SfdxUtil
#com.illuminatedcloud.util.CommandLineUtil

Note that on Windows, the maximum length of a command-line invocation is 8K characters, so when I detect that a selective deployment via the Salesforce CLI would exceed that, I degrade to a manifest-based deployment for the project. I already have a workaround in mind for that, but for the moment know that if you're on Windows and you attempt to deploy a large amount of metadata in a single operation, it might deploy your entire package.xml if the CLI invocation would otherwise fail.

I'll likely plan to post a few builds here with additional enhancements/fixes before I make this the official build. I'll post updates as they're ready.

Have at it!

[Scott Wells]

Here's a new beta/test build that fixes a large number of issues from the previous build and pretty much completes the general-purpose OAuth implementation. If you're trying out this functionality, please update to this build ASAP.

Conflict detection is not yet available for SFDX manifest projects, but I plan to incorporate that before general availability of this feature.

I've recorded short videos demonstrating OAuth and SFDX manifest projects.

As always, please let me know if you've installed this and, in particular, please let me know if you find any issues so that I can take care of them.

Thanks!

[Scott Wells]

Issue #1064 was marked as a duplicate of this issue.

[Scott Wells]

Just thought I'd see if anyone has installed this test build and is using it. If so, how are things working for you? The latest posted build has been working well for me, but I always like to hear from others who are test driving it.

I'd like to try to get this into the official build, so please let me know how it's going so I can address any issues you might have found. Thanks!

[Scott Wells]

New test build with minor fixes for IntelliJ IDEA/WebStorm 2019.1, better handling of long command-lines on Mac/Linux when invoking the Salesforce CLI for force:source:deploy/retrieve, and the ability to delete OAuth connections from within the IDE.

My goal is to add conflict detection and perhaps dynamic package.xml generation for larger deploys/retrieves in SFDX manifest projects, then release this in the official build early next week if possible. Feedback from those using it as a daily driver is greatly appreciated.

[Scott Wells]

New test build with the following major improvements since the last build:

• Support for integrated removal of OAuth connections.
• Selective deploy/retrieve in SFDX-deploy-against-any-org modules now creates a temporary package.xml file to specify just the selected contents. No more need for workarounds for maximum command-line length issues.
• Better understanding of the decomposed metadata structure when deploying, retrieving, etc. This is not full support for child metadata files yet, but it does address the main issues noticed when working with these types of projects.
• Support for conflict detection before deployment and OST regen for deployed SObjects after deployment.
• Added an SFDX config option to specify that SFDX push/pull should always be forced (really part of #852 but piggy-backing on this build).

I have noticed what seem to be some bugs/regressions in the most recent Salesforce CLI update. I'm speaking with Salesforce about them now. If any of these changes are in fact the way they should be and will be going forward, I'll address them in the next IC build. The most obvious one is around simple static resource file naming in SFDX/source form projects.

If you're using this, please let me know about any bugs/regressions you find so that I can address them before the GA release.

Here are the detailed change notes as I approach official release:

• Issue 159 - Support for creation and use of OAuth connections in Illuminated Cloud.
  • You may now create and manage SFDX OAuth connections using a new toolbar button in IC's connection manager. IC will prompt you for a connection name, org type (optional), and login URL (optional), then will require you to log in via your Web browser. The Salesforce CLI is used for all OAuth interaction and management of the access token.
  • OAuth connections can be used anywhere that configured IC connections can be used and can be configured similarly.
  • OAuth orgs can also be used for the SFDX-develop-against-any-org feature (see below).
  • NOTE: After updating to this version or higher, each project will be upgraded upon opening to include connection type information for each usage of a connection. This ensures that there is no ambiguity when an IC connection name and a Salesforce DX alias match. If you manage your IDE configuration files in version control, please check in these changes and have your entire team, if appropriate, align on a plugin build that includes this change.
• Issue 1184 - Support for SFDX-develop-against-any-org projects in Illuminated Cloud.
  • The Illuminated Cloud SFDX new project wizard can now create SFDX-develop-against-any-org projects by checking Generate Manifest in the first step.
  • The import project wizard automatically recognizes SFDX-develop-against-any-org projects and configures IC for them as appropriate.
  • The metadata subscription for an SFDX-develop-against-any-org project is automatically set to Package.xml and configured to use the package.xml file created during Salesforce CLI project creation. Subscription management is based mostly on manual package.xmlauthoring, but IC will prompt to add/remove package.xml entries as metadata files are added/removed in the IDE.
  • For the most part, SFDX-develop-against-any-org projects work exactly like IC projects using configured connections. Specifically the following features are not currently available in these types of projects:
    • Retrieve for merge is not supported because the CLI is responsible for retrieval of metadata directly into the project.
    • Namespace substitution and deployment/retrieval substitution rules are not supported for the same reason.
  • Tooling API-based deployment works for Apex classes, Visualforce pages, individual simple static resources (not bundles), and individual Aura bundle files in SFDX-develop-against-any-org projects. This can often result in much faster deployment times for these metadata types than sending them through the CLI/Metadata API.
  • Static resource bundles in these projects are managed by the Salesforce CLI.
• Issue 852 - Added an SFDX config option to specify that all push/pull operations against scratch orgs should use the -f flag to disable conflict detection and force overwrite the target copy. This option was added for some users who are seeing consistent false conflicts reported by push and pull operations and, as a result, are having to perform the operation twice each time. NOTE: If you are not seeing this type of issue, you should not enable this config option.
• Previously an IC SFDX project could have only one module, and it had to be an SFDX module. Now IC's SFDX support is module-oriented instead of project-oriented. This means that you may now have an IntelliJ IDEA project with multiple SFDX scratch org modules, SFDX-develop-against-any-org modules, and/or traditional username/password modules. The correct module-specific behavior should be applied at all times. NOTE: WebStorm does not support multi-module projects, so unfortunately this change only applies when IC is hosted by IntelliJ IDEA Community Edition or Ultimate Edition.
• The logic to auto-discover IC project/modules is now much more flexible and can find one or more SFDX modules nested below the project root, potentially as peers to username/password modules, and set up the project accordingly. NOTE: WebStorm does not support multi-module projects, so unfortunately this change only applies when IC is hosted by IntelliJ IDEA Community Edition or Ultimate Edition.

[Scott Wells]

Another day, another test build. This one addresses issues with static resources in SFDX projects as well as a few other bugs. I'm going to call this one a release candidate build because I think I've addressed all the known features and bugs required for a first full release of this functionality. Please update if you're testing this and let me know if you see any issues.

[Scott Wells]

Another test build--call this one RC2--that addresses some other problems with static resources, primarily around creation and removal, and accommodates for some event lifecycle timing changes that seem to have been introduced with 2019.1.

As before, please update to this build if you're helping with the burn-in and let me know of any issues you might see.

[Scott Wells]

New build. Nothing functionally new in this one, but I did revert a small change to how IC determines when it's time to refresh its list of SFDX orgs based on issues observed by a few users. Hopefully this build will resolve that problem.

[Doug Ayers]

Hi Scott,

This all looks very promising!

One of my older projects uses the traditional metadata format (/src directory) that is connected to a DE org, but I've recently added a sfdx-project.json file so that I could use the occasional CLI command. However, now that the sfdx-project.json file exists, IC2 expects my project's connection to be a scratch org and I'm unable to deploy metadata because IC2 is defaulting to force:source:push, which is invalid option for my DE org.

Are the new features you're creating here going to make it where the developer (aka, me) can choose how I want the project to deploy/retrieve rather than IC2 inferring based on my project layout? Meaning, even if I connect my project to a scratch org or a DE org, or I include or don't a sfdx-project.json, or my metadata is in a folder named /src or /force-app/main/default, I may want IC2 to use force:source:deploy/retrieve instead of force:mdapi:deploy/retrieve or force:source:push/pull commands.

In short, I think the problem I've run into on IC2 v2.0.5.6 is that the existence of a sfdx-project.json makes IC2 treat my project as if I'm developing against a scratch org despite the connection assigned to my module is not a scratch org.

Thanks!

[Scott Wells]

Doug, right now the type of module is inferred by IC so that it knows how to interpret the contents of the source folders--metadata form vs. source form. It infers as follows:

• If there's an sfdx-project.json file and a manifest/package.xml file, it's an SFDX-dev-against-any-org module.
• If there's an sfdx-project.json file and no manifest/package.xml file, it's an SFDX scratch org module.
• If neither of those and there are Salesforce source files, it's an MDAPI module.

I could certainly make this more explicit, though the capacity for misconfiguration concerns me.

Is the source for the project you're using here in metadata form or source form? If the former, what is the sfdx-project.json intended to do? If the latter, any reason not to restructure it to be consistent with the result of running sfdx force:project:create -x? I'm not at all challenging what you're doing...I'm trying to understand if there's a gap in the current assumptions I've made in IC around these three models.

[Scott Wells]

Delivered in 2.0.5.7. Consult the release notes for details on what is included in this release.

[Doug Ayers]

Thanks, Scott!

To answer your questions:

Is the source for the project you're using here in metadata form or source form?

Metadata form

If the former, what is the sfdx-project.json intended to do?

Some CLI commands require that it exists, and so I added it to the project. I don't often run CLI commands while working on this project, but occasionally I do and hit that dependency.

I could look at converting the project structure to source form, I just haven't yet. Perhaps a tech debt item I can work on for my project.

Thanks

[Scott Wells]

Thanks, Doug. That's interesting that there are CLI commands that require sfdx-project-json but that don't care about the project structure otherwise. Do you have any examples? I'd like to investigate and maybe talk to the CLI team. That seems like an impedance mismatch...either a CLI command should be 100% independent of a project--force:org:list, force:apex:execute, etc.--or it should be tied to the format of the project, e.g., force:source:push/deploy need source format and sfdx-project.json whereas force:mdapi:deploy needs metadata format and no sfdx-project.json. I'll be curious to see what I'm missing...