Is it possible to speed up metadata retrieval & deployment in large orgs?

[June Bischoff]

You did so great with speeding up the OST generation, I'll throw another one at you. It was taking us around 30 seconds to deploy a single class in our other orgs, and with the installation of Giant Managed Package, we're running around 2-1/2 minutes to deploy a single class to Salesforce. We've been bugging SF about trying to speed up our org, but no go. Giant Managed Package seems to be the root of the issue.

Retrieval is also slow, but since we do that somewhat less often than deployment it's mildly less problematic.

Any possibility of seeing speed increases in this department for large orgs?

[Scott Wells]

I'll give a "maybe/maybe not" answer. On the maybe side, it's definitely something I plan to investigate, though not until I knock out some other bugs and functional gaps that I've promised to other users. I'd also like to see that operation run faster. On the maybe not side, whereas OST generation is compromised of a series of API calls, many of which are against the same API for but subsets of data and therefore lend themselves to parallelization, metadata retrieval is a single operation in the Salesforce API, and the time is ultimately spent waiting for the retrieval operation to complete. I'm not sure if breaking a large request up into several parallel smaller requests would even work because of the way that Salesforce's async job scheduler works. It might just serialize those and return the batch results one at a time.

So there may be opportunities for some gains here, but my guess is that it's not going to yield the same types of major reductions that we saw with OST generation.

[June Bischoff]

Decreasing the priority of the issue, then.

[June Bischoff]

If it were possible to have retrieval and deployment run as a background process, that would be an improvement that would help mitigate the pain of the long retrieval times.

We also have Salesforce looking into our slow deployments in this complex environment, and while they're still investigating, they said: "We just checked and found that , you already have the Run test as False, however the deployment still taking part of analyzing the test run status (to check if test run is specified or not)."

I'm not sure if that information is something that would help you with speeding up the process or not.

(Still a minor issue compared to the other ones in the queue.)

[Scott Wells]

June, throw in a vote for issue #186 which calls for backgrounding of long-running operations such as deploy and retrieve.

As for the info you got from Salesforce, let me look at what I'm asking when the deployment completes and see if there's something I can do to make it more lean. I don't think I'm checking for test run status, but perhaps one of the checks I am doing is heavier than I'm thinking.

Oh, and #155 is next in my queue since I released the foundational deployment/retrieval enhancements this morning.

[June Bischoff]

That's great news re: #155!

Additional detail for the possibility of speeding up deployment: Most of the time my IC Settings are set to Compile with the Tooling API on Save. Deploying by right-clicking to the IC Menu and Deploying Modified Metadata takes about 2-1/2 minutes in our complex org. Changing my IC settings to Deploy with the Tooling API on Save speeds things up so that deployment takes about a minute. IC Settings of Deploy with the Metadata API are significantly slower again.

Which API is used by default when manually Deploying Modified Metadata if I'm only set to Compile on Save? Can I change that default API? I was looking at #240, which seems mildly related. Did I simply miss a setting? (I would prefer not to automatically deploy on Save when IntelliJ automatically saves every few seconds.)

[Scott Wells]

June, let me look at this a bit over the next few days and I'll see what opportunities exist for speeding this up for you and others. On a semi-related note (because it's what keeps me from looking at it sooner), I'm just about finished with #155 and expect that I'll release it tomorrow morning for you guys.

[Scott Wells]

June, I had to look closely in order to answer your question about which API is used when you have things configured the way you do and you ask for a bulk or force deployment. In the case of your configuration to use the Tooling API to compile (vs. deploy) on save, it'll use the Tooling API when deploying just Apex source (classes and/or triggers). If you're deploying Apex and Visualforce source and both of those are set to use the Tooling API, it'll also use the Tooling API. If either of those is set to use the Metadata API or if a deployment includes non-Apex/VF metadata, it'll degrade to the Metadata API which can handle all metadata types.

What I also noticed while in there is that if you set these to do nothing on save, it won't actually deploy. That's definitely a bug that I thought I'd fixed. I'm going to include a tactical fix for that now to have it use the Metadata API due to the issues described in issue #240 with deployment against the Tooling API. As a more long-term solution, I think I'm going to restructure that configuration screen a bit to simplify it. Some of those options can be collapsed into simpler ways to convey the same behavior. Anyhoo, based on what you described, you should be using the Tooling API as long as you're only deploying Apex. I can give you some info about how to enable debug logs if you want to confirm that and even get some timings for key deployment steps.

[Ruwantha Lankathialaka]

I started to use the plugin I have to spent some considerable amount of time compared to save. Whereas eclipse force.com (36V) plugin take lesser time. Is IlluminatedCloud use tooling API ? Can I change it to tooling API if it is not?

[Scott Wells]

Ruwantha, it should be using the Tooling API by default. You can configure that under Settings>Illuminated Cloud>Validation & Deployment. You can add diagnostic logging around deployment by adding the following to Help>Debug Log Settings:

#com.illuminatedcloud.intellij.builder.ForceComBuilder
#com.illuminatedcloud.intellij.builder.ForceComMetadataApiDeployer
#com.illuminatedcloud.intellij.builder.ForceComToolingApiDeployer

Perform a save and then send me the relevant portion of the log using Help>Show Log in Explorer/Finder. I'll take a close look at where the time is being spent and provide suggestions on how to improve things, or if it's an issue in IC, take a look at improving them internally.

[Tony White]

I work in a shared environment and need to pull down the entire SF environment locally on a daily basis (sometimes multiple times a day). Currently it takes about 15-25mins to pull down ~3227 items....

14/10/16 12:03 Retrieval Complete: Retrieved 3218 components from Insurance Dev in 16m 6s 282ms with status SUCCEEDED.

17/10/16 08:45 Retrieval Complete: Retrieved 931 components from Insurance Dev in 2m 12s 815ms with status SUCCEEDED.

09:10 Retrieval Complete: Retrieved 3227 components from Insurance Dev in 22m 10s 604ms with status SUCCEEDED.

Tooling API is preferred for all API settings, and I have just added the debug settings mentioned above to try and capture what it going on... (so next retrieval I can provide those details). I was using Mavensmate, and was able to retrieve the metadata much more quickly with that :(

IntelliJ IDEA 2016.2.4 Build #IC-162.2032.8, built on September 9, 2016 JRE: 1.8.0_112-release-b343 x86_64 JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o

Illuminated Cloud ver: 1.7.4.7

OS: Mac Os - Sierra (10.12) Ram: 16GB Processor: 2.6 GHz Intel Core i7

[Scott Wells]

Hi, Tony. For the most part metadata retrieval is an asynchronous job where the client (whether Illuminated Cloud, the Force.com Migration Tool, or whatever) is just polling for status after submitting the job. Having said that, let's go ahead and gather details logs of one of these retrievals to see where the time is truly being spent. It sounds like you've already added classes for logging, but just to be sure since these operations are so long-running, make sure you have the following in Help>Debug Log Settings:

#com.illuminatedcloud.intellij.builder.RetrieveAction
#com.illuminatedcloud.intellij.builder.ForceComBuilder
#com.illuminatedcloud.intellij.builder.ForceComMetadataRetriever

After one of these retrieves, please send me your idea.log (potentially all of the idea.log.* files) found in Help>Show Log in Explorer/Finder and I'll see where the time is being spent.

I do know that there have been issues for the past month or so with some async processes in some pods. Before I attribute this behavior to those issues, though, let me digest the log.

[Tony White]

Hi Scott

I have done a refresh this morning (twice), took about 19mins

Is there some way I can provide the log without making it public?

Cheers Tony

[Scott Wells]

Absolutely, Tony. Feel free to email them to [e-mail] and I'll take a look at them. Hopefully I'll be able to see what's happening pretty quickly.

[Scott Wells]

Just a quick follow-up here for those who are following the conversation. Based on Tony's logs, it looks like IC is configured to download all metadata in the org including all system metadata. The overwhelming bulk of the time in retrieval is spent polling Salesforce for the completed async retrieval result. I've recommended a more explicit metadata selection via either Package.xml or Selected metadata subscriptions.

[Tony White]

I have a suspicion that part of the problem is that it is pulling down old versions of process flows it does not need... I pulled down the UAT environment in just under 10mins So that is an issue of devs not cleaning up after themselves. We have found it adds significant delays if you have old versions of flows etc in the ant deployment...

[Scott Wells]

Not sure what else I can do on this one. If there are specific use cases that are still running slower than you'd expect...and in particular because of IC and not because of slowness in the APIs themselves...please let me know and I'll look into it.