Unable to pull down large project: LIMIT_EXCEEDED

[Steve MunLeeuw]

I am working on a large project, when I attempt to pull down all available metadata types I run into the following exception:

LIMIT_EXCEEDED: Too many files in retrieve call, limit is: 10000

I am following the Atlassian cookbook for continuous integration with Salesforce. One of the requirements that I have is that admin users should be able to make configuration modifications through the Salesforce Admin UI and then sync those changes down to a developer machine so that the xml file differences can be inspected and committed to source control.

[Scott Wells]

Thanks for the report, Steve. Right now it attempts to pull metadata from the org according to the subscription using a single retrieve request. If I can discern that the retrieve request would result in >10K files, I could batch it, but if you're using a package.xml with wildcards, that would be a bit more tricky. Still possible, but I'd have to call APIs to enumerate all of the org's metadata before deciding how to retrieve, and that would be quite expensive. Anyway, I'm pretty much thinking aloud here.

For the moment you could work around the limitation by performing multiple retrieves on subsets of the metadata. If you're not sure how to do that, please let me know and I'll describe it for you.

[Steve MunLeeuw]

I appreciate your prompt feedback. I believe I am going the multiple retrieves route. Rather than use the All option I select certain metadata, sync it, then select additional metadata types. There are some dependencies between the selections so it's been slow going. I have not been able to retrieve the full project using this technique yet. I am new to Illuminated Cloud, I'm a big fan of the IntelliJ IDEA and hope that this works out. MavensMate has similar issues.

[Scott Wells]

Okay. Let me know how it goes and feel free to provide any feedback on how it might be simplified/streamlined based on your experience.

[Steve MunLeeuw]

I went a different direction. I used the Force.com Migration Tool with an Ant script. I first use the sf:listMetadata to pull the types on the instance into a text file. I then use ant-contrib to read each line and call sf:bulkRetrieve. I checked out the Welkin Institute IDE as well, that only offers a subset of metadata.

[Scott Wells]

Duplicate of #107.

[Peter Lin]

Hi Scott, I also got the same error as below.

I am wondering whether it requires only a few hours to add a new feature similar to the following screenshot.

This way, when I selected 3 applications and 22 classes, IC would show the number of total selected metadata as 25. This might help to keep the total number of selected metadata files within 10,000 limit.

This is only my guessing. Do you think it will work like what I described as above?

Thanks a lot!

[Scott Wells]

Peter, the issue with the (naive approach to) the proposal is that child metadata types are also counted, e.g., custom fields and validation rules in .object files, distinct custom labels in custom label files, etc. You can't just sum up the top-level metadata, but you'd need to perform a deep dive on child metadata types to get the correct/accurate count. I'll need to keep thinking about whether there's a good/elegant way to handle this. I may also check with Salesforce to see if there is any intention to raise/remove this limit anytime since they do seem to be addressing some of the more arbitrary limits these days.

[Peter Lin]

Thank you so much, Scott, for your detailed explanation!