Retrieval needs to support batching of metadata when more than the governor-limited amount exists/is selected

[Petr Švestka]

Hi,

1. Noticed that reports are not retrieved unless you specifically tick all the relevant items in the "selected" approach.
2. In my org, there's lots of reports, so if I tick all reports manually I'll get an error that only 1,000 (or 10,000?) items are allowed.

So is there something you can do about the limit in 2? And subsequently retrieve all reports automatically in 1.?

Thanks

[Scott Wells]

Petr, have you managed to create a package.xml that can retrieve the subset that you care about? I ask because I'm just creating the equivalent of a package.xml manifest. Or is the problem that you don't really want a subset and really want/need it all? If the latter, you're correct that I'll need to do something to partition the manifest so that no single request is for more than the limit.

[Petr Švestka]

Hi Scott,

the idea was to retrieve all reports so that I could search items in the folder sharing settings (e.g., which folders are shared with a certain group).

Now, I need to select a few reports, retrieve, select some more, retrieve, etc.

[Scott Wells]

Gotcha. Makes sense. If you don't mind, I'm going to change the issue summary a bit. Please let me know if my changes don't reflect your intentions.

[Tyler Mowbrey]

I'm also experiencing this issue. For most of my development orgs this is a non-issue as I stay away from the options with a large amount of metadata records (emails, reports, etc). Where I find this an issue is when I want to pull everything from my production org to push to src prior to large project deployments.

Any thoughts on what release this might be looked at? I'm working around the issue by do a custom retrieve multiple times with different folders so it's not a huge issue on my end, just a time saver.

Thanks! Tyler

[Scott Wells]

Tyler, I haven't looked into this because of the small number of users who have bumped into it and the fact that there's a workaround, albeit manual. Right now this is prioritized after the big upcoming items like finishing up Lightning, static resource bundle enhancements, etc.

[Tyler Mowbrey]

Thanks Scott. I figured as much :)

[Scott Wells]

Issue #459 was marked as a duplicate of this issue.

[Nic Edwards]

Hey Scott,

Hope you get some time to look into this beast - as it makes it difficult to backup or search an org with confidence.

Is this a limitation with IC, Idea, or SF?

Cheers

Nic

[Scott Wells]

Nic, the issue here is with the Salesforce APIs. They impose a maximum 10K object restriction on a retrieval. I spoke with them about this a bit last week at TrailheaDX. It's possible that using the REST API instead of the SOAP API might help, but it wasn't clear what the change might be. I've also looked at ways what Illuminated Cloud might break up a retrieval request into batches of <10K objects each, but because of the ways in which metadata objects interrelate, it gets messy. Right now the best way to use IC's retrieval mechanism to back up an org with >10K objects is to pull down metadata in known chunks of no more than that size. Hopefully moving to the REST API will help. I'll post here when I know one way or the other.

[Nic Edwards]

Thanks for the swift response Scott. That makes sense. Hopefully SF presents some work around that will alleviate the issue before you end up switching APIs, as that would be a pretty major job.

[Scott Wells]

Issue tracker grooming. This has been such an infrequent topic of discussion in the past nine years that it's unlikely to be integrated directly into IC2.