Metadata Refresh Taking a Long Time

[Joshua Krinsky]

Metadata refreshes are taking a very long time – up to an hour or more, while metadata retrieval or comparison and merges seem to retrieve the metadata from the org much faster (< 5 minutes).

[Scott Wells]

Hi, Joshua. Can you please enable debug logging for metadata retrieval, reproduce the behavior, and provide the resulting idea.log either here or via email to [e-mail]? That will allow me to see where the time is being spent. It's possible that it's just the Salesforce API's retrieval queue, but given that you're not seeing the same thing with other actions that use the same API, it may be due to an issue that I've logged with JetBrains:

https://youtrack.jetbrains.com/issue/IDEA-250149

Let's see what the log shows.

[Joshua Krinsky]

Hi Scott,

Thanks for your reply. I am able to do metadata retrieves and server compares fairly quickly. However a metadata refresh does take about an hour and has generated tons of log content, which I have provided in my dropbox. I’m pretty certain it is not the Salesforce API's retrieval queue.

I did notice a couple of anomalies that could be related:

1. After concluding the refresh, I see the message “Refresh Complete: Refreshed 5500/5500 components from…” however there are certainly not 5500 components in my source tree currently. At one point in this project I did retrieve all of the objects and fields into the project and then removed them. At that time, I believe the component count was about 5500. However most of those resources have been removed from my local source tree and metadata subscription.
2. There is a second “Updating Indexes” task that appears in the Background Task queue when the metadata refresh is queued and remains “paused” for the duration of the refresh, but completes fairly quickly when the metadata refresh is completed.
3. There are different phases reported on in the background task view while the metadata refresh is in process. “Retrieving…” and “Preparing Retrieved files” do not take long, but then “Refreshing local metadata from retrieved metadata” seems to be the part that is taking a long time.

Thanks for looking into it, and please let me know if I can provide any additional info.

Cheers!

Josh

[Scott Wells]

Thanks for the log. Well, I am seeing that number of components, but it's because of how source format handles static resource bundles, and you have a few VERY large ones in the retrieval including what seems to be at least two full copies of SLDS, SurveyForce_SLDS and slds.

The retrieval itself takes ~50 seconds. Reconciling all of those extracted static resource bundle files is taking forever though. Obviously with two copies of SLDS, there are at least two files with any given filename, often more under other relative paths; IC2 has to determine the proper matches to local files based on contextual paths to avoid false correlations. There are likely a few optimizations I could do there, specifically removing files from future consideration once a match has already been found, but I don't think that's going to change the time here dramatically. At best it might cut it in half. Another optimization might be to recognize static resource bundles at the root level and, in a refresh situation, try to replace them en masse rather than based on each constituent file. That would likely yield much better results, but it's a non-trivial change relative to where things are now.

Let's do this...try to run the refresh metadata action on everything EXCEPT for static resources. I bet that happens relatively quickly. Then try it on all static resources except for the SLDS ones. Again, I imagine it'll be pretty quick, though I do see other potentially large static resource bundles listed including jquery. I don't really see that taking up much time in the log, though. Minimally this will allow us to confirm my suspicion, then we can see what to do about it.

[Scott Wells]

Issue tracker grooming. If this is still an issue, please feel free to reopen, ideally with a concrete reproduction scenario.