Deployment taking much longer in 2.2.9.2

[Scott Wells]

This was reported by a user here:

https://groups.google.com/a/illuminatedcloud.com/d/msgid/qanda/7e8f5aa5-2f11-479b-bf0f-d56971245bban%40illuminatedcloud.com?utm_medium=email&utm_source=footer

It appears to be because IC2 is enumerating all files unnecessarily for a (seemingly) larger metadata format project.

[Scott Wells]

This build includes a prospective fix for the performance regression. Please download (but do not extract) the attached archive, then install it using Settings/Preferences | Plugins | Install plugin from disk (under the gear drop-down menu). Allow the IDE to restart and then see if the issue is still present. If so, please attach new debug logs and thread dumps for review.

[Didier Beaufays]

I confirm installing this build fixes the issue for me. Thanks

[Scott Wells]

Okay, great! I will include that in the next build which will be this week. It might be tomorrow if others are running into this same issue, or it might be Thursday which is the typical day used for the weekly release. Feel free to update to 2.2.9.3 when prompted to do so.

[Maciej Mizera]

Hi Scott, this build resolves my freezing issue. Thank you! That’s why I love this product.

[Scott Wells]

Thanks for the additional confirmation. I’m working on (at least) one other change I’d like to get into this week’s build alongside the fixes in the attached build. Right now I’m targetting Thursday for a release that includes them. Users who have installed the attached build will still be prompted to update and should be able to do so safely.

[Scott Wells]

Delivered in 2.2.9.3.

[Scott Wells]

It appears that this wasn't fixed entirely. I'm attaching a build that changes the part that seems to be taking a long time in large projects -- effectively derivation of metadata types and names for files in a Local Files project -- to be multi-threaded instead of purely serial in nature.

Please download (but don't extract) the attached build and install it using Settings | Plugins | Install plugin from disk (under the gear drop-down menu). Allow the IDE to restart.

Please make sure that the following is in Help | Diagnostic Tools | Debug Log Settings:

#com.illuminatedcloud.intellij.builder.ForceComBuilder

Open Illuminated Cloud | Configure Module and click the Refresh toolbar button to force IC2 to re-enumerate the local and org metadata. That should happen on multiple threads, and there will be an entry in the log like:

Derivation of metadata types and names for XXX files took YYY time.

Click the Refresh button a few times if you'd like for multiple data points.

Now in that same dialog, click the edit button beside the module's assigned connection and double-click the module's row in the resulting dialog. That should open the connection properties. Uncheck Use parallel processing and click OK until you're back in the subscription editor. Again, click the Refresh button at least once but ideally a few times again. These are going to be purely serialized and will take longer. What I'd like to know is just how much longer they take and therefore how much of an improvement parallelization is providing.

With that done, please send me the resulting idea.log file (potentially multiple if the logs rolled during this testing) for review, or if you'd like, you can just extract the specific log entries identified above and provide those.

Oh, and you'll want to re-enable Use paralllel processing for the connection as well. That yields huge benefits all over the place.

[Didier Beaufays]

Hi , I completed this test and here are my logs, in my case I don’t see specific improvements.

2023-12-20 12:39:26,214 [  79178]   FINE - #com.illuminatedcloud.intellij.builder.ForceComBuilder - Derivation of metadata types and names for 5789 took 1 s 58 ms.
2023-12-20 12:39:27,137 [  80101]   FINE - #com.illuminatedcloud.intellij.builder.ForceComBuilder - Derivation of metadata types and names for 5789 took 107 ms.
2023-12-20 12:39:33,729 [  86693]   FINE - #com.illuminatedcloud.intellij.builder.ForceComBuilder - Derivation of metadata types and names for 5789 took 79 ms.
2023-12-20 12:39:46,300 [  99264]   FINE - #com.illuminatedcloud.intellij.builder.ForceComBuilder - Derivation of metadata types and names for 5789 took 113 ms.    
  --disable parallel processing
2023-12-20 12:43:12,462 [ 305426]   FINE - #com.illuminatedcloud.intellij.builder.ForceComBuilder - Derivation of metadata types and names for 5789 took 14 ms.
2023-12-20 12:43:31,739 [ 324703]   FINE - #com.illuminatedcloud.intellij.builder.ForceComBuilder - Derivation of metadata types and names for 5789 took 11 ms.

---

[Scott Wells]

Thank you for the feedback. I'm attaching another build that should resolve the issue when deploying via the Metadata API -- the prior fix was specifically for Tooling API-based deployment -- but won't address the issue categorizing local metadata files in the subscription editor.

Can you please install this build and confirm wherher it resolves any remaining slow deployments? Again, this build won't address refreshing the metadata list in the subscription editor, but you should no longer see slow deployments.

Also, for those of you seeing this issue with Local Files subscriptions, can you please characterize those projects in terms of the number of local metadata files? The largest test projects I have locally have ~10K local metadata files and I'm not seeing such an issue, so I'm wondering how large these projects might be. It could well be that projects over a certain size need to switch the use a Package.xml-based subscription to scale properly.

[Scott Wells]

Additional fix for deployment via the Metadata API delivered in 2.2.9.5. As stated in the linked release notes, this will probably be the last of the performance optimizations for local file metadata type discovery in extremely large projects using Local Files metadata subscriptions. Such projects should instead use Package.xml metadata subscriptions to avoid the need to enumerate and categorize very large numbers of local files.