Pseudo-wildcards that expand only for a filtered subset of metadata, e.g., excluding entries from installed packages

[Jason Clark]

We have an unlocked package (currently with NO namespace, though that may change) which we install into all consulting clients' orgs, which contains a set of common utilities. I work almost exclusively in sandboxes, and create SFDX projects for each code project before connecting IC2 (so I have a force-app/ dir and an sfdx-project.json, though I don’t usually create a manifest or project.xml).

I would like to set my module subscription to * for the standard dev types - ApexClass, ApexTrigger, LightningComponentBundle, etc. But when I do that, and try to Retrieve the module, the contents of my unlocked package appear in the list of files to be fetched, and will be fetched if I continue with the Retrieve operation. In the Retrieve dialog below, ApexUtil and CodeControl are both members of the unlocked package.

‌

This doesn’t seem to match the behavior reported in https://bitbucket.org/RoseSilverSoftware/illuminatedcloud/issues/2215/ignore-components-in-metadata-operations, but that’s what I’ve always observed. I would prefer to never fetch any package code, whether unlocked or managed, namespaced or not. I understand some folks might want to fetch it for debugging (I’ve skimmed https://bitbucket.org/RoseSilverSoftware/illuminatedcloud/issues/1165/unlocked-packages-no-namespace-metadata), but I’d prefer to keep it separate. Is this possible without managing individual ApexClass entries in the subscription?

[Scott Wells]

This one’s a bit tricky because Salesforce has a very specific notion of what a wildcard means for a metadata type in a package.xml file, and it doesn’t filter out entries of that type from installed packages with accessible components. It sounds like what you’re wanting--if I’m reading the request correctly--is to be able to configure whether your metadata subscription should include or, specifically, exclude components from installed packages, and still respect the intention behind the wildcard entries in your metadata subscription given that configuration setting, correct? IC2 would then need to expand those wildcards on bulk metadata operations based on the known ManageableState value for each entry so that INSTALLED, INSTALLED_EDITABLE, BETA, and UNMANAGED are excluded. One potential issue with that is that IC2 caches the org’s metadata list pretty aggressively to avoid having to query it repeatedly, but in this case it seems like it should always use the authoritative list--at least for these “filtered wildcard” metadata types--to ensure that it accurately behaves like a real wildcard. That could slow down these bulk metadata operations in densely-populated organizations. The alternative would be to use the cache, and the end user would need to ensure the cache is up-to-date or risk missing entries added to the org since the last time the cache was populated.

So those are my initial thoughts. Let me stop there to ensure we’re on the same page. Thoughts?

[Jason Clark]

@Scott Wells Sorry, this fell off my radar in the end-of-year holiday scheduling fun.

If I’m following you, the issue here is that when the package.xml subscription includes a * wildcard, that currently gets passed to the server for expansion, but to filter out packaged code, IC2 would need to do its own expansion, filter the metadata by ManageableState, and pass the expanded list(s) of metadata components to the server in the bulk metadata fetch request. Which is possible, but it raises the issue of stale cached metadata info possibly leading to missed metadata if the cache isn’t refreshed first.

I would be okay with that tradeoff; my normal workflow is something like:

1. Create project, using established SFDX connection
2. Fetch all code components with Tools > Illuminated Cloud > Retrieve… (or this dialog pops automatically during project creation?)
3. Edit in IC2 and push changes as needed.
4. Commit changes to git; pull changes from other team devs from git.
5. Repeat steps 2-3 as needed
6. If customer has internal developers working in the same sandbox who need to touch the same code we are working on, or if the sandbox is refreshed, or if we have to change sandboxes- commit all current work to git, then pull all code components again with Tools > Illuminated Cloud > Retrieve…, and finally review all changes reported by git to ensure no recent dev work has been lost. Finish with a “pulled changes from sandbox” commit and/or pushing work in the repo back to the sandbox.

So the only time I’m pulling metadata based on wildcards is when I’m intentionally pulling everything in the Retrieve dialog, which shows the timestamp of the metadata, so I always hit refresh if it’s more than a day old, which is most times since I don’t need to do it often.

Thanks,
JC