Unlocked Packages - No Namespace - Metadata not pulled into project

[Eric Alexander]

We have begun using Unlocked Packages in our production org. These packages have no namespace.

We attempted to troubleshoot an issue in production and needed to use IC's offline debugger.

When I went to pull the metadata down from production there was no ability to pull the classes from the unlocked package into IC.

[Scott Wells]

Eric, do you mind grabbing the metadata for a class from the installed unlocked package using Workbench (or another tool of your choice that can do the same thing), e.g.:

In particular I'm interested in seeing the value for ManageableState as that's what IC uses to determine what metadata to include/exclude during metadata retrieval.

[Eric Alexander]

@scotwells

Looks like the same as a managed package. One difference is the value for namespacePrefix is null / not present

[Scott Wells]

Okay, thanks. I've been scouring the metadata available through various APIs to try to figure out how to distinguish contents of installed managed packages from contents of installed unlocked packages. I agree that for unlocked packages without a namespace, looking for a manageable state of installed and no namespace prefix may be the best option...though that won't work for unlocked packages with namespaces...

The Tooling API does have some metadata about second-generation packages that includes the type of a package, but it doesn't appear that you can query it at this point. I'll check with Salesforce and see if they have more details on the best way to get this information from the org.

I'll keep you posted...

[Örn Ingvar Ásbjörnsson]

Has there been any update on this? I recently started using Unlocked Packages and after developing one in a DX project I installed it and wanted to see why the tests were failing in the installed org but couldn't pull the required classes to do so. All the classes are without namespace.

[Scott Wells]

Hi, Örn. Unfortunately there's no resolution to this yet. I have an open question to the folks at Salesforce as to how this particular state can be uniquely identified for tooling purposes. I'll plan to follow up with them because once I have that information (or can find a way to do it reliably myself if they don't have one), it will be a simple change to make to the way that IC filters metadata for inclusion in a metadata subscription and/or retrieval. I'll post any updates here as soon as I have them.

[Scott Wells]

Issue #1225 was marked as a duplicate of this issue.

[Scott Wells]

Okay, I do have an update. This has been discussed on the following group:

https://success.salesforce.com/_ui/core/chatter/groups/GroupProfilePage?g=0F93A000000Lg5U&s1oid=00D300000000iTz&s1nid=0DB30000000072L&emkind=chatterCommentNotification&s1uid=0053000000BNMQN&emtm=1549081542551&fromEmail=1&s1ext=0

Based on that discussion the Salesforce product manager for packaging, Dileep Burki, is now aware of this and has logged an issue internally to address it. I've also asked if there's any workaround until a proper fix is available assuming that won't happen for some time (Summer '19 at the earliest?). If there is a good workaround, I'll implement it. Otherwise I'll likely have to implement something hacky like "sampling" the bodies of packaged Apex classes to see whether they're obfuscated or not to know whether I should allow them to be retrieved.

I'll keep you all posted on progress here.

[Oleg Malitsky]

Hi Scott, do You found any 'hacky' workarounds for this issue? Looks like the only option for now is to use VsCode in parallel with IlluminantedCloud to deal with Unlocked packages.

[Scott Wells]

Oleg, I don't at present, but I do have one in mind. I was thinking about adding a config option to allow the user to include packaged metadata for retrieval, then let the user sort through it. I could even go a bit further and try to filter out things that are definitely from a managed package, e.g., any class or trigger with an obfuscated body. This option would default to disabled to keep folks from getting themselves into trouble unknowingly. Would that be helpful? If so, I could probably post a build here with that change to see how well it works, then once it's verified as useful, get it into the official build.

[Oleg Malitsky]

Scott, I believe it will help a lot. It sounds like a nice workaround untill we will have some updates from the Salesforce team.

[Scott Wells]

Okay. Let me see what I can do in short-cycle. I'll post here when something is available to try out.

[Scott Wells]

Okay, I had a bit of an epiphany...though I'll need your help to prove that out. It occurred to me that what I'm likely looking for here is metadata with a manageable state of installed and no namespace. I've adjusted IC's filters to allow selection of metadata meeting that description and have attached a build here. You can download it and install it using Settings/Preferences>Plugins>Install Plugin from Disk (which is now found under the gear drop-down). Once installed, see if the unlocked package metadata isn't available for retrieval and management.

Note that I have not tested this yet...It won't break anything, but it also may not be exactly what we're wanting (though I have high hopes!). Tomorrow I'll try to get an unlocked package together that I can use for more testing, but I wanted to give you all a chance to try it out as well in the interim. Please let me know how it goes for you.

[Sergey Ryabchikov]

Hi Scott! The updated plugin works, at least for me. It also resolve the issue with suggestions for objects and methods in a managed package. Thanks.

[Scott Wells]

Thanks for confirming, Sergey! Let's see if we can get some additional data points, but assuming it continues to pan out nicely, I'll just include it in the next build (which I may release Friday of this week).

[Eric Alexander]

Just tested and it works well for MDAPI orgs. For SFDX orgs the references are there but the bodies are missing which I guess is expected?

Can we get the alert for SFDX CLI update removed or an option to not show it? I am not able to update the CLI using SFDX update and have not been for over a year so this is going to get a bit intrusive popping up every time an SFDX CLI is executed: https://salesforce.stackexchange.com/questions/222566/sfdx-6-21-0-9db990b3c9-and-current-update-error-sfdx-no-longer-works

[Scott Wells]

Eric, the current change is to allow retrieval of metadata from unlocked packages, and I can't control how SFDX pull works in terms of what it will retrieve. The interfaces for those classes should still end up in the OST, though. Are they not?

And yes, you can disable the update notifications in the latest build already. Go into Illuminated Cloud>Configure Application>Salesforce DX and uncheck Show messages from the Salesforce DX error stream. That info will get logged but not displayed prominently in the IDE.

[Eric Alexander]

Scott, there reference resolved and you can navigate to the method declaration. Just the body is missing. Sounds like it is working as you expected.

Thanks for the option to turn it off...I wish they would fix it though :)

[Scott Wells]

Eric, it's working as I expected based on the current state of this change. Right now I've just added the ability to retrieve metadata from installed unlocked packages (without namespaces) into an MDAPI project. However, I can understand based on the original motivation for this issue why you'd want to have the full bodies of these classes in the OST as well...so that you can step into their implementations in the offline debugger to see what's going on. I don't think that would be a difficult change to make either now that I know how to identify this type of metadata. I'll see what I can do in the next few days.

As for the CLI warnings, they basically have fixed it...as much as they're going to. Previously when having the CLI output JSON for integration, success responses went to stdout and failure responses went to stderr. The issue with that is that not only did stderr get an error JSON document, but it would also get these warnings which resulted in malformed JSON. Now the CLI sends all structured output to stdout--success or failure--and stderr is reserved for free-form text messages such as upgrade notifications and internal errors. It's up to the calling application as to how to use that info. In IC I've made it show that info as a notification by default, but as you now know, that can be disabled so that it's logged but never presented. This is pretty consistent with how other CLIs that facilitate integration work, e.g., the AWS CLI, PMD, etc. I don't expect this behavior to change any further w.r.t. when/how these types of messages are issued.

[Oleg Malitsky]

Scott, thanks for resolving the issue that fast. Looks like the problem is solved, I'am able to see and edit unlocked package's classes and other stuff now. It's just awesome :)

[Scott Wells]

Glad to hear! Go ahead and stay on this build until I say otherwise here. I'll include at least this portion of the fix in the next released build--either tomorrow or Friday--and I'll also try to address Eric's request to have these classes rendered into the OST with their bodies in situations where they can't be retrieved as part of the project metadata.

[Örn Ingvar Ásbjörnsson]

Finally got a chance to test this out. Works as expected.

Thanks for the quick response Scott! :)

[Scott Wells]

Another verification. That's great! I'll get a small build out tomorrow with this so that you can all go back to the official build.

Eric, I looked at what will be required to convey the exact original source for classes from installed unlocked packages into the OST and it's non-trivial. It's very possible, don't get me wrong, but I don't want to rush the change. I'll get the first fix out tomorrow, and then I'll try to get the more complete fix out for you shortly. Just wanted to let you know that one doesn't preclude the other.

[Scott Wells]

Okay, the first part of this has been delivered with 2.0.5.6. Feel free to move back to the official build now. Eric, as mentioned previously, I'll try to get the OST aspect of this addressed shortly.

[Marko Čirič]

Hi Scott,

I have a namespaced unlocked managed package and it seems IC2 does not yet support retrieval of components from such packages? By my inspection its now possible to determine an editable package by the manageableState installedEditable.

‌

[Scott Wells]

Hi, Marko. While it's technically allowed, there are issues with that specific setup, e.g., the following from the IC2 code that handles this:

// TODO: I think that technically anything that's INSTALLED_EDITABLE is...well...editable,
//  but that can create serious issues with, e.g., Apex source files because they come back
//  with namespaces in their filenames that must be interpreted properly. For now disallow
//  that and we'll see if anyone specifically asks for it.

I'll consider this as "anyone specifically asks for it" and revisit the topic, but it won't likely be for a few weeks. I'll keep you posted when I circle back around to it.

[Marko Čirič]

Thank you!

[Cesar Parra]

Count me in as someone that is also interested in seeing this happen! (though I completely understand that this probably falls pretty low in the priority list seeing that the request history is few and far between)