Allow the user to attach local source to OST for installed packages

[Raihan Navroze]

Opened as "Package methods with @NamespaceAccessible are not available to orgs in the same namespace". Changed to be more clear that this is now about being able to attach local sources for types from installed packages.

---

Steps to reproduce:

1. Make a package with classes and methods marked as @NamespaceAccessible
2. Create a new project in IC, create a new scratch org within the same namespace and install the package to this new scratch org
3. Build the OST from the new scratch org with the plugin installed
4. Attempt to write code referring to the @NamespaceAccessible methods

Expected result:

The method is available in the OST and can be used to autocomplete code.

Actual result:

The method shows up in red and shows a warning of “Cannot resolve symbol”

‌

Example case:

This class and method is inside a package within the same namespace.

‌

This is in the new scratch org:

‌

(PS Same issue with namespace.TestNamespaceAccessible.testing())

[Scott Wells]

Raihan, I apologize but I don't think there's anything I'm going to be able to do about this right now. Any second-generation package with a namespace, whether managed or unlocked, applies code obfuscation to all packaged Apex types. In the case of global symbols in global types, the accessible skeleton is available, but for a class like the one in your example, the body is returned as (hidden). As a result, there's no way to access the @NamespaceAccessible interface of that type programmatically from the API.

Salesforce is aware of this, though I'm not sure when it will be resolved. Once they do resolve it, I will update IC's OST generation to ensure that the @NamespaceAccessible interface is available to callers in the same namespace.

[Matt Simonis]

As a workaround to this, I have been pulling down the outside project code annotated with @NamespaceAccessible into an untracked (gitignored) folder in the sfdx source format. This will allow the OST to pick up on those classes, but not pick it up when making commits.

[Cesar Parra]

Was running into this as well and ended up creating a tool as a (hopefully) temporary fix, so sharing here in case anyone else finds it useful:

https://www.npmjs.com/package/ost-generator

What it does is read parse through the dependent package source code (needs to be locally pulled down to your machine), and using that it updates the target OST and adds the missing definitions necessary for auto-complete to work.

[Scott Wells]

Thanks for sharing, Cesar. This got me thinking again about how to solve this until Salesforce stops applying code obfuscation to these packages that shouldn't be obfuscated. I was thinking that perhaps I could provide a way for you to attach source files to a namespace in the OST so that IC has more information about the contents of the package that produced that namespace. I think before I do that, I'll check with Salesforce to see if this is at all on their radar and, if so, whether they have any ETA for addressing it.

[Cesar Parra]

Not a problem! If this can be something directly integrated into IC that would be amazing, but I’ll be the first one to admit that this is a total workaround hack , so ideally Salesforce provides you the tools that you need for this to be done more reliably.

[Scott Wells]

I spoke with one of the Salesforce PMs today about this, but unfortunately this wasn't under his ownership. I'm reaching out to the correct PM shortly. Ultimately it comes down to the fact that installed unlocked packages with namespaces shouldn't be obfuscated but currently are. They've recognized that as a bug in the past, but I'm not sure if/when it will be addressed. We'll see what that PM says.

[Scott Wells]

Okay, I heard back from the correct PM and he said that this was fixed a bit back. So let me make sure that we're talking about the same issue here. Can you verify that you're doing/seeing the following?

1. Install an unlocked managed package with a namespace into an org.
2. Create an IC2 project against a connection to that org.
3. Generate your OST for that connection.
4. You do NOT not see the @NamespaceAccessible symbols in the OST.

Is that correct? If so, I'll play with it tomorrow and see whether I can reproduce that behavior or not. It's VERY possible that I need to update something in IC2 now that this is fixed on the server.

[Cesar Parra]

Difference is that Step 1 is a managed package and not an unlocked one.

[Scott Wells]

Gotcha. I was so focused on unlocked packages with namespaces that I missed that scenario completely. I've followed up again with the PM, but my guess is that it's not only not fixed but likely not scheduled to be fixed. Assuming I hear back from them along those lines, I'll investigate what would be required to allow users to "attach sources" for namespaces in the OST when they're available.

[Scott Wells]

Issue #1923 was marked as a duplicate of this issue.

[Anthony Sanchez]

@Scott Wells is this on your roadmap at all?

I recently switched from a monorepo with about 10 or so unlocked packages to a polyrepo where dependent unlocked packages are installed into a scratch org. I started to notice that @NamespaceAccessible classes are not coming back in the OST, although I do see references to them in idea.log during OST generation:

2023-07-11 11:06:27,182 [1016161]   INFO - #com.illuminatedcloud.symtab.ApexClassSymbolTableAdapter - Adapted source into symbol table for mvn__fflib_Application in 22 ms.

However, they remain unresolvable within the editor:

‌

[Scott Wells]

Hi, Anthony. I think first let’s see if there’s just something simple that can be done here. If I understand correctly, you have an installed managed package in namespace mvn, and your local project is presumably also in that namespace, correct? Otherwise @NamespaceAccessible wouldn’t provide access anyway. Assuming that’s the case, can you confirm that your scratch org has the mvn namespace and that the IC2 project is properly detecting that? It would show up on the scratch org connection and also in Illuminated Cloud > Configure Project. If that’s not the case, please ensure that your scratch org is created with that namespace and that IC2 is picking it up. You can use the Test button in the connection manager to force it to re-query the namespace from the org.

If that’s all good, are you seeing the mvn.fflib_Application.DomainFactory as having the @NamespaceAccessible annotation in its OST stub representation? I’m wondering if perhaps that’s getting lost.

Anyway, let’s start there. If any of my assumptions above are incorrect, please let me know. If not, please let me know if you’re having trouble getting the namespace to be recognized by IC2 for the project. If that all works, I’ll be curious to hear whether it’s just an absence of @NamespaceAccessible on the OST stubs…

[Anthony Sanchez]

If I understand correctly, you have an installed managed package in namespace mvn, and your local project is presumably also in that namespace, correct?

That’s correct. Technically, these are all unlocked packages within the mvn namespace that have been installed in the scratch org.

Assuming that’s the case, can you confirm that your scratch org has the mvn namespace and that the IC2 project is properly detecting that?

Yes, the scratch org is a namespaced org with mvn.

If that’s all good, are you seeing the mvn.fflib_Application.DomainFactory as having the @NamespaceAccessible annotation in its OST stub representation? I’m wondering if perhaps that’s getting lost.

I do not, I’m only seeing Apex stubs that are global scoped.

The fflib_Application class is not present in the mvn folder under the OST.

‌

‌

I was able to use Cesar’s npm tool to generate stubs, and that did generate stubs from the sources, which made the references resolvable within the editor.

[Scott Wells]

Okay. Thanks for answering all of the questions. Let me investigate a bit and see why it’s not including namespace-accessible symbols properly. It could still be an issue with that information being available during OST generation, but I bet it’s just a bug/oversight.

[Scott Wells]

I don’t suppose you have a scratch org with this namespace and unlocked package installed that I could access, do you? I can certainly create/set up everything needed to do try to reproduce this, but if you have such an environment that you could share safely, that would be absolutely wonderful. Zero issues if not, of course, especially if the org would contain anything sensitive. I assume, though, that a packaged version of fflib is pretty much already all open source anyway…

[Anthony Sanchez]

I can spin up another scratch org with less sensitive source that replicates the issue. How should I share access to it with you?

[Scott Wells]

That would be great and much appreciated if it’s only a small amount of effort for you. Once you’ve created the scratch org, probably the easiest thing to do is to create a local admin-level user with an email address of [e-mail] (pick whatever username you want, of course). That way I can log in, create a project, and generate an OST against the org so I can see whether the needed annotations are available.

[Anthony Sanchez]

@Scott Wells I just created a new scratch org and gave you access to it.

[Scott Wells]

Thanks! Logged in successfully, created a project, and generating the OST now. I’ll let you know what I find shortly.

[Scott Wells]

Okay, I have a fix for this, though it’s not perfect as it doesn’t explicitly use @NamespaceAccessible to determine what should be included in the OST when the org and org-only classes both have the same non-null namespace. Instead it includes everything that is protected or higher visibility. The reason is that, based on the way that it’s currently querying custom classes from the org, IC2 doesn’t have a distinction between an org-only class that’s from an installed managed package and one that’s just a free-floating org-only class. I could certainly update IC2 to query and use the ManageableState values for these classes to determine which is which, but that’s a much more extensive change than what I’ve made at this time.

Because this week’s build is already locked down, I won’t be including the fix for this issue in it. As a result, I may continue to refine it for a better signal-to-noise ratio before releasing it. I just wanted to let you know that I have made good progress, and technically with the changes I’ve made, you would have the missing symbols in the OST…but also more than those as a result of the current approach.

I’m happy to provide a pre-release build with these changes if you’d like to try them out. Just let me know.

[Anthony Sanchez]

Sure, I’d be happy to pilot the change in a pre-release.

[Scott Wells]

Sounds good. Because I’m a bit on the OCD side of things, I went ahead and implemented proper checking of manageable state as well. Fine-tuning things now, but it should give a more high-fidelity rendering of the accessible symbols in the OST based on what comes from installed unlocked packages and what is just free-floating in the org. I think what I’ll do is issue the new build tomorrow without this, then provide a build of that updated version with this for you to try out. I’ll attach it here sometime tomorrow.

[Scott Wells]

Okay, this is a build of 2.2.7.4 with enhancements to OST generation for installed unlocked packages with a namespace to include @NamespaceAccessible symbols.

To install it, download the archive (don't extract it) and use Settings / Preference > Plugins > Install plugin from disk (under the gear drop-down menu). Then restart the IDE and regenerate your OST.

Let me know if you run into any issues, and of course let me know whether it properly resolves the problem with those symbols missing from the OST.

[Anthony Sanchez]

I was able to install and pilot the pre-release build you provided. It now looks to be mostly resolved!

I did notice one quirk however on some of the method visibility resolution. The editor does show a visibility error on some methods, but they appear to me to be scoped fine.

‌

‌

This does mean that the method won’t show for code completion.

Not sure if this is specific to this codebase.

[Scott Wells]

Got it. Yeah, given that this is the first exposure of @NamespaceAccessible symbols like this, it’s not surprising that IC2 might not be considering them to be accessible from the point of reference during symbol resolution. I’ll check that out and provide a new build. Hopefully that’ll tie it off properly. I’ll update here when a new build is available.

[Scott Wells]

New build attached that should resolve the completion/resolution issues. Please let me know if you still have problems referencing/using symbols from the installed packages.

[Anthony Sanchez]

Looks to be working. I’ll continue piloting it and let you know if I find anything else. Thanks!

[Scott Wells]

Very good. Keep hammering on it, and I will as well, particularly with some large-scale test projects that optionally include a namespace, e.g., NPSP and fflib. Hopefully things will look good for inclusion in next week’s build.

[Scott Wells]

Delivered in 2.2.7.5.