Issues developing against NPSP source

[Daniel Fuller]

I’m currently having trouble with receiving an error for instanced methods and variables for pre-existing classes in my project. All instanced variables and instanced methods are showing a “Cannot resolve symbol” error. However, this isn’t happening for new classes that I create in the project.

Solutions Attempted

I have tried re-creating the offline symbol table, invalidating and restarting the cache, uninstalling and reinstalling Intellij and Illuminated Cloud.

One thing I have noticed during testing is that for any of these instance class references, when I use the cmd+B shortcut (on Mac) to navigate to the class reference, Intellij navigates me to the Offline Symbol reference for the class and not the actual class. This seems to occur anywhere I have seen this issue so I’m wondering if it’s related.

Also, all of these instances are with Apex *.cls file types.

I’m working with the NPSP open source codebase as a frame of reference.

I can provide any necessary logs, but I’m not sure where to start at this point.

[Scott Wells]

Hi, Daniel. The attempted solutions are what I'd generally recommend, but let's talk through the potential/likely problem. When you generate the OST, IC looks to see if classes found in the org are also found locally under configured source roots. If so, it won't generate stubs for those classes in the OST. That indicates to me that your project source roots aren't configured properly. Can you please attach or email your project's .iml file for review? Hopefully it's just a matter of adjusting how the source roots are configured, regenerating your OST one more time, and you'll be in good shape.

[Daniel Fuller]

@Scott Wells thanks for the reply. I’ve attached the .iml file above.

[Scott Wells]

Thanks. At a high-level the .iml file looks okay, but just for completeness' sake I'm trying to build out an org with NPSP in it so I can attempt to reproduce the behavior you're seeing and provide more concrete guidance. That's taking a bit, but I'll let you know what I find shortly.

[Daniel Fuller]

@Scott Wells

sounds great. thanks for the help! As an update, I also noticed that the symbols seem to be missing for custom labels as well when referencing in Apex via the System.Label dot notation.

‌

[Scott Wells]

Okay, after a heck of a grind getting this thing deployed, I think I have some insight into what's going on. The NPSP project from Git is....quite non-standard. It's stored in metadata format (as opposed to Salesforce DX's source format), but it also includes an sfdx-project.json file which indicates to IC2 that it's a Salesforce DX project and should therefore be stored in source format. As a result, things get...confused. If you don't mind, please try the following changes and see if things improve:

1. Rename sfdx-project.json to sfdx-project.jsonx so that the project is no longer determined to be in source format.
2. If you haven't already, check Use deploy/retrieve/delete instead of push/pull on your Cumulus__aerilon connection so that IC2 doesn't try to use SFDX push/pull when working against that org.
3. With the project closed, in your .iml file, change the ModuleContents block to:

        <option name="moduleContents">
          <ModuleContents>
            <option name="contentSelectionType" value="PACKAGE_XML" />
            <option name="packageXmlRelativePath" value="src/package.xml" />
          </ModuleContents>
        </option>

and then reopen the project. I have a locally that will allow you to do that from the UI, but for the moment you'll need to do it explicitly.

Then regenerate your OST and let me know whether that improves the situation or not.

[Scott Wells]

Can you provide an example of something that's resulting in this issue just so I can verify that I don't see the same problem in my local version of the same project?

[Scott Wells]

Oh, and regarding the unresolved label references, yes, I do see that locally. I'm going to debug that next. The ones that are found locally in the .labels files should be resolved properly. The ones that are sourced from installed managed packages via namespace qualification are a different beast, though, and I also have on my list to see whether I can add those to the OST so that they resolve properly as well.

[Scott Wells]

Well, the good news is that installing and configuring NPSP has yielded a pretty significant number of things I'd like to address. I'm just going to work through an NPSP-motivated set of fixes over the next several days (or whatever it takes, but none of this looks crazy). Let me know whether the steps I provided a few comments back help or not, and either way I'm going to be working on many of the things I'm seeing in this project as they will undoubtedly lead to improvements on more complex (and in this case, strange) projects such as this one. I may provide one or more test builds via this issue if you don't mind trying out key checkpoints along the way.

[Daniel Fuller]

@Scott Wells Thanks so much for all the troubleshooting and help! Updating the ModuleContentsblock worked. I did have to close IJ and reopen before the new OST generation took effect. I would also be happy to test and verify at any checkpoints along the way for any NPSP-motivated fixes that you add to a test build.

On a side note, do you think the sfdx-project.jsonfile will be continue to be an issue in the event that I create a new project from this repo again that has this sfdx file in it?

[Scott Wells]

I'm glad to hear that helped, Daniel, and I'm actively working through the various issues revealed by this project.

As for the sfdx-project.json file, I'm trying to decide what to do there. Technically speaking, this project is invalid as it's somewhat metadata format/old school and somewhat source format/new school. I noticed that VS Code doesn't seem to know what to do with it either...or at least it didn't seem to recognize it as a valid project even when it was configured against a scratch org.

As a result, I think I may follow up with the NPSP maintainers and ask them why it is the way it is given that it's going to cause issues for anyone working with it regardless of tooling. It's possible I'll learn from such a conversation that there is a good reason for it to be the way it is which would give weight to supporting this type of hybrid project in a more first-class manner. It wouldn't be hard to do so, but I hate to add a new feature for one project--albeit a broadly used one--that isn't technically valid.

I'll put a feeler out while I'm working on the other issues and see what comes of it.

[Daniel Fuller]

@Scott Wells I’m on the Salesforce.org team, so I might can steer you in the right direction for that conversation if you’d like.

[Scott Wells]

Daniel, that'd be great if you wouldn't mind. If an email connection is convenient, feel free to send a joint email to [e-mail] and whoever else might be relevant.

[Scott Wells]

Daniel, just a quick update on this work. I've already managed to implement fixes that resolve about two-thirds of the issues found in the NPSP project. The remaining ones fall primarily into two categories: 1) references to non-SObject/field metadata in installed managed packages, specifically labels and field sets; 2) dynamic Visualforce which isn't currently implemented in IC2. There are a few other small things, but those two dominate what's left.

I'm probably going to save dynamic Visualforce for last as it's a pretty huge undertaking; it effectively constitutes a separate object model that's proxied into Apex. Once I get everything but that in place I'll likely post a build here that you can install so you can see how things look as well. My guess is that won't happen until Tuesday of next week at the earliest. I'll post an update here when it's ready, though.

[Scott Wells]

Okay, here's a pre-release build that doesn't address everything, but it does address quite a bit including:

• Better (though still not perfect) handling of hybrid traditional/metadata format / SFDX/source format projects.
• Fixes for issues getting SObject/field describe metadata for certain local SObjects so that they are rendered properly into the OST. This eliminates a huge swath of unresolvable SObject/field symbols.
• Filled in missing Limits.getAsyncCalls()/getLimitAsyncCalls() system methods.
• Currently adding some referenced packaged metadata to the OST when generated, specifically custom labels, but also planning to add field sets from packages SObjects.
• Fixed a parser issue where string literals (specifically in string concatenation expressions) can start with a leading +. Not sure why one would do this, but it's technically legal and used in the NPSP source.
• Fixed an issue with type inference for Apex label expressions so that they properly evaluate as type String.

There are a few others as well. One thing I'm using as a litmus test is a full batch Apex code inspection run across the source base, and when I started I would get ~1800 reported issues. With the fixes that I've currently implemented, that's down to ~350 or so, the overwhelming majority of which are from unresolvable dynamic Visualforce references. That one is last on my list because, frankly, it's VERY involved and I've only heard about its current omission from one other user. I'll do something...but at least for now it might just be treating those references as exceptions in the unresolvable reference code inspection so that they're not flagged as errors; you still won't get full code completion for those symbols with that approach, though.

You'll also still need to move the sfdx-project.json file out of the way for IC2 to see NPSP as a metadata format project even when working against a scratch org. Otherwise it will try to treat it as source format which it's not. I'm also still trying to figure out exactly what I want to do with hybrid projects like this.

You can install this build by downloading the archive (don't extract it) and then using Settings / Preferences > Plugins > Install plugin from disk (under the gear drop-down menu). After installing and restarting the IDE, I recommend that you rebuild your OST completely as many of these fixes end up in the OST. Any feedback you may have is greatly appreciated.

I'm going to keep working on all of this but will probably do a release on Thursday of this week with all of the fixes implemented so far plus a few other things I already have ready to go.

[Daniel Fuller]

@Scott Wells I did some testing on the install you attached. I kept the sfdx-project.jsonx file renamed. It looks like I’m seeing ‘cannot resolve symbol errors' again. I know that we edited the .iml file before so I’m not sure if that would need to be updated. Let me know if there are any specific places I should check for the fixes or any additional changes I would need to make.

[Scott Wells]

Daniel, here's a new build with quite a few additional fixes including support for hybrid SFDX/metadata format projects. Install it the same way as before and then regenerate your OST again as many of these changes end up in a newly-generated OST.

My .iml file looks like:

<?xml version="1.0" encoding="UTF-8"?>
<module type="IlluminatedCloud" version="4">
  <component name="FacetManager">
    <facet type="IlluminatedCloud" name="Illuminated Cloud">
      <configuration>
        <option name="connectionName" value="npsp" />
        <option name="connectionType" value="SFDX" />
        <option name="defaultSourceRootUrl" value="file://$MODULE_DIR$/src" />
        <option name="moduleContents">
          <ModuleContents>
            <option name="contentSelectionType" value="PACKAGE_XML" />
            <option name="packageXmlRelativePath" value="src/package.xml" />
          </ModuleContents>
        </option>
      </configuration>
    </facet>
  </component>
  <component name="NewModuleRootManager" inherit-compiler-output="true">
    <exclude-output />
    <content url="file://$MODULE_DIR$">
      <sourceFolder url="file://$MODULE_DIR$/src" isTestSource="false" />
    </content>
    <orderEntry type="jdk" jdkName="IlluminatedCloud (NPSP_20210223/npsp)" jdkType="IlluminatedCloud" />
    <orderEntry type="sourceFolder" forTests="false" />
  </component>
</module>

As of this build, I can make the change below to sfdx-project.json which tells IC2 that even though that file is present, the project is in metadata format:

{
  "packageDirectories": [
    {
      "path": "src",
      "default": true
    }
  ],
  "namespace": "base_dev",
  "sourceApiVersion": "48.0",
  "plugins": {
    "illuminatedCloud2": {
      "format": "metadata"
    }
  }
}

The relevant portion is of course under plugins.

You may also notice that the namespace I'm using is base_dev. Because I can't create scratch orgs in the npsp namespace, I'm using my own namespace of base_dev and using IC2's support for namespace translation to develop in npsp but deploy to an org that uses base_dev. I'm assuming you're working against a scratch org with the npsp namespace, no? If not, what is the namespace of your scratch org?

And have you configured your scratch org to work like a normal org by enabling Use deploy/retrieve/delete instead of push/pull on that scratch org connection?

FWIW, with this build I'm down to about 4-5 errors aside from the ones from dynamic Visualforce.

[Scott Wells]

All of the fixes from the previous attached pre-release build have now been released in 2.1.6.5. I'm not resolving this one yet, though, as there are a few others things I may yet do. I'd also like to get feedback from you and others developing NPSP as to how effective these fixes have been when working with the project. Note, though, that it's going to be important that you configure the project as previously described and then regenerate your OST. That will leave the following remaining known issues:

1. Still no support for dynamic Visualforce in Apex.
2. Apex references to field sets for packaged SObjects don't yet resolve. I may knock this out next.
3. There's an unresolvable reference in the expression Schema.DataImportBatch__c.Name.getDescribe().getLabel() because Schema.DataImportBatch__c.Name is ambiguous and is resolving to Schema.DescribeSObjectResult.name instead of the field DataImportBatch__c.Name. This is more complicated because both are valid resolutions of the reference up to that point, so to disambiguate a level of lookahead and required to see which way it's being used.
4. This project is stored in metadata format overall (e.g., *.object files instead of decomposed *.object-meta.xml files), but the metadata under documents have file type-specific extensions instead of *.document. It's possible that this is valid in metadata format now and IC2 just doesn't handle that (yet), but right now those aren't recognized properly as documents.
5. A number of issues with the anonymous Apex scripts under the scripts folder:
   • They should be named *.apex instead of *.cls as they are not actually valid Apex types. The current names cause them to be evaluated as if they were Apex types which results in a number of error reports that wouldn't be reported in anonymous Apex.
   • They include constant declarations of the form public static final ... which are technically valid in anonymous Apex, but adding Apex parser support for IC2 has proven tricky. Those result in parser errors even when those files are renamed to *.apex, though that's easily worked around by just making them local variables instead of fields of a non-existent parent class.
   • The include macros of the form %%%NAMESPACE%%% and %%%NAMESPACE_RT%%% which are going to result in Apex parser errors no matter what. There are good ways to accomplish the same goal without using such macros. I've done this successfully in both of my ISV roles over the past decade, so let me know if you or the team want some input on that.
6. I also had trouble deploying some of the project metadata without modification, e.g., all of the objects/np*__*.object files required additional XML elements before they could be deployed successfully.
7. Some of the entries under featureParameters are incorrect, though they still deploy successfully (IC2 currently requires another fix coming in the next build), e.g., Data_CountErrorLog.featureParameterInteger has a top-level XML element type of FeatureParameterBoolean. This mismatch exists in several of the other files (seven total, I think).

Once you install the latest IC2 build, let me know if you aren't sure how to get the project configured for the best results. We can even have a screenshare to make sure you're seeing (no worse than) what I'm currently seeing. Then we can chat about next steps as I'd love to make sure that this project is as well-supported by IC2 as possible given how broadly it's used.

[Scott Wells]

Updated the title to more accurately reflect the nature of this issue.

[Scott Wells]

I just released 2.1.6.6 which fixes a few more of the issues found in the NPSP project:

1. Apex references to field sets for packaged SObjects now resolve properly. Note that the OST must be regenerated first.
2. The invalid compile-time SObject/field reference from item 3 above now resolves properly.
3. Feature parameters metadata now works properly.

The primary remaining issue is with dynamic Visualforce. I may look at that next, though it will be week-after-next as my kids' spring break is next week and I'll be going into support-only mode for the duration.

I'm happy to push a Git branch with what I've done to make this work as well as possible. That might be a good way to seed discussion with the NPSP maintainers. Thoughts?

[Scott Wells]

FYI, the next build (likely next week as my kids are on spring break this week) will include full support for dynamic Visualforce in Apex, e.g.:

That should knock out the last major swath of issues I've found in this project that seem to be specifically with Visualforce. Once that's out there I think it would be very useful to chat with one/some of the NPSP project maintainers about the remaining issues that seem to be with project structure and/or content to see what should happen there (if anything).

[Daniel Fuller]

@Scott Wells Thank you for all of the updates. I will install 2.1.6.6 today and start some more thorough testing, and I can let you know if I run into any issues. Once I’ve confirmed those fixes, I’ll reach out and see what we can do about having a chat with some more of the NPSP maintainers. I’ve already discussed several of these fixes with some of the maintainers and there is a lot of excitement around them. Thanks again!

[Scott Wells]

Daniel, if you continue to see many issues unrelated to dynamic Visualforce in the latest official build, don't fret. Let me know and perhaps I'll push my branch so you can see how I've configured the project and some other minor things I've had to do to resolve some issues. And then perhaps next week we can chat about what to do about those remaining issues (again, if anything).

[Daniel Fuller]

@Scott Wells I’ve been performing some testing of the fixes related to NPSP for 2.1.6.5 and 2.1.6.6 this week. Everything is looking good from what I can see except one lingering issue related to the “cannot find symbol” error on instance method references. This doesn’t seem to occur in all instances, but it appears there are some instances in the project where the class instantiation is resolving to the OST entry for the class instead of the actual Apex class in the module. One example of this is in the GE_PaymentServices_TEST apex class in the NPSP repo on line 62 in the master branch where paymentServices is an instance of the GE_PaymentServices class that’s calling the createTransaction() method, and the method is returning a “cannot resolve symbol” error while the class name resolves to the instance of the class in the OST.

[Scott Wells]

Daniel, are you on the absolute latest release build now? I fixed an issue with that in the most recent one. If you're a revision behind, please update; if not I'll investigate and see if perhaps there's still one lingering instance of that issue where it resolves to the OST instead of the local source.

I will likely post a new build tomorrow, but it's just small fixes and won't yet include the dynamic Visualforce enhancements. That will be coming next week, likely Thursday.

[Daniel Fuller]

@Scott Wells I’ve added the screenshot below from the plugin config. Looks like I’m on 2.1.6.6

‌

[Daniel Fuller]

@Scott Wells I performed some more testing on the issue I mentioned above regarding the instance names resolving to entities in the OST. I rolled back IC2 to version 2.1.6.4, which was the version before the hybrid directory structure changes in 2.1.6.5. I also used the same fix we first tried by renaming the sfdx-project.json file to sfdx-project.jsonx and then rebuilt the OST and reopened the project. This seemed to resolve the issue with OST entity references in the code where it’s currently happening, particularly in the GE_GiftEntryController in the NPSP repo, which is one class I found that seemed to have a number of these occurrences.

I then installed version 2.1.6.5 and performed the same steps above and the OST reference issue resurfaced. I tried in 2.1.6.5 both with the sfdx-project.json file with the plugins property in it as well as the file renamed with the .jsonx suffix and the issue was the same.

Not sure if that helps narrow anything down but wanted to share in case it might.

[Scott Wells]

Okay, I've just wrapped up full Apex support for dynamic VF components, both system and custom. That includes code completion, reference navigation, quick documentation, parameter info for component constructors, etc. With that in place, I get a 100% clean code inspection run (meaning no errors, though LOTS of warnings) across my local branch--at least the contents of src--which includes the following small changes (some of this is repeated from an earlier comment):

• I had to add the "plugins": { "illuminatedCloud2": { "format": "metadata" } } JSON block to sfdx-project.json so that IC2 knows that this project is in fact in metadata format even though it has an sfdx-project.json file. It sounds like perhaps this wasn't working for you, Daniel?
• I still have to comment out the LWC component usages in Aura components, e.g., the usage of c:utilIllustration in BDI_ManageAdvanceMapping.cmp. There are no local errors in the IDE when uncommented, but they just fail to deploy with, e.g., FIELD_INTEGRITY_EXCEPTION - ParseError at [row,col]:[25,23].
• My org doesn't have multi-currency enabled, so the dynamic SOQL queries that reference multi-currency fields, e.g., ConversionRate, can't resolve those fields since they're not rendered into the OST. I've suppressed the result code inspections with a suppression comment of the form //noinspection ApexUnresolvableReference.
• Several of the *.featureParameterInteger files use a root XML element of FeatureParameterBoolean that I've changed to FeatureParameterInteger. Interestingly these deploy just fine, but when you retrieve the same metadata form the org, the root element is corrected.
• I had to add minimal scaffolding to all np*__*__c.object files to allow them to deploy, typically just label, visibility, etc., elements that match exactly what's already in the org anyway if you retrieve the packaged versions of those files.
• I renamed the files under documents/NPSP_Documents to use *.document and *.document-meta.xml extensions as appropriate and change the values of the name elements in the meta.xml files to provide the real names of those files on disk so that they're processed correctly.

I plan to release these fixes on Thursday morning. I'm also going to see if I can reproduce the issue Daniel is seeing where it navigates to the OST version of a class instead of the local version. Between this and the fact that the change to sfdx-project.json to add the plugins clause didn't help Daniel, I'm wondering if it's due to how the project source roots are configured. For reference, here's my .iml file for this project now:

<?xml version="1.0" encoding="UTF-8"?>
<module type="IlluminatedCloud" version="4">
  <component name="FacetManager">
    <facet type="IlluminatedCloud" name="Illuminated Cloud">
      <configuration>
        <option name="connectionName" value="npsp" />
        <option name="connectionType" value="SFDX" />
        <option name="defaultSourceRootUrl" value="file://$MODULE_DIR$/src" />
        <option name="moduleContents">
          <ModuleContents>
            <option name="contentSelectionType" value="PACKAGE_XML" />
            <option name="packageXmlRelativePath" value="src/package.xml" />
          </ModuleContents>
        </option>
      </configuration>
    </facet>
  </component>
  <component name="NewModuleRootManager" inherit-compiler-output="true">
    <exclude-output />
    <content url="file://$MODULE_DIR$">
      <sourceFolder url="file://$MODULE_DIR$/src" isTestSource="false" />
    </content>
    <orderEntry type="jdk" jdkName="IlluminatedCloud (NPSP_20210223/npsp)" jdkType="IlluminatedCloud" />
    <orderEntry type="sourceFolder" forTests="false" />
  </component>
</module>

Once I get this next update out, let's schedule some time to chat about next steps, remaining issues, etc.

[Scott Wells]

Daniel, I've set up my local project so that all of the local Apex classes are also found in the OST as stubs. I'm not seeing the same behavior where it navigates to the OST stub, though. You said it happens in GE_GiftEntryController, but on what specific actions? Can you provide a few concrete examples of things that open the OST stub instead of the local class? I'm also wondering if this has to do with how your project is configured for namespaces, so if I'm unable to reproduce it using whatever specific examples you cite here, we'll look into that next.

[Scott Wells]

Apex support for dynamic VF component dev has been delivered in 2.1.6.8. With that I believe that all issues that I'd planned to address have been fixed and delivered. I'm resolving this issue, but I definitely still want to have a discussion about what I've had to to do to the project to get it working as cleanly as I have locally since I have a small changelist (described above) relative to the current master branch.

[Scott Wells]

FYI, I figured out the issue with the Aura components that contained LWC component references. It was in fact an IC2 issue with namespace translation (only when that's used), and I have a fix locally for the next build.

[Scott Wells]

I think I also just reproduced the issue where it navigates to the OST source instead. Seems to be an issue with how it determines the namespace of the local Apex source vs. what's been deployed to the server. I'm reopening this until I get both of these issues resolved properly.

[Scott Wells]

Okay, I've addressed all other known issues here. Resolving again.