Very slow offline symbol table generation - especially on "Loading SObject docs"

[Fabien Schiettecatte]

Loading the offline symbol table is frustratingly slow. Network is fine, using different networks does not change the issue.

Internet is fast.

[Scott Wells]

Hi. I’ve heard reports of some orgs/instances providing VERY slow API response times right now during the Winter ‘23 preview window. That’s the only reason that comes to mind for such slow OST generation as that process effectively calls a large number of Salesforce APIs to get the required information. Can you please enable debug logging for Offline Symbol Table Generation (don’t include the !trace line), reproduce the behavior, and either attach or email the resulting idea.log file for review?

[Fabien Schiettecatte]

Hi Scott Wells,

Thanks for your lightning fast response!

I also get

Deploy - ComponentSet - No source-backed components present in the package.

Whenever I create a new class.

Deploying the class after does work however.

Perhaps the issues are related?

I will provide the requested info asap.

[Fabien Schiettecatte]

Interesting information; my co-worker is working on the same organization, and has no issues generating the symbol table quickly.

[Scott Wells]

Let’s see what the debug log says. It should show where the time is going. As for the other error message, can you also attach your project’s .iml file for review? That error message is coming from the Salesforce CLI so the first thing I’d like to see is how the source roots are configured for the project.

[Fabien Schiettecatte]

.iml

<?xml version="1.0" encoding="UTF-8"?>
<module type="IlluminatedCloud" version="4">
  <component name="FacetManager">
    <facet type="IlluminatedCloud" name="Illuminated Cloud">
      <configuration>
        <option name="connectionName" value="elf-sandbox" />
        <option name="connectionType" value="OAUTH" />
        <option name="defaultSourceRootUrl" value="file://$MODULE_DIR$/code/@unpackaged" />
        <option name="moduleContents">
          <ModuleContents>
            <option name="contentSelectionType" value="PACKAGE_XML" />
            <option name="packageXmlRelativePath" value="manifest/package.xml" />
          </ModuleContents>
        </option>
      </configuration>
    </facet>
  </component>
  <component name="NewModuleRootManager" inherit-compiler-output="true">
    <exclude-output />
    <content url="file://$MODULE_DIR$">
      <sourceFolder url="file://$MODULE_DIR$/config" type="java-resource" />
      <sourceFolder url="file://$MODULE_DIR$/code/@triggers" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/@unpackaged" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/@utils" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/absence" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/client" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/common" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/connector" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/copy" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/dplan" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/email" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/fileImport" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/finance" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/googleMaps" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/housekeeper" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/location" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/mailchimp" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/marketing" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/organizationPart" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/planning" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/ponto" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/prisma" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/salesInvoice" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/signableDocument" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/snapshot" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/survey" isTestSource="false" />
      <sourceFolder url="file://$MODULE_DIR$/code/website" isTestSource="false" />
      <excludePattern pattern="*.md" />
    </content>
    <orderEntry type="jdk" jdkName="IlluminatedCloud (elf-sfdc/elf-sandbox)" jdkType="IlluminatedCloud" />
    <orderEntry type="sourceFolder" forTests="false" />
  </component>
</module>

‌

[Scott Wells]

Okay. That certainly looks curious. Is there a reason that so many subdirectories of code have been added as source roots vs. just adding code itself as a source root? I don’t know that it’s going to cause an issue, but certainly the less source roots you configure, the better. Also, do all of those directories contain Salesforce metadata/source code?

[Fabien Schiettecatte]

All the directories include metadata/source code.

I could try just including /code, I don’t really mind. Not sure if it will work.

We have this setup in most of our orgs, it never really creates any issues.

[Scott Wells]

Okay. If that’s working elsewhere let’s not make changes. Let’s start with the OST generation debug log and see what’s going on there, then once that’s figured out, we’ll look at debug logs for the failed deployments.

[Fabien Schiettecatte]

Will provide it as soon as … the bloody thing is ready :D. I just started the generation.

[Fabien Schiettecatte]

  <div class="preview-container wiki-content"><!-- loaded via ajax --></div>
  <div class="mask"></div>
</div>

</div> </form>

[Scott Wells]

Okay, that definitely helped me to see where the time is going. Here’s the relevant extraction:

2022-09-12 22:15:28,814 [1915854]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Starting activity Loading system classes with 1 steps.
2022-09-12 22:15:41,802 [1928842]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Completed activity Loading system classes in 12988 ms.

2022-09-12 22:15:42,223 [1929263]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Starting activity Augmenting system classes with 1 steps.
2022-09-12 22:15:47,737 [1934777]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Completed activity Augmenting system classes in 5514 ms.

2022-09-12 22:15:47,779 [1934819]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Starting activity Loading custom classes with 1 steps.
2022-09-12 22:15:56,073 [1943113]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Completed activity Loading custom classes in 8294 ms.

2022-09-12 22:15:56,073 [1943113]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Starting activity Loading SObject types with 1 steps.
2022-09-12 22:15:58,292 [1945332]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Completed activity Loading SObject types in 2219 ms.

2022-09-12 22:15:59,148 [1946188]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Starting activity Loading SObject types with 12 steps.
2022-09-12 22:16:54,829 [2001869]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Completed activity Loading SObject types in 55681 ms.

2022-09-12 22:16:54,834 [2001874]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Starting activity Loading SObject docs with 700 steps.
2022-09-12 22:22:58,237 [2365277]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Completed activity Loading SObject docs in 363403 ms.

2022-09-12 22:22:58,926 [2365966]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Starting activity Loading Tooling SObject types with 5 steps.
2022-09-12 22:23:23,803 [2390843]   INFO - #c.i.i.s.p.OfflineSymbolTableGenerator - Completed activity Loading Tooling SObject types in 24877 ms.

As you can see from above, almost every step is taking quite a long time, but the step to query docs for SObject types is the long pole taking >6 mins.

Since you mentioned that this is happening against an org that does not cause the same issues for a co-worker, you didn’t happen to disable Use parallel processing for your connection, did you? That would definitely cause everything to run long as each and every API call would be serialized against the others.

[Fabien Schiettecatte]

Oh dear, that setting is indeed off, heavens knows why! I recently recreated the oauth connection, so I guess from that point on it was off.

Thank you very much. Such a silly mistake to make.

In my defense I had no clue what this setting does, so I don’t usually touch it.

It’s super fast now!

[Scott Wells]

Hey, that’s great news! Allow me to provide the user guide topic that discusses that setting (and many others) for future reference:

https://bitbucket.org/RoseSilverSoftware/illuminatedcloud/wiki/User_Guide/Configuring_Connections

Now let’s move on to the deployment issue. Can you please enable debug logging for both Metadata Deployment and Salesforce DX, then reproduce that issue and provide the resulting log for review?

[Scott Wells]

Thanks. IC2 is just running the following Salesforce CLI command:

Running command line: 'C:/Program Files/Salesforce CLI/bin/sfdx.cmd force:source:deploy -u [e-mail].s1 -o -g -x C:/Users/fabie/AppData/Local/Temp/elf_sfdc-elf_sandbox-package.xml -w 0 --json' from working directory: 'C:\Users\fabie\OneDrive\Bureaublad\workspace\elf\elf-sfdc'.

with the following temporary package.xml file for a selective deployment:

<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
    <types>
        <members>PriApiTableAddress</members>
        <members>PriApiTableContract</members>
        <members>PriApiTableCostCentre</members>
        <members>PriApiTableDimona</members>
        <members>PriApiTableHousekeeper</members>
        <members>PriApiTableSalaryComponent</members>
        <members>PriApiTableTax</members>
        <name>ApexClass</name>
    </types>
    <version>52.0</version>
</Package>

The error message that you’re seeing was the result of a CLI bug a bit back:

https://github.com/forcedotcom/cli/issues/1151

As a result, let’s start with what may be the simple fix for this issue for you. Can you please do a full uninstall/reinstall of the Salesforce CLI? I hate to ask you to do that, but I can’t even begin to tell you how many CLI-based issues are resolved immediately upon doing so, especially in environments where the CLI has been updated in-place repeatedly over time.

If the problem still occurs on a completely clean/fresh CLI install, I’ll have you try to reproduce it from the CLI proper by running the same command that IC2 is running. Assuming that it reproduces, you’ll likely need to file an issue with the CLI team in their public GitHub issue tracker.

[Fabien Schiettecatte]

reinstalling now

[Fabien Schiettecatte]

after reinstall the error changed, log in next comment

[Fabien Schiettecatte]

(by the way, I changed the .iml to just /code and it works pretty well so far, so thanks, much better, much simplified!)

‌

[Scott Wells]

Okay, now let’s look at your sfdx-project.json. Does it include the parent (or ancestor) directory of the file PriApiTableHousekeeper.cls as a configured packageDirectory?

[Fabien Schiettecatte]

Oh dear. It does not! But never did, and still worked.

Oh well, looks like it’s all just some bad configuration, my apologies.

Thank you for all your very fast and generous help!

[Scott Wells]

Okay, everything’s making good sense now, at least:

• Slow OST generation = disabled parallel processing
• Initial deployment error = out-of-date Salesforce CLI (potentially masking the following)
• Current deployment error = misconfigured sfdx-project.json not covering files intended for deployment

Sounds like you should be okay now, right? I’m going to resolve this, but if you hit something else, don’t hesitate to ask.

‌