1. Scott Wells Avatar Scott Wells
  2. Illuminated Cloud
  3. IlluminatedCloud
  4. Issues

Generating offline Symbol table for Salesforce Sandbox stuck

Issue #129 resolved
Dusan Vrugic created an issue

While trying to generate offline symbol table, it always get stuck in process of "Loading class overlays:Method" I tried doing this with both of my user licenses, on two different machines, same result. Also tried to select only several components ( just apex classes and fields ), and with with whole package. When I am saying that it stays stuck, it stays in same spot for several hours.

Any help will be appreciated.

Comments (21)

  1. Scott Wells repo owner

    Hi, Dusan. My guess is that you're hitting the same problem as in issue #68 (OutOfMemoryError while generating the OST). You can verify that by looking in idea.log. You can access that by restarting IDEA and using Help>Show Log in Explorer/Finder. The solution is to give IntelliJ IDEA a little more breathing room by bumping the maximum heap size for its host JVM:

    https://www.jetbrains.com/idea/help/increasing-memory-heap.html

    Oh, and even on large orgs, OST generation shouldn't take more than several minutes. Hopefully with this change, you can generate the OST successfully and relatively quickly. Please let me know either way!

  2. Dusan Vrugic reporter

    Thanks Scott, I have bumped it to 1 Gb and had same issue. Will try to bump it a little bit more, and let you know.

  3. Scott Wells repo owner

    Okay. Can you see anything in idea.log that confirms whether this is in fact a memory issue? Do you mind sending me your idea.log, or at least any relevant subset that might provide more info about what's going on here? We'll get you past this and to the fun part ASAP!

  6. Scott Wells repo owner

    Yup, there's an OutOfMemoryError right where I've seen it in other cases like this. Should just be a matter of giving it enough memory.

    I'm going to address this soon...just to provide some context, it only really needs that memory because I'm writing the results of each API call into JSON files as I pull down the constituent pieces of the OST using the various APIs so that I can resume in the case of something like a transient network error during those API calls. Some of the API responses are quite large and marshalling them to JSON ends up needing all that memory just long enough to create the file contents.

    Let me know if you can't get the heap size adjusted adequately to get past this. Sorry for the inconvenience!

  8. Dusan Vrugic reporter

    Hi Scott, half/half. I was able to resolve it on Windows Machine by adjusting heap size in idea.exe.vmoptions to 1024Mb ( Although it still took good one hour ) Mac is still no go... (Although I just recently got Macbook Pro, and still learning how to utilize it ). Unfortunately haven't able to increase size of heap using your instructions, so I just used all mighty Google.

    P.S. I believe my issues was in different thread (#129) :)

  9. Scott Wells repo owner

    Wow, it took a full hour to generate your OST? I've definitely not seen one take more than five or so minutes, and that one had ~750 Apex classes, multiple installed managed packages, and a decent number of custom objects. I'm wondering if it was still bumping up against the maximum heap and thrashing the Java garbage collector or something. I may add some debug logging in the next build and, if you wouldn't mind, ask you to regenerate your OST during some down-time so that I can understand where that time is going. I don't like for you to have to wait more than a few minutes for that operation.

    As for getting things going on the Mac, see if this tech note helps:

    https://intellij-support.jetbrains.com/hc/en-us/articles/206827537-Configuring-JVM-options-and-platform-properties

    Okay, please keep me posted. I really apologize that this has been a bit rocky for you.

  10. Dusan Vrugic reporter

    yes, definitely, and I need to correct my old post, I bumped it to 2048m. Definitely took good chunk of a day, however, at least my Windows IntelliJ is up and running. Yes, I wouldn't mind running this again next time, however, probably during weekend, definitely not during work days anymore. I am guessing that it was still clashing with maximum heap ( although I thought that 2Gb is max? ) Debug log would be awesome, definitely let me know when you will manage to publish it.

    Thanks,

    Dusan

  11. Scott Wells repo owner

    Dusan, it turns out that I actually have a decent amount of debug logging in there now...certainly enough to give me an idea of where the time is going and how it's doing against the heap ceiling. When you do this again (like you said, perhaps over the weekend), you can enable that debug logging by going into Help>Configure Debug Log Settings... From there you can add the following line:

    #com.illuminatedcloud.symtab.OfflineSymbolTable

    Then when you generate the symbol table, you should see entries like the following in your idea.log:

    2015-10-08 09:36:09,415 [  44100]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 202924048/297336832 (477233152 max) - after prepareFileSystem 
2015-10-08 09:36:16,335 [  51020]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 182519552/287047680 (477233152 max) - after loadInstalledPackageNamespaces 
2015-10-08 09:36:18,039 [  52724]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 142926408/258015232 (477233152 max) - after initializeLocalSymbolsFromModules 
2015-10-08 09:36:21,099 [  55784]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 148949088/258015232 (477233152 max) - after loadSystemClasses 
2015-10-08 09:36:56,571 [  91256]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 189791504/336592896 (477233152 max) - after loadCustomClasses 
2015-10-08 09:38:33,465 [ 188150]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 181086856/328728576 (477233152 max) - after loadSObjectClasses 
2015-10-08 09:38:33,540 [ 188225]   INFO - loud.symtab.OfflineSymbolTable - Querying class summary information 
2015-10-08 09:38:34,253 [ 188938]   INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Class 
2015-10-08 09:38:38,418 [ 193103]   INFO - loud.symtab.OfflineSymbolTable - Processing 114 class identifiers. 
2015-10-08 09:38:38,527 [ 193212]   INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Interface 
2015-10-08 09:38:38,954 [ 193639]   INFO - loud.symtab.OfflineSymbolTable - Processing 8 class identifiers. 
2015-10-08 09:38:38,957 [ 193642]   INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Enum 
2015-10-08 09:38:39,578 [ 194263]   INFO - loud.symtab.OfflineSymbolTable - Processing 2 class identifiers. 
2015-10-08 09:38:39,582 [ 194267]   INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Exception 
2015-10-08 09:38:40,155 [ 194840]   INFO - loud.symtab.OfflineSymbolTable - Processing 5 class identifiers. 
2015-10-08 09:38:40,158 [ 194843]   INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Constructor 
2015-10-08 09:38:41,888 [ 196573]   INFO - loud.symtab.OfflineSymbolTable - Processing 82 class identifiers. 
2015-10-08 09:38:41,900 [ 196585]   INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Method 
2015-10-08 09:39:06,988 [ 221673]   INFO - loud.symtab.OfflineSymbolTable - Processing 1661 class identifiers. 
2015-10-08 09:39:07,156 [ 221841]   INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Variable 
2015-10-08 09:39:11,536 [ 226221]   INFO - loud.symtab.OfflineSymbolTable - Processing 242 class identifiers. 
2015-10-08 09:39:15,068 [ 229753]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 295055928/442761216 (477233152 max) - after applyClassOverlays 
2015-10-08 09:39:16,692 [ 231377]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 294213384/441974784 (477233152 max) - after reconcileSobjectClasses 
2015-10-08 09:39:18,750 [ 233435]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 292790712/440139776 (477233152 max) - after populateEnumConstants 
2015-10-08 09:39:20,976 [ 235661]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 289685768/437911552 (477233152 max) - after clearExistingOfflineSymbolTable 
2015-10-08 09:39:25,093 [ 239778]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 279063640/429654016 (477233152 max) - after generateApexSourceFiles

    That's the log for OST generation of a pretty large project with ~750 classes, ~75 triggers, ~100 custom objects, and multiple installed managed packages. Total time to generate was just over three minutes. That's what I want for you to be seeing!

    Two questions about your host JRE for IntelliJ IDEA. First, is it a 64-bit JRE? If not, you may be hitting the limits of a 32-bit address space which for Java is actually only 1.4GB. You should definitely make sure that IDEA is using a 64-bit JRE.

    Second, what JRE are you using to run IDEA on your Mac? I recently learned that IDEA has major issues running under anything but Apple's Java 6 on the Mac, so if you're on Oracle's Java 7/8, I would recommend you switch to Apple's JRE. This is just for the host VM for IntelliJ IDEA. You can set up any SDK you want for your own projects within IntelliJ IDEA. I just want to rule that out as a potential contributor here.

    If you're not sure how to switch IDEA's host JRE, see here:

    https://intellij-support.jetbrains.com/hc/en-us/articles/206827547-Selecting-the-JDK-version-the-IDE-will-run-under

    Let's get that info and see if we can figure out what's going on. Keep me posted!

  12. Scott Wells repo owner

    Hi, Dusan. Just thought I'd check in and see if you had anything new to report on this.

  13. Former user Account Deleted

    I'm also finding the same issue. I work inside a mid-size org with ~150 apex classes, ~20 triggers, and ~2500 objects (we use BMC Remedyforce which contributes quite a bit to this size). When generating the Offline Symbol Table it always seems to crash when showing "Method". I've increased the Heap size to 1GB but still no change. Any ideas or further logs I can provide?

  14. Scott Wells repo owner

    Hi, Tyler. Two recommendations:

    1. Bump up the heap size further. For some folks who are having this issue, a larger heap size than 1GB is needed only during OST generation. I know where this is going and will try to resolve it in one of the upcoming releases, but for now there's no harm in giving the JVM more breathing room during generation. You'll also want to make sure that you're using a 64-bit JVM in general.
    2. Update the debug log settings as described two comments back and send me your idea.log (Help>Show Log in Explorer/Finder) from the generation attempt. That will contain the duration and memory utilization of each step.

    Let me know!

  15. Former user Account Deleted

    Hi Scott,

    I finally got the Offline Symbol Table to generate. It took ~10 minutes after I upped the the mem to 2048 (which it used all of during the generation although hovered around 1 - 1.3GB most of the time).

    I've included the log file so you can review in the event it helps.

    Tyler

  16. Scott Wells repo owner

    Thanks, Tyler. Glad you got past it, but for the size project you have, I would be expecting half that time. I'll use the provided log to see what I can do to help you get it into that range and with much less memory required to do it!

  17. Scott Wells repo owner

    Okay, here's the relevant extract from that log:

    2015-10-16 11:18:27,771 [  75581]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 82485608/161452032 (2138767360 max) - after prepareFileSystem 
2015-10-16 11:18:33,869 [  81679]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 80384520/171626496 (2138767360 max) - after loadInstalledPackageNamespaces 
2015-10-16 11:18:36,128 [  83938]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 143378840/239824896 (2138767360 max) - after initializeLocalSymbolsFromModules 
2015-10-16 11:18:36,129 [  83939]   INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedSystemClasses.json 
2015-10-16 11:18:37,578 [  85388]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 140426928/239824896 (2138767360 max) - after loadSystemClasses 
2015-10-16 11:18:37,579 [  85389]   INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedCustomClasses.json 
2015-10-16 11:18:40,445 [  88255]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 347040120/495415296 (2138767360 max) - after loadCustomClasses 
2015-10-16 11:18:40,446 [  88256]   INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedSobjectDescribes.json 
2015-10-16 11:18:41,288 [  89098]   INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedSobjectDocumentation.json 
2015-10-16 11:18:43,774 [  91584]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 335414488/495415296 (2138767360 max) - after loadSObjectClasses 
2015-10-16 11:18:43,779 [  91589]   INFO - loud.symtab.OfflineSymbolTable - Querying class summary information 
2015-10-16 11:18:52,316 [ 100126]   INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedApexClassIdentifiers-Class.json 
2015-10-16 11:18:52,589 [ 100399]   INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedApexClassIdentifiers-Interface.json 
2015-10-16 11:18:52,596 [ 100406]   INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedApexClassIdentifiers-Enum.json 
2015-10-16 11:18:52,601 [ 100411]   INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedApexClassIdentifiers-Exception.json 
2015-10-16 11:18:52,613 [ 100423]   INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedApexClassIdentifiers-Constructor.json 
2015-10-16 11:18:52,693 [ 100503]   INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedApexClassIdentifiers-Method.json 
2015-10-16 11:18:54,783 [ 102593]   INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Variable 
2015-10-16 11:26:01,889 [ 529699]   INFO - loud.symtab.OfflineSymbolTable - Processing 38775 class identifiers. 
2015-10-16 11:26:34,507 [ 562317]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 1794099424/1971343360 (2138767360 max) - after applyClassOverlays 
2015-10-16 11:26:36,052 [ 563862]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 1796008224/1971343360 (2138767360 max) - after reconcileSobjectClasses 
2015-10-16 11:27:06,640 [ 594450]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 1795592256/1971343360 (2138767360 max) - after populateEnumConstants 
2015-10-16 11:27:08,121 [ 595931]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 1795792328/1971343360 (2138767360 max) - after clearExistingOfflineSymbolTable 
2015-10-16 11:27:26,585 [ 614395]  DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 1785849200/1971343360 (2138767360 max) - after generateApexSourceFiles

    The lion's share of time and memory is going into loading the Apex class identifiers for all of the variables (just under 40K). If you compare that to the log I included above for one of my largest projects, it's a pretty huge difference (~240 vs. ~40K). Those Apex class identifiers are CRITICAL to the construction of the OST, though. They fill in major gaps about the classes in the target org vs. the Tooling API by itself. I'm assuming this large number of symbols is coming from your installed managed packages.

    I'll install several large managed packages into one of my own dev orgs and generate the OST to see if I can reproduce this behavior. The APIs to query the Apex class identifiers are not the fastest in the world, so I'm not sure how much I'll be able to do about the time, but I do think I can tune the memory required during that phase. It's possible I can also affect the time a bit by seeing if I'm requesting anything that I don't end up using, but I wouldn't expect that to change much.

    Thanks again for the diagnostic info!

  18. Former user Account Deleted

    Hi Scott,

    Thanks for the update! That makes quite a bit of sense. We have close to 30 managed apps installed with the biggest being Remedyforce (>250 objects and ~1000 classes).

    I'm pretty satisfied with 10 minutes. Of course, anything you can do to improve on that is most welcome :)

    Tyler

  19. Scott Wells repo owner

    Wow! Yeah, that's quite a bit to have to sort through to build the OST. IC calls just about every API Salesforce has including some that are undocumented in order to build the best offline representation of the system/installed Apex classes and SObjects that it can. That many API calls returning that much structured data just takes time. I can probably tune your scenario a bit, but perhaps 5 minutes is a bit optimistic! I'll see what I can do, though!

  20. Scott Wells repo owner

    FYI, I have a change that should significantly lower the memory required while generating the OST, in particular while downloading the Apex class identifiers. I have one user evaluating a test build with this change and plan to release it in the next few days. It won't necessarily lower the time to generate the OST unless time was being spent thrashing the garbage collector, but it should lower the memory overhead significantly.

  21. Scott Wells repo owner

    In 1.6.4.0 I've implemented a number of significant improvements to OST. First, I found a place where the same data downloaded from the API was being transformed/copied into three different data structures when only the last one was needed, so now I'm clearing as I go along. This results in much less memory required when the class overlays are being processed. Second, I determined that the OST was including ALL symbols and not just those that are accessible, i.e., global from installed managed packages and >=protected from unpackaged classes. Now the OST only includes accessible symbols and filters for those along the way. This also reduces the amount of memory required during generation as well as creating a more compact and accurate OST in terms of what's actually available.

    I'm going to resolve this issue for now hoping that these changes address users' issues with OST generation. Feel free to reopen this issue or open a new issue if the problems are still present.

  22. Log in to comment
Assignee
Type
bug
Priority
major
Status
resolved
Component
Offline Symbol Table
Version
1.6.1.0
Votes
0
Watchers
1
Jira Software: the preferred issue tracker for Bitbucket. Join the team!