Generating offline Symbol table for Salesforce Sandbox stuck
While trying to generate offline symbol table, it always get stuck in process of "Loading class overlays:Method" I tried doing this with both of my user licenses, on two different machines, same result. Also tried to select only several components ( just apex classes and fields ), and with with whole package. When I am saying that it stays stuck, it stays in same spot for several hours.
Any help will be appreciated.
Comments (21)
-
repo owner -
reporter Thanks Scott, I have bumped it to 1 Gb and had same issue. Will try to bump it a little bit more, and let you know.
-
repo owner Okay. Can you see anything in idea.log that confirms whether this is in fact a memory issue? Do you mind sending me your idea.log, or at least any relevant subset that might provide more info about what's going on here? We'll get you past this and to the fun part ASAP!
-
reporter - attached idea.log
My idea.log
-
reporter Hi Scott, thank you for quick response, I am submitting you my idea.log
Thanks :)
-
repo owner Yup, there's an OutOfMemoryError right where I've seen it in other cases like this. Should just be a matter of giving it enough memory.
I'm going to address this soon...just to provide some context, it only really needs that memory because I'm writing the results of each API call into JSON files as I pull down the constituent pieces of the OST using the various APIs so that I can resume in the case of something like a transient network error during those API calls. Some of the API responses are quite large and marshalling them to JSON ends up needing all that memory just long enough to create the file contents.
Let me know if you can't get the heap size adjusted adequately to get past this. Sorry for the inconvenience!
-
repo owner Dusan, if you don't mind, please let me know if that gets you going or not.
-
reporter Hi Scott, half/half. I was able to resolve it on Windows Machine by adjusting heap size in idea.exe.vmoptions to 1024Mb ( Although it still took good one hour ) Mac is still no go... (Although I just recently got Macbook Pro, and still learning how to utilize it ). Unfortunately haven't able to increase size of heap using your instructions, so I just used all mighty Google.
P.S. I believe my issues was in different thread (
#129) :) -
repo owner Wow, it took a full hour to generate your OST? I've definitely not seen one take more than five or so minutes, and that one had ~750 Apex classes, multiple installed managed packages, and a decent number of custom objects. I'm wondering if it was still bumping up against the maximum heap and thrashing the Java garbage collector or something. I may add some debug logging in the next build and, if you wouldn't mind, ask you to regenerate your OST during some down-time so that I can understand where that time is going. I don't like for you to have to wait more than a few minutes for that operation.
As for getting things going on the Mac, see if this tech note helps:
Okay, please keep me posted. I really apologize that this has been a bit rocky for you.
-
reporter yes, definitely, and I need to correct my old post, I bumped it to 2048m. Definitely took good chunk of a day, however, at least my Windows IntelliJ is up and running. Yes, I wouldn't mind running this again next time, however, probably during weekend, definitely not during work days anymore. I am guessing that it was still clashing with maximum heap ( although I thought that 2Gb is max? ) Debug log would be awesome, definitely let me know when you will manage to publish it.
Thanks,
Dusan
-
repo owner Dusan, it turns out that I actually have a decent amount of debug logging in there now...certainly enough to give me an idea of where the time is going and how it's doing against the heap ceiling. When you do this again (like you said, perhaps over the weekend), you can enable that debug logging by going into Help>Configure Debug Log Settings... From there you can add the following line:
#com.illuminatedcloud.symtab.OfflineSymbolTable
Then when you generate the symbol table, you should see entries like the following in your idea.log:
2015-10-08 09:36:09,415 [ 44100] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 202924048/297336832 (477233152 max) - after prepareFileSystem 2015-10-08 09:36:16,335 [ 51020] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 182519552/287047680 (477233152 max) - after loadInstalledPackageNamespaces 2015-10-08 09:36:18,039 [ 52724] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 142926408/258015232 (477233152 max) - after initializeLocalSymbolsFromModules 2015-10-08 09:36:21,099 [ 55784] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 148949088/258015232 (477233152 max) - after loadSystemClasses 2015-10-08 09:36:56,571 [ 91256] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 189791504/336592896 (477233152 max) - after loadCustomClasses 2015-10-08 09:38:33,465 [ 188150] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 181086856/328728576 (477233152 max) - after loadSObjectClasses 2015-10-08 09:38:33,540 [ 188225] INFO - loud.symtab.OfflineSymbolTable - Querying class summary information 2015-10-08 09:38:34,253 [ 188938] INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Class 2015-10-08 09:38:38,418 [ 193103] INFO - loud.symtab.OfflineSymbolTable - Processing 114 class identifiers. 2015-10-08 09:38:38,527 [ 193212] INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Interface 2015-10-08 09:38:38,954 [ 193639] INFO - loud.symtab.OfflineSymbolTable - Processing 8 class identifiers. 2015-10-08 09:38:38,957 [ 193642] INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Enum 2015-10-08 09:38:39,578 [ 194263] INFO - loud.symtab.OfflineSymbolTable - Processing 2 class identifiers. 2015-10-08 09:38:39,582 [ 194267] INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Exception 2015-10-08 09:38:40,155 [ 194840] INFO - loud.symtab.OfflineSymbolTable - Processing 5 class identifiers. 2015-10-08 09:38:40,158 [ 194843] INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Constructor 2015-10-08 09:38:41,888 [ 196573] INFO - loud.symtab.OfflineSymbolTable - Processing 82 class identifiers. 2015-10-08 09:38:41,900 [ 196585] INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Method 2015-10-08 09:39:06,988 [ 221673] INFO - loud.symtab.OfflineSymbolTable - Processing 1661 class identifiers. 2015-10-08 09:39:07,156 [ 221841] INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Variable 2015-10-08 09:39:11,536 [ 226221] INFO - loud.symtab.OfflineSymbolTable - Processing 242 class identifiers. 2015-10-08 09:39:15,068 [ 229753] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 295055928/442761216 (477233152 max) - after applyClassOverlays 2015-10-08 09:39:16,692 [ 231377] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 294213384/441974784 (477233152 max) - after reconcileSobjectClasses 2015-10-08 09:39:18,750 [ 233435] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 292790712/440139776 (477233152 max) - after populateEnumConstants 2015-10-08 09:39:20,976 [ 235661] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 289685768/437911552 (477233152 max) - after clearExistingOfflineSymbolTable 2015-10-08 09:39:25,093 [ 239778] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 279063640/429654016 (477233152 max) - after generateApexSourceFiles
That's the log for OST generation of a pretty large project with ~750 classes, ~75 triggers, ~100 custom objects, and multiple installed managed packages. Total time to generate was just over three minutes. That's what I want for you to be seeing!
Two questions about your host JRE for IntelliJ IDEA. First, is it a 64-bit JRE? If not, you may be hitting the limits of a 32-bit address space which for Java is actually only 1.4GB. You should definitely make sure that IDEA is using a 64-bit JRE.
Second, what JRE are you using to run IDEA on your Mac? I recently learned that IDEA has major issues running under anything but Apple's Java 6 on the Mac, so if you're on Oracle's Java 7/8, I would recommend you switch to Apple's JRE. This is just for the host VM for IntelliJ IDEA. You can set up any SDK you want for your own projects within IntelliJ IDEA. I just want to rule that out as a potential contributor here.
If you're not sure how to switch IDEA's host JRE, see here:
Let's get that info and see if we can figure out what's going on. Keep me posted!
-
repo owner Hi, Dusan. Just thought I'd check in and see if you had anything new to report on this.
-
Account Deleted I'm also finding the same issue. I work inside a mid-size org with ~150 apex classes, ~20 triggers, and ~2500 objects (we use BMC Remedyforce which contributes quite a bit to this size). When generating the Offline Symbol Table it always seems to crash when showing "Method". I've increased the Heap size to 1GB but still no change. Any ideas or further logs I can provide?
-
repo owner Hi, Tyler. Two recommendations:
- Bump up the heap size further. For some folks who are having this issue, a larger heap size than 1GB is needed only during OST generation. I know where this is going and will try to resolve it in one of the upcoming releases, but for now there's no harm in giving the JVM more breathing room during generation. You'll also want to make sure that you're using a 64-bit JVM in general.
- Update the debug log settings as described two comments back and send me your idea.log (Help>Show Log in Explorer/Finder) from the generation attempt. That will contain the duration and memory utilization of each step.
Let me know!
-
Account Deleted - attached idea.offlinesymbol.log
Hi Scott,
I finally got the Offline Symbol Table to generate. It took ~10 minutes after I upped the the mem to 2048 (which it used all of during the generation although hovered around 1 - 1.3GB most of the time).
I've included the log file so you can review in the event it helps.
Tyler
-
repo owner Thanks, Tyler. Glad you got past it, but for the size project you have, I would be expecting half that time. I'll use the provided log to see what I can do to help you get it into that range and with much less memory required to do it!
-
repo owner Okay, here's the relevant extract from that log:
2015-10-16 11:18:27,771 [ 75581] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 82485608/161452032 (2138767360 max) - after prepareFileSystem 2015-10-16 11:18:33,869 [ 81679] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 80384520/171626496 (2138767360 max) - after loadInstalledPackageNamespaces 2015-10-16 11:18:36,128 [ 83938] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 143378840/239824896 (2138767360 max) - after initializeLocalSymbolsFromModules 2015-10-16 11:18:36,129 [ 83939] INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedSystemClasses.json 2015-10-16 11:18:37,578 [ 85388] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 140426928/239824896 (2138767360 max) - after loadSystemClasses 2015-10-16 11:18:37,579 [ 85389] INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedCustomClasses.json 2015-10-16 11:18:40,445 [ 88255] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 347040120/495415296 (2138767360 max) - after loadCustomClasses 2015-10-16 11:18:40,446 [ 88256] INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedSobjectDescribes.json 2015-10-16 11:18:41,288 [ 89098] INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedSobjectDocumentation.json 2015-10-16 11:18:43,774 [ 91584] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 335414488/495415296 (2138767360 max) - after loadSObjectClasses 2015-10-16 11:18:43,779 [ 91589] INFO - loud.symtab.OfflineSymbolTable - Querying class summary information 2015-10-16 11:18:52,316 [ 100126] INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedApexClassIdentifiers-Class.json 2015-10-16 11:18:52,589 [ 100399] INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedApexClassIdentifiers-Interface.json 2015-10-16 11:18:52,596 [ 100406] INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedApexClassIdentifiers-Enum.json 2015-10-16 11:18:52,601 [ 100411] INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedApexClassIdentifiers-Exception.json 2015-10-16 11:18:52,613 [ 100423] INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedApexClassIdentifiers-Constructor.json 2015-10-16 11:18:52,693 [ 100503] INFO - loud.symtab.OfflineSymbolTable - Loading cached state from CachedApexClassIdentifiers-Method.json 2015-10-16 11:18:54,783 [ 102593] INFO - loud.symtab.OfflineSymbolTable - Querying class identifiers for Variable 2015-10-16 11:26:01,889 [ 529699] INFO - loud.symtab.OfflineSymbolTable - Processing 38775 class identifiers. 2015-10-16 11:26:34,507 [ 562317] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 1794099424/1971343360 (2138767360 max) - after applyClassOverlays 2015-10-16 11:26:36,052 [ 563862] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 1796008224/1971343360 (2138767360 max) - after reconcileSobjectClasses 2015-10-16 11:27:06,640 [ 594450] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 1795592256/1971343360 (2138767360 max) - after populateEnumConstants 2015-10-16 11:27:08,121 [ 595931] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 1795792328/1971343360 (2138767360 max) - after clearExistingOfflineSymbolTable 2015-10-16 11:27:26,585 [ 614395] DEBUG - loud.symtab.OfflineSymbolTable - HEAP: 1785849200/1971343360 (2138767360 max) - after generateApexSourceFiles
The lion's share of time and memory is going into loading the Apex class identifiers for all of the variables (just under 40K). If you compare that to the log I included above for one of my largest projects, it's a pretty huge difference (~240 vs. ~40K). Those Apex class identifiers are CRITICAL to the construction of the OST, though. They fill in major gaps about the classes in the target org vs. the Tooling API by itself. I'm assuming this large number of symbols is coming from your installed managed packages.
I'll install several large managed packages into one of my own dev orgs and generate the OST to see if I can reproduce this behavior. The APIs to query the Apex class identifiers are not the fastest in the world, so I'm not sure how much I'll be able to do about the time, but I do think I can tune the memory required during that phase. It's possible I can also affect the time a bit by seeing if I'm requesting anything that I don't end up using, but I wouldn't expect that to change much.
Thanks again for the diagnostic info!
-
Account Deleted Hi Scott,
Thanks for the update! That makes quite a bit of sense. We have close to 30 managed apps installed with the biggest being Remedyforce (>250 objects and ~1000 classes).
I'm pretty satisfied with 10 minutes. Of course, anything you can do to improve on that is most welcome :)
Tyler
-
repo owner Wow! Yeah, that's quite a bit to have to sort through to build the OST. IC calls just about every API Salesforce has including some that are undocumented in order to build the best offline representation of the system/installed Apex classes and SObjects that it can. That many API calls returning that much structured data just takes time. I can probably tune your scenario a bit, but perhaps 5 minutes is a bit optimistic! I'll see what I can do, though!
-
repo owner FYI, I have a change that should significantly lower the memory required while generating the OST, in particular while downloading the Apex class identifiers. I have one user evaluating a test build with this change and plan to release it in the next few days. It won't necessarily lower the time to generate the OST unless time was being spent thrashing the garbage collector, but it should lower the memory overhead significantly.
-
repo owner - changed status to resolved
In 1.6.4.0 I've implemented a number of significant improvements to OST. First, I found a place where the same data downloaded from the API was being transformed/copied into three different data structures when only the last one was needed, so now I'm clearing as I go along. This results in much less memory required when the class overlays are being processed. Second, I determined that the OST was including ALL symbols and not just those that are accessible, i.e., global from installed managed packages and >=protected from unpackaged classes. Now the OST only includes accessible symbols and filters for those along the way. This also reduces the amount of memory required during generation as well as creating a more compact and accurate OST in terms of what's actually available.
I'm going to resolve this issue for now hoping that these changes address users' issues with OST generation. Feel free to reopen this issue or open a new issue if the problems are still present.
- Log in to comment
Hi, Dusan. My guess is that you're hitting the same problem as in issue
#68(OutOfMemoryError while generating the OST). You can verify that by looking in idea.log. You can access that by restarting IDEA and using Help>Show Log in Explorer/Finder. The solution is to give IntelliJ IDEA a little more breathing room by bumping the maximum heap size for its host JVM:https://www.jetbrains.com/idea/help/increasing-memory-heap.html
Oh, and even on large orgs, OST generation shouldn't take more than several minutes. Hopefully with this change, you can generate the OST successfully and relatively quickly. Please let me know either way!