Apex Editor incorrectly flags service invocation for managed package DLRS RollupService as invalid

[Alan Birchenough]

I am using managed package DLRS 2.11. The Apex editor is flagging calls to the rollup service as invalid. See screenshot:

This appears to be because the OST does not include the RollupService class. As you can see from this screenshot, only the SObjects within the package have been included:

What do I have to do to see the global/public services class(es) reflected in the OST and hence to avoid these spurious errors in the Apex editor?

I attach my most recent log, although I am not sure it will be very helpful since it doesn't mention RollupService anywhere.

Also I wasn't sure how to prioritize this issue since it is causing me grief but seems to belong somewhere between major and minor. Feel free to knock it back to "minor" depending on your own assessment.

[Scott Wells]

Alan, you may be hitting an issue that I've logged with Salesforce where they're not properly including all accessible symbols from installed managed packages in their API results. Looking at the provided log, We can confirm that pretty easily. Do you mind opening Developer Console and running the following SOQL query using the Tooling API?

SELECT Id, Name, NamespacePrefix, Body
FROM ApexClass
WHERE NamespacePrefix = 'dlrs'
ORDER BY Name

Then see what gets returned for things you'd expect to be present, e.g., RollupService. My guess is that either no row is returned for that class or the body returned for that class is incorrectly obfuscated.

Assuming that's the issue, the good news is that my understanding is that this bug is currently under review by the team, so I'll check and see if I can get a status for it.

[Alan Birchenough]

Actually, RollupService shows up nicely in the query result. The body is returned as follows:

/* This file is generated and isn't the actual source code for this managed global class. This read-only file shows the class's global constructors, methods, variables, and properties. To enable code to compile, all methods return null. */

global class RollupService {
    global static Exception LastMetadataAPIConnectionException { get; }
    global RollupService() { }
    global static Boolean checkMetadataAPIConnection() { return null; }
    global static List<SObject> rollup(List<SObject> childRecords) { return null; }
    global static void rollup(List<dlrs.RollupService.RollupToCalculate> rollupsToCalculate) { }
    global static void rollup(Map<Id, SObject> existingRecords, Map<Id, SObject> newRecords, Schema.SObjectType sObjectType) { }
    global static Id runJobToCalculate(Id lookupId) { return null; }
    global static Id runJobToCalculate(String lookupId) { return null; }
    global static Id runJobToCalculate(Id lookupId, String masterWhereClause) { return null; }
    global static Id runJobToCalculate(String lookupId, String masterWhereClause) { return null; }
    global static Id runJobToProcessScheduledItems() { return null; }
    global static void testHandler(SObject dummyChildRecord) { }
    global static void triggerHandler() { }
    global static void triggerHandler(Schema.SObjectType childObjectType) { }
    global class RollupToCalculate {
        global Id parentId;
        global String rollupSummaryUniqueName;
        global RollupToCalculate() { }
    }
}

[Scott Wells]

Hmmmmm...that's really odd then. Do you mind regenerating your OST with debug logging enabled as documented here:

http://www.illuminatedcloud.com/support/debuglogging

You can leave out the trace row. Then please either attach or email the resulting idea.log file(s) (there may be multiple for that operation given the logging verbosity).

[Alan Birchenough]

OK. I will just paste in the other three applicable settings for OST generation then.

[Alan Birchenough]

Attached is a zip of a folder containing a copy of the two log files appended / created when I ran full OST regeneration with those debugging options enabled. Thanks.

[Scott Wells]

Alan, thanks for providing that log. Unfortunately it didn't include some debug logging that I thought I'd put into the main release. I'm attaching a build with that logging here. Do you mind downloading it and installing it using Settings>Plugins>Install plugin from disk, then regenerating the OST and attaching all resulting logs? That should give me what I need...for real this time! Thanks.

[Alan Birchenough]

Sure. I will get to it as soon as.

[Alan Birchenough]

With apologies for the delay, I attach the log after installing the zipped plugin and re-establishing the OST debugging log settings. Please let me know if you need anything further. Thanks.

[Alan Birchenough]

Hi. Is there an update on this? Thanks.

[Scott Wells]

Hi, Alan. Sorry for the long delay. Between GDPR readiness and then Summer '18, I've been sidetracked by items with concrete deadlines. I will return to this for the next update and see if I can address it for you.

[Alan Birchenough]

Cool. It's not that urgent in the grand scheme of things so I would take it in stride. I just wanted to make sure that between us it didn't drift out of view.

[Alan Birchenough]

A similar issue came up in another org. (For my reference, this was the DE-FFDev org.)

References to the Order standard object and its fields are being flagged as erroneous. On examination of the OST, the Order object is not present. I have fully regenerated several times, to no avail.

I am aware that you weren't able to look at the previous occurrence of this incident yet, so when you are ready to investigate this second occurrence, let me know and I will provide all the necessary exhibits in terms of debug log and OST.

Thanks.

[Scott Wells]

Thanks, Alan. The next bug fix release is to address code inspection false negatives. This issue is already on the list for that effort. I'll take a look at both manifestations.

[Alan Birchenough]

Just wanted to let you know that this problem is making it difficult for me to keeping working with IC in certain orgs. Again, in DE-FFDev (my reference) I am seeing lots of objects to which I have full RCEDVM access and FLS for every field just not showing up at all in the OST. In this example, in addition to standard Order not showing up, which I wrote about earlier, I am not seeing any symbols related to pse__Proj__c (the most important object in the pse namespace) or to Sku_Effort_Map__c or Sku_Effort_Map_Entry__c. It is very hard to continue to use the IDE when the OST is providing so little information about the objects and fields, not to mention the noise generated by all the spurious error messages.

All of which is to say: If there is any diagnostic information I can provide that will help you fix this problem, I am highly motivated to do so. Just let me know.

Thanks.

[Scott Wells]

Alan, I definitely want to get this addressed for you as well. By far the simplest way for me to fix it is if I can reproduce it easily. Do you have some org where this happens that is also safe for me to access? I'm not asking for access to sensitive info, but if you perhaps had an org with just these problematic objects deployed and where you see the issue in IC, that would do it. If not I'll probably need to figure out how to get additional diagnostic from your install in those orgs, and that's totally fine...just trying to think about the quickest path to resolution.

[Alan Birchenough]

I'll see what I can come up with regarding an org in which to reproduce the issue(s).

[Alan Birchenough]

Having thought about this and slept on it, I honestly think that - initially at any rate - providing detailed diagnostics is going to work better than trying to provide an org that will make it fail. Since I don't know what it is about these orgs that is causing the problem, it would take me quite a bit of experimentation to get to the point where the problem can always be reproduced in a different org. Perhaps we could start with opening up the diagnostics and then, if we get a clue as to what is going wrong we could try setting up an org for you to work with directly at that point?

I have another related question. On a previous occasion when we were looking at OST issues, you had me go look at a particular file in the project that recorded which objects had been detected, and how they had been classified (managed, unmanaged, standard, custom, etc.). Now I am having a senior moment: Where do I go to find that information? I am curious to see why over half the pse package is missing and the standard Order object is missing and wonder if those same objects will be missing from this underlying list of objects, or whether the objects are there but somehow have incorrect attributes.

[Alan Birchenough]

FYI, I did just go to one of my dev orgs (Trailhead playground actually) to see what the OST contained, and FWIW the standard Order object is showing up just fine in there. (I don't have any managed packages installed there so I can't comment on that.)

[Scott Wells]

Thanks for the updates, Alan, and I'm fine debugging this using diagnostic logging. To your question about where to see what's happening during OST generation, I think the immediate next step is to get a new log on the absolute latest build of IC of your OST generating, send that over, and clearly call out what aspects of the OST you'd expect to be present that aren't after generation. First set up OST generation for debugging as documented here:

http://www.illuminatedcloud.com/support/debuglogging

Go ahead and add the !trace entry as well, and note that the leading hashes (#) are required. Then generate the OST, and finally send me all resulting logs using Help>Compress logs and show in Explorer/Finder. Feel free to email them assuming they'll contain potentially sensitive info. Again, in that email please provide some pointers on exactly what I'm looking for. I can certainly derive those details from the ongoing discussion here, but for the sake of expedience, the more (fresh) details provided, the better.

I'll see what I can grok from that, and if it doesn't yield the full answer, hopefully it'll let me know where additional logging would help to get the answer. I'll instrument accordingly and provide you a new build with those changes, then we'll run another cycle.

Thanks! Hopefully we'll get to the answer (and solution!) quickly with this approach.

[Scott Wells]

Alan, I've installed the package in one of my orgs to see if I can reproduce the same issue. After regenerating my OST post-package install, I do have access to dlrs.RollupService:

and of course both code completion and reference resolution work properly against it.

Do you mind running the following query against the org(s) with this issue:

SELECT Id, Name, NamespacePrefix, Body
FROM ApexClass
WHERE Name = 'RollupService'

I get the following results:

Id                    Name             NamespacePrefix    Body
01p61000008lbCmAAI    RollupService    dlrs               /* This file is generated and ...

That's pretty much how IC populates the custom classes-based stubs in the OST, so I'd be curious to see whether that query produces a useful result set for you.

Let's get that info and see whether it shows us anything useful.

[Alan Birchenough]

Hi -

I am due both to do this and gather the debugging information you asked for, hopefully this weekend. Thanks for your diligence in pursuing this issue.

Regards, -- Alan.

[Alan Birchenough]

Actually, if you look at an earlier entry for this issue, you will see that I did something very similar before, and got a full and correct query result for RolllupService out of it. The difficulty is, I no longer have access to that org - it was a client's org - and these failures to generate seem to happen randomly, so I can't guarantee to reproduce the issue specifically for dlrs in any other org. For example, now I have an org (let's call it MT FULL) in which the pse__Proj__c and pse__Project_Task__c objects don't show up in the OST, although others, such as pse__Project_Task_Assignment__c do. Also, whereas the OST for one org doesn't pick up the standard Order object, another one does. So far I have not been able to make any sense of when this happens and when it doesn't.

At any rate, if you want me to install DLRS in a further org and generate an OST for it, please let me know.

[Scott Wells]

No, that's fine. Sorry that I missed the previous instance of the same discussion! I just jumped back into this issue and wanted to try to reproduce the original manifestation of the issue as closely as possible, so I started with dlrs. Okay, so let's go back to what I requested on Thursday and get a full debug log of an OST generation in the current org that's causing you issues so that I can see what I can find in the log around the problem objects.

One more potentially dumb (and repeated) question while I'm here...are these objects that aren't showing up in the OST visible to your user profile? And is Ensure Field Read Access enabled on the connection you're using or not? Since the most recent manifestation is around objects and not classes, I just want to rule out authorization.

Thanks for your patience...I'm sure we'll get it figured out soon.

[Alan Birchenough]

I have RCEDVM and full FLS on all the objects that are not showing up in the OST. Where would I look to check the "Ensure Field Read Access" setting?

[Scott Wells]

It's an attribute of the connection. It defaults to enabled for non-production org types, so chances are it's enabled. Let's verify, though.

[Alan Birchenough]

Ensure Field Read Access is on in every sandbox where I have this issue.

BTW, I wanted to show you what my OST entries for the dlrs namespace look like in TV-ICON, which is the one in which dlrs.RollupService cannot be resolved. Note that, in contrast to your OST screenshot, there are no Apex classes showing up within that package in my OST, even though I can successfully query them in SOQL. Admittedly this doesn't tell us a whole lot more from a diagnostic POV so I will now go ahead and enable the debugging / tracing.

[Scott Wells]

Thanks for confirming that, Alan. I expected that would be the case, but always good to try out things that could resolve it with little additional effort. So yeah, next step is to get those SUPER verbose logs so I can review them. Feel free to email them to me if you'd prefer.

[Alan Birchenough]

Well I am a little dismayed by what just happened. Recall that there are two orgs included in this issue: one in which dlrs.RollupService cannot be resolved, and another in which all manner of objects seem to be missing.

First I fully regenerated the OST in the org where dlrs.RollupService could not be resolved. When the process completed, dlrs.RollupService was resolved! I looked in the OST, and this time all the classes had been included, but none of the objects as in my last screenshot. I took out the debug settings and regenerated again, and this time both classes and objects were generated. (Each time I am selecting full generation.) It seems that this operation is not behaving deterministically.

On to the second org. I ran full regeneration with the debug settings in place and many (for all I know, all) of the previously missing objects were still missing, so at least that behavior was consistent. However, 149MB seems large for an email attachment; are you sure it will reach you? If so, please remind me what email address I should use. Thanks.

[Scott Wells]

Ah, okay. So here's an important question based on something I found just yesterday...do you happen to have more than one copy of any of the .object files in the local project? I ask because I found that during OST generation, it looks for local versions of both .object files and .cls files to use when generating the OST. I observed some seemingly non-deterministic behavior during OST generation on one of my own projects where I had a work directory containing an older version of some of the metadata files. IC was finding both copies and always using the last one found, but that ordering was not consistent. As a result, things could change run-to-run.

I have this fixed locally for the next build such that it only includes files from under source roots. Is there any chance that might be what's happening to you? If so, you can move the additional copies of these files out of the way temporarily to see if you get a more consistent behavior, then you'll have the actual fix in this week's new build.

If that's definitely not what's happening to you, I'll need to find a network share for you to drop those logs as you're correct, I won't be able to receive an attachment that large.

[Alan Birchenough]

I will investigate the possibility that local files are causing a problem.

[Alan Birchenough]

Turns out that, as far as the second project is concerned, I have no local .object files whatsoever in the project directory or any of its subdirectories, and the only .cls files are in src/classes. I will go look at the first project next.

I look forward to receiving your suggestion for a network share where I can drop the debug logs. Thanks.

[Scott Wells]

Send an email to [e-mail] and I'll share a Google Drive network folder with your email address into which you can drop the logs.

[Alan Birchenough]

When looking at the logs for the second org, keep in mind that the observed anomalies are that the Account, Order and, I think, Contact objects are missing from the OST, along with examples such as pse__Project_Task__c but not, for some reason, pse__Project_Task_Assignment__c, which is a detail record to pse__Project_Task__c as master. There are other oddities where custom unmanaged objects are not picked up, e.g., SKU_Effort_Map__c and SKU_Effort_Map_Entry__c.

[Alan Birchenough]

How is it going? Can I try to grub up some more information that might be helpful? If you haven't been able to get much of value out of those logs, I might renew my search for an org you can access that exhibits the problem.

[Scott Wells]

Alan, last weekend I went dark on IC work (and all work) as it was my birthday and my twins' birthday. I will take your logs this weekend and see if I can get to the bottom of this. Sorry for the delay!

[Alan Birchenough]

Definitely not nagging, just trying to keep track. I hope you had a great birthday weekend. :-)

[Scott Wells]

Oh, I didn't take it as nagging at all! Just letting you know since I hadn't provided a status update in a bit. And it was a great weekend. Thanks!

[Alan Birchenough]

Using 2.0.3.5, I have updated OSTs in a number of orgs and at least the specific screenshot scenarios logged against this issue earlier are now resolved. I believe this issue can be closed.

[Alan Birchenough]

Thanks for all the work on it, BTW.

[Scott Wells]

Hah! Well, that's good news and bad news, I guess. The good news is that it seems to be fixed. The bad news is that I have no idea why it was happening to you. Don't hesitate to reopen this if the problem recurs. Thanks for your patience and help troubleshooting it!

[Alan Birchenough]

Yes, I was aware of the dual nature, but, LOL, I'll take it. :-)

[Alan Birchenough]

Unfortunately this bug returned - only what we should have expected I guess, since we had no explanation for why it appeared to go away.

Please advise as to next steps, and ETA for resolution.

Thanks.

[Scott Wells]

As odd as it may sound, I'm a bit glad that it's back. This felt like unfinished business. I'm going to need to reacquaint myself with where we were on troubleshooting as I evicted it from my mental cache when we resolved it. Let me re-review the logs you sent and see what that tells me, then I'll let you know what next steps look like. More info to come soon...

[Scott Wells]

Okay, so thanks to your log I have somewhat of a lead on this...at least what's happening if not why it's happening. Would you mind running the following SOQL query and providing the results:

SELECT NamespacePrefix, Name
FROM ApexClass
ORDER BY NamespacePrefix, Name

If you'd prefer not to share them here, feel free to do so via that same Google Drive folder I shared with you. I want to compare the results that IC is getting for that query with what you get since earlier in this thread you showed that a simpler query was in fact properly returning the classes that are missing from the OST. Thanks!

[Alan Birchenough]

I am stuck in an all-week on-site this week, but hopefully can send you something one evening "real soon now". Thanks.

Please keep in mind that what I am seeing does not just concern missing classes. More broadly, what I am seeing is that sometimes classes are missing, sometimes custom objects, sometimes standard objects. For example, in the latest incarnation, Contact is missing (a standard object), pse__Proj__c is missing (a managed custom object), and Project_Contact__c is missing (an unmanaged custom object), also Project_Contact_Sync__mdt (an unmanaged custom metadata type) is missing. Almost all my Apex is redlined. I have had to disable the relevant inspections so I can continue to work.

This problem appears to be random. At the weekend I had all my objects and classes, and no inspection redlines in my Apex code. Then I added the new custom metadata type and decided to regen the OST to pick up the new type. Only to find I had lost Project_Contact__c, giving me more redlines than ever. So I tried again, and this time I was additionally missing Contact and pse__Proj__c. Every time I regenerated it seemed to get worse. (Of course none of this circumstantial detail might be significant. Just painting a picture here.)

Anyway, given that both classes and objects are missing, including custom metadata and probably custom settings, I don't think you will be able to diagnose the problem by just investigating missing classes. Nevertheless I am happy to send you the query output that you requested. This is one part of the jigsaw after all.

[Alan Birchenough]

FYI, I am also looking for a sandbox to which you can log in, and hopefully experiment with reproducing the problem.

[Scott Wells]

Yeah, the random nature is similar to something I saw previously when retrieving SOQL query results via the partner API. Basically if the maximum page size wasn't set properly, it would return a random subset of the requested data instead of the entire data set...no errors or anything. I've extracted the full names of all custom Apex classes that are being returned by the query against your org used to build the OST. The fact that classes that we KNOW to be in your org aren't included--and the fact that the omissions are seemingly random--leads me to believe it's the same (or at least a VERY similar) issue.

No question that access to a sandbox org that evidences this issue would be invaluable, but I think that with what I know now, we can also get there via debug logs as well.

[Alan Birchenough]

That makes sense. One thing that puzzles me; am I the only user afflicted by this? That would be weird...

[Scott Wells]

Short version: Yes.

Long version: Somewhere between late February and mid-May I had at least one user complain of the issue I described above which is similar enough to what you're seeing that I believe it must be some variant of it. Basically when I was querying custom Apex class details from the org, I was providing a variable-sized batch of Apex class IDs based on the total number being queried. The batch size could be as low as 50 and as high as 500. It turns out that SOQL queries that filter by Id fields via a WHERE...IN clause can only handle a maximum of 200 values. If you provide more than that the query randomly chooses 200 out of what you provide and returns a result set for those 200. Now I use a maximum batch size of 200 when querying to ensure that doesn't happen.

What I'm seeing in the log you provided is that the query I run just before the one described above, used to get the header-level information for custom Apex classes, is behaving similarly. However, the big difference is that the query above is constrained by a WHERE...IN clause based on a batch of no more than 200 Apex class IDs, but the query that's misbehaving for you is unconstrained. It's just running SELECT Id, Name, NamespacePrefix FROM ApexClass.

A quick Google seems to reveal that the maximum result set that will be returned is 2K rows, so now I'm wondering if perhaps that's what's happening...you have >2K custom Apex classes in this org, and this query is getting back 2K randomly selected Apex classes. That may be the next best thing to try...partitioning this query as follows (pseudocode):

SELECT Id, Name, NamespacePrefix
FROM ApexClass
ORDER BY NamespacePrefix, Name
LIMIT 2000
OFFSET :(2000 * pageIndex)

where pageIndex starts at 0, and we keep incrementing it and repeating as long we get back a non-empty result set.

Let me put that together in a test build tonight and post it here for you. I have a sneaking suspicion that if that's not it exactly, it must be VERY close to what's going on. I'll let you know when that build is available and how to install/use it.

[Alan Birchenough]

Thanks. Definitely worth a try.

[Scott Wells]

Okay, here's the build with that change. Download it and install it using Settings/Preferences>Plugins>Install plugin from disk. After the IDE restarts, change Help>Debug Log Settings to:

#com.illuminatedcloud.symtab.OfflineSymbolTable

We don't need trace logging for this, just very tactical debug logging around these changes I made. If you want, make a backup of your current OST that you can use for comparison with the new one. Then regenerate your OST and attach or email the latest idea.log. We'll be looking for entries like:

Processing ### Apex class headers.

If the first one has 2000 for ### and there are subsequent ones, that's a pretty good indication that the issue was result set truncation. Assuming/hoping that we see messages like that, then the next step will be to verify that your OST contains all the custom classes you would expect. If you did back up your original OST, you can use a tool like Beyond Compare to compare the before/after archives showing only diffs. You can also dump both into my Google Drive share and I can do the diff for you.

If things do in fact look good, let's see if the behavior is repeatable. Again, make a backup of your OST archive and generate again. Those two should be identical aside from timestamps in the comment headers and the metadata files under META-INF. You might even do that a few times just to verify that it's stable across multiple generations. Note that there might be some variance in the Salesforce system types due to bugs I've reported to them. For this exercise we're focused on the custom Apex classes that have been flaky for you.

Let me know what you find! And if I somehow borked this change, let me know that as well and I'll spin a new, fixed build ASAP.

[Alan Birchenough]

Thanks - much appreciated.

I am stuck on-site with a client until the weekend, so it's touch and go if I get to do this before then. I will keep the issue up to date as usual.

Thanks again.

[Alan Birchenough]

I got a chance to have a go at this. Unfortunately immediately after initiating OST generation, I got this error dialog: .

I will upload the logs shortly as well.

[Alan Birchenough]

Here's the log that went with that error.

[Scott Wells]

Okay, that's actually good. It confirms my suspicion. Here's the key entry from the log:

2018-09-06 10:28:21,950 [  54468]  DEBUG - loud.symtab.OfflineSymbolTable - Processing 2000 Apex class headers. 

Unfortunately it looks like I can't use LIMIT/OFFSET to page through a result set of more than 2K rows, but that's fine...there are other ways to do that. For the moment I'd recommend you uninstall that test build and install the official build so you're not left in that broken state. This evening I'll implement another way of chunking the data to avoid the 2K issue.

This is great, though! It explains the behavior you've been seeing...now I just need to work around this API limitation and we should be good to go. Thanks for confirming.

[Scott Wells]

Hey, can you do one more thing for me in that org? Can you run the following queries and share the results?

SELECT COUNT()
FROM ApexClass

and:

SELECT COUNT()
FROM ApexClass
WHERE Status = 'Active'

and finally:

SELECT NamespacePrefix, COUNT(Id) NumClassesInNamespace
FROM ApexClass
WHERE Status = 'Active'
GROUP BY NamespacePrefix

I actually just came up with what I think should be a decent way to page through the data avoiding the 2K limitation, but I do want to see some raw metrics for your org just so I understand how it compares to these limits.

[Alan Birchenough]

Will do.

[Alan Birchenough]

• First count is 4943
• They are all active, so the second count is 4943 also
• Here are the counts by namespace:

NamespacePrefix	NumClassesInNamespace
CoveoV2	81
appirio_core	35
ffc	9
ffct	590
fferpcore	981
ffirule	341
ffr	966
ob1	12
psa_m	51
pse	1354
xecm	377
	146

[Scott Wells]

Thanks, Alan. That completely confirms what's going on here, and it also explains why you're seeing this and (almost) no one else is. Most orgs I've seen (including the ones I work on actively) have less than 2K classes in them even including installed managed packages, and my guess is that the folks who do have that many installed either aren't hitting this issue because they're not using the full contents of the org's installed packages during dev or they're hitting it and not letting me know. But I'm willing to bet that the number of people who even might encounter this issue is VERY proportionally small.

I'm never going to be able to rely on batching by namespace or whether things are active or not. You could still end up with an installed managed package that itself includes >2K classes. I have another thought on how that needs to work and plan to implement it very shortly. I'll post a new build once that's done and hopefully that will take care of this for you once and for all.

More to come soon. As always, thanks so much for helping to troubleshoot this!

[Alan Birchenough]

You're welcome regarding help with troubleshooting. I am both interested and strongly vested in a positive outcome. :-)

From one thing you said, I wonder if I am not following best practices. Should I somehow be scoping down the amount of metadata I am looking at in IC, and might that alleviate these issues as well? (I get that there is an actual defect here that needs to be fixed, but I want to use the tool as it is supposed to be used.) Usually I am only interested in a few of the namespaces, although TBH pse, which is FinancialForce PSA, is usually one of them and it is particularly large.

One reason for asking is that I must admit I don't find the various operations for populating a module at all self-explanatory even though I have been through the Youtube videos, so I may be doing things all wrong. I used the Eclipse-based Force.com IDE before, and then Mavensmate, and they each do things a bit differently, again without much explanation. As I started using IC - which I love, BTW, no problem there - I wondered whether to offer to help with documenting these things that I don't fully understand, i.e., contributing more content to your website / wiki for the sake of other people like me. Unfortunately I would then have to ask you a bunch of dumb questions. But this is probably a conversation for another day. [I wanted to talk to you about it at THDX this year, but we didn't coincide.]

[Scott Wells]

Well, OST generation is not currently something you can scope down. I've considered allowing the user to be able to select which subsets of metadata to include, e.g., which SObjects, which namespaces, etc., but right now it attempts to create a high-fidelity local rendering of everything in the org that can be referenced from Apex...well, that's not 100% true, but it does try to render everything around system types, custom Apex types, and SObjects at a minimum. The upside of this is that anything you might need should be available. The downside is that if you only really use a small portion of what's actually in the org, you can pay a pretty hefty time penalty for OST generation because it's including things you don't care about. It's something I might revisit in the future, though right now it would be behind a few other major initiatives I have in my queue.

Regarding the second question, you're hitting on something I hear about with some frequency...the lack of formal documentation is an issue for even some veteran users, much less for folks who aren't already quite familiar with JetBrains IDEs, the idiosyncrasies of Salesforce development, or perhaps just how things work in my head that led to a particular approach to a problem! That's one of those near-term major initiatives I referenced above. I'm planning to enhance the product docs via a set of HOWTO-style exercises. The initial focus will be on the main stumbling blocks for people, in particular new project creation of all forms and metadata management including how/when to manage the subscription and how/when to regenerate the OST.

I sincerely appreciate the interest in helping with the documentation work. I've had others make similar offers and I would like to find some way to take advantage of such kindness. What I'd like to do is get through this initial phase of new docs to set a framework for it and then I may well take you all up on those offers if they're still available. I'll definitely let you know.

I'm working on an updated build that will page this information properly I should be posting it shortly barring unforeseen issues.

[Alan Birchenough]

Sounds great. Thanks.

[Alan Birchenough]

Oh, and actually when you look at the numbers, selective OST generation would not generally benefit me very much because I am already working in and around the largest managed packages.

On Thu, Sep 6, 2018 at 12:46 PM, Alan Birchenough < alan.birchenough@ic

[Scott Wells]

Okay, Alan. Let's try this one. Same thing as before in terms of how to install it, and same steps to follow. I'd like to see the resulting idea.log, and if it does blow up again, just send me that log and I'll tune accordingly. Unfortunately I don't have any orgs with >2K classes in them, but I did test it with a smaller page size to verify that it paged through what I have properly. Hopefully it doesn't hit another boundary issue in your org with so much more content, but if it does, I'll adjust accordingly.

Fingers crossed!

[Alan Birchenough]

That seemed to work! Here is the resulting log.

[Scott Wells]

Oh, funny...it looks like that build still had my artificial page size of 100 instead of 2000. Let me get you one more build with the full 2000. That will be more efficient in terms of the number of queries/batches required. I also want to make sure that the extreme boundary of 2000/batch doesn't cause issues. But yeah, I think we're getting somewhere now!

[Scott Wells]

Okay, one more time if you don't mind. This should query 2000 at a time instead of 100 at a time. There's one more optimization I can do to avoid a final query that is guaranteed to be empty if the previous batch had <2K rows. I'll do that before I release this, but first let's confirm it.

[Alan Birchenough]

That also appeared to work. Here is the log.

[Scott Wells]

Yeah, that looks better:

2018-09-06 14:24:44,586 [  76029]  DEBUG - loud.symtab.OfflineSymbolTable - Processing 2000 Apex class headers. 
2018-09-06 14:24:46,157 [  77600]  DEBUG - loud.symtab.OfflineSymbolTable - Processing 2000 Apex class headers. 
2018-09-06 14:24:46,430 [  77873]  DEBUG - loud.symtab.OfflineSymbolTable - Processing 34 Apex class headers. 
2018-09-06 14:24:46,521 [  77964]  DEBUG - loud.symtab.OfflineSymbolTable - Processing 0 Apex class headers. 

So that seems to have gotten you past the previous limit. Sounds like your OST looks like it has everything you'd expect now, too, right? Assuming so, I'll optimize out that last query and plan to include this in the next official build, but feel free to sit on this build until I do so.

[Alan Birchenough]

Yes, I think I am good now. Many thanks.

[Alan Birchenough]

Unfortunately it seems there is still a problem with some orgs in connection with SObject types. Here is a screenshot from a sister org to the one we worked on this week - a recent direct sandbox clone of it, to be precise.

In this case the inspection errors are due to the fact that the Order standard object has not been included in the OST:

I will send you the log, but since you are currently logging Apex class processing rather than SObject processing I understand that this may not be useful, and you may want us to go through the same cycle of investigation and fixes for the SObjects as for the Apex classes. Just let me know. Thanks.

[Alan Birchenough]

Here are the logs for the org where the Order standard object is not making it into the OST.

[Alan Birchenough]

Further information: I ran the "Unresolvable reference" inspection project-wide in this sister org, and it found 4848 errors. They all related to missing SObjects. These include standard object such as Asset, Order and User. There are also custom metadata types like MapQueueBH__mdt and managed and unmanaged regular custom objects such as BUID__c, Execution_Log__c, pse__Proj__c and ShiftTwo__c that are missing.

[Alan Birchenough]

Out of curiosity I went back and ran this same inspection in the org we worked on this week and there are still 164 unresolvable reference errors in there - which admittedly is way better than 4848 - but even includes some missing Apex classes, which I thought we had a definitive solution for. This screenshot shows one missing class from the "pse" namespace.

[Alan Birchenough]

I guess I should have been alerted to the possibility of some classes being omitted when your debug log messages stated you were processing two lots of 2000 followed by 34 for a total of 4034, whereas the total should have been 4943.

[Scott Wells]

Okay. Let's try the attached build which uses different information for batching through the class headers. Install it, regenerate your OST, and let's see if it yields the correct/expected number of Apex classes.

[Alan Birchenough]

The new total number of active classes in this org stands at 4941.

I attach my latest log. This is from the org we have been working on all week. FWIW, on this iteration of OST generation, I did not get either Account or Contact in my OST and there are 4581 unresolvable references altogether in the project.

[Scott Wells]

Okay. So I'm not sure what two classes are missing from the set of custom classes, but that's much, much closer. For now let's focus on those unresolvable references you're seeing now. You're correct that they're due to SObjects such as Account and Contact. It looks like there were connection resets from your Salesforce org while querying SObject metadata:

[com.ctc.wstx.exc.WstxLazyException] com.ctc.wstx.exc.WstxIOException: Connection reset
    at com.ctc.wstx.exc.WstxLazyException.throwLazily(WstxLazyException.java:45)
    at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:728)
    at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3678)
    at com.ctc.wstx.sr.BasicStreamReader.getTextCharacters(BasicStreamReader.java:881)
    ...
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.lambda$process$0(OfflineSymbolTable.java:2094)
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(ForceComApiClient.java:220)
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(ForceComApiClient.java:318)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.process(OfflineSymbolTable.java:2088)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.process(OfflineSymbolTable.java:2084)
    at com.illuminatedcloud.util.ParallelBatchProcessorExecutor$1.call(ParallelBatchProcessorExecutor.java:70)

I see a number of those in the provided logs. SObjects are loaded completely separate from custom classes, and none of the logic to load them has been modified as result of working this issue (or at all recently). That part has never given you trouble before. I wonder if something changed as a result of Salesforce's weekend maintenance. Do you mind trying this again tomorrow to see if you see the same results? If so, do you mind disabling Use Parallel Processing for this connection and regenerating it to see if that goes through fine? Note that doing so will likely cause it to run much longer, but there have been some server-side changes in the past that have caused parallel API invocation to fail...I'm hoping they haven't implemented another one for this API.

Oh, and it also looks like you still have trace logging enabled for OST generation, so feel free to remove that entry from Help>Debug Log Settings to get a more concise log for review.

[Alan Birchenough]

Thanks. I will take your requested next steps, but first I want to emphasize that this issue, 991, for me has always been associated BOTH with SObjects and with classes. It has always been a toss-up which side would crap out and give me an incomplete OST. It's true that my initial report was in connection with Apex classes, but if you look at more recent comments you will see numerous mentions of missing SObjects, which are, if anything, more troublesome to me in my daily work than missing classes. As to recency of changes on the SObject side, this issue has been open since May 2018, and I was experiencing problems for a good while before I raised it - that's on me of course - so any changes on the SObject side would not need to be recent. My suspicion, based on our work so far, is that similar types of connection and perhaps paging issues can affect both retrieval of classes and of SObjects, but you will know much more than me about how you have implemented those connections so I will back off and leave the real diagnosis to the professional here.

Finally, note that there is no .trace suffix on any debugging level directives, so I can't remove it. I could remove the whole #com.illuminatedcloud.symtab.OfflineSymbolTable directive. Are you asking that I do that? I guess it's the word "trace" that is causing me confusion, since I don't have any trace requests.

[Scott Wells]

Ah, thanks for clarifying. I've been so myopically focused on the issue with custom classes that I guess I either never picked realized or forgot that SObjects were also giving you issues. As I mentioned above, it's entirely possible (probable even) that this org is causing some of the governor limits on the APIs used by IC to generate the SObjects portion of the OST to break just as with custom classes. Hold tight on any further testing. I'll instrument the SObject generator portion so I can see what kinds of numbers are being attempted on the threads that are failing. It could just be that I need to tune those down a bit. Right now it's set for 100 objects and 50K fields/request if memory serves, all based on known limits. Perhaps by lowering those this will all finally just work for you.

I'll post a new build a bit later today. As always, thanks for your amazing patience as we work through this!

[Scott Wells]

Okay, let's try this build now. With this one, you only need normal debug logging enabled for OST gen and not trace logging, and you only need to regnerate the SObjects portion of the OST, not the whole thing.

Assuming it fails the same way, it will log diagnostic information about the thread(s) that failed, and it will also automatically lower the maximum number of fields allowed per-partition by 50% each time. It will also tell you via a dialog that it's doing both of these things.

Basically what I'd like for you to do is run it so that it fails, then run it again and again until either it finally works because the values have been dropped low enough to work with your org, or because the value is down to something that I know for a fact works in other orgs, e.g., <500. That should only take ~7 iterations, I think.

With that done, please let me know what happened for you and also attach the latest logs for review.

Sorry to pass the onus to you for this, but as far I know, this is only happening in orgs loaded up like yours, so without access to such an org I'll still need assistance isolating the values that work vs. those that don't.

[Alan Birchenough]

Here is an interesting finding. I disabled parallel processing for the connection, and got way better results. I got only 125 unresolvable references and they were actually quite interesting. As for the missing classes and objects from last time, they appeared to be present this time.

One whole set of unresolvable reference errors was due to the inability to resolve Apex label references of the form System.Label.<labelName>, or just Label.<labelName>. I can provide further detail, but my guess is that IC isn't currently pulling labels or trying to resolve against them.

The second interesting class of errors was due to the inability to resolve SOQL selected columns that were defined using aliases. For example, someone had coded an unnecessary table alias as in SELECT proj.x FROM pse__Proj__c proj and IC wasn't able to resolve the proj.x selected column as being a reference to a field on pse__Proj__c. (I was actually surprised that the Salesforce SOQL parser did not complain about the unnecessary alias. It rejects them on fields in non-aggregate queries but apparently doesn't mind table/view aliases.)

In the end, I guess we don't know if this differing OST generation behavior was due to changes made by Salesforce or because parallel processing was disabled, but I thought you would want to know about it anyway, since the feedback is mostly positive.

[Alan Birchenough]

Got it. In view of my comment of a few minutes ago, I think I should re-enable parallel processing in order to get it to fail. It seems to work OK with parallel off.

[Scott Wells]

Okay, that's good to know. About a year ago when Salesforce started pushing out the new Apex compiler, they also implemented a "service protection" that prevented IC from issuing parallel requests against an API that it was using to build the custom classes portion of the OST. I had to switch to another API to keep things moving along decently fast. It's possible they've done something similar with the describeSObjects() API, but it's also possible that it's always done that and your org is densely-populated enough that it crosses a certain threshold. If you don't mind, please install the most recent build two comments above and follow those directions to see if we can find that threshold. That would allow you to take advantage of parallel processing even against this larger org which will always help with performance when it's a viable option.

As for labels, IC doesn't currently emit those into the OST. It resolves those locally against *.label metadata. If you haven't already, retrieve those into your project and they should resolve properly.

As for SOQL aliases, there's one known bug but otherwise those should work for the most part. I'll take your example and see if I can reproduce it. If so, I'll log a separate bug for that and fix it via that bug.

The good news is that we've almost completely characterized the various symbol resolution issues you've been seeing. I'm hoping we can find a good threshold for the SObject aspect so that you can keep parallel processing enabled. Then I think the only remaining bug would be the SOQL alias assuming that retrieving your label metadata addresses that one.

Let me know what you find with the newest test build I've posted, and thanks yet again for the assistance.

[Scott Wells]

If you haven't already started with the latest build, hold off for a few mins. It just occurred to me that because more than one thread fails on each run, it's going to halve the max number of fields more than once per-OST gen. Let me address that so it halves at most once per-run.

[Scott Wells]

Here you go. Hopefully this gives us the info we're looking for.

[Scott Wells]

Hey, Alan...just wanted to follow up and see if you had been able to try out this build, either to determine a working field count threshold or to produce a new log for me to digest. No hurry on my side at all, by the way. Just going through things that I'd flagged for follow-up.

[Alan Birchenough]

Not yet - still on my list. The last two days have been doozies.

[Alan Birchenough]

I hope to get a chance to do this at the weekend if I can't get to it before.

[Scott Wells]

Sure, that's fine. Just let me know once you've done it and hopefully it'll contain the final piece of the puzzle for us.

[Alan Birchenough]

Look forward to that. In the mean time I have a stupid question in connection with the labels. I have added both "CustomLabels" and "CustomLabel" to my subscription, and performed a "retrieve". All I get is this giant CustomLabels.labels file with all the labels inside. How do I persuade the API to give me the separate little label files against which the Apex editor will resolve? Thanks.

[Scott Wells]

Alan, you can't have it download distinct label files. You can only specify distinct labels that you want downloaded as part of the larger CustomLabels.labels file by using the more fine-grained CustomLabel metadata type instead of the broader CustomLabels metadata type. Unless there's a reason to do otherwise, I would generally recommend that you just download the entire message catalog as part of your project so that all labels are available for code completion.

[Alan Birchenough]

I guess that's what is confusing me then. I have downloaded all labels, but still the Apex editor is not resolving against them.

For example, I am seeing this error in connection with Label.System_Read_Only, but the CustomLabels.labels file does contain a label of that name:

[Alan Birchenough]

BTW, you gotta love the code here - NOT. Why couldn't they just write:

assetContactWrapper.isReadOnly = profileName.equalsIgnoreCase(Label.System_Read_Only);

(Just saying this so you know I didn't write it. <g>)

[Scott Wells]

Alan, can you show me the corresponding entry in your CustomLabels.labels file? Also, can you close/reopen the project in case something is caching too aggressively. I have a vague memory of that perhaps being the case at one point, and perhaps it still is.

[Scott Wells]

Fix for the custom Apex classes aspect delivered in 2.0.3.7. Still working with Alan to determine how to address the custom SObject fields aspect without having to disable parallel execution and whatever is going on with custom labels.

[Alan Birchenough]

Yes - my bad. The temporary deluge from the day job firehose continues. I am hoping it will diminish during this week.

[Scott Wells]

Still not an issue at all. Just wanted to let you know that the official build now contains all the fixes to-date, though it doesn't include the additional debug instrumentation around the SObject fields. I would recommend you stay on the build attached to this issue until you do get a chance to provide the resulting log since it should give us more data about the max custom fields threshold against your org.

[Alan Birchenough]

Will do.

[Alan Birchenough]

I tried the build with OST debugging (without tracing) and regenerating only the SObjects, but got this error dialog after the first cycle. Please advise. Thanks.

[Scott Wells]

Actually, Alan, I just had a potential epiphany about what's going on. Salesforce says you can query up to 100 SObjects with up to 50K fields in a single describeSObjects() call. When you run OST generation with parallel processing disabled, it completes successfully. When you run with it enabled, it fails. In this build, when running in parallel I divide the 50K limit by the number of allowed concurrent threads so that no more than 50K would ever be queried at the same time. I'm hoping that takes care of it as efficiently as possible.

Please try to generate your OST with this build. No need to repeat if it fails. Just send me the resulting log.

[Alan Birchenough]

I am trying to generate an OST, but generation is hanging during "Loading SObject types". It has been sitting at no progress on the bar for a full 15 minutes. Normally the entire operation will complete in less time than this.

[Scott Wells]

Sigh...I didn't expect it to just lock up. Can you email your idea.log, then restart the process entirely and try one more time while I digest the log. Sorry...I do feel like we're right on the cusp of having it all good for you, though.

[Alan Birchenough]

Here is the log from the operation that stalled. I will restart and retry.

[Scott Wells]

Okay, give me a sec. I may know what's going on. I'll post a new build very shortly if it's what I think it is.

[Scott Wells]

Okay, Alan...here's another build that corrects and issue in how the SObject fields are partitioned. It's a real issue and I'm glad your use case helped expose it! I've hopefully implemented a fix, and I've also instrumented it heavily with debug logging so I can see what's going on if it turns out that it's not quite right. Just regenerate the SObjects portion of your OST and, whether it works or not, please send me the latest idea.log file(s). Thanks (yet) again!

[Alan Birchenough]

This appeared to work, although it took a while, pausing for several minutes at a couple of different stages in the process. Also, unfortunately I could not restrict this operation to SObjects only. Since our previous iterations had trashed the OST on this project, it insisted on rebuilding the whole thing. The log is attached. Thanks.

[Alan Birchenough]

By way of further feedback, I'd like to confirm that only a very small number of unresolvable references were found; 149 in all, rather than the thousands we were seeing before, suggesting that you are definitely on to something with your current line of attack on resolving this issue. I attach the XML for the exported inspection report.

These errors fall into only a few groups:

• The label-related errors have gone away, thankfully, so we no longer need concern ourselves about those;

• There are errors related to the inability to resolve Flow.Interview.Contact_Search_and_Create and Flow.Interview.Unassociate_Associate_Order_to_Project_Processing. I have looked in the org to confirm that the first flow does exist, although I am having trouble running down the second. If you haven't had trouble with flow resolutions before, maybe let me know and I'll investigate further at my end.

• There are a large number of errors showing the inability to resolve column names defined in terms of object aliases as previously reported. Recall that the offending SOQL looks like 'SELECT x.a, x.b from O x'. It seems that in these cases x is not being correctly identified as being an SObject of type O, and each of the references x.a, x.b, is flagged as unresolvable.

• A whole service class seems to be missing from the OST. This one worries me because it's "where we came in" as they used to say in the movies when reaching the point in the plot for the second time where we first entered the theater. OST generation is failing to resolve pse.CreateProjectFromTemplateService.CreateProjectFromTemplateRequest and pse.CreateProjectFromTemplateService.CreateProjectResponse. I can confirm that both of these classes do exist, but the entire pse.CreateProjectFromTemplateService is missing from the OST. Since the flagged classes are managed service classes, the declarations are readable but of course the bodies are obfuscated.

That seems to exhaust the classification of these errors, which is good because there are not too many different cases to chase down. Since these are probably unrelated to the original issue - with the possible exception of the missing service class - I guess we may want to split off a couple of new issues for the Flow.Interview and the SOQL alias resolution problems.

In the mean time, if there is any further help I can give to resolve the current issue, please let me know.

[Scott Wells]

Thanks, Alan. Let me review the logs and in particular the timings and I'll see if there's anything else I can do. It may just be a limitation of the Salesforce API used to get this information when it's in this volume, though. If you get a chance, would you disable parallel processing and regenerate the OST for comparison? SObjects-only is fine as I just need that portion of the log for A/B comparison.

As for the remaining issues, the Flow issue is likely a bug/shortcoming of IC right now. At present it knows about compile-time constants for Page.<pageName>, Label.<labelName>, and all the variations of SObject/field compile-time constants. It does not have explicit support for Flow. in Apex, so that would be a separate bug. The SOQL aliases also sound like a potential bug. As I mentioned previously, except for one known corner-case, I thought I had implemented proper support for them, but it sounds like you may have exposed something missing. If so, again I think that should be splintered off into its own bug report.

I'll scan the log for mentions of the missing Apex service class and see if I can figure out why it's not present as well. Depending on what I see on this one, it likely does continue to belong in this bug since it's a manifestation of the original problem.

I'll let you know what I find as I review the log. Thanks!

[Scott Wells]

Alan, I figured out what's going on with the custom Apex service classes. Somehow when I merged changes from a branch where I was working on this for you into the main branch, I lost the last implementation of it that fixed how it pages. In just a few I'll post a build with that re-fixed. Glad you caught that!

[Scott Wells]

Alan, here's a breakdown of where the time is going in your OST generation:

Use Tooling API to get System Apex types and transform them into stub Apex classes for OST

2018-09-23 17:23:57,967 [  64115]   INFO - ct.OfflineSymbolTableGenerator - Starting activity Loading system classes with 1 steps.
2018-09-23 17:23:59,593 [  65741]   INFO - ct.OfflineSymbolTableGenerator - Completed activity Loading system classes in 1626 ms.
2018-09-23 17:23:59,600 [  65748]   INFO - ct.OfflineSymbolTableGenerator - Starting activity Loading system classes with 1 steps.
2018-09-23 17:23:59,639 [  65787]   INFO - ct.OfflineSymbolTableGenerator - Completed activity Loading system classes in 39 ms.
2018-09-23 17:23:59,640 [  65788]   INFO - ct.OfflineSymbolTableGenerator - Starting activity Loading system classes with 1103 steps.
2018-09-23 17:23:59,706 [  65854]   INFO - ct.OfflineSymbolTableGenerator - Completed activity Loading system classes in 66 ms.

Use Partner API to get custom Apex types and transform them into stub Apex classes for OST

2018-09-23 17:23:59,717 [  65865]   INFO - ct.OfflineSymbolTableGenerator - Starting activity Loading custom classes with 1 steps.
2018-09-23 17:24:34,076 [ 100224]   INFO - ct.OfflineSymbolTableGenerator - Completed activity Loading custom classes in 34359 ms.

Use Metadata API to enumerate header-level SObject type information

2018-09-23 17:24:34,077 [ 100225]   INFO - ct.OfflineSymbolTableGenerator - Starting activity Loading SObject types with 1 steps.
2018-09-23 17:24:57,769 [ 123917]   INFO - ct.OfflineSymbolTableGenerator - Completed activity Loading SObject types in 23692 ms.

Use Metadata API to get SObject type/field details

2018-09-23 17:25:12,254 [ 138402]   INFO - ct.OfflineSymbolTableGenerator - Starting activity Loading SObject types with 13 steps.
2018-09-23 17:29:32,951 [ 399099]   INFO - ct.OfflineSymbolTableGenerator - Completed activity Loading SObject types in 260697 ms.

Use Metadata API to get SObject type/field description metadata for integrated API documentation

2018-09-23 17:29:32,958 [ 399106]   INFO - ct.OfflineSymbolTableGenerator - Starting activity Loading SObject docs with 722 steps.
2018-09-23 17:39:45,008 [1011156]   INFO - ct.OfflineSymbolTableGenerator - Completed activity Loading SObject docs in 612050 ms.

Transform SObject type/field metadata into stub Apex classes for OST

2018-09-23 17:39:45,010 [1011158]   INFO - ct.OfflineSymbolTableGenerator - Starting activity Loading SObject types with 1017 steps.
2018-09-23 17:39:45,174 [1011322]   INFO - ct.OfflineSymbolTableGenerator - Completed activity Loading SObject types in 164 ms.

On 9/9 you mentioned that by disabling parallel OST generation and got much better results, presumably in line with these in terms of SObjects I would assume. I didn't see a log from that run for a timing comparison, though, so I'd still like to see one.

What I do see, though, is that a HUGE amount of the overall time is going into querying the SObject/field documentation...10 of the total 15 mins of your OST generation. If you'd like I could add an option to allow you to disable that. It would mean that SObjects in your OST would no longer have comment headers containing the object/field description metadata, but it looks like it would potentially cut your OST generation time by 2/3.

I'll get a build going now with the custom Apex class fix restored.

[Scott Wells]

Okay, here's the new build. It restores the correct logic for custom Apex classes, so hopefully you won't see any issues there anymore, and it also disables the SObject/field documentation step of OST generation. I just want to see how that affects the overall OST generation times, ideally two runs, one with parallel processing enabled and one with it disabled. I'll be happy to make that a configurable option in an upcoming build, but I want to see if what I'm seeing in your logs is accurate with regard to the proportion of time that step is consuming. And of course I'd like to see A/B time comparisons of the SObject portion of OST generation with parallel on/off.

[Alan Birchenough]

These are the logs for SObject generation alone with parallel processing enabled. This seemed to complete in reasonable time, although I wasn't really watching it that closely.

[Alan Birchenough]

BTW I am ambivalent about having an option to disable querying the SObject/field documentation. If it's easy to provide I would go ahead, but if it is anything other than easy I am not sure it's worth devoting much effort. I feel like most of the time I am going to want to keep the descriptions even if it takes longer to build/rebuild the OST. But I'm not sure. (Typical user - they don't know what they want, do they?)

[Alan Birchenough]

Here are the logs for SObject OST regeneration with parallel processing disabled.

[Scott Wells]

Thanks again, Alan. Here are the timings from the parallel run:

Completed activity Loading installed package namespaces in 2393 ms. 
Starting activity Initializing local symbols with 1 steps. 
Completed activity Initializing local symbols in 6 ms. 
Starting activity Loading SObject types with 1 steps. 
Completed activity Loading SObject types in 22333 ms. 
Starting activity Loading SObject types with 13 steps. 
Completed activity Loading SObject types in 604997 ms. 
Starting activity Loading SObject docs with 483 steps. 
Completed activity Loading SObject docs in 13664 ms. 
Starting activity Loading SObject types with 618 steps. 
Completed activity Loading SObject types in 94 ms. 
Starting activity Generating offline symbol table source files with 622 steps. 
Completed activity Generating offline symbol table source files in 359 ms. 
Offline symbol table generated for Dell FFDev Sandbox in 11 m 4 s 174 ms. 

and here are the timings from the run without parallel processing enabled:

Completed activity Loading installed package namespaces in 320 ms. 
Starting activity Initializing local symbols with 1 steps. 
Completed activity Initializing local symbols in 7 ms. 
Starting activity Loading SObject types with 1 steps. 
Completed activity Loading SObject types in 17550 ms. 
Starting activity Loading SObject types with 12 steps. 
Completed activity Loading SObject types in 49654 ms. 
Starting activity Loading SObject docs with 722 steps. 
Completed activity Loading SObject docs in 366036 ms. 
Starting activity Loading SObject types with 1118 steps. 
Completed activity Loading SObject types in 449 ms. 
Starting activity Generating offline symbol table source files with 1122 steps. 
Completed activity Generating offline symbol table source files in 1662 ms. 
Offline symbol table generated for Dell FFDev Sandbox in 7 m 17 s 894 ms. 

Ironically, the latter is quite a bit faster, presumably because while it loads SObject metadata in serial, it's able to do so in larger batches (up to 50K vs. up to 2K) which makes quite a difference in this org. I think perhaps the best way to proceed on this is to make OST generation selectively disable parallel processing for the SObject portion when there's a very large number of custom fields but still use it for everything else.

As for an option to disable SObject field/documentation, I'm also finding that to be quite a bit faster when not run in parallel with large amounts of metadata, so I'll likely apply the same strategy to that step and not add an option until it's something someone explicitly wants to see.

I'm about to jump on a flight to Dreamforce but will try to get a build with these characteristics posted for you today to try out. Let's get a final build with all of these fixes together and then we can point that build at your other org(s) and see what there is to see.

[Alan Birchenough]

Thanks. BTW, I won't be at DF18 this year, but I hope to catch you at THDX19. Have a good trip and conference.

Regards, -- A.

[Alan Birchenough]

Hope you had a great DF18. I noticed your positive Tweet about the platform roadmap.

Whenever you have a chance to post an updated build for me to try in connection with this issue, I should be able to turn it around pretty quickly. No hurry though.

Thanks, -- A.

[Scott Wells]

Sounds good, Alan. Let me see if I can put one together tonight for you that behaves as described just above and we'll see how well that one works for you.

[Scott Wells]

Okay, Alan. Here's a new test build with the Apex custom class stuff properly fixed and parallel processing disabled for the SObjects portion of OST generation only. So install this, make sure that parallel processing is enabled for your connection, and perform a full generation of your OST. Then please provide the resulting log files for review. What I'm hoping to see if the better times from parallel processing on everything but the SObjects portion and the better time from serial processing on SObjects. We'll see...

If this does pan out as expected/hoped, what I'll need to do is add an explicit configuration option under Enable parallel processing to disable that for the SObject portion of OST generation (with parallel processing enabled by default of course). I'd prefer to make it explicit rather than trying to use an object/field count as a heuristic. Let's verify that it does what I hope it does first, though.

Thanks!

[Alan Birchenough]

Thanks. I'll give it a try.

[Alan Birchenough]

OK. Here's the result from another large org, comparable in size to the other one I think. In any case I finished with that other project so no longer have access to that org. So I hope this log contains the information you need.

[Alan Birchenough]

Hmm. That worked fine the first time, but now every time I try to rebuild the OST I get "Read timed out". It may be a shaky internet connection, so I'll persevere.

Parenthetically, I never understand why some service providers seem to be able to provide services that are robust in the fact of WiFi blips or broadband hiccoughs, whereas others just throw up their hands as soon as anything slightly odd happens during a longer running operation. Salesforce APIs seem particularly fragile in this regard. Probably more of a beer conversation, so I'll leave it there...

[Alan Birchenough]

Tried again twice this morning; got "connection reset" both times after quite a long time running. I attach the log FWIW. If this is just a connection problem I have no idea what to do about it. Everything else on this WiFi is rock solid, except every so often Zoom will complain about instability. I wonder if your changes have somehow made OST generation more vulnerable to connection glitches. Anyway for now, I am going to have to try to get by with no OST at all since I just can't generate one.

[Alan Birchenough]

Oh, BTW, I had disabled parallel processing on this org connection in case it would help, in case the log seems confusing. I will try re-enabling it next time. I may send you another log.

[Alan Birchenough]

I re-enabled parallel and finally got an OST to generate just now. I am not sure if any of the recent posts about connection issues will prove significant, so I suggest we continue along the previously-agreed track, and meanwhile I will keep an eye on these connection issues and see if they appear to be repeatable or if I was just having a bad day yesterday and this morning.

[Scott Wells]

Thanks, Alan. Unfortunately I'm not sure what I can do about a volatile connection to Salesforce other than potentially build in retries on certain types of failures, in particular timeouts. I'll scan the latest logs this weekend and let you know what I find. I may be a bit delayed in getting feedback to you as I'm wrapping up the Winter '19 update to both IC1 and IC2 since it will be available in all orgs after this weekend. I'll let you know what I see once I've had a chance to review the logs thoroughly, though.

[Alan Birchenough]

Thanks. I would prioritize the steps we had already agreed before the connection issues came up. Like I said in my last comment, I am not sure how much significance to attach to them. I did go through 24 hours when I could not get an OST at all because of connection issues, but everything else, including everything else Salesforce-adjacent, including deploying Apex from IC to the org, was rock solid. But some issues are best not prioritized too highly lest they steal too much effort from others that would yield a much better return on the investment of effort.

Regarding the timing of when you can get new builds to me in connection with the original OST generation issues, this has been a long game from the start, and I am fine with continuing to take a measured (and sustainable) pace towards resolution.

[Alan Birchenough]

Just to let you know: I am continuing to have read timeout issues with OST generation (so apparently this is not some ephemeral problem) but no other noticeable connection issues on this machine / WiFi setup. I would estimate that one in ten attempts at OST generation succeeds at this point, and when the nine out of ten fail it usually happens many minutes into the generation process.

When you are ready to take this issue up again, do let me know if you want me to get you another log or two, otherwise I look forward to an update on this issue in due course.

The good news is that I think you are really close to giving me a definitive fix for my original issue. Yay!

[Scott Wells]

Alan, let's get some logs for the read timeout just so I can see which API calls are are failing and how long they're running before they time out.

[Alan Birchenough]

You bet. Of course the first one I tried right after updating this issue ran to completion, but I will be sure to send you the logs for the next few that fail. Thanks.

[Alan Birchenough]

Here's one that failed. Hopefully you can see what went on; I was doing a few things in different projects in the IDE while waiting. With this new org I am trying to OST, so far I have been unable. I have run it a couple of times, this time with parallel enabled on the connection.

[Scott Wells]

Alan, I can see what's going on. IC has a maximum timeout of 5 minutes for Salesforce API calls. I can see a single API call to your org to query SObject types hitting that timeout:

2018-10-24 17:21:49,068 [461642459]   INFO - ct.OfflineSymbolTableGenerator - Starting activity Loading SObject types with 15 steps. 
...
2018-10-24 17:22:35,170 [461688561]   WARN - ntellij.ui.tree.AsyncTreeModel - ignore duplicated child at 27: LineWrapper 
2018-10-24 17:27:18,547 [461971938]   WARN - xf.phase.PhaseInterceptorChain - Interceptor for {urn:partner.soap.sforce.com}PartnerService#{urn:partner.soap.sforce.com}describeSObjects has thrown exception, unwinding now 
org.apache.cxf.interceptor.Fault: Unmarshalling Error: Read timed out 
    at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:906)
    ...
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.lambda$process$0(OfflineSymbolTable.java:2135)
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(ForceComApiClient.java:220)
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(ForceComApiClient.java:318)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.process(OfflineSymbolTable.java:2129)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.process(OfflineSymbolTable.java:2125)
    at com.illuminatedcloud.util.SerialBatchProcessorExecutor.execute(SerialBatchProcessorExecutor.java:48)
    at com.illuminatedcloud.symtab.OfflineSymbolTable.lambda$loadDescribeSObjectResults$13(OfflineSymbolTable.java:2124)
    at com.illuminatedcloud.symtab.OfflineSymbolTable.runActivity(OfflineSymbolTable.java:630)
    at com.illuminatedcloud.symtab.OfflineSymbolTable.runActivity(OfflineSymbolTable.java:657)
    at com.illuminatedcloud.symtab.OfflineSymbolTable.loadDescribeSObjectResults(OfflineSymbolTable.java:2118)
    at com.illuminatedcloud.symtab.OfflineSymbolTable.loadSObjectClasses(OfflineSymbolTable.java:1213)
    at com.illuminatedcloud.symtab.OfflineSymbolTable.generateOfflineSymbolTable(OfflineSymbolTable.java:786)
    at com.illuminatedcloud.intellij.settings.project.OfflineSymbolTableGenerator$3.run(OfflineSymbolTableGenerator.java:206)

I can give you the ability to configure that timeout to something greater if you'd like. Assuming that Salesforce itself doesn't end up closing the request at the server, it would allow you to complete the OST generation, though it's going to be slow-running obviously since we know that at least this one step takes more than 5 minutes to completes (though we don't know how much more).

What we can do to get a feel for how this would work is get a you a build that hard-codes that timeout as, say, 15 minutes. That would allow us to understand the behavior of this org better so I can try to determine the best way to address this for you (and anyone else working against an org like this).

[Alan Birchenough]

No response after five minutes seems really excessive. The request must have floated off into the ether somewhere and I doubt it would ever complete. The thing is, I get this with nearly every org I work with, but I don't get it - at least not frequently - for any other scenarios within IC, or indeed, for other services on this machine. (In other words, I don't get it for metadata retrieves or refreshes. Occasionally I get a hiccup during test execution whereby an individual test is ignored because the status never comes back, and I also sometimes get a failed deploy, which however, I can easily recover from by hitting "Save" again. These are different from OST generation, which fails pretty much every time.)

If you would like me to proceed with a trial of the 15-minute timeout build I would be happy to do so.

[Scott Wells]

I've seen things like this in REALLY big orgs before and have chatted with Salesforce about it. The request is still being processed, but on their end for some of this setup metadata, it's just not well indexed on the back end for this type of access. As their own tooling needs kick over the same issues that third-party tooling solutions have hit, they've address them. Otherwise they tend to get lower levels of attention.

I'll get you a build with a much higher timeout this weekend and we can see if it completes and, if so, how long these operations take. It could be that it's 5m10s and it could be that it's 12m. And it could also be that the request is in fact abandoned at the server but not closed. It would be good to know either way.

[Alan Birchenough]

Agree. I look forward to finding out. Thanks.

[Scott Wells]

Alan, here's a new build with a 15 minute timeout instead of a 5 minute timeout. Let's give it a whirl and see if it changes the behavior. Either way please provide the resulting logs so I can see what's happening. Thanks!

[Alan Birchenough]

Will do. Thanks.

[Alan Birchenough]

This did eventually complete, but it took quite a while. Upon complete I appear to be missing a large number of SObjects in my OST so it is of no real use. (The "unresolvable reference" inspection has 1727 errors.)

[Scott Wells]

Thanks, Alan. I can still see a number of timeouts in the log from loading SObject types which is why your OST is missing information. These are all just calls to MetadataApi.describeSObjects() which are timing out, so there's not really anything I can do to tune a query or anything here. I did notice that this is using the parallel executor. Perhaps you should try again without parallel execution disabled just so we can see if things at least succeed that way. Perhaps Salesforce's service API protection layer is causing these parallel requests to queue up at the server which is causing them to time out? At least running things in serial would give an indication of whether that's the case or not.

[Alan Birchenough]

Here is a second attempt, which fared similarly, but managed to garner a few more SObjects, although not all. It seems to be random. This time the "Unresolvable reference" inspection found 1103 errors.

[Scott Wells]

Yeah, as long as there are API call timeouts there will be gaps. Was this one run with parallel execution disabled? I'm not able to review the logs just now but will this evening.

[Alan Birchenough]

This one failed with a "read timed out" error.

[Alan Birchenough]

The first two, with the "gaps" were run with parallel processing enabled. The last one was run with it disabled. Sorry, I lost track of which way you wanted to see it. Please advise if you want more examples and remind me whether you want parallel processing enabled or disabled.

[Alan Birchenough]

Just saw your email of earlier in which you recommend running with parallel disabled. This still didn't work.

I can't believe this whole struggle we have been engaged in over so many months might just falter at the last minute because Salesforce doesn't honor their API calls! And you say there is nothing either of us can do about it?

[Scott Wells]

Alan, the only thing we can do is try to adjust the partition size when invoking their API. Right now it tries to minimize the number of partitions and maximize the size of each partition based on Salesforce's own limitations around the invoked APIs. This is the call that IC is making:

https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/sforce_api_calls_describesobjects.htm

As you can see, that call is limited to no more than 100 SObject types per-invocation. Here is how has IC partitioned the way it invokes that API in the most recent logs you've provided:

2018-11-06 08:48:04,236 [57404925]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 6793 custom fields.
2018-11-06 08:48:04,238 [57404927]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 2180 custom fields.
2018-11-06 08:48:04,240 [57404929]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 1280 custom fields.
2018-11-06 08:48:04,242 [57404931]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 843 custom fields.
2018-11-06 08:48:04,244 [57404933]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 589 custom fields.
2018-11-06 08:48:04,246 [57404935]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 443 custom fields.
2018-11-06 08:48:04,247 [57404936]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 289 custom fields.
2018-11-06 08:48:04,249 [57404938]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 123 custom fields.
2018-11-06 08:48:04,251 [57404940]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 0 custom fields.
2018-11-06 08:48:04,252 [57404941]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 0 custom fields.
2018-11-06 08:48:04,256 [57404945]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 0 custom fields.
2018-11-06 08:48:04,258 [57404947]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 0 custom fields.
2018-11-06 08:48:04,259 [57404948]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 0 custom fields.
2018-11-06 08:48:04,261 [57404950]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 0 custom fields.
2018-11-06 08:48:04,262 [57404951]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 53 SObject types with a total of 0 custom fields.

It appears in this particular log that only one of the requests timed out. Since this last run was executed in serial fashion, based on the log it appears that it was the first request that timed out. When one of these requests times out in the serial executor, it short-circuits the entire operation.

Assuming that Salesforce support isn't really going to be able to be helpful here, I can think of two options that we can pursue ourselves:

1. Allow you to specify your own timeout and raise it until you find this passes. There are two downsides to this approach. First, there could end up being some type of server-side timeout after which Salesforce closes the request. Second, if it does work, you could find that it takes hours to generate this OST given how long it takes for Salesforce to issue the response.
2. Change the way these partitions are created so that they're quite a bit smaller. You'll have more partitions in the end that have to be evaluated, but perhaps each one will finish in a more reasonable time. I'm not sure exactly what Salesforce is having to do on their end to service these requests, so I can't say whether that will scale better, but I don't see any downside to trying that.

Let me know if you'd like to try either/both of these and I can provide another build for you.

[Alan Birchenough]

Thanks for the summary - that's great.

Can we first try the smaller partitions?

Thanks again.

[Alan Birchenough]

BTW, I am resigned to OST generation not necessarily being quick. What I can't adapt to is if it is unreliable in any way. I have to be able to get to a complete OST eventually for every org I work in. As you are seeing, most of them have a TON of metadata in them.

On Tue, Nov 6, 2018 at 12:07 PM Alan Birchenough < alan.birchenough@ic

[Alan Birchenough]

Hi -

Any time you have a chance to do a built with smaller partitions I would be happy to give it a test run.

Later, -- A.

[Scott Wells]

Alan, I've attached another build that sets the maximum number of custom fields per-batch to 2K, but I've also made it configurable so that you can try to tune it a bit yourself without requiring a new build each time. To do so, go to Help>Edit Custom VM Options... and then add a line like the following:

-Doffline.symbol.table.max.fields=2000

where you can replace 2000 with any value that's at least 500 (which is the maximum number of custom fields allowed on an object). Note that you will have to restart the IDE each time you make this change since it's a system property. I can make it a configurable setting in the app itself once we have characterized it better.

Based on what we saw previously, ~6K was causing a timeout, so I'd recommend starting with the default of 2K and, if that works, wonderful; if not, try 1K and then 500.

I would also recommend trying this with parallel execution enabled for the most part unless you simply can't get it to work configured that way. If that's the case, try turning off parallel execution and see what value works best. Those two pieces of information--whether you can get it working with parallel execution or not and what batch size works best--will tell me what I need to make configurable for orgs like this. It could very well be that I need to allow you to specify parallel execution for everything BUT SObjects/fields based on what we've seen previously.

Okay, let's get some data points and see where this takes us...

[Alan Birchenough]

Perfect. I'll give it a go. Thanks.

[Alan Birchenough]

First data point: This is org R-FULL (for my reference), running with the parameter set to 2000, parallel enabled, full OST regeneration. Result: Appeared to get "stuck", and now there are 1966 unresolvable references in the project.

[Scott Wells]

The server is still closing the connection and/or timing out:

org.apache.cxf.interceptor.Fault: Unmarshalling Error: Connection reset 
    at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:906)
    ...
    at com.sun.proxy.$Proxy202.describeSObjects(Unknown Source)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.lambda$process$0(OfflineSymbolTable.java:2155)
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(ForceComApiClient.java:220)
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(ForceComApiClient.java:318)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.process(OfflineSymbolTable.java:2149)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.process(OfflineSymbolTable.java:2145)
    at com.illuminatedcloud.util.ParallelBatchProcessorExecutor$1.call(ParallelBatchProcessorExecutor.java:70)

and:

org.apache.cxf.interceptor.Fault: Unmarshalling Error: Read timed out 
    at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:906)
    ...
    at com.sun.proxy.$Proxy202.describeSObjects(Unknown Source)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.lambda$process$0(OfflineSymbolTable.java:2155)
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(ForceComApiClient.java:220)
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(ForceComApiClient.java:318)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.process(OfflineSymbolTable.java:2149)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.process(OfflineSymbolTable.java:2145)
    at com.illuminatedcloud.util.ParallelBatchProcessorExecutor$1.call(ParallelBatchProcessorExecutor.java:70)

Note that until we figure out the correct combination to make this run without errors, you're going to have an incomplete OST and unresolvable references based on which calls did not complete successfully.

I would recommend that you first try generating the SObjects portion of the OST with the same default of 2000 but with parallel execution disabled. My guess is that we'll see the "Connection reset" errors go away but the "Read timed out" will still occur. Assuming that's the case, change the batch size to 1000 and leave parallel execution disabled and try again. If that fails, try 500. If either of those succeeds, leave the batch size as-is and re-enable parallel execution.

So basically run tests in this order:

• 2000 + parallel - you just sent the log for this and it failed
• 2000 + serial
• 1000 + serial - if 2000 + serial worked, you can skip this one
• 1000 + parallel - only if 2000/1000 + serial worked so we can see if/when you can enable parallel
• 500 + serial - only if 1000 + serial failed
• 500 + parallel - only if 500 + serial worked so we can see if/when you can enable parallel

If it still fails at 500 + serial, the only other thing that comes to mind is to allow you to tell it to fetch the metadata for each object individually, but that won't work because you have ~1.2K objects and we'll start to run into the maximum number of API requests that are allowed per-24 hour period. At that point we may need to get Salesforce support involved to understand why these API requests are failing...

Feel free to attach logs for each executed test so I can review them and see how it's progressing.

[Alan Birchenough]

Second data point: Same org, parameter set to 1000. Operation "stuck" for a while, but completed more quickly. Still 1334 unresolvable references, all SObjects, most of them the same.

[Scott Wells]

Still getting both "Connection reset" and "Read timed out". If you haven't already, please turn off parallel execution and let's see what 2000, 1000, and 500 look like.

[Alan Birchenough]

I will follow the recommended sequence and let you know what happens.

[Alan Birchenough]

2000 serial failed with "Read timed out". Here is the log.

[Scott Wells]

Thanks, Alan. Let's just go straight to 500 + serial and see what happens. That's going to be the one that gives the best chance of working that's still able to download all fields for a single object (given that a single object can have up to 500 custom fields), so it would be good to know if that one works.

[Alan Birchenough]

Actually, I did get a successful SObjects refresh of the OST with 1000 serial last night. Here's the log (although I am not sure if a log of nothing failing is very revealing). The only remaining unresolvable reference error was one single one, whereby it now can't find the reset() method on ApexPages.StandardController. I will go ahead and try 500 serial on another org. If that works I might try going back to 1000 serial to see if that works in that org as well, just so we have some level of repeatability. Of course this is all subject to your further recommendations about what to try.

[Scott Wells]

That's encouraging, Alan! So given that, let's run the following additional tests to try to find the sweet spot for you in this org (these orgs?):

• 1000 + parallel
• 500 + parallel
• 500 + serial

I'd like to look at what works and what doesn't, but also even if everything works, what provides the best performance for you. I think in the end what I'll likely end up doing is adding two additional connection-level config options for the batch size and whether SObjects should be loaded in parallel when parallel execution is enabled. But yeah, let's see the results of those tests.

As for ApexPages.StandardController.reset() missing, that sounds like a bug on the Salesforce side since I derive system types from the Tooling API completions REST resource. It wouldn't be the first such omission, and I have a provision in IC for "filling the gaps". Do you mind logging another bug for that in particular? I'll take care of it in the next build.

[Alan Birchenough]

I also got a successful OST gen in a different (large) org with 500 + serial, but it took 50 minutes or so - not good. Still, it's good that the information is complete. (There were no errors on the unresolvable references inspection).

[Scott Wells]

Okay, let's try parallel with both values, 1000 and 500. If either works with parallel, that should speed it up. Otherwise it may just be what it is...basically to get the info required to generate a complete OST in this org AND to do so in a way that doesn't result in connection drops/timeouts in the Salesforce APIs, it will just take a long time. The only recourse that comes to mind to try to speed that up would be a support case with Salesforce to understand why those API calls are taking so long. But before we jump to that conclusion, let's see if we can find a value that succeeds with parallel processing enabled since that definitely makes a big difference.

[Alan Birchenough]

1000 parallel gave me "connection reset"

[Scott Wells]

Okay. Let's try 500. I did think of another config variable we could try...right now I allow up to 25 threads for parallel execution. I could make that configurable to try to avoid an all-or-nothing proposition. Maybe it would be fine with 5 or 10 threads when it's not okay with 15. Again, not knowing how their API service protection works, and not having seen this anywhere else, I'm kind of throwing darts here. I think it's worth a try, though. Let's get the data point for 500 + parallel and then I'll look at providing another build that allows you to tweak the concurrent thread count.

[Alan Birchenough]

1000 serial in the same org gave me a bearable duration and complete generation.

[Alan Birchenough]

I will try 500 parallel.

[Alan Birchenough]

500 parallel in org 2 (M-FULL) gave me "connection reset".

[Scott Wells]

Okay. I'll get you a build with configuration of the number of parallel threads in a bit along with a description of how to configure that value. Hopefully we can find a combination that allows you to generate this OST in considerably less than 40-ish minutes (which is what it took in the 1000 + serial log).

[Alan Birchenough]

I just tried three 1000 serials on the second org and I just get connection reset every time. These were the parameters that were working best in both orgs I have tried today. There just doesn't seem to be any pattern.

[Alan Birchenough]

Here is the log from the most recent failed 1000 - Serial. A reproducible "connection reset" seems very odd, especially when this particular combination of parameters was working well. It is discouraging, plus this org now has no useful OST so it is all but impossible to work in.

[Scott Wells]

Does 500/serial not work for you anymore either? I certainly understand your frustration. If we can't find a reliable combination here, it'll be time to log a support case with Salesforce. There's no reason that a call to describeSObjects() should time out or result in a server-aborted connection.

[Alan Birchenough]

I will try 500 serial again, but it is hella slow.

[Scott Wells]

Understood. Unfortunately we're starting to hit a wall here against these orgs. These API calls must complete successfully to populate a usable OST. This org is rejecting those API calls unless we dial them down to the lowest acceptable values (given that an object can have up to 500 custom fields) and disable opportunities for parallelization which is what really makes a huge difference in OST generation in terms of speed.

As mentioned previously, I can provide you a build this evening that lets you specify the number of allowed threads to see if there's a sweet spot between 1 (parallel execution disabled) and 25 (parallel execution disabled) that completes successfully and doesn't take 40-50 minutes.

I've just never seen anything like this. I've even seen logs for OST generation against some of Salesforce's internal orgs that are MASSIVELY populated, and those take about 5-15 minutes to build out an OST without any timeout/connection reset issues. That's why I'm wondering if it's worth getting Salesforce involved. It would be interesting to isolate the describeSObjects() call(s) that result in this issue consistently and give those to Salesforce outside of IC...basically show them that this org isn't able to service API requests in a timely fashion or sometimes at all.

[Alan Birchenough]

500 serial worked this time.

[Alan Birchenough]

Let me know if you'd like me to collect any other data points. [It is hard to keep perfect track of it all, since I am multitasking with my day job, which is the best time to try all this.]

[Scott Wells]

Okay, glad you're back in (semi-)working state now. I think the next step is to get you the aforementioned build that lets you tune concurrency during parallel execution and try to find a configuration sweet spot. I'll see if I can get that build to you tonight or early tomorrow.

[Scott Wells]

Okay, Alan. Here's your new build. In this build you can set the following two properties to try to tune OST generation:

• offline.symbol.table.max.fields - Same property as before, but now you can set it as low as 0 in which case it will automatically configure itself to use the maximum number of custom fields on a single SObject. This will allow for the smallest partition size that can load everything while still allowing for some batching of SObjects with lower field counts.
• offline.symbol.table.max.threads - This property allows you to set the number of threads used when parallel execution is enabled. The default is 25, but because it appears that heavy concurrency is causing the server to drop connections, I recommend you start with a value of 5 and, if that works, perhaps also try a value of 10.

So after installing this build, I would recommend you adjust your custom VM properties to include:

-Doffline.symbol.table.max.fields=0
-Doffline.symbol.table.max.threads=5

and enable parallel execution for your connection, then regenerate only the SObjects portion of the OST.

If that doesn't work, you might even set the thread count to 2 to see if ANY attempts at parallelization are going to be allowed against this org.

I'll be very interested to see what this yields...

[Alan Birchenough]

You bet. Won't be until later today, at the earliest, but I am motivated and will get to this as soon as I can.

[Alan Birchenough]

I was able to generate a full OST with max fields of zero and max threads at 5. I was not able to limit the generation to the SObject portion only, since a previous failed generation had jacked up the OST as a whole. The generation seemed to roar along at quite a pace, then pause ominously for several minutes near the end - I thought it was going to time out - and then resume, and complete. Here is the log. Let me know if you want me to try anything else. If not, I will continue to use these settings and let you know over the next few days if they don't reliably work.

[Scott Wells]

Thanks, Alan. Reviewing the log, unfortunately I still see connection resets and timeouts on those same calls:

2018-11-14 16:12:27,350 [ 127315]  DEBUG - loud.symtab.OfflineSymbolTable - Created 43 partitions for 1455 SObject types and 12561 fields. 
2018-11-14 16:12:27,351 [ 127316]   INFO - ct.OfflineSymbolTableGenerator - Starting activity Loading SObject types with 43 steps. 
2018-11-14 16:12:27,351 [ 127316]  DEBUG - loud.symtab.OfflineSymbolTable - Creating a parallel batch processor executor with 5 maximum threads. 

org.apache.cxf.interceptor.Fault: Unmarshalling Error: Connection reset 
    ...
    at com.sun.proxy.$Proxy176.describeSObjects(Unknown Source)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.lambda$process$0(OfflineSymbolTable.java:2168)
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(ForceComApiClient.java:220)
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(ForceComApiClient.java:318)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.process(OfflineSymbolTable.java:2162)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.process(OfflineSymbolTable.java:2158)
    at com.illuminatedcloud.util.ParallelBatchProcessorExecutor$1.call(ParallelBatchProcessorExecutor.java:70)

...

org.apache.cxf.interceptor.Fault: Unmarshalling Error: Read timed out 
    ...
    at com.sun.proxy.$Proxy176.describeSObjects(Unknown Source)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.lambda$process$0(OfflineSymbolTable.java:2168)
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(ForceComApiClient.java:220)
    at com.illuminatedcloud.client.ForceComApiClient.runWithClient(ForceComApiClient.java:318)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.process(OfflineSymbolTable.java:2162)
    at com.illuminatedcloud.symtab.OfflineSymbolTable$5.process(OfflineSymbolTable.java:2158)
    at com.illuminatedcloud.util.ParallelBatchProcessorExecutor$1.call(ParallelBatchProcessorExecutor.java:70)

That means that you're not going to have a fully-realized OST.

Based on what we've seen before, the connection resets are due to too heavy of concurrency, and the timeouts are due to...well...the server just not really wanting to provide a response in a timely fashion. You could dial the threads down to 2 and see if that get rid of one or both of these, but I'm not sure that the current settings are going to give you what you want/need.

[Scott Wells]

Just in case you're curious, here's how it partitioned the SObjects this time:

2018-11-14 16:12:27,313 [ 127278]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 1 SObject types with a total of 382 custom fields.
2018-11-14 16:12:27,314 [ 127279]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 1 SObject types with a total of 300 custom fields.
2018-11-14 16:12:27,314 [ 127279]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 1 SObject types with a total of 290 custom fields.
2018-11-14 16:12:27,314 [ 127279]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 1 SObject types with a total of 220 custom fields.
2018-11-14 16:12:27,314 [ 127279]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 2 SObject types with a total of 381 custom fields.
2018-11-14 16:12:27,314 [ 127279]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 2 SObject types with a total of 263 custom fields.
2018-11-14 16:12:27,314 [ 127279]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 3 SObject types with a total of 379 custom fields.
2018-11-14 16:12:27,314 [ 127279]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 3 SObject types with a total of 334 custom fields.
2018-11-14 16:12:27,314 [ 127279]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 4 SObject types with a total of 382 custom fields.
2018-11-14 16:12:27,314 [ 127279]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 4 SObject types with a total of 361 custom fields.
2018-11-14 16:12:27,314 [ 127279]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 4 SObject types with a total of 335 custom fields.
2018-11-14 16:12:27,315 [ 127280]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 5 SObject types with a total of 349 custom fields.
2018-11-14 16:12:27,315 [ 127280]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 6 SObject types with a total of 359 custom fields.
2018-11-14 16:12:27,315 [ 127280]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 7 SObject types with a total of 377 custom fields.
2018-11-14 16:12:27,315 [ 127280]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 7 SObject types with a total of 340 custom fields.
2018-11-14 16:12:27,315 [ 127280]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 8 SObject types with a total of 355 custom fields.
2018-11-14 16:12:27,316 [ 127281]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 9 SObject types with a total of 356 custom fields.
2018-11-14 16:12:27,316 [ 127281]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 10 SObject types with a total of 352 custom fields.
2018-11-14 16:12:27,316 [ 127281]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 11 SObject types with a total of 362 custom fields.
2018-11-14 16:12:27,316 [ 127281]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 12 SObject types with a total of 363 custom fields.
2018-11-14 16:12:27,317 [ 127282]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 13 SObject types with a total of 369 custom fields.
2018-11-14 16:12:27,317 [ 127282]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 14 SObject types with a total of 369 custom fields.
2018-11-14 16:12:27,319 [ 127284]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 16 SObject types with a total of 376 custom fields.
2018-11-14 16:12:27,320 [ 127285]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 18 SObject types with a total of 371 custom fields.
2018-11-14 16:12:27,320 [ 127285]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 20 SObject types with a total of 369 custom fields.
2018-11-14 16:12:27,321 [ 127286]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 23 SObject types with a total of 376 custom fields.
2018-11-14 16:12:27,322 [ 127287]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 26 SObject types with a total of 382 custom fields.
2018-11-14 16:12:27,323 [ 127288]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 28 SObject types with a total of 372 custom fields.
2018-11-14 16:12:27,323 [ 127288]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 34 SObject types with a total of 376 custom fields.
2018-11-14 16:12:27,324 [ 127289]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 40 SObject types with a total of 381 custom fields.
2018-11-14 16:12:27,325 [ 127290]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 45 SObject types with a total of 375 custom fields.
2018-11-14 16:12:27,326 [ 127291]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 56 SObject types with a total of 377 custom fields.
2018-11-14 16:12:27,328 [ 127293]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 68 SObject types with a total of 379 custom fields.
2018-11-14 16:12:27,330 [ 127295]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 85 SObject types with a total of 381 custom fields.
2018-11-14 16:12:27,332 [ 127297]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 316 custom fields.
2018-11-14 16:12:27,334 [ 127299]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 152 custom fields.
2018-11-14 16:12:27,337 [ 127302]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 0 custom fields.
2018-11-14 16:12:27,341 [ 127306]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 0 custom fields.
2018-11-14 16:12:27,343 [ 127308]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 0 custom fields.
2018-11-14 16:12:27,344 [ 127309]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 0 custom fields.
2018-11-14 16:12:27,346 [ 127311]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 0 custom fields.
2018-11-14 16:12:27,349 [ 127314]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 100 SObject types with a total of 0 custom fields.
2018-11-14 16:12:27,350 [ 127315]  DEBUG - loud.symtab.OfflineSymbolTable -   Created a partition with 68 SObject types with a total of 0 custom fields.

As you can see, the top few entries are calls to describe the metadata for a single SObject type. One thing that just occurred to me that I could do is log the partition details for the API calls that are failing. Because this is all happening in parallel across multiple threads, I don't know if it's the big single object calls that are failing or the ones that try to cram many objects with smaller numbers of fields each. That information might be useful because it might allow me to see if there's some other threshold we might need to respect when running APIs against these orgs of yours.

I'll see if I can get you one more build with that type of additional diagnostic logging in place.

[Scott Wells]

Okay, new build that just adds logging of the failed SObject types when an exception is caught calling describeSObjects(). Let's see if we can fully characterize what's going on and see if there's any other tuning that could help.

[Alan Birchenough]

OK. I will take this build but otherwise leave the settings in place. Next few times I do an OST generation that fails, I will attach the log here so you can see what is annoying the API most. Thanks for continuing to wear away at this thing.

[Alan Birchenough]

I have done two or three OST gens since my last post, and none of them has failed, so I'm not sure what to send you. Obviously, though, this is a significant improvement from my POV. :-)

[Scott Wells]

Yep, that's certainly tremendous progress from where we started! Okay, just let me know the next time if fails so I can review the log and see which calls are timing out still.

[Alan Birchenough]

This was an interesting case, which took place in my AAsPSA sandbox org. I had increased the number of threads in the VM argument to 9 just to see if I could speed things up at all. I didn't see any errors during the process, but it did miss some SObjects, leading to 253 unresolvable reference errors in the project. I will set back to 5 and retry, hoping to get everything, albeit more slowly. Hope this is useful.

[Scott Wells]

Alan, these errors aren't presented to the end user. Basically what you'd want to do is run the OST gen and then scan the log for the duration of that process for those "connection timed out" and "connection reset" errors. If you have even one of those, you're going to end up with an incomplete OST in some form or another.

[Alan Birchenough]

Got it. I tried again with threads=5, but still couldn't get a complete OST. Looking at the logs I see several "connection reset" errors. Not sure which parameter is causing the problem though. The SObject size seems to work most of the time, and we have previously had good result with threads at 5. At any rate, here is the compressed log from the second attempt in that org.

[Alan Birchenough]

Unfortunately, the OST gen process is still so unreliable as to be almost unusable. I get connection resets and other glitches even with these optimized parameters, and that renders the OST unusable, which in turn renders IC2 awkward to use (since one can never rely on the feedback it gives). Sure, I can disable the Unresolvable References inspection, but I also won't get code completion and when/if I need to be sure to find all references to an entity or field it is always random whether that entity got included in the OST or not, so I can't have confidence in my analysis.

I think we did secure some improvements by tuning these parameters, but to make the whole process work, I think we are going to need complete reliability. For example, to be able to retry queries within failing partitions until either those partitions succeed, or the user cancels, or some retry limit is reached. I appreciate that this is much more complex logic, but I can't think of any other way of guaranteeing the user a complete OST. Certainly having some partitions fail silently is not going to be workable in the long term. (I realize we are still debugging here, but I am talking about the finalized version of the OST generation function.)

At any rate, I attach the latest log for a failing OST gen, which (for my reference) was in the Wy FFd sandbox. And of course my rambling feedback here is purely FWIW; as always I defer to your greater knowledge and experience of the design options available and applicable.

[Scott Wells]

Thanks for the update, Alan. Refresh my memory...does the OST generate successfully if you turn off parallel processing entirely? If not, what we'll need to do is get a log from such a run showing these timeouts and server-side connection resets and share that with Salesforce to understand why these orgs are responding in such a manner to legitimate API requests. I can add retry logic, but unfortunately I suspect that if the server rejects the request once, it's likely going to do so (mostly) consistently.

Let me know your thoughts...

[Alan Birchenough]

I will refresh my own memory first then get back to you. Thanks.

[Alan Birchenough]

I think we previously found that this combination of settings was best, namely, 5 threads, max fields=0 and running in parallel.

Before we go to Salesforce - a move about which I feel pessimistic because they have so seldom helped me in the past - I will try with different network connections in case there really is also an element of network instability. I will get back to you.

On Tue, Dec 11, 2018 at 11:30 AM Alan Birchenough < alan.birchenough@ic

[Alan Birchenough]

Just a quick update on this: Many of the client orgs I work with daily are working with this configuration, and I most likely won't have cycles to pursue any further investigation for a couple of weeks, so I plan to sit on this issue for a little while before getting back to it. Let's please keep it open and I'll get back with a further update when I can. Thanks for all your work on it so far.

[Scott Wells]

Sure, Alan. This is on your schedule. I'll keep the issue open as long as it takes. Good luck with these clients!

[Alan Birchenough]

Hi. Unfortunately I may no longer be able to appear as the reporter of this issue (or my other issues) since Icon has taken away our old Google App accounts and forced us to move to new ones.

Any any rate, my question now is about accepting IC2 plugin updates in IJ Idea. Did you put those OST gen parameters into the regular build, so I'll be able to keep them when I accept regular plugin updates, or do I have to stick to the version I downloaded in order to continue to have access to them?

In other news, I haven't had a chance to investigate status further with these updates, but my experience over the last few weeks has been focused on an org in which I don't seem to run into to trouble with OST updates. As promised previously, I will let you know of any further difficulties I run into when I get the chance.

Thanks as always for your work. IC is an awesome environment for Salesforce developers.

[Scott Wells]

Hey, Alan. I believe the additional debug logging should be part of the official build now, though I'm not sure if those params are. I'll have to check (not sitting in front of that workstation right now). If not, I'll go ahead and fold them in so you don't have to stay on a non-standard build.

[Alan Birchenough]

Thanks!

[Alan Birchenough]

Actually if you can drop an update here when you incorporate those parameters I will then be able to take the next update. Thanks again.

[Scott Wells]

Sure, no problem. I didn't get it into the build that will come out tomorrow because I wanted to be very conservative in what's going into the Spring '19 release. I'll provide a build with those two params a bit later in the week and let you know about it here.

[Scott Wells]

Alan, I just checked and these two config properties are already in the official build. You should be good to update to any official build and still have access to them.

[Alan Birchenough]

Perfect. Thank you.

[Alan Birchenough]

I wonder if we can close this for now? I don't seem to have run into any further problems recently. I can reopen if necessary.

Thanks for all your work on this - it was epic!

P.S. Hope to see you at THDX '19 if only for a quick chat.

[Scott Wells]

Alan, I defer to you completely. I'm certainly glad to hear that it's working well for you, though! If you'd like to resolve it for now and reopen if the problem recurs, that works for me, as does just leaving it open to hedge our bets.

And my pleasure, of course. I'm just so sorry that you hit so many issues with it! Having said that, the journey so far has only strengthened the product when used against huge orgs, so very much worth it.

Yep, I'll be at TDX. Definitely tap me on the shoulder if you see me wandering around. I'd love to say "hi" in person!

[Scott Wells]

Issue tracker grooming. If this is still an issue, please feel free to reopen, ideally with a concrete reproduction scenario.