Generating OST results in an Organizational Lock in SF for other developers.

[June Bischoff]

Whenever any of our developers needs to regenerate the OST, that environment is locked for all developers for the length of time it takes to generate. (15+ minutes)

Is there any way to make this a read-only process such that it doesn't produce an org lock?

Most of the time it's not a big deal, but it's a pain when we hit an all-hands-on-deck/many new fields/heavy development phase.

[Scott Wells]

That's really strange. It actually is a read-only process and just calls various aspects of the Tooling, Partner, and Metadata APIs. I can't think of any reason those calls would take out an org-wide lock, much less for the full duration of the operation. I'll need to ping my contacts at Salesforce about which calls might be causing the lock and what I might be able to do to avoid it (or at least minimize it). I'll let you know what I find.

[June Bischoff]

We ran into it most recently when setting up a new project, so he was going through the entire new connection/retrieve metadata/generate OST process, but I'm pretty sure I've hit it once previously when I was just regenerating the OST.

...I wouldn't swear to it, though; I might have been creating a new project. And it definitely held the lock during the entire process until the OST was generated. I can test and confirm whether it's an OST-only problem late today when we're not all in the middle of development.

[Scott Wells]

Okay. I've emailed my Salesforce contacts with details about the specific API calls that are made to build the OST and asked whether any of them might be taking out locks. Is it possible that the user was deploying to the org when the lock was held? That will definitely happen, and the new project wizard will allow you to deploy from local source automatically after creation of the OST.

[June Bischoff]

That's not part of our typical process, but I'll look into it. Thanks!

[Scott Wells]

June, here's Salesforce's documentation on the types of activities that can cause an org-wide lock:

https://help.salesforce.com/apex/HTViewSolution?id=000004045&language=en_US

and at the end it suggests using Setup > View Setup Audit Trail to determine which activities might have happened to cause the lock. The next time this happens, would you mind taking a look at the audit trail and trying to correlate anything there to the timeframe of the lock?

[June Bischoff]

I've just confirmed -- regenerating the OST alone causes the Organization Administration Lock. I'm the only developer still here, and there is no recent activity in the Setup Audit Trail.

Steps to recreate: Regenerate the OST from Illuminated Cloud. Log into the SF Org. Open any APEX class for editing. Add a comment or other minor change. Save the Class. Get Org Lock error.

[Scott Wells]

Thanks, June. That helps. I'll do some investigation on my own to see if I can figure out which operations are causing the lock while I wait for a response from my Salesforce contacts.

[Scott Wells]

June, the Salesforce folks are looking at why this might be happening based on the API calls that IC makes to build its OST. As I mentioned above, if I don't hear anything useful from them in the next few days, I'll also see if I can instrument things a bit to try to isolate it as well. I'll let you know when I have anything useful to share.

[Scott Wells]

I do have an update on this. The current thought is that this might be occurring due to a particular header that IC's Salesforce SOAP API client is using to get access to some org metadata that's not generally available through the API. To date I've been setting this request header all the time, but because it's potentially causing an org-wide lock to be taken out, I've changed the client to set the header only when absolutely needed to access this metadata. If this is in fact the case, while this won't eliminate the org-wide lock, it should minimize the duration of it as much as I can and still build the same OST.

I'm going to test these changes quite a bit before releasing a build with them since they're in the heart of the API client (though they're actually quite narrow in scope). I'll update this issue when I've released it and hopefully it'll help ameliorate the issue.

[Scott Wells]

June, what type of org are you using when this happens? Is it as a development edition org? production org? sandbox org? I ask because I've been unable to reproduce it even without the change I noted above, and I'm testing against a dev edition org. It's possible that the logic to lock the org only kicks in if you're in a prod org or sandbox org, so I wanted to make sure I'm testing in the same type of environment.

[June Bischoff]

Minimizing the duration of the lock would be wonderful, especially since the development environment for our current project is taking over an hour to generate the OST. (Compared to 15 or 20 minutes in our other sandboxes. Monstrous new managed package. Yes, we have a case open with SF to try to speed everything up.)

Thank you!

[June Bischoff]

I'm currently seeing it in our Full sandbox. I have also seen it in what I believe was a Developer Pro edition.

[Scott Wells]

Thanks for the quick response. Okay, I'm working against a straight up developer edition org but I do have a few partner dev edition orgs and test orgs sitting around from which I should be able to create sandboxes. I'll keep trying to see if I can reproduce it.

[Jacek Nakonieczny]

What's against adding OST to version control ?

Why not put ant lib for external tools like CI servers so they would be able to generate OST on demand/scheduled basis. That could be useful when whole team uses IC.

[Scott Wells]

Adding the OST to VCS is absolutely an option. Having OST generation available as an ant (or other command-line automation) tool is a very interesting idea. I'll open a separate enhancement request for that.

[Scott Wells]

June, I'm still researching this. I did manage to reproduce it, or at least I think I did, but only intermittently. It's even possible that I caused the lock by spamming the org with saves during OST generation. At this point perhaps some of the enhancements I'm making in the other logged issue around making OST generation faster will alleviate this pain for you. I'm also still chatting with Salesforce about it, but so far they're pretty perplexed as to why these calls would take out org locks. So anyway, still digging. I'll keep you posted!

[Scott Wells]

June, can you provide an update on the severity of this now with the OST generation improvements? Trying to figure out how to prioritize this relative to other items, especially since I've been having trouble reproducing it with any consistency.

[June Bischoff]

I think this is less critical now.

[Scott Wells]

Thanks, June. Just let me know if that changes, but for now I'm going to put it on the back-burner so I can concentrate on these other items that are higher priority for you and other users.

[Scott Wells]

I'm going to resolve this since it doesn't appear to have been an issue in quite a while. If it turns out that it is in fact still an issue, please feel free to reopen and let me know the specific situation(s) in which it's getting in the way.