Access denied errors to TFS causing extremely slow Bamboo startup

[Brian Andersen]

There are several TFS repositories configured on our server that happen to have "bad" username and passwords configured for them. When Bamboo is restarted it attempts to scan these TFS repositories and cannot putting errors such as this into catalina.out:

2017-06-20 19:23:03,470 ERROR [18-ImmutablePlanCacheServiceImpl.initialiseCache:pool-34-thread-4] [TfsRepository] com.microsoft.tfs.core.exceptions.TFSUnauthorizedException: Access denied connecting to TFS server http://{{obfuscation}}:8080/ (authenticating as {{obfuscation}})

After a certain number of these failures, Tomcat just stops trying to start for 30 minutes while it's waiting for "something" to time out from TFS.

2017-06-20 19:52:34,384 WARN [18-ImmutablePlanCacheServiceImpl.initialiseCache:pool-34-thread-1] [TFSTeamProjectCollection] Error getting data provider com.microsoft.tfs.core.exceptions.TECoreException: Read timed out

Can we shorten the connection timeout somehow? Full logs can be provided, I would just prefer not to attach them to a public issue

[Sergey Podobry]

Hi Brian,

We'll try to reproduce the issue in our lab. Meanwhile please send the full logs to [e-mail].

[Brian Andersen]

Support zip has been sent as requested.

[Sergey Podobry]

Writing here to confirm that we've received the logs and continue investigating.

[Sergey Podobry]

It's very unlikely that the issue is related to access denied exceptions. There are 5 read timed out exception while access denied exceptions are much more. Also we observed no issues with access denied exceptions in our lab.

Cause

Bamboo sends a request, a socket connection is established but no repsonse is given.

Troubleshoot

Go to a Bamboo administration page, click Log settings, change com.microsoft.tfs to DEBUG and click Save. This will enable extended TFS logging (note that the settings are not persistent and will be reset to defaults on restart). Do this after lowering socket timeout value so you don't need to wait 30 minutes.

Workaround

Set socket timeout to a lower value (How to set socket timeout). I think 60-120 seconds will be enough.

[Brian Andersen]

Sergey:

I will give it a try. That com.microsoft.tfs.core.socketTimeoutSeconds setting is probably what it is. We recently went through and purged several incorrectly configured TFS repos, and the most recent Bamboo restart went much more quickly. I will let you know.

[Sergey Podobry]

Ok.

[Brian Andersen]

Sergey: I think the com.microsoft.tfs.core.socketTimeoutSeconds is our solution. We had to restart Bamboo a couple of times yesterday to resolve another issue (as well as update the TFS plugin from 1.1.17dev1 to 1.1.18) and all restarts were quick. I think you can close this issue.

[Sergey Podobry]

Good.