TECoreException: Connection reset

Issue #67 resolved
Bobby Brown created an issue

Recently we upgraded our servers to sha2 certs and we use https on all applications -- the JDKs running the apps also have had their certs updated -- since this time we are no longer able to use the TFS plugin, all other functionality is working in bamboo.

error details follow

com.atlassian.bamboo.repository.RepositoryException: com.microsoft.tfs.core.exceptions.TECoreException: Connection reset at com.stellarity.bamboo.repository.TfsRepository.collectChanges(TfsRepository.java:404) at com.stellarity.bamboo.repository.TfsRepository.collectChangesSinceLastBuild(TfsRepository.java:289) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectionChangesWithRetry(DefaultChangeDetectionManager.java:556) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.access$400(DefaultChangeDetectionManager.java:76) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager$3.call(DefaultChangeDetectionManager.java:414) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager$3.call(DefaultChangeDetectionManager.java:358) at com.atlassian.bamboo.variable.CustomVariableContextImpl.withVariableSubstitutor(CustomVariableContextImpl.java:215) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceLastBuildInternal(DefaultChangeDetectionManager.java:357) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceRevisions(DefaultChangeDetectionManager.java:306) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceRevisions(DefaultChangeDetectionManager.java:198) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectAllChangesSinceLastBuild(DefaultChangeDetectionManager.java:155) at com.atlassian.bamboo.v2.trigger.ManualBuildDetectionAction.performDelayedChangeDetection(ManualBuildDetectionAction.java:172) at com.atlassian.bamboo.chains.ChainExecutionManagerImpl$2.getChainState(ChainExecutionManagerImpl.java:225) at com.atlassian.bamboo.chains.ChainExecutionManagerImpl.tryStartChainState(ChainExecutionManagerImpl.java:278) at com.atlassian.bamboo.chains.ChainExecutionManagerImpl.delayedStart(ChainExecutionManagerImpl.java:214) at sun.reflect.GeneratedMethodAccessor2061.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) at com.sun.proxy.$Proxy148.delayedStart(Unknown Source) at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$2$1$1$1.run(PlanExecutionManagerImpl.java:430) at com.atlassian.bamboo.util.CacheAwareness$2.call(CacheAwareness.java:83) at com.atlassian.bamboo.util.CacheAwareness$2.call(CacheAwareness.java:79) at com.atlassian.bamboo.util.CacheAwareness.whileReadingThroughCaches(CacheAwareness.java:101) at com.atlassian.bamboo.util.CacheAwareness.whileReadingThroughCaches(CacheAwareness.java:78) at com.atlassian.bamboo.util.CacheAwareness.whileReadingThroughCaches(CacheAwareness.java:118) at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$2$1$1.call(PlanExecutionManagerImpl.java:425) at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$2$1$1.call(PlanExecutionManagerImpl.java:417) at com.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:312) at com.atlassian.bamboo.plan.PlanExecutionLockServiceImpl.lock(PlanExecutionLockServiceImpl.java:85) at com.atlassian.bamboo.plan.PlanExecutionManagerImpl.doWithProcessLock(PlanExecutionManagerImpl.java:800) at com.atlassian.bamboo.plan.PlanExecutionManagerImpl.access$300(PlanExecutionManagerImpl.java:129) at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$2$1.run(PlanExecutionManagerImpl.java:411) at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:49) at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31) at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20) at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52) at java.lang.Thread.run(Thread.java:745) Caused by: com.microsoft.tfs.core.exceptions.TECoreException: Connection reset at com.microsoft.tfs.core.exceptions.mappers.TECoreExceptionMapper.map(TECoreExceptionMapper.java:99) at com.microsoft.tfs.core.exceptions.mappers.LocationExceptionMapper.map(LocationExceptionMapper.java:32) at com.microsoft.tfs.core.clients.framework.location.internal.LocationWebServiceProxy.connect(LocationWebServiceProxy.java:76) at com.microsoft.tfs.core.clients.framework.location.LocationService.connect(LocationService.java:754) at com.microsoft.tfs.core.clients.framework.location.LocationService.ensureConnected(LocationService.java:884) at com.microsoft.tfs.core.clients.framework.location.LocationService.getInstanceID(LocationService.java:943) at com.microsoft.tfs.core.TFSConnection.getInstanceID(TFSConnection.java:533) at com.stellarity.bamboo.repository.TfsRepository.cacheProjectInfo(TfsRepository.java:224) at com.stellarity.bamboo.repository.TfsRepository.collectChanges(TfsRepository.java:326) ... 42 more Caused by: com.microsoft.tfs.core.ws.runtime.exceptions.TransportException: Connection reset at com.microsoft.tfs.core.ws.runtime.client.SOAPService.executeSOAPRequestInternal(SOAPService.java:744) at com.microsoft.tfs.core.ws.runtime.client.SOAPService.executeSOAPRequest(SOAPService.java:473) at ms.ws._LocationWebServiceSoap12Service.connect(_LocationWebServiceSoap12Service.java:384) at com.microsoft.tfs.core.clients.framework.location.internal.LocationWebServiceProxy.connect(LocationWebServiceProxy.java:70) ... 48 more Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:209) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747) at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at com.microsoft.tfs.core.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:597) at com.microsoft.tfs.core.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2518) at com.microsoft.tfs.core.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1313) at com.microsoft.tfs.core.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:508) at com.microsoft.tfs.core.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:197) at com.microsoft.tfs.core.httpclient.HttpClient.executeMethod(HttpClient.java:464) at com.microsoft.tfs.core.httpclient.HttpClient.executeMethod(HttpClient.java:376) at com.microsoft.tfs.core.ws.runtime.client.SOAPService.executeSOAPRequestInternal(SOAPService.java:617) ... 51 more

Comments (12)

  1. Sergey Podobry

    Hi @bobby_brown,

    Could you find and post the whole exception stacktrace because the current one is not complete (it is ended "51 more")? It would be helpful.

    Note: If the full stacktrace is not available via UI it can be found in the bamboo log.

  2. Sergey Podobry

    Also please check your JRE/lib/security/local_policy.jar. Unpack it and take a look at default_local.policy. Does it look like this?

    // Some countries have import limits on crypto strength. This policy file
    // is worldwide importable.
    
    grant {
        permission javax.crypto.CryptoPermission "DES", 64;
        permission javax.crypto.CryptoPermission "DESede", *;
        permission javax.crypto.CryptoPermission "RC2", 128, 
                                         "javax.crypto.spec.RC2ParameterSpec", 128;
        permission javax.crypto.CryptoPermission "RC4", 128;
        permission javax.crypto.CryptoPermission "RC5", 128, 
              "javax.crypto.spec.RC5ParameterSpec", *, 12, *;
        permission javax.crypto.CryptoPermission "RSA", *;
        permission javax.crypto.CryptoPermission *, 128;
    }; 
    
  3. Bobby Brown reporter

    I tried your recommendation from above but seems to have the same error -- will do some more troubleshooting tomorrow and verify the java is the correct one -- thanks for your help and will get back tomorrow

  4. Sergey Podobry

    Also try adding -Djavax.net.debug=ssl to jvm options. The bamboo log will containt all SSL related info.

  5. Bobby Brown reporter

    So the issue is that the connection needs to use TLS SNI to make the connection, since TFS moved to using that, we did a dump of the cipher list and they were compatible.

  6. Sergey Podobry

    I see. Unfortunately TFS SDK doesn't support SNI. I'll take a look whether there is something that can be done. Meanwhile you can turn off SNI (if it is not required): e0a607fa-ef0d-11e6-9919-1a8d6f1534a0.png

  7. Log in to comment