1. Stellarity Software Avatar Stellarity Software
  2. TFS Repository
  3. TFS Repository Plugin
  4. Issues

java.lang.UnsatisfiedLinkError while running several agents on the same machine simultaneously

Issue #59 resolved
Chris Butwell created an issue

Just upgraded to Bamboo 5.10.3 build 51020 and Stellarity 1.1.15 and find the following error in the build log when trying to connect to TFS. Note that our developers report it appears to be randomly occurring. Still attempting to validate this is random rather than consistently failing.

Error occurred while running Task '(3)' of type com.atlassian.bamboo.plugins.vcs:task.vcs.checkout. 30-Mar-2016 13:17:16 java.lang.UnsatisfiedLinkError: com.microsoft.tfs.jni.internal.filesystem.NativeFileSystem.nativeGetAttributes(Ljava/lang/String;)Lcom/microsoft/tfs/jni/FileSystemAttributes; 30-Mar-2016 13:17:16 at com.microsoft.tfs.jni.internal.filesystem.NativeFileSystem.nativeGetAttributes(Native Method) 30-Mar-2016 13:17:16 at com.microsoft.tfs.jni.internal.filesystem.NativeFileSystem.getAttributes(NativeFileSystem.java:74) 30-Mar-2016 13:17:16 at com.microsoft.tfs.jni.FileSystemUtils.getAttributes(FileSystemUtils.java:39) 30-Mar-2016 13:17:16 at com.microsoft.tfs.core.clients.versioncontrol.engines.internal.GetEngine.processOperation(GetEngine.java:1841) 30-Mar-2016 13:17:16 at com.microsoft.tfs.core.clients.versioncontrol.engines.internal.GetEngine.processOperationsInternal(GetEngine.java:1204) 30-Mar-2016 13:17:16 at com.microsoft.tfs.core.clients.versioncontrol.engines.internal.GetEngine.processOperations(GetEngine.java:998) 30-Mar-2016 13:17:16 at com.microsoft.tfs.core.clients.versioncontrol.engines.internal.GetEngine.processGetOperations(GetEngine.java:823) 30-Mar-2016 13:17:16 at com.microsoft.tfs.core.clients.versioncontrol.soapextensions.Workspace.get(Workspace.java:2511) 30-Mar-2016 13:17:16 at com.microsoft.tfs.core.clients.versioncontrol.soapextensions.Workspace.get(Workspace.java:2389) 30-Mar-2016 13:17:16 at com.microsoft.tfs.core.clients.versioncontrol.soapextensions.Workspace.get(Workspace.java:2377) 30-Mar-2016 13:17:16 at com.microsoft.tfs.core.clients.versioncontrol.soapextensions.Workspace.get(Workspace.java:2353) 30-Mar-2016 13:17:16 at com.stellarity.bamboo.repository.TfsRepository.checkoutFilesSimple(TfsRepository.java:942) 30-Mar-2016 13:17:16 at com.stellarity.bamboo.repository.TfsRepository.tryToRetrieveSourceCode(TfsRepository.java:880) 30-Mar-2016 13:17:16 at com.stellarity.bamboo.repository.TfsRepository.retrieveSourceCode(TfsRepository.java:457) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.fillWorkingDirFromVcs(VcsCheckoutTask.java:280) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.execute(VcsCheckoutTask.java:137) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.task.TaskExecutorImpl.lambda$executeTasks$125(TaskExecutorImpl.java:273) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.task.TaskExecutorImpl$$Lambda$52/1863467737.call(Unknown Source) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.task.TaskExecutorImpl.executeTaskWithPrePostActions(TaskExecutorImpl.java:202) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.task.TaskExecutorImpl.executeTasks(TaskExecutorImpl.java:273) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.task.TaskExecutorImpl.executePreparationTasks(TaskExecutorImpl.java:83) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask.call(PrepareBuildTask.java:74) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build(DefaultBuildAgent.java:215) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl$1.call(BuildAgentControllerImpl.java:139) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl$1.call(BuildAgentControllerImpl.java:130) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.variable.CustomVariableContextImpl.withVariableSubstitutor(CustomVariableContextImpl.java:215) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:129) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:137) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:49) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20) 30-Mar-2016 13:17:16 at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52) 30-Mar-2016 13:17:16 at java.lang.Thread.run(Thread.java:745)

Comments (16)

  3. Sergey Podobry

    Hi Chris,

    UnsatisfiedLinkError means that someone cleared the temp folder while Bamboo was running (and deleted native libraries that TFS uses). Could you check that? Maybe there is a script that clears the temp folder by schedule?

  4. Chris Butwell reporter

    Resolved this by recycling the bamboo remote agent. Developer reported this was random - however we have 6 identical remote agents on the same host. Issue was consistent on one of the 6 agents. Recycled that agent and issue is resolved. Not sure what occurred other than possibly the download to the remote agent after the bamboo upgrade corrupted the TFS plugin.

  7. Chris Butwell reporter
    • changed status to open

    This error still occasionally occurs on our bamboo remote agents. The cause is always a recycle of the remote agent and the error will continue for every build on the remote agent until we recycle the agent again. There are no scripts that delete directories. I have also verified the TFS jar exists on the remote agent. I've run cksum to confirm the jar matches the other remote agents that are running successful and they are identical. Any pointers on what to look for or debug would be greatly appreciated. No other errors occur on these remote agents. Just issues with TFS plugin. If we pull from GIT the builds are successful - even on the same agent.

  11. Sergey Podobry

    Could you check whether the directory /tmp/bamboo-tfs-sdk-native exists on a buggy agent? To be more specific the file /tmp/bamboo-tfs-sdk-native/xxxxxx/linux/x86_64/libnative_filesystem.so.

  13. Chris Butwell reporter

    Sure, I can give that version a try. Also a few more details on the remote agent host. I have six bamboo remote agents running on the same host. Each are placed in their own directories etc. so they don't collide. A shell script starts each one in order - there are six. When the JNI issue appears, a few, but not all of them will have errors. For example, agent #2 was having problems with JNI but agents #1, #3, #4, #5, #6 are working properly. Odd if they all share /tmp/bamboo/.... right?

    I've disabled #2 for now rather than restart in case there is some evidence of why it is failing. Below is the contents of the /tmp directory you requested.

    [xxx@xxx] /tmp/bamboo-tfs-sdk-native/1469659975289/linux/x86_64 $ ls -ltr total 96 -rw-r--r-- 1 xxx 15279 Jul 27 17:52 libnative_synchronization.so -rw-r--r-- 1 xxx 13557 Jul 27 17:52 libnative_misc.so -rw-r--r-- 1 xxx 16299 Jul 27 17:52 libnative_filesystem.so -rw-r--r-- 1 xxx 13482 Jul 27 17:52 libnative_console.so -rw-r--r-- 1 xxx 31160 Jul 27 17:52 libnative_auth.so

  14. Sergey Podobry

    Oh, now the issue is perfectly clear. All agents share the same tmp directory and each one tries to cleanup it before use. If all required libraries loaded then it's ok. But if libnative_filesystem.so is not loaded (it happens if there is nothing new to checkout from a repo) then it gets deleted and JNI exception is thrown later.

    With the new version the issue should be gone.

  17. Log in to comment
Assignee
Type
bug
Priority
major
Status
resolved
Version
1.1.15
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!