UnsatisfiedLinkError when running several agents on the same Linux machine

Issue #83 resolved
Sandeep Nalam created an issue

All the TFS based plans in the Bamboo are failing with error. Can you please look into this issue and help to resolved. This is impacting all the build plans. Attached an output for the refrence.

Configuration: We have 1 Server which have 6 Agents are using and these are mainly for the Checkmarx Scan’s.

Comments (40)

  1. Sandeep Nalam reporter

    Still its not working. I have updated the version of TFS plugin and restart the Bamboo Server and the remote Agents too.

  2. Sandeep Nalam reporter

    I can see that:
    [prodcie@up15095] /tmp/bamboo-tfs-sdk-native1626692659681 $ ls -rtla
    total 4
    -rw-r----- 1 prodcie cie 0 Jul 19 06:04 lock
    drwxr-x--- 3 prodcie cie 17 Jul 19 06:04 aix
    drwxr-x--- 4 prodcie cie 31 Jul 19 06:04 freebsd
    drwxr-x--- 4 prodcie cie 36 Jul 19 06:04 hpux
    drwxr-x--- 6 prodcie cie 53 Jul 19 06:04 linux
    drwxr-x--- 2 prodcie cie 204 Jul 19 06:04 macosx
    drwxr-x--- 5 prodcie cie 44 Jul 19 06:04 solaris
    drwxr-x--- 9 prodcie cie 111 Jul 19 06:04 .
    drwxr-x--- 4 prodcie cie 31 Jul 19 06:04 win32
    drwxrwxrwt. 27 root root 4096 Jul 19 06:51 ..
    [prodcie@up15095] /tmp/bamboo-tfs-sdk-native1626692659681 $

  3. Sandeep Nalam reporter

    [prodcie@up15095] /tmp/bamboo-tfs-sdk-native1626692659681 $ uname -a
    Linux up15095 3.10.0-1160.25.1.el7.x86_64 #1 SMP Tue Apr 13 18:55:45 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
    [prodcie@up15095] /tmp/bamboo-tfs-sdk-native1626692659681 $ cat /etc/redhat-release
    Red Hat Enterprise Linux Server release 7.9 (Maipo)
    [prodcie@up15095] /tmp/bamboo-tfs-sdk-native1626692659681 $

  4. Sergey Podobry

    Let me check on that system. Meanwhile you can run:

    ldd /tmp/bamboo-tfs-sdk-native1626692659681/linux/x86_64/libnative_misc.so
    

  5. Sandeep Nalam reporter

    [prodcie@up15095] /tmp/bamboo-tfs-sdk-native1626692659681/linux/x86_64 $ ll
    total 96
    drwxr-x--- 2 prodcie cie 151 Jul 19 06:04 .
    drwxr-x--- 6 prodcie cie 53 Jul 19 06:04 ..
    -rw-r-xr-x 1 prodcie cie 31160 Jul 19 06:04 libnative_auth.so
    -rw-r-xr-x 1 prodcie cie 13482 Jul 19 06:04 libnative_console.so
    -rw-r-xr-x 1 prodcie cie 16299 Jul 19 06:04 libnative_filesystem.so
    -rw-r-xr-x 1 prodcie cie 13557 Jul 19 06:04 libnative_misc.so
    -rw-r-xr-x 1 prodcie cie 15279 Jul 19 06:04 libnative_synchronization.so
    [prodcie@up15095] /tmp/bamboo-tfs-sdk-native1626692659681/linux/x86_64 $
    [prodcie@up15095] /tmp/bamboo-tfs-sdk-native1626692659681/linux/x86_64 $
    [prodcie@up15095] /tmp/bamboo-tfs-sdk-native1626692659681/linux/x86_64 $ ldd /tmp/bamboo-tfs-sdk-native1626692659681/linux/x86_64/libnative_misc.so
    ldd: warning: you do not have execution permission for /tmp/bamboo-tfs-sdk-native1626692659681/linux/x86_64/libnative_misc.so' not a dynamic executable [prodcie@up15095] /tmp/bamboo-tfs-sdk-native1626692659681/linux/x86_64 $ ldd libnative_misc.so ldd: warning: you do not have execution permission for./libnative_misc.so'
    not a dynamic executable
    [prodcie@up15095] /tmp/bamboo-tfs-sdk-native1626692659681/linux/x86_64 $

  6. Sergey Podobry

    It may be caused by SELinux. You can temporarily change the SELinux mode from enforcing to permissive with the following command:

    sudo setenforce 0
    

    And then restart Bamboo and see whether the issue is gone.

  7. Sandeep Nalam reporter

    We don’t have the root permissions to change it. Please suggest any other alternative approach.

  8. Sandeep Nalam reporter

    Sorry for the delay.

    [prodcie@up15095] /usr/appl/cie/home/prodcie/bamboo-agent-2 $ getsebool user_exec_content
    user_exec_content --> on
    [prodcie@up15095] /usr/appl/cie/home/prodcie/bamboo-agent-2 $ sestatus
    SELinux status: enabled
    SELinuxfs mount: /sys/fs/selinux
    SELinux root directory: /etc/selinux
    Loaded policy name: targeted
    Current mode: permissive
    Mode from config file: disabled
    Policy MLS status: enabled
    Policy deny_unknown status: allowed
    Max kernel policy version: 31
    [prodcie@up15095] /usr/appl/cie/home/prodcie/bamboo-agent-2 $ ldd /tmp/bamboo-tfs-sdk-native1626959969218/linux/x86_64/libnative_misc.so
    ldd: warning: you do not have execution permission for `/tmp/bamboo-tfs-sdk-native1626959969218/linux/x86_64/libnative_misc.so'
    not a dynamic executable
    [prodcie@up15095] /usr/appl/cie/home/prodcie/bamboo-agent-2 $

    Dont have rights to look into the audit logs.

    They have rebooted the Node and then I have started the agent and tried still the same error.

  9. Sandeep Nalam reporter

    I have copied the /tmp/bamboo-tfs* to home dir to run the ldd

    Post that ldd is working:

    [prodcie@up15095] /usr/appl/cie/home/prodcie $ ldd bamboo-tfs-sdk-native1626960383332/linux/x86_64/libnative_misc.so
    ldd: warning: you do not have execution permission for `bamboo-tfs-sdk-native1626960383332/linux/x86_64/libnative_misc.so'
    linux-vdso.so.1 => (0x00007ffd46181000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f887f72e000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f887f512000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f887fafc000)
    [prodcie@up15095] /usr/appl/cie/home/prodcie $

  10. Sergey Podobry

    Yes, it’s definitely SElinux. I’ve got the same output:

    blocked by policy:

    [bamboo@centos x86_64]$ ldd libnative_misc.so 
    ldd: warning: you do not have execution permission for `./libnative_misc.so'
        not a dynamic executable
    

    allowed by policy:

    [user@centos x86_64]$ ldd libnative_misc.so 
    ldd: warning: you do not have execution permission for `./libnative_misc.so'
        linux-vdso.so.1 =>  (0x00007ffdb06c7000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fb64f95a000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb64f73e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb64fd28000)
    

    I think I’ll add a code to test if /tmp is executable and then fallback to /home.

  11. Sandeep Nalam reporter

    Is it possible to place in another directory rather than /tmp. Or any startup argument to change the path, which might help. Its just my point of view.

  12. Sandeep Nalam reporter

    The JAR is downloaded, but if I apply it, that needs bamboo restart which I will later today.

  13. Sergey Podobry

    Now *.so files are extracted to <bamboo_home>/temp that should be executable as it’s used by a lot of built-in bamboo tasks. I think you need to add your SELinux admin into the loop, so we’ll be able to find the best solution.

  14. Sandeep Nalam reporter

    Thanks for the input. there was no .so files under <bamboo_home>/temp.

    So I have added -Djava.io.tmpdir=<bamboo_home>/temp to the startup args and then I have restarted the agent. Post restart I saw build is getting triggered successfully. I will keep monitoring with different plans and I will update you.

  15. Sergey Podobry

    -Djava.io.tmpdir=<bamboo_home>/temp should work but the idea was to automatically select this location if /tmp is not executable.

  16. Log in to comment