Stopping Splunkd or SplunkForwarder using TA-metricator-for-nmon run into TimeoutStopSec=360

Issue #22 open
Jo Ru created an issue

Hi metricator team,

we are happily using your App. Thank you for the awesome work.

Sadly, we are running into problems with OracleLinux8 at the moment.

When stopping the SplunkForwarder with systemd, the splunk daemon reaches its default timeout after 360s configured in the system unit file.

When we kill the process with the pid from  /opt/splunkforwarder/var/log/metricator/var/fifo_reader_fifo1.pid, the splunkforwarder stops within a few seconds.

We are not sure, but it seems as if the SIGINT signal from systemd to splunk is not propagated down to all subprocesses and therefore splunkd never returns an exit code.

Maybe there is another reason as this only occurs on OL8 in our case.

We’d be happy to assist you in fixing the problem. 

Best regards and thank you again

Tested version infos

This issue is reproducible with following component versions:

Splunk Versions:

  • UF 9.1.1 and 9.1.2
  • Splunk Enterprise 9.1.2

OS Versions:

  • Oracle Linux 8 (8.9)
  • NOT on Oracle Linux 9 or Oracle 7

We further tested different binaries on OL8.9 from /opt/splunkforwarder/var/log/metricator/bin/linux with the very same issue:

  • ./ol/nmon_x86_64_ol8
  • ./generic/nmon_linux_x86_64
  • ./rhel/nmon_x86_64_rhel8

But it seems not to rely on the binary.

Comments (6)

  1. Jo Ru reporter

    Hi! Any ideas on that? We would really appreciate to get this one fixed and offer every help you need!

  2. Jo Ru reporter
    • changed status to open

    Hi Guilhem!

    Sadly, we tested the new fix on our OL8 machines and do still see a delayed stop, with the exact same behavior.

    Best regards,

    Johannes

  3. Guilhem Marchand

    Hi Jo!
    All right - I will have a look, I believe I had some confusion with other Oracle Linux issues related to some binary issues failing to start Nmon.

    I will have a look at this.

    Guilhem

  4. Jo Ru reporter

    Hi Guilhem!

    Many thanks in advance! Because we still see this behavior on hundreds of servers.

    Johannes

  5. Log in to comment