Technical Addon crashes universal forwarder

[Former user]

Hi Team,

we deployed the Ta to an AIX 7.1 instance.

Every time we start the universal forwarder and the nmon digest starts it crashes the whole universal forwarder with following error entry:

06-21-2021 14:46:43.643 +0200 ERROR ProcessRunner [37027871 ProcessRunner] - child's last words: cannot find portable_pid_t 35325136 in _pidToUniqMap
06-21-2021 14:46:43.645 +0200 FATAL ProcessRunner [37027871 ProcessRunner] - Unexpected EOF from process runner child!

This is our only 7.1 instance left and (of course) the production environment.

What can we do to debug the issue?

Thanks for any information how we can narrow it down. Best Regards, Jens

[Guilhem Marchand]

Hi Jens.

This looks like ulimits issues on the UF side.
AIX is very restrictive by default, way too much for the UF to be operating normally, and attempting to do anything besides very minor file monitoring or just setting there idle.

Some resources:
https://answers.splunk.com/answers/173567/why-does-splunk-612-forwarder-on-an-aix-71-machine.html

Can you review the ulimits being set on the instance?

Example of known good values:

Once you’ve set proper ulimits via Smitty, restart the UF and verify in splunkd.log if you see any complaints still.

[Guilhem Marchand]

Assumed to be fixed with the recommendations above.

[Guilhem Marchand]

assumed to be fixed