Trying to get Metricator add-on working on AIX VIO server

[Matthew Pearce]

We are trying to get the NMON metricator working on the splunk agent on an AIX VIO server.
https://splunkbase.splunk.com/app/3948/

The splunk agent appears to be reporting back ok via _internal index and other event indexes, but we're not seeing any metric data returning. The metrics index is fine as other AIX servers are reporting nmon metrics without issue. It’s only the VIO that is giving errors. We are receiving multiple errors, and wondered if anyone else has got this working, or if there are any gotchas with trying to run this add-on working on the VIO server? It appears possibly the nmon metrics aren’t being collected properly.

Here is a snippet of some of the logs coming from the os-unix-nmon-internal index.

NMON ID: 22-DEC-2020:09:29:10,vioswdmza.int.corp.sun,02104680T,147339,1608593350,,1608593350
22-12-2020 09:29:09, vioswdmza INFO: starting nmon : /usr/bin/topas_nmon -F /home/padmin/splunkforwarder/var/log/metricator/var/nmon_repository/fifo1/nmon.fifo -T -A -d -K -L -M -P -^ -p -yoverwrite=1 -s 60 -c 1440 in /home/padmin/splunkforwarder/var/log/metricator/var/nmon_repository/fifo1

22-12-2020 09:29:08, vioswdmza INFO: starting the fifo_reader fifo1

22-12-2020 09:29:08, vioswdmza INFO: Removing stale pid file (process not found)

22-12-2020 09:29:05 Reading NMON data: 2063 lines 147340 bytes
Splunk Root Directory ($SPLUNK_HOME): /home/padmin/splunkforwarder
addon type: /home/padmin/splunkforwarder/etc/apps/TA-metricator-for-nmon
addon version: 1.0.11
nmonparser version: 2.0.3
Collapse

22-12-2020 09:28:08, vioswdmza WARN: the current nmon process has an outdated definition of network interfaces. The current process with PID 34078980 will be terminated and a new process will be spawned at next occurrence.
22-12-2020 09:28:05 Reading NMON data: 2063 lines 147340 bytes
Splunk Root Directory ($SPLUNK_HOME): /home/padmin/splunkforwarder
addon type: /home/padmin/splunkforwarder/etc/apps/TA-metricator-for-nmon
addon version: 1.0.11
nmonparser version: 2.0.3
Show all 18 lines
Starting_epochtime: 1608593230
Ending_epochtime:
CONFIG section: will not be extracted (time delta of 60539 seconds is inferior to 86400 seconds)
WARN: hostname: vioswdmza.int.corp.sun :CPUnn section is not consistent: 0 fields in data, 5 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :CPUnn section is not consistent: 0 fields in data, 5 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency

========
Starting_epochtime: 1608593350
Ending_epochtime:
CONFIG section: will not be extracted (time delta of 60599 seconds is inferior to 86400 seconds)
WARN: hostname: vioswdmza.int.corp.sun :CPUnn section is not consistent: 0 fields in data, 5 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :CPU_ALL section is not consistent: 0 fields in data, 7 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :FILE section is not consistent: 0 fields in data, 9 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :MEM section is not consistent: 0 fields in data, 7 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :PAGE section is not consistent: 0 fields in data, 9 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :MEMNEW section is not consistent: 0 fields in data, 7 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :MEMUSE section is not consistent: 0 fields in data, 9 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :PROC section is not consistent: 0 fields in data, 14 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :PROCCOUNT section is not consistent: 0 fields in data, 2 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :LPAR section is not consistent: 0 fields in data, 22 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :POOLS section is not consistent: 0 fields in data, 10 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :DISKBUSY section is not consistent: 0 fields in data, 5 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency
WARN: hostname: vioswdmza.int.corp.sun :DISKBSIZE section is not consistent: 0 fields in data, 5 fields in header, extra fields detected (more fields in data than header), dropping this section to prevent data inconsistency

Also, here is a common error we find from the _internal index:

12-22-2020 11:22:04.754 +1000 ERROR ExecProcessor - message from "/home/padmin/splunkforwarder/etc/apps/TA-metricator-for-nmon/bin/metricator_consumer.sh" /home/padmin/splunkforwarder/var/log/metricator/var/nmon_repository/fifo1/nmon_timestamp.dat: A file or directory in the path name does not exist.

I have checked permissions and it all looks fine. Also, I've viewed the perl nmon process running.