1. pedro Avatar pedro
  2. FreshTomato
  3. freshtomato-arm
  4. Issues

SNMP still not get the internal interface speed when walked with cfgmaker/mrtg and others

Issue #81 closed
TheHiman created an issue

There is still the problem since a while, including with this SNMP 5.9-update, that on arm and mips the SNMP daemon is still incomplete after a bigger module change in the middle of the year at the makefile for snmp build config:

The following problems are still existing:

  • all physical interface eth1 - ethX still reports an interface speed of 0 - which let them exclude at the configmaker of mrtg, etc.
  • only the vlans got correctly solved (but they all get a “fake” interface speed of 1250 kbit instead of gigabit or megabit…)
  • The interface order for the physcial eth devices are now dynamicly assigned right at the end of the interface list and are often changed when some settings, like vpn, qos or other “ifb-block” virtual interfaces added, changed or removed.
  • eth1 and eth2, which is mostly used for the physical WLAN 2,4+5 GHz devices, was years long always on interface position 3 and 4, and reported the correct interface speed and a correct basic template was build - now they are near at the end of the interface list and are often changing when a new vlan is added, qos or other things are changed and become a new interface order on the fly, which is normal for tunnel and other dynamic device. After a cold reboot, the order is changed again.
  • There is still the issue that 64bit counters/access is no working any longer - the nessassary sub-modules was excluded for a while to save some space. This is for a today gbit home access in 2020 a show stopper in monitoring when the interfaces are accounted with mrtg in typical 5 minutes intervals. in 5 Minutes you get a lot more then 4096 MB. But Tomato itself ha no interface counter problem or 32 bit if wrap-issues at 4096 - so it can be read out correctly. Get back the 64bit support es easy by just re-include the nessassary submodules like it was in in the past. Normaly 64bit counters are used with snmpv2, but this is still no longer working, too.

The reported problems from the “snmpwalk” is “ “* has no ifSpeed property” and/or “it is operational DOWN” / “it is administratively DOWN”

actualy eth0 is correctly reported, but the same order problem here: is becomes interface6 and can vary, because the newer ifb0 - ifb3 devices now starts as “down” on position 2 - so the the same issue with the basic “hard” interfaces where the order can change in running state when one of “dynamic interfaces” are deleted, changed or reloaded.

When i remember correctly from the past, there is some sort of extra switch to fix the interface order at start time and add dynamicly created interfaces on the fly and add them always at the end of the list. I´am not sure, is this still true for snmp 5.9 ?

The old and good behavior was this standard-order:

localhost always as interface 0, then all physical eth0-ethX Interfaces, then all created vlans and all other dynamic and virtual interfaces, tunnels, etc comes always at the end of the interface list. So in case of modify more often vpn scenarios the bare basics newer changes the basic-order - and usualy only the basics are accounted and never virtual interfaces - which anyway reported mostly 0 traffic 😉

Comments (16)

  2. TheHiman reporter

    for the 64bit counter problem i have here one link:

    https://dev.archive.openwrt.org/attachment/ticket/8818/snmp.patch.html

    https://dev.archive.openwrt.org/changeset/25486

    basicly this modules must be included again:
    if-mib/ifXTable \

    and this needs to be added to the configure options:

    --enable-mfd-rewrites \` `

    i made some tests by adding this, but actualy it is still not working as expected, eventualy a look on an actual debian/ubuntu configure-script can help, which modules
    are all included. In my test-screnario snmp-walk see now that 64bit counters are working - but no interface itself was accountable, because many of them reported an invalid interface speed, see the problem above. (means all primary interfaces like eth0-eth2 and/or the vlans which reported a wrong if speed). Eventually some more submodules needs to be added to make it work again.

  3. TheHiman reporter

    Here i have a logfile snippet. after i added the if-mib/ifXtable modules and add --enable-mfw-rewrites to the configure options.

    This is the result of the mrtg cfgmaker by using snmpV2 in the request.

    So after adding the config options, 64bit are again available, but because of the wrong or not defined interface speeds internaly, no interface
    can be queryd correctly and mrtg makes automatic fallback to snmpV1 instead.

    BTW: the binary size increases with 10KB on a EA6700 by the way. i thinks, this is acceptable for a modern router configured as MegaVPN-Build 😉 

    --base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.1 -> 10 Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.1 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.2 -> unknown Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.2 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.3 -> unknown Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.3 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.4 -> unknown Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.4 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.5 -> unknown Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.5 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.6 -> 10 Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.6 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.7 -> 10 Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.7 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.8 -> 10 Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.8 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.9 -> 10 Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.9 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.10 -> 10 Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.10 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.11 -> 10 Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.11 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.12 -> 10 Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.12 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.13 -> unknown Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.13 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.14 -> unknown Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.14 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.15 -> unknown Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.15 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.16 -> unknown Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.16 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.17 -> unknown Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.17 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.18 -> unknown Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.18 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.19 -> unknown Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.19 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1
--base: snmpget snmp@testserver::::2::v4only for ifHighSpeed.21 -> unknown Mb/s
--base: snmpget snmp@testserver::::2::v4only for ifHCInOctets.21 -> unknown
--base: check for HighspeedCounters failed ... Dropping back to V1

    ‌

  5. Not Sure

    You’ll have much better luck looking at RHEL6 sources. They still support Kernel 2.6. That’s where I got the net namespaces code for backporting.

  6. TheHiman reporter

    Here is a makefile patch for more testing.

    64bit counter basicly WORKS now - but all the other issues are still there. It looks like the internal netlink handling is the main reason for getting no longer any interface information out.

    basicly i added 2 more modules: “if-mib/ifXTable,ip-mib/inetNetToMediaTable”
    and added one config option: --enable-mfd-rewrites
    and last but not leased changed the default snmp level from 3 to 2 !

    The change costs 15 KB more space in the binary - but i saw that basicly some other modules can be taken out, because some infos are really not needed.
    configuring snmpV3 option AND disable all crypto options makes no sense as sample. There is no configure option in the webinterface to basicly
    change the bare basics which are needed for a real v3 config with all the certificate and crypto options. So we should disable ALL snmpv3 options for now.

    With the attached patch the mrtg-cfgmaker still produces invalid and incomplete interface-configs, but with the hand tuned config mrtg is again working with 64bit counters.
    Having absolutely no real interface informations any longer is really boring.

    By the way, interfaces such ifbX and all the virtual interfaces which anyway have always 0 counter (lika apsta, etc…) can be hiden. i found one more patch on the openwrt repo for hiding interfaces with 0 counters.

    BTW: On old original advanced tomato from 2017 snmpV2/64bit counters and all interface informations & orders was completly working 🙂

  8. TheHiman reporter

    just reverting was not enough because you added a huge qos module patch, i have to exclude them atm 😉
    will see what changed shortly….

  9. TheHiman reporter

    OK, after successfully reverting and removing of “qos” module there is still no difference, physical interfaces again have no longer any “ifSpeed reported” and are excluded.
    the eth1+eth2 for wlan is still out commented and are placed dynamicly near the end of the list and changes his orders some times.
    i bet, there is a internal problem with the netlink or some other small issue which prevents the snmp daemon to request the corect speed of any interface corretly,
    maybe some issue with mibII, “ethtool” or some other small change which prevents a correct working.

    @pedro: can you plase took a closer look, why physical interfaces eth1 and eth2 no longer have an ifspeed defined, vlans becomes reported with just 10 mbit - but this
    are accountable. The including of the “ifb” and “qos” makes no sense for me - because, the basic snmp config is totaly rudimentary in the webinterface, and basicly
    in a view mode i see not a real benefit to have here bigger qos patches included which makes the dynamic list of extra “Interfaces” just longer.

    As i have chcked, when reverted your patch and the extra “64bit table”-commands are not added, 64bit counters are again working “blindly”. so they are enabled/included by default in newer SNMP. Just snmpv3 as DEFAULT makes no sense - unless you enhance the snmp webinterface in a way to ENABLE them, define user settable keys and all the extra security related things for v3 mode. So you should stick with snmp v2 as default for now.

    Right now my small patch is a badly working solution - and i have to check every time the interface orders for mrtg again when i make the smalest changes later in qos or other parts on the webinterface - then the orders of some interfaces changes again - this is mostly boring for eth1+eth2 , the native wlan interfaces, every time. 😞

  10. pedro repo owner

    If you want to help me with this and you can build your own version of images:

    • git checkout <commit no> - this one with merge master with sdk7 (go back six months or so)
    • build image
    • test it
    • if everything is ok, choose one which is newer
    • build, test again, etc

    to find out which commit breaks this functionality.

    ‌

  12. TheHiman reporter

    Not finaly. actualy i always add the ifxmibs tables as add-on to have the 64bit counters basicly working. the order of interfaces and the reporting of 0 interface speed i have not figured out what the root cause is here….

    I saw this problem on some debian/ubuntu 16.04 when virtualized on KVM or XEN, too, the last weeks. Reason here is the emulated ethernetdriver.
    When you have a real hardware, like e1000, tg3 or a realtek device you always get the correct if-speed. so my best bet is that the brcm is no longer 100% compatible with ethtool like rerquests somewhat…? Maybe an update of some broadcom related things from Asus the last months….

  13. pedro repo owner

    IMO, it’s simple to find what commit causes this problem.

    As I wrote earlier, go to the last working official image (and the corresponding commit). Then go forward for about 30 commits, build image and check if everything works as it should. If so - again go forward, etc. If not - go backward for about 15 commits, and check again.

    This way you’ll find commit causing problems.

  14. Vladislav Staroselskiy

    @TheHiman @pedro - gentlemen, it would be nice if you resolve this issue, SNMP certainly needs IF-MIB::ifXTable with 64-bit counters.
    32-bit counters (specifically - ifInOctets/ifOutOctets) are rolled over after 4G bytes traffic which completely sets off monitoring

    ‌

  17. Log in to comment
Assignee
–
Type
enhancement
Priority
minor
Status
closed
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!