PPTP server with CTF enabled causes kernel memory leak?
Issue #147
new
Hi!
FW: 2021.5 (2021.08.14) freshtomato-R7000-ARM_NG-2021.5-AIO-64K.trx
Model: netgear R7000
I just moved from AdvancedTomato to FreshTomato a couple days ago. I’ve made sure to check the “Erase all data in NVRAM” when flashing the FreshTomato firmware and re-did all my configuration by hand. Everything seemed to work fine for a few hours, but then at some time, I lost wifi and couldn’t connect anymore and no traffic was going through. I power-cycled the router, and it started working again, until the same thing happened. Over the course of 2 days, I had to power-cycle the router 5 of 6 times.
I setup a remote syslog server to get some details, and it looks like memory exhaustion.
I found traces like this:
2021-08-16T08:17:17 kernel "dnsmasq: page allocation failure. order:0, mode:0x4021"
2021-08-16T08:17:17 kernel [<c0042000>] (unwind_backtrace+0x0/0xf8) from [<c009d034>] (__alloc_pages_nodemask+0x61c/0x698)
2021-08-16T08:17:17 kernel [<c009d034>] (__alloc_pages_nodemask+0x61c/0x698) from [<c0397a24>] (__slab_alloc+0x794/0x814)
2021-08-16T08:17:17 kernel [<c0397a24>] (__slab_alloc+0x794/0x814) from [<c0397c58>] (__kmalloc_track_caller+0xf8/0x108)
2021-08-16T08:17:17 kernel [<c0397c58>] (__kmalloc_track_caller+0xf8/0x108) from [<c0399c80>] (__alloc_skb+0x54/0x104)
2021-08-16T08:17:17 kernel [<c0399c80>] (__alloc_skb+0x54/0x104) from [<c01d9a10>] (osl_alloc_skb+0x18/0x30)
2021-08-16T08:17:17 kernel [<c01d9a10>] (osl_alloc_skb+0x18/0x30) from [<c03a8d90>] (osl_pktget+0x4c/0x154)
2021-08-16T08:17:18 kernel [<c03a8d90>] (osl_pktget+0x4c/0x154) from [<bf362dfc>] (wlc_hdr_proc+0x54/0x264 [wl])
2021-08-16T08:17:18 kernel unwind: Index not found bf362dfc
2021-08-16T08:17:18 kernel Mem-info:
2021-08-16T08:17:18 kernel DMA per-cpu:
2021-08-16T08:17:18 kernel "CPU 0: hi: 42, btch: 7 usd: 0"
2021-08-16T08:17:18 kernel "CPU 1: hi: 42, btch: 7 usd: 37"
2021-08-16T08:17:18 kernel Normal per-cpu:
2021-08-16T08:17:18 kernel "CPU 0: hi: 42, btch: 7 usd: 42"
2021-08-16T08:17:18 kernel "CPU 1: hi: 42, btch: 7 usd: 35"
2021-08-16T08:17:18 kernel active_anon:2174 inactive_anon:2187 isolated_anon:0
2021-08-16T08:17:18 kernel active_file:1622 inactive_file:1723 isolated_file:0
2021-08-16T08:17:18 kernel unevictable:0 dirty:0 writeback:0 unstable:0
2021-08-16T08:17:18 kernel free:19087 slab_reclaimable:251 slab_unreclaimable:33615
2021-08-16T08:17:18 kernel mapped:978 shmem:699 pagetables:92 bounce:0
2021-08-16T08:17:18 kernel DMA free:4136kB min:11020kB low:13772kB high:16528kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:130048kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:122640kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
2021-08-16T08:17:18 kernel lowmem_reserve[]: 0 109 109
2021-08-16T08:17:18 kernel Normal free:72212kB min:9456kB low:11820kB high:14184kB active_anon:8696kB inactive_anon:8748kB active_file:6488kB inactive_file:6892kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:111616kB mlocked:0kB dirty:0kB writeback:0kB mapped:3912kB shmem:2796kB slab_reclaimable:1004kB slab_unreclaimable:11820kB kernel_stack:416kB pagetables:368kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
2021-08-16T08:17:18 kernel lowmem_reserve[]: 0 0 0
2021-08-16T08:17:18 kernel DMA: 0*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 4152kB
2021-08-16T08:17:18 kernel Normal: 517*4kB 222*8kB 309*16kB 218*32kB 62*64kB 12*128kB 9*256kB 3*512kB 4*1024kB 3*2048kB 9*4096kB = 72212kB
2021-08-16T08:17:18 kernel 4044 total pagecache pages
2021-08-16T08:17:18 kernel 0 pages in swap cache
2021-08-16T08:17:18 kernel "Swap cache stats: add 0, delete 0, find 0/0"
2021-08-16T08:17:18 kernel Free swap = 0kB
2021-08-16T08:17:18 kernel Total swap = 0kB
2021-08-16T08:17:18 kernel 65536 pages of RAM
2021-08-16T08:17:18 kernel 19333 free pages
2021-08-16T08:17:18 kernel 1642 reserved pages
2021-08-16T08:17:18 kernel 6502 slab pages
2021-08-16T08:17:18 kernel 4507 pages shared
2021-08-16T08:17:18 kernel 0 pages swap cached
2021-08-16T08:17:18 kernel SLUB: Unable to allocate memory on node -1 (gfp=0x21)
2021-08-16T08:17:18 kernel " cache: kmalloc_dma-1024, object size: 1024, buffer size: 1024, default order: 2, min order: 0"
2021-08-16T08:17:18 kernel " node 0: slabs: 5, objs: 80, free: 0"
or
2021-08-16T08:17:23 kernel __alloc_pages_slowpath: 853 callbacks suppressed
2021-08-16T08:17:23 kernel "swapper: page allocation failure. order:0, mode:0x4021"
2021-08-16T08:17:23 kernel [<c0042000>] (unwind_backtrace+0x0/0xf8) from [<c009d034>] (__alloc_pages_nodemask+0x61c/0x698)
2021-08-16T08:17:23 kernel [<c009d034>] (__alloc_pages_nodemask+0x61c/0x698) from [<c0397a24>] (__slab_alloc+0x794/0x814)
2021-08-16T08:17:23 kernel [<c0397a24>] (__slab_alloc+0x794/0x814) from [<c0397c58>] (__kmalloc_track_caller+0xf8/0x108)
2021-08-16T08:17:23 kernel [<c0397c58>] (__kmalloc_track_caller+0xf8/0x108) from [<c0399c80>] (__alloc_skb+0x54/0x104)
2021-08-16T08:17:23 kernel [<c0399c80>] (__alloc_skb+0x54/0x104) from [<c01d9a10>] (osl_alloc_skb+0x18/0x30)
2021-08-16T08:17:23 kernel [<c01d9a10>] (osl_alloc_skb+0x18/0x30) from [<c01d9cc8>] (osl_ctfpool_add+0x44/0x128)
2021-08-16T08:17:23 kernel [<c01d9cc8>] (osl_ctfpool_add+0x44/0x128) from [<bf361b18>] (wl_sendup+0x200/0x4e4 [wl])
2021-08-16T08:17:23 kernel unwind: Index not found bf361b18
2021-08-16T08:17:23 kernel Mem-info:
2021-08-16T08:17:23 kernel DMA per-cpu:
2021-08-16T08:17:23 kernel "CPU 0: hi: 42, btch: 7 usd: 2"
2021-08-16T08:17:23 kernel "CPU 1: hi: 42, btch: 7 usd: 45"
2021-08-16T08:17:23 kernel Normal per-cpu:
2021-08-16T08:17:23 kernel "CPU 0: hi: 42, btch: 7 usd: 41"
2021-08-16T08:17:23 kernel "CPU 1: hi: 42, btch: 7 usd: 37"
2021-08-16T08:17:23 kernel active_anon:2173 inactive_anon:2178 isolated_anon:0
2021-08-16T08:17:23 kernel active_file:1622 inactive_file:1723 isolated_file:0
2021-08-16T08:17:23 kernel unevictable:0 dirty:0 writeback:0 unstable:0
2021-08-16T08:17:23 kernel free:19096 slab_reclaimable:251 slab_unreclaimable:33605
2021-08-16T08:17:23 kernel mapped:978 shmem:691 pagetables:92 bounce:0
2021-08-16T08:17:23 kernel DMA free:4144kB min:11020kB low:13772kB high:16528kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:130048kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:122588kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
2021-08-16T08:17:23 kernel lowmem_reserve[]: 0 109 109
2021-08-16T08:17:23 kernel Normal free:72240kB min:9456kB low:11820kB high:14184kB active_anon:8692kB inactive_anon:8712kB active_file:6488kB inactive_file:6892kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:111616kB mlocked:0kB dirty:0kB writeback:0kB mapped:3912kB shmem:2764kB slab_reclaimable:1004kB slab_unreclaimable:11832kB kernel_stack:416kB pagetables:368kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
2021-08-16T08:17:23 kernel lowmem_reserve[]: 0 0 0
2021-08-16T08:17:23 kernel DMA: 2*4kB 1*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 4144kB
2021-08-16T08:17:23 kernel Normal: 524*4kB 222*8kB 309*16kB 218*32kB 62*64kB 12*128kB 9*256kB 3*512kB 4*1024kB 3*2048kB 9*4096kB = 72240kB
2021-08-16T08:17:23 kernel 4036 total pagecache pages
2021-08-16T08:17:23 kernel 0 pages in swap cache
2021-08-16T08:17:23 kernel "Swap cache stats: add 0, delete 0, find 0/0"
2021-08-16T08:17:23 kernel Free swap = 0kB
2021-08-16T08:17:23 kernel Total swap = 0kB
2021-08-16T08:17:23 kernel 65536 pages of RAM
2021-08-16T08:17:23 kernel 19349 free pages
2021-08-16T08:17:23 kernel 1642 reserved pages
2021-08-16T08:17:23 kernel 6505 slab pages
2021-08-16T08:17:23 kernel 4509 pages shared
2021-08-16T08:17:23 kernel 0 pages swap cached
2021-08-16T08:17:23 kernel SLUB: Unable to allocate memory on node -1 (gfp=0x21)
2021-08-16T08:17:23 kernel " cache: kmalloc_dma-4096, object size: 4096, buffer size: 4096, default order: 3, min order: 0"
2021-08-16T08:17:23 kernel " node 0: slabs: 3829, objs: 30604, free: 0"
and
2021-08-16T08:17:23 kernel osl_ctfpool_add: skb alloc of len 2256 failed
2021-08-16T08:17:23 kernel "swapper: page allocation failure. order:0, mode:0x4021"
2021-08-16T08:17:23 kernel [<c0042000>] (unwind_backtrace+0x0/0xf8) from [<c009d034>] (__alloc_pages_nodemask+0x61c/0x698)
2021-08-16T08:17:23 kernel [<c009d034>] (__alloc_pages_nodemask+0x61c/0x698) from [<c0397a24>] (__slab_alloc+0x794/0x814)
2021-08-16T08:17:23 kernel [<c0397a24>] (__slab_alloc+0x794/0x814) from [<c0397c58>] (__kmalloc_track_caller+0xf8/0x108)
2021-08-16T08:17:23 kernel [<c0397c58>] (__kmalloc_track_caller+0xf8/0x108) from [<c0399c80>] (__alloc_skb+0x54/0x104)
2021-08-16T08:17:23 kernel [<c0399c80>] (__alloc_skb+0x54/0x104) from [<c01d9a10>] (osl_alloc_skb+0x18/0x30)
2021-08-16T08:17:23 kernel [<c01d9a10>] (osl_alloc_skb+0x18/0x30) from [<c03a8d90>] (osl_pktget+0x4c/0x154)
2021-08-16T08:17:23 kernel [<c03a8d90>] (osl_pktget+0x4c/0x154) from [<c03a830c>] (_dma_rxfill+0x174/0x3d4)
2021-08-16T08:17:23 kernel [<c03a830c>] (_dma_rxfill+0x174/0x3d4) from [<bf0289bc>] (chiprx+0xf0/0x140 [et])
2021-08-16T08:17:23 kernel unwind: Index not found bf0289bc
2021-08-16T08:17:23 kernel Mem-info:
2021-08-16T08:17:23 kernel DMA per-cpu:
2021-08-16T08:17:23 kernel "CPU 0: hi: 42, btch: 7 usd: 7"
2021-08-16T08:17:23 kernel "CPU 1: hi: 42, btch: 7 usd: 39"
2021-08-16T08:17:23 kernel Normal per-cpu:
2021-08-16T08:17:23 kernel "CPU 0: hi: 42, btch: 7 usd: 41"
2021-08-16T08:17:23 kernel "CPU 1: hi: 42, btch: 7 usd: 37"
2021-08-16T08:17:23 kernel active_anon:2173 inactive_anon:2178 isolated_anon:0
2021-08-16T08:17:23 kernel active_file:1622 inactive_file:1723 isolated_file:0
2021-08-16T08:17:23 kernel unevictable:0 dirty:0 writeback:0 unstable:0
2021-08-16T08:17:23 kernel free:19096 slab_reclaimable:251 slab_unreclaimable:33605
2021-08-16T08:17:23 kernel mapped:978 shmem:691 pagetables:92 bounce:0
2021-08-16T08:17:23 kernel DMA free:4144kB min:11020kB low:13772kB high:16528kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:130048kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:122588kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
2021-08-16T08:17:23 kernel lowmem_reserve[]: 0 109 109
2021-08-16T08:17:23 kernel Normal free:72240kB min:9456kB low:11820kB high:14184kB active_anon:8692kB inactive_anon:8712kB active_file:6488kB inactive_file:6892kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:111616kB mlocked:0kB dirty:0kB writeback:0kB mapped:3912kB shmem:2764kB slab_reclaimable:1004kB slab_unreclaimable:11832kB kernel_stack:416kB pagetables:368kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
2021-08-16T08:17:23 kernel lowmem_reserve[]: 0 0 0
2021-08-16T08:17:23 kernel DMA: 0*4kB 2*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 4144kB
2021-08-16T08:17:23 kernel Normal: 524*4kB 222*8kB 309*16kB 218*32kB 62*64kB 12*128kB 9*256kB 3*512kB 4*1024kB 3*2048kB 9*4096kB = 72240kB
2021-08-16T08:17:23 kernel 4036 total pagecache pages
2021-08-16T08:17:23 kernel 0 pages in swap cache
2021-08-16T08:17:23 kernel "Swap cache stats: add 0, delete 0, find 0/0"
2021-08-16T08:17:23 kernel Free swap = 0kB
2021-08-16T08:17:23 kernel Total swap = 0kB
Right after boot, the router uses about 40MB of RAM, but when the problem happens, it quickly jumps to over 160MB and the system freezes from there. I’ve checked the list of processes when the problem starts occurring, but couldn't determine the culprit: the sum of individual processes' memory usage doesn’t account for all the memory used, which makes think that this is happening at the kernel level.
I’m not sure if it’s related, but I have CTF enabled.
I had to go back to AdvancedTomato for now, to get a stable connection again, but I thought I would report this, in case it could help.
Comments (36)
-
-
reporter
Thanks for the pointers!
can you please provide more infos about your router setup. Connections type, wifi setup, services you use, scripts…
R7000 with a Gigabit WAN connection, 2 wifi radios enabled (2.4 and 5GHz, same SSID with band steering), a guest wifi piggy-backing on the 2.4 interface, pretty default wireless options. No services enabled except 2 OpenVPN clients (each using their own Routing Policy) and a PPTP server. No stats collection (bandwidth monitoring disabled, as it’s pretty much useless with CTF anyway), no web monitoring, no QOS, no Web services/MySQL, no Bittorrent client, no Tor, no Captive portal, no USB, no FTP/CIFS/DLNA.
I’ll test with CTF off, as well as the other ctf binary over the weekend and send the results back.
-
reporter
Just a quick update: I’ve re-flashed FT 2021,5, and I haven’t got to a point where the router crashes with memory allocation failures, but there’s definitely higher-than-expected memory usage. Here’s the full list of processes running:
# top -b -n1 Mem: 122628K used, 132948K free, 2748K shrd, 6704K buff, 24240K cached CPU: 0.0% usr 0.0% sys 0.0% nic 100% idle 0.0% io 0.0% irq 0.0% sirq Load average: 0.35 0.36 0.39 1/54 11836 PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND 5082 1 nobody S 12396 4.8 0 0.0 dnsmasq -c 4096 --log-async 3774 1 root S 4512 1.7 1 0.0 /etc/openvpn/vpnclient1 --cd /etc/openvpn/client1 --config config.ovpn 3795 1 root S 4408 1.7 0 0.0 /etc/openvpn/vpnclient2 --cd /etc/openvpn/client2 --config config.ovpn 3697 1 root S 3716 1.4 1 0.0 httpd 3423 1 root S 1704 0.6 1 0.0 nas 1 0 root S 1676 0.6 0 0.0 /sbin/preinit 3497 1 root S 1672 0.6 1 0.0 blink eth2 5g 10 8192 769 1 root S 1664 0.6 0 0.0 buttons 772 1 root S 1600 0.6 0 0.0 console 3431 1 root S 1428 0.5 1 0.0 crond 30397 30396 root S 1412 0.5 1 0.0 -sh 4029 1 root S 1412 0.5 0 0.0 udhcpc -i vlan2 -b -s dhcpc-event -H rtr -m -S -p /var/run/udhcpc-wan.pid 3325 1 root S 1408 0.5 1 0.0 syslogd -D -R nas:514 -L -s 50 -O /var/log/messages -b 1 774 772 root S 1408 0.5 0 0.0 /bin/sh 11836 30397 root R 1408 0.5 1 0.0 top -b -n1 3327 1 root S 1400 0.5 0 0.0 klogd 3435 1 root S 1376 0.5 1 0.0 rstats 3466 1 root S 1204 0.4 1 0.0 /usr/sbin/bsd 30396 950 root S 1112 0.4 1 0.0 dropbear -p 22 -a 950 1 root S 1092 0.4 0 0.0 dropbear -p 22 -a 3421 1 root S 1048 0.4 1 0.0 eapd 4024 1 root S 832 0.3 0 0.0 miniupnpd -f /etc/upnp/config 3436 3434 root S 672 0.2 0 0.0 /usr/sbin/bcrelay -i disable -o ppp[0-9].* -n 732 1 root S 656 0.2 1 0.0 hotplug2 --persistent --no-coldplug 3434 1 root S 656 0.2 0 0.0 pptpd -c /etc/vpn/pptpd.conf -o /etc/vpn/pptpd_options -C 50 3 2 root SW 0 0.0 0 0.0 [ksoftirqd/0] 9 2 root SW 0 0.0 1 0.0 [ksoftirqd/1] 264 2 root SW 0 0.0 1 0.0 [mtdblock3] 299 2 root SW 0 0.0 0 0.0 [kworker/0:1] 781 2 root SWN 0 0.0 0 0.0 [jffs2_gcd_mtd6] 7 2 root SW 0 0.0 1 0.0 [migration/1] 6 2 root SW 0 0.0 0 0.0 [migration/0] 59 2 root SW 0 0.0 0 0.0 [sync_supers] 300 2 root SW 0 0.0 1 0.0 [kworker/1:1] 5 2 root SW 0 0.0 0 0.0 [kworker/u:0] 173 2 root SW< 0 0.0 1 0.0 [crypto] 117 2 root SW 0 0.0 1 0.0 [kswapd0] 259 2 root SW 0 0.0 1 0.0 [mtdblock2] 303 2 root SW 0 0.0 1 0.0 [mtdblock5] 62 2 root SW< 0 0.0 1 0.0 [kblockd] 313 2 root SW 0 0.0 1 0.0 [mtdblock7] 2 0 root SW 0 0.0 0 0.0 [kthreadd] 165 2 root SW 0 0.0 1 0.0 [fsnotify_mark] 4 2 root SW 0 0.0 0 0.0 [kworker/0:0] 249 2 root SW 0 0.0 1 0.0 [mtdblock0] 254 2 root SW 0 0.0 1 0.0 [mtdblock1] 802 2 root SW 0 0.0 1 0.0 [khubd] 8 2 root SW 0 0.0 1 0.0 [kworker/1:0] 269 2 root SW 0 0.0 1 0.0 [mtdblock4] 10 2 root SW< 0 0.0 1 0.0 [khelper] 11 2 root SW 0 0.0 1 0.0 [kworker/u:1] 14 2 root SW< 0 0.0 1 0.0 [netns] 308 2 root SW 0 0.0 1 0.0 [mtdblock6] 61 2 root SW 0 0.0 1 0.0 [bdi-default]Taken all together, they use maybe 40MB of memory, while the system reports about 100MB used. So the difference is likely taken by the kernel, but I’m not sure how to pinpoint that more precisely. Global usage continue to grow overt time, though (that capture is with ~1h uptime).
I’ll go ahead and flash the test images, to see if they behave the same way.
-
reporter
So I tested both test images (test_v001 and test_v002) and they both leak the same way. See this top output with test_v002:
Mem: 177168K used, 78408K free, 2736K shrd, 6596K buff, 12308K cached CPU: 4.5% usr 4.5% sys 0.0% nic 86.3% idle 0.0% io 0.0% irq 4.5% sirq Load average: 0.32 0.28 0.28 1/57 17816 PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND 2838 1 nobody S 12388 4.8 0 4.5 dnsmasq -c 4096 --log-async 17816 15652 root R 1408 0.5 0 4.5 top -n 1 -b 1364 1 root S 4468 1.7 1 0.0 /etc/openvpn/vpnclient1 --cd /etc/openvpn/client1 --config config.ovpn 1386 1 root S 4408 1.7 0 0.0 /etc/openvpn/vpnclient2 --cd /etc/openvpn/client2 --config config.ovpn 1186 1 root S 3716 1.4 1 0.0 httpd 947 1 root S 1704 0.6 1 0.0 nas 1 0 root S 1676 0.6 1 0.0 /sbin/preinit 1063 1 root S 1672 0.6 1 0.0 blink eth2 5g 10 8192 769 1 root S 1664 0.6 1 0.0 buttons 772 1 root S 1600 0.6 0 0.0 console 958 1 root S 1428 0.5 0 0.0 crond 1172 1 root S 1424 0.5 0 0.0 /usr/sbin/ntpd -l -t -S /sbin/ntpd_synced -d6 3868 3867 root S 1424 0.5 1 0.0 -sh 15652 15538 root S 1412 0.5 0 0.0 -sh 1345 1 root S 1412 0.5 1 0.0 udhcpc -i vlan2 -b -s dhcpc-event -H rtr -m -S -p /var/run/udhcpc-wan.pid 782 1 root S 1408 0.5 1 0.0 syslogd -D -R nas:514 -L -s 50 -O /var/log/messages -b 1 774 772 root S 1408 0.5 0 0.0 /bin/sh 784 1 root S 1400 0.5 0 0.0 klogd 961 1 root S 1376 0.5 1 0.0 rstats 1009 1 root S 1192 0.4 1 0.0 /usr/sbin/bsd 15538 943 root S 1112 0.4 0 0.0 dropbear -p 22 -a 3867 943 root S 1112 0.4 1 0.0 dropbear -p 22 -a 943 1 root S 1092 0.4 0 0.0 dropbear -p 22 -a 945 1 root S 1048 0.4 1 0.0 eapd 1338 1 root S 832 0.3 1 0.0 miniupnpd -f /etc/upnp/config 965 963 root S 672 0.2 0 0.0 /usr/sbin/bcrelay -i disable -o ppp[0-9].* -n 732 1 root S 656 0.2 0 0.0 hotplug2 --persistent --no-coldplug 963 1 root S 656 0.2 1 0.0 pptpd -c /etc/vpn/pptpd.conf -o /etc/vpn/pptpd_options -C 50 3 2 root SW 0 0.0 0 0.0 [ksoftirqd/0] 9 2 root SW 0 0.0 1 0.0 [ksoftirqd/1] 264 2 root SW 0 0.0 1 0.0 [mtdblock3] 6204 2 root SWN 0 0.0 1 0.0 [jffs2_gcd_mtd6] 299 2 root SW 0 0.0 0 0.0 [kworker/0:1] 6 2 root SW 0 0.0 0 0.0 [migration/0] 117 2 root SW 0 0.0 0 0.0 [kswapd0] 7 2 root SW 0 0.0 1 0.0 [migration/1] 59 2 root SW 0 0.0 0 0.0 [sync_supers] 300 2 root SW 0 0.0 1 0.0 [kworker/1:1] 5 2 root SW 0 0.0 1 0.0 [kworker/u:0] 173 2 root SW< 0 0.0 1 0.0 [crypto] 795 2 root SW 0 0.0 1 0.0 [khubd] 313 2 root SW 0 0.0 1 0.0 [mtdblock7] 2 0 root SW 0 0.0 1 0.0 [kthreadd] 303 2 root SW 0 0.0 1 0.0 [mtdblock5] 249 2 root SW 0 0.0 1 0.0 [mtdblock0] 254 2 root SW 0 0.0 1 0.0 [mtdblock1] 259 2 root SW 0 0.0 1 0.0 [mtdblock2] 4 2 root SW 0 0.0 0 0.0 [kworker/0:0] 269 2 root SW 0 0.0 1 0.0 [mtdblock4] 10 2 root SW< 0 0.0 1 0.0 [khelper] 11 2 root SW 0 0.0 1 0.0 [kworker/u:1] 8 2 root SW 0 0.0 1 0.0 [kworker/1:0] 308 2 root SW 0 0.0 1 0.0 [mtdblock6] 61 2 root SW 0 0.0 1 0.0 [bdi-default] 62 2 root SW< 0 0.0 1 0.0 [kblockd] 14 2 root SW< 0 0.0 1 0.0 [netns] 165 2 root SW 0 0.0 1 0.0 [fsnotify_mark]Sum of VSZ is 57,180K, mem used is 177,168K after 57 min of uptime.
-
reporter
With CTF disabled on the default image (stock 2021.5), memory usage is stable at under 40MB after 1h.
But of course, the router can only reach ~250-300Mbps on a 1Gbps connection, so disabling CTF is not a great solution
I also tried to disable Jumbo Frames, but that doesn’t seem to make much difference.
So, to summarize:
* 2021.5 (stock, or test_v001 or test_v002) w/ CTF enabled: memory usage grows over time, until memory allocation errors make the router unusable
* 2021.5 w/ CTF disabled and jumbo frames enabled: memory usage is stable
* 2021.5 w/ CTF enabled and jumbo frames disabled: memory usage grows over time (albeit more slowly)
* AdvancedTomato v3.5-140 w/ CTF enabled and jumbo frames enabled: memory usage is stableHope this can help pinpoint the problem and hopefully fix it.
Thanks! -
mhh ok. I have no problems with CTF on my side. I also asked a few more people. Can you crash the router with both test_v001 or test_v002 images ?
What functions/features else do you use? First time you mentioned Jumbo frames?
→ Jumbo Frames do not work on Advanced Tomato (or latest shibby 140) except you adjusted something. How do you check if its working ?
-
reporter
mhh ok. I have no problems with CTF on my side.
What’s your memory usage? Especially after a day or 2 of runtime? Is it stable over time, or is it increasing?
Can you crash the router with both test_v001 or test_v002 images ?
I haven’t waited long enough, but given they were both displaying the same pattern, it’s quite certain yes: memory usage started with both at ~40MB right after boot, and then steadily increased over time, to reach ~100MB after one hour, without starting any additional service.
What functions/features else do you use?
You can see all the processes that run on the router from the
topcapture I posted above. I can also share a sanitized nvram dump if it could help. Any particular variable of interest?First time you mentioned Jumbo frames?
Yeah, I didn’t think it would make much of a difference, and just thought about it because it’s on the same page as the CTF option.
→ Jumbo Frames do not work on Advanced Tomato (or latest shibby 140) except you adjusted something. How do you check if its working ?
I’m running stock Advanced Tomato from here: https://advancedtomato.com/downloads/router/r7000 (R7000-AT-ARM-3.5-140-AIO-64K.trx) without any modification. I don’t know how to check it is really working, but what I meant is that I’ve had the “Jumbo Frames” box checked in Advanced Tomato for the last 3 years.
Thanks!
-
reporter
Here’s an example of the memory leak, with test_v001: memory usage goes from 48k to 56k in just 10mn, without any notable network activity:
root@rtr:/tmp/home/root# while true; do printf "%s : " "$(date)"; free | awk '/Mem:/ {print $3}'; sleep 10; done Sun Aug 22 15:51:34 PDT 2021 : 48088 Sun Aug 22 15:51:44 PDT 2021 : 48080 Sun Aug 22 15:51:54 PDT 2021 : 48840 Sun Aug 22 15:52:04 PDT 2021 : 48872 Sun Aug 22 15:52:14 PDT 2021 : 48796 Sun Aug 22 15:52:24 PDT 2021 : 48984 Sun Aug 22 15:52:34 PDT 2021 : 49036 Sun Aug 22 15:52:44 PDT 2021 : 49020 Sun Aug 22 15:52:54 PDT 2021 : 49508 Sun Aug 22 15:53:04 PDT 2021 : 49600 Sun Aug 22 15:53:14 PDT 2021 : 49696 Sun Aug 22 15:53:24 PDT 2021 : 49696 Sun Aug 22 15:53:34 PDT 2021 : 49684 Sun Aug 22 15:53:45 PDT 2021 : 49684 Sun Aug 22 15:53:55 PDT 2021 : 50492 Sun Aug 22 15:54:05 PDT 2021 : 50492 Sun Aug 22 15:54:15 PDT 2021 : 50972 Sun Aug 22 15:54:25 PDT 2021 : 50908 Sun Aug 22 15:54:35 PDT 2021 : 50904 Sun Aug 22 15:54:45 PDT 2021 : 50876 Sun Aug 22 15:54:55 PDT 2021 : 51380 Sun Aug 22 15:55:05 PDT 2021 : 51396 Sun Aug 22 15:55:15 PDT 2021 : 51460 Sun Aug 22 15:55:25 PDT 2021 : 51384 Sun Aug 22 15:55:35 PDT 2021 : 51396 Sun Aug 22 15:55:45 PDT 2021 : 51464 Sun Aug 22 15:55:55 PDT 2021 : 52036 Sun Aug 22 15:56:05 PDT 2021 : 52100 Sun Aug 22 15:56:15 PDT 2021 : 52180 Sun Aug 22 15:56:25 PDT 2021 : 52320 Sun Aug 22 15:56:35 PDT 2021 : 52324 Sun Aug 22 15:56:45 PDT 2021 : 52336 Sun Aug 22 15:56:55 PDT 2021 : 52832 Sun Aug 22 15:57:05 PDT 2021 : 52816 Sun Aug 22 15:57:15 PDT 2021 : 52804 Sun Aug 22 15:57:25 PDT 2021 : 52924 Sun Aug 22 15:57:35 PDT 2021 : 52928 Sun Aug 22 15:57:45 PDT 2021 : 52976 Sun Aug 22 15:57:55 PDT 2021 : 53660 Sun Aug 22 15:58:05 PDT 2021 : 53692 Sun Aug 22 15:58:15 PDT 2021 : 53692 Sun Aug 22 15:58:25 PDT 2021 : 53876 Sun Aug 22 15:58:35 PDT 2021 : 53804 Sun Aug 22 15:58:45 PDT 2021 : 53880 Sun Aug 22 15:58:55 PDT 2021 : 54328 Sun Aug 22 15:59:05 PDT 2021 : 54376 Sun Aug 22 15:59:15 PDT 2021 : 54420 Sun Aug 22 15:59:25 PDT 2021 : 54516 Sun Aug 22 15:59:35 PDT 2021 : 54488 Sun Aug 22 15:59:45 PDT 2021 : 54504 Sun Aug 22 15:59:55 PDT 2021 : 55220 Sun Aug 22 16:00:05 PDT 2021 : 55244 Sun Aug 22 16:00:15 PDT 2021 : 55220 Sun Aug 22 16:00:25 PDT 2021 : 55376 Sun Aug 22 16:00:35 PDT 2021 : 55444 Sun Aug 22 16:00:45 PDT 2021 : 55440 Sun Aug 22 16:00:55 PDT 2021 : 55952 Sun Aug 22 16:01:05 PDT 2021 : 55952 Sun Aug 22 16:01:15 PDT 2021 : 56112 Sun Aug 22 16:01:25 PDT 2021 : 56076 ^C root@rtr:/tmp/home/root# grep AIO /etc/motd FreshTomato 2021.5 -15082021-ctf_test_v001 K26ARM USB AIO-64Kand here’s the comparison between global memory usage, and the sum of memory usage of all user-land processes:
root@rtr:/tmp/home/root# while true; do printf "%s | total used: %s | sum apps : %s\n" "$(date)" "$(free | awk '/Mem:/ {print $3}')" "$(top -b -n1 | awk '/root|nobody/ {s+=$5} END {print s}')"; sleep 10; done Sun Aug 22 16:17:08 PDT 2021 | total used: 71260 | sum apps : 55952 Sun Aug 22 16:17:18 PDT 2021 | total used: 71392 | sum apps : 55952 Sun Aug 22 16:17:28 PDT 2021 | total used: 71304 | sum apps : 55952 Sun Aug 22 16:17:38 PDT 2021 | total used: 71772 | sum apps : 55952 Sun Aug 22 16:17:48 PDT 2021 | total used: 72008 | sum apps : 55952 Sun Aug 22 16:17:59 PDT 2021 | total used: 73044 | sum apps : 55976 Sun Aug 22 16:18:09 PDT 2021 | total used: 73044 | sum apps : 55976 Sun Aug 22 16:18:19 PDT 2021 | total used: 73044 | sum apps : 55976 Sun Aug 22 16:18:29 PDT 2021 | total used: 73152 | sum apps : 55976 Sun Aug 22 16:18:39 PDT 2021 | total used: 73204 | sum apps : 55976 Sun Aug 22 16:18:49 PDT 2021 | total used: 73696 | sum apps : 55976 Sun Aug 22 16:18:59 PDT 2021 | total used: 73692 | sum apps : 55976 Sun Aug 22 16:19:10 PDT 2021 | total used: 73696 | sum apps : 55976 Sun Aug 22 16:19:20 PDT 2021 | total used: 73868 | sum apps : 55976 Sun Aug 22 16:19:30 PDT 2021 | total used: 73820 | sum apps : 55976 Sun Aug 22 16:19:40 PDT 2021 | total used: 74164 | sum apps : 55976 Sun Aug 22 16:19:50 PDT 2021 | total used: 74496 | sum apps : 55976 Sun Aug 22 16:20:00 PDT 2021 | total used: 74616 | sum apps : 55976 Sun Aug 22 16:20:10 PDT 2021 | total used: 74816 | sum apps : 55976 Sun Aug 22 16:20:20 PDT 2021 | total used: 74812 | sum apps : 55976 Sun Aug 22 16:20:31 PDT 2021 | total used: 74692 | sum apps : 55976 Sun Aug 22 16:20:41 PDT 2021 | total used: 74784 | sum apps : 55976 Sun Aug 22 16:20:51 PDT 2021 | total used: 75296 | sum apps : 55976 Sun Aug 22 16:21:01 PDT 2021 | total used: 75276 | sum apps : 55976 Sun Aug 22 16:21:11 PDT 2021 | total used: 75232 | sum apps : 55976 Sun Aug 22 16:21:21 PDT 2021 | total used: 75264 | sum apps : 55976 Sun Aug 22 16:21:31 PDT 2021 | total used: 75380 | sum apps : 55976 Sun Aug 22 16:21:41 PDT 2021 | total used: 75504 | sum apps : 55980 Sun Aug 22 16:21:52 PDT 2021 | total used: 76264 | sum apps : 55980 Sun Aug 22 16:22:02 PDT 2021 | total used: 76180 | sum apps : 55980 Sun Aug 22 16:22:12 PDT 2021 | total used: 76212 | sum apps : 55980You can see that the user-land applications memory footprint stays globally stable, while the overall memory usage keeps climbing. Which likely indicates that the kernel is the source of usage increase.
-
reporter
Here are the exact same measurements with CTF disabled (still used test_v001)
root@rtr:/tmp/home/root# while true; do printf "%s | total used: %s | sum apps : %s\n" "$(date)" "$(free | awk '/Mem:/ {print $3}')" "$(top -b -n1 | awk '/root|nobody/ {s+=$5} END {print s}')"; sleep 10; done Sun Aug 22 16:56:23 PDT 2021 | total used: 36360 | sum apps : 55944 Sun Aug 22 16:56:33 PDT 2021 | total used: 36304 | sum apps : 55944 Sun Aug 22 16:56:43 PDT 2021 | total used: 36364 | sum apps : 55944 Sun Aug 22 16:56:53 PDT 2021 | total used: 36364 | sum apps : 55944 Sun Aug 22 16:57:04 PDT 2021 | total used: 36364 | sum apps : 55968 Sun Aug 22 16:57:14 PDT 2021 | total used: 36392 | sum apps : 55968 Sun Aug 22 16:57:24 PDT 2021 | total used: 36420 | sum apps : 55968 Sun Aug 22 16:57:34 PDT 2021 | total used: 36364 | sum apps : 55968 Sun Aug 22 16:57:44 PDT 2021 | total used: 36392 | sum apps : 55968 Sun Aug 22 16:57:54 PDT 2021 | total used: 36468 | sum apps : 55968 Sun Aug 22 16:58:04 PDT 2021 | total used: 36408 | sum apps : 55968 Sun Aug 22 16:58:14 PDT 2021 | total used: 36364 | sum apps : 55968 Sun Aug 22 16:58:25 PDT 2021 | total used: 36392 | sum apps : 55968 Sun Aug 22 16:58:35 PDT 2021 | total used: 36392 | sum apps : 55968 Sun Aug 22 16:58:45 PDT 2021 | total used: 36436 | sum apps : 55968 Sun Aug 22 16:58:55 PDT 2021 | total used: 36364 | sum apps : 55968 Sun Aug 22 16:59:05 PDT 2021 | total used: 36408 | sum apps : 55968 Sun Aug 22 16:59:15 PDT 2021 | total used: 36392 | sum apps : 55968 Sun Aug 22 16:59:25 PDT 2021 | total used: 36420 | sum apps : 55968 Sun Aug 22 16:59:36 PDT 2021 | total used: 36424 | sum apps : 55968 Sun Aug 22 16:59:46 PDT 2021 | total used: 36408 | sum apps : 55968 Sun Aug 22 16:59:56 PDT 2021 | total used: 36428 | sum apps : 55968 Sun Aug 22 17:00:06 PDT 2021 | total used: 36500 | sum apps : 55968 Sun Aug 22 17:00:16 PDT 2021 | total used: 36508 | sum apps : 55968 Sun Aug 22 17:00:26 PDT 2021 | total used: 36492 | sum apps : 55968 Sun Aug 22 17:00:36 PDT 2021 | total used: 36420 | sum apps : 55968 Sun Aug 22 17:00:46 PDT 2021 | total used: 36448 | sum apps : 55968 Sun Aug 22 17:00:57 PDT 2021 | total used: 36448 | sum apps : 55968 Sun Aug 22 17:01:07 PDT 2021 | total used: 36520 | sum apps : 55968 Sun Aug 22 17:01:17 PDT 2021 | total used: 36448 | sum apps : 55968 Sun Aug 22 17:01:27 PDT 2021 | total used: 36448 | sum apps : 55968 Sun Aug 22 17:01:37 PDT 2021 | total used: 36464 | sum apps : 55968 Sun Aug 22 17:01:47 PDT 2021 | total used: 36448 | sum apps : 55968 Sun Aug 22 17:01:57 PDT 2021 | total used: 36476 | sum apps : 55968 Sun Aug 22 17:02:07 PDT 2021 | total used: 36476 | sum apps : 55968 Sun Aug 22 17:02:18 PDT 2021 | total used: 36460 | sum apps : 55968 Sun Aug 22 17:02:28 PDT 2021 | total used: 36460 | sum apps : 55968 Sun Aug 22 17:02:38 PDT 2021 | total used: 36404 | sum apps : 55968 Sun Aug 22 17:02:48 PDT 2021 | total used: 36448 | sum apps : 55968 Sun Aug 22 17:02:58 PDT 2021 | total used: 36460 | sum apps : 55968 Sun Aug 22 17:03:08 PDT 2021 | total used: 36416 | sum apps : 55968 Sun Aug 22 17:03:18 PDT 2021 | total used: 36420 | sum apps : 55968 Sun Aug 22 17:03:28 PDT 2021 | total used: 36464 | sum apps : 55968 Sun Aug 22 17:03:39 PDT 2021 | total used: 36392 | sum apps : 55968 Sun Aug 22 17:03:49 PDT 2021 | total used: 36376 | sum apps : 55968 Sun Aug 22 17:03:59 PDT 2021 | total used: 36452 | sum apps : 55968 Sun Aug 22 17:04:09 PDT 2021 | total used: 36448 | sum apps : 55968 Sun Aug 22 17:04:19 PDT 2021 | total used: 36392 | sum apps : 55968 Sun Aug 22 17:04:29 PDT 2021 | total used: 36420 | sum apps : 55968 Sun Aug 22 17:04:39 PDT 2021 | total used: 36408 | sum apps : 55968 Sun Aug 22 17:04:49 PDT 2021 | total used: 36376 | sum apps : 55968 Sun Aug 22 17:05:00 PDT 2021 | total used: 36336 | sum apps : 55968 Sun Aug 22 17:05:10 PDT 2021 | total used: 36392 | sum apps : 55968 Sun Aug 22 17:05:20 PDT 2021 | total used: 36276 | sum apps : 55968 Sun Aug 22 17:05:30 PDT 2021 | total used: 36336 | sum apps : 55968 Sun Aug 22 17:05:40 PDT 2021 | total used: 36220 | sum apps : 55968 Sun Aug 22 17:05:50 PDT 2021 | total used: 36276 | sum apps : 55968 Sun Aug 22 17:06:00 PDT 2021 | total used: 36188 | sum apps : 55968 Sun Aug 22 17:06:10 PDT 2021 | total used: 36288 | sum apps : 55968 Sun Aug 22 17:06:21 PDT 2021 | total used: 36188 | sum apps : 55968Here, the total memory usage stays stable at around 36k. The sum of memory usage for the user-land applications is very similar as with CTF enabled (55-56k).
So I guess the logical conclusion is that the memory leak comes from the kernel, and happens when CTF is enabled.
Doesn’t mean that CTF is the direct culprit, though, it may just be the enabling factor. I found a number of very similar memory leak reports with the stock R7000 firmware, like this: https://community.netgear.com/t5/Nighthawk-WiFi-Routers/Nighthawk-R7000-Firmware-1-0-9-64-memory-leak-lose-internet-and/m-p/1713956
There’s also this, which seem to incriminate the 5GHz radio: https://forum.dd-wrt.com/phpBB2/viewtopic.php?p=1172469 (but I tried the suggested workaround of disabling and re-enabling the 5GHz radio right after boot, but it didn’t help with this memory leak, unfortunately). -
reporter
Is there any other information I could provide to help diagnose, and hopefully fix that memory leak?
-
Same issue on 2021.8.
EA6700, normal starts with round 24MB memory usage, and raised to 160MB after large file transferring with CTF enabled. Then, all connections are down include 2.4 & 5G wifi. Router stuck, only cold reboot helps.
But tested with CTF disabled, all goes well, even if more than 30GB files are transferred to NAS. Memory still stays on 25MB or around.
-
BTW, I tested on cleaned NVRAM, with CTF on, others in default, no leak at all.
Then, I set other services on for normally use, pppoe, PPTP server, port forwarding including NAS port forwards. Memory leaked when I transferred some movies to NAS, files over 20GB.
Can not tell the bug exactly, seems like it relates not only CTF option, should be some combinations with multiple options.
I now changed to DD-WRT. Some web services need to go on.
-
Can you repeate with minimum config and find the combination with CTF? File Transfer from where to where using pppoe or pptp or … ?
-
I had been using 2021.8 for weeks, and versions bellow before. Never encountered the bug so badly until PPTP server was enabled. I did not watch the memory usage before because the router still worked.
CTF was always on, transferred movies from laptop to NAS under the same router, exactly, from 192.168.0.10 to 192.168.0.12.
But still need to find the minimal combination. CTF and PPTP server are the settings need to check. I’ll test it some day.
-
I thought my router was busted till I enabled the logs and found the same kernel warnings and memory leak as mentioned above.
For me I recently enabled PPTP Server on my side, and the memory leaks started shortly thereafter. Could be coincidence, but I can certainly do some testing.
FreshTomato version: 2021.8
-
Please test again with PPTP Server disabled. thx
-
what router do you use? I would like to test a small change
-
It reminds me that, it did not happen until PPTP server was on. But, sorry, I have been looking for a job for months, not tested it up till now.
-
I am using the same router as the reporter (Netgear R7000). I would be interested in testing.
-
for testing - small change to bypass CTF
-
Just tested it for about a day. The memory usage creeps up more slowly, but by the next night, it reaches capacity and leads to the same errors as shown above.
-
thx for feedback - next attempt
please test “v002”
-
I am dirty flashing these builds, would that hamper testing or is it okay?
For this build, same result. Memory usage creeps up till the router chokes.
-
I am dirty flashing these builds, would that hamper testing or is it okay?
thats ok - no need to clean nvram.
thx for feedback. further investigation needed for pptp server code
Support for PPP over IPv4.(Point-to-Point Tunneling Protocol) This driver requires pppd plugin to work in client mode or modified pptpd (poptop) to work in server mode. See http://accel-pptp.sourceforge.net/ for information how to utilize this module.
-
Confirmed, CTF on and PPTP server on will lead to memory leak. Only CTF on, all is OK.
-
thx for feedback. I see one patch from linksys src. it may can help here. Will add it later
-
repo owner
- changed title to PPTP server with CTF enabled causes kernel memory leak?
-
i would remove pptp support in favor of wireguard - no need to ipsec either
-
@sysres - wireguard is available in FT ARM https://www.linksysinfo.org/index.php?threads/wireguard-on-freshtomato.76295/ but ipsec it’s not (by default - needs extra modules)
-
I believe this bug should be closed with “won’t fix” - to force users off pptp.
Of course PR/MR from any pptp user are welcome
-
reporter
I agree that having a GUI interface to configure WiregGuard would be the ideal way forward.
Now, PPTP tunnels are already deprecated and can't be used on Android anymore, so I expect the user base to dwindle over time.
In the meantime, maybe displaying a warning on the PPTP server page when CTF is enabled (or just plain disable the option) would be a good start? A bit like the existing warnings for CTF and QOS.
-
Can someone please test 2022.5 release for this topic/problem.
→ There have been a lot of kernel updates and also to pptp code
thx & BR
-
Just tested, still exists in 2022.5.
-
thx for testing
-
But seems going more slowly than versions before.
Coping 6G data to other PC via local network, with PPTP on and CTF on, memory rises to 90MB from 24MB. But with PPTP on, CTF off, it stays on 24-25MB.
-
Still exists in 2024.1
- Log in to comment
hi,
can you please provide more infos about your router setup. Connections type, wifi setup, services you use, scripts…
What happens if you turn off CTF ? → please test that case
If CTF On can trigger the crash on your side → Different test - please use the following images (latest 2021-5 release and different ctf binary and latest branch)
Does the crash happen again with CTF turned on ?
https://bitbucket.org/M_ars/freshtomato-arm/downloads/freshtomato-R7000-ARM_NG-2021.5-15082021-ctf_test_v001-AIO-64K.trx
and
https://bitbucket.org/M_ars/freshtomato-arm/downloads/freshtomato-R7000-ARM_NG-2021.5-21082021-ctf_test_v002-AIO-64K.trx