After several days Wan throuput drops from 1Gbps down to 300Mbps

After several days Wan throuput drops from 1Gbps down to 300Mbps

After several days Wan throuput drops from 1Gbps down to 300Mbps
After several days Wan throuput drops from 1Gbps down to 300Mbps
2023-10-06 09:47:34 - last edited 2023-10-06 09:54:46
Model: Archer C5400  
Hardware Version: V2
Firmware Version: 1.2.9 Build 20230907 rel.96516(4341)

* Note the firmware version was just upgraded so this issue was with the previous version from 2022.

 

I have my AC5400 sitting behind an ISP-provided Fibre Modem that is running in routed mode (ISP Modem is running 192.168.0.x/24 and AC5400 is on 192.168.1.x/24). Except for my TV set-top box which pairs directly to the ISP modem over its WiFi, all wired and wireless devices in my house are connected to the AC5400 and on the 192.168.1.x subnet.

 

My traffic pattern means that several times throughout the day it is not unusual for my Linux server to download at network saturation for the 1Gbps hardware I am using in my house (the ISP Modem can run over 3Gbps uplink and downlink when using internal speed test tools meaning the WAN can handle this speed without issue). Running Ookla's speed test utility from Linux shows that I am getting the full 1Gbps that the wired network in my house allows in each direction...

 

bergang@hda:~/data_nbk_gh/Archive/temp/rename$ speedtest

   Speedtest by Ookla

      Server: Bell Canada - Ottawa, ON (id: 54836)
         ISP: Bell Canada
Idle Latency:     1.60 ms   (jitter: 0.23ms, low: 1.14ms, high: 1.69ms)
    Download:   938.39 Mbps (data used: 491.0 MB)
                 11.92 ms   (jitter: 0.87ms, low: 1.98ms, high: 23.21ms)
      Upload:   938.34 Mbps (data used: 871.4 MB)
                  5.48 ms   (jitter: 1.09ms, low: 1.71ms, high: 21.03ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/85f78478-b58f-4a0c-8f99-ce606e0b4a11

 

What I am experiencing is that over the course of several days (can be as many as 10) this speed will drop to about one-third of the 1Gbps which is normal (about 300-330 Mbps). Rebooting the AC5400 restores the network performance to normal. I have looked at the CPU and memory resources on the router and there does not seem to be a significant change from before and after the reboot. I am thinking that it could be related to socket management between the two routers but am wondering if there is some way that this can be diagnosed better. 

 

I have written a cron utility to catch this condition and send me an email when it occurs so I know to reboot the router. The test looks to see if uplink or downlink has fallen below 800 Mbps every 15 minutes and will trigger if 12 consecutive tests fail (this prevents normal network activity from appearing like a network slowdown if the test runs during a high-traffic event). Here is the pattern of its occurrence over the last several months (the utility generates a set/clear log event into a text file on the server)...

 


Aug 02 2023 09:00:27.038 : Internet speed test has fallen below threshold
Aug 02 2023 20:30:18.569 : Internet speed test is now above threshold
Aug 10 2023 06:30:23.412 : Internet speed test has fallen below threshold
Aug 10 2023 09:45:17.161 : Internet speed test is now above threshold
Aug 30 2023 21:30:28.428 : Internet speed test has fallen below threshold
Aug 31 2023 01:45:15.285 : Internet speed test is now above threshold
Sep 04 2023 01:45:22.502 : Internet speed test has fallen below threshold
Sep 04 2023 13:15:18.376 : Internet speed test is now above threshold
Sep 07 2023 22:00:31.355 : Internet speed test has fallen below threshold
Sep 08 2023 01:00:17.739 : Internet speed test is now above threshold
Sep 12 2023 12:00:17.949 : Internet speed test has fallen below threshold
Sep 13 2023 01:30:17.501 : Internet speed test is now above threshold
Sep 23 2023 12:30:28.748 : Internet speed test has fallen below threshold
Sep 23 2023 19:45:20.678 : Internet speed test is now above threshold
Sep 29 2023 03:00:32.344 : Internet speed test has fallen below threshold
Sep 30 2023 04:15:14.836 : Internet speed test is now above threshold
Oct 05 2023 00:15:33.428 : Internet speed test has fallen below threshold
Oct 06 2023 05:00:15.724 : Internet speed test is now above threshold
 

  0      
  0      
#1
Options
5 Reply
Re:After several days Wan throuput drops from 1Gbps down to 300Mbps
2023-10-13 10:22:14

  @Obelisk3847 Wow,you mean you wrote a program to detect this, which is really cool. I want to ask if the latest firmware version is still like this? Why does rebooting the router make the speed normal?  Is your Internet speed related to the time range when you use the Internet?

  0  
  0  
#2
Options
Re:After several days Wan throuput drops from 1Gbps down to 300Mbps
2023-10-13 18:09:02
The tool I wrote is a simple bash utility that runs the Ookla speed test program. I have it running every fifteen minutes and once it detects whether the uplink or downlink falls below 800Mbps it will then check file markers that it puts in the /tmp folder. Each time it runs if it fails the speed test it puts an additional file marker in the folder. If it finds that it has 13 file markers it sends me an email as it indicates the last 13 tests failed (it runs 4 times per hour). Any successful test will clean up the file markers in /tmp and reset everything this way if I am doing some heavy file transfers for a period of time that happened to affect the tests, if the speed goes back up it resets the history. I also drop a different marker into the file system when I send an email so it doesn't send me an email every fifteen minutes. There is also some additional code that prevents no more than 15 failed test markers from accumulating so if I don't get to the reboot in a timely manner it will not fill up /tmp with file markers. Once the test passes it cleans up /tmp including the email marker and sends me a notification saying the speed is back to normal (the set and clear events in the log above coincide with the email notifications). It just happened with the latest firmware this morning and I haven't been able to determine that there is any pattern to this behaviour. It appears to be quite random. The reason that I suspect it is related to sockets is because every socket opened up between the two routers will utilize a bit of memory and CPU. I was hoping that I would see that the CPU or memory resources would increase over time as reported by the TPLink router but it does not. Rebooting the router will force both routers to drop any connections that are between them that didn't get closed properly as well as drop any processes that could be running in memory on the TPLink side.
  1  
  1  
#3
Options
Re:After several days Wan throuput drops from 1Gbps down to 300Mbps
2024-04-07 03:30:36 - last edited 2024-04-07 03:31:13

  @Obelisk3847 

 

I had the same problem, I went through everything to find a solution, even the last one released > 1.2.9 Build 20230907 rel.96516 is not working well

to solve use this beta version > 1.2.5 Build 20191204 rel.63911(5553)

If you can't find it, call me in PM or bookmark here and I'll send you the zip file.

Escute cem vezes, pondere mil e fale apenas uma vez.
  1  
  1  
#4
Options
Re:After several days Wan throuput drops from 1Gbps down to 300Mbps
2024-04-07 18:31:12 - last edited 2024-04-07 18:31:41

  @RodrigoSQL Thanks for taking the time to respond. If you could upload a copy of the router software that would be helpful. My workaround has been to put the router on a smart plug and reboot it at 6 AM twice a week which has worked for the most part but is a terrible solution.

  1  
  1  
#5
Options
Re:After several days Wan throuput drops from 1Gbps down to 300Mbps
2024-04-10 01:39:21

  @Obelisk3847 

 

 

 I can't send it here, as there is a 10mb limit, if you want, call me in PM and I'll give you the link

Escute cem vezes, pondere mil e fale apenas uma vez.
  0  
  0  
#6
Options