Business Community > All Threads > ER7206 high latency, dropping packages - only from time to time
< All Threads

ER7206 high latency, dropping packages - only from time to time

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.

ER7206 high latency, dropping packages - only from time to time

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.
ER7206 high latency, dropping packages - only from time to time
ER7206 high latency, dropping packages - only from time to time
2023-06-20 09:54:47 - last edited 2023-06-20 13:32:32
Model: ER7206 (TL-ER7206)  
Hardware Version: V1
Firmware Version: 1.3.0

Hi!
I've been using Omada for couple of months now. Today's night I've observed my IoT devices miss internet access.

Firstly I though it's ISP, but after debug I found out ISP works stable. Further debug shown that there is a problem with ER7206 gateway.

 

The problem occurs in waves where gateway drops majority of packages - works perfectly fine for couple of minutes, and then fails for minute or two.

I've tried to restart all devices in my setup - it kept working for 10-15 minutes. Problem is back. Factory reset and adaptation didn't fix the issue.

After another restart it seems stable - very random. I didn't touch the infrastructure (cables, devices, etc.) for weeks. 

 

Using one PC I've connected to the Gateway (directly to one of it's ports) and the ISP's router using separate network cards. This gave me high confidence it's about gateway.

 

Latency to ISP's router is stable (less than 1ms, 0 packages dropped).

And here is the latency for ER7206:

 

  0      
 
  0      
 
#1
Options
1 Accepted Solution
Re:ER7206 high latency, dropping packages - only from time to time-Solution
2023-06-20 13:27:16 - last edited 2023-06-20 13:32:32

This is quite interesting, and this could be it. I can't say what exactly caused such increasing activity.

As I get it - DEV1 spammed the network with something.

 

I'll post if I'll figure it out.


Thank you!

Recommended Solution
  0  
  0  
#6
Options
10 Reply
Re:ER7206 high latency, dropping packages - only from time to time
2023-06-20 10:14:53
Did I say "After another restart it seems stable"? not anymore, problem is back
  0  
  0  
#2
Options
Re:ER7206 high latency, dropping packages - only from time to time
2023-06-20 10:24:10 - last edited 2023-06-20 10:34:02

  @folfix 

 

Have you tried disconnecting all other devices and only using a PC directly in the router? there may be a loop or a broadcast storm in the network that is causing problems. I have an ER7206 with the same firmware and I have 1 ms when I ping the router

 

you should also check if you have a device with the same IP as the router, 192.168.0.1 is a very common IP that is standard on many devices.

 

 

  0  
  0  
#3
Options
Re:ER7206 high latency, dropping packages - only from time to time
2023-06-20 12:15:39

  @folfix 

 

Any messages in the logs?  Does the CPU %'ge change between working normally and super-slow mode?

<< Paying it forward, one juicy problem at a time... >>
  0  
  0  
#4
Options
Re:ER7206 high latency, dropping packages - only from time to time
2023-06-20 13:13:53

Thank you all for your suggestions!

I've restarted every device in my whole setup (1 gateway, 3 switches, 8 EAPs). So far so good (couple of hours).

 

I've checked the consumption this morning (for 24h) - but it didn't take my attention as it was quite ok (based on % values).

But, it looks strange - memory building up. Still, 27% in peak doesn't looks bad.

 

7 days chart:


 

Last 24h (1min resolution):

First outage was captured around 00:30 - which aligns with CPU peak. and IoT kept restarting for 3-4 hours. It was super unstable.

 

No alerts/event is Omada in the 00:30 - 09:00 timeframe.

(I mean I've seen alerts but that was just after I've factory reset gateway and restarted devices - I expected them to happen)

 

I'll then monitor further and in case of problem I'll check CPU and mem consumption.

Worth to mention - I don't use VPN (it's configured, but no users) and ACL.

 

Thank you,

Jan

  0  
  0  
#5
Options
Re:ER7206 high latency, dropping packages - only from time to time-Solution
2023-06-20 13:27:16 - last edited 2023-06-20 13:32:32

This is quite interesting, and this could be it. I can't say what exactly caused such increasing activity.

As I get it - DEV1 spammed the network with something.

 

I'll post if I'll figure it out.


Thank you!

Recommended Solution
  0  
  0  
#6
Options
Re:ER7206 high latency, dropping packages - only from time to time
2023-06-20 13:42:06

  @folfix 

 

Looks like a spanning tree issue between ports 1 and 3.   Spiralling packet storm leading to port shut down, cooling off, and re-enabling.

 

There should be something in the Events pointing to this (unless you've disabled messaging for such events).

 

 

<< Paying it forward, one juicy problem at a time... >>
  0  
  0  
#7
Options
Re:ER7206 high latency, dropping packages - only from time to time
2023-06-20 13:59:53

I see nothing interesting in Events - only that devices were connected/disconnected frequently.

 

The cut off at ~3:45 (chart time) was a restart of the gateway - which I did by hand. This solved the issue for a while as problematic devices (clients) couldn't recover the connection. And then after ~9:30 when I started work, the traffic was back - as devices woke up.

 

I have something like a cluster setup which involves server-to-server communication (within a LAN). This must be it. It just went wild.

In between I've restarted client devices and this restarted all services in the OS.

 

Thank you all for the support!

 

  0  
  0  
#8
Options
Re:ER7206 high latency, dropping packages - only from time to time
2023-06-20 14:03:58

  @folfix 

 

Is it possible you're getting DDOS'd from outside?

<< Paying it forward, one juicy problem at a time... >>
  0  
  0  
#9
Options
Re:ER7206 high latency, dropping packages - only from time to time
2023-06-20 18:30:27

  @folfix dev1 link is up to something, most of the time nothing malicious, broken equpiment installed that behaves like a old hub, broken cabling, assume incompetence rather than malice.

  0  
  0  
#10
Options
Re:ER7206 high latency, dropping packages - only from time to time
2023-06-23 15:26:06

Hi!

Issue is gone permanently.
I've verified as much as I could and I doubt it was a DDOS attack.

DEV1, DEV2, and DEV3 are connected together. They're part of the cluster. When I've restarted the cluster services - all works.

 

 

Thank you all!

 

  0  
  0  
#11
Options

Information

Helpful: 0

Views: 942

Replies: 10

Related Articles
ISP Load latency is sky high. Real latency is normal.
397 0
ER7206 - Dropping Connections
1006 0
Latency / Bad Ping Configuration for ER7206
591 0
ER7206 OpenVPN
766 0
ER7206 won't connect
692 0
Snmpwalk through ER7206 router
740 0
About Us
Press
Copyright © TP-Link Systems Inc. All rights reserved.