Bug report - ER8411 - One vlan loses lan>wan communication
I have noticed a very strange bug with one particular vlan on my network. This issue has persisted from ER8411 firmware 1.2.2 at least, and 1.2.3. And OC200v1 firmware from January and the very latest march 10th beta.
This issue has persisted across factory resets of both gateway and controller, and complete rebuilds from scratch of the entire network. It does not matter if the ER8411 is doing internal routing on this particular VLAN or i have it switch routed through an SVI on the lan side switches for internal routing. And it only effects one particular vlan with the same IP range each and every time!
Issue-
VLAN with tag 8, with IP range 192.168.8.0/23 will lose internet connectivity at random and will never come back by itself. Internal routing still works. It can ping the gateway. It can ping and recieve response from an IP address on the internet. However, it cannot traverse the NAT and use the internet at all for normal browsing. DHCP for clients set to either cloudflare DNS servers directly, or to internal DNS proxy service of the gateway makes no difference. Problem effects both LAN and WiFi clients on this vlan, rehgardless of which switch path they are on from the gateway and core switch. Traceroutes fail at the gateway, but it can still ping an internet IP such as 8,8,8,8 ? weird
How i resolve it-
To fix it, all i need to do is assign it to another WAN on policy routing, then reassign it back to the WAN i actually want it to use. This resolves its internet connectivity every time without needing any reboots of anything. Very Strange!
What is weird, is it only effects this one single IP range vlan. none of the others at all, ever. All the other VLANs policy routed to the same WAN are not effected, at all, ever. I cannot reliably replicate the problem as it seeming happens at random, and is hard to monitor as this particular vlan has very infrequent use.
I would welcome some ideas to test!
Things i have tried:
IDS/IPS disabled
Different DNS servers on that vlans DHCP
client with static IP and statically set DNS
Deleted and recreated that one vlan
Deleted and recreated all vlans
Factory reset everything, readopted from existing configuration
Factory reset everything, including OC200, rebuild everything from scratch
Disabled all gateway and switch ACLs
Thrown candies at the ER8411 when testing due to frustration with this
Shouting and Swearing at it
Threatening to replace it with a Cisco unit
Bear with me, this is a bit complex!
I have all these vlans
192.168.0.0/23 (management, all omada lives here)
192.168.2.0/24 (internet transit vlan)
192.168.6.0/23
192.168.8.0/23
192.168.10.0/23
192.168.100.0/23
SG3428X is downlink from ER8411 on 10gbit link, and has 6-link LAGs to each of the switches below. SVIs on all vlans on the lower half of each /23 at .253 (eg 192.168.6.253)
SG3452 - For LAN Devices. SVI on all vlans is the lower half of the /23 on each vlan at .254. (eg, 192.168.6.254)
SG2428LP - For POE / WiFi devices via access points, SVI on all vlans is in the upper half of all vlans on .254 (eg, 192.168.7.254)
Each Switch VLANs are the gateway for clients on that half of the network. No clients on the SG3428X, its juse a L3 routing bridge.
SG3452 and SG2428LP have static routes 0.0.0.0/0 to SG3428X transit vlan SVI for internet hop (192.168.2.253)
SG3428X has 0.0.0.0/0 route to ER8411 Transit vlan interface at 192.168.2.1
ER8411 has supernet 192.168.0.0/16 default route to SG3428X transit VLAN svi
All internet and inter vlan traffic is handled by L3 switching at each entry point, using the SVIs to directly route it to and from all clients.
Tracert to internet IP from a wifi client on 192.168.7.1 would be:
192.168.7.254 (SVI on SG2428LP) > 192.168.2.253 (transit SVI on core switch) > 192.168.2.1 (ER8411 transit interface)
(however, because the switches are IP routing aware as im using SVIs and the source packet IP remains intact, the actual tracert shows the middle hop as happening on the SVI for the actual vlan, which is expected of course)
External Tracert to a device i temporarily opened up through NAT follows the same path but in reverse.
This works perfectly for all vlans, all the time. Never an issue. EXCEPT, the 192.168.8.0/23 network just sometimes stops at the gateway, and when that happens, it wont go any further. Only ever that vlan. Never any others. It does reach the gateway. So i dont think my switch routing is the issue here.
All SVIs are defined with a 255.255.254.0 subnet mask so all are aware of the full .23 subnet of each vlan. I have double and triple checked i have no IP overlaps on the switch interfaces.
Client on vlan 6 192.168.6.0/23
Client on vlan 8 192.168.8.0/23 (when its working). This is the vlan that occasionally just....stops.
As i indicated, as soon as i change the WAN that this vlan is set to in policy routing, and then back to the one i want, traffic once again flows through the ER8411 and back normally, until it just stops again at some random future point.
I managed to catch it when the 192.168.8.0 /23 network breaks
In this first image, you can see that tracert to 1.1.1.1 traverses the inter-switch routing, and finally hits the gateway. If goes off to the internet correctly.
You can then see this i can ping google.com. However, in the browser, google.com will not load - even if i put in its resolved IP address directly.
In this second image, you can see that the same scenario, on a different VLAN policy routed to the same WAN as above, with identical (other than IP range) settings, traverses the switches and hits the gateway, pings, and can load the web page
And lastly, this image below. Same vlan as the first image - the one that is problematic - 192.168.8.0 /23. All i did was change the WAN port its policy routerd to, and back again, and now working....
Currently, ER8411 is on 1.3.0 beta. I was seeing the same behaviour on 1.2.2 and 1.2.3
There is only one difference between the network the breaks .8.0/23 and the ones that dont - .8.0/23 has no clients on it at all 24/7. Its a network for visiting technicians to use in the theatre, and only has light occasional use. Is this some sort of timeout / keepalive problem perhaps ?
EDIT:
My policy routing. I have modified this several times to try and mitigate the problem. I have tried: Combining networks into one rule per WAN, seperate rules per VLAN, Used "Networks" and "IPGroups" as the source, no difference. The Problematic VLAN is the one called "Event Tech"
Hi @GRL
Thanks for posting in our business forum.
I am curious why it is different from the tracert you have above?
Without anything touched, simplify it, what's the tracert like?
And this is different from the time when 192.168.8.0/23 worked. It was never sent to 9.254 and 9.253. What are they? These IPs?
After tons of reading on your writings, I am more inclined to believe this is not a router/switch problem.
If it exists in 1.2.2, 1.2.3 and 1.3.0, it means this is a problem that has existed long time. Not a firmware issue.
Same WAN exit, but the connection was reset, DNS and ping both work, could be the server stops you.
Except for the connection to Google. If you access any other servers that are not using Google CDN or servers, will you be able to get a proper connection?
Did you try to recreate the 192.168.8.0/23? Will the problem persist?