@Clive_A 

Thanks for responding.

The ISP is Astound, formerly Wave-G, a smaller regional ISP. The service is provided via ethernet in a multi-tenant mix use building. Each group of floors has a switch feeding each unit. Each unit is on its own port. The building is serviced by a 10Gb wireless backhaul. I have a screen shot of some of the logs from the Cisco switch confirming the psecure-failure and some of the MAC addresses. Prior to installing this router last week I didn't have any issues. Before the ER605 was installed the Deco X60, mentioned below in the diagram, as acting as the router but has since been switched to AP mode.

My topology is currently:

I have since deviated from this topology a little on a hunch that the ER605 port 3, a WAN/LAN port, connected to the Deco X60 is problem. I moved the Deco off the ER605 and have both WAN/LAN ports unused. Since the issue happens about a dozen times a day I should know in the morning if this has made a difference. The current topology is:

Thanks,

Jake

This reconfiguration has been in place for ~12 hours now without issues. I hate to jinx it but I am confident it was something to do with having a device on the WAN/LAN port.

One more change I made that I forgot to include in my previous message was I also disable all 802.1q VLAN tagging, which wasn't currently being used yet anyway. Prior to putting the ER605 in place the Deco was setup with a tagged guest network. In AP mode the Decos don't tag the guest network, making it completely useless. I hadn't removed the VLAN tagging from the switch that it had been attached to. There were no VLANs configured on the ER605 or TL-SG105E and nothing would have been on that VLAN during all these outages issues though, so don't think it is the culprit. If this current configuration survives 24 hours then I am going to put the VLAN back just to be certain.

Either way this is looking like a bug on the port of the router software still, no ethernet frames from the LAN side should EVER be coming out the WAN side.

  @Clive_A sorry it has been a while. I have some captured frames from the WAN interface but first let's catch up. I had though things had cleared up after removing VLANs and factory resetting the switches, but it is was just time before it happend again. I then decided to factory reset the router and run it as default as possible, only a DHCP pool range change, so no Omada controller or anything else. It continued to happen. So today I was able to get a machine in the closet and capture it in action. 

I configured the router such that port 3 was not in th default VLAN, so no natural traffic was flowing to that port. I then setup mirroring to mirror egress only from port 1 (WAN) to port 3. I setup tcpdump to capture all ethernet frames not having my router's WAN MAC address. Over the course of the next few ours it capture a few frames, in two distinct events. The second event triggered the Cisco's psecure and cut me off. In both events the frames are IP, with the source IP set to my WAN IP. The destination IP are different for the two events. The source and destination ethernet addresses differ in each event. The very odd thing is the the destination ether address is not the MAC for the ISPs router but some seamingly random MAC in the universal administered block. The two source MACs are not my router's WAN MAC and like before they seem to are the local administered block. None of these 4 MACs show up on my LAN. I have checked ARP tables on a few hosts and the router's DHCP client list. Those randomized local administered MACs would typically show up in the DHCP table but none of the ones in the table match the two in this capture. If you know of a way to dump the forwarding table for the ER605, TL-SG108E and TL-SG105E then I can double check those.

I think we can safely rule out corrupt frames from the wire since:
a) the frames captured make up a sequence of IP packets.
b) are being recored from port mirroring, not by capturing what is flowing on the wire.
 

What is very odd is that as thes packets leak out something is rewriting the IP header such that WAN IP is ending up on the packets. I suppose something on the LAN could be sending these wacked packets but I would expect the router to drop them on the LAN side if the source IP was WAN IP. To be safe I will setup a mirror on the LAN ports and cature anyting with a non LAN IP.

  @jake-at-home 

Interestingly enough I am hitting almost the exact same issue that you are...

I have Astound/WaveG, and ER605 and Deco Wifi Mesh system...  My Connectivity has been horrible for the last few months.  Finally got WaveG support out and we identified that i am suffering from the same issue of additional mac addresses leaking out the WAN port, triggering port to disconnected.

After an exhaustive Google Search, I discovered this thread, as well as this thread of very similar behaviours being seen on the Ubiquity EdgeRouter:

• https://community.ui.com/questions/Blocked-by-ISP-multiple-MAC-addresses-in-their-logs/763d41fd-b884-4e16-89f6-223698d0b57e
• https://community.ui.com/questions/EdgeRouter-X-SFP-leaking-MAC-addresses/0099c9c9-a2d3-42c4-817e-71806094f7b3

It seems that ER605 V2 and the Ubiquity Edge Router are built atop the same SoC: MediaTek MT7621AT

The threads referenced with regard to EdgeRouter above, while not entirely spelling out the solution, implied that disabling Hardware Offload may have mitigated the issue...  
I am going to give this a try on my side this evening... 

  @Its-A-Trap we are probably neighbors! That is some great information. I had seen some of that but hadn't put it together. I have read that turning of HW offloading kills the throughput pretty bad. It also looks like I need to put this back under an Omada controller to even get that option, I can't find it anywhere in the direct management interface. I look forward to hearing if you have any luck.

  @Clive_A thanks for picking this up! Some more info, I was able to capture the LAN side of the network at the same time as the WAN and linked the packets back to the source on my network. So the MAC addresses that end up on the WAN side are completely random. I had thought maybe the fact the destination MACs seem to always be in the local admin block, like randomized private MACs, that maybe the source of the traffic was in randomized block. That is not true, in the latest capture it is my wired host with it's true MAC address on the LAN. When it leaves the router the source and destination macs are just seemingly randomized bytes. 

If you want any of these capture files please send me a direct message with an email address or other place to send them. I don't want to post them on a public forum. 

Thanks!!

-Jake

  @jake-at-home , Quite possible we are neighbors!

Quick update, with a few small caveats...

• My setup is slightly different than yours in that i have Two Internet connections.  I added a XFinity cable modem as a backup when i thought that the issue was on the WaveG Side. I have been experimenting with maintaining both and with having only WaveG connected
• I have not yet set up the testing infrastructure to directly observe the mac address leaking, and am thus relying on observing my port being shut down, so the testing takes quite some time...

I need to do further testing, however at this point i am fairly confident that the issue is at least partially associated with the configuration of Hardware Offload.  I am suspicious if there are either potentially multiple initialization code paths that results in different configurations, or if the IPs have some order dependent initialization requirements not being fully respected.

• Previously, My port would be shut off for approximately 11 minutes each time this defensive measure was triggered by the upstream switch.  I would encounter this 11 minute gap every 20-30 minutes.
• After Disabling Hardware Offload (With only WaveG Active) my connection was stable for over 2 hours
• After Re-enabling Hardware Offload (with only WaveG Active) my connection continued to remain stable for over 2 hours.
• I then took a risk and re-enabled the second WAN/Lan port for my backup internet connection, after the system was re-provisioned, my system once again exhibited the disconnect behavior.  This behavior did however seem to continue even when i disabled hardware offload (Confused on this one, and need to dig in more...)
• I was able to restore long term stability by unplugging my backup internet connection and disabling Hardware offload 

I would like to get to a true root cause as I am sure that it will come back and bite me again soon, as if it is an artifact of the different initialization code paths or some other order of operations. I expect that this is not yet resolved.  Also, I do not currently have a stable configuration that supports my second WAN.

In order to really make progress, i likely need to invest the time to setup an upstream switch to enable mirroring and capture so i can have a more reliable signal than waiting for my port to get turned off...  that may be next weekend...