ER8411 Loss of WAN connectivity after power outage
Hello,
I faced another issue with ER8411 last night. I had a power outage at my site that lasted 3 hours. After the power restoration (ER8411 has no backup power unit as of now, both power sockets are connected), the ER8411 was unable to raise again the PPPOE connection.
SFP+ WAN1: PPPoE connection was disconnected by peer. (AC-MAC=08-b2-58-40-fa-98, Session-ID=0x00d0) Jun 10, 2024 08:24:38 Gateway PPPoE Module Information SFP+ WAN1: PPPoE Discovery phase negotiation succeeded. (AC-MAC=08-b2-58-40-fa-98, Session-ID=0x00d0) Jun 10, 2024 08:24:33 Gateway PPPoE Module Information SFP+ WAN1: PPPoE Discovery phase negotiation succeeded. (AC-MAC=08-b2-58-40-fa-98, Session-ID=0x00d0) Jun 10, 2024 08:23:50 Gateway PPPoE Module Information SFP+ WAN1: PPPoE Discovery phase negotiation succeeded. (AC-MAC=08-b2-58-40-fa-98, Session-ID=0x00d0) Jun 10, 2024 08:23:08 Gateway PPPoE Module Information SFP+ WAN1: PPPoE Discovery phase negotiation succeeded. (AC-MAC=08-b2-58-40-fa-98, Session-ID=0x00d0) Jun 10, 2024 08:22:26 Gateway PPPoE Module Information SFP+ WAN1: PPPoE Discovery phase negotiation succeeded. (AC-MAC=08-b2-58-40-fa-98, Session-ID=0x00d0)
I though this was caused by the power outage, maybe affecting the ISP connectivity but as test I moved the SFP+ transceiver to two different routers and PPPoE connectivity went immediately up.
This ER8411 is managed by Omada SDN Controller Linux 5.14.20.9.
I tried to move the WAN connection to the 3 SFP+ 1G port of the ER8411 and also to LAN 1 (RJ45), configuring the proper settings for PPPoE connection but had the same errors reported above. At this point, I tried multiple reboots of the ER8411.
Omada SDN reported no connectivity issues towards ER8411 and the WAN interfaces were just marked as "not connected" (connection was still KO also after manually trying to bring it up via connect button).
Then, I tried to re-adopt the device from the controlled = nothing changed.
Lastly, I downloaded diagnostic file of ER8411 and let the controller to forgot it.
As soon as I booted to default/standalone mode, I reconfigured the PPPoE connection, with same SFP+ IF and transceiver and it went up immediately. I performed the adoption from Omada Controller and everything was back up as before. So we are definelly facing a software (ER8411 1.2.1) or Omada SDN bug which prevented raising a PPPOE connection, even after multiple software reboots. I did not perform any hard reset so far.
As said, I have both the ER8411 diagnostic and the controlled export of running logs (I did not reboot yet the controller) and config export available.
I look forward to your support as this device should be TP-Link's diamond tip router and such issues shouldn't really happen. I'm available to provide every log/diagnostic needed.
Attached some PPP logs about the disconnections and the last connection brought up with no further events or config changes on my end.
Thank you.
Hi @Bianco8
Thanks for posting in our business forum.
I cannot say this is a problem with the router.
As long as the PPPoE is sent out, it is not up to the router to determine whether to connect or disconnect. It is your ISP to determine that.
This log should not be a problem with us at all as it is disconnected by the peer server. The router should just send the discovery and wait for the peer server to allow its connection.
Understanding PPPoE Interaction and PPPoE PADI Timeout
And, based on your log,
(Did you manually censor the MAC address?)
The same thing happened to the SFP+ WAN as well. I think it should be a line issue or the ISP. The only way to find this out is to Wireshark during the down time and see if the ISP actually sends the LCP response.
So far I cannot judge it to be a problem with the router.
Clive_A wrote
Hi @Bianco8
Thanks for posting in our business forum.
I cannot say this is a problem with the router.
As long as the PPPoE is sent out, it is not up to the router to determine whether to connect or disconnect. It is your ISP to determine that.
This log should not be a problem with us at all as it is disconnected by the peer server. The router should just send the discovery and wait for the peer server to allow its connection.
Understanding PPPoE Interaction and PPPoE PADI Timeout
And, based on your log,
(Did you manually censor the MAC address?)
The same thing happened to the SFP+ WAN as well. I think it should be a line issue or the ISP. The only way to find this out is to Wireshark during the down time and see if the ISP actually sends the LCP response.
So far I cannot judge it to be a problem with the router.
Many thanks for your prompt reply.
I saw the peer disconnect message, which led me to think about ISP issues.
I forgot to add some context: ER8411 is connected to a Nokia XGS-PON ONT 10G (provided by ISP).
After several hours of outage, I took a look on the setup and saw that the ONT lights were reporting nominal status. So I connected 2 different routers and PPPoE went up immediately. As soon as I plugged the ER8411, PPPOE was stuck down with such messages. I also physically rebooted the ONT but nothing changed on ER8411 end (ONT state was OK).
The only device I did not physically reboot was the ER8411 only. These details let me think of a software issue and not to ISP/line anomaly and let me confirm this:
Checking the debug logs downloaded from ER8411 (just before removing it from controller), syslog.log is reporting an additional message:
2024-06-10 12:11:56 pppoe_client[7484]: <5> 08214206 WAN4: PPPoE session is over 2024-06-10 12:11:56 pppoe_client[7484]: <5> 08214212 [OMADA]WAN4: PPPoE connection was disconnected by peer. (AC-MAC=MAC_ADDR, Session-ID=0x0548) 2024-06-10 12:11:56 ppp[7484]: <4> 02313107 [OMADA]<null>: PPPOE PPPOE_USERNAME failed to connect to the server because IPCP negotiation failed and local IPv4 addresses duplicate. 2024-06-10 12:11:55 ppp[7484]: <5> 02313103 [OMADA]WAN4: PPPOE_USERNAME authenticated. 2024-06-10 12:11:55 ppp[7484]: <5> 02313101 [OMADA]WAN4: LCP negotiation succeeded. (MTU=1492, AUTH=<null>) 2024-06-10 12:11:55 pppoe_client[7484]: <5> 08214211 [OMADA]WAN4: PPPoE Discovery phase negotiation succeeded. (AC-MAC=MAC_ADDR, Session-ID=0x0548) 2024-06-10 12:11:52 pppoe_client[6282]: <5> 08214202 WAN4: PPPoE began connecting automatically.
###
2024-06-10 12:24:48 pppoe_client[28082]: <5> 08214211 [OMADA]WAN1: PPPoE Discovery phase negotiation succeeded. (AC-MAC=MAC_ADDR, Session-ID=0x0590) 2024-06-10 12:24:45 pppoe_client[26839]: <5> 08214202 WAN1: PPPoE began connecting automatically. 2024-06-10 12:24:33 pppoe_client[24702]: <5> 08214206 WAN1: PPPoE session is over 2024-06-10 12:24:33 pppoe_client[24702]: <5> 08214212 [OMADA]WAN1: PPPoE connection was disconnected by peer. (AC-MAC=MAC_ADDR, Session-ID=0x0590) 2024-06-10 12:24:33 ppp[24702]: <4> 02313107 [OMADA]<null>: PPPOE PPPOE_USERNAME failed to connect to the server because IPCP negotiation failed and local IPv4 addresses duplicate. 2024-06-10 12:24:33 ppp[24702]: <5> 02313103 [OMADA]WAN1: PPPOE_USERNAME authenticated. 2024-06-10 12:24:32 ppp[24702]: <5> 02313101 [OMADA]WAN1: LCP negotiation succeeded. (MTU=1492, AUTH=<null>) 2024-06-10 12:24:32 pppoe_client[24702]: <5> 08214211 [OMADA]WAN1: PPPoE Discovery phase negotiation succeeded. (AC-MAC=MAC_ADDR, Session-ID=0x0590) 2024-06-10 12:24:30 pppoe_client[23724]: <5> 08214202 WAN1: PPPoE began connecting automatically. 2024-06-10 12:24:17 pppoe_client[16986]: <5> 08214206 WAN1: PPPoE session is over 2024-06-10 12:24:17 ppp[16986]: <4> 02313122 [OMADA]WAN1: LCP sending CONFIG-REQUEST timeout.
MAC_ADDR / PPPOE_USERNAME are censored by me for posting the comment here in the forum.
I have one more question, my ONT is the default gateway once the PPPOE connection is established, the default gateway IP is 192.168.100.1. I setup a MGMT VLAN weeks ago with VLAN ID 100 and the ER8411 has IP 192.168.100.254. This could cause conflicts being the ONT and another VLAN on the same subnet or it is correctly segregated/virtualized at Omada level? Otherwise, the system should prevent using the same IPs or provide an error message:
/**************************************************************************************************/
/* vnete ifconfig */
/**************************************************************************************************/
kern0.1 Link encap:Ethernet HWaddr MAC_ADDR1
inet addr:192.168.1.1 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: IPV6_ADDR1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1458111 errors:0 dropped:310 overruns:0 frame:0
TX packets:684744 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:109193077 (104.1 MiB) TX bytes:56727782 (54.0 MiB)
kern0.100 Link encap:Ethernet HWaddr MAC_ADDR1
inet addr:192.168.100.254 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: IPV6_ADDR1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:2600 (2.5 KiB)
kern0.200 Link encap:Ethernet HWaddr MAC_ADDR1
inet addr:192.168.200.1 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: IPV6_ADDR1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:35 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:2562 (2.5 KiB)
kern0.99 Link encap:Ethernet HWaddr XXX
inet addr:192.168.99.1 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: IPV6_ADDR1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:37 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:2762 (2.6 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:10 errors:0 dropped:0 overruns:0 frame:0
TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:680 (680.0 B) TX bytes:680 (680.0 B)
veth1 Link encap:Ethernet HWaddr MAC_ADDR_USED_BY_ER8411_FOR_PPPOE
inet addr:169.254.11.22 Bcast:0.0.0.0 Mask:255.255.255.252 <--- Fallback?
inet6 addr: fe80::IPV6_ADDR2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:682325 errors:0 dropped:0 overruns:0 frame:0
TX packets:775696 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:56391692 (53.7 MiB) TX bytes:61942817 (59.0 MiB)
I'll dig a bit on my available logs and let you know in case of further finding.