Switch suddenly disables stops delivering power to POE
Last night, the switch suddenly stopped delivering any power to POE devices. This is the second time this happened this quarter.
Only a hardware switch restart seems to help (or at least was the mehod that worked for me).
Here a CPU graph. At 1:01 AM, the controller logged the unavailability of the POE access points. The switch was restarted at just before 8 AM.
Any suggestions to avoid this in future?
- Copy Link
- Subscribe
- Bookmark
- Report Inappropriate Content
Hi @sb0373
Thanks for posting in our business forum.
sb0373 wrote
Hello @Clive_A ,
just now it happened again. This time during daytime, which is a bit inconvenient to reboot. Also firmware is 5.20.0 Build 20230818 Rel.72032.
Again, I forgot to check the LEDs. I really have to remember doing so.
This time, a soft restart did not help (I assume I did it via the controller all the other times).
After the soft restart, POE was still not available and all LEDs apart from the power LED (green) were off (literally no other LED was alight on the whole switch).
Hard restart (power cable out/in) made it come back.
Any suggestions what else I can send to support the analysis?
I have the logs before soft restart and before hard restart.
Best wishes,
sb0373
Judging from the log again, you might start an RMA with the local support team because it looks like this is a failure in the hardware. Replace one and check if this issue happens again or not.
- Copy Link
- Report Inappropriate Content
Hi @sb0373
Thanks for posting in our business forum.
How many PoE devices are connected to this switch? Number and model of them?
So you said this happened twice in a quarter, was this stable before? Did you add anything to the switch prior to the first time it happened?
Was there any firmware update before the first time?
My guess is out of the PoE budget. Can you take a screenshot of the remaining PoE power on the switch?
- Copy Link
- Report Inappropriate Content
Hello @Clive_A,
good questions!
> How many PoE devices are connected to this switch? Number and model of them?
4 PoE devices.
Port 12: Instar Camera
Port 15: EAP610-Outdoor
Port 17: Olimex ESP23-PoE Board
Port 18: EAP660 HD
> So you said this happened twice in a quarter, was this stable before? Did you add anything to the switch prior to the first time it happened?
> Was there any firmware update before the first time?
These two are the only occurrences so far. Unfortunately, I cannot really tell anymore what has changed when from the first time. If there were firmware updates available, I would have done it. I really don't remember when it first happened as I brushed it off and didn't think much of it. But it was recent enough to claim within a quarter.
One significant change I did was add an RTSP loop to port 1. So now port 1is disabled automatically. Unfortunately, I don't remember whether the first occurrence is before or after that change.
Temperatures started to drop this week but I doubt this was the case of the first time. Could this behavior occur if one PoE connection is damaged by water? The EAP610-Outdoor is (surprisingly) outside. Maybe water ran into the connection?
> My guess is out of the PoE budget. Can you take a screenshot of the remaining PoE power on the switch?
Is there a way to pull a detailed switch log of the last X weeks from somewhere? I couldn't figure it out.
- Copy Link
- Report Inappropriate Content
Hi @sb0373
Thanks for posting in our business forum.
Two files might be helpful.
1. Device info. Click the device and go to config, find the Device Info.
2. Running log. Global View.
Both are in plaint text.
- Copy Link
- Report Inappropriate Content
Thank you for that! Seems like the switch logs were cleared after the restart so I couldn't find anything interesting.
The controller logs only indicate the event by showing:
status change from Connected to Heartbeat Missed
Apart from that, no unexpected message.
Would you like me to send you the logs (I don't really want to post them here publicly).
I would otherwise wait until the issue shows up again and grab the logs before restarting the switch.
From the logs, I would expect the first occurrence of the issue to have happened either on
2023-07-23 at 15:00 UTC or 2023-10-16 13:37 UTC.
- Copy Link
- Report Inappropriate Content
Hi @sb0373
Thanks for posting in our business forum.
sb0373 wrote
Thank you for that! Seems like the switch logs were cleared after the restart so I couldn't find anything interesting.
The controller logs only indicate the event by showing:
status change from Connected to Heartbeat Missed
Apart from that, no unexpected message.
Would you like me to send you the logs (I don't really want to post them here publicly).
I would otherwise wait until the issue shows up again and grab the logs before restarting the switch.
From the logs, I would expect the first occurrence of the issue to have happened either on
2023-07-23 at 15:00 UTC or 2023-10-16 13:37 UTC.
Wait for your further reply and monitor the results. Will also update this information with the dev & test team and see what they suggest.
Also, please monitor the LED changes if you can catch this "down" time.
- Copy Link
- Report Inappropriate Content
Hello,
this night, this happened again after an uptime of 40day(s) 23h 54m 34s. Hopefully that matches with my previous past time frame :)
- Issue was detected by the controller at 2024-01-09 01:57:46 (CET) by indicating the disconnected APs. Interestingly, the missing AP message does not show up on the controller global alerts. Only on the site alerts. In my opinion, the device alerts should show up globally if you can also see the devices in the global known device list...
- The exported logs show a file time stamp of 0:03 although the containing syslog.log file seems to indicate an issue at a later time.
- cpuUtilizationThread.txt contains some high cpu usage but no time stamp. Also not sure how relevant this really is.
- syslog.log contains this:
#2024-01-09 02:22:28,[Link]/5/Gi1/0/16 changed state to up.
#2024-01-09 02:22:26,[Link]/5/Gi1/0/16 changed state to down.
#2024-01-09 02:22:18,[Link]/5/Gi1/0/16 changed state to up.
#2024-01-09 01:54:39,[LLDP]/6/Delete a neighbor from port Gi1/0/15.
#2024-01-09 01:54:28,[LLDP]/6/Delete a neighbor from port Gi1/0/18.
#2024-01-09 01:53:02,[PoE]/3/I2C read/write fail occurs on PSE 1.
#2024-01-09 01:53:02,[PoE]/3/I2C read/write fail occurs on PSE 0.
#2024-01-09 01:52:57,[PoE]/3/I2C read/write fail occurs on PSE 4.
#2024-01-09 01:52:55,[PoE]/3/I2C read/write fail occurs on PSE 3.
#2024-01-09 01:52:53,[PoE]/3/I2C read/write fail occurs on PSE 2.
#2024-01-09 01:52:52,[Link]/5/Gi1/0/18 changed state to down.
#2024-01-09 01:52:52,[Link]/5/Gi1/0/15 changed state to down.
#2024-01-09 01:52:52,[Link]/5/Gi1/0/17 changed state to down.
#2024-01-09 01:52:52,[Link]/5/Gi1/0/12 changed state to down.
#2024-01-09 01:52:52,[PoE]/3/I2C read/write fail occurs on PSE 5.
#2024-01-09 01:04:09,[Link]/5/Gi1/0/16 changed state to down.
#2024-01-09 01:03:05,[Link]/5/Gi1/0/16 changed state to up.
For this point, ignore "[Link]/5/Gi1/0/16". I only included it to have the continuous time series. - Overall, I do seem to have a strange "[Link]/5/Gi1/0/16" as it goes up and down multiple times very often. Can you tell me which port it is?
6 Sorry, I missed your message "Also, please monitor the LED changes if you can catch this "down" time.". I didn't do that... Will remember for next time as the switch is in a separate building.
How can I get the logs to you, if required? I can't upload them (I get a "Failed to upload" message here.
Thanks and best wishes,
sb0373
- Copy Link
- Report Inappropriate Content
Hello @Clive_A ,
just now it happened again. This time during daytime, which is a bit inconvenient to reboot. Also firmware is 5.20.0 Build 20230818 Rel.72032.
Again, I forgot to check the LEDs. I really have to remember doing so.
This time, a soft restart did not help (I assume I did it via the controller all the other times).
After the soft restart, POE was still not available and all LEDs apart from the power LED (green) were off (literally no other LED was alight on the whole switch).
Hard restart (power cable out/in) made it come back.
Any suggestions what else I can send to support the analysis?
I have the logs before soft restart and before hard restart.
Best wishes,
sb0373
- Copy Link
- Report Inappropriate Content
Hi @sb0373
Thanks for posting in our business forum.
sb0373 wrote
Hello @Clive_A ,
just now it happened again. This time during daytime, which is a bit inconvenient to reboot. Also firmware is 5.20.0 Build 20230818 Rel.72032.
Again, I forgot to check the LEDs. I really have to remember doing so.
This time, a soft restart did not help (I assume I did it via the controller all the other times).
After the soft restart, POE was still not available and all LEDs apart from the power LED (green) were off (literally no other LED was alight on the whole switch).
Hard restart (power cable out/in) made it come back.
Any suggestions what else I can send to support the analysis?
I have the logs before soft restart and before hard restart.
Best wishes,
sb0373
Judging from the log again, you might start an RMA with the local support team because it looks like this is a failure in the hardware. Replace one and check if this issue happens again or not.
- Copy Link
- Report Inappropriate Content
Thank you, I also started thinking along the lines but hoping it could be fixed via software.
- Copy Link
- Report Inappropriate Content
Information
Helpful: 0
Views: 924
Replies: 9
Voters 0
No one has voted for it yet.