Ocassional Dead Loop in Omada Controller runtime on Linux

Ocassional Dead Loop in Omada Controller runtime on Linux

Ocassional Dead Loop in Omada Controller runtime on Linux
Ocassional Dead Loop in Omada Controller runtime on Linux
a week ago - last edited a week ago
Hardware Version:
Firmware Version: 5.15.24.19

Our Omada Application on Rocky Linux 8.10 (Giada Mi-NAS25 Motherboard) seem to go in a infinite dead loop occupying full CPU and cannot come out of it, until Omada is killed and restarted. We notice that when this happens the Mongo DB usage climbs up abruptly till it peaks out at some value.



We noticed that their are some heap related exceptions in startup.log file. Their are also lot of exceptions in the server log file.  If memory is less, how much it should be (just not increase randomly by 256 MB or double) as this type of configuration only shifts the problem away by few weeks and this happens even when 2 GB heap space is configured.

Unable to upload log files here as the Forum Interface would not permit the same.

 

  0      
  0      
#1
Options
2 Reply
Re:Ocassional Dead Loop in Omada Controller runtime on Linux
Friday

  @APRC-P3-Tel 

Thank you for your post.
Was this issue present the very first time you installed the Omada Controller, or did it only start after the system had been running normally for a while?
Also, could you describe in detail what exactly you mean by “infinite dead-loop”? (For example: one particular Java thread pegs a core at 100 % forever, the web UI becomes unresponsive, MongoDB keeps recreating the same cursor, etc.)

As a quick test you might back up the data and work folders, completely remove the controller, re-install the same (or a newer) version, restore the backup, and see if the symptom disappears.
If the log files are too large to attach here, please upload them to any file-sharing service (Google Drive, Dropbox, WeTransfer, etc.) and post the link—we’ll be happy to take a look.

  0  
  0  
#2
Options
Re:Ocassional Dead Loop in Omada Controller runtime on Linux
Yesterday

  @Ethan-TP : When the controller starts up completely, Its CPU usage is low (5%). It tuns nornally at 5-10% CPU if their are no managment plane interactions. IF we interact using mobile app, web app, etc with the controller, its CPU usage momentarilty shoots up to 100% and within a few seconds falls back to the nor,al (5-10%) level. It runs like this for days, weeks and even months, if left undisturbed.

However if their are heavy ineractions from mangement plane, at some instant the CPU usage becomes 100% and does not come down to the normal level even if left alone for hours. Obsviously at this time problems happen in network, web UI and mobile app become unresponsive, wifi traffic falls (means its main function impacted) and we have to restart this controller to restre the network health. THis is what I mean by infinite dead loop. I also noticed when this event of dead loop happens, the mongo DB usage has also preaked abnormally (from the 400MB+ we see mongo db using it shoots up to ~650 MB)


The logs are uploaded and shared here:

https://drive.google.com/drive/folders/1S2hxhueh2bWZo_TbIyzsuY2hW3cnTCSr?usp=sharing

 



 

  0  
  0  
#3
Options