Ocassional Dead Loop in Omada Controller runtime on Linux
Ocassional Dead Loop in Omada Controller runtime on Linux
Our Omada Application on Rocky Linux 8.10 (Giada Mi-NAS25 Motherboard) seem to go in a infinite dead loop occupying full CPU and cannot come out of it, until Omada is killed and restarted. We notice that when this happens the Mongo DB usage climbs up abruptly till it peaks out at some value.
We noticed that their are some heap related exceptions in startup.log file. Their are also lot of exceptions in the server log file. If memory is less, how much it should be (just not increase randomly by 256 MB or double) as this type of configuration only shifts the problem away by few weeks and this happens even when 2 GB heap space is configured.
Unable to upload log files here as the Forum Interface would not permit the same.
- Copy Link
- Subscribe
- Bookmark
- Report Inappropriate Content
We have identified the root cause as follows:
- Auto-backup is enabled and the options to back up client data, logs, etc. are checked.
- The large volume of data being exported triggers the issue.
- The anomaly occurs around the scheduled auto-backup time.
Work-arounds:
- Edit the auto-backup settings: un-check “Client”, “Alert”, and “Audit Log” export options. This will eliminate the problem immediately; a permanent fix will be included in a later release.
- Copy Link
- Report Inappropriate Content
Thank you for your post.
Was this issue present the very first time you installed the Omada Controller, or did it only start after the system had been running normally for a while?
Also, could you describe in detail what exactly you mean by “infinite dead-loop”? (For example: one particular Java thread pegs a core at 100 % forever, the web UI becomes unresponsive, MongoDB keeps recreating the same cursor, etc.)
As a quick test you might back up the data and work folders, completely remove the controller, re-install the same (or a newer) version, restore the backup, and see if the symptom disappears.
If the log files are too large to attach here, please upload them to any file-sharing service (Google Drive, Dropbox, WeTransfer, etc.) and post the link—we’ll be happy to take a look.
- Copy Link
- Report Inappropriate Content
@Ethan-TP : When the controller starts up completely, Its CPU usage is low (5%). It tuns nornally at 5-10% CPU if their are no managment plane interactions. IF we interact using mobile app, web app, etc with the controller, its CPU usage momentarilty shoots up to 100% and within a few seconds falls back to the nor,al (5-10%) level. It runs like this for days, weeks and even months, if left undisturbed.
However if their are heavy ineractions from mangement plane, at some instant the CPU usage becomes 100% and does not come down to the normal level even if left alone for hours. Obsviously at this time problems happen in network, web UI and mobile app become unresponsive, wifi traffic falls (means its main function impacted) and we have to restart this controller to restre the network health. THis is what I mean by infinite dead loop. I also noticed when this event of dead loop happens, the mongo DB usage has also preaked abnormally (from the 400MB+ we see mongo db using it shoots up to ~650 MB)
The logs are uploaded and shared here:
https://drive.google.com/drive/folders/1S2hxhueh2bWZo_TbIyzsuY2hW3cnTCSr?usp=sharing
Their are Java exceptions in both startup and server.log which surface once controller has been running for a day or two. Despite the exceptions the controller appears to run fine, till one day it goes into that dead loop type of state.
- Copy Link
- Report Inappropriate Content
1. Is this issue reproducible on a consistent basis?
2. Kindly provide the log files from the path opt/tplink/EAPController/data for further analysis.
- Copy Link
- Report Inappropriate Content
@Ethan-TP
1. The issue is reproducible
2. The Logs have been shared in Google drive in previous message post.
- Copy Link
- Report Inappropriate Content
The log you just provided does not seem to come from the path
opt/tplink/EAPController/data.
- Copy Link
- Report Inappropriate Content
@Ethan-TP :
In my setup their are no logs in the folder you indicated:
- Copy Link
- Report Inappropriate Content
Please provide the following dump file so R&D can identify the root cause of the issue:
/opt/tplink/EAPController/logs/java_heapdump.hprof
Also, could you tell me how many devices are currently managed under this organization?
- Copy Link
- Report Inappropriate Content
@Ethan-TP : Here is the dump you requested:
https://drive.google.com/file/d/1n6ZMV0N5h0waWXcwPw0ALc4lMyKI5pEN/view?usp=drive_link
Its a 1.4 GB file.
62 APs are being managed by this partcular controller, plus 2 switch and 1 Gateway.
- Copy Link
- Report Inappropriate Content
Thank you so much for taking the time to post the issue on the TP-Link community!
To better assist you, I've created a support ticket via your registered email address and escalated it to our support engineer to look into the issue. The ticket ID is TKID251133498 please check your email box and ensure the support email is well received. Thanks!
Once the issue is addressed or resolved, welcome to update this topic thread with your solution to help others who may encounter the same issue as you did.
Many thanks for your great cooperation and patience!
- Copy Link
- Report Inappropriate Content
We have identified the root cause as follows:
- Auto-backup is enabled and the options to back up client data, logs, etc. are checked.
- The large volume of data being exported triggers the issue.
- The anomaly occurs around the scheduled auto-backup time.
Work-arounds:
- Edit the auto-backup settings: un-check “Client”, “Alert”, and “Audit Log” export options. This will eliminate the problem immediately; a permanent fix will be included in a later release.
- Copy Link
- Report Inappropriate Content
Information
Helpful: 0
Views: 314
Replies: 11
Voters 0
No one has voted for it yet.