Omada Software Controller running out of RAM

Omada Software Controller running out of RAM
Omada Software Controller running out of RAM
2022-06-23 14:24:01 - last edited 2022-06-26 13:03:58
Model:Adapter
Hardware Version: V1
Firmware Version: 5.1.7

I'm running Omana Software Controller on a RPi 3B (1GB RAM) for years now. And Version4 was running really well.

But since the upgrade to Omada Version 5, which needed a clean install, my RPi runs out of RAM very fast and then isn't responsive at all, no SSH, nothing... (no screen connected, so I don't know what would be there).

 

Do you know any configs to size down the controllers resource hunger?

 

My RPi is running on 64bit Ubuntu.

I used htop to inspect the process and it eats up all the RAM.

 

I just run a very small network with 2 EAPs and a Portal for guest to login

0
0
#1
Options
1 Accepted Solution
Re:Omada Software Controller running out of RAM-Solution
2022-06-26 10:33:01 - last edited 2022-06-26 13:03:58

  @Khaos 

 

Hey

 

Yeah the 1gb of RAM is going to be your issue here, no amount of cutting or fiddling is going to reduce it that low.     Moores Law would indicate that hardware doubles every 18 to 24 months, this has proven to be true to an extent and the Pi3 came out in 2016

 

Unfortunately its simply too old a device now, you should be looking for one with at least 4gb RAM to run version 5 of SDN

Recommended Solution
0
0
#2
Options
11 Reply
Re:Omada Software Controller running out of RAM-Solution
2022-06-26 10:33:01 - last edited 2022-06-26 13:03:58

  @Khaos 

 

Hey

 

Yeah the 1gb of RAM is going to be your issue here, no amount of cutting or fiddling is going to reduce it that low.     Moores Law would indicate that hardware doubles every 18 to 24 months, this has proven to be true to an extent and the Pi3 came out in 2016

 

Unfortunately its simply too old a device now, you should be looking for one with at least 4gb RAM to run version 5 of SDN

Recommended Solution
0
0
#2
Options
Re:Omada Software Controller running out of RAM
2022-06-26 11:12:28 - last edited 2022-06-26 11:26:56

  @Khaos I can confirm that without additional steps rpi3 is running out of memory with v5. 

 

Solution: create and use larger swap file, 1gb is sufficient.

 

Here is one way of how you can create it:

 

1. If you already use some swap file like some small 64mb, you need first to swap off olf swapfile, if you do not use swap, skip this step

 

# swap off

swapoff /swapfile

 

1.1 remove swapped off swapfile from /etc/fstab manually or with sed and delete it

sed "/swapfile/d" /etc/fstab

# delete old swap

rm -f /swapfile

 

2. create new swap file of one GB

fallocate -l 1G /swapfile

chmod 600 /swapfile

mkswap /swapfile

swapon /swapfile

 

3. add it to the fstab

 

cat << EOF | tee -a /etc/fstab

# Swapfile on Root

/swapfile swap swap defaults 0 0

EOF

 

 

That would be it, if you want to add it as script, then set SWAPFILE and SWAPSIZE vars and run this:

 

SWAPFILE=swapfile

SWAPSIZE=1G

 

if [ -f /$SWAPFILE ]; then

echo "Swap off old swapfile"

swapoff /$SWAPFILE

echo "remove entry from fstab if exists"

sed "/$SWAPFILE/d" /etc/fstab

echo "delete old swap"

rm -f /$SWAPFILE

fi

echo "Fallocate swapfile in choosen size and create a swap file"

fallocate -l ${SWAPSIZE} /$SWAPFILE

chmod 600 /$SWAPFILE

mkswap /$SWAPFILE

swapon /$SWAPFILE

echo "Add new swap to fstab entry so that it is mounted on each reboot"

cat << EOF | tee -a /etc/fstab

# Swapfile on Root

/$SWAPFILE swap swap defaults 0 0

EOF

echo "Show free memory"

free -h

 

For those who want to tune their swapfile, check online guides for swappiness and other possible settings on how to tell your rpi3 to use swap as less as possible, especially because sd cards are not good for swapping memory. However, with 1G (probably 128MB would be sufficient too) rpi3 runs smoothly without crashes.

2
2
#3
Options
Re:Omada Software Controller running out of RAM
2022-06-26 11:20:01

  @Philbert 

Philbert wrote

Unfortunately its simply too old a device now, you should be looking for one with at least 4gb RAM to run version 5 of SDN

 

I would probably disagree with 4GB to be "at least", it is simply expensive, 1GB (if still on sale) is a lot cheaper compared to ocXXX, rpi's 4 cpu and performance beats in dust as example oc200. Beside that, rpi4 is much faster and when you take a possibility to buy radxa's rock (they are now expensive) with 1G, then you can have your swapfile on an m2 disk too. That way rpi4 with 1G RAM can work for years without issue. RAM is most expensive on those SoC boards and not only rpi but other too hesitated to bring it up to 8GB because the price gets then that high, which contradicts to reasons why most use those SoC boards.

0
0
#4
Options
Re:Omada Software Controller running out of RAM
2022-06-26 11:37:20

  @btx 

 

Hey mate

 

Perhaps it's the network engineer in me, but I totally disagree with that.     A 4gb Pi is like what, €45?  that will be good for another 4-6 years it seems like a done deal to me.  

 

Chugging a load of swop file at it, while this may work it's only a fudge in my opinion.  Ultimately if you had sufficient hardware then you wouldn't need to start increasing swop files.   Yes I have done that before for other things previously, but as you say yourself SDs are not reliable for swop and you may well start burning them.     Yes a radxa's rock  with a M2 would be more than perfect, but as you say its going to cost more than a Pi 4gb or a OC200.

 

For all the cost involved of replacing a 6 year old piece of hardware, just seems a waste of time personally.    Yes the OC200 is more expensive, but again it's not mega money

0
0
#5
Options
Re:Omada Software Controller running out of RAM
2022-06-26 11:45:57 - last edited 2022-06-26 11:52:25

  @Philbert hello Philbert, thanks for sharing your view. Of course you are correct from your perspective, but it is still subjective even if from engineer. My point comes out of developers sole as we still speak about software and its requirements which will not change in quite some time in regards of omada and rpi3, the cpu of rpi3 is more than powerful enough. If RAM can be reduced, of course it can be on tp-links costs and maybe some stability issues (which we do have anyway with omada) but that could potentially bring new bugs and kill some features. As example, strip it of facebook and all those other things which users do not want, use maybe system binaries instead of java code, shorten your variables to math. min possible size, etc .... . Considering everybody can create a swap file, I guess tp-link will not optimize it specifically for low ram devices, but they could bring out lite version of it, especially after they launched cloud based controller.

 

I do agree with your view, but as a developer or engineer one needs to define min. requirements and rpi3 still fits the purpose, rpi4 with 1G is more than sufficient for omada for next ten years, that is at least my subjective opinion on min. requirements.

 

About costs: I developed in crypto few things, and in few of my teams people invested into renting servers where I invested into purchasing, that's how I ended with quite a big bunch of rpi3's, pi zero's, rock4's, hystou's .... , amazingly they all ROI'd in around 3-6 months, basically making them free for me. So far, again, costs view is again subjective and so far hardware does not cost me anything. But there is something I would badly disagree with you, if someone uses min. requirements, like rpi3, then it is mostly not due to the fact that the owner has not $ 50 to throw away, you seem to forget that you were student too. On other side, rpi3 is and will never be used in a business, meaning we really talk about customers for who $ 50 might be their monthly wage or simply somebody who has some rpi3 laying around and does not want to spend additional $ 50 for controller, standalone would be good, omada is luxus, therefore, why  spending anything at all instead of letting it run to see how it works and when it works, there is no need for hardware replacement?

2
2
#6
Options
Re:Omada Software Controller running out of RAM
2022-06-26 12:00:27

  @btx 

 

As I was reading your reply I thought, "this is like being in work!"..  laugh  Pretty much sums up the daily debate between dev and networks.

 

Again you are also correct too, it would be good to have the option to stop certain services and features on-demand.  I know on one of the other vendors you could stop the tomcat server, that would bring down the RAM and CPU massively but still allow the controller to function, all be it without GUI.  When you needed it you could SSH in and re-start it.   Things like that are perfect for this, but seems unlikely to happen sadly. 

 

 

 

 

1
1
#7
Options
Re:Omada Software Controller running out of RAM
2022-06-26 12:10:12

  @Philbert you can take any openwrt capable device, flash first some release fw on it (that is with gui, luci), then measure performance etc.. . Next flash snapshot version (without gui, no luci) and do the same. You will quickly recognize things like on stable vpn gives 40Mbit, on snapshot which should be unstable 80Mbit. The lower the specifications of device, the bigger is the difference, best is to take old 4MB devices to see hige differences. Same would apply to omada, but omada can not be controller over terminal/ssh and their api is still in works which potentially could allow everybody to run terminal like controller. Even if tp-link ever releases something in that direction, that will not happen in next time except if they extend controller software to kinda outsource some heavy tasks to cloud. That is already now very makable and quickly makable solution for low ram devices.

2
2
#8
Options
Re:Omada Software Controller running out of RAM
2022-06-26 13:03:45

  @btx thank you for the input on this issue. I really appreciate it!

I just went and bought a rpi4 with 8gb ram to throw money at the problem :'D 

And it worked very nicely.

 

I'm sure I find another purpose for the rpi3 soon

0
0
#9
Options
Re:Omada Software Controller running out of RAM
2022-06-26 13:11:31 - last edited 2022-06-26 13:34:14

  @Khaos congrats on new rpi ;), I am also sure that rpi3 will find some purpose, just make sure it does not get dusty

 

@Philbert do you maybe have some broken tablet, phone, pc or anything with emmc card in it? If yes, take it out and place it into emmc to sd adapter, they are quite cheap and available if you do not have any (I could send you one, but post fee would probably be X times of its value). Now consider using emmc card over adapter on rpi3 instead of SD card :). Nothing will burn, no replacement since ... I think 2014. Beside that, biggest bottleneck on rpi3 is actually its storage, sd card and its low performance. However, using emmc card brings huge difference for a user, rpi3 is much more responsive and faster, resulting in very smooth experience. Like I said, I never purchased emmc cards, I just took them out of broken devices which were thrown away, but you can buy 32gb, they are quite expensive.

 

 

In fact, the only purpose for which I use rpi3's are to give them for free to all those people who asked me to help them setup their omada where they had tp-link hardware but no clue about omada, sparing them costs. All together I think already over 10 rpi3's are running omada controller, no issues for now, nobody of them is looking/asking for new or oc device, no crashes, no issue in logs, all fine. Biggest network rpi3 manages is with 10 ap's and around 250 users.

0
0
#10
Options
Re:Omada Software Controller running out of RAM
2022-06-26 13:43:42

  @btx you seam really nice.

I was really happy with the rpi3 running omada controller version 4. But I like to stay up to date.

 

Did you change the swap file on all of those rpi3s? 

0
0
#11
Options