Business Community > Controllers > Controller offline (port 27217 is already in use)
< Controllers

Controller offline (port 27217 is already in use)

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.
12

Controller offline (port 27217 is already in use)

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.
Controller offline (port 27217 is already in use)
Controller offline (port 27217 is already in use)
2021-03-08 08:15:34
Model: Omada Software Controller (Linux)  
Hardware Version:
Firmware Version: 4.2.11

I'm running the controller on Ubuntu 18.04 server. From time to time, the controller goes offline. It probably crashes or tries to restart itself but it does not stop also Mongo and when it tries to start again and Mongo is already running, it stops. Then "tpeap start" fails with:

 

2021-03-08 00:07:57 [log4j-thread] [INFO]-[SourceFile:29] - success to load configuration omada.properties
2021-03-08 00:07:58 [main] [INFO]-[SourceFile:89] - going to start local mongod.
2021-03-08 00:07:59 [main] [ERROR]-[SourceFile:109] - Failed to init_port 27217 is already in use.
com.tplink.omada.start.a.a: port 27217 is already in use.
    at com.tplink.omada.start.b.b.a(SourceFile:45) ~[omada-start.jar:?]
    at com.tplink.omada.start.b.d.d(SourceFile:106) ~[omada-start.jar:?]
    at com.tplink.omada.start.b.b.e(SourceFile:72) ~[omada-start.jar:?]
    at com.tplink.omada.start.OmadaBootstrap.d(SourceFile:223) ~[omada-start.jar:?]
    at com.tplink.omada.start.OmadaBootstrap.q(SourceFile:287) ~[omada-start.jar:?]
    at com.tplink.omada.start.OmadaBootstrap.a(SourceFile:100) [omada-start.jar:?]
    at com.tplink.omada.start.OmadaBootstrap.e(SourceFile:245) [omada-start.jar:?]
    at com.tplink.omada.start.OmadaLinuxMain.b(SourceFile:80) [omada-start.jar:?]
    at com.tplink.omada.start.OmadaLinuxMain.start(SourceFile:43) [omada-start.jar:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_282]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
    at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243) [commons-daemon-1.0.15.jar:1.0.15]

When I kill Mongo and try "tpeap start" again, it succeeds. This is quite annoying . Is there something that I can do about it or it needs to be fixed in the controller?

  0      
 
  0      
 
#1
Options
11 Reply
Re:Controller offline (port 27217 is already in use)
2021-04-07 10:22:16

It happens again and again... No one else has the same problem? No official reaction?

  0  
  0  
#2
Options
Re:Controller offline (port 27217 is already in use)
2021-04-08 03:34:12 - last edited 2021-04-08 04:04:58

Dear @ZdenekM,

 

ZdenekM wrote

I'm running the controller on Ubuntu 18.04 server. From time to time, the controller goes offline. It probably crashes or tries to restart itself but it does not stop also Mongo and when it tries to start again and Mongo is already running, it stops. Then "tpeap start" fails with:

When I kill Mongo and try "tpeap start" again, it succeeds. This is quite annoying . Is there something that I can do about it or it needs to be fixed in the controller?

 

Could you please export the running log for checking?

 

Edit: Port 27217 is used for MongoDB. To address the problem, you may also check if there are other applications/processes occupying this port. If port 27217 is already occupied by some other applications/processes, the MongoDB cannot be initialed, and thus it will lead to the failure of the Controller.

>> Omada EAP Firmware Trial Available Here << *Try filtering posts on each forum by Label of [Early Access]*
  0  
  0  
#3
Options
Re:Controller offline (port 27217 is already in use)
2021-04-08 06:54:53

@Fae Thank you for your response. I'm attaching the logs. Sure, I know that port 27217 is used by Mongo :) What periodically happens is this:

  • I find that the controller is not running.
  • Try to start it but it fails.
  • The log says it is because port 27217 is occupied.
  • "lsof -i:27217" tells me (see below) that the port is used by mongod started by the user omada
  • When that instance is killed, the controller can start ok.

 

My suspicion is that the controller tries to restart itself or crashes from time to time... When it goes down, it fails (or even does not attempt) to stop the mongod. When it tries to start up again, it tries to also start mongod, which is already running. It should take down mongod beforehand or it should be able to use the running instance.

 

 

 

# sudo lsof -i:27217
COMMAND   PID  USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
mongod  15370 omada   11u  IPv4 734626926      0t0  TCP localhost:27217 (LISTEN)
File:
logs_2021-04-08_08-45-13.zipDownload
  0  
  0  
#4
Options
Re:Controller offline (port 27217 is already in use)
2021-04-08 10:20:56

Dear @ZdenekM,

 

ZdenekM wrote

I'm attaching the logs. Sure, I know that port 27217 is used by Mongo :) What periodically happens is this:

  • I find that the controller is not running.
  • Try to start it but it fails.
  • The log says it is because port 27217 is occupied.
  • "lsof -i:27217" tells me (see below) that the port is used by mongod started by the user omada
  • When that instance is killed, the controller can start ok.

 

The main issue is that the Controller crashed and you try to restart, resulting in the conflict with MongoDB.

From the running log, I don't find any clues why the controller stopped working.

 

For your case, if you have MongoDB set to auto startup, you may disable the MongoDB started by the system, making Omada Controller start MongoDB by itself. If it doesn't help, you may try to reinstall the Omada Controller by referring to the guide below.

How to install Omoada SDN controller on Linux system(above Controller 4.1.5)

>> Omada EAP Firmware Trial Available Here << *Try filtering posts on each forum by Label of [Early Access]*
  0  
  0  
#5
Options
Re:Controller offline (port 27217 is already in use)
2021-04-08 14:01:53

@Fae The system-started instance of mongod runs on a different port (27017). You may see from the output of "lsof" command, that port 27217 is blocked by mongod started by the "omada" user (e.g. started by the controller itself).

  0  
  0  
#6
Options
Re:Controller offline (port 27217 is already in use)
2021-04-08 14:06:48

Btw, it would be nice if the controller can be automatically restarted when it crashes. Making it a systemd service would do the trick quite easily and will be the more standard approach I would say :-) Also, it would be nice if the configuration could be retained when updating the controller version - now when I e.g. change the port number in /opt/tplink/EAPController/properties/omada.properties, the change is lost when I update the controller package.

  0  
  0  
#7
Options
Re:Controller offline (port 27217 is already in use)
2021-04-13 10:56:49 - last edited 2021-04-13 10:59:44

Dear @ZdenekM,

 

ZdenekM wrote

Btw, it would be nice if the controller can be automatically restarted when it crashes. Making it a systemd service would do the trick quite easily and will be the more standard approach I would say :-) 

 

Sorry that the Omada Controller doesn't support being auto-restarted when it crashes at present.

I'll forward this feedback to the R&D team for further investigation.

 

Back to your case, if the issue still exists, please try to check the service status(command: service tpeap status) and tpeap log(command: journalctl -u tpeap) after the controller crashes, we'll see if we can find something and address the problem.

 

In addition, when you try to fix the problem, did you kill MongoDB by "tpeap stop" or how do you kill it?

>> Omada EAP Firmware Trial Available Here << *Try filtering posts on each forum by Label of [Early Access]*
  0  
  0  
#8
Options
Re:Controller offline (port 27217 is already in use)
2021-04-13 11:30:31

@Fae When controller crashes, "tpeap status" returns something like "The controller is not running". "journalctl -u tpeap" returns no log items, but currently the controller has not crashed so maybe there will appear something after the crash - I will try it next time. I checked quite carefully logs in /opt/tplink/EAPController/logs and found nothing suspicious. I'm killing mongo by "kill mongo_pid" (where mongo_pid is found using lsof). The crash usually happens at night.

  0  
  0  
#9
Options
Re:Controller offline (port 27217 is already in use)
2021-04-14 02:55:12

Dear @ZdenekM,

 

ZdenekM wrote

When controller crashes, "tpeap status" returns something like "The controller is not running". "journalctl -u tpeap" returns no log items, but currently the controller has not crashed so maybe there will appear something after the crash - I will try it next time. I checked quite carefully logs in /opt/tplink/EAPController/logs and found nothing suspicious. I'm killing mongo by "kill mongo_pid" (where mongo_pid is found using lsof). The crash usually happens at night.

 

I see, thank you for your kind reply. If you have any update on the case, please feel free to share with us.

>> Omada EAP Firmware Trial Available Here << *Try filtering posts on each forum by Label of [Early Access]*
  0  
  0  
#10
Options
Re:Controller offline (port 27217 is already in use)
2021-04-16 06:46:42

@Fae Ok, so now I know where the problem is! The controller crashed and dmesg shows that the system ran out of memory and jsvc was killed. So maybe it could be a memory leak? But it is hard to say, the memory could be occupied by another process. Also, the system has only 2 GB of RAM so I will try to add more to see if it helps.

 

I'm using the following java version:

 

$ java -version
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-8u282-b08-0ubuntu1~18.04-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)

 

 

File:
dmesg-oom.txtDownload
  0  
  0  
#11
Options
12

Information

Helpful: 0

Views: 12782

Replies: 11

Related Articles
 Clients to use controller hostname
576 1
 Omada software controller 4.1.5 - Port 80 and port 443 are already in use
3396 0
Use Fixed IP Address not working.
4473 0
how is the correct Syntax to use Custom DDNS
3105 0
IPv4 WAN shows being offline - DynDns not updating - ER8411
1263 0
Can't start controller after port change
2180 0
About Us
Press
Copyright © TP-Link Systems Inc. All rights reserved.