Business Community > Controllers > ER605 Needs weekly Reboot Issue - does it have a memory leak?
< Controllers

ER605 Needs weekly Reboot Issue - does it have a memory leak?

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.
123

ER605 Needs weekly Reboot Issue - does it have a memory leak?

This thread has been locked for further replies. You can start a new thread to share your ideas or ask questions.
26 Reply
Re:ER605 Needs weekly Reboot Issue - does it have a memory leak?
2023-03-16 02:25:46 - last edited 2023-03-16 02:27:08

  @d0ugmac1 I don't experience this at all or as much probably due to different number of clients and workload. My ER605v1 is still sitting at 39% memory with 40 days of uptime.

 

I use standalone mode which has an option to "export diagnostic information" which includes the output of "top" to see per-process memory usage. Does the controller include a similar option so that you could export the diagnostic info immediately after a reboot and then again a few days later to see the difference?

 

Here's a snippet from my diagnostic export today:

  PID USER       VSZ STAT COMMAND

 8098 root     11772 S    nginx: master process /usr/sbin/nginx
 8099 root     13636 S    nginx: worker process

 8999 root     16292 S    /usr/sbin/omadad

 

The largest memory consumers are usually the nginx processes (nginx) and the omada daemon process (omadad) so I suspect that's where the difference would be observed.

  0  
  0  
#13
Options
Re:ER605 Needs weekly Reboot Issue - does it have a memory leak?
2023-03-20 16:26:41 - last edited 2023-03-20 16:27:01

  @d0ugmac1 

 

Well this is a new development today...processor spikes and unresponsive periods

 

unresponsive

 

 

 

 

<< Paying it forward, one juicy problem at a time... >>
  0  
  0  
#14
Options
Re:ER605 Needs weekly Reboot Issue - does it have a memory leak?
2023-03-20 16:37:29

  @d0ugmac1 

Reported Physical and Virtual Memory continue to rise:

 

rising

 

 

and the buffers are behaving weirdly

 

jitter

 

<< Paying it forward, one juicy problem at a time... >>
  0  
  0  
#15
Options
Re:ER605 Needs weekly Reboot Issue - does it have a memory leak?
2023-03-20 18:16:05 - last edited 2023-03-20 18:20:18

  @d0ugmac1 

 

I believe I am now in 'end stage' disconnect-itis (I am at 21 days 11h since last reboot)

 

Controller seems to have lost ability to determine/report latency as of 3AM today (exactly 21 days since last reboot)

 

latency lost

 

Controller now also starting to report disconnects:

 

disconnects

 

I am now also seeing regular drops of the L2TP/IPsec tunnel to the remote site (logs too sensitive to post)

 

yet Controller logs don't look alarming at all...

 

not alarming?

 

Router still shows as Connected, internet access still working....nothing alarming looking

 

boring

 

 

More updates very soon I'm sure.

 

<< Paying it forward, one juicy problem at a time... >>
  0  
  0  
#16
Options
Re:ER605 Needs weekly Reboot Issue - does it have a memory leak?
2023-03-21 12:37:56

  @d0ugmac1 

 

Router has started to disconnect, ~8-10 events in the last 24h, largest was about 1h around midnight last night.  It continues to pass traffic, and has come back online after each outage so far.  Pings continue to work, and it continues to forward traffic and self collect stats, but controller, http and ssh are all offline when disconnected.

 

I pulled an SNMP walk and found some messages about low swap and high loads.  That is corroborated by some of the SNMP polled data shown below:

 

Some of the spikes may be related to the re-establishment of an L2TP/IPsec tunnel each time the router reconnects after being disconnected.  Periods of initial disconnections hard to determine but typically < 5min.

 

high loads

 

<< Paying it forward, one juicy problem at a time... >>
  0  
  0  
#17
Options
Re:ER605 Needs weekly Reboot Issue - does it have a memory leak?
2023-03-21 17:39:48 - last edited 2023-03-21 18:22:43

  @d0ugmac1 

 

Looks like something snapped after the last 1hr disconnect...router is showing as Disconnected, and I don't expect it to come back

 

fds

 

...though SNMP is still active.

 

<< Paying it forward, one juicy problem at a time... >>
  0  
  0  
#18
Options
Re:ER605 Needs weekly Reboot Issue - does it have a memory leak?-Solution
2023-03-22 13:21:06 - last edited 2023-03-22 13:22:30

Conclusion.  OpenVPN logging resulting in memory starvation.  After spending some quality time online with the TPlink Support team last night, it appeared that I had amassed ~30MB of OpenVPN logs in just 3 weeks (a problem on a device with only 128M of RAM), despite not a single vpn connection during that time...ironic if you think how many people would have like to have seen what was in those logs, but regardless, it looks to be the culprit.  Interestingly, it was only HTTP and the Controller interface that went offline, SNMP and SSH were working fine.  I have since deleted my OpenVPN settings from the router and have rebooted.  TPlink should be issuing a beta firmware version with the verbosity dialed back and hopefully some kind of log rotation increase as well.  I'll monitor for another few weeks just to be sure, but I think we can close the book on this 'colourful' episode.

<< Paying it forward, one juicy problem at a time... >>
Recommended Solution
  0  
  0  
#19
Options
Re:ER605 Needs weekly Reboot Issue - does it have a memory leak?
2023-05-01 15:54:00

@Hank21 

 

Flagging that memory behaviour is getting worrisome again. Will continue to monitor and will request ticket if it hits starvation levels.

Memory buffers are already basically back to nothing again which was the precursor to the last Disconnect event.

 

(Note, I have installed the latest 1.2.3 beta software for the ER605v1 some weeks ago)

Here's where we are at today for overall memory levels...which creep up inexorably:

 

 

 

<< Paying it forward, one juicy problem at a time... >>
  0  
  0  
#20
Options
Re:ER605 Needs weekly Reboot Issue - does it have a memory leak?
2023-05-02 04:18:43 - last edited 2023-05-02 04:21:25

  @d0ugmac1 

 

More evidence that my ER605 is leaning towards disk/swap/cache starvation and eventual disconnect...I/O waits have really started to tick upwards now...

 

<< Paying it forward, one juicy problem at a time... >>
  0  
  0  
#21
Options
Re:ER605 Needs weekly Reboot Issue - does it have a memory leak?
2023-05-04 03:38:07

Hello @d0ugmac1 

 

Are you using the 1.2.3 Build 20230413 Beta firmware? 

ER605 V1_1.2.3_Build 20230413 Beta Firmware For Trial (Released on Apr 14th, 2023)

>> Omada EAP Firmware Trial Available Here << *Try filtering posts on each forum by Label of [Early Access]*
  0  
  0  
#22
Options
123
Related Articles
ER605 slow to reboot
605 0
Scheduled reboot for ER605 v2
371 0
ER605 web access issue
1552 0
ER605 prefers secondary wan on reboot
378 0
is it possible to set up a second ER605 in Omada?
909 0
ER605 V2.0 - OpenVPN Client Issue
379 0
About Us
Press
Copyright © TP-LINK CORPORATION PTE. LTD. All rights reserved.