WARNING: System memory is running low. Client device doesn't get an IP address.

ghstrhtrsbhtr · ‎05-13-2019

Hi,

I've got 9 2802i access points setup using mobility express that works quite well for the most part - on average we have 170 devices connected and working properly. We are using ME ver. 8.8.120

However, we are regularly having trouble with devices associating to the AP, but not getting an IP address. If I remote into the AP and show the log, I see many warnings about running out of memory. If I reboot the AP, devices can join the AP properly again. I've seen this issue on about 4 of our APs, so I don't think hardware is the issue.

May 14 07:11:58 apsw_watchdog: WARNING: System memory is running low (free memory: 88 MB)!!
May 14 07:11:58 kernel: [*05/13/2019 21:41:58.8066]
May 14 07:11:58 kernel: [*05/13/2019 21:41:58.8066] !!!!! {watchdogd} WARNING: System memory is running low (free memory: 88 MB)!!
May 14 07:11:58 kernel: [*05/13/2019 21:41:58.8066]
May 14 07:22:49 apsw_watchdog: WARNING: System memory is running low (free memory: 88 MB)!!
May 14 07:22:49 kernel: [*05/13/2019 21:52:49.0347]
May 14 07:22:49 kernel: [*05/13/2019 21:52:49.0347] !!!!! {watchdogd} WARNING: System memory is running low (free memory: 88 MB)!!
May 14 07:22:49 kernel: [*05/13/2019 21:52:49.0347]
May 14 07:33:39 apsw_watchdog: WARNING: System memory is running low (free memory: 88 MB)!!
May 14 07:33:39 kernel: [*05/13/2019 22:03:39.2628]
May 14 07:33:39 kernel: [*05/13/2019 22:03:39.2628] !!!!! {watchdogd} WARNING: System memory is running low (free memory: 88 MB)!!
May 14 07:33:39 kernel: [*05/13/2019 22:03:39.2628]
May 14 07:44:29 apsw_watchdog: WARNING: System memory is running low (free memory: 89 MB)!!
May 14 07:44:29 kernel: [*05/13/2019 22:14:29.4878]
May 14 07:44:29 kernel: [*05/13/2019 22:14:29.4878] !!!!! {watchdogd} WARNING: System memory is running low (free memory: 89 MB)!!
May 14 07:44:29 kernel: [*05/13/2019 22:14:29.4879]

Any thoughts on how to workaround this possible memory leak?

Travis

Sandeep Choudhary · ‎05-14-2019

its a bug: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvo75757/?rfs=iqvred

contact cisco TAC.

Regards

Dont forget to rate helpful posts

Some_Guy · ‎05-14-2019

Me too! Same setup - ME 8.8 on 2800s. Seems 8.8 is a bit of a dumpster fire. The "TAC recommended version" for ME is 8.5 or 8.8. Unfortunately I have one site that needed a feature of 8.7 so I decided to move towards standardizing on 8.8 at all sites. So far I am trying to wait it out for 8.8 fixes, but if 8.5 meets your needs then maybe consider a downgrade. My sites on 8.5 have been fine.

One of the bugs says the workaround is "Proactively, Reboot AP before it hit below 150MB." How can I check the current memory usage of an AP?

ghstrhtrsbhtr · ‎05-14-2019

Glad to hear it's not just me - just to confirm, are you not having any trouble stability-wise with 8.5.140.0?

Some_Guy · ‎05-14-2019

@ghstrhtrsbhtr wrote:

Just to confirm, are you not having any trouble stability-wise with 8.5.140.0?

I was happy with 8.5.135 before.

I do have some sites on 8.5.140, but those sites are mostly 2600 APs running subordinate off a 2800 ME master with its radios disabled. So, not really a useful comparison for judging this specific issue.

Some_Guy · ‎05-16-2019

I hacked together an Expect script to reboot APs approaching low memory. Edit the values in the top section as needed for your environment. It will SSH to the specified controller(s), get a list of 2800/3800 APs, then SSH to each AP and check its "show memory." If MemFree is below the set threshold then it will reload the AP.

Use at your own risk! This is the first time I've ever written anything in Expect so it could have some issues.

#!/usr/bin/expect -f

set timeout 12

set WLC_IP_LIST "192.0.2.21 192.0.2.31 192.0.2.41"
set WLC_USERNAME "admin"
set WLC_PASSWORD "P@ssw0rd"
set AP_USERNAME "yourusername"
set AP_PASSWORD "P@ssw0rd"
set AP_ENABLE_PASSWORD "P@ssw0rd"
set AP_MEM_THRESHOLD "196608"


foreach WLC_IP_ADDR $WLC_IP_LIST {
	spawn ssh -l $WLC_USERNAME -o StrictHostKeyChecking=no $WLC_IP_ADDR
	
	expect {
		"User: $" {
			send "$WLC_USERNAME\r"
			expect "assword:$"
			send "$WLC_PASSWORD\r"
		}
		 "$WLC_USERNAME@$WLC_IP_ADDR's password: $" {
			send "$WLC_PASSWORD\r"
		}
	}
	
	expect "(Cisco Controller) >$"
	send "config paging disable\r"
	
	expect "(Cisco Controller) >$"
	send "show ap summary\r"
	
	set AP_IP_LIST ""
	expect {
		-re " AIR-AP\[23\]8..\[IE\]-.... *..:..:..:..:..:.. *.........................?.?.? *(\[0-9\]*\\.\[0-9\]*\\.\[0-9\]*\\.\[0-9\]*) " {
			lappend AP_IP_LIST "$expect_out(1,string)"
			exp_continue
		}
		"(Cisco Controller) >$"
	}
	
	send "logout\r"
	
	expect {
		"Would you like to save them now\? (y/N) $" {
			send "n"
			expect "Connection to $WLC_IP_ADDR closed."
		}
		"Connection to $WLC_IP_ADDR closed."
	}
	
	
	#TODO error handling? idkwtf
	foreach AP_IP_ADDR $AP_IP_LIST {
		spawn ssh -l $AP_USERNAME -o StrictHostKeyChecking=no $AP_IP_ADDR
	
		expect "$AP_USERNAME@$AP_IP_ADDR's password: $"
		send "$AP_PASSWORD\r"
	
		expect ">$"
		send "enable\r"
	
		expect "assword: $"
		send "$AP_ENABLE_PASSWORD\r"
	
		expect "#$"
		send "show memory\r"
	
		expect -re "MemFree: *(\[0-9\]*) kB"
		set MemFree $expect_out(1,string)
		expect "#$"
	
		if {$MemFree < $AP_MEM_THRESHOLD} {
			send "reload reason CISCO SUCKS - CSCvo75757, CSCvp57188, CSCvp36540, etc.\r"
			expect "reload\? \\\[confirm\\\]$"
			#send "nexit\r" # just testing!
			send "y"
		} else {
			send "exit\r"
		}
	
		expect "Connection to $AP_IP_ADDR closed."
	}
}

exit