cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

1134
Views
0
Helpful
11
Replies
Beginner

Help with 5520 running 8.5.140.0 and memory buffer issues

I am hoping to get help from the Cisco Community on an issue we are currently having with our Campus Controller running 8.5.140.0.  We have a pair of 5520's in HA with SSO.  During busier daytime periods we started getting complaints of slow response time from users several days ago, while in the middle of a two week period where we are migrating Access points from our old 5508 controllers to this new HA pair. We have two main WLAN's configured and in use, one an 802.1X enterprise WLAN for corporate users, and one Guest WLAN that is anchored to two DMZ controllers. All AP's are CAPWAP attached to the controller from campus buildings connected over 10Gig ethernet private fiber, and we do not believe there is congestion anywhere on the LAN. Basically a classic Cisco Controller setup.

 

When users complain, and we investigate, what we are seeing is very high ping times to the client machine on our corporate network, going as high as 400 to 500 msec for extended periods when it typically is less than 10msec round trip.

 

One thing we found was repeated occurrences of messages similar to the following being logged in the msg log of the controller:

[PA]broffu_fp_dapi_cmd.c:7104 Warning: DP Early PacketBuffer Buildup warning. DP0 PacketsInBuffer = 16827 Prev PacketsInBuffer = 13892 WM time = 320 Secs

 

While investigating, we uncovered what looks to be buffer memory issues that looks like a memory leak or non-released buffer memory than can be seen in the following command output:

 

(bewlcs4) >show memory pools

-------------------------- System Memory Pools Summary -------------------------
System Name:bewlcs4 Primary SW Ver:8.5.140.0
Current Time:Fri Feb 8 14:26:50 2019 System UP Time:14 days 17 hrs 45 mins 37 secs (Time: 1249237 Secs)
Index Name PID 16-byte 64-byte 128-byte 256-byte 384-byte 512-byte 1024-byte 2048-byte 4096-byte Raw-Pool Total-Pools
001 SNMPTask 2519 453923 665 272 23 4 1 10 2 2 1 454903
002 fp_main_task 1917 15337 21130 664 581 63 13 124101 62 24586 1806 188343
003 emWeb 3884 6443 3432 33149 6167 29 5 18 2 5 32 49282
004 Client Profiler 3763 4365 0 0 0 0 4365 8730 0 0 0 17460
005 spamApTask1 3620 3393 7965 1208 1014 663 39 389 197 0 109 14977
006 spamApTask2 3621 3307 7757 1178 990 647 38 379 191 0 116 14603
007 spamApTask4 3623 3132 7349 1115 936 612 36 359 182 0 100 13821
008 spamApTask3 3622 2872 6736 1023 858 561 33 327 173 0 105 12688
009 spamApTask7 3626 2700 6352 962 810 529 31 313 158 0 93 11948
010 spamApTask6 3625 2523 5925 897 754 493 29 281 151 0 83 11136
011 spamApTask0 3619 2178 5127 777 654 427 25 249 127 0 81 9645
012 spamApTask5 3624 2088 4905 742 624 408 24 239 122 0 78 9230
013 apfProbeThread 3739 0 0 0 0 0 0 4056 4056 0 0 8112
014 spectrumDataTask 3804 1 1 0 510 0 0 6322 2 0 7 6843

 

In the above output, we observe the 16-byte buffer memory pool continuously increasing and the process that is contributing to this continuously increasing buffer pool usage is the SNMPTASK process.  The 16-byte buffer memory pool counter never decreases, it either stays constant during low traffic periods or increases typically during higher traffic periods. It looks like the memory buffers are never released. In the above, the counter is currently at 453,923 buffers used ny the SNMPTASK process. When looking at our other controllers, this number typically seen for this 16-byte buffer pool is magnitudes smaller, and the SNMPTASK typically using a few thousand.  In addition, when performing a debug snmp all enable on the affected controller, there is no debug output being displayed related to SNMP, as if the SNMP process is not functioning correctly.  When performing this debug on other controllers, we see debug output related to SNMP being displayed.

 

Can anyone with model 5520 WLAN controllers and running 8.5.140.0 code please check their controllers similarly and let me know if they are seeing increasing counters in the SNMPTASK usage of 16 byte memory pools or observe any other unusual buffer memory statistics or slow response time from users?

 

We have 255 Access points associated to this controller, and approximately 2200 users associated to the controller in the middle of the several hour period when users were experiencing slow response.

 

We have a TAC case opened with Cisco but have not gotten confirmation from them that this is due to an existing bug nor have they confirmed this is a new bug. Your help would be greatly appreciated as we are trying to get more information to support our feeling that this is a bug and the cause of the slow response for our users.

 

Thanks,

Ken Nadsady

CCNA-Wireless, CWNP

Everyone's tags (4)
11 REPLIES 11
Hall of Fame Community Legend

Re: Help with 5520 running 8.5.140.0 and memory buffer issues

I have not used 8.5.140.0 but I have used 8.5.135.0 and I've never had issues like this before.
So you might try to downgrade to 8.5.135.0 if things get out of hand.
Beginner

Re: Help with 5520 running 8.5.140.0 and memory buffer issues

Thanks, but 8.5.135.0 has several bugs that affected us and are fixed in 8.5.140.0.  It's an option but it exposes us in other ways. Right now I would just like to get anyone who is running 8.5.140.0 and 5520's to check their show memory pools output and see if the 16 byte buffer is growing like we see in our controller.

 

VIP Advocate

Re: Help with 5520 running 8.5.140.0 and memory buffer issues

This is what it's looking like in my installation, which is fairly similar to yours (no Anchor controllers though):
(Cisco Controller) >show memory pools

-------------------------- System Memory Pools Summary -------------------------
System Name:wlc-5520-1 Primary SW Ver:8.5.140.0
Current Time:Wed Feb 13 11:12:25 2019 System UP Time:19 days 12 hrs 51 mins 37 secs (Time: 1654402 Secs)
Index Name PID 16-byte 64-byte 128-byte 256-byte 384-byte 512-byte 1024-byte 2048-byte 4096-byte Raw-Pool Total-Pools
001 fp_main_task 1902 4591 20216 657 575 60 13 124079 61 24636 1803 176691
002 emWeb 3225 94189 571 33131 6166 29 5 18 2 3 23 134137
003 Client Profiler 3095 1292 2267 1555 369 0 5483 8708 0 0 0 19674
004 spamApTask4 2978 2343 5173 810 756 432 27 269 135 0 73 10018
005 spamApTask7 2981 2170 4845 752 704 402 25 248 125 0 75 9346
006 spamApTask2 2976 1989 4453 690 644 368 24 231 116 0 71 8586
007 spamApTask0 2974 1899 4166 661 618 353 22 229 110 0 70 8128
008 spamApTask6 2980 1730 3865 600 560 320 20 206 101 0 56 7458
009 spamApTask1 2975 1737 3810 602 564 322 20 203 100 0 60 7418
010 spamApTask3 2977 1648 3622 570 532 304 19 197 95 0 63 7050
011 spamApTask5 2979 1470 3292 510 476 272 17 176 86 0 55 6354
012 spamReceiveTask 2973 0 0 0 0 0 0 3002 0 0 5 3007
013 SISF BT Process 3151 485 770 0 484 0 0 0 0 0 0 1739
014 SNMPTask 2490 702 633 261 15 1 0 3 0 2 1 1618


Please note, it's currently holiday here, so only around 35% of the users are around. Using ~175 APs and around 600 clients at the moment.
Hall of Fame Community Legend

Re: Help with 5520 running 8.5.140.0 and memory buffer issues

Can you please post the complete output to the command "sh memory mbuf summary" and "sh memory statistics"?
VIP Advocate

Re: Help with 5520 running 8.5.140.0 and memory buffer issues

In my case (without any issues, for comparison):

(Cisco Controller) >show memory mbuf summary

System Up Time................................... 19 days 13 hrs 19 mins 33 secs
Maximum number of Mbufs.......................... 24576
Number of Mbufs Free............................. 24560
Number of Mbufs In Use........................... 16
                                                    Top 10 Mbuf Usage:
802.11 Mgmt...................................... 65535
Fast Path Control................................ 65529
IPv6............................................. 65514
DOT1X............................................ 2
IAPP............................................. 2
802.11 Probe..................................... 1
CAPWAP........................................... 1
DHCP............................................. 1
CDP.............................................. 1
Orphan........................................... 1

(Cisco Controller) >show memory statistics


System Memory Statistics:
Total System Memory............: 33158025216 bytes (30.88 GB)
Used System Memory.............: 2976354304 bytes (2.77 GB)
Free System Memory.............: 30181670912 bytes (28.10 GB)
Effective Free Memory..........: 29902340096 bytes (27.84 GB)
Bytes allocated from RTOS......: 873503040 bytes (833.09 MB)
Chunks Free....................: 103 bytes 
Number of mmapped regions......: 28
Total space in mmapped regions.: 929271808 bytes (886.28 MB)
Total allocated space..........: 808951728 bytes (771.53 MB)
Total non-inuse space..........: 64551312 bytes (61.56 MB)
Top-most releasable space......: 133136 bytes (130.01 KB)
Total allocated (incl mmap)....: 1802774848 bytes (1.67 GB)
Total used (incl mmap).........: 1738223536 bytes (1.61 GB)

(Cisco Controller) >
Beginner

Re: Help with 5520 running 8.5.140.0 and memory buffer issues

Don't downgrade to 8.5.135. I 'm having the same issue on pair of 8510 in HA running 8.5.135. 

For comparison:

show memory mbuf summary

System Up Time................................... 18 days 0 hrs 24 mins 44 secs
Maximum number of Mbufs.......................... 24576
Number of Mbufs Free............................. 24560
Number of Mbufs In Use........................... 16
Top 10 Mbuf Usage:
Fast Path Control................................ 40
IPv6............................................. 5
DOT1X............................................ 4
802.11 Mgmt...................................... 2
802.11 Probe..................................... 1
IAPP............................................. 1
CAPWAP........................................... 1
DHCP............................................. 1
CDP.............................................. 1
Orphan........................................... 1

(Cisco Controller) >

show memory statistics


System Memory Statistics:
Total System Memory............: 16282812416 bytes (15.16 GB)
Used System Memory.............: 5421293568 bytes (5.04 GB)
Free System Memory.............: 10861518848 bytes (10.11 GB)
Effective Free Memory..........: 10583502848 bytes (9.85 GB)
Bytes allocated from RTOS......: 1947444480 bytes (1.81 GB)
Chunks Free....................: 129 bytes
Number of mmapped regions......: 31
Total space in mmapped regions.: 18446744071678341120 bytes (382.10 GB)
Total allocated space..........: 1882917152 bytes (1.75 GB)
Total non-inuse space..........: 64527328 bytes (61.54 MB)
Top-most releasable space......: 77520 bytes (75.70 KB)
Total allocated (incl mmap)....: 18446744073625785600 bytes (383.92 GB)
Total used (incl mmap).........: 18446744073561258272 bytes (383.86 GB)

Hall of Fame Community Legend

Re: Help with 5520 running 8.5.140.0 and memory buffer issues

I don't see anything wrong with the output. Number of MBufs free is very high (good) and MBufs in Use is low (good).
Beginner

Re: Help with 5520 running 8.5.140.0 and memory buffer issues

We are at times running into the same issues with our 8510's running 8.5.135.x

- VoIP Clients experiencing issues with dropped calls, roaming issues.

- Ping times from wireless client/wired client to WLC higher than usual.

- This message appears on the WLC logs from time to time: %BROFFU-0-DP_BUFFER_POOL_EARLY_WARNING: broffu_fp_dapi_cmd.c:7101 Warning: DP Early PacketBuffer Buildup warning. DP0 PacketsInBuffer = 15367 Prev PacketsInBuffer = 16850 WM time = 260 Secs

 

 

Beginner

Re: Help with 5520 running 8.5.140.0 and memory buffer issues

Hi,

 

I have an update. After opening case with TAC, they suggested that we change our TCP MSS value to 1250 or use flexconnect with local switching even though we do not have any branch sites. After changing the TCP MSS to 1250 things are away better and I'm not seeing any of those syslog messages. Thanks.

Highlighted
Beginner

Re: Help with 5520 running 8.5.140.0 and memory buffer issues

Hi, 

 

thanks for the update.  That's interesting, we already had our TCP MSS value set to 1250 globally when this issue was happening.

 

Beginner

Re: Help with 5520 running 8.5.140.0 and memory buffer issues

To be exact this was TAC reply:

 

Those alarms are informational. These alarms alert users for buffers getting exhausted.

Those alarms are related to the traffic the WLC is receiving. A high amount of traffic can overwhelm the WLC operation.

As action plan I suggested you to enable TCP MSS. What we are doing with this feature is defining the segment length clients will send to wireless infrastructure.

When a big amount of traffic is being sent to the WLC with high segment length retransmissions could occur thus WLC could get overwhelmed.

With TCP MSS enabled (1250) what we did was reducing the segment length then we reduced retransmissions and an improvement in the network traffic management is expected. So you stopped seeing those messages.

No issue has been identified in the WLC. No crashes, no core-dumps.

 

How to avoid those messages?

We have two options (being the second one the best one for this scenario):

 

     Enabling TCP MSS == 1250. (This is a feature we always recommend to enable globally with value 1250)

     Modifying APs to work as flexConnect.

 

In this moment all your APs (589) are working as local and your WLANs are configured as central. That means all clients traffic will be sent to the WLC.

With flexConnect-localSwitching enabled, clients traffic will remain in the LAN where the APs are connected.

CreatePlease to create content
Content for Community-Ad
August's Community Spotlight Awards