Thanks, but 184.108.40.206 has several bugs that affected us and are fixed in 220.127.116.11. It's an option but it exposes us in other ways. Right now I would just like to get anyone who is running 18.104.22.168 and 5520's to check their show memory pools output and see if the 16 byte buffer is growing like we see in our controller.
... View more
I am hoping to get help from the Cisco Community on an issue we are currently having with our Campus Controller running 22.214.171.124. We have a pair of 5520's in HA with SSO. During busier daytime periods we started getting complaints of slow response time from users several days ago, while in the middle of a two week period where we are migrating Access points from our old 5508 controllers to this new HA pair. We have two main WLAN's configured and in use, one an 802.1X enterprise WLAN for corporate users, and one Guest WLAN that is anchored to two DMZ controllers. All AP's are CAPWAP attached to the controller from campus buildings connected over 10Gig ethernet private fiber, and we do not believe there is congestion anywhere on the LAN. Basically a classic Cisco Controller setup.
When users complain, and we investigate, what we are seeing is very high ping times to the client machine on our corporate network, going as high as 400 to 500 msec for extended periods when it typically is less than 10msec round trip.
One thing we found was repeated occurrences of messages similar to the following being logged in the msg log of the controller:
[PA]broffu_fp_dapi_cmd.c:7104 Warning: DP Early PacketBuffer Buildup warning. DP0 PacketsInBuffer = 16827 Prev PacketsInBuffer = 13892 WM time = 320 Secs
While investigating, we uncovered what looks to be buffer memory issues that looks like a memory leak or non-released buffer memory than can be seen in the following command output:
(bewlcs4) >show memory pools
-------------------------- System Memory Pools Summary ------------------------- System Name:bewlcs4 Primary SW Ver:126.96.36.199 Current Time:Fri Feb 8 14:26:50 2019 System UP Time:14 days 17 hrs 45 mins 37 secs (Time: 1249237 Secs) Index Name PID 16-byte 64-byte 128-byte 256-byte 384-byte 512-byte 1024-byte 2048-byte 4096-byte Raw-Pool Total-Pools 001 SNMPTask 2519 453923 665 272 23 4 1 10 2 2 1 454903 002 fp_main_task 1917 15337 21130 664 581 63 13 124101 62 24586 1806 188343 003 emWeb 3884 6443 3432 33149 6167 29 5 18 2 5 32 49282 004 Client Profiler 3763 4365 0 0 0 0 4365 8730 0 0 0 17460 005 spamApTask1 3620 3393 7965 1208 1014 663 39 389 197 0 109 14977 006 spamApTask2 3621 3307 7757 1178 990 647 38 379 191 0 116 14603 007 spamApTask4 3623 3132 7349 1115 936 612 36 359 182 0 100 13821 008 spamApTask3 3622 2872 6736 1023 858 561 33 327 173 0 105 12688 009 spamApTask7 3626 2700 6352 962 810 529 31 313 158 0 93 11948 010 spamApTask6 3625 2523 5925 897 754 493 29 281 151 0 83 11136 011 spamApTask0 3619 2178 5127 777 654 427 25 249 127 0 81 9645 012 spamApTask5 3624 2088 4905 742 624 408 24 239 122 0 78 9230 013 apfProbeThread 3739 0 0 0 0 0 0 4056 4056 0 0 8112 014 spectrumDataTask 3804 1 1 0 510 0 0 6322 2 0 7 6843
In the above output, we observe the 16-byte buffer memory pool continuously increasing and the process that is contributing to this continuously increasing buffer pool usage is the SNMPTASK process. The 16-byte buffer memory pool counter never decreases, it either stays constant during low traffic periods or increases typically during higher traffic periods. It looks like the memory buffers are never released. In the above, the counter is currently at 453,923 buffers used ny the SNMPTASK process. When looking at our other controllers, this number typically seen for this 16-byte buffer pool is magnitudes smaller, and the SNMPTASK typically using a few thousand. In addition, when performing a debug snmp all enable on the affected controller, there is no debug output being displayed related to SNMP, as if the SNMP process is not functioning correctly. When performing this debug on other controllers, we see debug output related to SNMP being displayed.
Can anyone with model 5520 WLAN controllers and running 188.8.131.52 code please check their controllers similarly and let me know if they are seeing increasing counters in the SNMPTASK usage of 16 byte memory pools or observe any other unusual buffer memory statistics or slow response time from users?
We have 255 Access points associated to this controller, and approximately 2200 users associated to the controller in the middle of the several hour period when users were experiencing slow response.
We have a TAC case opened with Cisco but have not gotten confirmation from them that this is due to an existing bug nor have they confirmed this is a new bug. Your help would be greatly appreciated as we are trying to get more information to support our feeling that this is a bug and the cause of the slow response for our users.
... View more
Please provide more detail on this Bug. The Bug details state that Cisco Prime Network Control System Version 3.5(0.0) is affected, and it is fixed in Version 3.5(0.0). There are the same versions! Also, is this Cisco Prime INFRASTRUCTURE? "Network Control System" is a very old name for a previous product. We customers cannot make sense of Bug release info that is inaccurate or incomplete.
... View more
I know this is an old post, but I wanted to share what I recently found while troubleshooting management from wireless issues. The behavior I was seeing was this. I have several controllers with management interfaces all on the same vlan, say vlan100. I have wireless user vlans configured on several controllers, say vlan200,201,202, etc. All controllers are configured to allow management from wireless. There is a firewall between the wlan controllers and the rest of the LAN, and the firewall is the default gateway for all controller vlan interfaces. The WLAN controllers are running 184.108.40.206 code.
When associated to an AP on Controller 1, on vlan201, I can access controller 1 management interface via HTTPS and ssh, no problem. However, I could not access Controller 2 management interface via HTTPS or ssh, but I could ping it. I don't believe the reason is that Cisco "locked down" the ability to manage a different controller from wireless other than the one the wireless client is associated to.
What I found is that when my client machine sent packets to controller 2 mgmt interface, the path was from wifi adapter thru AP thru Controller 1 to FW on vlan201, then out FW on vlan100 to controller 2 mgmt. The return path from controller 2 to client was from controller vlan201 dynamic interface directly to controller vlan201 dynamic interface to the AP and to the client.
For ping (ICMP) echo and echo reply traffic, this will work just fine even though the forward path was thru the firewall and the reverse was not. For TCP traffic however, the behavior of the firewall caused the communication to fail. In this case, the firewall used TCP randomization to change the sequence numbers on the client and server sides as the packets passed thru it. This is a security feature that protects against attackers trying to predict future sequence and acknowledgement numbers and intercept/hijack communications. The FW was a Cisco ASA running 9.3 code.
The result was a TCP connection attempt where the client sent SYN with seq "abc", the FW changed it to seq "jki", and forwarded on to controller. The controller sent ACK with seq"pqr" and ack "jki" (acknowledging the SYN packet) directly to client via the common vlan201 path between the controllers, bypassing the firewall. The client received the ACK with ack "jki", when it was expecting and acknowlegement for "abc", and immediately sent a RST back to the controller.
The same behavior was seen with both ssh and HTTPS management traffic as they both use TCP. The reason the asymmetric path was taken, is for the return path, controller 2 broadcast an ARP request for the mac address for my client machine, on vlan201. The arp request is directed to my client by the wireless infrastructure (the controller or the AP, I have not confimed which yet) and my client replies with it's mac back to controller 2. So before controller 2 sent the SYN ACK, it inserted the client MAC into the frame and forwarded it out the vlan201 dynamic interface, directly through controller 1 and up to the AP, instead of inserting the FW mac address for the mgmt vlan100 gateway and sending the packet out vlan100.
I believe this issue would still exist even if the firewall did not use TCP randomization, as long as the firewall tracked state in the flows. Without TCP randomization, my client would have accepted the SYN ACK instead of sending a RST, and sent the final ACK in the handshake, thru the firewall, but the firewall would have blocked the ACK since it never saw the SYN ACK, and the RST would likely have come from the firewall as it killed the invalid connection. If there were no firewall, or if the "firewall" was really just ACL=based and not stateful, then I do not think this asymmetry would prevent the management the same way.
I am still trying to think of a scenario where this behavior might break communication other than Administration of the controllers from wireless vlans, which is more of a nuisance than an issue for us, as we can either connect via ethernet or remote into another machine when trying to manage a different controller than the one you are associated to. It seems this might cause issues if one roams from an AP on one controller to an AP on another controller if the controllers share the same dynamic vlans, and if there is TCP traffic traversing a firewall between clients on either controller. However, in well-designed wireless networks, I don't think this would commonly occur. In our case, we are preparing to migrate from three 5508 controllers to an HA pair of 5520's, so that is why we have shared dynamic vlans between the controllers. Once migrated, we will not have the old 5508's to manage and this will not be an issue any more.
... View more
Has anyone configured The Feature Template for Zone Based Firewall in Cisco Prime Infrastructure? We are trying to configure a Policy for ZBFW on ISR 4K routers, and when I go to add a Rule, I can select the Source Zone I previously defined in the Shared Policy Objects, but it does not look like I can select the default zone as the destination, it is not in the list. Self zone is in the list but not default. So how is a policy written between a security zone we define and the rest of the network that is not in a zone?
We are using Prime 3.1.1.
... View more
The 8.1 MR2 release notes list this bug as open. Does anyone know if this is a common issue with MR2? The bug notice only mentions 220.127.116.11. I am concerned about upgrading to MR2 if this bug would cause multicast to stop working on even-numbered WLAN ID's.
... View more
Does anyone know how to see in Cisco prime Infrastructure the specific Access Points that a particular scheduled AP task will be applied to? I created a Lightweight AP Template that disables the A radio and B/G radio, and then scheduled the deployment to occur daily and selected the AP I wanted to apply it to. Then I created another LWAP Template to enable the radios and scheduled it to occur daily at a different time. It seems to work, it disabled the radios and re-enabled them for the AP I selected when I created it, but I can't see where in the UI it shows the scheduled task is attached to that Access Point. Anyone familiar with this? Thanks
... View more
DId you doublecheck the controller name you configured for the Primary or secondary controller for the AP? Not sure about your older version 4.2, but newer versions require the primary controller name configured for the AP to match exactly the name configured for the controller, including upper/lower case characters. If they don't match, I have seen a problem similar to yours where the AP does not like the response from the controller.
... View more
We are trying to determine if it is possible to install our Servers running a NICE VoIP recording Application in our Virtual environment on Cisco UCS. We have filtering TAPS installed in our network that provide as an output a copy of the RTP media streams (similar to SPAN output) to a server with a promiscuous NIC that records the VoIP streams, like a sniffer. Can we connect the output of the tap to a port on the Fabric Interconnect, configure the port as an uplink with a dedicated vlan, and direct the traffic to a VM with a vNIC in promiscuous mode and associated with the dedicated vlan? If this would work, what configuration details are needed? Do we need to disable MAC learning on the dedicated vlan, if that is possible? Monitoring of source interfaces, servers, etc within the UCS environment and directing the destination out to an external device is documented clearly in the configuration guide, but we are essentially trying to do the opposite. Ken Nadsady
... View more