unusual traffic load on WLC ports

[Temur Kalandia]

Hello everyone, 

i have installed Cisco 9800-CL virtual WLC on vmWare ESXi. 

current setup lools like : 

• WLC has dedicated MGMT interface(GIG1) and trunk port (GIG2).
• Cisco VPC is configured with ESXi standard switch, with active passive uplink ports. no LACP, or etherchannel. 
• WLC is centrally switching user traffic
• promiscius mode and forged transmit rejected at vSwitch level and accepted at port-group level. 

 problem description:

we see unusuall traffic load on trunk interface (GIG2)(RX) , it is two times higher compared to MGMT interface (TX). 

i assume that tunnell established between AP and MGMT port is tranfering control and data traffic and at least same amound of data should be on trunk side. i have doubt that something is duplicating packets but dont understand how to troubleshoot the problem.

 

[MHM Cisco World]

You use LAG? If yes what is hash you use in SW

MHM

[Temur Kalandia]

hi @MHM Cisco World , 

there is no LAG, two dedicated 10G Trunks ports from Nexus is connected to the server. here is the config from VMWare side. uplinks are configured active and passive.

 

[marce1000]

  - Have start with a checkup of the 9800-CL configuration with the CLI command show tech wireless and feed the output into:   Wireless Config Analyzer

 M.

[Temur Kalandia]

Hi @marce1000 , 

i did this, report doesnot show anything related to WLC interfaces. 

there are logs regarding Duplicated MACs :

Dynamic mac 0454.530C.71F3 from GigabitEthernet2 conflict with WlClient, please check the network topology and make sure there is no loop.

this was originally one of the reason of current post, we started troubleshooting why there is duplicated MACs on GIG2 , which is trunk. after i noticed those unusual traffic on GIG2 

[marce1000]

  - FYI : https://medium.com/@wirelesslab.io/cisco-9800-wlc-swport-4-mac-conflict-dynamic-mac-mac-addr-from-interface-conflict-with-cd0a94824aa1

 M.

[Temur Kalandia]

hi @marce1000 , 

i have 17.9.4a firmware and there no another Cisco WLC in the netwrok. also there is no loop, becous uplink switches are quiet . 

there is a part in above mentioned URL , whoch tells following  :

When a client device connects to the Access Point (AP) joined to a 9800 WLC, a client entry is created under Monitoring > Clients. When a device roams away from a WLC or disconnects from Wi-Fi, they are expected to send out disassociation frame to let the AP and WLC that it is leaving. Once the WLC receives a disassociation frame, it removes the client entry.

However, if a device ungracefully roams away (without sending disassociation frame), the 9800 will be left with a stale client entry that is only removed when the idle timeout expires.

this was mentioned, in case of another WLC being in the network, but does it means that without another WLC we can have same duplicated logs in the network ? 

anyway this message should not couse so much traffic on trunk interface. there must be some other explanation to that.

[marce1000]

                  >...there must be some other explanation to that.
                     - Try to analyze the traffic with related tools.

 M.

[Temur Kalandia]

additionally from this, i have SNMP info from the gateway device and it shows exacly the same amount of traffic as it in WLCs MGMT port. based on this i can assume that MGMT is showing reall traffic passing WLC , but why it is doublet at trunk port not clear for me. 

[Scott Fella]

I'm a bit confused here.... your management is management and control plan which is Gig1. Your all other traffic for wireless is Gig2.  I would assume if you have ap's and clients, you would see more traffic on the trunk port by default.

[Temur Kalandia]

hi @Scott Fella , 

what type of controll plane utilizes 500mb/ps traffic ? see screen in original post.  control plane and data plane tunnels are terminated on MGMT interface and aftre WLC passing data plane traffic to trunk ports. so if i am recieving 500mb/ps , ,then i should transfer same amount of traffic via trunk ports, but WLC shows that it tranmitting 2x more traffic

[Scott Fella]

Take a packet capture so you know what you are seeing.  That is the only way to know what is happening.  You never mentioned how many ap's and or clients are on this controller also.

[Temur Kalandia]

hi @Scott Fella  , 

there are about 300 APs and 2-3 k users connected. 

already captured packers from both interfaces, trying to find any unusual packets

[Temur Kalandia]

Maybe we need to move WMI from GIG1 to GIG2 , GIG2 is trunk interface. From the best practice docs, I see that GIG1 is for OOB and WMI should be tagged inside GIG2 Trunk, where WMI and Client VLANs will be tagged

[JPavonM]

That's correct, Giga1 is for OoB, Giga2 is for Prod and Giga3 is for HA, this is by design and not a best practice (https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-cl-wireless-controller-cloud/nb-06-cat9800-cl-wirel-cloud-dep-guide-cte-en.html#Introduction:~:text=9800%2DCL%20network%20interface%20mappings)