4500 High CPU issue due to MAC Address Flapping

MANISH Yadav · ‎01-02-2017

A MAC flap is caused when a switch receives packets from two different physical/logical interfaces with the same source MAC address.
When flapping occurs it causes this table to be updated whenever a packet is sent/received. The more data flowing through the interface that is flapping the higher your CPU Utilization is going to be which can have serious potential negative side effects.
Use below command to identify high CPU issue.

Switch# show process cpu sorted

CPU utilization for five seconds: 17%/0%; one minute: 16%; five minutes: 16%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

60 29111481 13351616 2180 9.27% 8.06% 7.97% 0 Cat4k Mgmt LoPri

59 25067966 35373845 708 8.00% 8.10% 8.16% 0 Cat4k Mgmt HiPri

41 323666 430147 752 0.15% 0.14% 0.15% 0 IDB Work

131 36 200 180 0.07% 0.01% 0.00% 0 Exec

107 83017 4301245 19 0.07% 0.05% 0.07% 0 UDLD

192 5072 3186984 1 0.07% 0.00% 0.00% 0 PM Callback

6 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Receive

5 0 1 0 0.00% 0.00% 0.00% 0 Retransmission o

340921 58127 5865 0.00% 0.11% 0.06% 0 Check heaps

This output delineates two processes that use the CPU: Cat4k Mgmt HiPri and Cat4k Mgmt LoPri).

Other command :- show platform health

cisco4500#show platform health

%CPU %CPU RunTimeMax Priority Average %CPU Total

Target Actual Target Actual Fg Bg 5Sec Min Hour CPU

VSI channel Slot-01 1.00 0.27 6 1 100 500 0 0 0 17:45

VSI channel Slot-04 1.00 0.05 6 0 100 500 0 0 0 5:40

VSI channel Local Ja 1.00 0.01 6 0 100 500 0 0 0 3:28

VSI channel Remote J 1.00 0.04 6 0 100 500 0 0 0 3:16

GalChassisVp-review 3.00 3.79 10 157 100 500 3 3 3 233:53

Lj-poll 1.00 0.01 2 0 100 500 0 0 0 1:29

StatValueMan Update 1.00 0.07 1 0 100 500 0 0 0 4:05

GalK5TatooineStatsMa 0.70 0.02 4 0 100 500 0 0 0 2:12

K5L3FlcMan Consisten 2.00 0.48 15 7 100 500 0 0 0 55:28

K5L3FlcMan NI Regs & 1.00 0.42 5 4 100 500 0 0 0 22:51

K5L3AdjStatsMan Revi 2.00 0.03 10 6 100 500 0 0 0 15:00

K5FlcHitMan review 2.00 0.01 5 2 100 500 0 0 0 20:41

K5PortMan Regular Re 2.00 0.15 15 11 100 500 0 0 0 21:35

K5PortMan Ondemand L 3.00 0.34 30 0 100 500 0 0 0 18:51

%CPU Totals 238.72 22.42

Each of the platform-specific processes has a target or expected CPU utilization. When that process is within range, the CPU executes the process in the high-priority context and the show processes cpucommand output displays that utilization under Cat4k Mgmt HiPri . If a process exceeds the target, it runs under the low-priority context and the show processes cpu command output counts that additional utilization under Cat4k Mgmt LoPri. You can also use Cat4k Mgmt LoPri to run background and other low-priority processes (such as consistency check and reading interface counters). This mechanism allows the CPU to run high-priority processes when necessary. The remaining idle CPU cycles are used for the low-priority processes. Marginally exceeding the target CPU utilization (or a brief utilization spike) does not reflect a problem that requires investigation.

To resolve issue use below command to generate the logs on switch for mac address flapping issue. Generally does not show mac flap logs on switch. As to avoid too many logs in switch.

cisco4500(config)#mac address-table notification mac-move.

This command will generate logs as below.

Cisco4500# show log

Syslog logging: enabled (0 messages dropped, 1 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

...

*Oct 3 08:51:28.149: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (10.10.10.236)

*Oct 3 09:43:46.437: %C4K_EBM-4-HOSTFLAPPING: Host 00:60:48:1B:01:15 in vlan 400 is moving from port Gi2/40 to port Gi2/30

*Oct 3 09:43:48.629: %C4K_EBM-4-HOSTFLAPPING: Host 00:60:48:1B:01:15 in vlan 400 is moving from port Gi2/30 to port Gi2/40

*Oct 3 09:43:48.717: %C4K_EBM-4-HOSTFLAPPING: Host 00:60:48:1B:01:15 in vlan 400 is moving from port Gi2/40 to port Gi2/30

*Oct 3 09:43:49.581: %C4K_EBM-4-HOSTFLAPPING: Host 00:60:48:1B:01:15 in vlan 400 is moving from port Gi2/30 to port Gi2/40

Logs shows that MAC address bouncing between two different physical ports.

1.Finally to disable one of the two interfaces or configure LACP as per requirement. It may also between two different ESXI/blade servers /firewall consisting same mac address connected on two different switches which are in common LAN segment.
2.Remove command after issue resolved.
3.cisco4500(config)#no mac address-table notification mac-move