09-29-2012 06:27 AM - edited 03-04-2019 05:42 PM
Hello everyone,
here is a brief description of a problem:
On a Cisco 7600 with SUP720 BXL sudden increase of CPU utilization (from 35% to 98%) appeared.
This utilisation happend and still happens on the SP CPU (not RP) and it is interrupt based.
router-sp#sh proc cpu
CPU utilization for five seconds: 99%/82%; one minute: 98%; five minutes: 98%
I also got output from ibc
router-sp#show ibc
Interface information:
Interface IBC0/0(idb 0x44E47588)
Hardware is Mistral IBC (revision 5)
5 minute rx rate 24000 bits/sec, 44 packets/sec
5 minute tx rate 68000 bits/sec, 122 packets/sec
1186057298 packets input, 80461312123 bytes
1179948920 broadcasts received
3254930405 packets output, 224120381448 bytes
85350832 broadcasts sent
0 Inband input packet drops
0 Bridge Packet loopback drops
0 Packets CEF Switched, 0 Packets Fast Switched
0 Packets SLB Switched, 0 Packets CWAN Switched
IBC resets = 2; last at 00:28:58.792 CET Wed Nov 30 2011
Using SPAN I collected packets that were punted to SP CPU.
Statistics showed that 96,82% of all pakets were STP BPDUs and second next were CDP packets with 0,43% share.
It can be seen from the collected packets that 70% off all STP BPDUs are from 2 source MAC addresses.
Since this layer 2 network consists of few hundreds Cisco switches (series 7600, 6500, 2960, 3560,3750, and Linksys) it is very hard to trace this 2 MAC addreses.
questions:
-----------------
1) is there a way to somehow trace these two MAC addreses/switches besides loggin into every switch in the network?
Problem is that these 2 MAC addresses don't appear in CAM or ARP tables on network, And also, as far as I know, huge problem is that 2960,3560 series for BID (bridge identifier) use special MAC address which can be seen only with "show version" (and this MAC is also source MAC address for BPDU)
2) regarding the output of "show ibc", the traffic comming and going to/from SP CPU shouldn't be significant for this CPU utilisation of 98%.
5 minute rx rate 24000 bits/sec, 44 packets/sec
5 minute tx rate 68000 bits/sec, 122 packets/sec
That leads me to conclusion that this could be some kind of loop in processing on CPU and reload could help. Have anyone seen maybe something like this? Am I misled or 122 BPDUs per second can realy overwhelm CPU?
3) Generaly have anyone experienced problem with high SP CPU utilisation on SUP720 And what was the cause usualy?
(and if someone can recommend what else should I look for)
Thanks in advance,
A.
10-15-2012 04:43 AM
Hi Giuseppe,
I totaly agree with your statements. Let me answer step by step
a) Here is the logical topology of the network
____________
| \
bunch of PVST_SWITCHES-------MST_CLOUD PVST_SWITCH(with_problem)
|____________/
affected switch (Cisco7600) has PVST configured and is connecetd to MST domain to two different switches in the domain (also two Cisco7600).
I also captured packets punted to CPU with SPAN and wireshark and it shows:
packets with source MAC address 0015.2B0D.62A1 shows in STP body BID 0015.63F3.4180
packets with source MAC address 0015.6305.170D shows in STP body BID 0015.63F3.3B40
Now 0015.63F3.4180 and 0015.63f3.3b40 are switches directly connected to problematic switch.
following output shows that CAM table is ok.
router01#sh mac-address-table count
MAC Entries for all vlans :
Dynamic Address Count: 2646
Static Address (User-defined) Count: 292
Total MAC Addresses In Use: 2938
Total MAC Addresses Available: 65536
And here are the outputs which shows that all 3 switches don't know about src MAC addresses
0015.2B0D.62A1 0015.6305.170D
problematic switch:
---------------------------
router#sh mac-address-table | i 0015.2b0D.62a1
router#
router#sh mac-address-table | i 0015.6305.170d
router#
MST switch with BID 0015.63F3.4180
-------------------------------------------------------
MST_switch1#sh mac-address-table | i 0015.2b0D.62a1
MST_switch1#
MST_switch1#sh mac-address-table | i 0015.6305.170d
MST_switch1#
MST switch with BID 0015.63F3.3B40
-------------------------------------------------------
MST_switch2#sh mac-address-table | i 0015.2b0D.62a1
MST_switch2#
MST_switch2#sh mac-address-table | i 0015.6305.170d
MST_switch2#
Does switch learn source MACs for system packets like BPDU?
Also does MST change source MAC for BPDUs (because I can't find those two MACs on swithches that sends BPDUs to problematc switch)? Because only answer could be that BPDU is relayed from other PVST switches connected to MST domain.
b) on interface TenGigabitEthernet7/1 is MST_switch1
It has Bridge ID 0015.63F3.4180, and as I showed earlier in a) it doesn't know about
MAC address 0015.2b0D.62a1.
I suppose that when MST_switch1 sends BPDU it sends it with src MAC which it should be found in CAM table or with show modules or show version? Am I wrong?
Thank you and regards,
A.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide