03-20-2012 01:18 AM - edited 03-07-2019 05:40 AM
Hi ,
what should be the problem?
SUP2T-D#sh proce cpu hist
11111111111 1111 1111 1111 1111
0000000000099999000099999000099999999990000999900009999999
0000000000099999000099999000099999999990000999900009999999
100 **********************************************************
90 **********************************************************
80 **********************************************************
70 **********************************************************
60 **********************************************************
50 **********************************************************
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
SUP2T-D#sh proce cpu sorted
CPU utilization for five seconds: 100%/83%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
416 3324188 879928 3777 13.05% 14.42% 14.45% 0 Spanning Tree
633 104408 5091 20508 1.50% 0.53% 0.45% 0 Env Poll
75 22000 298 73825 1.10% 0.13% 0.07% 0 Per-minute Jobs
168 69696 163563 426 0.39% 0.23% 0.22% 0 slcp process
2 532 1010 526 0.07% 0.00% 0.00% 0 Load Meter
712 504 2467 204 0.07% 0.00% 0.00% 0 NF AM Main Proc
352 9224 1410 6541 0.07% 0.01% 0.00% 0 Lif stats hw rea
413 82300 238061 345 0.07% 0.03% 0.00% 0 MOP Protocols
10 28304 1454 19466 0.07% 0.07% 0.07% 0 Check heaps
9 0 2 0 0.00% 0.00% 0.00% 0 LC Registry Expo
11 0 1 0 0.00% 0.00% 0.00% 0 Pool Manager
8 0 2 0 0.00% 0.00% 0.00% 0 PF Redun ICC Req
13 4 165 24 0.00% 0.00% 0.00% 0 WATCH_AFS
14 0 1 0 0.00% 0.00% 0.00% 0 License Client N
15 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest
7 0 1 0 0.00% 0.00% 0.00% 0 PF Redun ICC Req
12 0 2 0 0.00% 0.00% 0.00% 0 Timers
SUP2T-D#show ibc
====================================
Inband counters and statistics
====================================
Interface information:
Interface IBC0/0(idb 0x5172A510)
5 minute rx rate 132671000 bits/sec 215989 packets/sec
5 minute tx rate 5324000 bits/sec 6655 packets/sec
841634035 Packets input, 61939066968 bytes
674810428 broadcasts received
31998854 Packets outputs, 3210901354 bytes
0 broadcasts sent
0 Inband input packet drops
0 Inband output packet drops
1 IBC resets
*** Inband Generic Counters***
841634104 Packets Input, 1809524824 Bytes
674810466 Broadcasts Input
31999239 Packets Output
816543102 Total Drops
0 software bridged paks
0 Packets Fast switching paks
Potential/Actual paks CEF switched 30197067/49692
Potential/Actual paks tag CEF switched 0/0
0 packets xconnect CEF switched
0 Packets xconnect L2 switched
0 packets xconnect dropped
0 Packets immediately punted
Potential/Actual paks copied to process level 826935947/10304396
Potential/Actual paks copied to L2 process level 14645067/14645067
14647635 Packets L2 processed
2568 packets L2 switched
0 Encapsulation corrections
Internally intercepted packets:
0 exc, 151275503 dindex (14647635 sp, 0 old index), 796790606 cap1, 0 cap2
Earl8 shimmed paks rcvd:
0 pseudoipv6 0 svchdr 30195034 shim
*** INBAND PACKET DROPS & FAILURES ***:
0 shim_processing 0 shim_error
0 no idb 0 get_l2_idb failures 14644875 if_input changes
0 intercept 88449 spd
0 Number of times intercept function(s) returning incorrect values
0 bridge loopback 0 diag
0 fastsend 0 soutput 0 no mac descriptors 0 NULL if_outputs
816543102 throttle 580 throttle sneaks
0 encapsulation 24769 get_lif
0 if_input corrections 0 uncorrectable if_inputs
0 corr ecc errors
0 null parse packet
17360486 shim insertions attempted at tx, 24769 failed
0 times les cef vector not set
0 times les cef vector disabled
SUP2T-D#Show netdr captu
A total of 4096 packets have been captured
The capture buffer wrapped 0 times
Total capture capacity: 4096 packets
------- dump of incoming inband packet -------
l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
20820000 000A0000 00020000 51080000 06000002 8E000004 00000008 200A9360
destmac AB.00.00.02.00.00, srcmac 00.21.D8.C6.5F.80, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8C6
5F806400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
28820100 000A0000 00000000 51080000 06000052 8E000004 00000008 200AA3DA
destmac AB.00.00.02.00.00, srcmac 00.14.1B.30.50.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 00141B30
50006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
30820000 000A0000 00020000 51080000 06000002 0E000004 00000008 200A69C3
destmac AB.00.00.02.00.00, srcmac 00.21.D8.C6.5F.80, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8C6
5F806400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
40820100 000A0000 00000000 51080000 06000002 0E000004 00000008 200A3567
destmac AB.00.00.02.00.00, srcmac 00.21.D8.C6.5F.80, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8C6
5F806400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
48820000 000A0000 00020000 51080000 06000012 0E000004 00000008 200AD5C0
destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8B2
2B006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
58820100 000A0000 00000000 51080000 06000012 0E000004 00000008 200A2961
destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8B2
2B006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
60820000 000A0000 00020000 51080000 06000052 8E000004 00000008 200A577E
destmac AB.00.00.02.00.00, srcmac 00.14.1B.30.50.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 00141B30
50006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
68820100 000A0000 00000000 51080000 06000012 0E000004 00000008 200AB961
destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8B2
2B006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
70820000 000A0000 00020000 51080000 06000002 8E000004 00000008 200AA365
destmac AB.00.00.02.00.00, srcmac 00.21.D8.C6.5F.80, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8C6
5F806400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
78820100 000A0000 00000000 51080000 06000052 8E000004 00000008 200A93DF
destmac AB.00.00.02.00.00, srcmac 00.14.1B.30.50.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 00141B30
50006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
80820000 000A0000 00020000 51080000 06000002 0E000004 00000008 200A79C9
destmac AB.00.00.02.00.00, srcmac 00.21.D8.C6.5F.80, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8C6
5F806400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
98820000 000A0000 00020000 51080000 06000012 0E000004 00000008 200A65CF
destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8B2
2B006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
A0820000 000A0000 00020000 51080000 06000012 0E000004 00000008 200ACDCF
destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8B2
2B006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
A8820100 000A0000 00000000 51080000 06000012 0E000004 00000008 200A796E
destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8B2
2B006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/3, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x2(2), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
B0820000 000A0000 00020000 51080000 06000052 8E000004 00000008 200AE771
destmac AB.00.00.02.00.00, srcmac 00.14.1B.30.50.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 00141B30
50006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
B8820100 000A0000 00000000 51080000 06000012 0E000004 00000008 200A096E
destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8B2
2B006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
------- dump of incoming inband packet -------
l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
C8820100 000A0000 00000000 51080000 06000052 8E000004 00000008 200A83D5
destmac AB.00.00.02.00.00, srcmac 00.14.1B.30.50.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 00141B30
50006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
03-20-2012 01:37 AM
Hello,
CPU is high due to interrupts (traffic) and it is good you have taken nedtr capture.
Ethertype 6002 is MOP for incoming packets
1. It looks like your router may transmit DECnet MOP (DEC Maintenance Operation Protocol (MOP)) packets (ethertype 6002) to multicast
MAC address ab00.0002.0000
2. Those packets sourced from Vlan 10(MAC 0023.33C5.AD80) were massively punted to CPU -
a part of netdrive capture below:
------- dump of incoming inband packet -------
l2idb Gi1/1, l3idb Vl10, routine inband_process_rx_packet, timestamp 00:11:47.943
dbus info: src_vlan 0xA(10), src_indx 0x0(0), len 0x51(81)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x200A(8202)
cap1 1, cap2 0
B8820100 000A0000 00000000 51080000 06000012 0E000004 00000008 200A096E
destmac AB.00.00.02.00.00, srcmac 00.21.D8.B2.2B.00, ethertype 6002
layer 3 data: 3D000700 00000100 03030000 02000221 00030006 00000000
00000400 023C0005 0002D805 06000200 01070006 0021D8B2
2B006400 01799001 01019101 02EE0500 00000000 00000000
00000000 00
http://www.cisco.com/en/US/tech/tk870/tk136/tk885/technologies_tech_note09186a0080093cd1.s
html
3.You can jcheck if MOP protocol and MOP sysid enabled on on Vlan4005 or inout interfaces Gi1/1 and Gi1/3 and consider if you want to remove.
4. You can also trace the source MAcs of coming packets and understand why MOP traffic is coming from it.
Hope this helps,
Nik
03-20-2012 08:59 AM
Hi,
thank you for your answer.
I have disable mop on vlan interfaces, but still same.
how can i completely disabe mop?
03-20-2012 09:04 AM
Hey.
Did you check it on physical interfaces as well? It might be needed to be disabled there. Also you can see where those source MAcs 00.21.D8.B2.2B.00 and 00.14.1B.30.50.00 are coming from. I do see these two sending MOP packets.
check where those are learnt from with "show mac-addr addr 0014.1B30.5000" or similar (based on platform) command and go to the device connected on that interface to do same check until you find the end host with that MAC.
Hope that helps.
Nik
03-20-2012 09:07 AM
hi,
i have diasble mop on physical interfaces also. g1/1 ang g1/3 are also 6500. I have disable mop on their phsical and vlan interfaces but netdr output still shows ether 6002.. can i police these packets via COPP?
03-20-2012 09:16 AM
I guess CoPP wont help here as ethertype 6002 is not ip so IP acl wont help and Copp afaik does not support MAC acl.
You can try to create MAC ACL or VACL and apply that on port just to drop those MAC addresses as sources.
I guess now even with MOP disabled switch treat destination MAC of those packets like some mcast MAC and send copy ofr CPU analysis.
Nik
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide