High CPU utilization in core switch

vipinrajrc · ‎12-20-2011

Hi Experts,

I am experiencing high cpu utilization in my 4000 series core switch.

I checked the loggs. i saw some strange loggs.

Please see the below loggs and advice

Core1#sh ver
Cisco IOS Software, Catalyst 4000 L3 Switch Software (cat4000-I5S-M), Version 12
.2(25)EWA14, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Tue 20-May-08 19:28 by chendah
Image text-base: 0x10000000, data-base: 0x114BE208

ROM: 12.2(31r)SG2
Dagobah Revision 226, Swamp Revision 34

Core1 uptime is 49 weeks, 2 hours, 59 minutes
System returned to ROM by power-on
System image file is "bootflash:cat4000-i5s-mz.122-25.EWA14.bin"

cisco WS-C4503 (MPC8245) processor (revision 4) with 524288K bytes of memory.
Processor board ID FOX104902U5
MPC8245 CPU at 400Mhz, Supervisor V
Last reset from PowerUp
27 Virtual Ethernet interfaces
14 Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

==============================================================

Core1#sh logg

1d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 1 times)Packet received

with invalid source MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1

40w1d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou

rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 129

40w3d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou

rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1

40w3d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou

rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 129

40w4d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou

rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1

40w4d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 1 times)Packet recei

ved with invalid source MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 12

9

40w6d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou

rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1

41w2d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou

rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 129

41w3d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou

rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 129

41w3d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou

rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 129

42w1d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou

rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 129

47w3d: %IP-4-DUPADDR: Duplicate address 172.17.113.2 on Vlan113, sourced by 5061

.3d4e.6748

47w3d: %IP-4-DUPADDR: Duplicate address 172.17.113.2 on Vlan113, sourced by 5426

.190b.7179

47w3d: %IP-4-DUPADDR: Duplicate address 172.17.113.2 on Vlan113, sourced by 5675

.2c7b.581f

48w5d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 13 times)Packet rece

ived with invalid source MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1

11

48w6d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 144 times)Packet rec

eived with invalid source MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan

111

48w6d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 13 times)Packet rece

ived with invalid source MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1

48w6d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou

rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 111

1d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 1 times)Packet received
with invalid source MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1
40w1d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou
rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 129
40w3d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou
rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1
40w3d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou
rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 129
40w4d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou
rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1
40w4d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 1 times)Packet recei
ved with invalid source MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 12
9
40w6d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou
rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1
41w2d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou
rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 129
41w3d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou
rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 129
41w3d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou
rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 129
42w1d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou
rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 129

47w3d: %IP-4-DUPADDR: Duplicate address 172.17.113.2 on Vlan113, sourced by 5061
.3d4e.6748
47w3d: %IP-4-DUPADDR: Duplicate address 172.17.113.2 on Vlan113, sourced by 5426
.190b.7179
47w3d: %IP-4-DUPADDR: Duplicate address 172.17.113.2 on Vlan113, sourced by 5675
.2c7b.581f

48w5d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 13 times)Packet rece
ived with invalid source MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1
11
48w6d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 144 times)Packet rec
eived with invalid source MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan
111
48w6d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 13 times)Packet rece
ived with invalid source MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1
48w6d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: Packet received with invalid sou
rce MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 111

can anyone please explain what is the "invalidsourceaddresspacket" ?

Is this is due to some virus attack or something?? Also one more thing this switch is the Active router in HSRP.

Please advice.

Thanks in Advance

Vipin

Thanks and Regards, Vipin

nkarpysh · ‎12-20-2011

Hi Vipin,

If you check the error message

48w5d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 13 times)Packet rece

ived with invalid source MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1 48w5d: %C4K_L2MAN-6-INVALIDSOURCEADDRESSPACKET: (Suppressed 13 times)Packet rece
ived with invalid source MAC address (00:00:00:00:00:00) on port Gi3/2 in vlan 1

You will see that source MAC address is (00:00:00:00:00:00) - that is indeed no correct address. Some some device sending this kind of address to your 4500.

You can see those packets coming from Gi3/2 so you need to trace further - possibly there is a hub connected to this port or down through network connected to that port which sends pakcets with incorrect source MAC.

Trace it and fix.

Nik

HTH,
Niko

vipinrajrc · ‎12-20-2011

Hi Nikolay,

Are you sure the CPU utilization is due to this packetss???

Thanks

Vipin

Thanks and Regards, Vipin

vipinrajrc · ‎12-20-2011

Hi Nikolay,

Also can you identify this???? is this due to any attack??

47w3d: %IP-4-DUPADDR: Duplicate address 172.17.113.2 on Vlan113, sourced by 5061
.3d4e.6748
47w3d: %IP-4-DUPADDR: Duplicate address 172.17.113.2 on Vlan113, sourced by 5426
.190b.7179
47w3d: %IP-4-DUPADDR: Duplicate address 172.17.113.2 on Vlan113, sourced by 5675
.2c7b.581f

Thanks

Vipin

Thanks and Regards, Vipin

vipinrajrc · ‎12-20-2011

Hi,

The interface G3/2 is connected to a distibution switch.Seven access layer switch is connected to this switch.

I have checked the loggs in distibution switches. But no trace regarding this invalid source mac-address.

Anyone have any idea to solve this issue?

Thanks

Vipin

Thanks and Regards, Vipin

vipinrajrc · ‎12-20-2011

Hi,

Anybody know the answer of the above posts?

Please advice

Thanks

Vipin

Thanks and Regards, Vipin

nkarpysh · ‎12-20-2011

Hi

47w3d: %IP-4-DUPADDR: Duplicate address 172.17.113.2 on Vlan113, sourced by 5061
.3d4e.6748
47w3d: %IP-4-DUPADDR: Duplicate address 172.17.113.2 on Vlan113, sourced by 5426
.190b.7179
47w3d: %IP-4-DUPADDR: Duplicate address 172.17.113.2 on Vlan113, sourced by 5675
.2c7b.581f

these messages mean that same ip belong to different hosts. Trace those mac address and see why they send packets with same source ip. This can cause High CPU as will trigger ARP/MAC table to flap between ports.

To troubleshoot High CPU in general follow this doc:

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

You may paste commands shown there to this thread for all to help to check it.

P.S. regarding the interface Gi3/2 - you will need to setup SPAN to see what those packets eith incorrect source MAC are - possibly that can give you a clue how to trace those. Those packets usually don't cause High CPU.

Nik

HTH,
Niko

vipinrajrc · ‎12-20-2011

Hi Nikolay,

Thanks for the reply.

This 172.17.113.2 is the vlan113 Ip address in core switch. And vlan113 is dedicated for an access-layer switch.

I tried a mac-address to vendor search. But it is showing invalid(no vendor found). I think some one is using some tool to generate these things purposefully. Other wise how can it be like this?

Please suggest your ideas

Thanks

Vipin

Thanks and Regards, Vipin

nkarpysh · ‎12-20-2011

Hi Vipin,

Also not sure what are those mac addresses are. What you can do is trace those MAcs through your network towards the edge port and see what is connected there. You can do "show mac addres-table address" command (or equivalent based on platform) with those MACs to see where it learnt from. And then go to that switch and do the same until you locate the edge port sending these packets. But those packets need to come still to your switch otherwise MAC entries would age out.

In general to prevent the spoofing of router IP you have some options:

1. Unicast Reverse Path Forwarding will prevent IP spoofing on the routed interfaces.

Configuring Unicast Reverse Path Forwarding

http://www.cisco.com/en/US/partner/docs/switches/lan/catalyst4500/12.2/44sg/configuration/
guide/schrpf.html

2. You can turn on "IP source guard" in your access or aggregation level switches, which

prevents IP spoofing closer to the source. More information in the following link:

Configuring DHCP Snooping, IP Source Guard, and IPSG for Static Hosts

http://www.cisco.com/en/US/partner/docs/switches/lan/catalyst4500/12.2/44sg/configuration/
guide/dhcp.html

Hope this helps,

Nik

HTH,
Niko

Leo Laohoo · ‎12-20-2011

5061.3d4e.6748

5426.190b.7179

5675.2c7b.581f

These MAC address don't belong to any specific NIC manufacturer in the IEEE database. So this means that you are possibly looking at a DoS.

Trace these MAC address to their final port and disable the port.

vipinrajrc · ‎12-26-2011

Hi,

Sorry for the late response.

But how can i trace these mac address. from the switches i didnt any mac address like this. that is what confusing me.

Have anyone had this kind of experince before?

Thanks

Vipin

Thanks and Regards, Vipin

Leo Laohoo · ‎12-26-2011

But how can i trace these mac address. from the switches i didnt any mac address like this. that is what confusing me.

From your core switch, use the command (depending on your IOS) "sh mac-address address ". This command will show you the interface where the MAC is heard from. If this is another switch, then run the same command until you reach an access port.

Rajmohan Badiger · ‎12-28-2011

Hi Vipin

I saw your post regarding this high utilization on switch.what I saw from the logs and suggest is your network is being effected by a user (nasty one).It seems someone is trying to play with the network if its in a production.

Its a form of hacking or prank .Wherein

"

When the attacker starts to send the ARP packets to the targeted victim, those ARP packets cannot be verified by the receiver. The receiver ARP table is filled with the forged details of the ARP packets sent by the attacker. The attacker is then able to gather all the information about the receiver and even tries to resemble as the receiver to other devices in the network "

Your problem seems to be bit similar ,where you swithc is going crazy.

Find out abt this culprit , use wireshark or network monitor kndaa software to catch hold of this ..

For your refernce check out "

http://www.dis9.com/attack/vlan-hacking.html"

This is my suggestion.kindly correct me if I am Worng.

Thanks

RajM