Re: CISCO C3750E CPU utilization 100% - Page 2

nishhhant · ‎09-02-2013

Dear Team,

I am experiencing an issue with my core switch C3750E (L3 switch) with the cpu utilization going 100% and in cpu processes I see the ARP input is using more than 61% of the cpu.

CPU utilization for five seconds: 99%/25%; one minute: 99%; five minutes: 99%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

12 1313789569 66834997 19657 60.10% 61.13% 61.48% 0 ARP Input

85 75586512 18520293 4081 1.76% 1.46% 1.44% 0 RedEarth Tx Mana

84 27740856 23022207 1204 1.44% 1.00% 0.94% 0 RedEarth I2C dri

214 51056595 110992559 460 1.28% 1.65% 1.64% 0 IP Input

129 26284746 4806428 5468 0.96% 0.83% 0.82% 0 hpm counter proc

Please suggest what I can provide from my end from show run and help me out resolving this issue.

Warm Regards,

Nishant.

Leo Laohoo · ‎09-03-2013

Do you want me to downgrade the IOS version from 12.2(58)SE2 to 12.2(55)SE8 ?

My recommendation is for you to downgrade to 12.2(55)SE8.

nishhhant · ‎09-04-2013

Do you have any other suggestion of resolving this issue ...

Leo Laohoo · ‎09-04-2013

Do you have any other suggestion of resolving this issue

Aside from downgrading the IOS? Sure, create a Cisco TAC Case.

nishhhant · ‎09-04-2013

Hi Leo,

Lol I could have raised a Cisco TAC but my switch is out of warranty..

Leo Laohoo · ‎09-04-2013

The entire 12.2(58)SE train is very buggy. I've used this version before and downgraded all of my 3560/3750 fleet to 12.2(55)SE8.

Seb Rupik · ‎09-04-2013

Hi Nishant,

Are the hosts on your network not under the management of you IT services group? I would locate the offending machine and talk with the owner about why it is trying to contact so many hosts on the network. If it can't be explained then disconnect it from the network until they can.

This is the sort of behavior you might expect from a machine riddled with viruses!

cheers,

Seb.

nishhhant · ‎09-04-2013

Hi Seb,

The hosts not in management vlan but they have full access to internet through proxy.

There are around 325 users in this vlan that is 142 where most of all the hosts trying to reach the few hosts which are not in the network. Due to which ARP entries are incomplete.

Seb Rupik · ‎09-04-2013

What is the 'ip route' config on this switch?

If you are specifiying the next-hop as an interface and not an IP then the switch will generate an ARP request for every packet which it does not have a more specfic route.

nishhhant · ‎09-04-2013

Hi Seb,

Please find the show ip route output below :-

D*EX 0.0.0.0/0 [170/28416] via 192.168.24.137, 1d15h, GigabitEthernet1/0/24

10.0.0.0/8 is variably subnetted, 7 subnets, 4 masks

S 10.70.4.0/23 [1/0] via 192.168.24.158

S 10.70.6.0/24 [1/0] via 192.168.24.158

S 10.70.56.0/21 [1/0] via 192.168.24.158

S 10.70.66.0/23 [1/0] via 192.168.24.162

C 10.70.68.0/23 is directly connected, Vlan207

L 10.70.68.2/32 is directly connected, Vlan207

S 10.70.70.0/23 [1/0] via 192.168.24.174

192.168.24.0/24 is variably subnetted, 11 subnets, 2 masks

C 192.168.24.136/30 is directly connected, GigabitEthernet1/0/24

L 192.168.24.138/32 is directly connected, GigabitEthernet1/0/24

D 192.168.24.140/30 [90/28416] via 192.168.150.3, 6d12h, Vlan150

[90/28416] via 192.168.149.3, 6d12h, Vlan149

[90/28416] via 192.168.24.137, 6d12h, GigabitEthernet1/0/24

[90/28416] via 10.70.68.3, 6d12h, Vlan207

C 192.168.24.156/30 is directly connected, GigabitEthernet1/0/21

L 192.168.24.157/32 is directly connected, GigabitEthernet1/0/21

C 192.168.24.160/30 is directly connected, GigabitEthernet1/0/18

L 192.168.24.161/32 is directly connected, GigabitEthernet1/0/18

C 192.168.24.168/30 is directly connected, GigabitEthernet1/0/1

L 192.168.24.169/32 is directly connected, GigabitEthernet1/0/1

C 192.168.24.172/30 is directly connected, GigabitEthernet1/0/16

L 192.168.24.173/32 is directly connected, GigabitEthernet1/0/16

S 192.168.135.0/24 [1/0] via 192.168.24.158

S 192.168.136.0/22 [1/0] via 192.168.24.158

S 192.168.140.0/23 [1/0] via 192.168.24.162

C 192.168.142.0/23 is directly connected, Vlan142

192.168.142.0/32 is subnetted, 1 subnets

L 192.168.142.2 is directly connected, Vlan142

S 192.168.144.0/23 [1/0] via 192.168.24.174

192.168.146.0/24 is variably subnetted, 2 subnets, 2 masks

C 192.168.146.0/24 is directly connected, Vlan146

L 192.168.146.1/32 is directly connected, Vlan146

192.168.147.0/24 is variably subnetted, 2 subnets, 2 masks

C 192.168.147.0/24 is directly connected, Vlan147

L 192.168.147.1/32 is directly connected, Vlan147

192.168.148.0/24 is variably subnetted, 2 subnets, 2 masks

C 192.168.148.0/24 is directly connected, Vlan148

L 192.168.148.1/32 is directly connected, Vlan148

192.168.149.0/24 is variably subnetted, 2 subnets, 2 masks

C 192.168.149.0/24 is directly connected, Vlan149

L 192.168.149.2/32 is directly connected, Vlan149

192.168.150.0/24 is variably subnetted, 2 subnets, 2 masks

C 192.168.150.0/24 is directly connected, Vlan150

L 192.168.150.2/32 is directly connected, Vlan150

192.168.151.0/24 is variably subnetted, 2 subnets, 2 masks

C 192.168.151.0/24 is directly connected, Vlan151

L 192.168.151.2/32 is directly connected, Vlan151

192.168.152.0/24 is variably subnetted, 2 subnets, 2 masks

C 192.168.152.0/24 is directly connected, Vlan152

L 192.168.152.1/32 is directly connected, Vlan152

nishhhant · ‎09-04-2013

Hi Seb,

When you specify IP for a route you have to specify for all segments on the switch...which in turn will increase the processes on the switch and the table length will increase the cpu of switch..so we are working on how to restrict unwanted arp request..If you can provide a resolution to restrict please let me know

Cheers,

Nishant.

nishhhant · ‎09-05-2013

For the time being I have managed to minimize the utilization.

Now its as below "-

show proc cpu sorted | in ARP

12 1461177163 88623776 16487 14.07% 16.87% 19.47% 0 ARP Input

13 232156 4977278 46 0.00% 0.00% 0.00% 0 ARP Background

41 2791625 4801887 581 0.00% 0.00% 0.00% 0 Dynamic ARP Insp

40 8083154 6113999 1322 0.00% 0.00% 0.00% 0 ARP Snoop

212 449891 127484 3528 0.00% 0.00% 0.00% 0 IP ARP Adjacency

213 731410 109891119 6 0.00% 0.00% 0.00% 0 IP ARP Retry Age

216 164817 9172622 17 0.00% 0.00% 0.00% 0 IP ARP Track

277 15835 503671 31 0.00% 0.00% 0.00% 0 Cluster RARP

285 0 12 0 0.00% 0.00% 0.00% 0 RARP Input

show arp | in Incomplete

Internet 192.168.142.147 0 Incomplete ARPA

Internet 192.168.142.193 0 Incomplete ARPA

Internet 192.168.149.23 0 Incomplete ARPA

show proc cpu | in CPU

CPU utilization for five seconds: 42%/14%; one minute: 41%; five minutes: 46%