Cisco Switch 4503 high CPU issue

[HCL Support]

Hi Team,

We facing issue of high CPU utilization & need the suggestion on the case.

Now attaching here the show version, show logging, show process memory, show process cpu, show process cpu history for analyze the issue & give the suggestion on the same.

CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   1         104      2663         39  0.00%  0.00%  0.00%   0 Chunk Manager   
  13       74128    139363        531  0.95%  0.49%  0.57%   0 ARP Input 
  42      615904   1178468        522  4.47%  5.08%  4.66%   0 Cat4k Mgmt HiPri
  43     5916828    608479       9723 59.42% 58.36% 60.72%   0 Cat4k Mgmt LoPri
  76        8472      1546       5479  4.95%  4.77%  2.00%   1 SSH Process
  79     1885596    453959       4153 21.40% 20.74% 21.35%   0 IP Input
  82      821128    147344       5572  6.54%  7.94%  7.99%   0 Spanning Tree
 

Regards

Ashutosh

[rakeshvelagala]

Hi

The issue suddenly started happening?

Have you added more acls?

[HCL Support]

Hi,

We have removed all vlan’s ACL to avoid TCAM exhaustion due to s/w switching, but still getting high CPU (99%).

Attaching here the CLI output for checking the case.

Show process cpu sorted

Show process cpu history

Show logging

show platform health

show platform cpu packet statistics

sh platform cpu packet buffered

Need suggestion here if the issue can be software related means the bug hit.

[Leo Laohoo]

  79     1885596    453959       4153 21.40% 20.74% 21.35%   0 IP Input

One of the high utilization is caused by the process called "IP Input".  

Experience have taught me that "IP Input" happens when you're chassis is trying to push as much as the inbound traffic is pushing down.  It's like saying that you're pushing 10 Gbps traffic but the chassis or supervisor card has a backplane of 6 Gbps.  

[Madhukrishnan Gopinathan Nair]

I am not seeing the output of the below infact anything excpet the first post.

Can you please share it here.

show platform health

show platform cpu packet statistics

sh platform cpu packet buffered

Also just to ensure there is no Layer 2 Instability, you could turn on mac flap notification.

(config)#mac address-table notification mac-move

Hope this helps

Thanks,

Madhu.

[Madhukrishnan Gopinathan Nair]

Also please ensure you run the below command before you take sh platform cpu packet buffered

debug platform packet all receive buffer

Only then it will capture packets going to cpu.

Thanks,

Madhu.

[HCL Support]

Hi,

Providing here CLI output as below in attachment.

Show process cpu sorted

Show process cpu history

Show logging

show platform health

show platform cpu packet statistics

sh platform cpu packet buffered

Regards

Ashutosh

[Madhukrishnan Gopinathan Nair]

What is on Gi2/38 ? Can you please check if cpu goes down by shutting it down?

Also do you have a lot of ACLs on this?

Thanks,

Madhu.

[Madhukrishnan Gopinathan Nair]

Also please try enabling mac move notification as below

(config)#mac address-table notification mac-move

HTH

Madhu

[HCL Support]

Hi,

Also attaching the show tech of the Switch 4503-E.

Suggest me the cause of the high cpu event for the Switch from show tech if possible.

Regards,

Ashutosh

[Madhukrishnan Gopinathan Nair]

Hi,

It is difficult to comment from show tech.

Did you check my above post and attemted that?

Thanks,

Madhu

[Jon Marshall]

I agree with Madhu, the server on port gi2/38 is accounting for most of the traffic but that could be because of time you caught the conversation.

Can you post the output of -

"show platform software acl input summary interface vlan 172"

Edit - you could also try temporarily removing the acl from the server vlan 172 but not before you run the above command.

Jon