CPU processes Heartbeat and SpanTree Flush on Cat6500/sup7203B

[jvrooden]

Hi,

I'm running into a problem with the CPU-utilization of the SP on two cat6500's with Sup720-3B running 12.2(33)SXI5. The average load on this processor had rissen from 40% to 60% within one month. We did not make any significant changes to the network, but are still investigating the issue.

When I do a "show processes cpu" I see that the Spanning-tree process is using 22,52%, which seems normal to me. However I also see the "Heartbeat proces"  and the "Spantree Flush process" at the top of the list.

The Heartbeat proces is raising to 100% form time to time, but than falls back to 0%

The Spantree Flush process seems to be running continually at around 5%

Both of the processes are unknown to me and I could not find any documentation on them. Do you know what these processes are doing? There seems to be a relation between the rissen load on the SP and these two processes as I do not see them on other 6500's in our netwerk which are running normally.

Is there perhaps a list somewehere available explaining all possible processes running on the Cat6500? I was not able to find such a list anywhere..

Hope you can help me!

Kind regards,

Joris

Switch#remote command switch sh proc cpu sorted

CPU utilization for five seconds: 77%/35%; one minute: 58%; five minutes: 57%

PID      Runtime(ms)        Invoked      uSecs   5Sec        1Min             5Min      TTY      Process

  97           292872         3183633         91      100.00%      33.22%      23.10%        0 Heartbeat Proces                         

295        241924348      322862806        749      22.55%      22.44%      22.52%        0 Spanning Tree

488         4078788        1274098       3201       5.90%       0.88%       0.92%        0 NDE - IPV4

112        21941380        82096647        267       4.01%       5.71%       5.45%        0 SpanTree Flush

354         36106700       84968102        126       1.96%       2.42%       2.42%        0 mls-igmp Process

109         16528352       84528505        195       1.96%       2.17%       2.17%        0 slcp process

262         16335084        879064      18582       1.74%       1.67%       1.74%        0 Vlan Statistics

[danishahammed015]

Did you find a solution for this?

[Naveen Venkateshaiah]

Cpu heartbeat is used by the CPU receiving the packets it sends to itself.

If you can login to SP ,

enable terminal capturing

conf t >Remote login switch

capture below logs.

sp>show spann det | in executing|changes|from

sp> terminal length 0

sp>show process cpu sorted
sp>debug netdr capture rx
sp>show netdr captured-packets

sp>u all

After the capture,u can disable this debug after the capture using  command  "undebug all"

And upload the logs.

Regards,Naveen

****rate if it is helpful****

[danishahammed015]

Dear Naveen

Thanks for your reply.

Just to brief our issue. We have 6500 VSS running as core. One of the SUP shows high cpu utilization in SP. We used to check the utilization on RP and it was normal. Today only we noticed about this from LMS showing that chasis 1 switching processor of module 5 with high CPU. As per logs it seems it was running like this for a long time(97% CPU). But we didnt face any performance issue. Now if i run the debug you mentioned, there might be a chance of utilization going high as the utilization is 97% and its neither increasing nor decreasing. I have taken some logs attached here. Please let me know if you are able to see any errors from the logs.Logs are based on CPU utilization only.If required we can take debug output after working hours.

Thanks

-Danesh Ahammad

[Naveen Venkateshaiah]

Share this below logs,But Capture it when cpu utilization is more than 90%

remote command switch-id 1 sh process cpu

CPU utilization for five seconds: 97%/88%; one minute: 97%; five minutes: 97%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

103    63160776 431973649        146  0.00% 34.66% 36.61%   0 Heartbeat Proces 

Can you share

show stack 103    //3 times

remote command switch show ibc  /// 3 times

show spann det | in executing|changes|from

show spanning-tree summary

Regards,

Naveen

***rate if it is helpful***

[danishahammed015]

HI 

I was engaged in different  project and was not able to update. CPU went down below 10% by itself without changing any thing.

Now the same issue appear again.Please find attached requested files.

Please let me know if you able to find something.Only thing i noticed is dual active detection not configured. 

Regards

-Dani

[Naveen Venkateshaiah]

Hi ,

High cpu is due to Interrupt.

Core#remote command switch-id 1 module 5 show process cpu

CPU utilization for five seconds: 98%/83%; one minute: 97%; five minutes: 97%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

From the logs i could see STP is stable,NO TCN's observred.

VLAN0007 is executing the rstp compatible Spanning Tree protocol
  Number of topology changes 158 last change occurred 5w2d ago
          from Port-channel111

From Netdr i could see, below are the  top talkers  , We see source 0000.0800.0000 and 0000.0300.0000 both are from Xerox cop mac addresses.Which are hitting to the CPU 0100.0ccc.cccc is used as a destination address for CDP, VTP and DTP.

L2 Source Mac

Value                    Count
0000.0800.0000    1487
0000.0300.0000    1233

000003    Xerox Corporation


Kindly look into these source mac address ,try to find those interfaces which this source are connected to  and shut it down and look if the cpu get reduced.



Regards,

Naveen

[danishahammed015]

Hello

I saw this mac-address and search for all (00003,00005,00008,00006,00007).But no such mac address in network

Also no Xerox machines connected in Network. 

Regards

Dani

[martinwins1]

me also having the same issue, its shows xerox with similar MAC