High Cpu Spike in 6509

[amol1976]

Hi,

We are getting continuous high cpu spikes while we are checking in Show Cpu History.

7877696987799868787898889899899988999999998999978799689879999998988988

    8287546933471147759488242667705722847693587263081432870793575134890331

100        *   *        *     **  **  * *** **  *             ***   *    

90      * *   **  * *  **  ********  *************   ** *** ****** ** * 

80 **** * **  *** ********************************** ** *****************

70 ************** *******************************************************

60 **********************************************************************

50 **********************************************************************

40 **********************************************************************

30 **********************************************************************

20 *******#***#**********************###*******#*************************

10 ######################################################################

   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

             0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours

The STP is stable as no TCN are generating, we have checked that the OSPF processes are reflecting causing High CPU.

They are 8 OSPF processes configured for 8 VRFs,and 3 are causing for high cpu spikes.

But I have checked the SPF and neighbour changes ,they are stable.

Kindly suggest.

[Ivan Krimmel]

Hi Amol,

coudl you please elaborate a bit more on what is the issue, especially at what extent OSPF is involed?

I understand that current CPU utilization is high. What do you see in 'sh process cpu sort | ex 0.00%' taken off that box?

Cheers,

Ivan.

[amol1976]

Hi Ivan,

Thanks for your reply,I am not able to collect the 'sh process cpu sort | ex 0.00% at the time of instance as it is intermittant.

I have configured the CPU threshould to capture the processes.as below.

Dec 14 10:00:26.646: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 90%/15%, Top 3 processes(Pid/Util):  524/21%, 402/17%, 528/7%

Dec 14 10:34:16.677: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 27%/10%.

Dec 14 11:07:56.736: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 87%/16%, Top 3 processes(Pid/Util):  524/23%, 402/17%, 528/6%

Dec 14 11:08:06.740: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 29%/12%.

Dec 14 11:38:01.772: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 80%/14%, Top 3 processes(Pid/Util):  524/20%, 402/16%, 528/6%

Dec 14 11:38:11.780: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 24%/8%.

Dec 14 12:11:21.822: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 97%/18%, Top 3 processes(Pid/Util):  524/23%, 402/19%, 528/7%

Dec 14 12:45:06.889: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 29%/10%.

Dec 14 13:18:41.983: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 96%/17%, Top 3 processes(Pid/Util):  524/23%, 402/20%, 528/7%

Dec 14 13:42:37.111: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 29%/8%.

Dec 14 13:52:32.146: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 94%/17%, Top 3 processes(Pid/Util):  524/20%, 402/19%, 528/7%

Dec 14 13:52:42.150: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 21%/8%.

Dec 14 14:59:27.378: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 95%/16%, Top 3 processes(Pid/Util):  524/21%, 402/18%, 528/8%

Dec 14 14:59:37.390: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 23%/8%.

Dec 14 15:33:37.413: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 86%/14%, Top 3 processes(Pid/Util):  524/23%, 402/17%, 528/6%

Dec 14 15:33:47.421: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 25%/9%.

Dec 14 16:07:32.471: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 96%/17%, Top 3 processes(Pid/Util):  524/22%, 402/20%, 528/7%

Dec 14 16:17:57.614: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 26%/9%.

Dec 14 16:41:12.637: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 77%/14%, Top 3 processes(Pid/Util):  524/18%, 402/15%, 528/5%

Dec 14 16:41:22.642: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 22%/8%.

Dec 14 17:14:42.736: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 75%/13%, Top 3 processes(Pid/Util):  524/18%, 402/15%, 528/5%

Dec 14 17:14:52.740: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 22%/9%.

The processes are

524     -  OSPF-4 Router

402     -  OSPF-2 Router

528     -  OSPF-1 Router

But the OSPF database is not so unstable to cause so much spikes.

Also Please find attached..

Show Version

Show ip ospf

show proc cpu hist

show proc cpu

[rsimoni]

Hi Amol,

CPU is ok. What you see is expected as you have BGP peerings - I see from the process BGP scanner.

This process once per minute takes all the CPU cycles it can grasp to check the availibility of all BGP next hops.  Since it is a low priority process should a higher process needs cycles it relinquishes them immediately.

CPU issues are not when you see cpu peaks but wehn you see high averages in cpu utilization - the ones shown by the symbol #.

So don't be worried at all as your device is not misbehaving.

Riccardo

[amol1976]

Hi Riccardo,

I have configured the CPU threshould , the BGP scanner process is not reflecting in HIgh cpu usage only OSPF processes are reflecting.

The high CPU spikes are not occuring over a minute, it is once in a hour.

See the logs attached earlier.

I understand that it is not causing any outage uptill but it is not normal also.