topic Re: CCS11050 High CPU. in Application Networking

CCS11050 High CPU.

colin.lowe — Fri, 07 Dec 2007 12:53:25 GMT

Please can you help me identify what is causing high CPU on a CSS11050 device?

Details of device are -

chdccss01# sh ver

Version: ap0610306s (6.10 Build 306)

Flash (Locked): 4.01 Build 2

Flash (Operational): 6.10 Build 306

Type: PRIMARY

Licensed Cmd Set(s): Standard Feature Set

Secure Management

chdccss01# sh chassis

Configuration for CSS 11150:

Name: CSS 11150 SW Version: 6.10 Build 306

HW Major Version: 03 HW Minor Version: 0

Serial Number: 21480063243

Base Mac Address: 00-10-58-03-50-45

Module Number Module Name Status

1 FEM primary

2 FEM primary

5 SCFM-PLUS primary

Port Number Port Name Status

1 e1 online

2 e2 online

3 e3 online

4 e4 online

5 e5 online

6 e6 online

7 e7 online

8 e8 online

9 e9 online

10 e10 online

11 e11 online

12 e12 online

13 e13 online

14 e14 online

15 e15 online

16 e16 online

chdccss01# sh system-resources

System Resources:

Installed Memory: 134,217,728 (128 MB)

Free Memory: 20,833,744 (19 MB)

CPU: 98%

Buffer Statistics:

Buffer Pool: 0

Size: 256 Total:1000 Available:1000 Failures: 0 Low Buffer Count: 688

Buffer Pool: 1

Size:1600 Total:9000 Available:8128 Failures: 0 Low Buffer Count: 7585

chdccss01#

Re: CCS11050 High CPU.

Gilles Dufour — Fri, 07 Dec 2007 14:02:00 GMT

do the following.

go to debug mode - command is 'llama'

load symbol-table.

execute cpu hog.

Press 'd' several times to get a list of active processes.

then do unload symbol-table.

gduf-css11000-1(debug)# symbol-table load

Loading string table... 583,460

Loading symbols........ 21,561

Load complete.......... 1,100,924

gduf-css11000-1(debug)# cpu hog

CPU Hog Debug Options

d) Display data

Please enter selection [q=Quit]:d

Checking CPU Hog

TID Name Milliseconds

--- ---- ------------

0x836d92e0 PingKal 0

0x8768b4b0 tASSync 0

0x8375e5f0 AS_00000080.I 0

0x8584d3a0 consoleOB 0

0x876b9bc0 apRdyActvLed 0

Re: CCS11050 High CPU.

colin.lowe — Fri, 07 Dec 2007 14:15:14 GMT

Thanks for quick response.

Output is in attached file.

Thanks,

Colin.

Re: CCS11050 High CPU.

Gilles Dufour — Fri, 07 Dec 2007 15:12:43 GMT

all these processes are responsible to handle traffic.

Seems like you have reach some limit.

Get a 'flow stat' from llama mode so we can see amount of connections.

Gilles.

Re: CCS11050 High CPU.

colin.lowe — Fri, 07 Dec 2007 15:57:48 GMT

Thanks Giles,

Details are -

chdccss01(debug)# flow statistics

Flow Manager Statistics:

Cur High Avg

UDP Flows per second 22 502 15

TCP Flows per second 42 310 39

Total Flows per second 64 624 55

Hits per second 0 0 0

Number of Allocated Flows (non-purged) 3451

Number of Free Flows 94853

Number of Flow Drops 0

Max Number of Flow Control Blocks 159359

Number of Flows with SynAckPending 83

Accumulated Port Flow Statistics:

Current Number of Active Flows 3474

Total Flow Accounting Reports received 25690664

Total Out of Sequence Packet Received 165

Total Spoof Queue Mis-Hits 0

-------------------------------------------------------------------------------

Port CE Active Total Acct TCP UDP Rst FCBs

-------------------------------------------------------------------------------

#e1 1f00 2 7 5 2 0 0 15231

#e13 3001f00 1290 12715311 12714022 1027 263 0 26216

#e14 3401f00 0 12125 12121 0 0 0 26195

#e15 3801f00 1296 6241815 6240727 1171 125 0 26211

#e16 3c01f00 886 6722438 6723789 757 137 0 26214

chdccss01(debug)#

Cheers,

Colin

Re: CCS11050 High CPU.

Gilles Dufour — Fri, 07 Dec 2007 23:11:02 GMT

Colin,

that's not that much of flows, but it depends what kind of rules you have.

Any L7 rule ?

Do you have some acl logging ?

Do you have a lot of keepalives ?

I can confirm the processes consuming the cpu are the ones handling the traffic.

So, this box is running at its max.

Gilles.

Re: CCS11050 High CPU.

colin.lowe — Thu, 13 Dec 2007 08:39:11 GMT

Gilles,

Thankyou for the information. As a result we looked at the traffic flow in more detail through the CSS and it appears that we have some type of looping FIN/ACK traffic which is generating high volumes of small packets.

When this traffic is blocked the CSS CPU returns to normal (4-5%).

Thankyou for your assistance.

Kind Regards,

Colin.