CPU utilisation on 11503

[gmiiller]

I recently upgraded on of our L4 units to an 11503. Following the upgrade (identical configurations, only interface numbers changing etc) the 11503 is sitting on 100% cpu pretty much constantly. - Everything works, but the box is flat-out.

It's very vanilla L4, about 30 mb/s bi-directional traffic, most of which requires routing only by the L4

There are substantial (100+) static routes leading to 60+ different firewalls on the directly connected ethernet segments.

Nothing particularly apparent as a cpu hog. lenstr and bcopy have greatest cpu usage at 4 - 6%. multiple instances of playtask appear at 1-2 milliseconds each..

Anyone got any ideas? are the implicit service tests burning all the cpu, and if so, why wasn't the previous box (a 11152) having the same problem only more so? What can be done to reduce cpu usage if anything?

[seilsz]

Hi-

Do you see any suspicious log messages on the CSS?

What version of code is the CSS running?

Are you managing the CSS through the web interface?

Is the CSS under attack (show dos)?

Can you post the config and a 'flow stat all' from debug mode?

~Zach

[gmiiller]

Zach,

Nope, nothing in logs

Running 7.40.0.04

No to the web interface, console only

A couple of spurious hits in show dos, (it's logically behind some IPS kit) nothing of any consequence.

Sorry, can't post the config (angry-looking ITSM looking over my shoulder)

(debug)# flow stat all

Flow Manager Statistics - Slot 1, Subslot 1:

Cur High Avg

UDP Flows per second 52 1312 59

TCP Flows per second 158 9555 155

Total Flows per second 210 9597 216

Hits per second 8 193 8

Number of Allocated Flows (non-purged) 4179

Number of Free Flows 125357

Number of Flow Drops 0

Max Number of Flow Control Blocks 530942

Accumulated Port Flow Statistics:

Current Number of Active Flows 4179

Total Flow Accounting Reports received 36909177

Total Out of Sequence Packet Received 0

Total Spoof Queue Mis-Hits 0

-------------------------------------------------------------------------

Port CE Active Total Acct TCP UDP

-------------------------------------------------------------------------

# 1/2 200081 1804 15640101 15635309 903 901

# 1/1-13 200340 732 9002319 8983939 377 355

# 1/1-14 200380 677 7608997 7608230 552 125

# 1/1-16 200400 968 4699268 4681699 196 772

Aggregate Flow Manager Statistics:

Cur High Avg

UDP Flows per second 52 1312 59

TCP Flows per second 158 9555 155

Total Flows per second 210 9597 216

end of buffer.

The other thing that I'm noticing is that even when the unit reports that it is at 100% cpu, there are none of the symptoms I've seen previously to do with cpu issues eg: the console is still responsive, and the VRRP standby unit is not trying to assume active status when the active unit has no spare cycles to process the keepalives.

I'm wondering whether the problem might be that the "show sys" results themselves are inaccurate?

[seilsz]

Are there performance problems with client/server connections while the CPU utilization is @ 100%?

Check for large amounts of multicast traffic or errors with 'sh mibii'.

Check for interface errors with 'sh ether-errors'.

~Zach

[gmiiller]

Zach,

There's no sign of any connection issues under load.

No interface errors

Bucketloads of multicast traffic on one particular interface. This is an interface that connects to a pair of routers (running hsrp) and the switch also runs vrrp to its partner on that segment.

[seilsz]

Can you confirm that HSRP and VRRP is the only multicast on that segment? Was this multicast traffic present in the previous configuration?

Capture the output (in debug mode) from CSS the following the next time CPU utilization reaches 100%:

symbol-table load SPRITZ

shell 1 1 spy

shell 1 1 spyReport

shell 1 1 spyStop

symbol-table unload SPRITZ

~Zach

[gmiiller]

Hi Zach,

Yes to HSRP/VRRP only on that segment. The L2 switch that the CSS connects to on that segment also filters bpdus on the CSS port.

There have been no changes to config during the swapover, (other than interface numberings)

It stood out because the same arrangements exist on other interfaces (VRRP on switches, HSRP on routers) but I don't see the same amount of Multicast traffic.

[Gilles Dufour]

how many keepalives ?

Any script keepalive ?

Can we have the output of the cpu hog you captured ?

Also which CPU is high ?

All of them ?

Only SCM ?

Thanks,

Gilles.

[gmiiller]

Gilles,

Around 100, mostly http script, some dns, and a handful of mail.

The unit only has an SCM present

Here's output from some fairly typical cpuhog queries.

Checking CPU Hog

TID Name Milliseconds

--- ---- ------------

0x8a15cd60 tSpare126 1

0x8bb90a70 tIcpRx 1

0x8a0bedf0 PlayTask 1

0x8bb86f50 tTcpRx 1

0x8a6340c0 PlayTask 1

Checking SemID = 0x00000000

TID Name sem MilliSecs

--- ---- -------------

Checking SemID = 0x00000000

TID Name sem MilliSecs

--- ---- -------------

CPU Hog Debug Options (1/1)

d) Display data

Please enter selection [q=Quit]:d

Checking CPU Hog

TID Name Milliseconds

--- ---- ------------

0x8a0e8270 AS_03001ed4.I 1

0x89fd3e50 PlayTask 1

0x8a0c9310 PlayTask 1

0x8a162570 PlayTask 1

0x8bc6fc90 tFlowMgrPktRx 1

Checking SemID = 0x00000000

[Gilles Dufour]

playtask means script.

So it looks like the high cpu comes from all your script.

Try to find a way to reduce number of scripts [use standard kal as much as possible] or increase frequency.

Regards,

Gilles.

[gmiiller]

Gilles,

Yep, I accept that there are more scripts in use than I like. It doesn't explain why a newer (larger) L4 switch is running at sustained higher CPU than the older model it replaced.

Any reasons you can think of?

[Gilles Dufour]

the new switch will have better performance in amount of traffic it can switch due to its distributed architecture.

However, local tasks like keepalives are still perform only by the SCM.

So you're still very limited here to what you can do.

Regards,

Gilles.