how to tell whether a router/switch is overloaded

[insert7992]

Hi, all

I have an experiment system consist of a 2610XM, a catalyst2950, several clients and one Linux server. Those clients are running programs that keep sending large amount of http requests(may up to one or two thousands per second) to the server. WCCP is used on the router interface to encapasulate packets and send to the server. The server will deal with those packets and reply.

When the client sending rate is low, the system works fine. However, when the rate is high, performance becames bad. We observed following:

(1). at the server side, the network history chart(sending and receiving rate) have peak-valley curves. Most of the time it runs at the peak(say 1.2MB/sec); then suddently, the curve drop to zeroKB/sec and runs for a short period of time(vary from several seconds to half a minute); after that it eventually goes back to the peak ...

(2), when the rate is high, the router can not be pinged. I guess some buffer/queue is overloaded and can not accept new packets. Am I right?

(3), after a high rate experiment, I issue two command (a)"show interfaces" and (b)"show c2600". Here is the output:

For (a), at the header part, it shows "Input queue: num1/num2/num3/num4 ...", where num3 represent "drops" and it is increased compared to the number before that experiment.

For (b), at the IDMA status section, "Drops = num5", and num5 is increased compared to before.

(4). The best experiment I got so far is that the whole system is being able to process 740 http requests per seconds. And we hope to make this rate even better. By the way, the destination html page is very small, less than 1KB.

Based on above observation, can we say that, in the whole system, the router/switch is definitely the bottle neck? (I'm not sure if there is a problem on the server side, but I want to make sure first that the problem doesn't come from router/switch). How can I find the capacity of a router/switch.

Thanks in advance.

Joe

[dmooreami · Accepted solution]

this will work with any IOS device:

sho process cpu | exc 0.00

For routers:

sho process cpu history

that give a ASCII graph of the cpu history over several hours and

Make sure you have IP Cef enabled on the router. I think the 2610 supports that

Your bottleneck is going to be the router. The 2950 is an L2 device and its ports can be pushed to almost 100meg.

The router might have a 100meg port, but the actual "routing" speed is much less.

Check into IP CEF:

Cisco's Express Forwarding (CEF) is an advanced, Layer 3 switching

technology inside a router. It defines the fastest method by which a

Cisco router forwards packets from ingress to egress interfaces.

Process switching is the lowest common denominator in switching paths;

it is available on every version of IOS, on every platform.

I beleave this is the defaul method used.

Process switching uses the CPU on every packet, CEF only needs to the

CPU for the first packet of each session.

------

[insert7992]

Thank a lot, dmooreami.

[insert7992]

Hi, dmooreami,

I tried your suggestion and found that the CPU does reach more than 95% in a tough experiment.

Some more questions please.

(1). As 2600 series router documentation says, the max scalability of the router should be 25000packets/sec. I guess in my experiment, the packet number is much less than that(the highest rate of dealing with http request is about 740req/sec). And the html page each http request trying to get is very small--about 800Bytes. It doesn't make sense to me that each http transaction cost 30~40 packets. What would be the problem?

(2). as you mentioned, the actual routing speed is much less than 100MB/sec, how would the actual routing speed could be? At the server side, the max sending/receing rate(per second) I observed is 1.2MB/800KB, which sum up to 2MB/sec at most. Does it sound reasonable?

Thanks.

Joe

[sirdudesly]

what are the processes that are causing high CPU usage?

Are you inspecting the packets etc?

When you start implementing more things obviously your MAX starts to lower.

[insert7992]

Hi, sirdudesly,

I attach the output of some commands here. Not quite sure how to read those processes. The over all 5 sec process usage is about 90%, but the highest CPU usage for a process is at most 3%, and most of others are 0%. Am I missing something? Could you please help me out of this.

Thanks a lot.

Joe