Cisco 2801 with CPU Overload Issue

[dialtelusa]

Hello,

We are doing SIP Termination and have no need for any voice solutions built into the router. Rather we have our own SIP Proxy that needs to talk to carriers. We need the cisco to configure our IP Block but we are just looking to use bandwidth to send our traffic out.... Well we noticed that when we run only 100 channels we were getting bad voice quality. I noticed that the router CPU was running at 100% while memory and bandwidth were still rather low.

Is there a setting to help with this? Are we in need of a different device? I have figured out that the SIP Signalling is not the issue and that is seems the load is created due to the media RTP packets that need to communicate back and forth with our carriers.

We want to handle about 20,000 channels atleast as on our private network this is no issue currently because we send our traffic from our switch to the carrier thus bypassing our router. But now that we are testing public termination ourselves we were expecting to run about 1200-1500 lines of G.711 through the Cisco2801 figuring that would be about 80Mbps on a 100Mbps rated router.

Any help would be greatly appreciated.

Thx

Erik

[paolo bevilacqua · Accepted solution]

You need a much faster router for that. Like a 3925E, ot 7200  NPE G1 or G2.

[dialtelusa]

Its so interesting you say that because I was just looking at the Cisco 7200VXR Series NPE-G2. Do you happen to know however if that is a 6 slot capable of running (6) NPE-G2 at one time? I think just one NPE-G2 although rated to do 2 million PPS probably would only handle around 2000 lines...

[ADAM CRISP · Accepted solution]

Hi Eric,

A 7200 has one processing engine only - the slots are for line interface cards.

20,000 calls @ 100pps = 2,000,000pps as you already stated, that's a lot of calls by the way ;-)

Also using overly basic calculation of 10 calls per mb of traffic, that that's 2Gbit stream.

It's possible if you're connecting to lots of carriers and each of these carriers has a separate circuit, than the 7200 might be OK for connecting a circuit to each carrier, but you'd have more that one router. You'll also need to consider your LAN switching infrastructure and make sure that's up to the job.

Adam

[dialtelusa]

Hey Adam,

I figured out the part about 1 processing engine only.

However I am still confused about the numbers and how people are interpreting them. I called Cisco and I assume I spoke with someone that was not cisco certified because they did not completely understand my needs. Let me go into more detail about some confusions that I have...... There might be some random things mentioned here but I hope you can decipher what I say as a whole.

Off the cuff the guy at cisco was talking to me about 3 devices... The Cisco2801, Cisco3945, and the Cisco7204VXR-NPE-2.

He mentioned at first that the devices could do the following.....

3000 Voice Packets (Cisco2801)

10,000 Voice Packets (Cisco3945)

20,000 Voice Packets (Cisco7204VXR-NPE-G2)

At first he was recommending the Cisco3945 to me. I explained that I could only run about 50 lines before my processor maxed out. So this made me think that the 3945 would handle 150 lines and the 7204 would handle about 2000 channels....

Later in the conversation he mentioned I was actually right and we were talking about pps numbers....

Anyway later I found some good docs which I will attach here..

In the router poster the only similar product I found was the Cisco3945... I wont even touch on the 3945E which seemed much better... Anyway it says that can handle 1000 sessions of SIP Termination. I didnt find anything as fas as what the pps is rated on for this model...

So lets now jump to the routerperformace pdf. This one was very interesting and will ultimately lead me to my real question...

In routerperformance doc I found two columns. One for Process Switching and the other for Fast/CEF Switching.

Only one model was identical to mine and there was the 7200 NPE-1 model... Those stated the following...

Cisco2801 can do 3000pps of Process Switching and 70,000pps of Fast Switching.....

I have that model and it only runs 50 lines before reaching about 90 percent CPU so I have to assume that I am doing Process Switching.

So with that math I now look at the 7200-NPE-G1 which can handle 79,000 process switching and 1,000,000 Fast Switching. That makes me think my traffic would only handle 1250 lines even on that high end model... The NPE-2 I hear is twice as fast with 2,000,000 Fast Switching but I found nothing about Process switching for that unit.... Lets assume it is 150,000. That would mean that its 50 times as fast as my 2801 with process switching so I could only run 2500 channels on that device....

That is hard to understand for me because in the Routing Poster the 3945E is said to handle 2500 Channels of SIP Termination... But I have no Process or Fast Switching stats to compare that number to....

So at this point I do not know how to measure what our traffic is like. You mentioned 100pps per call which I find very interesting because that seems to be about right when looking at my PPS Charts for my test runs. However your math was based on the Fast Switching rather than the process switching.

We technically have no need for Cisco modules or software to handle SIP Termination as we run from our own proxy servers. We just need to pass our traffic through the device.

Is there a way for me to not use the processor on my cisco but rather just the fast switching. Maybe some special command or routing protocol? Because mine is rated at 70,000 and that would technically make me think going by your math that I should be able to do 700 calls.

Or maybe SIP Termination because of the type of traffic the cisco automatically has to use processing power to handle the sessions..... If thats the case then I think I should be basing your 100pps off the routers processing pps capability and not the Fast Switching pps capability.

I really appreciate your taking the time here. It means so much to me. I have been losing sleep the last few nights trying to understand. Cisco has so much information but it is spread around so much that I wish they would have a chart that would have all the stats such as Fast Switching, Process Switching, and SIP Sessions all in one chart as this would make my life so much easier.

[dialtelusa]

Just to further reiterate... We dont need to connect physically for each carrier. We actually program the IP of the carriers on our internal servers and send traffic out from our SIP Proxies through the switch and router. Our SIP Platform has been running privately to one carrier for years and we are now trying to do public termination on our own so we can connect to multiple carriers for lower rates. We never ran into a problem in the past because our traffic went from a disjoint switch to the carrier and our data ran through the Cisco 2801 seperate from the SIP traffic. With the introduction of this new router I would seperate the SIP again onto the 7204VXR and have a disjoint network for customers to access our site for quality purposes. All we would need from the router is simply to program a block of IP's and route traffic from our internal servers to our carriers around the country.

[ADAM CRISP · Accepted solution]

Hi ,

Ok, so here's my simple explanation. The technicalities of each type of forwarding depend on the architecture of the router but in general we have:

Process switching - Each packets as it arrives is processed by the CPU of the router. This is slow.

Fast switching - Here the first packet of a flow is process switched, and then the path is stored in a cache that is written to the line cards, so future packets can be sent to the outgoing interface directly

CEF - Here a forwarding base is built based on the routing table and pushed out into the cach for quick forwarding all the time.

What switching is configured on your router, I would have thought it's unusual for switching to be disabled on your router.

Look at your config and look for any commands - no ip route cache - or you can look at the switching features with the show ip interface command..................

Adam

[dialtelusa]

IP Cef is enabled which is why it has the no ip route-cache cef command below

interface FastEthernet0/0
description $ES_WAN$$FW_OUTSIDE$
ip address ******************************************************
ip address ********************************
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
no ip route-cache cef
ip route-cache flow
duplex auto
speed auto
no mop enabled

[ADAM CRISP · Accepted solution]

Please can you post the OP of show ip interface fast 0/0 ??

Adam

[dialtelusa]

Thanks

#show interface

FastEthernet0/0 is up, line protocol is up

  Hardware is Gt96k FE, address is 000a.b8d8.fbde (bia 000a.b8d8.fbde)

  Description: $ES_WAN$$FW_OUTSIDE$

  Internet address is**********************

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, 100BaseTX/FX

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:00, output 00:00:00, output hang never

  Last clearing of "show interface" counters never

  Input queue: 1/75/39597595/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 5000 bits/sec, 10 packets/sec

  5 minute output rate 1000 bits/sec, 2 packets/sec

     241528901 packets input, 1090672429 bytes

     Received 90575745 broadcasts, 0 runts, 0 giants, 0 throttles

     376 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog

     0 input packets with dribble condition detected

     151304281 packets output, 4223810560 bytes, 0 underruns

     0 output errors, 0 collisions, 10 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

FastEthernet0/1 is up, line protocol is down

  Hardware is Gt96k FE, address is 000a.b8d8.fbdf (bia 000a.b8d8.fbdf)

  Description: $ETH-LAN$$ETH-SW-LAUNCH$$INTF-INFO-FE 0$$ES_LAN$$FW_INSIDE$

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Auto-duplex, Auto Speed, 100BaseTX/FX

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output never, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

     0 packets input, 0 bytes

     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog

     0 input packets with dribble condition detected

     295100 packets output, 17706000 bytes, 0 underruns

     0 output errors, 0 collisions, 1 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

NVI0 is up, line protocol is up

  Hardware is NVI

  MTU 1514 bytes, BW 10000000 Kbit, DLY 0 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation UNKNOWN, loopback not set

  Last input never, output never, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

     0 packets input, 0 bytes, 0 no buffer

     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

     0 packets output, 0 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 output buffer failures, 0 output buffers swapped out

[ADAM CRISP · Accepted solution]

Erik  "show IP interface fast 0/0" , not show interface...

While you're there, please post "show interace stats" and "show interfaces switching"

Adam

[dialtelusa]

Show IP Interface Fast 0/0

CAuthorized access only!
Disconnect IMMEDIATELY if you are not an authorized user!

User Access Verification

Username: dialtel
Password: C
% Password expiration warning.

dialtel#show ip interface fast0/0
FastEthernet0/0 is up, line protocol is up
  Internet address is ****************
  Broadcast address is *****************************
  Address determined by non-volatile memory
  MTU is 1500 bytes
  Helper address is not set
  Directed broadcast forwarding is disabled
  Secondary address *****************************
  Outgoing access list is not set
  Inbound  access list is not set
  Proxy ARP is disabled
  Local Proxy ARP is disabled
  Security level is default
  Split horizon is enabled
  ICMP redirects are never sent
  ICMP unreachables are never sent
  ICMP mask replies are never sent
  IP fast switching is enabled
  IP fast switching on the same interface is disabled
  IP Flow switching is enabled
  IP CEF switching is disabled
  IP Flow switching turbo vector
  IP multicast fast switching is enabled
  IP multicast distributed fast switching is disabled
  IP route-cache flags are Fast, Flow cache, No CEF, Full Flow
  Router Discovery is disabled
  IP output packet accounting is disabled
  IP access violation accounting is disabled
  TCP/IP header compression is disabled
  RTP/IP header compression is disabled
  Policy routing is disabled
  Network address translation is enabled, interface in domain outside
  BGP Policy Mapping is disabled

Show Interface Stats

FastEthernet0/0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor  241216988 1022719761  150965074 4154214208
             Route cache     342996   69968545     342996   69943193
                   Total  241559984 1092688306  151308070 4224157401
FastEthernet0/1
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor          0          0     295224   17713440
             Route cache          0          0          0          0
                   Total          0          0     295224   17713440
NVI0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor          0          0          0          0
             Route cache          0          0          0          0
                   Total          0          0          0          0

Show Interfaces Switching

FastEthernet0/0 $ES_WAN$$FW_OUTSIDE$
          Throttle count          0
                   Drops         RP   39597595         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs   88577885      Drops          0

    Protocol  IP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process  152637375 4284088476  150477852 4124983131
            Cache misses          0          -          -          -
                    Fast     342996   69968545     342996   69943193
               Auton/SSE          0          0          0          0

    Protocol  ARP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process   88533452 1017039854     192090   11525400
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      49125   16751625     295240   17714400
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.
FastEthernet0/1 $ETH-LAN$$ETH-SW-LAUNCH$$INTF-INFO-FE 0$$ES_LAN$$FW_INSIDE$

    Protocol  Other
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0     295240   17714400
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.
NVI0

    All statistics for this interface are zero.

-------------------------------------------------------------------------------------------------------------

Thx

[ADAM CRISP · Accepted solution]

OK,

So the stats show a lot of process switching - are you using the Netflow feature?

Do you know why CEF is switched off ?

If this is a lab enviroment - try enabling it and disable netflow accounting.

Also look at your other interfaces - disabling cef has an effect on ingress traffic only.

Adam

[dialtelusa]

Very interesting. I do not know why CEF is switched off. When I show the running config IP CEF is listed. I dont know if this is a interface specific command as the IP CEF is listed outside of the interface configs. Also I have 2 IP Addresses on the same interface and the secondary address is actually our IP block. I saw in that   IP fast switching on the same interface is disabled. So I wonder know if I can enable fast switching for each interface somehow...

THanks for the help!

[dialtelusa]

This is currently a lab environment so I am free to make changes...