cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2277
Views
0
Helpful
5
Replies

Cisco 7204 VXR w/ NPE-G1 cpu maxing out

wmusselwhite
Level 1
Level 1

I have a cisco 7204 vxr that terminates a 300 meg ethernet circuit asn well as an mpls DS-3.  CPU increases along with utilization of the ethernet circuit.  When the utilization gets to around 150 Mbps on the receive, the cpu is maxed out at 100%.  I am wondering if the router can support the amount of traffic coming through it.  The majority of the traffic is voip using g729 codec, so packet size is small.  We are no where close to peak utilization and cpu is at 39%.   Here is what I see currently:

#sh ver
Cisco IOS Software, 7200 Software (C7200-ADVIPSERVICESK9-M), Version 12.4(15)T4, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Thu 13-Mar-08 10:40 by prod_rel_team

ROM: System Bootstrap, Version 12.3(4r)T3, RELEASE SOFTWARE (fc1)
BOOTLDR: 7200 Software (C7200-KBOOT-M), Version 12.3(15), RELEASE SOFTWARE (fc3)

uptime is 3 years, 1 week, 3 days, 6 hours, 40 minutes
System returned to ROM by Reload Command
System restarted at 08:26:49 UTC Wed May 14 2008
System image file is "disk2:c7200-advipservicesk9-mz.124-15.T4.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 7204VXR (NPE-G1) processor (revision B) with 491520K/32768K bytes of memory.
Processor board ID 26817669
SB-1 CPU at 700MHz, Implementation 1025, Rev 0.2, 512KB L2 Cache
4 slot VXR midplane, Version 2.6

Last reset from power-on

PCI bus mb1 (Slots 1, 3 and 5) has a capacity of 600 bandwidth points.
Current configuration on bus mb1 has a total of 100 bandwidth points.
This configuration is within the PCI bus capacity and is supported.

PCI bus mb2 (Slots 2, 4 and 6) has a capacity of 600 bandwidth points.
Current configuration on bus mb2 has a total of 0 bandwidth points.
This configuration is within the PCI bus capacity and is supported.

Please refer to the following document "Cisco 7200 Series Port Adaptor
Hardware Configuration Guidelines" on Cisco.com <http://www.cisco.com>
for c7200 bandwidth points oversubscription and usage guidelines.


3 Gigabit Ethernet interfaces
1 Serial interface
509K bytes of NVRAM.

62720K bytes of ATA PCMCIA card at slot 2 (Sector size 512 bytes).
16384K bytes of Flash internal SIMM (Sector size 256K).
Configuration register is 0x2102

#sh proc cpu            
CPU utilization for five seconds: 38%/36%; one minute: 39%; five minutes: 39%

#sh inventory
NAME: "Chassis", DESCR: "Cisco 7204VXR, 4-slot chassis"
PID: CISCO7204VXR      , VID:    ,

NAME: "NPE-G1 0", DESCR: "Cisco 7200 Series Network Processing Engine NPE-G1"
PID: NPE-G1            , VID:    ,

NAME: "disk2", DESCR: "64MB Compact Flash Disk for NPE-G1"
PID: MEM-NPE-G1-FLD64  , VID:    , SN:          

NAME: "module 1", DESCR: "Serial T3+"
PID: PA-T3+=           , VID:    ,

NAME: "Power Supply 1", DESCR: "Cisco 7200 AC Power Supply"
PID: PWR-7200-AC       , VID:    , SN:          

NAME: "Power Supply 2", DESCR: "Cisco 7200 AC Power Supply"
PID: PWR-7200-AC       , VID:    , SN:          

GigabitEthernet0/1 is up, line protocol is up
  Hardware is BCM1250
  Description: (MPLS LAN)
  Internet address is x.x.x.x
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:52, output 00:00:00, output hang never
  Last clearing of "show interface" counters 04:48:49
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 87000 bits/sec, 164 packets/sec
  5 minute output rate 2697000 bits/sec, 243 packets/sec
     2304373 packets input, 185375406 bytes, 0 no buffer
     Received 336 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 9151 multicast, 0 pause input
     0 input packets with dribble condition detected
     3199907 packets output, 4063355247 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

#sh int gi 0/2
GigabitEthernet0/2 is up, line protocol is up
  Hardware is BCM1250
  Description: "WAN (300M)"
  Internet address is x.x.x.x
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 3/255, rxload 12/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, link type is force-up, media type is ZX
  output flow-control is unsupported, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 04:48:51
  Input queue: 0/75/1/561 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 50072000 bits/sec, 80406 packets/sec
  30 second output rate 14841000 bits/sec, 12179 packets/sec
     844705896 packets input, 1777600166 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     199 input errors, 0 CRC, 0 frame, 199 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     115272242 packets output, 482335841 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

#sh int gi 0/3
GigabitEthernet0/3 is up, line protocol is up
  Hardware is BCM1250
  Description: "LAN 300M"
  Internet address is x.x.x.x
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 12/255, rxload 3/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 00:01:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 04:48:52
  Input queue: 0/75/1/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 14931000 bits/sec, 12365 packets/sec
  30 second output rate 50159000 bits/sec, 80590 packets/sec
     118538597 packets input, 680510291 bytes, 0 no buffer
     Received 41003 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 16981 multicast, 0 pause input
     0 input packets with dribble condition detected
     848042180 packets output, 1976487006 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

#sh int serial 1/0
Serial1/0 is up, line protocol is up
  Hardware is M1T-T3+ pa
  Description:  DS3 MPLS WAN
  Internet address is x.x.x.x
  MTU 4470 bytes, BW 10500 Kbit, DLY 200 usec,
     reliability 255/255, txload 2/255, rxload 58/255
  Encapsulation PPP, LCP Open
  Open: IPCP, crc 16, loopback not set
  Keepalive set (10 sec)
  Restart-Delay is 0 secs
  Last input 00:00:17, output 00:00:00, output hang never
  Last clearing of "show interface" counters 04:53:01
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/31/256 (active/max active/max total)
     Reserved Conversations 4/4 (allocated/max allocated)
     Available Bandwidth 6947 kilobits/sec
  5 minute input rate 2429000 bits/sec, 231 packets/sec
  5 minute output rate 89000 bits/sec, 162 packets/sec
     3257899 packets input, 4100695534 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
              0 parity
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     2344248 packets output, 164998094 bytes, 0 underruns
     0 output errors, 0 applique, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
   rxLOS inactive, rxLOF inactive, rxAIS inactive
   txAIS inactive, rxRAI inactive, txRAI inactive

5 Replies 5

vmiller
Level 7
Level 7

Had a similar experience with a '06 running with a G2. The culprit was the small packets.

We ended up migrating to a 7600. I don't think (but can't prove) a g2 upgrade will fix everything.

Joseph W. Doherty
Hall of Fame
Hall of Fame

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind.  Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

In theory, G1 offers 1 Mpps, which should allow up to about 672 Mbps, simplex, for Ethernet min sized packets.  Theoretical performance doesn't account for other things router needs to do.  You might carefully review your configuration and see if configured for optimal performance.  However, in the end, you may just need a "faster" router to deal with the offered load.

wmusselwhite
Level 1
Level 1

I forgot to mention earlier that we see a lot of errors on the interfacing for the 300 meg circuit (int gi 0/2).  It shows as input errors and overruns.  During peak times we are seeing close to 1 million errors every 10 minutes.  Are these errors a result of the load on the router, or could the errors be coming from something else and causing the load on the router?

I'd check the physical path real carefully.

My guess is its some sort of subrate mechanisim that ties into a sonet service (anda..)

possibly some timing issues...

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind.  Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting


The overruns are likely just another symptom of your router unable to keep up with the offered traffic load.  To quote Cisco:

overrun

Shows the number of times that the receiver hardware was incapable of  handing received data to a hardware buffer because the input rate  exceeded the receiver's capability to handle the data.

What might help deal with these errors, if possible, is to slow some/all your gig interfaces to 100 Mbps.  This will throttle some burst demand on the CPU.

BTW, we already are seeing your CPU run flat out, so slowing gig interfaces shouldn't really reduce the performance you're obtaining now.

PS:

The input interface queue drops/flushes is another symptom of a router that's overtaxed.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Review Cisco Networking products for a $25 gift card