Cisco 7204VXR OC3 Output drops increasing - Page 2

ahmad82pkn · ‎07-30-2012

Hello Team,

Any idea why my OC3 interface is showing increasing output drops its starts happening as soon traffic exceed 60Mbps ( FYI my traffic never exceeded 75Mbps MAX, and circuit is 150Mbps from carrier , full OC3)

Any thoughts how i can fix this? Below are few stats as starter.

IOS i am using is c7200p-is-mz.122-31.SB10.bin

sh int pos1/1

POS1/1 is up, line protocol is up

Hardware is Packet over Sonet

Internet address is x.x.x.x

MTU 4470 bytes, BW 155000 Kbit, DLY 100 usec,

reliability 255/255, txload 83/255, rxload 36/255

Encapsulation PPP, LCP Open

Stopped: CDPCP

Open: IPCP, crc 16, loopback not set

Keepalive set (10 sec)

Scramble enabled

Last input 00:00:00, output never, output hang never

Last clearing of "show interface" counters 1w1d

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 57260637

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 21988000 bits/sec, 27871 packets/sec

5 minute output rate 51053000 bits/sec, 26240 packets/sec

1754515820 packets input, 3457578272 bytes, 0 no buffer

Received 0 broadcasts (0 IP multicasts)

1 runts, 18 giants, 0 throttles

0 parity

422 input errors, 418 CRC, 0 frame, 0 overrun, 0 ignored, 4 abort

1667079744 packets output, 2951488384 bytes, 0 underruns

0 output errors, 0 applique, 242 interface resets

0 output buffer failures, 0 output buffers swapped out

0 carrier transitions

----------------------------------------------------------------------------------------------

sh controllers pos 1/1

POS1/1

SECTION

LOF = 0 LOS = 0 BIP(B1) = 0

LINE

AIS = 0 RDI = 0 FEBE = 0 BIP(B2) = 0

PATH

AIS = 0 RDI = 0 FEBE = 0 BIP(B3) = 0

PLM = 5 UNEQ = 0 TIM = 0 TIU = 0

LOP = 0 NEWPTR = 0 PSE = 3 NSE = 0

Active Defects: None

Active Alarms: None

Alarm reporting enabled for: SF SLOS SLOF B1-TCA B2-TCA PLOP B3-TCA

Framing: SONET

APS

COAPS = 0 PSBF = 0

State: PSBF_state = False

Rx(K1/K2): 00/00 Tx(K1/K2): 00/00

S1S0 = 00, C2 = 16

Remote aps status (none); Reflected local aps status (none)

CLOCK RECOVERY

RDOOL = 0

State: RDOOL_state = False

PATH TRACE BUFFER: STABLE

Remote hostname : dca-priv-03 so-7/1/2

Remote interface:

Remote IP addr :

Remote Rx(K1/K2): / Tx(K1/K2): /

BER thresholds: SF = 10e-3 SD = 10e-6

TCA thresholds: B1 = 10e-6 B2 = 10e-6 B3 = 10e-6

Clock source: line

Interface POS1/1

dma mode : push

Hardware is Packet over SONET (hwidb=0x3EB91F0)

fastsend = 0x0085CE44, turbofs = 0x05DFBF60

multicast fs = 0x00000000

SFP is SFP-OC3-MM OC3 MM

Serial Number : YAAY0016

Framer is PMC PM5379 S/UNI-4x155 1 0

Receive side:

1455581673090 bytes, 10348619754 frames, 1 overrun interrupts

Transmit side:

3297935862601 bytes, 10260849591 frames, 0 aborts

PHY link is UP

Device instance = 0x03EBB3D8

PCI device base = 0xE0000000, Framer base = 0xE0010000

TX FPGA base = 0xE00A0000, RX FPGA base = 0xE04A0000

TX SRAM base = 0xE0200000, RX SRAM base = 0xE0600000

TX SRAM size (KB) = 0 , RX SRAM size (KB) = 0

TX PLX base = 0xE0000000, RX PLX base = 0xE0400000

PA TX ring base = 0x00000000, PA RX ring base = 0xE04B0000

TX ring base = 0x00000000, RX ring base = 0x385A5E80

tx_upd_addr = 0x38377240, tx_write_addr = 0xE00C0000

txr_mp_shadow = 0x00000000, txr_pak_shadow = 0x00000000

rx_upd_addr = 0x38387340, rx_upd_addr (val) = 91

rxr_shadow = 0x03EBB784, rx_pak = 0x03CA4274

max_particles=36, input_err_rate=0, tx_fullring=0

tx_limited=0, mci_txcount_enabled=0, mci_txcount=256

rx_head=75, tx_head=0, tx_tail=0, tx_count=1

rx_int=3435852815, tx_int=578546419

rx_enabled=1

TX FPGA registers:

reset = 0x2C000100, abrt_cnt = 0x00000000

wr_ptr = 0x0005EAE0, rd_ptr = 0x0005EAE0

part_size = 0x00000000, bd_wr_cnt = 0x02050700

act_bd_idx = 0xAA55AA55, lst_rcvd_bd_idx = 0xAA55AA55

fifo_full = 0xAA55AA55, fifo_full_upd_addr = 0x38377240

led_cntrl = 0x00000000, pak_cnt_back = 0x6398E6AE

pak_cnt_front = 0x6398E6AE, no_traf_lat_timer = 0xAA55AA55

dbg_reg_front = 0x00000000, dbg_reg_back = 0x00000000

RX FPGA registers:

reset = 0x43000008, fifo_full_cnt = 0xAA55AA55

wr_ptr = 0x0004BAD8, rd_ptr = 0x0004BAD4

pkt_index_addr = 0x38387340, hst_rng_strt = 0x385A5E80

part_size = 0xAA55AA55, bd_wr_cnt = 0xAA55AA55

actv_bd_idx = 0x80000077, lst_rcvd_bd_idx = 0x0000006D

lst_part_wr_ptr = 0x0004BD78, int_idle_tmr = 0xAA55AA55

bad_bd_rx_fifo = 0x00000000, clock_select = 0x00000000

sfp_reg = 0x00000020, pak_cnt_host = 0x0072C13D

pak_fmr_good = 0x0072DA6F, pak_fmr_err = 0x00000000

dbg_reg_int_sm = 0x003A0042, dbg_reg_pl3_mict = 0xAA55AA55

TX PLX registers:

cfid = 0x905610B5, cfcs = 0x02B00006, cfrv = 0x068000BA

cflt = 0x0000FF00, cbmem = 0xE0000000, cbio = 0x00000001

cblocal0 = 0xE0000000, cblocal1 = 0x00000000, cbrom = 0x00000000

cfit = 0x00000100

range_pci_local0 = 0xFFC00000, remap_pci_local0 = 0x00010001

mode_arb_reg = 0x03200000, big_little_endian_reg = 0x00300500

range_pci_rom = 0x00000000, remap_pci_rom = 0x0000001F

bus_desc_pci_local0 = 0x410301C3, range_local_pci = 0x00000000

cb_local_pci = 0x00000000, cb_local_pci_io = 0x00000000

remap_local_pci = 0x000004A9, cfg_addr_reg_lcl_pci_io = 0x00000000

dma_chan0_mode = 0x000001C3, dma_chan1_mode = 0x000001C3

range_pci_local1 = 0xFFFFFFF0, remap_pci_local1 = 0x00000000

bus_desc_pci_local1 = 0x0000010B

RX PLX registers:

cfid = 0x905610B5, cfcs = 0x02B00006, cfrv = 0x068000BA

cflt = 0x0000FF00, cbmem = 0xE0400000, cbio = 0x00000001

cblocal0 = 0xE0400000, cblocal1 = 0x00000000, cbrom = 0x00000000

cfit = 0x00000100

range_pci_local0 = 0xFFC00000, remap_pci_local0 = 0x00420001

mode_arb_reg = 0x03200000, big_little_endian_reg = 0x00300500

range_pci_rom = 0x00000000, remap_pci_rom = 0x0000001F

bus_desc_pci_local0 = 0x41030143, range_local_pci = 0x00000000

cb_local_pci = 0x00000000, cb_local_pci_io = 0x00000000

remap_local_pci = 0x000004A9, cfg_addr_reg_lcl_pci_io = 0x00000000

dma_chan0_mode = 0x000001C3, dma_chan1_mode = 0x000001C3

range_pci_local1 = 0xFFFFFFF0, remap_pci_local1 = 0x00000000

bus_desc_pci_local1 = 0x0000010B

paolo bevilacqua · ‎08-02-2012

ahmad82pkn wrote:

Above Gi0/1 information is my LAN side of the router, so you mean they is so much traffic entring the router from LAN ?

note: POS1/1 is my WAN side.

so i need two WAN circuits? if so then what is the benefit of having OC3 card/ circuit on 7204 which is half utilized?

Half utilized on average, but also unable to keep up with bursts coming from three gigabit interface.

Performances aren't cheap

Joseph W. Doherty · ‎08-01-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Input queue drops aren't unusual on 7200 gig interfaces; often indicates CPU can't keep up with a burst of packets at gig rate. BTW, what "engine" are you using?

Like egress queuing, one possible solution is to increase input queue depth. Assuming traffic is bursty, this gives the chance for the CPU to catch up between bursts.

Another possible solution might be to run gig interfaces (especially if you have multiple active) at 100 Mbps, or if possible, have other LAN side device shape slower sending to your 7200. Here the idea is to allow the 7200 more time to process each packet.

NB: running gig interfaces at 100 Mbps shouldn't be too detrimental if your average traffic is only 60 to 75 Mbps, but if you can't shape to the 7200 and you want more than 100 Mbps (to take full advantage of your OC3), another possible solution might be to run two LAN interfaces at 100 Mbps in an Etherchannel configuration.

NB: gig (duplex) Ethernet can require up to about 3 MPPS to maintain wirerate, and even the NPE-G2 only provides 2 MPPS (fast path) - this is why its forwarding can fall behind with gig bursts.

ahmad82pkn · ‎08-10-2012

Hi Team, Sorry for coming back late on this.

Thank you for your useful input on this thread.

after changing queue value from 40 to 100, my one issue is resolved that is previously i had 5 to 8% packet loss on circuit, now packet loss is gone its a big step forward for me.

Although outdrops are still there, i somehow didnt read Joseph second last reply, in which he suggested 600 queue size, i just read it.

i will definitely try it and see if output drops also vanishes with this, as per joseph suggestion i have hold my carrier for juniper to cisco movement on their end for the circuit.

will get back tou you guys soon after chaning queue to 600

here are my current stat at peak time

sh int pos1/1

POS1/1 is up, line protocol is up

Description WAN SIDE

Hardware is Packet over Sonet

MTU 4470 bytes, BW 155000 Kbit, DLY 100 usec,

reliability 255/255, txload 119/255, rxload 50/255

Encapsulation PPP, LCP Open

Stopped: CDPCP

Open: IPCP, crc 16, loopback not set

Keepalive set (10 sec)

Scramble enabled

Last input 00:00:00, output never, output hang never

Last clearing of "show interface" counters 5d11h

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1780162

Queueing strategy: fifo

Output queue: 0/100 (size/max)

5 minute input rate 30751000 bits/sec, 36164 packets/sec

5 minute output rate 72895000 bits/sec, 36605 packets/sec

1510929192 packets input, 3528806243 bytes, 0 no buffer

Received 0 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

0 parity

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

1745501896 packets output, 1331683932 bytes, 0 underruns

0 output errors, 0 applique, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

0 carrier transitions

#######################LAN SIDE######################

sh int gi0/1

GigabitEthernet0/1 is up, line protocol is up

Hardware is MV64460 Internal MAC, address is 0004.9bd6.cc1b (bia 0004.9bd6.cc1 b)

Internet address is 10.80.0.8/24

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 9/255, rxload 17/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full Duplex, 1000Mbps, RJ45, media type is RJ45

output flow-control is XON, input flow-control is XON

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 1w1d

Input queue: 0/75/1/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

30 second input rate 69428000 bits/sec, 29735 packets/sec

30 second output rate 35568000 bits/sec, 32126 packets/sec

3981696301 packets input, 1131224550 bytes, 0 no buffer

Received 20508 broadcasts (166776 IP multicasts)

0 runts, 0 giants, 0 throttles

2 input errors, 0 CRC, 0 frame, 0 overrun, 2 ignored

0 watchdog, 625454 multicast, 0 pause input

4278761853 packets output, 492860961 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out