Experiencing large output packet drops on Cisco 3945e router

[Rohit Mangotra]

Hi,

I am experiencing slow network performance. I have got two C3900e Software (C3900e-UNIVERSALK9-M), Version 15.2(3)T, routers. One is at our Data center router connected and other is at our headoffice. Both are connected via 500 Mbps fibre (Single Mode). At any given time the router at data center is not pushing more then 200 Mbps. which is making the overall network very slow. When I did some troubleshooting I found output drops on Data Centre router (interface g0/0/0 --> connecting to the fibre link and g0/0 --> interface connected to the wan switch which is further connected to remote sites).

For g0/0/0 --> sh int 

 MTU 1500 bytes, BW 500000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 54/255, rxload 3/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is LX
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 20:45:48
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 338
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/338 (size/max total/drops)
  5 minute input rate 6207000 bits/sec, 7194 packets/sec
  5 minute output rate 107041000 bits/sec, 11479 packets/sec
     697540945 packets input, 3705880110 bytes, 0 no buffer
     Received 11177 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     1087255331 packets output, 1231873543 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

For g0/0 --> sh int

 MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 2/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 20:47:13
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1190964
  Queueing strategy: Class-based queueing
  Output queue: 16/1000/1190923 (size/max total/drops)
  5 minute input rate 1279000 bits/sec, 969 packets/sec
  5 minute output rate 8061000 bits/sec, 1123 packets/sec
     120053809 packets input, 2503345718 bytes, 0 no buffer
     Received 111110 broadcasts (2423 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 111110 multicast, 0 pause input
     158568974 packets output, 4116883448 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     2494 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

These errors are increasing a lot. Any ideas of what is going on here? any help would be really appreciated.

Please see attached network topology for better idea.

Thanks 

Rohit.

[Rohit Mangotra]

Hi Joseph,

Thanks for the reply. we are using EHWIC-1GE-SFP-CU on both routers with 2 GB of DRAM memory to connect via 500 Mbps. We are experiencing very high output packet drops on G0/0 of CoM-DC-Router which then connects to WAN switch (WAN SWITCH) and further it connects to the remote site routers. The packet drop on this interface is increasing a lot. However, there are no errors on any switch ports of WAN switch itself. I am planing to do some more troubleshooting for this issue and will let you know the results. Meanwhile, any recommendation would be appreciated.

Please see more details in the network topology attached above.

Thanks & Regards

Rohit.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Again, how do you run a fiber link at 500 Mbps when your interface is running at gig?  This is an important question, because if you're running over some kind of service that provides subrate bandwidth, then the question becomes, what happens to your packets if you exceed the CIR (i.e. 500 Mbps)?  If they are dropped, then you'll likely want to shape.

Yes, I see how your drops are increasing, but assuming my math is correct, the drop rate is only .7%.  Normally, drops are not considered really adverse until they exceed about 1%.  More concerning is, again, your interface showing queued packets with such an overall low transmission rate; or even the drop rate you're are obtaining.  Also again, this might be indicative of TCP flows being sent into slow start over and over again.

PS:

BTW, I'm unsure of the full specs of the EHWIC, but sometime the module cards and/or module interfaces are bottlenecks compared to the built-in ports.  I.e. there may be benefit to only using this interface for what's expected to be the least busy port.

[Rohit Mangotra]

Hi Joseph,

We have identified some network issue (output drops) on the Data center router’s interface which connects to the  WAN switch.

We also found that there were some packet drops on all the remote sites router WAN interface.

Troubleshooting task done

• If we  Shut down the remote sites router LAN interfaces then there are no total output drops.
• Then we reduced the bandwidth command (from 30000 to 25000) on WAN interface of one of the remote site. On the data center router, we also reduced the bandwidth (from 30000 to 25000) the sub-interface as well as reduced shape average command (from 30000 to 25000) on the policy-map. We found that the number of output drops on the WAN interface of the router at the remote site is less then before.
• We also configured the flow control send desirable command on both the core switches 1,2 interfaces (G2/3, G0/0) connecting to DC and DR routers.
• Then we rebooted both WAN routers (DC and DR).
• Each core switch have SAN connected to. We also resync both SANs.
• We found that the sync speed performance is much better compared to what it was before and the main 500 Mbps fiber link is being fully used.
•  
•  However, we are still experiencing total output drops on WAN interface(G0/0) of the DC router. Any suggestions !!!!

Thank you very much,

KInd Regards

Rohit.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

re: suggestions

Yes, when posters ask for more information, if possible, provide it.  It's difficult to make suggestions without sufficient information.

Also when posters do make suggestions, provide the results, if tried, or your reasons why not tried.

PS:

"Ordinary" Ethernet flow control, might indeed reduce drops, but it does this by generally suspending all data transmission, which can sometime being even more adverse then some drops.

[Rohit Mangotra]

Hi Joseph,

Sorry for not telling you the information that you asked for. Actually we connect the 500 Mbps point to point via single mode fibre. The ISP then does the traffic shaping of 500 Mbps for us on both ends. We are still seeing packet drops including input packet drops as well on one of the remote site router.

FastEthernet0/0 is up, line protocol is up
   Description: --- WAN Link ---
    MTU 1500 bytes, BW 25000 Kbit, DLY 100 usec,
     reliability 255/255, txload 10/255, rxload 142/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 22:15:22
  Input queue: 0/75/126/0 (size/max/drops/flushes); Total output drops: 7099
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 13950000 bits/sec, 1446 packets/sec
  5 minute output rate 981000 bits/sec, 888 packets/sec
     96864577 packets input, 1819719831 bytes
     Received 27450 broadcasts, 0 runts, 0 giants, 15 throttles
     1031 input errors, 0 CRC, 0 frame, 0 overrun, 1031 ignored
     0 watchdog
     0 input packets with dribble condition detected
     57306295 packets output, 2745962933 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     6519892 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

Thank You,

Regards

Rohit.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

It's unusual that your provider shapes, most would police.  If they do shape, then the question arises, how do they manage any queued congestion?  Do they use FIFO or something else?  If something else, what are the criteria?

It's better if you would manage your own shaping.  This both allows you full control over congestion management, both prioritization and drops, and also allows you to "see" how much congestion you're dealing with (often "hidden" from you when a provider does it).

Your latest posting has WAN interface f0/0 but your earlier posting had g0/0 for the WAN interface?  In any case you had 126 ingress drops over 96,864,577 packets, which is such a low percentage, it shouldn't be much adverse to your traffic, but might be mitigated by increasing your queue depth.  You also show 7,099 egress drops over 57,306,295 packets, a much higher percentage, but at 0.012%, this too shouldn't normally be too much of a problem.

You do realize that, at least for TCP, drops are often a major part of TCP flow's rate control?  I.e. some are to be expected.

Although I've noted your drop percentages are low enough that generally they shouldn't be a problem, that's not always true, because we're looking at stats over time.  Your drop rate might be very bad during short term bursts which might throw flows into slow start.  This can be difficult to diagnose (you might want to research microbursts).

Your first posting's g0/0 shows CBQ for the interface, what's the policy, and its stats?