cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2899
Views
0
Helpful
20
Replies
Highlighted
Beginner

Experiencing large output packet drops on Cisco 3945e router

Hi,

I am experiencing slow network performance. I have got two C3900e Software (C3900e-UNIVERSALK9-M), Version 15.2(3)T, routers. One is at our Data center router connected and other is at our headoffice. Both are connected via 500 Mbps fibre (Single Mode). At any given time the router at data center is not pushing more then 200 Mbps. which is making the overall network very slow. When I did some troubleshooting I found output drops on Data Centre router (interface g0/0/0 --> connecting to the fibre link and g0/0 --> interface connected to the wan switch which is further connected to remote sites).

For g0/0/0 --> sh int 

 MTU 1500 bytes, BW 500000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 54/255, rxload 3/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is LX
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 20:45:48
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 338
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/338 (size/max total/drops)
  5 minute input rate 6207000 bits/sec, 7194 packets/sec
  5 minute output rate 107041000 bits/sec, 11479 packets/sec
     697540945 packets input, 3705880110 bytes, 0 no buffer
     Received 11177 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     1087255331 packets output, 1231873543 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

 

For g0/0 --> sh int

 MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 2/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 20:47:13
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1190964
  Queueing strategy: Class-based queueing
  Output queue: 16/1000/1190923 (size/max total/drops)
  5 minute input rate 1279000 bits/sec, 969 packets/sec
  5 minute output rate 8061000 bits/sec, 1123 packets/sec
     120053809 packets input, 2503345718 bytes, 0 no buffer
     Received 111110 broadcasts (2423 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 111110 multicast, 0 pause input
     158568974 packets output, 4116883448 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     2494 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

 

These errors are increasing a lot. Any ideas of what is going on here? any help would be really appreciated.

 

Please see attached network topology for better idea.

 

Thanks 

Rohit.

 

20 REPLIES 20
Highlighted
Hall of Fame Community Legend

One is at our Data center router connected and other is at our headoffice. Both are connected via 500 Mbps fibre (Single Mode). 

3945E is NOT designed to push past 175 Mbps in a single direction.  

 

For 500 Mbps, an ASR 1001-X or 4451-X would be a good fit.  This way, if you need to grow, you can just upgrade the license (instead of buying a brand-new router).

Highlighted

Hi Leo,

Thanks for the reply. Please see wan switch int conf that is connected to the DC router.

interface GigabitEthernet0/1
 description --- Link to WAN Router ---
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 255
 switchport mode trunk
 speed 1000
 duplex full
 mls qos trust cos

 

DC router interface that is connected to this wan switch 

interface GigabitEthernet0/0
 description - To CoM-DC-SW1 G0/1 -
 no ip address
 ip flow ingress
 logging event subif-link-status
 duplex full
 speed 1000
 

I just read a document on Cisco that 3945 router can handle approximately 800 Mbps of data. some time it pushes data up to 390Mbps but for short time then goes back to 200 or less. The only thing we changed is we switched off the flow control command on both Core switches connected to these routers (have a look in the topology for clear view). Do you recon that this might be the issue for large packet drops and slow network?

Thank You,

Kind Regards

Rohit

 

 

Highlighted
Hall of Fame Community Legend

interface GigabitEthernet0/0
 description - To CoM-DC-SW1 G0/1 -
 no ip address
 ip flow ingress
 logging event subif-link-status
 duplex full
 speed 1000

Any reason why the link to your switch has duplex & speed hard-coded?

 

If the switch is Cisco, post the output to the command "sh controller e <SWITCH PORT>".  

Highlighted

Hey Leo,

As far as speed/duplex, shouldn't both sides match, either both hard coded or both set to auto? 

 

Other question, Do routers support DTP?  I see unknown protocols drops.

Highlighted
Hall of Fame Community Legend

Other question, Do routers support DTP?  I see unknown protocols drops.

Yes and no.  All I know is if the port has Dot1Q enabled, unknown protocol drop counters will increment.  

As far as speed/duplex, shouldn't both sides match, either both hard coded or both set to auto? 

Yes and no.  If you're talking about an old switch, like 1900, 2900XL/3500XL, 2940/2950/2955, 3550, then the answer is yes.  Nowadays, speed/duplex settings between Cisco appliances is better if set to "auto".  

 

Setting the speed/duplex is "old school".  

Highlighted

Hi,

Yes it is an old switch WS-C3560-24PS      12.2(35)SE5 6 years old. we want to make sure that the speed and duplex matches from the switch interface connecting to the DC router interface (which is a 1gig) that is why we had it hardcoded.

 

CoM-DC-SW1#sh controller e gigabitEthernet 0/1

     Transmit GigabitEthernet0/1              Receive
    501129633 Bytes                        630423739 Bytes
   1645257630 Unicast frames              2178154631 Unicast frames
   1331728654 Multicast frames             268313943 Multicast frames
         7823 Broadcast frames               1416974 Broadcast frames
            0 Too old frames              3144944737 Unicast bytes
            0 Deferred frames             3868572289 Multicast bytes
            0 MTU exceeded frames           90686336 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames                   7 FCS errors
            0 3 collision frames                   7 Oversize frames
            0 4 collision frames          1891247467 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames          1754650824 Minimum size frames
            0 8 collision frames          2368601050 65 to 127 byte frames
            0 9 collision frames          2150667367 128 to 255 byte frames
            0 10 collision frames         2222968552 256 to 511 byte frames
            0 11 collision frames         4088320917 512 to 1023 byte frames
            0 12 collision frames         2747578735 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  2 Pause frames
            0 15 collision frames
            0 Excessive collisions                 0 Symbol error frames
            0 Late collisions                      7 Invalid frames, too large
            0 VLAN discard frames                  0 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
   3153954846 64 byte frames              1891247467 Valid frames, too small
     30448599 127 byte frames
   1839963663 255 byte frames                      0 Too old frames
   1274072935 511 byte frames                      0 Valid oversize frames
    767249639 1023 byte frames                     0 System FCS error frames
    206271721 1518 byte frames                     0 RxPortFifoFull drop frame
            0 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames

 

Thanks

Kind Regards

Rohit

Highlighted
Hall of Fame Community Legend

12.2(35)SE5

Wow.  Very, very old IOS. 

 

Can you post the output to the command "sh interface Gi0/1"?

 

 

Highlighted

Hi Leo,

I don's see any drops here !!

sh interface Gi0/1

 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 4/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX SFP
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:45, output 00:00:00, output hang never
  Last clearing of "show interface" counters 41w0d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 19391000 bits/sec, 2333 packets/sec
  5 minute output rate 6332000 bits/sec, 2330 packets/sec
     38682528058 packets input, 46253579316131 bytes, 0 no buffer
     Received 46622297 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 46621862 multicast, 0 pause input
     0 input packets with dribble condition detected
     32441671666 packets output, 11196112926344 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

 

Thanks & Regards

Rohit.

 

Highlighted
Hall of Fame Community Legend

Total output drops came from the 3560 switchport.  Total output drops is registered by the SENDING interface (the router) but originates from the receiving port (the switch).

Highlighted

Hi Leo,

Do you think that this might be only reason for overall network performance? But my 1st question where the router are not able to push more then 200 Mbps? Because of this Backups are taking very long time. what do you recon I should do?

 

Thanks & Regards

Rohit.

Highlighted
Hall of Fame Community Legend

But my 1st question where the router are not able to push more then 200 Mbps?

One-way-traffic, yes.  Two-way-traffic, no.

Because of this Backups are taking very long time.

Do you have policy-shaping rules to ensure your router does not attempt to push WAN traffic above your subscribed bandwidth?

Highlighted

Hi Leo,

On both routers we have 

interface GigabitEthernet0/0/0
 description - DR-Router G0/0/0 -
 bandwidth 500000

 

interface GigabitEthernet0/0/0
 description - DC-Router G0/0/0 -
 bandwidth 500000

 

Don't know what is actually going on the network???

Thanks & Regards

Rohit

 

Highlighted
Hall of Fame Community Legend

  Last clearing of "show interface" counters 20:47:13
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1190964

Hold on ... You've just cleared the counters and you get this much output drops?  


Your bottleneck is your LAN.

Full Duplex, 1Gbps, media type is RJ45

Next, the link between the 3945E and your switch is copper.  So why is the speed and duplex setting hard-coded? 

 

What line errors can you see on the interface port of the switch?  Do you see speed/duplex mismatch errors?  Can you see if the switch port is seeing collision errors?

Highlighted
Hall of Fame Expert

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

So how do you have a SM fiber link with just 500 Mbps?  I see the port is running at (the expected) gig.

Your posted g0/0 stats are interesting as it shows 16 packets queued yet a 5 minute transmission rate of only 8 Mbps (on gig)!  I'm wondering whether some TCP flow or flows keeps being thrown back into slow start.  Your overall drop rate, .7%, isn't horrible, although it's generally higher than I like.

PS:

Regarding a 3945e performance, Cisco recommends it for up to 350 Mbps (duplex).  Cisco recommendation is for general all purpose usage, but with a "simple" config, you should be able to get more, even perhaps much more.  What does your CPU usage history look like?

Content for Community-Ad