Hi Leo,Thanks for the reply.

Rohit Mangotra · ‎03-09-2015

Hi,

I am experiencing slow network performance. I have got two C3900e Software (C3900e-UNIVERSALK9-M), Version 15.2(3)T, routers. One is at our Data center router connected and other is at our headoffice. Both are connected via 500 Mbps fibre (Single Mode). At any given time the router at data center is not pushing more then 200 Mbps. which is making the overall network very slow. When I did some troubleshooting I found output drops on Data Centre router (interface g0/0/0 --> connecting to the fibre link and g0/0 --> interface connected to the wan switch which is further connected to remote sites).

For g0/0/0 --> sh int

MTU 1500 bytes, BW 500000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 54/255, rxload 3/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is LX
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 20:45:48
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 338
Queueing strategy: Class-based queueing
Output queue: 0/1000/338 (size/max total/drops)
5 minute input rate 6207000 bits/sec, 7194 packets/sec
5 minute output rate 107041000 bits/sec, 11479 packets/sec
697540945 packets input, 3705880110 bytes, 0 no buffer
Received 11177 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
1087255331 packets output, 1231873543 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

For g0/0 --> sh int

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 2/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 20:47:13
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1190964
Queueing strategy: Class-based queueing
Output queue: 16/1000/1190923 (size/max total/drops)
5 minute input rate 1279000 bits/sec, 969 packets/sec
5 minute output rate 8061000 bits/sec, 1123 packets/sec
120053809 packets input, 2503345718 bytes, 0 no buffer
Received 111110 broadcasts (2423 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 111110 multicast, 0 pause input
158568974 packets output, 4116883448 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
2494 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

These errors are increasing a lot. Any ideas of what is going on here? any help would be really appreciated.

Please see attached network topology for better idea.

Thanks

Rohit.

Leo Laohoo · ‎03-09-2015

One is at our Data center router connected and other is at our headoffice. Both are connected via 500 Mbps fibre (Single Mode).

3945E is NOT designed to push past 175 Mbps in a single direction.

For 500 Mbps, an ASR 1001-X or 4451-X would be a good fit. This way, if you need to grow, you can just upgrade the license (instead of buying a brand-new router).

Rohit Mangotra · ‎03-09-2015

Hi Leo,

Thanks for the reply. Please see wan switch int conf that is connected to the DC router.

interface GigabitEthernet0/1
description --- Link to WAN Router ---
switchport trunk encapsulation dot1q
switchport trunk native vlan 255
switchport mode trunk
speed 1000
duplex full
mls qos trust cos

DC router interface that is connected to this wan switch

interface GigabitEthernet0/0
description - To CoM-DC-SW1 G0/1 -
no ip address
ip flow ingress
logging event subif-link-status
duplex full
speed 1000

I just read a document on Cisco that 3945 router can handle approximately 800 Mbps of data. some time it pushes data up to 390Mbps but for short time then goes back to 200 or less. The only thing we changed is we switched off the flow control command on both Core switches connected to these routers (have a look in the topology for clear view). Do you recon that this might be the issue for large packet drops and slow network?

Thank You,

Kind Regards

Rohit

Leo Laohoo · ‎03-09-2015

interface GigabitEthernet0/0
 description - To CoM-DC-SW1 G0/1 -
 no ip address
 ip flow ingress
 logging event subif-link-status
 duplex full
 speed 1000

Any reason why the link to your switch has duplex & speed hard-coded?

If the switch is Cisco, post the output to the command "sh controller e <SWITCH PORT>".

Charles Hill · ‎03-09-2015

Hey Leo,

As far as speed/duplex, shouldn't both sides match, either both hard coded or both set to auto?

Other question, Do routers support DTP? I see unknown protocols drops.

Leo Laohoo · ‎03-09-2015

Other question, Do routers support DTP?  I see unknown protocols drops.

Yes and no. All I know is if the port has Dot1Q enabled, unknown protocol drop counters will increment.

As far as speed/duplex, shouldn't both sides match, either both hard coded or both set to auto?

Yes and no. If you're talking about an old switch, like 1900, 2900XL/3500XL, 2940/2950/2955, 3550, then the answer is yes. Nowadays, speed/duplex settings between Cisco appliances is better if set to "auto".

Setting the speed/duplex is "old school".

Rohit Mangotra · ‎03-09-2015

Hi,

Yes it is an old switch WS-C3560-24PS 12.2(35)SE5 6 years old. we want to make sure that the speed and duplex matches from the switch interface connecting to the DC router interface (which is a 1gig) that is why we had it hardcoded.

CoM-DC-SW1#sh controller e gigabitEthernet 0/1

Transmit GigabitEthernet0/1 Receive
501129633 Bytes 630423739 Bytes
1645257630 Unicast frames 2178154631 Unicast frames
1331728654 Multicast frames 268313943 Multicast frames
7823 Broadcast frames 1416974 Broadcast frames
0 Too old frames 3144944737 Unicast bytes
0 Deferred frames 3868572289 Multicast bytes
0 MTU exceeded frames 90686336 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 7 FCS errors
0 3 collision frames 7 Oversize frames
0 4 collision frames 1891247467 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 1754650824 Minimum size frames
0 8 collision frames 2368601050 65 to 127 byte frames
0 9 collision frames 2150667367 128 to 255 byte frames
0 10 collision frames 2222968552 256 to 511 byte frames
0 11 collision frames 4088320917 512 to 1023 byte frames
0 12 collision frames 2747578735 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 2 Pause frames
0 15 collision frames
0 Excessive collisions 0 Symbol error frames
0 Late collisions 7 Invalid frames, too large
0 VLAN discard frames 0 Valid frames, too large
0 Excess defer frames 0 Invalid frames, too small
3153954846 64 byte frames 1891247467 Valid frames, too small
30448599 127 byte frames
1839963663 255 byte frames 0 Too old frames
1274072935 511 byte frames 0 Valid oversize frames
767249639 1023 byte frames 0 System FCS error frames
206271721 1518 byte frames 0 RxPortFifoFull drop frame
0 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames

Thanks

Kind Regards

Rohit

Leo Laohoo · ‎03-09-2015

12.2(35)SE5

Wow. Very, very old IOS.

Can you post the output to the command "sh interface Gi0/1"?

Rohit Mangotra · ‎03-09-2015

Hi Leo,

I don's see any drops here !!

sh interface Gi0/1

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 4/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:45, output 00:00:00, output hang never
Last clearing of "show interface" counters 41w0d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 19391000 bits/sec, 2333 packets/sec
5 minute output rate 6332000 bits/sec, 2330 packets/sec
38682528058 packets input, 46253579316131 bytes, 0 no buffer
Received 46622297 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 46621862 multicast, 0 pause input
0 input packets with dribble condition detected
32441671666 packets output, 11196112926344 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Thanks & Regards

Rohit.

Leo Laohoo · ‎03-09-2015

Total output drops came from the 3560 switchport. Total output drops is registered by the SENDING interface (the router) but originates from the receiving port (the switch).

Rohit Mangotra · ‎03-09-2015

Hi Leo,

Do you think that this might be only reason for overall network performance? But my 1st question where the router are not able to push more then 200 Mbps? Because of this Backups are taking very long time. what do you recon I should do?

Thanks & Regards

Rohit.

Leo Laohoo · ‎03-10-2015

But my 1st question where the router are not able to push more then 200 Mbps?

One-way-traffic, yes. Two-way-traffic, no.

Because of this Backups are taking very long time.

Do you have policy-shaping rules to ensure your router does not attempt to push WAN traffic above your subscribed bandwidth?

Rohit Mangotra · ‎03-10-2015

Hi Leo,

On both routers we have

interface GigabitEthernet0/0/0
description - DR-Router G0/0/0 -
bandwidth 500000

interface GigabitEthernet0/0/0
description - DC-Router G0/0/0 -
bandwidth 500000

Don't know what is actually going on the network???

Thanks & Regards

Rohit

Leo Laohoo · ‎03-09-2015

  Last clearing of "show interface" counters 20:47:13
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1190964

Hold on ... You've just cleared the counters and you get this much output drops?

Your bottleneck is your LAN.

Full Duplex, 1Gbps, media type is RJ45

Next, the link between the 3945E and your switch is copper. So why is the speed and duplex setting hard-coded?

What line errors can you see on the interface port of the switch? Do you see speed/duplex mismatch errors? Can you see if the switch port is seeing collision errors?

Joseph W. Doherty · ‎03-10-2015

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

So how do you have a SM fiber link with just 500 Mbps? I see the port is running at (the expected) gig.

Your posted g0/0 stats are interesting as it shows 16 packets queued yet a 5 minute transmission rate of only 8 Mbps (on gig)! I'm wondering whether some TCP flow or flows keeps being thrown back into slow start. Your overall drop rate, .7%, isn't horrible, although it's generally higher than I like.

PS:

Regarding a 3945e performance, Cisco recommends it for up to 350 Mbps (duplex). Cisco recommendation is for general all purpose usage, but with a "simple" config, you should be able to get more, even perhaps much more. What does your CPU usage history look like?

Experiencing large output packet drops on Cisco 3945e router