Here are interface statistics

kevinwong11 · ‎08-12-2014

I have a cisco 1921 router performing routing and NAT function. Cisco 3750 is performing layer 3 core switching duties. Users are complaining about large file transfers being stalled (not dropped) down to 0Mbps and then the connection bounces back to normal. The internet connection is 100/20 and my graphs peak around 20/5Mbps. CPU and Memory Usage are under 5% and logs show no signs of failure. Everything was working smoothly at first and then now I have trouble maintaining a stable remote connection. Can somebody provide some input...DHCP Lease Time? ARP Timeout? ACL? MTU? NAT? Packet Fragmentation? Crap device?

Router-1#show running-conf
Building configuration...

Current configuration : 5169 bytes
!
! Last configuration change at 04:19:46 UTC Wed Aug 13 2014 by admin
! NVRAM config last updated at 06:45:31 UTC Tue Aug 5 2014 by admin
! NVRAM config last updated at 06:45:31 UTC Tue Aug 5 2014 by admin
version 15.2
service timestamps debug datetime
service timestamps log datetime
no service password-encryption
!
hostname Router-1
!
boot-start-marker
boot-end-marker
!
!
logging buffered 64000
logging console emergencies

!
no aaa new-model
!
ip cef
!
!
!
!
!
!
ip domain name yourdomain.com
ip name-server 8.8.8.8
ip name-server 8.8.4.4
no ipv6 cef
multilink bundle-name authenticated
!
!
crypto pki trustpoint TP-self-signed-4171117453
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-4171117453
revocation-check none
rsakeypair TP-self-signed-4171117453
!
!
crypto pki certificate chain TP-self-signed-4171117453
certificate self-signed 01
3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 34313731 31313734 3533301E 170D3134 30323237 32303534
30365A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D34 31373131
31373435 3330819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
8100A02B 57173408 B3535863 614AF02D 3FF39DF3 B5E8FCE5 258C64D7 39624691
6877BCB7 987D8A1C 9587952B 35FE21E3 09511E5F 6CD67D38 2001AB79 902A3BC1
D3A5939F 36259B74 6477457C 03191EBC 6F5AA818 3A3497D5 4CE79785 C7719EE2
77265F87 6846AB0D 5C118CC7 E198A96D FB662C71 996F129F BD95B8C3 CFB468E2
0B870203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF 301F0603
551D2304 18301680 141CFDA8 54FB8657 3913F1A2 4D19BBCB 9E1802E4 68301D06
03551D0E 04160414 1CFDA854 FB865739 13F1A24D 19BBCB9E 1802E468 300D0609
2A864886 F70D0101 05050003 8181007A 47596796 4F7D5AAA A9675522 F75D6865
983C6F66 1679E529 172844F1 C461DD08 1FEECEC1 423F5C62 6A7A4256 0263D93A
2D4B71EE 90FC86E6 289C5EBC 72261F19 291C40A3 F194C8C3 BC0A6D8B 1AE50986
89A13B32 85BF51C4 BF16A225 13012099 EF3B8C63 65976CEC D1387E6D 79FEDAD3
8225467E AC6C290F 3AC6D57E B45854
quit
license udi pid CISCO1921/K9 sn FGL18092403
!
!
!
!
!
!
!
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description LAN
ip address 192.168.1.2 255.255.255.0
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto
!
interface GigabitEthernet0/1
ip address *WAN*
ip access-group 101 in
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
!
ip forward-protocol nd
!
ip http server
ip http access-class 23
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip nat inside source list 7 interface GigabitEthernet0/1 overload
ip route 0.0.0.0 0.0.0.0 *WAN*
ip route 192.168.2.0 255.255.254.0 192.168.1.1
ip route 192.168.4.0 255.255.255.0 192.168.1.1
ip route 192.168.5.0 255.255.255.0 192.168.1.1
!
access-list 7 permit 192.168.1.0 0.0.0.255
access-list 7 permit 192.168.2.0 0.0.0.255
access-list 7 permit 192.168.3.0 0.0.0.255
access-list 7 permit 192.168.4.0 0.0.0.255
access-list 7 permit 192.168.5.0 0.0.0.255
access-list 23 permit 10.10.10.0 0.0.0.7
access-list 101 remark Standardized inbound anti-spoofing list
access-list 101 deny ip 192.168.0.0 0.0.255.255 any
access-list 101 deny ip 172.16.0.0 0.15.255.255 any
access-list 101 deny ip 10.0.0.0 0.255.255.255 any
access-list 101 deny ip 127.0.0.0 0.255.255.255 any
access-list 101 deny ip 255.0.0.0 0.255.255.255 any
access-list 101 deny ip 224.0.0.0 7.255.255.255 any
access-list 101 deny ip 14.0.0.0 0.255.255.255 any log
access-list 101 deny ip 169.254.0.0 0.0.255.255 any log
access-list 101 deny ip 198.18.0.0 0.0.255.255 any log
access-list 101 deny ip host 0.0.0.0 any
access-list 101 deny ip 50.202.143.128 0.0.0.31 any
access-list 101 deny udp any any eq snmp log
access-list 101 deny udp any any eq snmptrap log
access-list 101 deny tcp any any range 135 139 log
access-list 101 deny udp any any range 135 netbios-ss log
access-list 101 deny tcp any any eq 6666 log
access-list 101 deny tcp any any eq 6667 log
access-list 101 deny tcp any any eq 445 log
access-list 101 deny udp any any eq 445 log
access-list 101 permit ip any any
access-list 101 deny ip any any log
!
!
snmp-server community
snmp-server enable traps entity-sensor threshold
!
control-plane
!
!
!
line con 0
login local
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
access-class 7 in
privilege level 15
password
login local
transport input telnet ssh
line vty 5 15
access-class 23 in
privilege level 15
login local
transport input telnet ssh
!
scheduler allocate 20000 1000
ntp server 128.138.141.172
ntp server 216.228.192.69
!
end

kevinwong11 · ‎08-13-2014

Here are interface statistics from my Router and Switch. Noticed a lot of dropped packets in the output queue.

1921 Router:

GigabitEthernet0/1 is up, line protocol is up

Hardware is CN Gigabit Ethernet, address is 18e7.287c.c141 (bia 18e7.287c.c141)

Internet address is *****************

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full Duplex, 1Gbps, media type is RJ45

output flow-control is unsupported, input flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 04:32:42

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 172000 bits/sec, 75 packets/sec

5 minute output rate 387000 bits/sec, 87 packets/sec

1469144 packets input, 775266818 bytes, 0 no buffer

Received 498326 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

1173631 packets output, 600838806 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

Router-1#sho interfaces g0/0

GigabitEthernet0/0 is up, line protocol is up

Hardware is CN Gigabit Ethernet, address is 18e7.287c.c140 (bia 18e7.287c.c140)

Description: LAN

Internet address is 192.168.1.2/24

MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full Duplex, 100Mbps, media type is RJ45

output flow-control is unsupported, input flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 04:33:30

Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 1266

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 431000 bits/sec, 98 packets/sec

5 minute output rate 149000 bits/sec, 50 packets/sec

1183706 packets input, 579526099 bytes, 1265 no buffer

Received 283 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 274 multicast, 0 pause input

978123 packets output, 743563818 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

Catalyst 3750

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

-------------------------------------------------------------------------

Vlan1 0 0 0 0 0 0 0 0 0

* Vlan10 0 0 0 0 0 0 0 0 0

* Vlan20 0 0 0 0 0 0 0 0 0

* Vlan30 1 0 0 0 2000 3 3000 3 0

* Vlan40 0 79 0 0 0 0 0 0 0

* Vlan50 0 0 0 0 0 0 0 0 0

* FastEthernet1/0/1 0 0 0 155 38000 33 340000 79 0

* FastEthernet1/0/2 0 0 0 1513 0 0 5000 5 0

* FastEthernet1/0/3 0 0 0 1 1000 1 2000 3 0

* FastEthernet1/0/4 0 0 0 0 0 0 0 0 0

* FastEthernet1/0/5 0 0 0 0 314000 62 18000 27 0

* FastEthernet1/0/6 0 0 0 0 1000 0 2000 4 0

FastEthernet1/0/7 0 0 0 0 0 0 0 0 0

FastEthernet1/0/8 0 0 0 0 0 0 0 0 0

* FastEthernet1/0/9 0 0 0 3501 0 0 0 0 0

FastEthernet1/0/10 0 0 0 0 0 0 0 0 0

* FastEthernet1/0/11 0 0 0 2718 0 0 0 0 0

FastEthernet1/0/12 0 0 0 3755 0 0 0 0 0

FastEthernet1/0/13 0 0 0 231 0 0 0 0 0

* FastEthernet1/0/14 0 0 0 1461 0 1 2000 1 0

* FastEthernet1/0/15 0 0 0 0 0 0 0 0 0

* FastEthernet1/0/16 0 0 0 0 0 0 0 0 0

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

-------------------------------------------------------------------------

* FastEthernet1/0/17 0 0 0 0 11000 4 8000 6 0

* FastEthernet1/0/18 0 0 0 0 0 0 0 0 0

FastEthernet1/0/19 0 0 0 0 0 0 0 0 0

FastEthernet1/0/20 0 0 0 714 0 0 0 0 0

FastEthernet1/0/21 0 0 0 0 0 0 0 0 0

* FastEthernet1/0/22 0 0 0 3504 3000 2 1000 2 0

FastEthernet1/0/23 0 0 0 1129 0 0 0 0 0

FastEthernet1/0/24 0 0 0 0 0 0 0 0 0

GigabitEthernet1/0/1 0 0 0 0 0 0 0 0 0

GigabitEthernet1/0/2 0 0 0 0 0 0 0 0 0

Joseph W. Doherty · ‎08-13-2014

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Your number of output drops, as a percentage of output packets, is only around .1%, which isn't considered high. They seem to be buffer drops, so manual or auto buffer tuning might help.

As the link is 100/20, what does your ISP do when you try to exceed 20 Mbps? There might be benefit in shaping your egress to the 20 Mbps.

Is your default route statement using an IP or interface (unable to tell as you just have *WAN*)?

BTW, 100/20 is somewhat more than what Cisco recommends for a 1921 (15 Mbps [full duplex]).

davethehedgehog · ‎08-13-2014

I presume that's 100Mbps download 20 Mbps upload?

If that's the case the 1921 doesnt really have enough beef for that. You might want to consider 2951 or bigger really. Actual performance of a 1921 is around 15Mbps as Joseph says. There's nothing wrong with the 1921, it's not a crap device. It's just making sure you get the right router for the right job really. There's a useful Cisco document bouncing around the web on this. I won't link as I'm not sure if it's legit or not.

kevinwong11 · ‎08-13-2014

Yes the internet connection is routed through an IP interface on port G0/1 (removed the ip for security purposes). The internet connection is 100/20. I find it hard to believe that a $500 router cannot push more than 15M upload. I might believe that about a linksys home router but not enterprise quality. I work with 10 year old switches that can push 100M.

davethehedgehog · ‎08-13-2014

That's because routers route and switches switch, two very different functions all together. CCNA101. You're comparing massively different things. You could try plugging your 10 year old switch into your WAN connection, but you couldn't do much with it.

Here is a useful link...

https://supportforums.cisco.com/discussion/11684596/performance-specs-throughput-maximums-isr-g2

kevinwong11 · ‎08-13-2014

Ya I understand CCNA101. 15Mbps is rated for the router under full load. My CPU is under 7% and Bandwidth isn't even topping 10Mbps. You can see from my config that the only things I am running on the 1921 is NAT and ACL's.

davethehedgehog · ‎08-13-2014

Cool, so you understand why comparing a 10 year old 10/100 switch to a router platform capable of providing routing, VPN and voice is a bit of a strange comparison!

Remember, CPU stats from the router are based on averages. They don't necassarily show the peaks that can occur from wait. It's also massively dependant on packet size. I've tested most routers under varying conditions and I've found that PDF joseph posted to be very accurate. I've also had this conversation with many ISP's over the years who were adamant the routers can do more, until I showed the Y.1564 results of the routers the installed. Pay particular attention to the last chart. That's Cisco's recommendation on WAN bandwidth and model based on real world testing.

At the end of the day, there's not enough info there to precisely identify your problem, but what i personally can say from my experience is that the 1921 router is in no way capable of routing 100Mbps of real world traffic.

kevinwong11 · ‎08-13-2014

It was just a statement not necessarily a comparison. I understand the Cisco documented specifications but those are clearly understated and represent a router under full load with all services turned on. My connection is peaking up to the full 100Mbps on download and 20Mbps on upload. Then once it hits the peak it starts throttling back down to next to nothing. My average bandwidth graph shows that traffic isn't even anywhere close to 15Mbps. I just can't believe a $4,000 setup would not match up with a $300 Asus Router from Fry's. That cheap box can at least theoretically route over 15mbps right?

If anything, I would think my Cisco 3750 10/100 core switch is the bottleneck.

Could there be another issue other than the advertised performance of a device. Things such as a bottleneck between Gig ports and 10/100 ports, Auto-On negotiation dropping packets, ACL's blocking legititmate packets, NAT overload configured?

how to fix stalled connection on large file transfers