11-09-2015 03:22 PM - edited 03-05-2019 02:42 AM
Hi,
To give some backgound, we have a 20Mbps up/down fiber circuit from our ISP. The fiber lines comes into our office, hits a fiber>copper media converter, and then connects into our edge router (Cisco 2911). From there, the traffic goes through our Cisco ASA firewall, and then hits another Cisco 2911 and is routed internally from there.
I recently just upgraded to both of these 2911 routers. Prior to that the WAN edge router was a Cisco 1761 (max WAN thruput at 10Mbps) and a Cisco 2851 to handle the internal routing. I upgraded the routers so we could take advantage of the 20Mbps that we are paying for.
Since the upgrade, up/down thruput is at 20Mbps as expected. The issue is, when anyone attempts a download or upload on the network, the entire network crawls to a halt until that download/upload is finished. I can watch our traffic in the ASA and start a download, and even though I may only be DL'ing the file at 2-3 Mbps, the bandwidth instantly jumpts to 20Mbps used and all other outbound traffic on the network stops completely until the download ends. The bandwidth levels then drop back down normal levels. I have tested this multiple times, including on weekends when there is hardly any traffic on the network.
I can't have one user bring the entire network down because they need to download a file. What's going on here?
11-09-2015 03:27 PM
Below is the config for the edge router and also the interfaces. I took out any sensitive information and replaced first 3 IP octets with 1.1.1.x or 2.2.2.x.
ROUTER1#show running-config
Building configuration...
Current configuration : 3925 bytes
!
!
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
no service dhcp
!
hostname ROUTER1
!
boot-start-marker
boot-end-marker
!
!
logging buffered 51200 warnings
enable secret 5 $1$H9Mq$m7VSvzI0rqVpOIv43YYCg/
!
no aaa new-model
clock timezone EST -4 0
!
!
!
no ip domain lookup
ip name-server 208.67.222.222
ip name-server 8.8.8.8
ip cef
no ipv6 cef
multilink bundle-name authenticated
!
cts logging verbose
!
redundancy
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description WAN
ip address 1.1.1.166 255.255.255.252
duplex auto
speed auto
no keepalive
!
interface GigabitEthernet0/1
description IP Block
ip address 2.2.2.65 255.255.255.192
duplex auto
speed auto
no keepalive
!
interface GigabitEthernet0/2
no ip address
shutdown
duplex auto
speed auto
!
ip default-gateway 1.1.1.165
ip forward-protocol nd
!
no ip http server
ip http access-class 23
ip http authentication local
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip route 0.0.0.0 0.0.0.0 1.1.1.165
!
access-list 50 permit 2.2.2.64 0.0.0.63
!
control-plane
!
line con 0
login local
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
access-class 23 in
privilege level 15
login local
transport input telnet ssh
line vty 5 15
access-class 23 in
privilege level 15
login local
transport input ssh
!
scheduler allocate 20000 1000
!
end
ROUTER1#show int
Embedded-Service-Engine0/0 is administratively down, line protocol is down
Hardware is Embedded Service Engine, address is 0000.0000.0000 (bia 0000.0000.0000)
MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/64/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/0 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is 188b.9dc2.2620 (bia 188b.9dc2.2620)
Description: WAN
Internet address is 1.1.1.166/30
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 2/255, rxload 8/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full Duplex, 100Mbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:16, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 108213
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3513000 bits/sec, 494 packets/sec
5 minute output rate 893000 bits/sec, 377 packets/sec
333882510 packets input, 4021350135 bytes, 108208 no buffer
Received 39298 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
283751011 packets output, 915184340 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/1 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is 188b.9dc2.2621 (bia 188b.9dc2.2621)
Description: IPBlock
Internet address is 2.2.2.2.65/26
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:02:34, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 905000 bits/sec, 377 packets/sec
5 minute output rate 3511000 bits/sec, 495 packets/sec
283274492 packets input, 1675614373 bytes, 0 no buffer
Received 8400 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
333483661 packets output, 3604797528 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/2 is administratively down, line protocol is down
Hardware is CN Gigabit Ethernet, address is 188b.9dc2.2622 (bia 188b.9dc2.2622)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto Duplex, Auto Speed, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
ROUTER1#
11-10-2015 06:49 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
On an ISR, egress FQ would probably mostly eliminate the issue you're seeing for uploads. Unfortunately, the download issue is not so easy to deal with using ISR QoS.
The download issue migth be easy to mitigate with QoS on the ISP's egress, but almost all ISPs won't provide it (although they will often be happy to sell you more bandwidth).
There are some ISR QoS techniques that can somewhat regulate ingress bandwidth, but they don't generally work exactly as we desire.
There are also 3rd party traffic management applicances that can do much more than an ISR to regulate ingress bandwidth.
What you migth try:
policy-map shapedOut
class class-default
shape average 17000000 !I'm shaping 15% slower to allow for L2 overhead
policy-map shapedIn
class class-default
shape average 10000000 !I'm shaping much slower to congest and drop so to slow sender before downstream bandwidth is saturated - you can experiment with this bandwidth - lower bandwidths will likely better control ingress saturation but at the cost of not using all your downstream bandwidth when you want
service-policy FQ
policy-map FQ
class class-default
fair-queue
interface (toward ISP)
service-policy output shapedOut
interface (toward LAN)
service-policy output shapedIn
11-10-2015 07:08 AM
Joseph,
Thanks for the reply! I am going to add the traffic shaping this evening to see if that will help out. I've never added traffic shaping, but it looks fairly straight-forward.
I will reply back once I'm able to get that added and tested, hopefully tonight or tomorrow AM.
Thanks
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide