cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
842
Views
0
Helpful
3
Replies

Cisco 2911 - Entire network bandwidth being consumed on any upload/download on network

crescigno
Level 1
Level 1

Hi,

To give some backgound, we have a 20Mbps up/down fiber circuit from our ISP.  The fiber lines comes into our office, hits a fiber>copper media converter, and then connects into our edge router (Cisco 2911).  From there, the traffic goes through our Cisco ASA firewall, and then hits another Cisco 2911 and is routed internally from there.

I recently just upgraded to both of these 2911 routers.  Prior to that the WAN edge router was a Cisco 1761 (max WAN thruput at 10Mbps) and a Cisco 2851 to handle the internal routing.  I upgraded the routers so we could take advantage of the 20Mbps that we are paying for.

Since the upgrade, up/down thruput is at 20Mbps as expected.  The issue is, when anyone attempts a download or upload on the network, the entire network crawls to a halt until that download/upload is finished.  I can watch our traffic in the ASA and start a download, and even though I may only be DL'ing the file at 2-3 Mbps, the bandwidth instantly jumpts to 20Mbps used and all other outbound traffic on the network stops completely until the download ends.  The bandwidth levels then drop back down normal levels.  I have tested this multiple times, including on weekends when there is hardly any traffic on the network.

I can't have one user bring the entire network down because they need to download a file.  What's going on here?

3 Replies 3

crescigno
Level 1
Level 1

Below is the config for the edge router and also the interfaces.  I took out any sensitive information and replaced first 3 IP octets with 1.1.1.x or 2.2.2.x.

ROUTER1#show running-config
Building configuration...

Current configuration : 3925 bytes
!
!
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
no service dhcp
!
hostname ROUTER1
!
boot-start-marker
boot-end-marker
!
!
logging buffered 51200 warnings
enable secret 5 $1$H9Mq$m7VSvzI0rqVpOIv43YYCg/
!
no aaa new-model
clock timezone EST -4 0
!
!
!
no ip domain lookup
ip name-server 208.67.222.222
ip name-server 8.8.8.8
ip cef
no ipv6 cef
multilink bundle-name authenticated
!
cts logging verbose
!
redundancy
!
interface Embedded-Service-Engine0/0
 no ip address
 shutdown
!
interface GigabitEthernet0/0
 description WAN
ip address 1.1.1.166 255.255.255.252
 duplex auto
 speed auto
 no keepalive
!
interface GigabitEthernet0/1
 description IP Block
 ip address 2.2.2.65 255.255.255.192
 duplex auto
 speed auto
 no keepalive
!
interface GigabitEthernet0/2
 no ip address
 shutdown
 duplex auto
 speed auto
!
ip default-gateway 1.1.1.165
ip forward-protocol nd
!
no ip http server
ip http access-class 23
ip http authentication local
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip route 0.0.0.0 0.0.0.0 1.1.1.165
!
access-list 50 permit 2.2.2.64 0.0.0.63
!
control-plane
!
line con 0
 login local
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
line vty 0 4
 access-class 23 in
 privilege level 15
 login local
 transport input telnet ssh
line vty 5 15
 access-class 23 in
privilege level 15
 login local
 transport input ssh
!
scheduler allocate 20000 1000
!
end

ROUTER1#show int
Embedded-Service-Engine0/0 is administratively down, line protocol is down 
  Hardware is Embedded Service Engine, address is 0000.0000.0000 (bia 0000.0000.0000)
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/64/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
	 0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/0 is up, line protocol is up 
  Hardware is CN Gigabit Ethernet, address is 188b.9dc2.2620 (bia 188b.9dc2.2620)
  Description: WAN
  Internet address is 1.1.1.166/30
  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec, 
     reliability 255/255, txload 2/255, rxload 8/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full Duplex, 100Mbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:16, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 108213
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 3513000 bits/sec, 494 packets/sec
  5 minute output rate 893000 bits/sec, 377 packets/sec
	333882510 packets input, 4021350135 bytes, 108208 no buffer
     Received 39298 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     283751011 packets output, 915184340 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/1 is up, line protocol is up 
  Hardware is CN Gigabit Ethernet, address is 188b.9dc2.2621 (bia 188b.9dc2.2621)
  Description: IPBlock
  Internet address is 2.2.2.2.65/26
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:02:34, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 905000 bits/sec, 377 packets/sec
  5 minute output rate 3511000 bits/sec, 495 packets/sec
     283274492 packets input, 1675614373 bytes, 0 no buffer
     Received 8400 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     333483661 packets output, 3604797528 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/2 is administratively down, line protocol is down 
  Hardware is CN Gigabit Ethernet, address is 188b.9dc2.2622 (bia 188b.9dc2.2622)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Auto Duplex, Auto Speed, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
ROUTER1#
                                          

Joseph W. Doherty
Hall of Fame
Hall of Fame

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

On an ISR, egress FQ would probably mostly eliminate the issue you're seeing for uploads.  Unfortunately, the download issue is not so easy to deal with using ISR QoS.

The download issue migth be easy to mitigate with QoS on the ISP's egress, but almost all ISPs won't provide it (although they will often be happy to sell you more bandwidth).

There are some ISR QoS techniques that can somewhat regulate ingress bandwidth, but they don't generally work exactly as we desire.

There are also 3rd party traffic management applicances that can do much more than an ISR to regulate ingress bandwidth.

What you migth try:

policy-map shapedOut

class class-default

shape average 17000000 !I'm shaping 15% slower to allow for L2 overhead

policy-map shapedIn

class class-default

shape average 10000000 !I'm shaping much slower to congest and drop so to slow sender before downstream bandwidth is saturated - you can experiment with this bandwidth - lower bandwidths will likely better control ingress saturation but at the cost of not using all your downstream bandwidth when you want

service-policy FQ

policy-map FQ

class class-default

fair-queue

interface (toward ISP)

service-policy output shapedOut

interface (toward LAN)

service-policy output shapedIn

Joseph,

Thanks for the reply!  I am going to add the traffic shaping this evening to see if that will help out.  I've never added traffic shaping, but it looks fairly straight-forward.

I will reply back once I'm able to get that added and tested, hopefully tonight or tomorrow AM.


Thanks