Cisco 2811 behind bridged modem randomly dropping ISP link

[Areyouserious]

We have two Cisco 2811 routers on two sites, with a an IP SEC GRE tunnel linking each site.

On each end we have a broadband modem in bridge mode connected to each 2811.

Each modem connected to interface fa0/0, with DHCP  and NAT set in place.

The links work but occasionally the modem link stops passing traffic.

The internet goes down, but yet the interface still remains up. shutting the interface down and then opening it again resolves the problem.

Just wondering if anyone else has experienced this problem before?

Thanks

[Georg Pauwen]

Hello,

post the full configs of both 2811 routers...

[Areyouserious]

Site A

service password-encryption
!
hostname SITE-A
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
!
!
dot11 syslog
ip source-route
!
!
ip cef
ip dhcp excluded-address 10.1.1.110 10.1.1.250
!
ip dhcp pool Telephones
network 10.1.2.0 255.255.255.0
default-router 10.1.2.1
option 150 ip 192.168.1.6
!
ip dhcp pool OutsideSwitch
network 10.1.9.0 255.255.255.0
default-router 10.1.9.1
dns-server 10.1.9.1
option 150 ip 192.168.1.6
!
ip dhcp pool WiredLaptops
network 10.1.5.0 255.255.255.0
default-router 10.1.5.1
dns-server 203.0.178.191
!
ip dhcp pool WiFi
network 10.1.7.0 255.255.255.0
default-router 10.1.7.1
dns-server 203.0.178.191
!

!
!
!
ip name-server 203.0.178.191

ip inspect tcp reassembly alarm on
ip inspect name NBN udp audit-trail on
ip inspect name NBN tcp audit-trail on
ip inspect name NBN rtsp audit-trail on
ip inspect name NBN http audit-trail on
ip inspect name NBN https audit-trail on
ip inspect name NBN isakmp audit-trail on
ip inspect name NBN ntp audit-trail on
ip inspect name NBN sip audit-trail on
ip inspect name NBN ssh audit-trail on
login block-for 300 attempts 3 within 60
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
voice-card 0
!
crypto pki token default removal timeout 0
!
!
!
!
username XXXX privilege 15 secret XXXXXXXXXXXXXXXXXXXXXXXXXX
!
redundancy
!
!
!
!
crypto isakmp policy 1
encr aes 256
authentication pre-share
group 2
crypto isakmp key XXXXXXXXX address XXX.XXX.XXX.XXX
!
!
crypto ipsec transform-set XXXXXXXXXXXX esp-aes esp-sha-hmac
!
crypto ipsec profile IPSEC_PROFILE
set transform-set XXXXXXXXXXXXX
!
!
!
!
!
!
!
interface Tunnel0
ip address 172.16.1.2 255.255.255.0
ip mtu 900
ip nat inside
ip virtual-reassembly in
tunnel source XXX.XXX.XXX.XXX
tunnel mode ipsec ipv4
tunnel destination XXX.X.XX.XXX
tunnel protection ipsec profile XXXXXXXXX
!
interface FastEthernet0/0
no ip address
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
!
interface FastEthernet0/0.1
encapsulation dot1Q 1 native
ip address 10.1.1.1 255.255.255.0
!
interface FastEthernet0/0.2
encapsulation dot1Q 2
ip address 10.1.2.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
!
interface FastEthernet0/0.5
encapsulation dot1Q 5
ip address 10.1.5.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface FastEthernet0/0.6
encapsulation dot1Q 6
ip address 10.1.6.1 255.255.255.0
!
interface FastEthernet0/0.7
encapsulation dot1Q 7
ip address 10.1.7.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface FastEthernet0/0.8
encapsulation dot1Q 8
ip address dhcp
ip nat outside
ip inspect NBN out
ip virtual-reassembly in
!
interface FastEthernet0/0.9
ip nat inside
ip virtual-reassembly in
!
interface FastEthernet0/1
ip address 10.1.9.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto
!
ip forward-protocol nd
no ip http server
no ip http secure-server
!
!
ip nat inside source list 1 interface FastEthernet0/0.8 overload
ip route 192.168.1.0 255.255.255.0 Tunnel0
ip route 0.0.0.0 0.0.0.0 FastEthernet0/0.8 dhcp
!

!
access-list 1 permit 10.1.5.0 0.0.0.255
access-list 1 permit 10.1.7.0 0.0.0.255
access-list 1 permit 10.1.9.0 0.0.0.255
access-list 1 permit 10.1.6.0 0.0.0.255
!
!
!
!
!
!
control-plane
!
!
!
!
mgcp profile default
!
!
!
!
!
line con 0
logging synchronous
login local
line aux 0
line vty 0 4
logging synchronous
login local
transport input ssh
!
scheduler allocate 20000 1000
end

SITE B 

service password-encryption
!
hostname SITE-B
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
!
no network-clock-participate wic 0
!
dot11 syslog
ip source-route
!
!
ip cef
no ip dhcp use vrf connected
ip dhcp excluded-address 192.168.1.1 192.168.1.15
!
ip dhcp pool XXXXXXXXXX
network 192.168.1.0 255.255.255.0
option 150 ip 192.168.1.6
dns-server 203.215.29.191
default-router 192.168.1.8
!
!
!
ip name-server 203.0.178.191
ip inspect tcp reassembly alarm on
ip inspect name NBN udp audit-trail on
ip inspect name NBN tcp audit-trail on
ip inspect name NBN rtsp audit-trail on
ip inspect name NBN http audit-trail on
ip inspect name NBN https audit-trail on
ip inspect name NBN isakmp audit-trail on
ip inspect name NBN ntp audit-trail on
ip inspect name NBN sip audit-trail on
ip inspect name NBN ssh audit-trail on
login block-for 300 attempts 3 within 300
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
voice-card 0
!
crypto pki token default removal timeout 0
!
!
!
!

username XXXXXXXXX privilege 15 secret XXXXXXXXXXXXXXXXX
!
redundancy
!
!
!
!
crypto isakmp policy 1
encr aes 256
authentication pre-share
group 2
crypto isakmp key XXXXXXXXXXX address XXX.XXX.XXX.XXX
!
!
crypto ipsec transform-set XXXXXXXXXXXXXXX esp-aes esp-sha-hmac
!
crypto ipsec profile IPSEC_PROFILE
set transform-set XXXXXXXXXXXX
!
!
!
!
!
!
!
interface Tunnel0
ip address 172.16.1.1 255.255.255.0
ip mtu 900
ip nat inside
ip virtual-reassembly in
tunnel source XXX.XXX.XXX.XXX
tunnel mode ipsec ipv4
tunnel destination XXX.XXX.XXX.XXX
tunnel protection ipsec profile IPSEC_PROFILE
!
interface FastEthernet0/0
ip address dhcp
ip nat outside
ip inspect NBN out
no ip virtual-reassembly in
no ip route-cache
duplex auto
speed auto
no cdp enable
!
interface FastEthernet0/1
ip address 192.168.1.8 255.255.255.0
ip nat inside
no ip virtual-reassembly in
duplex auto
speed auto
!
interface Serial0/0/0
no ip address
shutdown
clock rate 2016000
!
interface Serial0/0/1
no ip address
shutdown
clock rate 2016000
!
interface Serial0/0/2
no ip address
shutdown
clock rate 2016000
!
interface Serial0/0/3
no ip address
shutdown
clock rate 2016000
!
ip default-gateway 203.59.224.255
no ip forward-protocol nd
no ip http server
no ip http secure-server
!
!
ip nat inside source list 1 interface FastEthernet0/0 overload
ip route 10.1.1.0 255.255.255.0 Tunnel0
ip route 10.1.2.0 255.255.255.0 Tunnel0
ip route 10.1.6.0 255.255.255.0 Tunnel0
ip route 10.1.9.0 255.255.255.0 Tunnel0
ip route 0.0.0.0 0.0.0.0 FastEthernet0/0 dhcp
!
!
access-list 1 permit 192.168.1.0 0.0.0.255
!
!
!
!
!
!
control-plane
!
!
!
!
mgcp profile default
!
!
!
!
!
banner login XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
line con 0
line aux 0
line vty 0 4
logging synchronous
login local
transport input ssh
!
scheduler allocate 20000 1000
end

[Georg Pauwen]

Hello,

try and implement the changes marked in bold (the dhcp exclusions only in Site A router):

service password-encryption
!
hostname SITE-A
!
boot-start-marker
boot-end-marker
!
no aaa new-model
!
dot11 syslog
ip source-route
!
ip cef
!
ip dhcp excluded-address 10.1.1.110 10.1.1.250
ip dhcp excluded-address 10.1.2.1
ip dhcp excluded-address 10.1.9.1
ip dhcp excluded-address 10.1.5.1
ip dhcp excluded-address 10.1.7.1
!
ip dhcp pool Telephones
network 10.1.2.0 255.255.255.0
default-router 10.1.2.1
option 150 ip 192.168.1.6
!
ip dhcp pool OutsideSwitch
network 10.1.9.0 255.255.255.0
default-router 10.1.9.1
dns-server 10.1.9.1
option 150 ip 192.168.1.6
!
ip dhcp pool WiredLaptops
network 10.1.5.0 255.255.255.0
default-router 10.1.5.1
dns-server 203.0.178.191
!
ip dhcp pool WiFi
network 10.1.7.0 255.255.255.0
default-router 10.1.7.1
dns-server 203.0.178.191
!
ip name-server 203.0.178.191

ip inspect tcp reassembly alarm on
ip inspect name NBN udp audit-trail on
ip inspect name NBN tcp audit-trail on
ip inspect name NBN rtsp audit-trail on
ip inspect name NBN http audit-trail on
ip inspect name NBN https audit-trail on
ip inspect name NBN isakmp audit-trail on
ip inspect name NBN ntp audit-trail on
ip inspect name NBN sip audit-trail on
ip inspect name NBN ssh audit-trail on
login block-for 300 attempts 3 within 60
no ipv6 cef
!
multilink bundle-name authenticated
!
voice-card 0
!
crypto pki token default removal timeout 0
!
username XXXX privilege 15 secret XXXXXXXXXXXXXXXXXXXXXXXXXX
!
redundancy
!
crypto isakmp policy 1
encr aes 256
authentication pre-share
group 2
crypto isakmp key XXXXXXXXX address XXX.XXX.XXX.XXX
!
crypto ipsec transform-set XXXXXXXXXXXX esp-aes esp-sha-hmac
!
crypto ipsec profile IPSEC_PROFILE
set transform-set XXXXXXXXXXXXX
!
interface Tunnel0
ip address 172.16.1.2 255.255.255.0
--> no ip mtu 900
--> no ip nat inside
ip virtual-reassembly in
tunnel source XXX.XXX.XXX.XXX
tunnel mode ipsec ipv4
tunnel destination XXX.X.XX.XXX
tunnel protection ipsec profile XXXXXXXXX
!
interface FastEthernet0/0
no ip address
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
!
interface FastEthernet0/0.1
encapsulation dot1Q 1 native
ip address 10.1.1.1 255.255.255.0
!
interface FastEthernet0/0.2
encapsulation dot1Q 2
ip address 10.1.2.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface FastEthernet0/0.5
encapsulation dot1Q 5
ip address 10.1.5.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface FastEthernet0/0.6
encapsulation dot1Q 6
ip address 10.1.6.1 255.255.255.0
!
interface FastEthernet0/0.7
encapsulation dot1Q 7
ip address 10.1.7.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface FastEthernet0/0.8
encapsulation dot1Q 8
ip address dhcp
ip nat outside
ip inspect NBN out
ip virtual-reassembly in
!
interface FastEthernet0/0.9
ip nat inside
ip virtual-reassembly in
!
interface FastEthernet0/1
ip address 10.1.9.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto
!
ip forward-protocol nd
no ip http server
no ip http secure-server
!
ip nat inside source list 1 interface FastEthernet0/0.8 overload
ip route 192.168.1.0 255.255.255.0 Tunnel0
ip route 0.0.0.0 0.0.0.0 FastEthernet0/0.8 dhcp
!
access-list 1 permit 10.1.5.0 0.0.0.255
access-list 1 permit 10.1.7.0 0.0.0.255
access-list 1 permit 10.1.9.0 0.0.0.255
access-list 1 permit 10.1.6.0 0.0.0.255
!
control-plane
!
mgcp profile default
!
line con 0
logging synchronous
login local
line aux 0
line vty 0 4
logging synchronous
login local
transport input ssh
!
scheduler allocate 20000 1000
end

[Areyouserious]

Thankyou very much,

I'll give you an update soon.

[Richard Burts]

I had an issue with a customer site which sounds like it might have been similar. Site to site vpn where periodically the link to the ISP would stop passing traffic but was still up/up. After a bunch of troubleshooting we determined that it was an issue with arp timeout. Changing the arp timeout on the router to something less than the default of 4 hours was successful in resolving this issue.

HTH

Rick

[paul driver]

Hello Rick

Very interesting- can you elaborate on the reason being - lack of traffic? 

[Richard Burts]

Paul

It has been a while and some details are no longer clear in my mind (can not remember which ISP, or what kind of equipment they used, etc) but in general the issue was not lack of traffic but was arp timeout. If the ISP device on the connection would time out its arp entry it then would not forward traffic until it had relearned the mac address. This would take down the vpn connection. Part of the confusion was that it was very sporadic. It would work for some period of time and then would fail. Then work for some longer or shorter period of time and then fail.  

For a long time I took it as an article of faith that Cisco IOS arp time out is 4 hours exactly. But in the TAC case on this I learned that Cisco introduces some variation in the arp timeout (sometimes it is a bit less than 4 and sometimes a bit more than 4). The purpose of the variation is to prevent synchronization of entries in the arp table. What was happening was that some times our Cisco would time out first, and since Cisco sends an arp request before it actually times out the entry we would refresh the arp entry on both sides and the connection was good. But sometimes the ISP timed out first. In that case they just stopped forwarding traffic till a new arp entry was created. We changed the arp timeout on our Cisco to a shorter value and the vpn became very stable.

HTH

Rick

[paul driver]

+5 stars rick

been very useful information 

[Richard Burts]

Paul

Thanks. It was a very interesting case. Glad I could share information about it.

HTH

Rick

[paul driver]

Hello

---

@Areyouserious wrote: 

The links work but occasionally the modem link stops passing traffic.

The internet goes down, but yet the interface still remains up. shutting the interface down and then opening it again resolves the problem.

---

Has this always been like this or  has it recently started to happen, and if the latter have you changed anything because if you haven't and it was working perfectly in the past then it would suggest a hardware issue of some kind otherwise can you explain what has been changed recently?

[Areyouserious]

Hard to say really.

It seems to be a little better since I applied that new config that you guys reccommended.

It also may have been due to a faulty WAP which has since been replaced.

Another issue is that I have recently applied a deny ip any any statement to my outside wan port with the DHCP link to my bridged modem.

I have also enabled CBAC ip inspect going out on that WAN port.

However, this is dropping the link after 5 mins.

Could DHCP traffic be blocked perhaps?

[Richard Burts]

When you include deny ip any any on an ACL, or when you enable CBAC, it is certainly possible that this would result in denying DHCP. But if DHCP were being denied I think that the outside interface would not work at all. And if I am reading the post correctly you are saying that it works ok for some time and then fails. So I doubt this is due to issues with DHCP.

Glad to know that it is better after you made the changes, or maybe it was replacing the faulty WAP. When you say it is better seems to indicate that it still is having problems. Can you provide some details on the problems you are now experiencing?

HTH

Rick