Solved: Re: Odd But Consistent Packet Dropping

frank.trampe · ‎10-02-2019

I have a Cisco 1921 router running 15.7(3)M2 that is consistently dropping certain packets but not others for reasons that are not clear. As an example, a DNS response for stltoday.com (from any server) gets completely dropped (no return packet on Wireshark), but the response for amd.com always gets through. Both response packets are under 200 bytes and thus vastly smaller than the MTU. The problem also affects rsync, but it seems not to affect anything else. This is not a hardware fluke. This router passes hardware tests, and the problem started on the previous router (an 891W). This is also not an upstream connectivity issue. This router and other devices connected directly to the modem are able to complete the DNS requests (and so the work-around right now is for the router to work as a pass-through DNS server). I can't imagine this being anything other than a bug, but I'm eager to get it solved either way.

!
! Last configuration change at 18:34:02 UTC Wed Oct 2 2019 by cisco
!
version 15.7
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname ht312-router
!
boot-start-marker
boot system usbflash0 c1900-universalk9-mz.SPA.157-3.M2.bin
boot-end-marker
!
!
enable secret 5 $4$MExE$REDACTED/REDACTED/XXX/
!
no aaa new-model
!
!
!
!         
!
!
!
!
!
ip dhcp excluded-address 10.10.10.1
ip dhcp excluded-address 192.168.0.1 192.168.0.63
ip dhcp excluded-address 192.168.2.1 192.168.2.16
ip dhcp excluded-address 192.168.4.1 192.168.4.15
!
ip dhcp pool cvo-pool
 import all
 network 10.10.10.0 255.255.255.248
 default-router 10.10.10.1 
 lease 0 2
!
ip dhcp pool dhcp0
 network 192.168.0.0 255.255.255.0
 default-router 192.168.0.1 
 dns-server 192.168.0.1 208.67.222.123 208.67.220.123 8.8.8.8 
 accounting default
!
ip dhcp pool dhcp2
 network 192.168.2.0 255.255.255.0
 default-router 192.168.2.1 
 dns-server 192.168.2.1 8.8.8.8 8.8.4.4 
 option 43 hex f104.c0a8.020e
 accounting default
!
ip dhcp pool dhcp12
 network 192.168.4.0 255.255.255.0
 dns-server 192.168.0.1 208.67.222.123 208.67.220.123 
 accounting default
!
!
!
ip domain name local
ip name-server 8.8.8.8
ip name-server 8.8.4.4
no ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
cts logging verbose
!
!
license udi pid CISCO1921/K9 sn XXXXXXXXXXX
!
!
username cisco privilege 15 secret 5 $4$JE78$REDACTED0REDACTED0XXX.
!
redundancy
!
!         
!
!
!
! 
!
!
!
!
!
bridge irb
!
!
!
!
interface Embedded-Service-Engine0/0
 no ip address
 shutdown
! Trunk to main switch.
interface GigabitEthernet0/0
 no ip address
 duplex auto
 speed auto
!         
interface GigabitEthernet0/0.1
 encapsulation dot1Q 1 native
 bridge-group 1
!
interface GigabitEthernet0/0.2
 encapsulation dot1Q 10
 bridge-group 2
!
interface GigabitEthernet0/0.12
 encapsulation dot1Q 12
 bridge-group 12
! Uplink to modem.
interface GigabitEthernet0/1
 ip address dhcp
 ip access-group 101 in
 no ip redirects
 no ip proxy-arp
 ip flow ingress
 ip nat enable
 ip virtual-reassembly in
 duplex auto
 speed auto
! Main internal network.
interface BVI1
 ip address 192.168.0.1 255.255.255.0
 ip access-group 100 in
 no ip redirects
 ip nat enable
! Wireless infrastructure network.
interface BVI2
 ip address 192.168.2.1 255.255.255.0
 ip access-group 100 in
 no ip redirects
 ip nat enable
! Guest network.
interface BVI12
 ip address 192.168.4.1 255.255.255.0
 ip access-group 100 in
 no ip redirects
 ip nat enable
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!         
ip dns server
ip nat source list 1 interface GigabitEthernet0/1 overload
ip nat source list 65 interface GigabitEthernet0/1 overload
ip nat source list 101 pool globalnet overload
ip nat source static tcp 192.168.0.35 22 interface GigabitEthernet0/1 2930
ip nat source static tcp 192.168.0.35 3030 interface GigabitEthernet0/1 3030
!
logging history size 128
!
!
access-list 1 permit 10.10.10.0 0.0.0.7
access-list 23 permit 10.10.10.0 0.0.0.7
access-list 65 permit 192.0.0.0 0.255.255.255
access-list 72 permit 192.0.0.0 0.255.255.255
access-list 100 deny   ip host 255.255.255.255 any
access-list 100 permit ip any any
!
control-plane
!
bridge 1 protocol ieee
bridge 1 route ip
bridge 2 protocol ieee
bridge 2 route ip
bridge 12 protocol ieee
bridge 12 route ip
!
no vstack
!
line con 0
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
line vty 0 4
 login local
 transport input ssh
line vty 5 15
 access-class 23 in
 privilege level 15
 login local
 transport input telnet ssh
!
scheduler allocate 20000 1000
event manager environment _internet_route_established_phrase LINK-3-UPDOWN
event manager directory user policy "usbflash0:/"
! Work-around for the NVI rule-dropping bug.
event manager policy fix_nvi.tcl
!
end

Georg Pauwen · ‎10-03-2019

Hello,

looking at your configuration, the first thing that looks odd is that you turned off CEF, which means everything gets process switched (which usually slows down everything). Try to enable process switching (ip cef).

I have made some changes (marked in bold) to your configuration, as a lot of things are reddundant:

! Last configuration change at 18:34:02 UTC Wed Oct 2 2019 by cisco
!
version 15.7
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname ht312-router
!
boot-start-marker
boot system usbflash0 c1900-universalk9-mz.SPA.157-3.M2.bin
boot-end-marker
!
enable secret 5 $4$MExE$REDACTED/REDACTED/XXX/
!
no aaa new-model
!
ip dhcp excluded-address 10.10.10.1
ip dhcp excluded-address 192.168.0.1 192.168.0.63
ip dhcp excluded-address 192.168.2.1 192.168.2.16
ip dhcp excluded-address 192.168.4.1 192.168.4.15
!
ip dhcp pool cvo-pool
import all
network 10.10.10.0 255.255.255.248
default-router 10.10.10.1
lease 0 2
!
ip dhcp pool dhcp0
network 192.168.0.0 255.255.255.0
default-router 192.168.0.1
dns-server 8.8.8.8 208.67.222.123 208.67.220.123 192.168.0.1
accounting default
!
ip dhcp pool dhcp2
network 192.168.2.0 255.255.255.0
default-router 192.168.2.1
dns-server 8.8.8.8 8.8.4.4 192.168.2.1
option 43 hex f104.c0a8.020e
accounting default
!
ip dhcp pool dhcp12
network 192.168.4.0 255.255.255.0
dns-server 8.8.8.8 208.67.222.123 208.67.220.123 192.168.0.1
accounting default
!
ip domain name local
ip name-server 8.8.8.8
ip name-server 8.8.4.4
--> ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
cts logging verbose
!
license udi pid CISCO1921/K9 sn XXXXXXXXXXX
!
username cisco privilege 15 secret 5 $4$JE78$REDACTED0REDACTED0XXX.
!
redundancy
!
bridge irb
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
! Trunk to main switch.
interface GigabitEthernet0/0
no ip address
duplex auto
speed auto
!
interface GigabitEthernet0/0.1
encapsulation dot1Q 1 native
bridge-group 1
!
interface GigabitEthernet0/0.2
encapsulation dot1Q 10
bridge-group 2
!
interface GigabitEthernet0/0.12
encapsulation dot1Q 12
bridge-group 12
! Uplink to modem.
interface GigabitEthernet0/1
ip address dhcp
--> no ip access-group 101 in
no ip redirects
no ip proxy-arp
ip flow ingress
ip nat enable
ip virtual-reassembly in
duplex auto
speed auto
! Main internal network.
interface BVI1
ip address 192.168.0.1 255.255.255.0
--> no ip access-group 100 in
no ip redirects
ip nat enable
! Wireless infrastructure network.
interface BVI2
ip address 192.168.2.1 255.255.255.0
--> no ip access-group 100 in
no ip redirects
ip nat enable
! Guest network.
interface BVI12
ip address 192.168.4.1 255.255.255.0
--> no ip access-group 100 in
no ip redirects
ip nat enable
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip dns server
ip nat source list 1 interface GigabitEthernet0/1 overload
--> no ip nat source list 65 interface GigabitEthernet0/1 overload
--> no ip nat source list 101 pool globalnet overload
ip nat source static tcp 192.168.0.35 22 interface GigabitEthernet0/1 2930
ip nat source static tcp 192.168.0.35 3030 interface GigabitEthernet0/1 3030
!
logging history size 128
!
access-list 1 permit 10.10.10.0 0.0.0.7
--> access-list 1 permit 192.168.0.0 0.0.255.255
access-list 23 permit 10.10.10.0 0.0.0.7
--> no access-list 65 permit 192.0.0.0 0.255.255.255
--> no access-list 72 permit 192.0.0.0 0.255.255.255
--> no access-list 100 deny ip host 255.255.255.255 any
--> no access-list 100 permit ip any any
!
control-plane
!
bridge 1 protocol ieee
bridge 1 route ip
bridge 2 protocol ieee
bridge 2 route ip
bridge 12 protocol ieee
bridge 12 route ip
!
no vstack
!
line con 0
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
login local
transport input ssh
line vty 5 15
access-class 23 in
privilege level 15
login local
transport input telnet ssh
!
scheduler allocate 20000 1000
event manager environment _internet_route_established_phrase LINK-3-UPDOWN
event manager directory user policy "usbflash0:/"
! Work-around for the NVI rule-dropping bug.
event manager policy fix_nvi.tcl
!
end

View solution in original post

Deepak Kumar · ‎10-03-2019

Hi,

As you assigned the primary DNS server as your Local Router. Can you change primary DNS server to 8.8.8.8 and test it the same?

!
ip dhcp pool dhcp0
 network 192.168.0.0 255.255.255.0
 default-router 192.168.0.1 
 no dns-server 192.168.0.1 208.67.222.123 208.67.220.123 8.8.8.8 
 dns-server 8.8.8.8 208.67.222.123 208.67.220.123 192.168.0.1
 accounting default

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

frank.trampe · ‎10-03-2019

Hi, Deepak.

Per my original message, the problem is that the router is not passing some DNS packets between the NAT outside and inside, so setting 8.8.8.8 as the first DNS server for internal DHCP clients would cause them to be unable to access some sites.

Georg Pauwen · ‎10-03-2019

Hello,

looking at your configuration, the first thing that looks odd is that you turned off CEF, which means everything gets process switched (which usually slows down everything). Try to enable process switching (ip cef).

I have made some changes (marked in bold) to your configuration, as a lot of things are reddundant:

! Last configuration change at 18:34:02 UTC Wed Oct 2 2019 by cisco
!
version 15.7
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname ht312-router
!
boot-start-marker
boot system usbflash0 c1900-universalk9-mz.SPA.157-3.M2.bin
boot-end-marker
!
enable secret 5 $4$MExE$REDACTED/REDACTED/XXX/
!
no aaa new-model
!
ip dhcp excluded-address 10.10.10.1
ip dhcp excluded-address 192.168.0.1 192.168.0.63
ip dhcp excluded-address 192.168.2.1 192.168.2.16
ip dhcp excluded-address 192.168.4.1 192.168.4.15
!
ip dhcp pool cvo-pool
import all
network 10.10.10.0 255.255.255.248
default-router 10.10.10.1
lease 0 2
!
ip dhcp pool dhcp0
network 192.168.0.0 255.255.255.0
default-router 192.168.0.1
dns-server 8.8.8.8 208.67.222.123 208.67.220.123 192.168.0.1
accounting default
!
ip dhcp pool dhcp2
network 192.168.2.0 255.255.255.0
default-router 192.168.2.1
dns-server 8.8.8.8 8.8.4.4 192.168.2.1
option 43 hex f104.c0a8.020e
accounting default
!
ip dhcp pool dhcp12
network 192.168.4.0 255.255.255.0
dns-server 8.8.8.8 208.67.222.123 208.67.220.123 192.168.0.1
accounting default
!
ip domain name local
ip name-server 8.8.8.8
ip name-server 8.8.4.4
--> ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
cts logging verbose
!
license udi pid CISCO1921/K9 sn XXXXXXXXXXX
!
username cisco privilege 15 secret 5 $4$JE78$REDACTED0REDACTED0XXX.
!
redundancy
!
bridge irb
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
! Trunk to main switch.
interface GigabitEthernet0/0
no ip address
duplex auto
speed auto
!
interface GigabitEthernet0/0.1
encapsulation dot1Q 1 native
bridge-group 1
!
interface GigabitEthernet0/0.2
encapsulation dot1Q 10
bridge-group 2
!
interface GigabitEthernet0/0.12
encapsulation dot1Q 12
bridge-group 12
! Uplink to modem.
interface GigabitEthernet0/1
ip address dhcp
--> no ip access-group 101 in
no ip redirects
no ip proxy-arp
ip flow ingress
ip nat enable
ip virtual-reassembly in
duplex auto
speed auto
! Main internal network.
interface BVI1
ip address 192.168.0.1 255.255.255.0
--> no ip access-group 100 in
no ip redirects
ip nat enable
! Wireless infrastructure network.
interface BVI2
ip address 192.168.2.1 255.255.255.0
--> no ip access-group 100 in
no ip redirects
ip nat enable
! Guest network.
interface BVI12
ip address 192.168.4.1 255.255.255.0
--> no ip access-group 100 in
no ip redirects
ip nat enable
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip dns server
ip nat source list 1 interface GigabitEthernet0/1 overload
--> no ip nat source list 65 interface GigabitEthernet0/1 overload
--> no ip nat source list 101 pool globalnet overload
ip nat source static tcp 192.168.0.35 22 interface GigabitEthernet0/1 2930
ip nat source static tcp 192.168.0.35 3030 interface GigabitEthernet0/1 3030
!
logging history size 128
!
access-list 1 permit 10.10.10.0 0.0.0.7
--> access-list 1 permit 192.168.0.0 0.0.255.255
access-list 23 permit 10.10.10.0 0.0.0.7
--> no access-list 65 permit 192.0.0.0 0.255.255.255
--> no access-list 72 permit 192.0.0.0 0.255.255.255
--> no access-list 100 deny ip host 255.255.255.255 any
--> no access-list 100 permit ip any any
!
control-plane
!
bridge 1 protocol ieee
bridge 1 route ip
bridge 2 protocol ieee
bridge 2 route ip
bridge 12 protocol ieee
bridge 12 route ip
!
no vstack
!
line con 0
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
login local
transport input ssh
line vty 5 15
access-class 23 in
privilege level 15
login local
transport input telnet ssh
!
scheduler allocate 20000 1000
event manager environment _internet_route_established_phrase LINK-3-UPDOWN
event manager directory user policy "usbflash0:/"
! Work-around for the NVI rule-dropping bug.
event manager policy fix_nvi.tcl
!
end

frank.trampe · ‎10-03-2019

Thanks, Georg!

I applied a few changes at a time to see the critical point, if any. DNS packets started transiting properly after the second of these steps below, with the others included since they seem closely related.

access-list 1 permit 192.168.0.0 0.0.255.255
no ip nat source list 65 interface GigabitEthernet0/1 overload
no ip nat source list 101 pool globalnet overload

rsync is still not working, so I guess that that's a separate issue, maybe MTU instead of NAT.

frank.trampe · ‎10-03-2019

Hi, Georg.

Upon further testing, I do not think that the remaining problem (just with rsync so far as I can tell) is an MTU issue after all. Could you take another look at my patched configuration in the new issue for other possible sources of erratic behavior?