Hello, all
Although I read the forums frequently, this is my first time posting. Please know that I'm primarily a sysadmin and developer, not a network engineering specialist, and that I've worked much more with Cisco switches rather than routers. IOW, if I'm committing an act of stupidity in my configs, please feel free to constructively enlighten me.
The problem summary is that outbound Internet connectivity works reliably for the overwhelming majority of public IP addresses, but intermittently fails for certain domains. I am not certain, but there may be a correlation that the failing domains use content delivery networks (CDNs) such as Akamai and their competitors. The sites that are failing are major companies, such that it's unlikely their servers are simply down. The symptom is sometimes the address being unpingable from our network as a whole, but sometimes it is pingable from the router but not from our internal hosts via NAT. In the latter case, a traceroute from the internal host shows exactly the same hops as a traceroute from the router itself, except that the internal host adds the router's inside IPv4 address as a first hop. The ISP's eBGP peer address is the first hop from the router and the second hop from the internal host.
Below is a simplified and redacted network diagram. We have two routers, with each connected over an Ethernet eBGP peering to one ISP. The primary router (01) connects to the primary ISP, and the secondary router (02) connects to the secondary ISP. Both primary and secondary ISPs advertise only default routes to us via eBGP; we are a small organization with neither the resources nor the need for larger routing tables.
Although each router is provisioned for both ISPs and has Layer 2 connectivity to them, the ISPs are on separate VLANs, and each router's subinterface is administratively shutdown for the ISP to which it doesn't normally connect. This extra provisioning is intended to allow us to quickly (but manually) adapt to unusual situations in which one router fails and the opposite ISP fails during the same outage.
Each router can reach the public Internet through the other one if its own eBGP connection is down; I have verified this by admin shutdown of the eBGP subinterfaces.
For private and DMZ VLANs, the VIPA for HSRP is the first usable address in the subnet, and this is configured as the default gateway for all hosts on that subnet. The device-specific IPv4 addresses are the last two usable addresses in the subnet. HSRP is not in use on the OSPF backbone nor on the [effectively] point-to-point eBGP segments; we rely the routing protocols to fail us over, and on metrics/priority to ensure that router01 is preferred for each path. The intended behavior is:
| router01 | router02 | ISP 1 | ISP 2 | Internal routing | Public routing |
| up | (any) | up | (any) | router01 | router01 + isp1 |
| up | up | down | up | router01 | router02 (via router01) + isp2 |
| down | up | (any) | up | router02 | router02 + isp2 |
| down | up | up | down | router02 | router02 + isp1 (manual ISP failover) |
For internal hosts connecting to the public Internet, I've set up two non-overlapping NAT pools (one on each router) in the U.V.W.0/24 subnet, which is our registered public IPv4 address block. We have a VoIP server that is an exception; I found that our trunk provider didn't like dynamic IP addresses at our end, so that server has a hardwired NAT outside address.
Following is my excerpted config file, again with placeholder IP addresses matching the above diagram:
hostname router01 ! redundancy mode none ! interface Loopback0 description Always-up pingable address ip address 10.250.1.1 255.255.255.0 ipv6 address ************************** ! interface Port-channel1 description Data trunk no ip address ! interface Port-channel1.3 description OSPF Backbone encapsulation dot1Q 3 ip address 10.0.5.1 255.255.255.0 no ip redirects ip nat inside ip ospf priority 100 ipv6 address ************************** ip virtual-reassembly ! interface Port-channel1.6 description Core production servers encapsulation dot1Q 6 ip address 10.1.3.254 255.255.252.0 ip nat inside standby version 2 standby 6 ip 10.1.0.1 standby 6 priority 110 standby 6 preempt delay minimum 60 ip virtual-reassembly ! interface Port-channel1.665 description DMZ encapsulation dot1Q 665 ip address 10.0.7.254 255.255.255.0 ip nat inside ip access-group from_dmz_ipv4 in standby version 2 standby 665 ip 10.0.7.1 standby 665 priority 110 standby 665 preempt delay minimum 60 ipv6 address ****:****:****:****:****.0.7.254/112 ! interface Port-channel1.667 description External IP segment encapsulation dot1Q 667 ip address U.V.W.254 255.255.255.0 no ip redirects ip access-group from_external_ipv4 in standby version 2 standby 667 ip U.V.W.1 standby 667 priority 110 standby 667 preempt delay minimum 60 ipv6 address ****:****:****:0:U.V.W:254/112 ip virtual-reassembly ! interface Port-channel2 description ISP trunk no ip address negotiation auto ! interface Port-channel2.6 shutdown ! interface Port-channel2.1997 description BGP to secondary ISP encapsulation dot1Q 1997 ip address E.F.G.106 255.255.255.252 ip nat outside ip access-group from_isp2_ipv4 in shutdown ! interface Port-channel2.1998 description BGP to primary ISP encapsulation dot1Q 1998 ip address A.B.C.146 255.255.255.252 ip nat outside ip access-group from_isp1_ipv4 in ! interface Port-channel2.1999 description BGP to Primary ISP IPv6 encapsulation dot1Q 1999 ipv6 address ****:****:****:****::941/127 ipv6 enable ipv6 traffic-filter from_isp1_ipv6 in ! router ospfv3 100 router-id 10.250.1.1 ! address-family ipv4 unicast redistribute static redistribute bgp OUR_ASN metric 120 passive-interface default no passive-interface Port-channel1.3 exit-address-family ! address-family ipv6 unicast passive-interface default no passive-interface Port-channel1.3 redistribute bgp OUR_ASN metric 120 redistribute static exit-address-family ! router ospf 100 router-id 10.250.1.1 priority 100 redistribute connected metric 2 redistribute static metric 1 redistribute bgp OUR_ASN metric 120 passive-interface default no passive-interface Port-channel1.3 network 10.250.1.0 0.0.0.255 area 0 network 10.0.5.0 0.0.0.255 area 0 network 10.0.7.0 0.0.0.255 area 0 network 10.1.0.0 0.0.3.255 area 0 network U.V.W.0 0.0.0.255 area 0 default-information originate ! router bgp OUR_ASN bgp router-id 10.250.1.1 bgp log-neighbor-changes no bgp default ipv4-unicast neighbor E.F.G.105 remote-as ISP2_ASN neighbor E.F.G.105 description Secondary ISP neighbor ****:****:****:****::940 remote-as ISP1_ASN neighbor ****:****:****:****::940 description Primary ISP neighbor ****:****:****:****::940 update-source Port-channel2.1999 neighbor A.B.C.145 remote-as ISP1_ASN neighbor A.B.C.145 description Primary ISP neighbor A.B.C.145 update-source Port-channel2.1998 ! address-family ipv4 network U.V.W.0 neighbor E.F.G.105 activate neighbor E.F.G.105 weight 20000 neighbor E.F.G.105 soft-reconfiguration inbound neighbor E.F.G.105 prefix-list IPV4_ANY in neighbor E.F.G.105 prefix-list IPV4_EXT out neighbor A.B.C.145 activate neighbor A.B.C.145 weight 21000 neighbor A.B.C.145 soft-reconfiguration inbound neighbor A.B.C.145 prefix-list IPV4_ANY in neighbor A.B.C.145 prefix-list IPV4_EXT out default-information originate exit-address-family ! address-family ipv6 redistribute static network ****:****:****::/48 neighbor ****:****:****:****::940 activate neighbor ****:****:****:****::940 weight 21000 neighbor ****:****:****:****::940 soft-reconfiguration inbound neighbor ****:****:****:****::940 prefix-list IPV6_ANY in neighbor ****:****:****:****::940 prefix-list IPV6_ARIN out exit-address-family ! ip forward-protocol nd ! ip nat pool natpool U.V.W.240 U.V.W.249 prefix-length 24 ip nat inside source static udp OUR_VOIP_SERVER 5060 U.V.W.37 5060 extendable ip nat inside source static OUR_VOIP_SERVER U.V.W.37 ip nat inside source list ipv4_rfc1918_nat pool natpool overload ip route DEVLAB_TEST_SEGMENT 255.255.255.0 DEVLAB_TEST_ROUTER ip route U.V.W.0 255.255.255.0 Null0 ! ip access-list extended from_external_ipv4 10 deny ip host 0.0.0.0 any 20 deny ip 127.0.0.0 0.255.255.255 255.0.0.0 0.255.255.255 30 deny ip 10.0.0.0 0.255.255.255 255.0.0.0 0.255.255.255 40 deny ip 172.16.0.0 0.15.255.255 255.240.0.0 0.15.255.255 50 deny ip 192.168.0.0 0.0.255.255 255.255.0.0 0.0.255.255 60 deny ip 192.0.2.0 0.0.0.255 255.255.255.0 0.0.0.255 70 deny ip 169.254.0.0 0.0.255.255 255.255.0.0 0.0.255.255 80 deny ip 240.0.0.0 15.255.255.255 any 90 deny icmp any any fragments 100 permit udp any host 224.0.0.102 eq 1985 110 permit icmp any any echo-reply 120 permit icmp any any packet-too-big 130 permit icmp any any source-quench 140 permit icmp any any time-exceeded 150 permit icmp U.V.W.0 0.0.0.255 any 160 deny icmp any any 170 permit tcp any any established 180 deny ip any 10.0.0.0 0.255.255.255 190 deny ip any 172.16.0.0 0.15.255.255 200 deny ip any 192.168.0.0 0.0.255.255 210 deny ip any 169.254.0.0 0.0.255.255 220 deny ip any 240.0.0.0 15.255.255.255 230 permit ip U.V.W.0 0.0.0.255 any 240 deny ip any any ! ip access-list extended from_isp1_ipv4 10 deny ip host 0.0.0.0 any 20 deny ip 127.0.0.0 0.255.255.255 255.0.0.0 0.255.255.255 30 deny ip 10.0.0.0 0.255.255.255 255.0.0.0 0.255.255.255 40 deny ip 172.16.0.0 0.15.255.255 255.240.0.0 0.15.255.255 50 deny ip 192.168.0.0 0.0.255.255 255.255.0.0 0.0.255.255 60 deny ip 192.0.2.0 0.0.0.255 255.255.255.0 0.0.0.255 70 deny ip 169.254.0.0 0.0.255.255 255.255.0.0 0.0.255.255 80 deny ip 240.0.0.0 15.255.255.255 any 90 deny icmp any any fragments 100 permit icmp any U.V.W.0 0.0.0.255 echo 110 permit icmp any any echo-reply 120 permit icmp any any packet-too-big 130 permit icmp any any source-quench 140 permit icmp any any time-exceeded 150 permit icmp any A.B.C.144 0.0.0.3 160 deny icmp any any 170 permit tcp any host A.B.C.146 eq bgp 180 permit tcp any any established 190 permit tcp any U.V.W.0 0.0.0.255 eq www 200 permit tcp any U.V.W.0 0.0.0.255 eq 443 210 permit tcp any U.V.W.0 0.0.0.255 eq smtp ! ! --- Some permit rules here for specific protocols ! --- to app public-facing app servers (no deny ! --- rules in this section) ! 250 permit tcp any U.V.W.0 0.0.0.255 eq domain 260 permit udp any U.V.W.0 0.0.0.255 eq domain ! ! --- Some permit rules here for specific protocols ! --- to app public-facing app servers (no deny ! --- rules in this section) ! 300 permit udp any U.V.W.0 0.0.0.255 320 deny ip any any ! ip access-list extended from_private_ipv4 10 permit tcp any any established 20 permit udp any host 224.0.0.102 eq 1985 30 permit icmp any any 40 permit ip any 10.0.0.0 0.255.255.255 60 remark Internal hosts not allowed outbound SMTP ports 60 deny tcp any any eq smtp 70 deny tcp any any eq 10025 80 deny tcp any any eq 465 90 deny tcp any any eq 587 100 permit ip 10.0.0.0 0.255.255.255 any 120 remark Anything not allowed is forbidden 120 deny ip any any ! ip access-list extended ipv4_rfc1918_nat 10 deny ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255 20 deny ip 192.168.0.0 0.0.255.255 172.16.0.0 0.15.255.255 30 deny ip 192.168.0.0 0.0.255.255 10.0.0.0 0.255.255.255 40 deny ip 172.16.0.0 0.15.255.255 192.168.0.0 0.0.255.255 50 deny ip 172.16.0.0 0.15.255.255 172.16.0.0 0.15.255.255 60 deny ip 172.16.0.0 0.15.255.255 10.0.0.0 0.255.255.255 70 deny ip 10.0.0.0 0.255.255.255 192.168.0.0 0.0.255.255 80 deny ip 10.0.0.0 0.255.255.255 172.16.0.0 0.15.255.255 90 deny ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255 94 deny udp host OUR_VOIP_SERVER any 100 permit ip 192.168.0.0 0.0.255.255 any 110 permit ip 172.16.0.0 0.15.255.255 any 120 permit ip 10.0.0.0 0.255.255.255 any 130 deny ip any any ! ipv6 access-list from_isp1_ipv6 sequence 10 permit tcp any any established sequence 20 permit icmp any ****:****:****::/64 sequence 30 permit tcp ****:****:****:****::940/127 any eq bgp sequence 40 deny ipv6 any any ! ip prefix-list IPV4_ANY seq 5 permit 0.0.0.0/0 ! ip prefix-list IPV4_EXT seq 5 permit U.V.W.0/24 ! ipv6 route ****:****:****::/48 Null0 ! ! ipv6 prefix-list IPV6_ANY seq 5 permit ::/0 ! ipv6 prefix-list IPV6_ARIN seq 5 permit ****:****:****::/48 ! end
What I have tested and (I think) ruled out so far:
- The secondary router is not taking control of the HSRP interfaces; "show standby" on both routers shows the expected status.
- The internal hosts having problems getting outside have correct routing tables, with the HSRP VIPA as their default gateway.
- The unreachable public IPv4 addresses do not overlap any addresses in our internal ranges (we use only the RFC 1918 private addresses internally).
- Both ISPs are advertising their default route to us, as expected.
- Our OSPF and BGP routing tables look sensible and don't show things going via router02 that should be via router01.
- In cases where an external host is unreachable from either an internal host or from our router, the external host is pingable from a cloud server that is not on our network.
- Disabling IPv6 on our internal host doesn't change the situation. I had originally considered that we might be getting AAAA records for public hosts, and our internal hosts trying to use IPv6 when we don't yet have it fully deployed at the router.
Any insight into why I might have problems connecting to certain external hosts would be appreciated. Also, if there's anything I've configured that is bad network design, please let me know that as well, so I can improve the config in general.
Many thanks for any suggestions or advice.
