INTERNET CONNECTIVITY ISSUE

Raja_D · ‎01-07-2017

Hi,

There is an issue reported in my office network where our remote location is facing internet sites access issue when diverting on the backup internet link. However, the internet works fine when traffic is enabled to pass through the Primary internet link.

There are static route entries on the internet router and the natting for the private lan network with public ip's is being done on the Firewall.

Default route is pointed on Primary internet link with AD value of 1 and the same default route is pointed on the backup internet link with AD value of 10

so if the primary link goes down the default route automatically points towards the backup internet link with AD value 10 and the access to the public dns server with source as Public wan ip is reachable however when tried to open the websites user's report "Page cannot be displayed" error.

The dns server is also reachable from the user pc and as said the public dns server 4.2.2.2 and 8.8.8.8 are also reachable.

can someone guide as of what could be possibility of where the issue is happening and what can be checked at this stage.

I am suspecting that the Proxy is misbehaving.. Please suggest/Advise.

An outline of the connectivity setup is shared as well.

paul driver · ‎01-07-2017

Hello

can you bypass the proxy and test again also have a client flush its dns and reregister

res

Paul

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Raja_D · ‎01-08-2017

Paul,

As mentioned initially the same proxy is being used when user's are able to access the internet through Primary internet link .

Georg Pauwen · ‎01-08-2017

Post your config(s), we might be able to detect something.

Raja_D · ‎01-08-2017

Here is the config ...

track 1 ip sla 1 reachability
delay down 5 up 10

interface Loopback100
description "Loopback for MGMT"
ip address 10.10.x.x 255.255.255.255
!
!
interface Tunnel10
description Tunnel to DC
ip address 10.15.45.194 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
ip tcp adjust-mss 1300
load-interval 30
keepalive 10 3
tunnel source 86.201.12.86
tunnel destination 115.113.x.x
!
!
interface Tunnel11
description Tunnel to DC
ip address 10.15.45.206 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
ip tcp adjust-mss 1300
load-interval 30
keepalive 10 3
tunnel source 81.22.18.13
tunnel destination 115.113.x.x
!
!
interface GigabitEthernet0/0
description Connected to LAN
ip address 82.19.46.33 255.255.255.248 secondary
ip address 81.15.16.137 255.255.255.248
no ip redirects
no ip unreachables
no ip proxy-arp
ip accounting output-packets
ip flow ingress
load-interval 30
duplex auto
speed auto
no mop enabled
!
!
interface GigabitEthernet0/1
description Primary internet link
bandwidth 7168
ip address 81.22.18.13 255.255.255.252
ip access-group INTERNET-IN in
ip access-group INTERNET-OUT out
no ip redirects
no ip unreachables
no ip proxy-arp
load-interval 30
duplex auto
speed auto
no mop enabled
!
!
interface FastEthernet0/0/0
description Secondary Internet link
bandwidth 3072
ip address 86.201.12.86 255.255.255.252
ip access-group INTERNET-IN in
ip access-group INTERNET-OUT out
no ip redirects
no ip unreachables
no ip proxy-arp
load-interval 30
duplex auto
speed auto
no mop enabled
!

!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip route 0.0.0.0 0.0.0.0 81.22.18.14 track 1
ip route 61.x.x.x 255.255.255.255 81.22.18.14 track 1
ip route 200.x.x.x 255.255.255.255 81.22.18.14 track 1
ip route 201.x.x.x 255.255.255.0 81.22.18.14 track 1
ip route 203.27.x.x 255.255.255.255 81.22.18.14 track 1
ip route 203.27.x.x 255.255.255.255 81.22.18.14 track 1
ip route 203.27.x.x 255.255.255.255 81.22.18.14 track 1

ip route 0.0.0.0 0.0.0.0 86.201.12.85 10
ip route 10.24.x.x 255.255.255.255 Tunnel11
ip route 10.24.x.x 255.255.255.255 Tunnel10 10
ip route 10.60.x.x 255.255.255.255 Tunnel11
ip route 10.60.x.x 255.255.255.255 Tunnel10 10
ip route 71.x.x.x 255.255.192.0 86.201.12.85 10
ip route 61.x.x.x 255.255.255.255 86.201.12.85 10
ip route 200.x.x.x 255.255.255.255 86.201.12.85 10
ip route 201.x.x.x 255.255.255.0 86.201.12.85 10
ip route 203.27.x.x 255.255.255.255 86.201.12.85 10
ip route 203.27.x.x 255.255.255.255 86.201.12.85 10
ip route 203.27.x.x 255.255.255.255 86.201.12.85 10

deny ip any any log
ip access-list extended INTERNET-IN
deny ip 10.0.0.0 0.255.255.255 any
deny ip 172.16.0.0 0.15.255.255 any
deny ip 192.168.0.0 0.0.255.255 any
deny ip 127.0.0.0 0.255.255.255 any
deny ip host 0.0.0.0 any
deny ip 224.0.0.0 31.255.255.255 any
deny udp any eq ntp any
permit ip any any
ip access-list extended INTERNET-OUT
permit ip any host 203.27.x.xlog-input
permit ip any any

ip sla 1
icmp-echo 81.22.18.14 source-ip 81.22.18.13
frequency 15
ip sla schedule 1 life forever start-time now
logging 10.24.x.x

Georg Pauwen · ‎01-08-2017

Hello,

your 'secondary' route is not really secondary, since you have numerous static routes that are not tracked pointing to the other tunnel. Unless there is a specific reason for all those static routes, try to simplify your routing:

ip route 0.0.0.0 0.0.0.0 Tunnel11 track 1

ip route 0.0.0.0 0.0.0.0 Tunnel10 10

Also, add the below (in bold) to your tunnel configurations:

interface Tunnel10
description Tunnel to DC
ip address 10.15.45.194 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
ip tcp adjust-mss 1360
ip mtu 1400
load-interval 30
keepalive 10 3
tunnel source 86.201.12.86
tunnel destination 115.113.x.x
tunnel path-mtu-discovery
!
interface Tunnel11
description Tunnel to DC
ip address 10.15.45.206 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
ip tcp adjust-mss 1360
ip mtu 1400
load-interval 30
keepalive 10 3
tunnel source 81.22.18.13
tunnel destination 115.113.x.x
tunnel path-mtu-discovery

chrihussey · ‎01-09-2017

When the primary link fails, there is still access to the data center via the secondary tunnel, but Internet access (browsing) fails to many (not all) sites even though the public DNS servers are successful. Correct?

Could this be an Internet routing issue where when your primary link fails routing still occurs to both ISPs even though you are not accessible via the primary?

You could check some of the Looking Glass sites to see how your NAT address space is being advertised to the Internet. You can also do trace routes from many of these sites to see how things are working in normal operation. If you have other remote sites with a similar configuration, perform trace routes from them to this site too. Especially note ones that use your primary link.

At some point, with the primary link failed, check the Looking Glass sites for the NAT block again and perform trace routes just to confirm the secondary ISP is the only route.

Raja_D · ‎01-09-2017

Hi chrihussey

When the primary link fails, there is still access to the data center via the secondary tunnel, but Internet access (browsing) fails to many (not all) sites even though the public DNS servers are successful. Correct? Yes correct.

will perform trace and revert.

Thanks for your suggestion...

Raja_D · ‎01-10-2017

Hi Chrihussey,

Can you please guide me on Looking Glass sites for the NAT block. Trace to which ip and from where you suggest to perform the trace ?

chrihussey · ‎01-10-2017

You should check the routing and trace to the NAT IP(s) that the proxy is using.

If you google "Internet Looking Glass Sites" or go to traceroute.org you'll get plenty of sites.

One example is if you go to the Sprint Looking Glass site (https://www.sprint.net/lg/) you put in the IP of your LAN interface (81.15.16.137) and query the Internet routing table (sh bgp route) you get the following result:

Tue Jan 10 17:42:23.904 UTC
BGP routing table entry for 81.15.0.0/17
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker          157674813   157674813
Last Modified: Jan  9 10:24:20.801 for 1d07h
Paths: (5 available, best #1)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Not advertised to any peer
  1273 12969, (aggregated by 12969 217.151.190.225)
    144.228.241.49 (metric 120) from 144.228.241.16 (144.228.241.49)
      Origin IGP, metric 4294967294, localpref 90, valid, internal, atomic-aggregate, best, group-best
      Received Path ID 0, Local Path ID 1, version 157674813
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1006
      Originator: 144.228.241.49, Cluster list: 144.228.241.16
  Path #2: Received by speaker 0
  Not advertised to any peer
  1273 12969, (aggregated by 12969 217.151.190.225)
    144.228.241.49 (metric 120) from 144.228.241.17 (144.228.241.49)
      Origin IGP, metric 4294967294, localpref 90, valid, internal, atomic-aggregate
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1006
      Originator: 144.228.241.49, Cluster list: 144.228.241.17
  Path #3: Received by speaker 0
  Not advertised to any peer
  1273 12969, (aggregated by 12969 217.151.190.225)
    144.228.241.128 (metric 780) from 144.228.241.128 (144.228.241.128)
      Origin IGP, metric 4294967294, localpref 90, valid, internal, atomic-aggregate
      Received Path ID 0, Local Path ID 0, version 0
      Community: internet 1239:666 1239:667 1239:1000 1239:1015
  Path #4: Received by speaker 0
  Not advertised to any peer
  1273 12969, (aggregated by 12969 217.151.190.225)
    144.228.243.251 (metric 728) from 144.228.241.132 (144.228.243.251)
      Origin IGP, metric 4294967294, localpref 90, valid, internal, atomic-aggregate
      Received Path ID 0, Local Path ID 0, version 0
      Community: internet 1239:500 1239:666 1239:667 1239:1000 1239:1014
      Originator: 144.228.243.251, Cluster list: 144.228.241.132
  Path #5: Received by speaker 0
  Not advertised to any peer
  1273 12969, (aggregated by 12969 217.151.190.225)
    144.228.243.251 (metric 728) from 144.228.241.133 (144.228.243.251)
      Origin IGP, metric 4294967294, localpref 90, valid, internal, atomic-aggregate
      Received Path ID 0, Local Path ID 0, version 0
      Community: internet 1239:500 1239:666 1239:667 1239:1000 1239:1014
      Originator: 144.228.243.251, Cluster list: 144.228.241.133

So you can see it is advertised as a 81.15.0.0 /17 netblock originating from AS 12969.

You can also trace route to it from various Sprint sites:

Not sure if your IPs in your config are the ones you are using but this is what you get if you trace route to it from the Sprint site:

Sprint Source: Anaheim, CA (sl-crs3-ana)
User-defined destination: 81.15.16.137
Performing: ICMP Traceroute
IP Version: IPv4

Tracing the route to  (81.15.16.137)

 1  144.232.13.244 4 msec  4 msec  3 msec 
 2  144.232.6.104 [sl-st50-la-.sprintlink.net] 2 msec  1 msec  4 msec 
 3  144.232.12.209 0 msec  3 msec  2 msec 
 4  144.223.54.190 4 msec  3 msec  2 msec 
 5  195.2.28.49 [xe-1-1-0-xcr1.ash.cw.net] [MPLS: Label 386669 Exp 0] 145 msec  152 msec  151 msec 
 6  195.2.28.33 [xe-0-1-1-xcr1.nyh.cw.net] [MPLS: Label 579765 Exp 0] 135 msec 
    195.2.30.46 63 msec  67 msec 
 7  195.2.28.170 [ae3-xcr1.slo.cw.net] [MPLS: Label 538128 Exp 0] 135 msec 
    195.2.25.1 [ae25-xcr1.lns.cw.net] 135 msec  143 msec 
 8  195.2.24.34 [ae4-xcr1.ltw.cw.net] [MPLS: Label 360604 Exp 0] 135 msec 
    166.63.223.22 [vodafone-ice-gw.lns.cw.net] 131 msec  139 msec 
 9  217.151.190.147 [te3-5-D01-Sidumuli.c.is] 175 msec  *  * 
 10  *  *  * 
 11 166.63.223.22 [vodafoneiceland-gw-xcr1.lns.cw.net] 129 msec  *  *

You can go the various Looking Glass sites and find the one you are most comfortable with. If you have any other questions just let me know.

Raja_D · ‎01-10-2017

Hi Chrihussey,

So you mean to say that I first need to get the natted pool range of my lan network 81.15.16.137 255.255.255.248 & 82.19.46.33 255.255.255.248 that is reserved in the Proxy server and then perform a trace to that nat pool range ip from the glass site ?

If I do a "show bgp route" from the sprint site it would show best path but how would i conclude from it if its pointing towards the Primary internet link or the secondary internet link .? And does this output would show me the reverse traffic flow from internet back to my enterprise network ?

Below is the a sample out that have taken for lan network from the glass site. please guide me with this output as of what things that i need to look into this and what nformation in this would be helpful to me in isolating the issue.

BGP routing table entry for 81.22.18.0/24
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker          156061629   156061629
Last Modified: Jan  7 15:55:03.846 for 3d02h
Paths: (17 available, best #2)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Not advertised to any peer
  6453 48237 5416 39273
    144.228.241.8 (metric 700) from 144.228.241.8 (144.228.241.8)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:666 1239:667 1239:1000 1239:1026
  Path #2: Received by speaker 0
  Not advertised to any peer
  6453 48237 5416 39273
    144.228.242.51 (metric 120) from 144.228.241.16 (144.228.242.51)
      Origin IGP, metric 4294967294, localpref 90, valid, internal, best, group-best
      Received Path ID 0, Local Path ID 1, version 156061629
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1006
      Originator: 144.228.242.51, Cluster list: 144.228.241.16
  Path #3: Received by speaker 0
  Not advertised to any peer
  6453 48237 5416 39273
    144.228.242.51 (metric 120) from 144.228.241.17 (144.228.242.51)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1006
      Originator: 144.228.242.51, Cluster list: 144.228.241.17
  Path #4: Received by speaker 0
  Not advertised to any peer
  6453 48237 5416 39273
    144.228.241.39 (metric 146) from 144.228.241.44 (144.228.241.39)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1011
      Originator: 144.228.241.39, Cluster list: 144.228.241.44
  Path #5: Received by speaker 0
  Not advertised to any peer
  6453 48237 5416 39273
    144.228.241.39 (metric 146) from 144.228.241.45 (144.228.241.39)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1011
      Originator: 144.228.241.39, Cluster list: 144.228.241.45
  Path #6: Received by speaker 0
  Not advertised to any peer
  6453 48237 5416 39273
    144.228.243.98 (metric 550) from 144.228.241.124 (144.228.243.98)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1007
      Originator: 144.228.243.98, Cluster list: 144.228.241.124
  Path #7: Received by speaker 0
  Not advertised to any peer
  6453 48237 5416 39273
    144.228.243.98 (metric 550) from 144.228.241.125 (144.228.243.98)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1007
      Originator: 144.228.243.98, Cluster list: 144.228.241.125
  Path #8: Received by speaker 0
  Not advertised to any peer
  6453 48237 5416 39273
    144.228.242.37 (metric 384) from 144.228.241.126 (144.228.242.37)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1002
      Originator: 144.228.242.37, Cluster list: 144.228.241.126
  Path #9: Received by speaker 0
  Not advertised to any peer
  6453 48237 5416 39273
    144.228.242.37 (metric 384) from 144.228.241.127 (144.228.242.37)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1002
      Originator: 144.228.242.37, Cluster list: 144.228.241.127
  Path #10: Received by speaker 0
  Not advertised to any peer
  6453 48237 5416 39273
    144.228.241.128 (metric 780) from 144.228.241.128 (144.228.241.128)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:666 1239:667 1239:1000 1239:1015
  Path #11: Received by speaker 0
  Not advertised to any peer
  3257 48237 5416 39273
    144.228.241.129 (metric 781) from 144.228.241.129 (144.228.241.129)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:666 1239:667 1239:1000 1239:1015
  Path #12: Received by speaker 0
  Not advertised to any peer
  6762 48237 5416 39273
    144.228.243.250 (metric 728) from 144.228.241.132 (144.228.243.250)
      Origin IGP, metric 4294967294, localpref 90, valid, internal, group-best
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1014
      Originator: 144.228.243.250, Cluster list: 144.228.241.132
  Path #13: Received by speaker 0
  Not advertised to any peer
  6762 48237 5416 39273
    144.228.243.250 (metric 728) from 144.228.241.133 (144.228.243.250)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1014
      Originator: 144.228.243.250, Cluster list: 144.228.241.133
  Path #14: Received by speaker 0
  Not advertised to any peer
  6453 48237 5416 39273
    144.228.241.104 (metric 415) from 144.228.243.201 (144.228.241.104)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:666 1239:667 1239:1000 1239:1010
      Originator: 144.228.241.104, Cluster list: 144.228.243.201
  Path #15: Received by speaker 0
  Not advertised to any peer
  3257 48237 5416 39273
    144.228.241.4 (metric 597) from 144.228.243.241 (144.228.241.4)
      Origin IGP, metric 4294967294, localpref 90, valid, internal, group-best
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1004
      Originator: 144.228.241.4, Cluster list: 144.228.243.241
  Path #16: Received by speaker 0
  Not advertised to any peer
  3257 48237 5416 39273
    144.228.241.4 (metric 597) from 144.228.243.246 (144.228.241.4)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:500 1239:666 1239:667 1239:1000 1239:1004
      Originator: 144.228.241.4, Cluster list: 144.228.243.246
  Path #17: Received by speaker 0
  Not advertised to any peer
  6453 48237 5416 39273
    144.228.241.104 (metric 415) from 144.228.243.252 (144.228.241.104)
      Origin IGP, metric 4294967294, localpref 90, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
      Community: 1239:666 1239:667 1239:1000 1239:1010
      Originator: 144.228.241.104, Cluster list: 144.228.243.252

Raja_D · ‎01-10-2017

Chrihussey,

Does this require me to co-ordinate with the local isp once after we have the AS number details to confirm it from the SP end as i mentioned earlier from my internet router on the branch location we just have static routes in place and no bgp protocol being.

Internet links are point to point links.

please advise

chrihussey · ‎01-10-2017

Your two Internet links are point to point to two different ISPs and both ISPs should have routing in place to the site's public IP space, part of which is used by the proxy. Correct? They also should obviously advertise this space to the rest of the Internet.

If you do a trace route from a Looking Glass site to an IP in that range, you should see at some point the IP address of your site's Internet link IPs. That's what you want to see to determine which ISP you are routing through. In some cases, if you go to a different looking glass site and different ISP you may see routing to the site via your secondary ISP.

Once this is determined you can do the fail over test to check your address space on the Internet and see how you route. In this respect it would be good if you tested from a few sites, because some may work and others may not.

The BGP table shows the originating AS, in the case above it is 39273. So if you do a trace route from that site and you see your ISP link at the end of the trace, you know the AS. The secondary ISP has a different AS, so again when you fail the primary you should then see the other AS when doing a similar query.

If testing reflects it is a routing issue, then it should just be a matter of dealing with the local ISP.

chrihussey · ‎01-11-2017

Question:

Did the site get a netblock from one or both ISPs and does the proxy use one or both to provide the users Internet access?

Or, did the site have its on Internet address space that the ISPs are advertising on its behalf?

Raja_D · ‎01-11-2017

Just to confirm when you say as site does that mean the remote location for which the two internet links are existing ?