INTERNET CONNECTIVITY ISSUE

[Raja_D]

Hi, 

There is an issue reported in my office network where our remote location is facing internet sites access issue when diverting on the backup internet link. However, the internet works fine when traffic is enabled to pass through the Primary internet link.

There are static route entries on the internet router and the natting for the private lan network with public ip's is being done on the Firewall.

Default route is pointed on Primary internet link with AD value of 1 and the same default route is pointed on the backup internet link with AD value of 10

so if the primary link goes down the default route automatically points towards the backup internet link with AD value 10 and the access to the public dns server with source as Public wan ip is reachable however when tried to open the websites user's report "Page cannot be displayed" error.

The dns server is also reachable from the user pc and as said the public dns server 4.2.2.2 and 8.8.8.8 are also reachable.

can someone guide as of what could be possibility of where the issue is happening and what can be checked at this stage.

I am suspecting that the Proxy is misbehaving.. Please suggest/Advise. 

An outline of the connectivity setup is shared as well. 

[chrihussey]

Yes, the remote location that has the issue when the primary link fails.

[Raja_D]

As said earlier there is no igp or bgp protocol being used in my router. we just have the static routing set on the Internet router.

There are two internet links with two different public ip's for two different isp's being in use, does that mean proxy should set both Public ip's in it for the user's to enable internet links ? please advise.

[chrihussey]

OK, then this might be the problem. I understand how you are routing and how your proxy is working. No issues there. The question is, is the IP address(s) used by the proxy, for the users, address space provided to the site from ISP #1?

If so, chances are very good ISP #2 does not route that to you on it's network. So when the link to ISP #1 fails, you are getting out to the Internet on ISP #2, but there is no return path. The specifics of how and why DNS and some sites work might be just because the site/proxy is using cached information.

Your secondary tunnel to the data center stays up because your are using the interface IP to ISP #2 which uses ISP #2's address space.

I hope that makes sense. Please verify.

[Raja_D]

so as per my understanding of your analysis the Lan pool 81.15.16.136/29 if provided by the ISP 1 then this lan pool should need to be advertised by the ISP 2 in their internet cloud ?

I am trying to get the details with the local IT shortly. please let me know if my understanding is correct or not about your analysis.

[chrihussey]

That is correct. Also, if it is the case then unfortunately the solution may not be that simple.

ISP #1 does not advertise the 81.15.16.136/29 to the rest of the Internet, it advertises 81.15.0.0/17. If you wanted to have ISP #2 advertise your netblock, ISP #1 would also have to change it's advertising to the same, and that would be unlikely.

ISP1 & 2 will not be able to advertise a /29 because generally the the Internet routing tables do not advertise anything smaller than a /24.

You would also need to get both ISPs to agree to this. I could probably go on a lot more on this but you get the point.

Assuming our diagnosis is correct, probably the quickest solution would be to get a second link to ISP #1. If possible, have it diversely routed and to a different POP than the primary. This way if the primary link fails the ISP will have no issue with the alternate route since it is all their network.

Mind you there are plenty ways to skin a cat, all varying in complexity and cost, but aside from that, this would be the simplest to implement.

[Raja_D]

Will confirm if the ISP 1 has advertised 81.15.0.0/17 or 81.15.16.136/29 in their internet cloud. If ISP 1 has advertised 81.15.0.0/17 then will get this verified with ISP 2 and will ask them to check if they can advertise the bip ip pool 81.15.0.0/17 in their internet cloud or not.

Although both links of two isp's are connected onto the same router but I guess that both of them would have a seperate POP that hold each of their telecommunication devices individually.

When you say this statement "Assuming our diagnosis is correct, probably the quickest solution would be to get a second link to ISP #1."  do you mean to have a second internet link that belongs to the same ISP#1 ?

If so I don't think so that it would not be possible as my company has been paying to the secondary ISP which is existing since from long time although not being able to use. And the issue is noticed recently when the Primary internet got failed. However, as reported to you earlier that the laptop is connected directly to the link after removing the link from router and got it tested to confirm that the internet is working or not.

So it worked, however issue raises only when connected on router for which we are trying to get a solution.

[chrihussey]

An ISP has their own network and advertises its routes to other ISPs. ISP #1 has the /29 route on it's network (and many others on the 81.15.0.0/17 range) but only advertises the 81.15.0.0/17 to the other Internet ISPs. I will not be possible for ISP #2 to advertise the 81.15.0.0/17 address space since they don't hold any of the routes on their network and it is not their space.

Yes, if this is indeed the issue, a possible solution is to have two links to one ISP.

If I understand you correctly, when the primary failed, you connected a laptop directly to the link to the second ISP. Correct? If so, when you did this, the laptop had to be given the 86.201.12.86 255.255.255.252 address. Which is ISP #2 space. That is why the laptop worked.

[Raja_D]

K then i will check with isp if they can advertise /29 in their internet cloud or not.

As per my knowledge not to have dependancy on same isp my company would have aligned a different provider for secondary link.

Infact with your guidance i got to know most of the possible causes of this issue so i feel that i could insist service provider to get this fixed from their end on priority as my company is paying for them.

Yes internet through isp#2 was tested on laptop with public ip of second isp n its associated gateway n dns server as given by isp.

This issue does not seem to be normal , looks like there is lot of exercise to be done..:)

Any more suggestions on this issue please advise..

[chrihussey]

Generally when one has connectivity to two different ISPs, they have their own registered IP space which is at least a /24. They can ask the ISPs to advertise it on their behalf or can peer directly with the ISPs and advertise it themselves.

Doubtful that it will be possible to advertise a /29. The Internet table is large enough and for that reason anything smaller than a /24 is not advertised or accepted by other ISPs, because it would make the route table impossibly large. Right now it is at about 650k routes.

Don't know if this is possible, but you could ask ISP #1 if they could provide you with a /24 that ISP#2 would be allowed to advertise. Might be a solution.

Best of luck.

[Raja_D]

Chrihussey,

As discussed with my proxy team, i got to know that there is no setting in proxy done for link configuration, it is just forwarding the traffic towards the default gateway.

Default gateway is the firewall ip and the interface public ip of it (81.15.16.138) .

[paul driver]

Hello

I assume the resiliency was tested previous and was successful, If so and you suspect the proxy then can you bypass it and test clients as I suggested, Also are you successful via IP instead or the FQDN.

If you have internet connectivity from client ( able to ping public dns etcc.) as you stated, then I am doubting at this time its a reachability issue.

Are you about to analyse from src/dst ( wireshark) and post the output

res
Paul

[Georg Pauwen]

Hello,

in addition to Paul's suggestions, web page display problems can also be caused by MTU size settings. Can you post the configs of the DC and the remote site routers ?

[chrihussey]

Some observations and thoughts.

1. Since you are static routing to the provider there may be asymmetrical routing going on in normal operation. You're traffic will use the primary link but you don't control the return traffic. So DNS requests go out the primary link, but may come back via the secondary link. If the primary fails the connectivity to the DNS servers remains. There are probably other web sites that follow this pattern.
2. What is the connection to the ISP? If your primary link goes down, does the interface at the ISP also fail? Or could it be accomplished via some sort of shared or switched network, where if your interface fails theirs may stay up and return traffic gets black holed.
3. With the primary failed, perform trace routes to various sites, it may lead to where the problem actually lies.
4. Are your connections to 1 ISP or two? And if the primary fails, does it still advertise your address space?

[Raja_D]

Hi Chrihussay, 

Thank you for your observations. 

Here are my answers to your queries. 

2. Its a Point to point connection with isp. Actually we have the GRE tunnel iin place for both the internet links, while tunnel source is the Physical interface ip of the internet link and the destination is the DC end Physical interface of the Internet link. 

Static route entries are in place and the default static route is pointing to Primary internet link and with higher AD value for the secondary link. 

The Physical interface goes down making the GRE tunnel to go down. Recently during a failover test i have shut the Primary internet link manually and the secondary internet link that was existing has taken its default route in the routing table. However, as said the internet is not accessible infact the public dns server is reachable. 

3. I did not go with the trace which i will need to recheck again when the primary goes down. But is there any other option to identify the issue cause when both the links are up as getting down time has to planned again. 

4. There are connections to two different ISP's but both of their connections are made on only single internet router. There is no igp or ebgp protocol used in setup just static routing entry of all the entries. 

I will try to get the configs as well but any suggestions or advise with these inputs will be of very helpful for me.