BGP Failover didn't fail back? Help!

davidbnbf · ‎11-08-2017

We have a netblock of IPs that are configured for BGP between 2 ISPs for failover and redundancy. We had an issue with our primary site ISP and brought it back online. Everything seemed fine and existing IPs and NATs are working properly for the primary site. However I tried to set up a new NAT at that site with another public IP that should be tied to this ISP and I can't get any reply packets back from the internet. Our theory is that the IP is tied to the secondary ISP for some reason during a BGP failover and never came back.

Unfortunately the person who configured BGP is long gone and I am not network expert. I know enough to be dangerous :)

Can someone help me figure out how to reset the BGP so that the IPs are associated with the primary ISP instead of the secondary? How can I restore this? It was my understanding this configuration was supposed to be pretty automatically but something has clearly gone awry.

Jon Marshall · ‎11-08-2017

If the public IP that is not working is part of the same IP block that has switched back to your preferred ISP then it is unlikely to be a BGP issue as ISPs generally don't accept anything less than a /24 subnet and certainly don't accept host advertisements.

Which means you need to check whether it is a BGP problem or a NAT configuration issue, so go to this site-

https://tools.keycdn.com/traceroute

and do a traceroute to the IP address in question. You may see the trace going via both ISPs depending on the source location but most if not all should be going via one or other of the ISPs.

If the last hop is via the preferred ISP then it is more likely a configuration issue with the NAT setup.

Try it and let us know the results.

Jon

davidbnbf · ‎11-08-2017

Thank you for the insights. You are indeed correct. When I tracert they both still go to primary site. The last IP I see when tracing any of the newly created NATs is an IP assigned to a vlan interface on the BGP switch. It doesn't seem to be getting past that. Any further thoughts? I'd love to pin this back on the firewall but I already worked with support and they said contact my ISP. They saw outbound ICMP going out the correct interface but no return packets from internet traffic.

I think this VLAN interface IP which is the last hop in the tracert is just for a port channel between the BGP router and BGP switch. Does this mean the outside interface of the firewall which is also plugged in to the same BGP switch is not accepting the traffic?

Jon Marshall · ‎11-08-2017

The firewall may well not respond to a traceroute especially on the outside interface.

Just to clarify, you are trying to use a spare public IP from an existing block and the NAT is presumably setup on the firewall ?

If so is this new IP being used to NAT internal IPs for access to internet or to present a server etc. for access from the internet ?

Either way if the traceroute for the IP is going via the primary ISP it is not a BGP issue as far as I can see.

Jon

davidbnbf · ‎11-08-2017

I would like to open up a port from the internet to the server on the inside. But I also want the server to surf on the same NAT IP and not the main surfing IP of the office. I am using a spare IP from the existing public NETblock we own. I've tried several spare IPs actually and I can't get any working. These are only off by 1 address from working ones for example.

When we do this we use static NATs and that usually makes the internet surfing from from the same NATed IP.

However this time around when we configure the NAT the server immediately cannot route to the internet. However if we do a packet trace on the firewall the cisco guys said they can see the packets leaving the correct NAT IP but no replies.

I am stumped because there are other internal servers on the same subnet with static NATs that are working fine.

Jon Marshall · ‎11-08-2017

It sounds very much like a configuration issue on the firewall.

I can see why you thought it was BGP if the firewall guys are saying they can see packets leaving with the right IP but if the traceroute you ran showed the path via the primary ISP I don't think it is routing.

Have you told them about the traceroute results ?

Without seeing the firewall configuration it is going to be hard to help further.

Jon

Tinashe Ndhlovu · ‎11-08-2017

what are you using to NAT if i may ask and may we get the new NAT configs..

On another note if you are using a statefull firewall e.t.c... your Routing Tables as they pertain to BGP are only concerned with outgoing traffic... yes you can use MED to influence return traffic etc and other BUT.. your routing table has routes for outgoing not incoming traffic.. goodness knows how return traffic is getting to you... could be coming back via ISP 2 and if using a state-full firewall this could cause issues

T

davidbnbf · ‎11-08-2017

@Tinashe Ndhlovu wrote:

what are you using to NAT if i may ask and may we get the new NAT configs..

On another note if you are using a statefull firewall e.t.c... your Routing Tables as they pertain to BGP are only concerned with outgoing traffic... yes you can use MED to influence return traffic etc and other BUT.. your routing table has routes for outgoing not incoming traffic.. goodness knows how return traffic is getting to you... could be coming back via ISP 2 and if using a state-full firewall this could cause issues

T

I am not sure on the answers to this. It is a 5515X ASA firewall.

davidbnbf · ‎11-08-2017

object network internalIP
nat (any,any) static publicIP net-to-net

I have other internal IPs on the same subnet as the one I am trying to setup. I also have other public IPs on the same subnet working fine. I can't fathom why this new NAT is not working. We recently updated firmware on ASA and had that BGP incident last week so just thinking of possible causes and solutions.

davidbnbf · ‎11-08-2017

Very simple NAT. I can route out but no return packets.

8 (any) to (any) source static VM-STOCKIQ VM-STOCKIQ-PUBLIC net-to-net
translate_hits = 7, untranslate_hits = 0

Georg Pauwen · ‎11-08-2017

Hello,

which ASA (e.g. 9.8) version are you running ? Can you post the static entry and the objects for a working entry, as well as the objects for the non-working entry ?

davidbnbf · ‎11-08-2017

@Georg Pauwen wrote:

Hello,

which ASA (e.g. 9.8) version are you running ? Can you post the static entry and the objects for a working entry, as well as the objects for the non-working entry ?

!

object network VM-EXCHANGE

nat (inside,outside) static EXCHANGE-PUBLIC net-to-net dns

object network VM-DNSDHCP

nat (any,any) static DNSDHCP-PUBLIC net-to-net

object network VIDEO-CONF-PRIVATE

nat (inside,outside) static VIDEO-CONF-PUBLIC net-to-net

object network VM-WEBPROXY1

nat (DMZ,outside) static VM-WEBPROXY1-PUBLIC net-to-net dns

object network VM-WEBPROXY2

nat (DMZ,outside) static VM-WEBPROXY2-PUBLIC net-to-net dns

object network Cisco-SocialMiner-DMZ

nat (DMZ,outside) static Cisco-SocialMiner-Public dns

object network Cisco-Expressway-DMZ

nat (DMZ,outside) static Cisco-Expressway-Public dns

object network VM-STOCKIQ

nat (any,any) static VM-STOCKIQ-PUBLIC net-to-net

!

Everything is working except the last entry for VM-STOCKIQ. It is using the an available internal IP on the same subnet as all the other entries and also using an available public IP on the same subnet as the other entries. The only thing different is we updated the firmware on the ASA from 9.5.3 to ASA Version 9.6(3)14 a week ago before we put this last NAT in for STOCKIQ.

When the NAT is configured we can capture traffic leaving properly on the NAT IP but no reply traffic.

Georg Pauwen · ‎11-08-2017

Hello,

what I meant was the objects themselves (VM-STOCKIQ-PUBLIC and net-to-net).

Is it possible to post the full configuration ? We might be able to spot something...

davidbnbf · ‎11-08-2017

I am hesitant to post the full configuration for security reasons.

here are the associated network objects for some of these NAT rules including working ones and the broken one. As you can see they are incredibly similar so I cannot figure out why the old ones work and the new NAT does not. Let me know if there are more pieces I can include to help from the config.

object network VM-STOCKIQ-PUBLIC
host 204.152.150.207
description VM-STOCKIQ-PUBLIC
object network VM-STOCKIQ
host 172.16.30.164

object network EXCHANGE-PUBLIC
host 204.152.150.209
object network VM-EXCHANGE
host 172.16.30.50
object network DNSDHCP-PUBLIC
host 204.152.150.208
object network VM-DNSDHCP
host 172.16.30.100

I tried to create more NATs using other free IPs both inside and out and it also resulted in failure. It seems I cannot create any new NATs successfully other than the ones that exist already.. I am stumped...

Jon Marshall · ‎11-08-2017

Have you allowed the access to the new server on the port you were referring to ?

If so then can you run this -

"packet-tracer input outside tcp 8.8.8.8 12345 <public IP of server> <port num>"

where <port num> is the port you are allowing through (assuming TCP).

Can you also run the same command against an existing server that works and we can compare.

Jon