BGP Failover didn't fail back? Help!

[davidbnbf]

We have a netblock of IPs that are configured for BGP between 2 ISPs for failover and redundancy.  We had an issue with our primary site ISP and brought it back online.  Everything seemed fine and existing IPs and NATs are working properly for the primary site.  However I tried to set up a new NAT at that site with another public IP that should be tied to this ISP and I can't get any reply packets back from the internet.  Our theory is that the IP is tied to the secondary ISP for some reason during a BGP failover and never came back.

Unfortunately the person who configured BGP is long gone and I am not network expert.  I know enough to be dangerous :)

Can someone help me figure out how to reset the BGP so that the IPs are associated with the primary ISP instead of the secondary?  How can I restore this?  It was my understanding this configuration was supposed to be pretty automatically but something has clearly gone awry.

[davidbnbf]

---

@Jon Marshall wrote:

Have you allowed the access to the new server on the port you were referring to ? 

If so then can you run this - 

 "packet-tracer input outside tcp 8.8.8.8 12345 <public IP of server> <port num>"

where <port num> is the port you are allowing through (assuming TCP). 

Can you also run the same command against an existing server that works and we can compare. 

Jon

---

I can do that.  I have not even gotten that far yet.  I am on the host device and when I turn on the NAT I immediately lose the ability to get to the internet.  I haven't configured any inbound ACL yet.  My expectation is that as soon as I configure the NAT I should be able to go to www.whatismyip.com on the host and see the public IP listed like it does on all other servers we configure static NATs for.

[Jon Marshall]

Can you describe the setup ie. is it - 

internal -> ASA -> switch -> router -> internet 

if so you mentioned that the last hop you saw was on the switch which is unusual ie. I would have thought the switch would be L2 only but if it is an SVI on the switch then it sounds like it is acting as L3 switch. 

Can you confirm the setup ?

Jon

[davidbnbf]

This is correct: internal -> ASA -> switch -> router -> internet
Switch is layer 2.
There is a 2911 router. The IP is for a vlan interface that handles the port channel between the BGP router and BGP switch.

[Jon Marshall]

Does the ASA outside interface have an IP from the 204.152.150.0/24 subnet ? 

Jon

[davidbnbf]

Yes. ASA outside is 204.152.150.249/28

[Jon Marshall]

Okay it is definitely a /28 not a /24 ? 

If so what is the mask on the router interface that connects to the BGP switch ? 

If it is a /28 also can you see if you have static routes configured on the router for any of your public IPs pointing to the firewall IP ie. on your 2911 - 

"sh ip route static"

Jon

[davidbnbf]

It is /28 on the ASA.

kkbgprtr01#sh ip route static
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is 162.211.2.65 to network 0.0.0.0

66.0.0.0/32 is subnetted, 1 subnets
S 66.195.152.102 [1/0] via 162.211.2.65

[Jon Marshall]

Do you mean it is a /28 on the router ? 

If so then the IPs you are using for your NAT are not part of that subnet ie. 204.152.150.240/28 so how does the router know where to send the traffic. 

For one of the public IPs that works, on the 2911 - 

"sh ip route <public IP>" 

then same again for the public IP that is not working. 

Jon

[davidbnbf]

No the outside interface on the ASA is the /28.

.209 works. .208 works. .207 does not.

kkbgprtr01#sh ip route 204.152.150.209
Routing entry for 204.152.150.0/24
Known via "ospf 1", distance 110, metric 20
Tag 666, type extern 2, forward metric 2
Advertised by bgp 27542
Last update from 204.152.150.223 on Port-channel1.663, 2w0d ago
Routing Descriptor Blocks:
* 204.152.150.223, from 204.152.150.2, 2w0d ago, via Port-channel1.663
Route metric is 20, traffic share count is 1
Route tag 666
kkbgprtr01#sh ip route 204.152.150.208
Routing entry for 204.152.150.0/24
Known via "ospf 1", distance 110, metric 20
Tag 666, type extern 2, forward metric 2
Advertised by bgp 27542
Last update from 204.152.150.223 on Port-channel1.663, 2w0d ago
Routing Descriptor Blocks:
* 204.152.150.223, from 204.152.150.2, 2w0d ago, via Port-channel1.663
Route metric is 20, traffic share count is 1
Route tag 666



NOT WORKING:
kkbgprtr01#sh ip route 204.152.150.207
Routing entry for 204.152.150.0/24
Known via "ospf 1", distance 110, metric 20
Tag 666, type extern 2, forward metric 2
Advertised by bgp 27542
Last update from 204.152.150.223 on Port-channel1.663, 2w0d ago
Routing Descriptor Blocks:
* 204.152.150.223, from 204.152.150.2, 2w0d ago, via Port-channel1.663
Route metric is 20, traffic share count is 1
Route tag 666

[Jon Marshall]

We need to understand your layout in more detail ie. a traceroute to a working IP (last 3 hops) - 

13.|-- 162.211.2.66 0.0% 4 85.7 85.7 85.6 85.9 0.0
14.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0
15.|-- 204.152.150.209 0.0% 4 87.6 87.7 87.4 88.0 0.0

13 is your router, 14 your firewall (presumably) and 15 the host 

traceroute to 204.152.150.207 - 

13.|-- 162.211.2.66 0.0% 4 90.9 91.0 90.9 91.1 0.0
14.|-- 204.152.150.223 0.0% 4 91.7 92.8 91.7 93.6 0.6
15.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0

13 is your router, 14 is ? and 15 is presumably firewall ? 

Jon

[davidbnbf]

Correct. Work hop 14 is firewall.

Not working hop 14 is the vlan interface of the port channel that connects the router to the switch outside the firewall.

[Jon Marshall]

What is the subnet mask on the interface on the 2911 that connects to the switch ? 

Also you say the vlan interface for the port channel, where is this vlan interface, it cannot be on the switch because that is L2 not L3 switch. 

Jon

[Georg Pauwen]

Hello,

try and add 'no-proxy-arp' to the entry:

object network VM-STOCKIQ

nat (any,any) static VM-STOCKIQ-PUBLIC net-to-net no-proxy-arp

[davidbnbf]

---

@Georg Pauwen wrote:

Hello,

try and add 'no-proxy-arp' to the entry:

object network VM-STOCKIQ

nat (any,any) static VM-STOCKIQ-PUBLIC net-to-net no-proxy-arp

---

Did not help unfortunately.

[Georg Pauwen]

Hello,

not sure if this has already been mentioned in this thread, but which block have you actually been given ? Did they give you 204.152.150.0/28 ?