Problem with ASA active/standby set-up after migrating to new ISP circuits

[mitchen]

We have an Active/Standby ASA5540 firewall set-up with the Primary Active unit at our head office site (Site A) and the Secondary Standby unit at our DR site (Site B)

Both sites had their "outside" interfaces directly connected to our ISP (We connect the ASA outside interface to the provider's NTE at each site)   This all seemed to work reasonably well - our active traffic would go through Site A and, in the event of a failure with Site A firewall or interface, comms would failover to Site B.

We recently decided to upgrade the bandwidth of our outside links to the ISP.  This meant getting completely new circuits installed and new NTEs but we requested that we keep the same IP Addressing for the new circuits (we have a number of VPN connections so didn't want to have to be changing configuration)

So, come time to move to the new circuits, we presumed it would just be a case of changing the interface speed on the ASA interface (from 10 to 100) and moving the cables across from old NTE to new NTE.  Meanwhile the ISP would activate the "new" ports on their network switch and shutdown the "old"      ports.   And this could be carried out relatively quickly to minimise any disruption.

However, this is not how it panned out.  It seems that when the ISP activates the new ports, Site B takes over as Active firewall and the Site A firewall has its outside interface marked as "failed"  - The ISP had to shutdown the Site B link in order to allow us to pass traffic through the Site A firewall and circuit again.  And we are left with the situation where we effectively DON'T have our Active/Standby set-up with automatic failover any longer!  We can either have Site A active and passing traffic and Site B marked as "failed" on its outside interface or vice versa.

I don't know too much about the ISP's set-up to be honest but, as far as I'm aware, the ISP connects both the circuits for Site A and Site B to the same network switch in their datacentre and to the same VLAN.

Can anyone suggest what the problem might be and how to resolve it?  I'm assuming it has to be something at the ISP end since I don't really understand what else could be necessary from our point of view (i.e. what else would we need to do other than move the cables and configure the new interface speed)?   Its as if there is some sort of conflict on the ISP's network switch - I don't know if it is something to do with the way the standby ASA takes over the active ASA IP and MAC address and that somehow gets the ISP network switch in a state of confusion?

Does anyone have any ideas/suggestions?  Naturally we are a bit disappointed since we hoped this would be a relatively straightforward task to migrate to our new circuits with increased bandwidth!

Thanks.

[mitchen]

Hi Barry,

Thanks for the input - I've been tied up with other things and all has gone quiet on the ISP front so haven't had the chance to chase this up with them since last week.

yes, our ASAs are definitely in routed (L3) mode though we are only running version 7.2(5)10 software on these particular firewalls.  (I don't know if there is any significance in that, regarding the bugs you mentioned, or whether they are specific to 8.2 releases)

Interesting info on the BPDU's in routed (L3) mode - can anyone confirm this in the documentation?   I had found the information previously on ASAs forwarding BPDUs but hadn't appreciated that this was only in transparent (L2) mode.

So, if we have our ASAs running in routed (L3) mode and we can therefore confirm that they will just discard BPDUs so can't possibly create the L2 loop then we can say with 100% certainty that the issue is with the ISP and the fact we are using ASAs to connect to them is of absolutely no significance? (although they haven't said it directly I get the impression they are trying to hide behind the fact we connect using ASAs)  i.e. they presumably have something connected incorrectly (simply cabled wrongly maybe?) and have created a loop on their network themselves? 

Am I correct in that assumption? (just want to be sure of my facts first!)   In which case, the ISP  need to investigate that rather than simply disabling STP? (which I must admit, their suggestion made me very nervous anyway!)

[Jouni Forss]

Hi,

Just out of interest, has the ISP been able to solve this issue for you? Has anything happened so far?

- Jouni

[mitchen]

Hi Jouni, unfortunately the issue is still outstanding. The ISP has conceded that they have an STP issue of some sort and are discussing with the carrier.  That's the last I heard and that was over a week ago but admittedly I havent chased up with them as I've been sidetracked with other things.

I will be sure to post back on the forum when/if we finally get it resolved though!

[mitchen]

Hi all,

Just thought I’d let you know, we finally got this issue resolved and both our new circuits are up and running with our ASAs in Active/Standby mode and capable of automatic failover again.  (Phew!)

The ISP weren’t particularly forthcoming with the details of what the issue actually was but I believe some sort of spanning tree issue between the ISP switch(es) and the carrier switch(es)  but, certainly, it was something at “their” end, as we all suspected, rather than anything with our ASA set-up.

Thanks to all for the input which undoubtedly helped provide evidence to the ISP that the fault was of their making not ours!  

[Jouni Forss]

Hi,

Glad to hear it got sorted out.

If nothing else this was probably a great learning expirience. Usually learn the most when trying to determine the cause of a problem and solve it.

- Jouni