Getting redundancy, but between 2 separate switches

[WallaceVanDunk1494]

So I'm trying to think of a way to do this and have been messing around in gns3 a bit but I figured I'd post here for ideas while I putz around.  

2 internet routers that connect to 2 different ISP's.  They share the BGP tables.  Right now, those go into a layer 2 switch stack, which then goes to a firewall.  From the firewall, it goes back to the layer 2 switch stack, and then into the inside network.  So this is weird and messy and not ideal obviously.

So I'm replacing the layer 2 switch stack with 2, separate, 9300 switches.  The idea being that I want to be able to upgrade 1 of these 9300's without losing internet connectivity, and I'm trying to figure out the best way to do that.

So in GNS3 I setup 2 layer 3 switches, each with an SVI in the same subnet, call it 10.1.1.3 on one switch for the SVI and 10.1.1.4 on the other.  I connected one of the switches to a router, and the router's interface was 10.1.1.1.  Then I setup HSRP on both SVI's and trunked the vlan between the 2 layer 3 switches.  HSRP came up, all good.  Then I connected a port on the one layer 3 switch to the router interface as an access port.  The router could then ping 10.10.1.3, but it couldn't ping 10.10.1.2, which is the virtual IP I'm using for HSRP.  

So I thought I'd try a sub interface on the router, and change the access port on the L3 switch to a trunk....and same thing.  I could ping 10.1.1.3, but not the HSRP IP...at least not without a static route to 10.1.1.2 that pointed to 10.1.1.3.

So I'm just messing around, trying to figure out a way to do this.  There's only 1 available port on each of the ISP routers, and the switches don't plan to be stacked at least not currently, because again, I want to be able to update the switches one at a time without losing internet access.  So one cable is going to have to run from 1 ISP router to 1 9300, and another cable from the second ISP router to the other 9300.

How would you guys go about achieving redundancy with this type of setup?  I'm thinking maybe a /30 between the ISP routers and the new 9300's and then some IGP to keep the outbound connectivity?

[balaji.bandi]

what is the existing arrangement with your FW, that is a very key part here. do you run any IGP between uplink Routers and FW ?

how is your traffic Load-balancing between ISP, PBR, or LB between ISP round robin?

Where is your NAT take place ? on FW?

Make a small network diagram for your understanding, so you get more idea of a cutover plan, and also you can view more failure scenarios.

I can understand your future maintenance part of 2 X 9300 stack, so you can patcc job will be easy rather downtime.

but why do you need stacking on each side if only 1 cable going to each stack, 1 x 9300 is good enough for each side as standalone- if only layer2 between Router and FW. (correct me do you have any other device connected here).

[WallaceVanDunk1494]

No other device.  The edge routers connect to the switch on a separate VLAN for each to differentiate between the 2 ISP connections and the edge routers have an OSPF link between the 2 of them.  Most of the traffic goes in and comes out a single ISP and a fractional amount is routed over the other.  

So say the IP we get from one edge router is 10.1.1.1...that connects to VLAN 10 for example, which then connects to a L3 interface on the firewall of 10.1.1.2, same for the other edge router, just a different VLAN.  NAT is done on the firewall.  I'm building the topology out in GNS3 but I haven't made it far enough to get to the firewall.  I'm still trying to get some kind of failover/redundancy scenario to work between the edge routers and the switches.  

[balaji.bandi]

Ok nice Life is easy with a simple setup, So you have Failover in Place if ISP1 goes down you shift the traffic to ISP2, and vice versa?

On the FW they are 2  L3 Interfaces connected to each segment? what FW is this ASA  or CP  or Palo?

if they are separated and each segment, you have 1 x 9300 on each side just extend the network and remove old switch.

  I'm still trying to get some kind of failover/redundancy scenario to work between the edge routers and the switches.  

If the switch is Pure Layer 2, what kind of redundancy you expecting here. 

[WallaceVanDunk1494]

Yes if ISP1 goes down all the traffic goes out and comes in ISP2.  FW is palo.  Yes there are 2 layer 3 interfaces on the FW that connect to each segment.  The current switch is just a passthrough, but it's 2 switches stacked together.  The 9300's are layer 3 capable.  What I'm trying to do is replace the current stack of 2 layer 2 switches, with the 9300's.  Basically there will be 2 9300's for the outside connection, that is that they will connect to the ISP's, and 2 9300's for the inside connection.  There are no plans to stack the 9300's.

What I want to do is have it setup so that I can update any singular 9300 without losing internet connectivity.  

[balaji.bandi]

As long as you are not changing Layer 3 hop your task is simple. you are splitting the stack in to 2 standalone devices.(with 9300).

You Migration plan is easy here (correct me if any deviations)

1. Prepare both Cat 9300 with respected VLAN connecting to respected ISP

2. bring the Devices online

3. Move 1 Link at a time to cat 9300, so another link still is available forr the internet.

[WallaceVanDunk1494]

Sorry, I don't see how that provides the redundancy I'm talking about.  I'm trying to figure out a way to do it so that both outbound and inbound paths to both ISP's remain up.  If one of the ISP paths goes down, then all of the BGP shifts to the other router, which takes time and causes a "blip".  I'm trying to do this so there's no blip.

[balaji.bandi]

As per the Orginal post - 

1. You have BGP peering with 2 routers connected outside.

2. you looking to change the old switch to a new switch? - this where i was focussing.

3., if this Layer 2 you can not do much here. (this is standard setup)

I have advice only replacing the Switch.

Looks now the discussion going interesting side - as I suggested in the original, Make a small Diagram  - are you aiming to change your routing topology? Do you have HSPR?  How is Palo failover between ISP? 

There are other ways to track and reduce the timers, that only can be advised by seeing some configuraitons in place. ( as per original the post you already mechanism Failover in place) "Yes if ISP1 goes down all the traffic goes out and comes in ISP2."

[WallaceVanDunk1494]

My concern isn't the failover between the routers...maybe I'm not being clear.  The routers are going to connect to the 9300s, which will then connect to the firewall...what I want is to be able to update or reboot either 9300 and keep internet connectivity without any blips.  This has nothing to do with failover on the routers themselves or BGP.  

One edge router connects to one 9300.  The other to another 9300.  Both 9300s connect to the palo.  I need to be able to pull out, shut off, reboot, take down a single 9300, without any hinderance to internet connectivity.  

Here's a rough drawing of the intended topology:

[Jon Marshall]

Have read through the post and still not entirely sure I follow. 

Is the basic aim that if you reboot the switch the primary ISP is connected to then you still use the primary ISP for internet traffic ie. the path does not failover to the other ISP ? 

I am not sure what blip you are trying to avoid or how your routing currently works. 

I suspect you may be able to do something IBGP but don't want to confuse the issue if I have not understood the problem. 

Jon

[balaji.bandi]

Much better - Looks like your Palo Alto FW doing all the work for you.

Switch is just Layer 2 here - when you do maintenance of  any switch the BGP peer will  not go down, since it is connected to Router outside.

Router do not have any visibility (for now) until you have any track machanism in place to trace inside interface go down or FW IP not reachable and bring down the BGP peer arrangements in place.

Stil we do not have 1 clarity is PALO and Router ( static Routing or any IGP  running between those ?)

Rather make more engineering complicated setup as it is running (may be we can do better by changing design - but at this stage i will not advised or requirement i see here, since you have all in place).

When you doing each side switch upgrade, Suggest tot  shutdown Router interface to bring down the BGP or Do the Traffic engineer at Palo side to send traffic to active ISP, same way you do other side maintenance of the swtich, that is the reason we always make resiliance setup so maintenance will be easy ?

is this make sense ?