Need to Transition to a new core Cisco L3 switch, but need smooth transition. Possible, with no down time?

[StillMaKiniT]

I'm replacing our old Cisco L3 core switch with a newer one.  As the outfit I'm involved with can't have ANY down time, I'm looking for a smooth transitional method involving little or none.  I'm not as concerned with the other branch switches it interconnects everything with, as I am with the ASA 5525 its using to connect to the WAN side of things.  Normally I'd just do an after hours swap: confirm environment compatibility and the config's, flush the ARP and then move on.  But keeping the other one connected while activating the new one with similar IP structures and other goodies makes me nervous to say the least.  There are numerous VLAN's, Trunks and potential IP issues that could cause problems. I do have an available spare port on the ASA.  I was considering connecting the new Cisco switch while leaving the one it's replacing still servicing the environment. Is it possible to do this and eventually ween off of the old switch even with the similar configuration between them?  I'm concerned about IP conflicts and other unknowns.         

[Martin L]

First of all I think we need more info; what are sw models, what stuff you have running (protocols/services), what and how many devices are connected, etc.

It seems swapping old with new switch will be easier and less risky comparing to migration from old to new. You are correct about migration and IP conflicts and problems; you will have to assign different IP to new sw.

What amount of down time is acceptable?

Regards, ML
**Please Rate All Helpful Responses **

[Georg Pauwen]

Hello,

you might want to have a look at Nonstop Forwarding with Stateful Switchover (NSF/SSO), which reduces downtime to (mili)seconds. I agree that we need to get more details on what model switches you have, and what IOS versions you are running, as this is crucial to recommending any features...

https://www.cisco.com/c/en/us/td/docs/solutions/Verticals/EttF/EttFDIG/ch6_EttF.html

[paul driver]

Hello
Even though you haven’t elaborated on what type of L3 core you are presently running ( Firewall HA, Stackwise, VSS, VPC ) the concept of migration can be discussed.

You have basically from my experience two options of migration:
Straight forward all in one migration or a staged migration, both I have found have benefits and caveats

The all in change will require you to build the new core to the same prerequisite of the exiting core and at time of migration relocate all the L2/l3 interconnects at the same time, the benefit of this would be you have the old core still active and a known working backout solution, The down side is you once completed you’ll need the client to post-test their services and if for some reason something doesn’t work and it cannot be resolved then you would most probably have to relocate back all the interconnects/routing at the same time which is very time consuming in terms of outage to the client.
This option would be the only viable option if you didn’t have the rack space to have the old/new core able to running along side each other and low on budget in terms of man hrs etc.

The other option would to stage the migration, This will require you initially setup L2 interconnects between old/new core to allow traffic to flow between them, Once this is in place you can slowly migrate off the L2 distribution/access switches over to the new core whilst the old core would be still running the L3 routing.

With the L3 routing change you have lots of options but it would depend on what routing process you have running (static routing or an IGP) but envisage a bit of downtime/service interruption whilst your routing is changed.

Again, you could go with a straight forward change, like for like, shutting down all l3 of the old core and activating on the new core relocating any wan connection, static routing etc… or you could stage this part but then you would require a L3 interconnection between old and new core and advertise the addressing like any FHRP into the IGP or changing the static routing then lastly relocating any wan connection and disabling the routing on the old core.

The staged migration is best for the client as then they have time to review each change before you can carry on with the next part of the migration, the downside is it takes more time to complete and a lot more hrs booked to the project

You do have options and I have found a Lan core migration is all about the planning and knowing the steps you wish to take and being open to the client with this in mind you should be fine  whichever path you take.

[StillMaKiniT]

I apologize for not replying sooner and thanking all of you and providing more details. There was a delay due to having 3 heart attacks and 2 strokes in one night, and getting treatment. All I know is that I returned to an absolute MESS with regard to the networks condition, in terms of no updated documentation and an environment that is within an inch of bricking itself. Essentially, I have a Cisco Catalyst 3850 that needs to replace the current core switch and we have a VERY tenuous situation with this network.  I will get the details and post and any assistance would be greatly appreciated.

[Georg Pauwen]

Sounds like you are working in a health care environment ? Good luck with your efforts. It would help if you could post the running configuration (sh run) of your current core switch, that would make it a lot easier to recommend any sort of strategy...

[paul driver]

Hello
Futher information will equate to what sort of core you runing at present, be it a standalone core, dual core with first hop redundency,Stackwise, VSS etc...the kind of routing (static,IGP)

Having just re-read your OP you have only 1 spare port on the asa to play with and I assume with this you would probably want to keep the ASA for the secuirty policys and wan connection and just migrate the inter-vlan routing off onto the new 3850?

[meisemann7]

Paul,

I just wanted to say, I found your post above helpful as I am faced with a similar issue right now but must be doing something wrong.  Both my L3 switches have similar configs, but when I get to where I enable L3 routing on the new core, I start getting odd results.  ARP incomplete entries and only certain VLANs work.  I am assuming the 'ip default-gateway' command needs to be removed and all SVIs need to have the same IPv4s as the old L3 core, right?  All my routing is static and going to 1 physical 'no switchport' interface.  Currently, it is working with the new L3 core as an L2 switch (no IP routing) and all traffic trunked into the old L3 core, which does the routing.  I just can not seem to get past this step.  Any thoughts are helpful.  

[paul driver]

Hello
Okay you have L2 interconnectivity and you have already tried to migrate the L3 unsuccessfully.

The Arp incomplete could be resolved by appending the “old core” SVI arp addressing to the “new core” SVI’s however even without doing this reconvergence should work as/when arp is flushed and relearned (manually if need be)
Are you using a First Hop redundancy?

When you enable L3 on the new core you should have L3 routing running and all L3 SVI’s active, The SVIs on the old core need to disabled and routing needs to be turned off (no ip routing), this will delete any static routes on the old core and make it a host switch, it will then just require a default-gateway for mgt purposes (if you don’t do this correctly you will begin to receive duplicate ip address conflicts as both old/new cores will see either others ip addressing.

[meisemann7]

Paul,

I don't think we are using FHRP or First Hop Redundancy. Should I use FHRP in this situation when replacing an old L3 switch? Also, when you talk about manually relearning ARP entries, does that manual command need to execute on the L2 access switches hanging off the new L3 switch or just on the L3 switch? See Drawing1 and let me know if there are other suggestions/ things to check that might cause incomplete ARPs beyond what you have mentioned.  My DNS servers are on the other side of the virtual router but I have a static route of 0.0.0.0 sending all traffic I think to the virtual router so... does it take a while for things get 'relearned'?

[Giuseppe Larosa]

Hello @meisemann7 ,

when changing the L3 core switch and using the new one it happens that all MAC addresses related to SVIs are different.

So a clear arp on each device for its own default gateway would be needed.

As a way to make it easier you could enforce on new L3 switch SVI interfaces the SAME MAC address used on the old

So first on the old one switch you collect

show int vlan x | inc address

Most of cases all SVIs use the same base MAC address in a switch .Let's say it is AAAA.BBBB.CCCC

on the new switch you can configure the MAC to be manually set with

int vlan x

mac   AAAA.BBBB.CCCC

for each VLAN x of interest.

This way the change can be smoother as no device needs to re-ARP for its default gateway in each client VLAN.

For the routed uplink interface this should not be necessary as you need to unplug it to move it on the new switch.

Hope to help

Giuseppe

[meisemann7]

Thanks Giuseppe Larosa,  I will try to issue the first and/or maybe hardcode the MACs in the 'int vlan x' I appreciate it.

[meisemann7]

Sorry for the delayed response. One quick question of clarification, when you say, "So a clear ARP on each device for its own default gateway would be needed." Are you referring to the downstream or access (L2) switches I have connected to the new L3 switch, all those need a manual 'clear arp' or similar flush command. Sorry for my ignorance, this is the first time I am replacing a production L3 switch on my own so any helpful hints/troubleshooting thoughts are helpful.  Thanks.

[Giuseppe Larosa]

Hello @meisemann7 ,

if you don't "simulate" the MAC address(es) of the old L3 switch on the new one, all devices on all VLANs will have to learn the new MAC address of their default gateway not only the downstream L2 access switches also end user devices.

Hope to help

Giuseppe

[meisemann7]

Makes sense, and probably why my downstream L2 access switches can't reach the 'default gateway' when I transfer routing to the new L3 switch, cause the MAC isn't matching.  How long does the 'learning' process take usually if I didn't 'simulate' it.  Just curious.

Thanks, you have been a great help.