Network migration

[Julio Garcia]

Hi all,

I have a large layer 2 network with quite a lot of vlans and hosts , they all aggregate to a pair of 3750s (stacked) which are doing the layer 3 routing and are the default gateway of all the vlans

i need to migrate from these 3750s onto new kit on which i will configure the ips that the old 3750 stacks used to have.

I dont really want too much downtime for this , assuming my old 3750 stack goes down , what is the average time hosts will  hold the old arp entries for the default gateways? the hosts are made up of quite a variety of things , mainly windows servers,    and can anyone think of a better of doing this than shutting down my old switches and reconfiguring my new kit to take the old role.

i thought of enabling hsrp , but i will still get same problem as above when i configure hsrp on the old 3750s.

many thanks

[Peter Paluch · Accepted solution]

Hello Rob,

Using the command clear arp-cache forces a Cisco router/multilayer switch forces it to clear the ARP cache on its interfaces (not needed for your purpose), and in addition, to send gratuitous ARP replies on all cleared Layer3 interfaces. This will enable the stations to refresh their ARP caches.

You may select a combination of these approaches:

1. Introduce the new 3750 without using HSRP and use the clear arp-cache command on the new switch to send gratuitous ARP replies
2. Configure HSRP on the old 3750 and use the clear arp-cache to make stations learn the new vIP/vMAC mapping. Then introduce the new 3750 using the HSRP and simply perform active/standby router switchover.

Relying on ARP cache expiration is not reliable - it can take minutes, and it depends very much on the operating system. I believe that Linux and Windows expire the entries after around 60 seconds.

Best regards,

Peter

[Julio Garcia]

Hi Peter,

i didnt know clear arp cache command sent a gratuitous arp aswell, thanks very much

[Jason Masker]

One way you could accomplish this with little to no downtime is with HSRP.

On your existing 3750 stack, configure your SVIs as follows:

interface vlan100

ip address 10.1.1.3 255.255.255.0

standby 1 ip 10.1.1.1 255.255.255.0

standby 1 timers msec 200 msec 750

standby 1 preempt delay minimum 180

The HSRP configuration is represented in bold.

The first line tells this interface to participate in the HSRP virtual address 10.1.1.3 (I am assuming you are using the .1 address for the gateway today.)

The second line is optional and is used to tune the HSRP timers for subsecond failover. This will be useful to you if you are running VOIP on your network or have some other high availability requirement.

The third line sets up the virtual address to preempt which I'll explain later.

You will want to do something similar for every SVI and you will want to paste the configuration in so that the switch very quickly changes the SVI address and then immediately sets it to participate in the virtual address. The reason you want to paste is to minimize the time that something could arp on the network while no interface is configured with the .1 gateway address. After it is configured when machines arp they will get the virtual HSRP mac address instead of the SVI mac address. This is ok because your SVI interface will respond to both its MAC and the HSRP MAC. In the meantime, if devices have still cached the SVI MAC the 3750 will have no problems servicing packets destined for the SVI MAC.

When you begin to configure the SVIs on your new switch stack, configure them as follows:

interface vlan100

ip address 10.1.1.2 255.255.255.0

standby 1 ip 10.1.1.1 255.255.255.0

standby 1 timers msec 200 msec 750

standby 1 priority 200

standby 1 preempt delay minimum 180

Again, I've bolded the HSRP configuration. Notice it is exactly the same except for the SVI address and the standby priority command.  This priority command tells HSRP that you prefer this SVI to the SVI on your old stack and because we have set the preempt directive, after 180 seconds the new device will begin responding to the HSRP mac address and your old 3750 stack will stop.  For this reason, you want to be sure your new device is completely configured before entering the HSRP configuration on it.

Once you have completed this for every SVI on the new device you have completed the migration. HSRP will automatically issue a gratuitous arp when the new device takes over because of the preempt directive which will update the arp entries of all devices on the network.

Here is the HSRP guide:

http://www.ciscosystems.com/en/US/docs/switches/lan/catalyst3750/software/release/12.2_35_se/configuration/guide/swhsrp.html

[Peter Paluch]

Jason,

You have provided a very nice explanation and a very well presented configuration examples. Good work!

I have a comment, though: You wrote that HSRP will send gratuitous ARPs because of the preempt directive. This is something I do not agree with. The HSRP essentially sends gratuitous ARPs whenever a router becomes active for a HSRP standby group. It does not matter whether the change to the active state took place because of preemption or because the current active router died and the hold time has expired.

Best regards,

Peter

[Jason Masker]

Agreed Peter. I guess I could have been more clear. I was trying to get the idea across that the final grat arp would be issued by the new devices when it preempts resulting in the desired blissful completion of the migration, but you are correct HSRP uses gratuitous arp whenever a router takes over regardless of the reason.  The only instance I was unsure of is when the standby commands are first issued and the HSRP virtual address is first created--I was not sure if a gratuitous arp would be sent out or not, but now that I think of it, this would have to be the case because the SVI would quickly realize it was the only participating member and 'take over' with a gratuitous arp.

Jason

[Peter Paluch]

Jason,

The only instance I was unsure of is when the standby commands are
first issued and the HSRP virtual address is first created--I was not
sure if a gratuitous arp would be sent out or not

Well, that was my original doubt as well That is why I recommended using the clear arp-cache in my first reply to get around this particular issue

Note that if the HSRP did not do the gratuitous ARPs when first coming up then we would have a problem here: currently, the gateway IP is bound to the real MAC. If you just configured the HSRP group having the same IP but a different vMAC, you would end up with the very problem Rob originally came here with: how to migrate the same IP from one MAC to another with the least downtime? I silently worked around it by the clear arp-cache but now it seems that the HSRP is smart enough. Still, the clear arp-cache will do no harm.

Best regards,

Peter

[Jason Masker]

Here too I would say we're ok because the SVI mac still exists and the SVI will still respond to requests sent to this mac address. For example, once you have HSRP set up, it matters not if you set your gateway to the SVI 10.1.1.2 address or the HSRP 10.1.1.1 address--in both cases the SVI interface will respond and correctly route traffic. So some end hosts may be confused and think that 10.1.1.1 is still the old SVI address, but the switch doesn't care. In fact, the switch has no idea the arp entry of the client is wrong. For all the switch knows, the client could have 10.1.1.2 set as its gateway for some reason.

[Peter Paluch]

Jason,

You are right and I am fully aware of all things you have described but I am pointing to a different issue: apparently, Rod did not want to modify the IP stack settings on any machine in the network. His concern was that without touching the configuration of the nodes on his network, he needs to swap two 3750 stacks (or standalone switches, doesn't matter) while retaining their IP addressing and not modifying the default gateway on the machines in the network. This would obviously pose a problem as either moving directly from one 3750 to another, or from 3750 to HSRP would constitute a change in the MAC address of the default gateway and temporary outage of internetwork connectivity until the new MAC address would be relearned that could take a minute or more without gratuitous ARP.

Best regards,

Peter

[Jason Masker]

Odd, I emailed in a response, but I guess it did not post..

Yeah, you're not understanding what I'm getting at. My config above
does not require a change in gateway nor does it cause a detectable
outage (should be at least as fast as HSRP itself.) Try it.

On the SVI interfaces configured as the gateway I suggested moving the
gateway address to an HSRP address and changing the SVI address. The
reason this does not cause an outage is because the SVI will route
both packets destined for it's SVI MAC and also packets destined for
the HSRP MAC. So it does not matter if devices have cached the old MAC
for 10.1.1.1 because that MAC is now 10.1.1.2 which still routes. As
the gateway IP is contained nowhere in the packet, from the SVI
perspective there is no way of telling if a packet was directed at the
SVI as gateway because of a stale arp entry or because 10.1.1.2 was
actually configured as the gateway--that's what I was getting at, the
SVI will happily route any packet directed at it. The reason I
recommended pasting the config is to eliminate the time where 10.1.1.1
is not owned by any interface because we want to reply in case
anything happens to arp for the address. There will be no outage and
no change necessary for the default gateway.

[Peter Paluch]

Hey Jason,

Now I see where you're going. Nicely done! Sorry - introducing that new IP came slightly out of blue and I was not quite sure (or aware) of your intent but it's clear now. Thanks!

Best regards,

Peter

[Julio Garcia]

Hi Jason,

thanks very much for your post, very detailed, i gave it 5 stars for effort --  i would of tried this out if it wasnt for the fact i have about 211 svis and i wanted to keep things a little bit simple  for this put of hours work , i think to  dish out the clear arp cache when the new hardware is changed is the most suitable solution for my situation

Many thanks

[Jason Masker]

No worries, Rob. I can understand not exerting more effort than the business requires. (Or more than they are willing to pay for. ) Maybe I'm a junkie, but I strive to never lose a packet every time. I like the challenge and it keeps me sharp.

As to how to handle hundreds of SVIs, I've dealt with that scenario before as well.  I would download a copy of Notepad++ (http://notepad-plus-plus.org/) and paste the SVI interface portion of the existing config into it, use regular expression find/replace to make the changes and paste it all back to the switch. After you spend a few hours familiarizing yourself with the ins and outs of regular expression, it's amazing just how fast you can manufacture configs. ETA on this change for 6 SVIs? About 15 minutes. ETA for 300 SVIs? About 15 minutes.

[Jason Masker]

Hey, gents.

If the prospect of rapid reconfiguration using regular expression peaked either of your interests at all, I just put up a document on how I used it for VLAN configuration in the UCS.  Forgive my zeal in being a bit of a regex evangelist, but this really saved me hours of punching in VLAN and vNIC template configs manually through the gui in our UCS build-up.

http://bit.ly/atVlPZ