Procedures for keeping cold spare equipment

[ColForbin]

---

I have what appears on one level to be a basic question. I was tasked with coming up with a plan to support 2 remote sites that don’t have IT people. Each site has an ISR4400 series and a 9200 switch stack with 2 switches. There is a spare 4400 and a spare 9200 at each site, boxed up in storage. The configs rarely change but they do occasionally. They want someone to make a trip out to each and drop the current config on the spares. The idea being if any of the current devices die someone with minimal experience could swap them out. 
This is not about whether or not it’s a good idea lol

The configs include AAA and radsec is configured. On the production router obviously I can do a show run to a text file and paste it in. Or export a copy and go the config replace route. But what I’m not clear on is what happens with the certs, ssh keys, and essentially anything along those lines that may be an issue moving a config from one device to another. 
I’m also wondering the same with the switch but there’s the added difficulty of not knowing which switch in the stack dies, 1 or 2, and how that would be handled. 

Any thoughts or references would be greatly appreciated. Thank you

[Joseph W. Doherty · Accepted solution]

On the switch side, add the 3rd switch to the stack.  Setup, if used, multiple VLANs, "vertically".  I.e. a column of ports, across the stack, are same VLAN.  In any column, always keep one stack's worth of ports available.  So, if any stack member fails, all you need to do is repatch failed switch member ports to another available stack member's ports.

From my experience, you can often talk anyone through something like move cable from middle switch to bottom switch, port 1 to port 1, port 2 to port 2, etc.

The stack, keeps all stack members synced.

For the router, it too could be placed on-line, for remote configuration updates, but ideally, if possibly, perhaps using VPN across an inexpensive Internet connection so remote access isn't totally lost (although remote performance perhaps degraded).

One company I worked at had about 80 remote sites, most without on-site IT support, but company wanted to avoid or minimize any remote network outages at that would often very much impact work of branch users (especially as corporate servers were at HQ).

[ColForbin]

Thanks fellas this is all some fantastic thinking. Much appreciated. Been furloughed so just getting back to stuff

[Leo Laohoo]

Keeping spares in boxes is a waste of time because there is no guarantee that when they are needed they will be fine during the bootup.  I have received several RMA units direct from Cisco arrived DoA.  

Switches are easy because if they are spare, just make sure they are part of the stack because the firmware and the config are synchronized all the time.

Routers will be difficult but that is what the Management port is for, i.  e.  Copy the config from the production router, change the IP address of the Management port and upload it into the secondary/standby router.  

[Joseph W. Doherty]

---

@Leo Laohoo wrote:

Routers will be difficult but that is what the Management port is for, i.  e.  Copy the config from the production router, change the IP address of the Management port and upload it into the secondary/standby router.  

---

Yup, you can use the management port, but how's that accessed, via some dedicated telcom link, supporting 9.6Kbps?  Again, for the price of such, you might be able to have a multi megabit ISP connection, which, besides being used for remote access to the router, is possibly fast enough to support users.  I.e. possibly no full user outage at all (assuming fast dynamic fail over).

For one company I worked at, we had a dedicated primary router, using a dedicated commercial WAN/MAN connection, and a "backup" router using an Internet VPN connection.

The "backup" VPN, was originally planned to just be a warm backup, but then it was suggested it might be used to carry low priority traffic (also removing such traffic from the "expensive" service provider WAN/MAN link).

However, I discovered, if you shaped the VPN for the available end-to-end bandwidth, and you didn't share it with general Internet traffic (i.e. only used it for VPN traffic), it usually performed as well as the "expensive" service provider link (of like bandwidth).

So, if you have the option of bringing in an "inexpensive" Internet connection, to connect to your "backup" router, it's so much "better", starting with, possibly, no user outage at all if the primary router, or its link, fails.

BTW, with the "spare" switch in the stack, you have a couple of additional options.  For one, rather than connecting to two of your 3 switches (which if one of the two fails, you lose up to half your local network), you can distribute connections across the 3 switches (you'll only lose 1/3 of your local network, of course if you only connect to two switches, you have a 1/3 chance that the switch that fails is the spare).  (If you use all 3 switches, again, remember to leave 1/3 ports available in the column.)

For another option, certain hosts that are "critical" might have a second jack, at the host location, wired to a different stack switch member.  I.e. someone at the host can immediately repatch or use LACP or multi-home.

Again, at the one company I worked at, the general policy was a network failure shouldn't take down more one stack switch member, chassis line card or FEX ports' number of hosts.

If you're going to have cold spares sitting at sites in boxes, you might as well save the capital cost and use the 4 hour replacement option.  As you don't have IT support at the sites anyway, you could work to arrange that too, maybe to coincide with the 4 hour replacement.

However, as you already have the spare router and switch, leverage them.  I assume having them is to reduce network outage time, from multiple hours.  If so, why have an outage you can avoid or just limit to a few minutes?

[balaji.bandi]

Sure, I agree that some critical sites need this kind of arrangement due to factors like travel and minimal downtime.

Service restoration is important, Certs (local one have already,) ssh also have already in the switch.

you may need to more focus on the Operational configuration when the device failed and engineer able to replace and you have management access, rest all can be fixed soon you have access.

backup of the configuration is key here (if you are changing regularly on the device)

if one time setup it easy of replication of configuration, remove failed device and swap with standby one.

Note - this can be argued with different views, some time this is very helpfull, if you have budget and needed.

Cisco do have contract 4 hours replacement, but does not carry any config.

[pieterh]

currently many Cisco devices support cloud-based PnP 
if the device at least has internet access, it can register itself at a Cisco could service from where you can remotely configure it,
just like a local DNAC deployment.