Cisco 3750X Failed Stack Member Insertion - Config Back-flow

[CiscoMedMed]

We tried to insert a new stack member - the primary - into a stack. 

I wasn't there to see the precise order cabling and power on. But

my understanding is that racking the replacement primary

stack member and cabling it up as the previous primary 

stack member should allow it to join the stack as the primary. 

What ended up happening was i received a late call 

and nothing was working. Looking at the upstream router

sho cdp nei detail showed that the neighbor 3750X 

no longer had an IP address and showing VLAN 1. 

So it appears the new switch propagated its default

configuration to both stack members. To add insult

to injury the Cisco USB to serial adapter was so old

that it wasn't recognized by Windows 10 so I had

no way to recover the stack. As a work around 

I moved routing to the upstream ISR and at least

the remote site could work. 

Was there some other step needed to be performed
to make the new switch take on the configuration of

the existing stack rather than bestowing its 

default vlan 1 for everyone and little else? 

The stack did indeed form. But everything 

was VLAN 1. What do you think likely 

happened?

[Leo Laohoo]

1. Is the replacement switch loaded with the exact same IOS version as the rest of the stack? 
2. Does the replacement switch have the same LICENSE as the rest of the stack? 

NOTE:  If the License of the replacement switch is not the same as the rest of the stack, then the replacement switch will go into a non-stop boot-loop.

[CiscoMedMed]

I sent the show ver to the reseller and they put the same IOS. Show cdp nei detail
on the ISR confirmed that the IOS was identical. . We know the new switch didn't

go into a non-stop boot-loop because a stack was formed - albeit a default all

one vlan 1 stack with auto negotiation.

Thanks for the thoughts!

[Leo Laohoo]

Wait a second ... During the installation of the stacking cable, did the replacement switch have power when the stacking cable was attached?

[CiscoMedMed]

The instructions were for it to NOT have the power on when the stacking cables were plugged in. 

But I was 2500 miles away from where this actually happened. So it would seem like the most

likely scenario would be they turned it on and then plugged in the cables to end up with this

result. Would you agree? 

[Leo Laohoo · Accepted solution]

Well, that would explain why the replacement switch "took over":  The entire stack had an "election" and the replacement switch probably had the lowest hardware MAC address (among the stack) and became stack master.  

If the stack is reachable, post the complete output to the command "sh version" to determine if the entire stack has indeed rebooted.

Also post the command "sh switch detail" to determine if switch priority was configured (my guess is no and all switch members had a priority of "1").

[balaji.bandi · Accepted solution]

Looks like they have plugged in the stack cable while Switch is on, that is not recommended, can you tell me what is the status to assists here better.

did you lose the config? if not ask them to the power of the new switch was inserted. (if nothing was issued command wr any with new switch)

see if the second switch has config, take the config out of the box.

Step 1: Power up the switch and wait for it to boot.
* DO NOT attach any stacking cable.
Note, attaching a stack cable to a powered switch will cause the entire stack to reboot! 
Step 2: Renumber the new switch accordingly with the old (faulty) member switch
Step 3: Ensure that same IOS firmware as the master, 
Step 4: Unplug the new (replacement) switch from power; make sure it's OFF
Step 5: Plug the stack wise cable in.( make sure intact)

Step 6: Power on the new device. ( connect console cable to see if the switch boot normally) with out any ROMMON.
Step 7: The new (replacement) switch will boot and downloads its new firmware from the master if required and join the stack.
Step 8: The new (replacement) switch will upload the existing configuration from the master switch
Step 9: Check the new stack: #show switch

#show interfaces status

[CiscoMedMed]

This is a remote site and the tech who is hours away had all he could do to make sure all the phones and printers and computers were running before finally returning home at 3am. And he won't be able to make it back for at least another week. 

Based on show CDP nei and the fact that all ports on switch 1 and 2 are reachable from the inside interface of the ISR it is clear that the switch stack is now a flat VLAN 1 switch in default config more. We'll slated to replace the ISR with a Meraki MX in the next couple of weeks. At that time I'll have to try to recover the switch stack - at least put a few VLANs on it and authentication. Thank you much for the detailed reply. 

[Leo Laohoo]

Out of curiousity, what is the exact version of the switch. 
There is still a way for the router to push a config into the switch.  If the switch is running the vulnerable version then someone will need to reboot the switch and ZeroTouch will do the rest.

[CiscoMedMed]

From show cdp nei det:

Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(2)SE6, RELEASE SOFTWARE (fc2)

I even see the switch grabbed an IP address and default route from the inside interface of the ISR. But alas, I can't SSH nor Telnet to is - probably because line vty has not been configured. That'd be my guess. So close.

[Leo Laohoo]

15.0(2)SE6 -- Oh good.  That's a vulnerable version -- ZeroTouch will absolutely work.  The router can be the "vstack master".