09-25-2020 05:49 PM - edited 09-25-2020 06:23 PM
We tried to insert a new stack member - the primary - into a stack.
I wasn't there to see the precise order cabling and power on. But
my understanding is that racking the replacement primary
stack member and cabling it up as the previous primary
stack member should allow it to join the stack as the primary.
What ended up happening was i received a late call
and nothing was working. Looking at the upstream router
sho cdp nei detail showed that the neighbor 3750X
no longer had an IP address and showing VLAN 1.
So it appears the new switch propagated its default
configuration to both stack members. To add insult
to injury the Cisco USB to serial adapter was so old
that it wasn't recognized by Windows 10 so I had
no way to recover the stack. As a work around
I moved routing to the upstream ISR and at least
the remote site could work.
Was there some other step needed to be performed
to make the new switch take on the configuration of
the existing stack rather than bestowing its
default vlan 1 for everyone and little else?
The stack did indeed form. But everything
was VLAN 1. What do you think likely
happened?
Solved! Go to Solution.
09-25-2020 11:40 PM
Well, that would explain why the replacement switch "took over": The entire stack had an "election" and the replacement switch probably had the lowest hardware MAC address (among the stack) and became stack master.
If the stack is reachable, post the complete output to the command "sh version" to determine if the entire stack has indeed rebooted.
Also post the command "sh switch detail" to determine if switch priority was configured (my guess is no and all switch members had a priority of "1").
09-26-2020 12:25 AM - edited 09-26-2020 12:29 AM
Looks like they have plugged in the stack cable while Switch is on, that is not recommended, can you tell me what is the status to assists here better.
did you lose the config? if not ask them to the power of the new switch was inserted. (if nothing was issued command wr any with new switch)
see if the second switch has config, take the config out of the box.
Step 1: Power up the switch and wait for it to boot.
* DO NOT attach any stacking cable.
Note, attaching a stack cable to a powered switch will cause the entire stack to reboot!
Step 2: Renumber the new switch accordingly with the old (faulty) member switch
Step 3: Ensure that same IOS firmware as the master,
Step 4: Unplug the new (replacement) switch from power; make sure it's OFF
Step 5: Plug the stack wise cable in.( make sure intact)
Step 6: Power on the new device. ( connect console cable to see if the switch boot normally) with out any ROMMON.
Step 7: The new (replacement) switch will boot and downloads its new firmware from the master if required and join the stack.
Step 8: The new (replacement) switch will upload the existing configuration from the master switch
Step 9: Check the new stack: #show switch
#show interfaces status
09-25-2020 07:01 PM
NOTE: If the License of the replacement switch is not the same as the rest of the stack, then the replacement switch will go into a non-stop boot-loop.
09-25-2020 07:44 PM - edited 09-25-2020 07:57 PM
I sent the show ver to the reseller and they put the same IOS. Show cdp nei detail
on the ISR confirmed that the IOS was identical. . We know the new switch didn't
go into a non-stop boot-loop because a stack was formed - albeit a default all
one vlan 1 stack with auto negotiation.
Thanks for the thoughts!
09-25-2020 08:07 PM
Wait a second ... During the installation of the stacking cable, did the replacement switch have power when the stacking cable was attached?
09-25-2020 10:19 PM
The instructions were for it to NOT have the power on when the stacking cables were plugged in.
But I was 2500 miles away from where this actually happened. So it would seem like the most
likely scenario would be they turned it on and then plugged in the cables to end up with this
result. Would you agree?
09-25-2020 11:40 PM
Well, that would explain why the replacement switch "took over": The entire stack had an "election" and the replacement switch probably had the lowest hardware MAC address (among the stack) and became stack master.
If the stack is reachable, post the complete output to the command "sh version" to determine if the entire stack has indeed rebooted.
Also post the command "sh switch detail" to determine if switch priority was configured (my guess is no and all switch members had a priority of "1").
09-26-2020 12:25 AM - edited 09-26-2020 12:29 AM
Looks like they have plugged in the stack cable while Switch is on, that is not recommended, can you tell me what is the status to assists here better.
did you lose the config? if not ask them to the power of the new switch was inserted. (if nothing was issued command wr any with new switch)
see if the second switch has config, take the config out of the box.
Step 1: Power up the switch and wait for it to boot.
* DO NOT attach any stacking cable.
Note, attaching a stack cable to a powered switch will cause the entire stack to reboot!
Step 2: Renumber the new switch accordingly with the old (faulty) member switch
Step 3: Ensure that same IOS firmware as the master,
Step 4: Unplug the new (replacement) switch from power; make sure it's OFF
Step 5: Plug the stack wise cable in.( make sure intact)
Step 6: Power on the new device. ( connect console cable to see if the switch boot normally) with out any ROMMON.
Step 7: The new (replacement) switch will boot and downloads its new firmware from the master if required and join the stack.
Step 8: The new (replacement) switch will upload the existing configuration from the master switch
Step 9: Check the new stack: #show switch
#show interfaces status
09-28-2020 07:16 AM
This is a remote site and the tech who is hours away had all he could do to make sure all the phones and printers and computers were running before finally returning home at 3am.
Based on show CDP nei and the fact that all ports on switch 1 and 2 are reachable from the inside interface of the ISR it is clear that the switch stack is now a flat VLAN 1 switch in default config more. We'll slated to replace the ISR with a Meraki MX in the next couple of weeks. At that time I'll have to try to recover the switch stack - at least put a few VLANs on it and authentication. Thank you much for the detailed reply.
09-28-2020 03:24 PM
Out of curiousity, what is the exact version of the switch.
There is still a way for the router to push a config into the switch. If the switch is running the vulnerable version then someone will need to reboot the switch and ZeroTouch will do the rest.
09-28-2020 03:40 PM
From show cdp nei det:
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(2)SE6, RELEASE SOFTWARE (fc2)
I even see the switch grabbed an IP address and default route from the inside interface of the ISR. But alas, I can't SSH nor Telnet to is - probably because line vty has not been configured. That'd be my guess. So close.
09-28-2020 05:44 PM
15.0(2)SE6 -- Oh good. That's a vulnerable version -- ZeroTouch will absolutely work. The router can be the "vstack master".
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide