A new added member at stack 3650 stays in „Initializing state” permanently.

[rico.nego.rico.nego]

First of all I have four stack and I noticed the same problem with all of them when I’m adding a new member to stack. The scenario of action is as below.

1. There is stack with for example 5 members.
2. Adding a new member with the same software version and license
3. On the console I can see logs as below.

Interface GE 0 link down***ERROR: PHY link is down Getting rest of image Reading full image into memory...Check base package header ...: done = 16384 Getting rest of image Reading full image into memory....done Reading full base package into memory...: done = 22171522 Bundle Image --------------------------------------
Kernel Address : 0x5342f384 Kernel Size : 0x34e70e/3467022
Initramfs Address : 0x5377da92
Initramfs Size : 0x117dcf0/18341104 Compression Format: mzip Bootable image at @ ram:0x5342f384
Bootable image segment 0 address range [0x81100000, 0x81b8adc0] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 377 Loading Linux kernel with entry point 0x81690230 ...
Bootloader: Done loading app on core_mask: 0xf
### Launching Linux Kernel (flags = 0x5)
%IOSXEBOOT-c728d3d5f6906183ecb6e668a0c523cb-new_cksum: (rp/0): 4
%IOSXEBOOT-c728d3d5f6906183ecb6e668a0c523cb-saved_cksum: (rp/0): 4
Waiting for 120 seconds for other switches to boot
####Switch is in STRAGGLER mode, waiting for active switch to boot
Active switch has booted up, starting discovery phase
########################################################################################################################
Switch number is 6

4. Show switch shows that a new member is in initialization state. The switch stays in this state permanently.

Switch/Stack Mac Address : 2c5a.0f50.8a80 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
*1 Active 2c5a.0f79.0800 1 V04 Ready
2 Standby 2c5a.0f50.8a80 1 V04 Ready
3 Member 2c5a.0f50.8780 1 V04 Ready
4 Member f80b.cbcf.0a00 1 V04 Ready
5 Member 64f6.9d56.2b80 1 V01 Ready
6 Member 2c5a.0f3a.3a00 1 V04 Initializing

5. To recover from this state I have to reboot a new member.
6. After that the switch boot and works without any problem. On the console I see exactly the same as before and prompt.

Booting...
Interface GE 0 link down***ERROR: PHY link is down
Getting rest of image
Reading full image into memory...Check base package header ...: done = 16384
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 22171522
Bundle Image
--------------------------------------
Kernel Address : 0x5342f384
Kernel Size : 0x34e70e/3467022
Initramfs Address : 0x5377da92
Initramfs Size : 0x117dcf0/18341104
Compression Format: mzip
Bootable image at @ ram:0x5342f384
Bootable image segment 0 address range [0x81100000, 0x81b8adc0] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 377
Loading Linux kernel with entry point 0x81690230 ...
Bootloader: Done loading app on core_mask: 0xf
### Launching Linux Kernel (flags = 0x5)
%IOSXEBOOT-c728d3d5f6906183ecb6e668a0c523cb-new_cksum: (rp/0): 4
%IOSXEBOOT-c728d3d5f6906183ecb6e668a0c523cb-saved_cksum: (rp/0): 4
Waiting for 120 seconds for other switches to boot
####Switch is in STRAGGLER mode, waiting for active switch to boot
Active switch has booted up, starting discovery phase
########################################################################################################################
Switch number is 6

sw1 console is now available

Press RETURN to get started.

Sw1#sho switch
Switch/Stack Mac Address : 2c5a.0f50.8a80 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
*1 Active 2c5a.0f79.0800 1 V04 Ready
2 Standby 2c5a.0f50.8a80 1 V04 Ready
3 Member 2c5a.0f50.8780 1 V04 Ready
4 Member f80b.cbcf.0a00 1 V04 Ready
5 Member 64f6.9d56.2b80 1 V01 Ready
6 Member 2c5a.0f3a.3a00 1 V04 Ready

Could you help me understand the issue.
All the best
Rico Negro

[Philip D'Ath]

What software version are you running?

[rico.nego.rico.nego]

16.3.1 CAT3K_CAA-UNIVERSALK9 INSTALL

[Philip D'Ath · Accepted solution]

That release is a bit "bleeding edge" new.  At a minimum you should move to the current patch release in that train, 16.3.3.

[rico.nego.rico.nego]

I didn't notice that . I wil upgrade IOS.

[philnet100]

Did that resolve the issue for you?

I can see the exact same problem on 3650 Stacks (stuck at "initializing" on first bootup of new member switch) also with 16.3.3..

[Nathan Hancock]

I also have the same problem with a 2 switch stack of Cisco Catalyst 3650 switches running 16.3.3.  Bug??

[Leo Laohoo]

Unless there is an operational reason, stick with 3.6.X and avoid 16.X.X.

[philnet100]

But how? I can find no other version than 16.X.X for Cisco 3650..?

[Leo Laohoo]

What is the exact model number of the switch?

[philnet100]

we do use WS-C3650-24PDM and WS-C3650-48FQM-L.

[Leo Laohoo]

Oh dear.  

Well, I've started testing 16.6.1.  16.5.1a is a bad release.  I have a 3850 that crashes every few days even when the switch does NOT have any config.  I have a TAC Case with them.

I have not tried out 16.3.3 yet.

[maeh.ayi01]

#switch clear stack-mode 

to clear the switches and start from scratch and before stacking, set the priority of the the switch you want to be master to be 15, then stack the switches and power them on.