Catalyst 3650 stack issue

[Rohit Mangotra]

Hi,

I am trying to stack 2 3650 switch with same IOS both of them are L2 LAN based switches. When I connect the two switches I get the error mentioned below.

Bootable image at @ ram:0x5342f388
Bootable image segment 0 address range [0x81100000, 0x81b8adc0] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 377
Loading Linux kernel with entry point 0x816903a0 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

Final tar file: mcu_ucode_bundle_8_1_0.tar

*** Stack adapter authentication failed on stack port 1 on switch 2 ***

Stack Adapter Auth Fail : SIF_SERDES_CABLE_EASTBOUND
Waiting for 120 seconds for other switches to boot
All switches in the stack have been discovered. Accelerating discovery

Chassis 2 reloading, reason - active removed before switch became ready
Apr 4 10:49:44.008 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested
Apr 4 10:49:52.640 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: rp processes exit with reload switch code

octeon_wdt: WDT device closed unexpectedly. WDT will not stop!
reboot: Restarting system

Switch#sh switch
Switch/Stack Mac Address : 58f3.9cac.0f80 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
1 Member 0000.0000.0000 0 V04 Removed
*2 Active 58f3.9cac.0f80 1 V01 Ready

Have anyone seen this issue before. Anything I can do to fix this issue?

Thanks in advance

Kind Regards

Rohit.

[Leo Laohoo]

What firmware is the stack running on?
CSCvo12056

[Rohit Mangotra]

Hi Leo,
This is the software version I am using in all the switches. I am having the issue when I am using 2 Catalyst 3650 switches 48 ports or one 48 ports and other 24 ports.
Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9- M),
Version 16.3.7, RELEASE SOFTWARE (fc4)
Thanks & Regards
Rohit.

[Deepak Kumar]

Hi,

What is the firmware version? We noticed the same type of bug in C9300 switches.

[Leo Laohoo]

---

@Deepak Kumar wrote:

We noticed the same type of bug in C9300 switches.

---

Deepak, 

I think you're hitting this:  CSCvn30950

[Rohit Mangotra]

Hi Deepak,
This is the software version I am using in all the switches. I am having the issue when I am using 2 Catalyst 3650 switches 48 ports or one 48 ports and other 24 ports.
Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9- M),
Version 16.3.7, RELEASE SOFTWARE (fc4)
Thanks & Regards
Rohit.

[Deepak Kumar]

Hi,

Let please confirm a few things:

1. What is happening if you are booting the same switch as standalone?

2. Are both switches are having same IOS version?

[Rohit Mangotra]

Hi Deepak,

When we reboot the switch independeltly the stack configuration is still there and after reboot it still have got the same. So I have to hard reset the switch to factory default. If they work standalone they work fine.

Yes both the switches have got the same IOS version.

Thanks & Regards

Rohit.

[paul driver]

Hello

Just like to add and confirm how are you stacking these two switches?

Have you provisioned the master switch prior to stacking it -

Do you shutdown the slave switch prior to attaching it to the master?

Have you made sure the stack cable cables are okay and are seated correctly  

[Rohit Mangotra]

Hi Paul,

Yes I had provisioned the master switch prior to stacking it.

Yes I shutdown the salve before I connected it to the master switch.

The stack cables are brand new.

I did stacking for three 3650 48 port LAN based and it went just fine. No issues. I am only experiencing problem with the same model switched that I need to deploy in different location. Not sure what's going on.

Thanks & Regards

Rohit.

[Philip Denton]

Did anyone find a fix to this?  I'm deploying a bunch of 3650s and am running into the same issue.  I'm running 16.3(8) on all stacks.  4 stacks are OK and 3 are showing the following error:

*** Stack adapter authentication failed on stack port 1 on switch 1 ***

Jul 10 15:20:07.377 R0/0: %PMAN-3-PROCHOLDDOWN: The process sif_mgr has been helddown (rc 134)
Jul 10 15:20:07.952 R0/0: %PMAN-0-PROCFAILCRIT: A critical process sif_mgr has failed (rc 134)
Jul 10 15:20:08.456 R0/0: %PMAN-3-RELOAD_RP: Reloading: Switch will be reloaded

Chassis 0 reloading, reason - Non participant detected
Jul 10 15:20:17.840 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested
Jul 10 15:20:28.268 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: rp processes exit with reload sw itch code
Jul 10 15:20:47.050 R0/0: %PMAN-3-PROCESS_NOTIFICATION: System report /crashinfo/system-report_local_20190 710-152041-Universal.tar.gz (size: 5621 KB) generated

octeon_wdt: WDT device closed unexpectedly. WDT will not stop!
reboot: Restarting system

The switches are just rebooting over and over.  If I remove the stack adapters they come up in standalone mode just fine.  Then when I re-insert the stack adapters and reboot they get stuck in a boot loop again.

What is going on here?  Does anyone have a fix for this?

[Leo Laohoo]

---

@Philip Denton wrote:

Does anyone have a fix for this?

---

Update the firmware to, like, 16.6.6.

[Philip Denton]

I thought 16.3(8) was just as new as 16.6(6).  Do you know for sure if this bug has been fixed in the 16.6 train (where it hasn't been in 16.3?).

I only ask because I'm pretty sure some of my switches came running 16.6(6) and still didn't boot properly with the stack modules installed but I'm certainly happy to try upgrading.  I will post my results as soon as I can.

[Philip Denton]

Upgrading from 16.3(8) to 16.6(6) did not fix the issue.  What else can I try?

*** Stack adapter authentication failed on stack port 1 on switch 1 ***

Jul 12 16:52:05.819 RP0/0: %PMAN-3-PROCHOLDDOWN: The process sif_mgr has been helddown (rc 134)
Jul 12 16:52:06.025 RP0/0: %PMAN-0-PROCFAILCRIT: A critical process sif_mgr has failed (rc 134)
Jul 12 16:52:06.257 RP0/0: %PMAN-3-RELOAD_RP: Reloading: Switch will be reloaded

Chassis 0 reloading, reason - Non participant detected
Jul 12 16:52:14.650 FP0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested
Jul 12 16:52:19.048 RP0/0: %PMAN-5-EXITACTION: Process manager is exiting: rp procesJul 12 16:52:51.101 RP0/0: %PMAN-3-PROCESS_NOTIFICATION: System report /crashinfo/system-report_local_20190712-165244-Universal.tar.gz (size: 7426 KB) generated

octeon_wdt: WDT device closed unexpectedly. WDT will not stop!
reboot: Restarting system

[Leo Laohoo]

Raise a TAC Case. This could be CSCvn96821.