C9500-16X-2Q Pair - no boot when StackWise Virtual enabled

[simoncope]

Hi,

I have a pair of new C9500-16X-2Q switches that I am attempting to pair using StackWise Virtual. They are running 17.03.03 in INSTALL mode and have the requisite Advantage license (show version output attached for the 1st switch). The intention is to link them via the two 40G ports in the C9500-NM-2Q modules using QSFP-H40G-AOC cables.  For my initial attempts to set up stacking I have been using only one link.

I have been referring to the High Availability Configuration Guide, Cisco IOS XE Amsterdam 17.3.x (Catalyst 9500 Switches) at:  https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/17-3/configuration_guide/ha/b_173_ha_9500_cg/configuring_cisco_stackwise_virtual.html

My StackWise Virtual commands:

Switch(config)#stackwise-virtual
Please reload the switch for Stackwise Virtual configuration to take effect
Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-stackwise-virtual)#domain 2
Switch(config-stackwise-virtual)#exit
Switch(config)#int Fo1/1/1
Switch(config-if)#stackwise-virtual link 1
Switch(config-if)#exit

I believe that is all that is required (along with a write mem of course). The problem I have is that when I then issue the reload command, the switches will then not boot back into the OS again. They just kind of get stuck:

Proceed with reload? [confirm]

Chassis 1 reloading, reason - Reload command
May 18 10:47:06.531: %PMAN-5-EXITACTION: F0/0: pvp: Process manager is exiting: reload fp action requested
May 18 10:47:07.895: %PMAN-5-TACTION: R0/0: pvp: Process manager is exiting: process exit with reload stack code

Initializing Hardware......

System Bootstrap, Version 17.3.2r, RELEASE SOFTWARE (P)
Compiled Tue 08/25/2020 23:46:12.85 by rel

Current ROMMON image : Primary
Last reset cause : CpuReset
C9500-16X platform with 16777216 Kbytes of main memory

boot: attempting to boot from [flash:packages.conf]
boot: reading file packages.conf
#
###############################################################################################################################################################################################################################################################################################################################################################################################################################################################################

Waiting for 120 seconds for other switches to boot

Waiting for 120 seconds for other switches to boot
#######################################################################################################################
Switch number is 1

At this point I have no command prompt (connected via RS232 by the way) and cannot issue any commands. There is no further output. The only way out of that state is to reboot and break into ROMMON and unset the parameters relating to stacking.

The same output can be seen regardless of whether the second switch is up and with StackWise Virtual enabled. I have also tried various combinations including using multiple ports for the link, adding DaD, trying 10G ports etc etc but I always end up at the same point. What am I doing wrong?

[marce1000]

                  - Make sure that the  IOS Software License types are the same on both devices (check licenses)

 M.

[simoncope]

Hi, thanks for the reply. Both switches have C9500 Network Advantage ver 1.0 and C9500-16X DNA Advantage ver 1.0:

Switch#sh license all
Smart Licensing Status
======================

Smart Licensing is ENABLED

Export Authorization Key:
Features Authorized:
<none>

Utility:
Status: DISABLED

Smart Licensing Using Policy:
Status: ENABLED

Data Privacy:
Sending Hostname: yes
Callhome hostname privacy: DISABLED
Smart Licensing hostname privacy: DISABLED
Version privacy: DISABLED

Transport:
Type: cslu
Cslu address: <empty>
Proxy:
Not Configured

Miscellaneous:
Custom Id: <empty>

Policy:
Policy in use: Merged from multiple sources.
Reporting ACK required: yes (CISCO default)
Unenforced/Non-Export Perpetual Attributes:
First report requirement (days): 365 (CISCO default)
Reporting frequency (days): 0 (CISCO default)
Report on change (days): 90 (CISCO default)
Unenforced/Non-Export Subscription Attributes:
First report requirement (days): 90 (CISCO default)
Reporting frequency (days): 90 (CISCO default)
Report on change (days): 90 (CISCO default)
Enforced (Perpetual/Subscription) License Attributes:
First report requirement (days): 0 (CISCO default)
Reporting frequency (days): 0 (CISCO default)
Report on change (days): 0 (CISCO default)
Export (Perpetual/Subscription) License Attributes:
First report requirement (days): 0 (CISCO default)
Reporting frequency (days): 0 (CISCO default)
Report on change (days): 0 (CISCO default)

Usage Reporting:
Last ACK received: <none>
Next ACK deadline: Aug 15 13:52:51 2021 UTC
Reporting push interval: 30 days
Next ACK push check: <none>
Next report push: May 18 12:04:53 2021 UTC
Last report push: <none>
Last report file write: <none>

Trust Code Installed: <none>

License Usage
=============

network-advantage (C9500 Network Advantage):
Description: network-advantage
Count: 1
Version: 1.0
Status: IN USE
Export status: NOT RESTRICTED
Feature Name: network-advantage
Feature Description: network-advantage
Enforcement type: NOT ENFORCED
License type: Perpetual

dna-advantage (C9500-16X DNA Advantage):
Description: C9500-16X DNA Advantage
Count: 1
Version: 1.0
Status: IN USE
Export status: NOT RESTRICTED
Feature Name: dna-advantage
Feature Description: C9500-16X DNA Advantage
Enforcement type: NOT ENFORCED
License type: Subscription

Product Information
===================
UDI: PID:C9500-16X,SN:XXXXXXXXX

Agent Version
=============
Smart Agent for Licensing: 5.0.9_rel/68

License Authorizations
======================
Overall status:
Active: PID:C9500-16X,SN:FOC2506LLE6
Status: NOT INSTALLED

Purchased Licenses:
No Purchase Information Available

[balaji.bandi]

not sure what is the other side switch config :

I have followed below document and did many SWV works as expected step by step :

https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9000/nb-06-cat-9k-stack-wp-cte-en.html

[simoncope]

Hi. The other switch was configured with the same SWV commands as detailed in my original post. I have been using that White Paper also, but it doesn't explain what is going wrong in my case.

[balaji.bandi]

can you post both the side config.

[simoncope]

Sure, both config files attached now. Note that you will not see any Stackwise Virtual commands in there due to the problem outlined in my original post. You will have to trust me that at this point they have been applied (see original post) and saved, but they will not show up in the running config until after reload. And when I reload....well, that's the problem!

[balaji.bandi]

Not that we are forcing here to show what you did, we try to understand the issue and help you : with the below template as basic, i have builkd many SWV with Cat 9500.

Once other switch elected as Primary then ( Seconday console will have Limited access) - connect your console other switch and check)

Configure SW1:
en
conf t
BB(config)#stackwise-virtual
BB(config-stackwise-virtual)#domain 100

Create the stackwise virtual links (SVL's)
BB(config)#inter ten 1/0/47
Switch(config-if)#description ***SVL***
BB(config-if)#stackwise-virtual link 1
BB(config)#inter ten 1/0/48
Switch(config-if)#description ***SVL***
BB(config-if)#stackwise-virtual link 1

Create the "dual-active-detection link"
BB(config)#inter ten 1/0/1
Switch(config-if)#description ***DAD***
BB(config-if)#stackwise-virtual dual-active-detection
end
wr

Do the same for SW2:

en
conf t
BB(config)#stackwise-virtual
BB(config-stackwise-virtual)#domain 100

Create the stackwise virtual links (SVL's)
BB(config)#inter ten 1/0/47
Switch(config-if)#description ***SVL***
BB(config-if)#stackwise-virtual link 1
BB(config)#inter ten 1/0/48
Switch(config-if)#description ***SVL***
BB(config-if)#stackwise-virtual link 1

Create the "dual-active-detection link"
BB(config)#inter ten 1/0/1
Switch(config-if)#description ***DAD***
BB(config-if)#stackwise-virtual dual-active-detection
end
wr

Conenct the cables while booting or before  and the switches rebooted :

BB#show stackwise-virtual

[simoncope]

It is fine, I understand. It is my first time attempting to set up Stackwise Virtual, so I accept that I may have got something wrong.

From those basic switch configs I then add the commands (for example):

NET-SW-1200-R04-01(config)#stackwise-virtual

NET-SW-1200-R04-01(config-stackwise-virtual)#domain 100

NET-SW-1200-R04-01(config-stackwise-virtual)#exit
NET-SW-1200-R04-01(config)#int Fo1/1/1
NET-SW-1200-R04-01(config-if)#stackwise-virtual link 1
NET-SW-1200-R04-01(config-if)#exit

NET-SW-1200-R04-01#write mem

NET-SW-1200-R04-01#reload

I have also tried the above but adding Fo1/1/2 as the dual activation link (my understanding is that DAD is recommended but optional):

NET-SW-1200-R04-01(config)#int Fo1/1/1
NET-SW-1200-R04-01(config-if)#stackwise-virtual dual-active-detection
NET-SW-1200-R04-01(config-if)#exit

I have also tried using both Fo1/1/1 and Fo1/1/2 in the SVL, with no DAD.

I have also tried using one of the 10G interfaces as the SVL instead.

All of the above were done replicating the commands on the 2nd switch, and reloading with the relevant cables attached. I always get the same outcome. I never see any further messages in the CLI indicating that the stack is being formed.

[balaji.bandi]

NET-SW-1200-R04-01(config)#int Fo1/1/1
NET-SW-1200-R04-01(config-if)#stackwise-virtual dual-active-detection
NET-SW-1200-R04-01(config-if)#exit

Looks above config conflict with SVL Links. you need to use a different GB port for DAD if you like to use it.

If you need further assistance then we need a complete Log right from the beginning to end reload from the console - from bot the switch? (is this possible to post) ?

[simoncope]

Sorry, that was a typo. The DAD command would have been:

NET-SW-1200-R04-01(config)#int Fo1/1/2
NET-SW-1200-R04-01(config-if)#stackwise-virtual dual-active-detection
NET-SW-1200-R04-01(config-if)#exit

As obviously Fo1/1/1 was being used for SVL.

Anyway, ignore the DAD configuration. Just doing the below should be enough to configure Stackwise Virtual, correct? Why do my switches hang on reload after doing these basic commands:

NET-SW-1200-R04-01(config)#stackwise-virtual

NET-SW-1200-R04-01(config-stackwise-virtual)#domain 100

NET-SW-1200-R04-01(config-stackwise-virtual)#exit
NET-SW-1200-R04-01(config)#int Fo1/1/1
NET-SW-1200-R04-01(config-if)#stackwise-virtual link 1
NET-SW-1200-R04-01(config-if)#exit

NET-SW-1200-R04-01#write mem

NET-SW-1200-R04-01#reload

[balaji.bandi]

Obviously we do not know, since we do not see what you see there on the console. Until 16.12.X Cat 9500 SWV works for me with good installation base.

I have not installed any SWV with out DAD Links (but technically may be not required - not tested myself).

couple of things come in to mind for testing :

1. Is the ports working and come up, without SWV configured ?

2. When you reload, ( what is the console output of both ? (this was asked before also - we expected to see the full boot).

3. hope you have checked and verified both the version of code is same.

4. is the both the console of the devices freezes - after reload ? and you not able to input any commands (any one of them).

Since you are doing  with 17.3.3 code - may be bug but not that any one reported here for sure.

[simoncope]

1. Is the ports working and come up, without SWV configured ? - yes

2. When you reload, ( what is the console output of both ? (this was asked before also - we expected to see the full boot). - I posted this output in my original post (well, just for one switch, but it is the same on the other)

3. hope you have checked and verified both the version of code is same. - yes, I mentioned in my original post that they are both running 17.03.03

4. is the both the console of the devices freezes - after reload ? and you not able to input any commands (any one of them). - yes, this is correct

Since you are doing  with 17.3.3 code - may be bug but not that any one reported here for sure. - it could be, yes. However I was seeing the same issues with 16.12.05. I upgraded to 17.3.3 this week to see if that resolved the problem.

[balaji.bandi]

Cat 9500 with 16.12.04 - more on network with SWV i do not see any issue around for sure.  - there may be some bit piece of information we missing here for sure.

Once you configure SWV before reload after wriite - can you post show run ( FULL config here )

[simoncope]

Hi. show run outputs for both switches posted yesterday at your request: https://community.cisco.com/t5/switching/c9500-16x-2q-pair-no-boot-when-stackwise-virtual-enabled/m-p/4404789/highlight/true#M505075