06-18-2021 09:13 AM
Hello,
I had an 2960x access stack go down in the middle of the night (of course). On the 9500 core, I found the following logs:
015356: Jun 18 2021 03:15:24.087 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0400. Port consistency restored. 015357: Jun 18 2021 03:15:24.087 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0999. Port consistency restored. 015358: Jun 18 2021 03:15:37.208 CDT: %SPANTREE-2-BLOCK_PVID_PEER: Blocking Port-channel1 on VLAN0750. Inconsistent peer vlan. 015359: Jun 18 2021 03:15:37.208 CDT: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking Port-channel1 on VLAN0350. Inconsistent local vlan. 015360: Jun 18 2021 03:15:39.211 CDT: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking Port-channel1 on VLAN0401. Inconsistent local vlan. 015361: Jun 18 2021 03:15:39.212 CDT: %SPANTREE-2-BLOCK_PVID_PEER: Blocking Port-channel1 on VLAN0999. Inconsistent peer vlan. 015362: Jun 18 2021 03:15:39.213 CDT: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking Port-channel1 on VLAN0900. Inconsistent local vlan. 015363: Jun 18 2021 03:15:41.231 CDT: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking Port-channel1 on VLAN0600. Inconsistent local vlan. 015364: Jun 18 2021 03:15:43.227 CDT: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking Port-channel1 on VLAN0155. Inconsistent local vlan. 015365: Jun 18 2021 03:15:53.281 CDT: %SPANTREE-2-BLOCK_PVID_PEER: Blocking Port-channel1 on VLAN0700. Inconsistent peer vlan. 015366: Jun 18 2021 03:15:54.212 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0750. Port consistency restored. 015367: Jun 18 2021 03:15:56.232 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0350. Port consistency restored. 015368: Jun 18 2021 03:15:56.232 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0600. Port consistency restored. 015369: Jun 18 2021 03:15:58.228 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0401. Port consistency restored. 015370: Jun 18 2021 03:15:58.228 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0155. Port consistency restored. 015371: Jun 18 2021 03:16:08.281 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0700. Port consistency restored. 015372: Jun 18 2021 03:16:08.281 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0900. Port consistency restored. 015373: Jun 18 2021 03:16:08.281 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0999. Port consistency restored. 015374: Jun 18 2021 03:16:17.366 CDT: %SPANTREE-2-BLOCK_PVID_PEER: Blocking Port-channel1 on VLAN0401. Inconsistent peer vlan.
I was not able to get the logs from the stack side. There has not been any changes to the port channel since it was installed over a year ago. One thing I did find is that on the core side, the port-channel had 'switchport mode trunk' on it, and the stack side did not. So when running a 'show int trunk' the mode on the core side was "on" and on the stack side was "auto".
I am not sure what the ramifications are of that being mismatched. As I said, the stack has been up for over a year and I have not fully determined what caused this issue to start last night. currently we have had to shut down one of the links in the 2 link port-channel in order to regain access to the stack.
Here are the configs for each PO:
Stack 2960-X
interface Port-channel1 description <== Uplink to PLB-RMT-MDF-SW-1 (po1) ==> switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999 switchport trunk native vlan 999 ip dhcp snooping trust end
Core C9500-16
interface Port-channel1 description <== to PLB-RMT-IDF1-SW-1 ==> switchport trunk native vlan 999 switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999 switchport mode trunk
I am happy to provide any thing else to help.
06-18-2021 09:21 AM
Hi,
show int trunk' the mode on the core side was "on" and on the stack side was "auto".
If you are using mode "on" it should be "on" both sides of the connection. The same for LACP. If you want to use LACP, make the core mode active and the stack passive.
HTH
06-18-2021 09:22 AM
if both are Cisco switch, i would suggest use LACP, mode Active on both the side.
CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT:
Can you post interface config part of the port-channel ?
check the spanning tree any topology changes ?
06-18-2021 06:35 PM
06-18-2021 08:10 PM - edited 06-18-2021 08:27 PM
Hello everyone,
I apologize for lack of response. I was going through the configs and found what I thought was some config mismatch. I put the port channel memberships on the ports and I lost connectivity to the switch stack.
version 15.2 no service pad service timestamps debug datetime msec localtime show-timezone year service timestamps log datetime msec localtime show-timezone year service password-encryption service sequence-numbers ! ! boot-start-marker boot-end-marker ! logging userinfo aaa new-model ! ! ! ! ! ! aaa session-id common process cpu threshold type total rising 80 interval 5 clock timezone CST -6 0 clock summer-time CDT recurring ! ! ! ! ip dhcp limit lease log ! ! ip dhcp snooping vlan 155,350,400,401,500,600,700,750,800 no ip dhcp snooping information option ip name-server 10.32.69.11 ip name-server 10.40.69.11 login on-failure log login on-success log vtp mode transparent ! ! ! ! ! ! authentication mac-move permit epm logging cts sxp log binding-changes mls qos map cos-dscp 0 8 16 24 32 46 48 56 mls qos srr-queue output cos-map queue 1 threshold 3 4 5 mls qos srr-queue output cos-map queue 2 threshold 1 2 mls qos srr-queue output cos-map queue 2 threshold 2 3 mls qos srr-queue output cos-map queue 2 threshold 3 6 7 mls qos srr-queue output cos-map queue 3 threshold 3 0 mls qos srr-queue output cos-map queue 4 threshold 3 1 mls qos srr-queue output dscp-map queue 1 threshold 3 32 33 40 41 42 43 44 45 mls qos srr-queue output dscp-map queue 1 threshold 3 46 47 mls qos srr-queue output dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23 mls qos srr-queue output dscp-map queue 2 threshold 1 26 27 28 29 30 31 34 35 mls qos srr-queue output dscp-map queue 2 threshold 1 36 37 38 39 mls qos srr-queue output dscp-map queue 2 threshold 2 24 mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55 mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63 mls qos srr-queue output dscp-map queue 3 threshold 3 0 1 2 3 4 5 6 7 mls qos srr-queue output dscp-map queue 4 threshold 1 8 9 11 13 15 mls qos srr-queue output dscp-map queue 4 threshold 2 10 12 14 mls qos queue-set output 1 threshold 1 100 100 50 200 mls qos queue-set output 1 threshold 2 125 125 100 400 mls qos queue-set output 1 threshold 3 100 100 100 400 mls qos queue-set output 1 threshold 4 60 150 50 200 mls qos queue-set output 1 buffers 15 25 40 20 mls qos ! archive log config logging enable logging size 200 notify syslog contenttype plaintext hidekeys memory free low-watermark processor 20000 memory free low-watermark IO 20000 ! spanning-tree mode rapid-pvst spanning-tree logging spanning-tree extend system-id spanning-tree portfast edge default spanning-tree portfast edge bpduguard default spanning-tree vlan 1-4069 priority 4096 auto qos srnd4 errdisable recovery cause link-flap errdisable recovery cause dhcp-rate-limit errdisable recovery interval 120 ! ! ! ! vlan internal allocation policy ascending ! vlan 100 name LEGACY-DATA ! vlan 155 name MGMT ! vlan 350 name GUEST ! vlan 400 name USER-1 ! vlan 401 name DATA-1 ! vlan 500 name VOICE-1 ! vlan 600 name SERVER ! vlan 700 name SECURITY ! vlan 750 name SCAN ! vlan 800 name MILLS-1 ! vlan 900 name WIFI ! vlan 999 name TRANSIT ! ! ! ! ! ! ! ! ! ! ! interface Port-channel1 description <== Uplink to RMT-MDF-SW-1 (po1) ==> switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999 switchport trunk native vlan 999 switchport mode trunk ip dhcp snooping trust ! interface GigabitEthernet1/0/49 description <== Uplink to RMT-MDF-SW-1 (po1) ==> switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999 switchport trunk native vlan 999 switchport mode trunk channel-group 1 mode active ! ! interface GigabitEthernet2/0/50 description <== Uplink to RMT-MDF-SW-1 (po1) ==> switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999 switchport trunk native vlan 999 switchport mode trunk channel-group 1 mode active ! interface Vlan1 no ip address shutdown ! interface Vlan155 description <MS>,10.96.15.101 ip address 10.96.15.101 255.255.255.0 ! ip default-gateway 10.96.15.100 ! no ip http server no ip http secure-server ip ssh version 2 ip tacacs source-interface Vlan155 ! ip access-list standard VTYACCESS remark <== Admin access ==> permit 10.32.0.0 0.0.255.255 permit 10.36.0.0 0.0.255.255 permit 10.44.0.0 0.0.255.255 permit 192.168.220.0 0.0.3.255 permit 10.0.0.0 0.255.255.255 permit 192.168.224.0 0.0.3.255 ! logging host 10.32.65.103 logging host 10.32.65.29 logging host 10.60.40.53 transport udp port 4514 ! ! ! line con 0 exec-timeout 5 0 logging synchronous login authentication LOCAL_ONLY line vty 0 4 access-class VTYACCESS in exec-timeout 60 0 authorization commands 1 AAA authorization commands 12 AAA authorization commands 15 AAA authorization exec AAA logging synchronous login authentication AAA length 0 transport input ssh line vty 5 15 access-class VTYACCESS in exec-timeout 60 0 authorization commands 1 AAA authorization commands 12 AAA authorization commands 15 AAA authorization exec AAA logging synchronous login authentication AAA transport input ssh ! ntp logging ntp source Vlan155 ntp server 10.32.255.16 prefer ntp server 10.40.255.16 mac address-table notification mac-move ! end
I ended up having to go on site, long story short, the port channels are suddenly being err-disabled by BPDU Guard, which is not configured on the ports? Here is the full stack config minus the irrelevant ports:
Versions are:
2960X 15.2(7)E3
C9500 16.9.4
06-18-2021 11:14 PM
Can you share the logs related to BPDU guards and also if you took the output of sh int status err
Can you post the same?
Also %SPANTREE-2-BLOCK_PVID_PEER is mostly due to mi matching configs.
## Make sure to mark post as helpful, If it resolved your issue. ##
06-19-2021 01:36 AM
suddenly being err-disabled by BPDU Guard
post more logs here.
Look at the document miss-configuration protection
also, post another side config also.
06-19-2021 09:18 AM
Here is a section of the log from when I was working on it yesterday. Unfortunately, I do not have the logs from the first time it happened. Currently the stack is having to be run off a single uplink. Any attempt to put it into a port-channel causes it to be err-dis.
I have tried removing the lines
spanning-tree portfast edge default spanning-tree portfast edge bpduguard default
And also tried directly disabling BPDU guard on the trunk interfaces, but it would still err as soon as i would put it up.
Port Name Status Reason Err-disabled Vlans Gi1/0/50 err-disabled bpduguard Gi2/0/50 <== Uplink to PLB- err-disabled bpduguard Po1 <== Uplink to PLB- err-disabled bpduguard
%PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:no spanning-tree portfast 002591: .Jun 18 2021 13:31:20.568 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:shutdown 002592: .Jun 18 2021 13:31:22.385 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:no shutdown 002593: .Jun 18 2021 13:31:22.445 CDT: %PM-4-ERR_DISABLE: bpduguard error detected on Gi1/0/49, putting Gi1/0/49 in err-disable state 002596: .Jun 18 2021 13:31:56.213 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:exit 002597: .Jun 18 2021 13:32:03.508 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:no spanning-tree portfast edge default 002606: .Jun 18 2021 13:32:47.525 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:interface GigabitEthernet1/0/49 002607: .Jun 18 2021 13:32:48.493 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:shutdown 002608: .Jun 18 2021 13:32:50.492 CDT: %LINK-5-CHANGED: Interface GigabitEthernet1/0/49, changed state to administratively down 002609: .Jun 18 2021 13:32:50.939 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:no shutdown 002610: .Jun 18 2021 13:32:51.037 CDT: %PM-4-ERR_DISABLE: bpduguard error detected on Gi1/0/49, putting Gi1/0/49 in err-disable state 002611: .Jun 18 2021 13:32:52.935 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/49, changed state to down 002612: .Jun 18 2021 13:33:06.315 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:shutdown 002613: .Jun 18 2021 13:33:08.329 CDT: %LINK-5-CHANGED: Interface GigabitEthernet1/0/49, changed state to administratively down 002614: .Jun 18 2021 13:33:09.611 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin logged command:no shutdown 002615: .Jun 18 2021 13:33:09.692 CDT: %PM-4-ERR_DISABLE: bpduguard error detected on Gi1/0/49, putting Gi1/0/49 in err-disable state 002616: .Jun 18 2021 13:33:11.611 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/49, changed state to down
06-19-2021 11:10 AM
I would like to see other side 9500 config too
06-19-2021 05:35 PM
@nkingsbury wrote:
Port Name Status Reason Err-disabled Vlans Gi1/0/50 err-disabled bpduguard Gi2/0/50 <== Uplink to PLB- err-disabled bpduguard Po1 <== Uplink to PLB- err-disabled bpduguard
Wait. WTF. Gi1/0/50?
If your previous response (LINK) with the config for the 2960X, Po1 members were Gi1/0/49 and Gi2/0/50. What is Gi1/0/50?
06-19-2021 08:37 PM
Sorry for the confusion, this is a production stack, so I cant leave it it in an errored state. The stack is currently running off of 1/0/49 its just not in the group channel.
06-19-2021 08:40 PM - edited 06-19-2021 08:41 PM
I think my main question at this point is why would BPDU guard be shutting down these ports if it is not enabled? Is there something that would be putting BPDU guard on every port?
Here is the ports on the C9500 side:
interface Port-channel1 description <== to RMT-IDF1-SW-1 ==> switchport trunk native vlan 999 switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999 switchport mode trunk interface TenGigabitEthernet1/0/2 description <== To RMT-IDF1-SW-1 (Te1/0/1) ==> switchport trunk native vlan 999 switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999 switchport mode trunk channel-group 1 mode active interface TenGigabitEthernet2/0/2 description <== To RMT-IDF1-SW-1 (Te2/1/1) ==> switchport trunk native vlan 999 switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999 switchport mode trunk channel-group 1 mode active !
06-19-2021 09:41 PM - edited 06-19-2021 09:46 PM
Have a look at CSCvt31437.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide