cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2639
Views
0
Helpful
12
Replies

Port-channel, spanning tree, and vlan mismatch

nkingsbury
Level 1
Level 1

Hello,

I had an 2960x access stack go down in the middle of the night (of course). On the 9500 core, I found the following logs:

015356: Jun 18 2021 03:15:24.087 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0400. Port consistency restored.
015357: Jun 18 2021 03:15:24.087 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0999. Port consistency restored.
015358: Jun 18 2021 03:15:37.208 CDT: %SPANTREE-2-BLOCK_PVID_PEER: Blocking Port-channel1 on VLAN0750. Inconsistent peer vlan.
015359: Jun 18 2021 03:15:37.208 CDT: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking Port-channel1 on VLAN0350. Inconsistent local vlan.
015360: Jun 18 2021 03:15:39.211 CDT: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking Port-channel1 on VLAN0401. Inconsistent local vlan.
015361: Jun 18 2021 03:15:39.212 CDT: %SPANTREE-2-BLOCK_PVID_PEER: Blocking Port-channel1 on VLAN0999. Inconsistent peer vlan.
015362: Jun 18 2021 03:15:39.213 CDT: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking Port-channel1 on VLAN0900. Inconsistent local vlan.
015363: Jun 18 2021 03:15:41.231 CDT: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking Port-channel1 on VLAN0600. Inconsistent local vlan.
015364: Jun 18 2021 03:15:43.227 CDT: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking Port-channel1 on VLAN0155. Inconsistent local vlan.
015365: Jun 18 2021 03:15:53.281 CDT: %SPANTREE-2-BLOCK_PVID_PEER: Blocking Port-channel1 on VLAN0700. Inconsistent peer vlan.
015366: Jun 18 2021 03:15:54.212 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0750. Port consistency restored.
015367: Jun 18 2021 03:15:56.232 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0350. Port consistency restored.
015368: Jun 18 2021 03:15:56.232 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0600. Port consistency restored.
015369: Jun 18 2021 03:15:58.228 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0401. Port consistency restored.
015370: Jun 18 2021 03:15:58.228 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0155. Port consistency restored.
015371: Jun 18 2021 03:16:08.281 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0700. Port consistency restored.
015372: Jun 18 2021 03:16:08.281 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0900. Port consistency restored.
015373: Jun 18 2021 03:16:08.281 CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel1 on VLAN0999. Port consistency restored.
015374: Jun 18 2021 03:16:17.366 CDT: %SPANTREE-2-BLOCK_PVID_PEER: Blocking Port-channel1 on VLAN0401. Inconsistent peer vlan.

I was not able to get the logs from the stack side. There has not been any changes to the port channel since it was installed over a year ago. One thing I did find is that on the core side, the port-channel had 'switchport mode trunk' on it, and the stack side did not. So when running a 'show int trunk' the mode on the core side was "on" and on the stack side was "auto". 

 

I am not sure what the ramifications are of that being mismatched. As I said, the stack has been up for over a year and I have not fully determined what caused this issue to start last night. currently we have had to shut down one of the links in the 2 link port-channel in order to regain access to the stack.

 

Here are the configs for each PO:

Stack 2960-X

interface Port-channel1
 description <== Uplink to PLB-RMT-MDF-SW-1 (po1) ==>
 switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999
 switchport trunk native vlan 999
 ip dhcp snooping trust
end

Core C9500-16

interface Port-channel1
 description <== to PLB-RMT-IDF1-SW-1 ==>
 switchport trunk native vlan 999
 switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999
 switchport mode trunk

I am happy  to provide any thing else to help. 

12 Replies 12

Reza Sharifi
Hall of Fame
Hall of Fame

Hi,

show int trunk' the mode on the core side was "on" and on the stack side was "auto". 

If you are using mode "on" it should be "on" both sides of the connection. The same for LACP. If you want to use LACP, make the core mode active and the stack passive.

HTH

balaji.bandi
Hall of Fame
Hall of Fame

if both are Cisco switch, i would suggest use LACP, mode Active on both the side.

 

CDT: %SPANTREE-2-UNBLOCK_CONSIST_PORT:

Can you post interface config part of the port-channel ?

 

check the spanning tree any topology changes ?

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Leo Laohoo
Hall of Fame
Hall of Fame
  1. What exact firmware is the 9500 running on? 
  2. What exact firmware is the 2960X running on? 
  3. Post the configuration of each trunk port (not the etherchannel) from the 2960X and 9500.

nkingsbury
Level 1
Level 1

Hello everyone,

I apologize for lack of response. I was going through the configs and found what I thought was some config mismatch. I put the port channel memberships on the ports and I lost connectivity to the switch stack. 

version 15.2
no service pad
service timestamps debug datetime msec localtime show-timezone year
service timestamps log datetime msec localtime show-timezone year
service password-encryption
service sequence-numbers
!
!
boot-start-marker
boot-end-marker
!
logging userinfo
aaa new-model
!

!
!
!         
!
!
aaa session-id common
process cpu threshold type total rising 80 interval 5
clock timezone CST -6 0
clock summer-time CDT recurring
!
!
!
!
ip dhcp limit lease log
!
!
ip dhcp snooping vlan 155,350,400,401,500,600,700,750,800
no ip dhcp snooping information option
ip name-server 10.32.69.11
ip name-server 10.40.69.11
login on-failure log
login on-success log
vtp mode transparent
!
!
!
!
!
!
authentication mac-move permit
epm logging
cts sxp log binding-changes
mls qos map cos-dscp 0 8 16 24 32 46 48 56
mls qos srr-queue output cos-map queue 1 threshold 3 4 5
mls qos srr-queue output cos-map queue 2 threshold 1 2
mls qos srr-queue output cos-map queue 2 threshold 2 3
mls qos srr-queue output cos-map queue 2 threshold 3 6 7
mls qos srr-queue output cos-map queue 3 threshold 3 0
mls qos srr-queue output cos-map queue 4 threshold 3 1
mls qos srr-queue output dscp-map queue 1 threshold 3 32 33 40 41 42 43 44 45
mls qos srr-queue output dscp-map queue 1 threshold 3 46 47
mls qos srr-queue output dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23
mls qos srr-queue output dscp-map queue 2 threshold 1 26 27 28 29 30 31 34 35
mls qos srr-queue output dscp-map queue 2 threshold 1 36 37 38 39
mls qos srr-queue output dscp-map queue 2 threshold 2 24
mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55
mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63
mls qos srr-queue output dscp-map queue 3 threshold 3 0 1 2 3 4 5 6 7
mls qos srr-queue output dscp-map queue 4 threshold 1 8 9 11 13 15
mls qos srr-queue output dscp-map queue 4 threshold 2 10 12 14
mls qos queue-set output 1 threshold 1 100 100 50 200
mls qos queue-set output 1 threshold 2 125 125 100 400
mls qos queue-set output 1 threshold 3 100 100 100 400
mls qos queue-set output 1 threshold 4 60 150 50 200
mls qos queue-set output 1 buffers 15 25 40 20
mls qos
!
archive
 log config
  logging enable
  logging size 200
  notify syslog contenttype plaintext
  hidekeys
memory free low-watermark processor 20000
memory free low-watermark IO 20000
!
spanning-tree mode rapid-pvst
spanning-tree logging
spanning-tree extend system-id
spanning-tree portfast edge default
spanning-tree portfast edge bpduguard default
spanning-tree vlan 1-4069 priority 4096
auto qos srnd4
errdisable recovery cause link-flap
errdisable recovery cause dhcp-rate-limit
errdisable recovery interval 120
!
!
!
!
vlan internal allocation policy ascending
!
vlan 100
 name LEGACY-DATA
!
vlan 155
 name MGMT
!
vlan 350
 name GUEST
!
vlan 400
 name USER-1
!
vlan 401
 name DATA-1
!
vlan 500
 name VOICE-1
!
vlan 600
 name SERVER
!
vlan 700  
 name SECURITY
!
vlan 750
 name SCAN
!
vlan 800
 name MILLS-1
!
vlan 900
 name WIFI
!
vlan 999
 name TRANSIT
!
!
! 
!
!
!
!
!
!
!
!
interface Port-channel1
 description <== Uplink to RMT-MDF-SW-1 (po1) ==>
 switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999
 switchport trunk native vlan 999
 switchport mode trunk
 ip dhcp snooping trust
!

interface GigabitEthernet1/0/49
 description <== Uplink to RMT-MDF-SW-1 (po1) ==>
 switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999
 switchport trunk native vlan 999
 switchport mode trunk
 channel-group 1 mode active
!

!
interface GigabitEthernet2/0/50
 description <== Uplink to RMT-MDF-SW-1 (po1) ==>
 switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999
 switchport trunk native vlan 999
 switchport mode trunk
 channel-group 1 mode active
!

interface Vlan1
 no ip address
 shutdown
!
interface Vlan155
 description <MS>,10.96.15.101
 ip address 10.96.15.101 255.255.255.0
!
ip default-gateway 10.96.15.100
!
no ip http server
no ip http secure-server
ip ssh version 2
ip tacacs source-interface Vlan155
!
ip access-list standard VTYACCESS
 remark <== Admin access ==>
 permit 10.32.0.0 0.0.255.255
 permit 10.36.0.0 0.0.255.255
 permit 10.44.0.0 0.0.255.255
 permit 192.168.220.0 0.0.3.255
 permit 10.0.0.0 0.255.255.255
 permit 192.168.224.0 0.0.3.255
!
logging host 10.32.65.103
logging host 10.32.65.29
logging host 10.60.40.53 transport udp port 4514
!
!

!
line con 0
 exec-timeout 5 0
 logging synchronous
 login authentication LOCAL_ONLY
line vty 0 4
 access-class VTYACCESS in
 exec-timeout 60 0
 authorization commands 1 AAA
 authorization commands 12 AAA
 authorization commands 15 AAA
 authorization exec AAA
 logging synchronous
 login authentication AAA
 length 0
 transport input ssh
line vty 5 15
 access-class VTYACCESS in
 exec-timeout 60 0
 authorization commands 1 AAA
 authorization commands 12 AAA
 authorization commands 15 AAA
 authorization exec AAA
 logging synchronous
 login authentication AAA
 transport input ssh
!
ntp logging
ntp source Vlan155
ntp server 10.32.255.16 prefer
ntp server 10.40.255.16
mac address-table notification mac-move
!
end

 

 

I ended up having to go on site, long story short, the port channels are suddenly being err-disabled by BPDU Guard, which is not configured on the ports? Here is the full stack config minus the irrelevant ports:

 

Versions are:

2960X 15.2(7)E3

C9500  16.9.4

 

 

 

 

Can you share the logs related to BPDU guards and also if you took the output of sh int status err
Can you post the same?

Also %SPANTREE-2-BLOCK_PVID_PEER is mostly due to mi matching configs. 

 


## Make sure to mark post as helpful, If it resolved your issue. ##

 





## Make sure to mark post as helpful, If it resolved your issue. ##

suddenly being err-disabled by BPDU Guard

post more logs here.

 

Look at the document miss-configuration protection

 

also, post another side config also.

 

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst2960l/software/15-2_5_e/config-guide/b_1525e_consolidated_2960l_cg/b_1525e_consolidated_2960l_cg_chapter_010001.html

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

nkingsbury
Level 1
Level 1

Here is a section of the log from when I was working on it yesterday. Unfortunately, I do not have the logs from the first time it happened. Currently the stack is having to be run off a single uplink. Any attempt to put it into a port-channel causes it to be err-dis. 

 

I have tried removing the lines  

spanning-tree portfast edge default
spanning-tree portfast edge bpduguard default

And also tried directly disabling BPDU guard on the trunk interfaces, but it would still err as soon as i would put it up.

Port      Name               Status       Reason               Err-disabled Vlans
Gi1/0/50                     err-disabled bpduguard
Gi2/0/50  <== Uplink to PLB- err-disabled bpduguard
Po1       <== Uplink to PLB- err-disabled bpduguard

 

%PARSER-5-CFGLOG_LOGGEDCMD: User:admin  logged command:no spanning-tree portfast 
002591: .Jun 18 2021 13:31:20.568 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin  logged command:shutdown 
002592: .Jun 18 2021 13:31:22.385 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin  logged command:no shutdown 
002593: .Jun 18 2021 13:31:22.445 CDT: %PM-4-ERR_DISABLE: bpduguard error detected on Gi1/0/49, putting Gi1/0/49 in err-disable state
002596: .Jun 18 2021 13:31:56.213 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin  logged command:exit 
002597: .Jun 18 2021 13:32:03.508 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin  logged command:no spanning-tree portfast edge default 
002606: .Jun 18 2021 13:32:47.525 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin  logged command:interface GigabitEthernet1/0/49 
002607: .Jun 18 2021 13:32:48.493 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin  logged command:shutdown 
002608: .Jun 18 2021 13:32:50.492 CDT: %LINK-5-CHANGED: Interface GigabitEthernet1/0/49, changed state to administratively down
002609: .Jun 18 2021 13:32:50.939 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin  logged command:no shutdown 
002610: .Jun 18 2021 13:32:51.037 CDT: %PM-4-ERR_DISABLE: bpduguard error detected on Gi1/0/49, putting Gi1/0/49 in err-disable state
002611: .Jun 18 2021 13:32:52.935 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/49, changed state to down
002612: .Jun 18 2021 13:33:06.315 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin  logged command:shutdown 
002613: .Jun 18 2021 13:33:08.329 CDT: %LINK-5-CHANGED: Interface GigabitEthernet1/0/49, changed state to administratively down
002614: .Jun 18 2021 13:33:09.611 CDT: %PARSER-5-CFGLOG_LOGGEDCMD: User:admin  logged command:no shutdown 
002615: .Jun 18 2021 13:33:09.692 CDT: %PM-4-ERR_DISABLE: bpduguard error detected on Gi1/0/49, putting Gi1/0/49 in err-disable state
002616: .Jun 18 2021 13:33:11.611 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/49, changed state to down

  

I would like to see other side 9500 config too

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help


@nkingsbury wrote:
Port      Name               Status       Reason               Err-disabled Vlans
Gi1/0/50                     err-disabled bpduguard
Gi2/0/50  <== Uplink to PLB- err-disabled bpduguard
Po1       <== Uplink to PLB- err-disabled bpduguard

Wait.  WTF.  Gi1/0/50?

If your previous response (LINK) with the config for the 2960X, Po1 members were Gi1/0/49 and Gi2/0/50.  What is Gi1/0/50?

Sorry for the confusion, this is a production stack, so I cant leave it it in an errored state. The stack is currently running off of 1/0/49 its just not in the group channel. 

nkingsbury
Level 1
Level 1

I think my main question at this point is why would BPDU guard be shutting down these ports if it is not enabled? Is there something that would be putting BPDU guard on every port? 

 

Here is the ports on the C9500 side:

interface Port-channel1
 description <== to RMT-IDF1-SW-1 ==>
 switchport trunk native vlan 999
 switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999
 switchport mode trunk

interface TenGigabitEthernet1/0/2
 description <== To RMT-IDF1-SW-1 (Te1/0/1) ==>
 switchport trunk native vlan 999
 switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999
 switchport mode trunk
 channel-group 1 mode active

interface TenGigabitEthernet2/0/2
 description <== To RMT-IDF1-SW-1 (Te2/1/1) ==>
 switchport trunk native vlan 999
 switchport trunk allowed vlan 100,155,350,400,401,500,600,700,750,800,900,999
 switchport mode trunk
 channel-group 1 mode active
!

Have a look at CSCvt31437.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: