Added switch to network and err-disabled part of the network, wh

Andy White · ‎09-07-2013

Hello,

Still not sure how I did this even after looking at the log files which are below. Anyway we have a LAN which has 2 Nortel 8600 Core switch which are trunked together (IST - Nortel Etherchannel). Off each Nortel are etherchannels to 2 Cisco 3750 stacks where our VMware farm and Storage SANs are connected.

I plugged the Cisco 3750 into the Nortel switch shown above (port 1/27) and at that point I was fine, but when I on to the Nortel GUI manager and selected port 1/27 and these 3 options, band ports 12,24 on the 2 Cisco stacks went into error disable:

Perform Tagging
Discard Untagged Frames
Loop Detection

I then clicked apply and that did the trick and all went down. I have a feeling it was the Loop Detection that cause the issue but why?

Here are the logs and the Cisco switch config I added.

Did the 2 Cisco stacks see the mac from 2 loctions? What is strange I've don this one before on the other Nortel Core and it was fine. I hate those Nortels.

# Generated by Syslog Viewer

# Version 2013.1.0

# 06/09/2013 18:06:42

#

# Syslog Messages

# 06 September 2013 06:06 PM

Gi3/0/33" SW_MATM-4-MACFLAP_NOTIF 320517892

"06/09/2013 15:43:44" 192.168.28.251 192.168.28.251 Error "26108: Interface GigabitEthernet1/0/24, changed state to down" LINK-3-UPDOWN 320517882

"06/09/2013 15:43:44" 192.168.28.251 192.168.28.251 Error "26107: Interface Port-channel1, changed state to down" LINK-3-UPDOWN 320517891

"06/09/2013 15:43:43" 192.168.28.251 192.168.28.251 Notice "26103: Line protocol on Interface GigabitEthernet1/0/12, changed state to down" LINEPROTO-5-UPDOWN 320517883

"06/09/2013 15:43:43" 192.168.28.251 192.168.28.251 Notice "26104: Line protocol on Interface GigabitEthernet1/0/24, changed state to down" LINEPROTO-5-UPDOWN 320517885

"06/09/2013 15:43:43" 192.168.28.251 192.168.28.251 Notice "26105: Line protocol on Interface Port-channel1, changed state to down" LINEPROTO-5-UPDOWN 320517887

"06/09/2013 15:43:43" 192.168.28.251 192.168.28.251 Error "26106: Interface GigabitEthernet1/0/12, changed state to down" LINK-3-UPDOWN 320517889

"06/09/2013 15:43:42" 192.168.28.251 192.168.28.251 Warning "26101: channel-misconfig (STP) error detected on Po1, putting Gi1/0/24 in err-disable state" PM-4-ERR_DISABLE 320517888

"06/09/2013 15:43:42" 192.168.28.251 192.168.28.251 Warning "26100: channel-misconfig (STP) error detected on Po1, putting Gi1/0/12 in err-disable state" PM-4-ERR_DISABLE 320517886

"06/09/2013 15:43:42" 192.168.28.251 192.168.28.251 Warning "26099: channel-misconfig (STP) error detected on Gi1/0/24, putting Gi1/0/24 in err-disable state" PM-4-ERR_DISABLE 320517884

"06/09/2013 15:43:42" 192.168.28.251 192.168.28.251 Warning "26098: channel-misconfig (STP) error detected on Gi1/0/12, putting Gi1/0/12 in err-disable state" PM-4-ERR_DISABLE 320517881

"06/09/2013 15:43:42" 192.168.28.251 192.168.28.251 Warning "26102: channel-misconfig (STP) error detected on Po1, putting Po1 in err-disable state" PM-4-ERR_DISABLE 320517890

"06/09/2013 15:43:31" 192.168.28.250 192.168.28.250 Warning "29404: 032029: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)." CDP-4-NATIVE_VLAN_MISMATCH 320517880

"06/09/2013 15:43:31" 192.168.28.250 192.168.28.250 Warning "29405: 000045: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)" CDP-4-NATIVE_VLAN_MISMATCH 320517879

"06/09/2013 15:43:31" 192.168.28.251 192.168.28.251 Warning "26097: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)." CDP-4-NATIVE_VLAN_MISMATCH 320517878

"06/09/2013 15:42:32" 192.168.28.250 192.168.28.250 Warning "29403: 000044: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)" CDP-4-NATIVE_VLAN_MISMATCH 320517875

"06/09/2013 15:42:31" 192.168.28.251 192.168.28.251 Warning "26096: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)." CDP-4-NATIVE_VLAN_MISMATCH 320517873

"06/09/2013 15:42:31" 192.168.28.250 192.168.28.250 Warning "29402: 032028: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)." CDP-4-NATIVE_VLAN_MISMATCH 320517874

"06/09/2013 15:42:27" 192.168.28.250 192.168.28.250 Notice "29401: 032027: Line protocol on Interface Vlan23, changed state to down" LINEPROTO-5-UPDOWN 320517872

"06/09/2013 15:42:26" 192.168.28.251 192.168.28.251 Notice "26095: Line protocol on Interface Vlan23, changed state to down" LINEPROTO-5-UPDOWN 320517871

"06/09/2013 15:42:26" 192.168.28.251 192.168.28.251 Notice "26094: Line protocol on Interface Vlan1, changed state to down" LINEPROTO-5-UPDOWN 320517870

"06/09/2013 15:42:26" 192.168.28.251 192.168.28.251 Critical "26093: Blocking Port-channel1 on VLAN0023. Inconsistent local vlan." SPANTREE-2-BLOCK_PVID_LOCAL 320517869

"06/09/2013 15:42:26" 192.168.28.251 192.168.28.251 Critical "26092: Blocking Port-channel1 on VLAN0001. Inconsistent peer vlan." SPANTREE-2-BLOCK_PVID_PEER 320517868

"06/09/2013 15:42:26" 192.168.28.251 192.168.28.251 Critical "26091: Received BPDU with inconsistent peer vlan id 1 on Port-channel1 VLAN23." SPANTREE-2-RECV_PVID_ERR 320517867

"06/09/2013 15:42:26" 192.168.28.250 192.168.28.250 Critical "29400: 032026: Blocking Port-channel1 on VLAN0023. Inconsistent local vlan." SPANTREE-2-BLOCK_PVID_LOCAL 320517866

"06/09/2013 15:42:26" 192.168.28.250 192.168.28.250 Critical "29399: 032025: Blocking Port-channel1 on VLAN0001. Inconsistent peer vlan." SPANTREE-2-BLOCK_PVID_PEER 320517865

"06/09/2013 15:42:26" 192.168.28.250 192.168.28.250 Critical "29398: 032024: Received BPDU with inconsistent peer vlan id 1 on Port-channel1 VLAN23." SPANTREE-2-RECV_PVID_ERR 320517864

"06/09/2013 15:41:32" 192.168.28.250 192.168.28.250 Warning "29397: 000043: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)" CDP-4-NATIVE_VLAN_MISMATCH 320517861

"06/09/2013 15:41:31" 192.168.28.251 192.168.28.251 Warning "26090: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)." CDP-4-NATIVE_VLAN_MISMATCH 320517860

"06/09/2013 15:41:31" 192.168.28.250 192.168.28.250 Warning "29396: 032023: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)." CDP-4-NATIVE_VLAN_MISMATCH 320517859

"06/09/2013 15:40:31" 192.168.28.250 192.168.28.250 Warning "29394: 032022: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)." CDP-4-NATIVE_VLAN_MISMATCH 320517856

"06/09/2013 15:40:31" 192.168.28.250 192.168.28.250 Warning "29395: 000042: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)" CDP-4-NATIVE_VLAN_MISMATCH 320517855

"06/09/2013 15:40:31" 192.168.28.251 192.168.28.251 Warning "26089: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1)." CDP-4-NATIVE_VLAN_MISMATCH 320517854

From one of the heldesk cisco switch I added that caused the issue.

*Mar 1 00:29:09.961: %PARSER-5-CFGLOG_LOGGEDCMD: User:console logged command:ip dhcp snooping vlan 80

*Mar 1 00:29:22.997: %PARSER-5-CFGLOG_LOGGEDCMD: User:console logged command:ip dhcp snooping

*Mar 1 00:30:10.050: %SYS-5-CONFIG_I: Configured from console by console

*Mar 1 00:40:11.588: %SYS-5-CONFIG_I: Configured from console by console

*Mar 1 00:42:02.462: %PARSER-5-CFGLOG_LOGGEDCMD: User:console logged command:ip default-gateway 192.168.23.253

*Mar 1 00:42:03.435: %SYS-5-CONFIG_I: Configured from console by console

*Mar 1 00:54:38.807: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/12, changed state to up

*Mar 1 00:54:40.828: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/12, changed state to up

*Mar 1 00:57:44.483: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 23 on GigabitEthernet1/0/12 VLAN1.

*Mar 1 00:57:44.483: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking GigabitEthernet1/0/12 on VLAN0001. Inconsistent local vlan.

*Mar 1 00:59:11.969: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking GigabitEthernet1/0/12 on VLAN0001. Port consistency restored.

*Mar 1 00:59:27.907: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/12, changed state to down

*Mar 1 00:59:28.922: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/12, changed state to down

*Mar 1 03:28:31.432: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: Andyw] [Source: 0.0.0.0] [localport: 0] at 03:28:31 UTC Mon Mar 1 1993

Helpdesk_Switch#sh run

Building configuration...

Current configuration : 3563 bytes

!

version 12.2

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname Helpdesk_Switch

!

boot-start-marker

boot-end-marker

!

logging console informational

enable secret 5 $1$gyXtMMoalpx1SAcS7T.

!

username ***

!

no aaa new-model

switch 1 provision ws-c3750g-12s

system mtu routing 1500

ip domain-name ***

!

ip dhcp snooping vlan 80

ip dhcp snooping

login on-failure log

login on-success log

!

archive

log config

logging enable

logging size 200

notify syslog contenttype plaintext

hidekeys

!

spanning-tree mode rapid-pvst

spanning-tree extend system-id

!

vlan internal allocation policy ascending

!

ip ssh version 2

!

interface GigabitEthernet1/0/1

switchport access vlan 80

switchport mode access

spanning-tree portfast

spanning-tree bpduguard enable

!

(removed)

!

interface GigabitEthernet1/0/12

description Trunk to Nortel

switchport trunk encapsulation dot1q

switchport mode trunk

ip arp inspection trust

ip dhcp snooping trust

!

interface Vlan1

no ip address

shutdown

!

interface Vlan23

ip address 192.168.23.9 255.255.255.0

!

ip default-gateway 192.168.23.253

ip classless

no ip http server

no ip http secure-server

!

logging trap notifications

logging source-interface Vlan23

logging 192.168.*.*

access-list 20 permit 192.168.*.* log

access-list 24 permit 192.168.*.*

snmp-server community *** RO 24

snmp-server host 192.168.*.* ***

!

line con 0

exec-timeout 15 0

logging synchronous

login local

line vty 0 4

access-class 20 in

exec-timeout 15 0

logging synchronous

login local

transport input ssh

line vty 5 15

access-class 20 in

exec-timeout 15 0

logging synchronous

login local

transport input ssh

!

ntp logging

ntp clock-period 36029310

ntp source Vlan23

ntp server 192.168.*.*

end

Helpdesk_Switch#

ALIAOF_ · ‎09-07-2013

Ah STP issues, get rid of them Nortels lol.

Ok so some things to check here. Make sure that all switches are running same STP mode from your config Help Desk is running rstp so make sure all the others are running the same. Also do you have the native VLAN configured as a different VLAN on the Cisco Stacks and Nortel vs the Help Desk switch (Cisco Help Desk switch looks like is using the default VLAN1 as the native VLAN but others aren't ? Make sure that matches.

Jeff Van Houten · ‎09-07-2013

I'm no nortel expert but it looks like the nortels are expecting native vlan 23 whereas the Cisco is configured as native vlan 1 on the trunk port. If that's correct you need to update the trunk port on the ciscos to switchport trunk native vlan23.

Sent from Cisco Technical Support iPad App

Andy White · ‎09-07-2013

It is using Native VLAN 1, but so is the other Cisco switch I installed on the other Nortel core that didn't cause any issues. Why would a native vlan mismatch cause an issue on the 2 separate Cisco switch stacks and cause them to go into err-disable?

I've had native vlan mismatches before and it never caused the ports to shutdown or a STP mismatch.

Thanks

Rolf Fischer · ‎09-07-2013

Hi Andy,

the Cisco per-VLAN STP modes send so-called SSTP BPDUs to the destination address 01-00-0C-CC-CC-CD.

Third party switches normally don't recognize them as BPDUs (IEEE destination address is 01-80-C2-00-00-00) but as the group bit is set, treat them as multicast and flood them. So the Nortel switches act as virtual trunks for the per-VLAN ST instances between the c3750 stacks and the helpdesk switch.

Anyway we have a LAN which has 2 Nortel 8600 Core switch which are trunked together (IST - Nortel Etherchannel).

Are you saying the mode in this part is MSTP?

If so, the the receipt of SSTP BPDUs on switchports running MSTP will result in ST inconsistent state and errdisable the port for ~2 minutes, then the same procedure starts again.

If the rest of the multi-vendor LAN is running MSTP, I'd recommend to configure MSTP on the helpdesk switch as well.

Useful link: Troubleshooting Spanning Tree PVID- and Type-Inconsistencies

Hope that helps

Rolf

Aninda Chatterjee · ‎09-07-2013

Hi Rolf,

Good day.

If so, the the receipt of SSTP BPDUs on switchports running MSTP will result in ST inconsistent state and errdisable the port for ~2 minutes, then the same procedure starts again.

I'm not sure if this is being stated with specifically Nortel switches in mind, but the above is not true for Cisco switches. Cisco switches running MST can interoperate with other switches running PVST+ (or Rapid-PVST+) without any problems. It does get a little complicated, I won't deny that, but in most situations it works fine without any problems.

Regards,

Aninda

Rolf Fischer · ‎09-08-2013

Hi Aninda,

thanks for joining us.

Cisco switches running MST can interoperate with other switches running PVST+ (or Rapid-PVST+) without any problems.

Of course that's correct, but here we could have a very special scenario where some of the ports receive SSTP- and MST BPDUs at the same time so the switch cannot figure out the correct mode.

Anyway, MSTP is obviously not the point here. I've been reading a Nortel Layer-2 configuration guide and found that "IST" stands for "InterSwitch Trunking" (link aggregation).

Best regards

Rolf

ALIAOF_ · ‎09-07-2013

Here are couple of links to explain more in detail:

Basically with the mismatch you are adding inconsistency to the ports. If you run "show spanning-tree inconsistentports" you will see the ports will show up there.

Not sure about the other switch are you pruning VLAN1, do you may be have loopguard setup? Also then there an issue of inter operability between Nortel and Cisco's implementation of STP.

https://supportforums.cisco.com/thread/2151150

http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a00801d11a0.shtml

http://www.informit.com/library/content.aspx?b=CCNP_Studies_Switching&seqNum=25

Hope this helps a bit.

Note: It is a good idea I believe to keep things simple and standard. Such as matching Native VLAN's, port configurations, port channel configs, STP modes etc. Will make troubleshooting much easier.

Aninda Chatterjee · ‎09-07-2013

Hi Andy,

Both Rolf and Mohammad have given very good information here. It is quite important to understand how the Cisco proprietary PVST+ (or Rapid-PVST+) really works and more interestingly, how they interoperate with third party switches - in this case, Nortel switches.

Both PVST+ and Rapid-PVST+ send their BPDUs using a destination mac-address which is Cisco proprietary and which is not understood by most third party switches (there are some third party switches which have the capability to turn on a feature to understand these BPDUs). This creates a problem with Nortel switches in the middle because they will not understand BPDUs for VLANs 2 and above (VLAN 1 would work fine because an IEEE BPDU would be sent out for this as well) and they simply flood these out without processing them.

Any Cisco switches behind the Nortels would now be receiving these BPDUs and will be processing them.

A snippet of the errors you saw:

*Mar 1 00:57:44.483: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 23 on GigabitEthernet1/0/12 VLAN1.

*Mar 1 00:57:44.483: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking GigabitEthernet1/0/12 on VLAN0001. Inconsistent local vlan.

This indicates that the local switch received a BPDU which was untagged, however, the VLAN ID in the TLV of the BDPU was set to 23. This is being compared to the native VLAN on the local switch's receiving interface which is 1. Because of the mismatch, spanning-tree blocks the interfaces for both the VLANs.

From your topology, you have two Cisco switches on top - can you post the show run interface output for all the interfaces from these switches that go to the Nortel switches please?

Regards,

Aninda

Aninda Chatterjee · ‎09-07-2013

I've had native vlan mismatches before and it never caused the ports to shutdown or a STP mismatch.

Native VLAN mismatches would not cause a port to be shut down, but it would most certainly cause spanning-tree to block the mismatched VLAN pair for that interface.

There was quite an interesting discussion on how this happens sometime back -

https://supportforums.cisco.com/message/4001268#4001268

Please feel free to ask further!

Regards,

Aninda

Andy White · ‎09-08-2013

Thanks guys for you info, I have a couple of questions. What confuses me I'm already done this and it is production.

I configured the Cisco switch first and plugged in and plugged in and all was ok, then I went on to the Nortel manager and selected the 4 options and hit apply and then the issue started.

The Cisco switch I added must be using native VLAN 1 but so is the other Cisco switch I added that worked

1.) So in summary the Nortel switch didn't know what to do with the this BPDU and flooded it using multicast so both Cisco stacks (virtual server farm) received it and saw it as a native vlan mismatch between vlan 1 and 23 (should of been 23?) and shutdown the etherchannes which were ports 12 and 24?

2.) Is a native VLAN mismatch's default behavior to err-disable the port and shut it down then?

3.) When this happened I pulled the Cisco switch I added off the network then I consoled onto one of the Cisco 3750 stacks where the etherchannel was down (ports 12 & 24) and did a 'shut' and 'no shut' on the ports to find they err-disabled immediately again, why?

4.) I then pulled the etherchannel cables out (trunk ports 12 & 24) so there was no connection to the Nortels and did a 'shut' then 'no shut' and the ports err-disabled again, why would they do they if they we no longer connect to anything?

5.) I had to reload the switch (stack) to get it all working. I guess it clear the mac address table, is there a 'soft' way to clear the table?

Thanks in advance

Rolf Fischer · ‎09-08-2013

Andy,

I've been reading some Nortel documentation to understand their Layer-2 concepts and I have to say that there are some things I still don't understand entirely.

Nortel Switches also have some proprietary STP enhencaments, like STG (ST groups). STG1 BPDUs are normal IEEE-untagged BPDUs, STG 2-64 send tagged BPDUs with IEEE-format. So the only common ST is between Cisco VLAN1 and Nortel STG1, the rest (Cisco PVST-instances and Nortel STG >1) are partitioned. I wonder what a Catalyst Switch does when it receives a tagged IEEE BPDU ...

Could you tell us if you use STG others than STG1?

The screenshot above shows the trunkport configuration to the added helpdesk-switch, right?

The "DiscardUntaggedFrames" option is enabled, so I'd expect that received control traffic like CDP will be dropped here.

However, we can find syslog messages like:

"06/09/2013 15:40:31" 192.168.28.250 192.168.28.250 Warning "29395: 000042: Native VLAN mismatch discovered on GigabitEthernet1/0/24 (23), with Helpdesk_Switch.CBSO GigabitEthernet1/0/12 (1). (ISCSI-3750-A-1)" CDP-4-NATIVE_VLAN_MISMATCH

Almost all Layer-2 control protocols are always send untagged, even if VLAN1 is manually pruned or another VLAN is configured as Native-VLAN. So how could that happen ...

The very first syslog message is a SW_MATM-4-MACFLAP notification, could you tell us if that MAC-address came form the helpdesk-switch or from the original LAN (Nortel, c3750 stacks)?

After that, member-ports of the etherchannel went down:

"06/09/2013 15:43:44" 192.168.28.251 192.168.28.251 Error "26108: Interface GigabitEthernet1/0/24, changed state to down" LINK-3-UPDOWN 320517882

This could have been caused by the Nortel's LoopDetection Feature:

On a per-port basis, the Loop Detection feature detects MAC addresses that are looping from one port to other ports. After a loop is detected, the port on which the MAC addresses were learned is disabled. Additionally, if a MAC address is found to loop, the MAC address is disabled for that VLAN.

Perhaps analysing the syslogs can help us reconstructing what happend and what was cause and effect.

Generally, a port is put in ErrDisabled to avoid even worse situations. I think there must have been more than just a mismatch of native VLANs .

Best regards

Rolf

[edit]

I just noticed that the syslog messages in the original post are not in ascending timestamps-order.

So the MAC flap was not the first event ...

Andy White · ‎09-08-2013

I'd like to lab this topology, but don't have any spare Nortel switches, I have spare Cisco switches, is there a way to cause this behavour between 2 Cisco switches?

I just trunked 2 switches and the native vlan's were different and all was good still.

This still confuses me, I guess a multi vendor LAN is every mans nightmare, we hope to replace the 2 Nortels witch 2 x 3850x's

Any ideas on these?

1.) So in summary the Nortel switch didn't know what to do with the this BPDU and flooded it using multicast so both Cisco stacks (virtual server farm) recieved it and saw it as a native vlan mismatch between vlan 1 and 23 (should of been 23?) and shutdown the etherchannes which were ports 12 and 24?

2.) Is a native VLAN mismatch's default behavior to err-disable the port and shut it down then?

3.) When this happened I pulled the Cisco switch I added off the network then I consoled onto one of the Cisco 3750 stacks where the etherchannel was down (ports 12 & 24) and did a 'shut' and 'no shut' on the ports to find they err-disabled immediately again, why?

4.) I then pulled the etherchannel cables out (trunk ports 12 & 24) so there was no connection to the Nortels and did a 'shut' then 'no shut' and the ports err-disabled again, why would they do they if they we no longer connect to anything?

5.) I had to reload the switch (stack) to get it all working. I guess it clear the mac address table, is there a 'soft' way to clear the table?

Rolf Fischer · ‎09-08-2013

1) The Native VLAN mismatch made the Catalysts block VLAN 1 and 23 on Po1

15:42:26 SPANTREE-2-RECV_PVID_ERR (Result: VLAN 1 and 23 are blocked on Po1)

Since Po1 was the only ST active port for VLANs 1 and 23, the SVIs went down at the same moment.

15:42:26 LINEPROTO-5-UPDOWN (Interface VLAN 1)
15:42:26 LINEPROTO-5-UPDOWN (Interface VLAN 23)

1 1/2 minutes later, the etherchannel was errdisabled because a EC misconfiguration was assumed.

15:43:42 PM-4-ERR_DISABLE (Po1, channel-misconfig STP)

2) No, the mismatched VLANs will be blocked but the ports are not errdisabled. Technically it was an etherchannel/STP protection mechanism that put the EC in errdisable.

3-4) Perhaps the Nortels unbundeld the channel. Another interesting link: Errdisable Port State Recovery on the Cisco IOS Platforms

Recover a Port from Errdisabled State

After you discover why the ports were disabled, fix the root problem. The fix depends on the triggering problem. (...)

I know, not really a satisfactroy reply, but the best I can offer.

Best regards

Rolf

P.S.: Setting up this scenario in a lab would be very interesting, but I think you'll need some third-party switches.

Andy White · ‎09-09-2013

Thanks for the time you have spent on this issue.

As you know I pulled te trunk cables out of the 2 x 3750 stacks and tried a 'shut' & 'no shut' on ports 12 & 24 and it still err-disabled, rather than reload the 2 switches which fixed the issue, could I of don't something less drastic? I can only think the reload cleared the Mac/CAM table which helped.

So in brief this was down to the Nortel's not understanding the BPDU sent from the helpdesk switch and flooding it so the 2 Cisco stacks received on their po1 etherchannels and as it was their default behavior to shut down because they expected a Native VLAN of 1, but got 23?

Regards

Added switch to network and err-disabled part of the network, why?