Catalyst 3750 master switch fail cause network down

[jackson.ku]

Hi,

We have a compus network, please reference the attached file :

1. core layer switch is 6509-VSS

2. distribute layer switch is Catalyst 3750-12S. Totally we have 18 Catalyst 3750-12S, stack 2 Catalyst 3750-12S switches as one distribute layer switch and service 4-6 VLANs.

3. access layer switch is 2960 switch. each wire closet use 2 uplink interface to connect distribute layer switch, the spanning-tree protocol is rstp.

4. The 2 stacked Catalyst 3750-12S used Layer 3 Multichassis Etherchannel to connected to 6509-VSS. the routing protocol between 3750 & 6509 is ospf.

5. The stacked Catalyst 3750-12S have many SVI interface as the default gateway of each access layer VLAN.

Today all master switches in each stacking Catalyst 3750-12S power off due to the UPS issue, is cause the network down, PC at access layer switch can not ping to server at 6509-VSS, the maximum network recover time some VLAN up to 30 minutes ...

Something stranges :

1. only master switch power off, not all of stacking member power off, the layer 3 multichassis etherchannel still up ( only master switch to VSS link down ) why the OSPF neighbor status change?

2. why the network down up to 30-40 minutes?? Is it cause by MAC for the SVI change because the master switch change?

I also attached syslog for one of 3750 and VSS, please mention what happen of the problem..

Best Regards,

Jackson Ku

[Leo Laohoo]

According to your VSS-Syslog, power outage started from 1512 to 1646.  Are you sure the network outage was for 30 minutes only?

What is connected to VSS G2/1/1 up to G2/1/12?

What power supply do you have for your VSS chassis?

Regarding the 3750, is the entire stack running the same IOS and the same feature set?

How many 3750 in a stack?

[jackson.ku]

Hi,

1. Yes, the network down up to 30-40 minutes only. The VSS switch have dyal power supply

2. The G2/1/1 - 2/1/12 was connected to stacking 3750 switches

3. Yes, the entire 3750 use same IOS version and feature set.

4. 2 3750 switch in a stack.

Best Regards,

Jackson Ku

[jackson.ku]

Hi,

1. Yes, the network down up to 30-40 minutes only. The VSS switch have dual power supply, and access switch - stacking distribute switch - VSS have dual attached connection.

2. The another failed UPS come online at 16:46 ..

3. The G2/1/1 - 2/1/12 was connected to stacking 3750 switches

4. Yes, the entire 3750 use same IOS version and feature set.

5. 2 3750 switch in a stack.

Best Regards,

Jackson Ku

[Leo Laohoo]

3. The G2/1/1 - 2/1/12 was connected to stacking 3750 switches

I think this could be the problem.  Let me clarify things that all the ports from the 3750 stack, switch 2 from ports 1 up to 12 is going to your VSS?  And they are NOT configured as EtherChannel, is this correct?

[jackson.ku]

Hi,

The VSS G1/1/1-2, G2/1/1-2 is connected to 3750-1 G1/0/11-12, G2/0/12 ( the 3750-1 are 2 stacking 3750 switches ), the VSS G1/1/3-4, G2/1/13-4 is connected to 3750-2 G1/0/11-12, G2/0/12 ( the 3750-2 also th 2 stacking 3750 switches ), and we have config Etherchannel..

[Leo Laohoo]

3. The G2/1/1 - 2/1/12 was connected to stacking 3750 switches

I've checked your config.  GigabitEthernet1/0/11 and 12,  GigabitEthernet2/0/11 and 12 are configured as EtherChannel.

[Kishore Chennupati]

when the Master switch in the stack fails..i believe the stack  flushes the control plane but contiue layer 2 traffic .Hence re-learns the adjacencies.

I can be corrected if I am wrong. Also, from your VSS logs looks like your power supply on the 6k also has failed over. Did all of this happen at once?

As Leo suggested the outage is for more than an hour frmo what I can see...

regards

[jackson.ku]

Hi,

Yes, all of this happen at once. The UPS failed at 15:12, and all master switch at 3750 stack and one of the VSS power supply down. The VSS have dual power supply, another member in 3750 stack elect to new master switch.

Today we build a lab to emulate the problem ( only 2 2960 switches, 2 stacking 3750 switches, 1 3560 switch as VSS ), but no problem happening ....

Best Regards,

Jackson Ku