Static port-channel forwarding on both Nexus 5000

[pablo3112]

Hi,

We are setting up two Nexus 5672 that will act as the new core managing SVIs and replacing two 6500 switches that will remain as layer 2 access switches. These two Cisco 6500 will be connected to the new core switches with two 20 Gbps VPCs and will be used as access switches.

There are 8 more access switches that will be connected using vpc to the 5672 core switches. Last, there are some stacked switches that do not support LACP and will be connected using static port channels (mode on). These switches are part of a "special" storage system and need to have two 30 Gbps port channels to each one of the Nexus switches, each Nexus will connect to the storage with two port-channels configured in mode access with vlans 16 and 18 respectively.

These two VLANs are also allowed on trunks to access switches connected to Nexus using vpc. Are these two VLANs (16 and 18) vpc or non-vpc vlans? they are configured in mode access in po 10,11,20 and 21 to the storage switches that are orphaned ports but also allowed on vpc trunks to access switches so not sure about this.

We tried to deploy this configuration and had issues because STP was not blocking the port-channels in mode on to the storage system, we expected port-channels connected to 5672_2 (highest STP priority) would be blocked by STP but as these VLANs are allowed on vpc members port-channels and allowed on the peer link i am not sure if this is expected behaviour. VPC switches are configured with peer switch.

Thanks!

[Peter Paluch]

Hi Pablo,

All the best in 2019! :)

By definition, every VLAN that is allowed on the vPC peer-link is a vPC VLAN. It does not matter whether it is used or allowed on any particular vPC; what matters is that it is permitted from both ends of the vPC peer-link. Based on this, and also confirmed by the show spanning-tree outputs, VLANs 16 and 18 are definitely vPC VLANs.

However, based on the same outputs, you did not configure the same STP priorities on both 5672 switches. Instead, your 5672_1 has the STP priority of 0 while 5672_2 has the STP priority of 4096 in all VLANs. This renders the peer-switch configuration inactive - for peer-switch to become active, it is mandatory that both your 5672 switches are configured with the same STP priority on all vPC VLANs. I strongly encourage you to configure both 5672 switches with the same STP priority.

Now, regarding the unexpected STP behavior in your case: I must admit that I cannot entirely make up the topology based on description alone; my apologies for that. Do you think you could draw and share a quick sketch of your topology that includes the two 5672 switches and the storage switches? It would help us to better understand the links and the physical redundancy, and then discuss whether the observed STP behavior was expected.

Thank you! Looking forward to hearing from you.

Best regards,
Peter

[pablo3112]

Hello Peter,

Many thanks for your reply. I attached a diagram with the STP priority changed to 0 in the two 5672 core switches. The storage switches are in the upper side (ISIS storage system).

Regards,

Pablo

[Peter Paluch]

Hi Pablo,

Thank you very much for your patience! I have reviewed the diagram - it helped me a lot.

Let us focus only on the leftmost three ISIS switches, and for my purposes, let's call them i1, i2, and i3 from left to right. Now, based on your diagram, the swBT010204 connects to i1, i2, and i3 through Po10, one link to each ISIS switch, and swBT010205 connects to i1, i2, and i3 through Po11, again one link to each ISIS switch. You have mentioned that these i1, i2, and i3 are stacked switches so I assume that from outside, the stack consisting of i1+i2+i3 acts as a single switch.

Now my question is: Do i1/i2/i3 speak STP or RSTP? If not, do they at least pass the BPDUs transparently?

The expected behavior in these scenarios would be:

• If i1/i2/i3 speak STP or RSTP, then they should put one of the port-channels (Po10 or Po11) on their end to Alternate Discarding (Blocking) port role and state. This is because, thanks to the peer-switch feature, both swBT010204 and swBT010205 send out BPDUs on orphan ports where the root bridge is the same virtual vPC-derived Bridge ID, and the sender Bridge ID is the switches' own physical BID. One of these sender BIDs will be numerically higher, hence less preferred, and the port-channel toward that switch should become blocking on the ISIS stack side.
• If i1/i2/i3 pass BPDUs transparently, then one of Po10 or Po11 should become Alternate Discarding on one of swBT010204 / swBT010205 (the one with the higher physical Bridge ID). The logic here is exactly the same as above.

So now, the issue boils down to understanding how exactly do the ISIS switch stacks handle STP/RSTP. Can you comment on this?

Thank you!

Best regards,
Peter

[pablo3112]

Hi Peter,

Your reply was really useful. I checked in the current configuration with ISIS switches connected to 6500 (see attachment) and it shows the ports blocked on the 6500-core2  switch which acts as STP secondary. Also found out that when running "show CDP neighbors" in 6500-core1 i see in the interfaces connected to the ISIS switches the 6500-core2 as neighbor and not the ISIS switches.  See below info from the output only related to the switches you named i1,i2 and i3. These switches are connected to both Cisco 6500 in Slot 1 ports 1,2 and 3 respectively. 6500 switches are connected between them through Po1 that uses interfaces 1/8 and 5/5.

sh cdp neigh

SWiBT0102-02     Ten 1/3           158             R S I  WS-C6509- Ten 1/1

SWiBT0102-02     Ten 1/3           158             R S I  WS-C6509- Ten 1/2

SWiBT0102-02     Ten 1/3           160             R S I  WS-C6509- Ten 1/3

SWiBT0102-02     Ten 1/8           167             R S I  WS-C6509- Ten 1/8

SWiBT0102-02     Ten 5/5           158             R S I  WS-C6509- Ten 6/5

swRTSCBT010201.rtsc.bcn

The local switch is the 6500-core1 (SWiBT010201) and the remote switch is the 6500-core2 (SWiBT010202). Based on this, i assume that ISIS switches are BPDU pass-through but not generating or processing the BPDUs.

Regards,

Pablo

[Peter Paluch]

Hi Pablo,

Thank you for the clarification!

It indeed seems that the ISIS switches are transparent for BPDUs. In that case, I would expect one of the N5Ks to put the port toward the ISIS stack into Alternate Discarding (Blocking) state. Was your observation different? Do you remember what states those ports were in?

Best regards,
Peter

[pablo3112]

Hi Peter,

The four port-channels were forwarding causing loops and really slow performance when accessing the ISIS storage from clients connected to 6500.

We will try again Monday 7, hope it goes better than last time ;)

Many thanks for your help. 

[pablo3112]

Hi Peter,

The four port-channels were forwarding causing loops and really slow performance when accessing the ISIS storage from clients connected to 6500.

Today we tested connecting port chaanels in Nexus 1 directly to Nexus 2 and were blocked in Nexus 2 so STP config seems to be ok. If ISIS is bpdu transparent we should see the same behaviour as in the tests.

We will try again Monday 7, hope it goes better than last time ;)

Many thanks for your help. 

[Peter Paluch]

Hi Pablo,

I was wondering how did your tests yesterday (Monday, Jan 7th) go. I hope you have been more lucky than the last time - please let us know!

Best regards,
Peter

[pablo3112]

Hi Peter,

It worked really well on Monday and we did not find issues with STP and ISIS, it blocked in the secondary switch side as expected. The only changes in the config since last time we tried was disabling bpdu guard and udld on the links from 5672 switches to ISIS as it was recommended by the storage provider.

Many thanks for your help.

Regards,

Pablo