Connecting a Switch to a 2232 FEX

visitor68 · ‎09-05-2011

It is my understanding that the host ports on the FEXs can only have servers connected to them and not switch uplinks - like, say, from a blade switch. The reason, as I understand it, is that the host ports are hard-coded for PortFast and cannot be changed. So, of course you would never connect a dot1q inter-switch link to a port configured for PortFast. You can, but its certainly not recommended for obvious reasons.

But is that the ONLY reason?

Jerry Ye · ‎09-05-2011

It is not hard-coded for portfast/edge port. It has bpduguard is enabled and cannot be turn off.

One way to do it is configure the switch not to send BPDU, the HIF of the FEX will be able to connect to a switch. However, you can have a protential STP loop in your networ. Hense, this is not recommended.

HTH,

jerry

Viral Bhutta · ‎09-05-2011

From 5.2.1, by default the host interfaces are Layer 3. Hence you can connect it to switch if you want to keep it just Layer 3.

But if you want to keep the port as layer 2, then as Jeye mentioned, bpduguard is enabled by default.

Now when you connect a switch to FEX, it will send out a BPDU and hence the FEX port will get err-disabled.

If you want to connect a switch to those port then you need to enable bpdu filter on the switch interface which connects to FEX. This will prevent any BPDU going from switch to FEX and hence it will work out for you.

Always care should be taken when you enable bpdu filter since that will not help you if you have spanning-tree loop since you are not passing BPDUs.

For more reference:

http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_521/Configuring_the_Cisco_Nexus_2000_Series_Fabric_Extender_rel_5_2_chapter1.html#con_1046083

Hope this helps.

visitor68 · ‎09-05-2011

Thank you , gentlemen!

Let's see if I got this straight....

BPDUGUARD is enabled by default on each host port and it cannot be disabled. By the way, since BPDUGUARD is typically enabled when a port is placed in PortFast mode, I got confused with my thinking....

Anyway, so if I did want to connect a switch to a Host port, I theoretically can achieve this by enabling BPDUFILTER on the switch's uplink port. This way the switch will not send a BPDU and the host port will not be forced into err-disable mode.

Correct?

Why did Cisco take this route in the design? Why did they intend to not have any switches connected to the Host ports? If indeed the FEX modules are supposed to emulate a linecard in a chassis-based switch, why not allow them to be configured as regular access or trunk ports?

Also, if the point is to exclude a port from the spanning-tree convergence process, why not hard-code the host ports for PortFast, too? When we connect servers (non-bridges in general) to switch ports, we enable PortFast for convenience purposes since the hosts do not pose a bridging loop possibility.

I would love to have these questions answered.

Thank you!

Viral Bhutta · ‎09-05-2011

FEX host interfaces are edge ports (portfast enabled) as well as BPDUGUARD enabled.

However, the main reason you wont be able to connect a switch to FEX host interfaces is because of BPDUGUARD because that will err-disabled the port.

Hence you need to enable the bpdu filter.

FEX was introduced mainly to be at the access layer.

Architecture Flexibility

• Unified server access architecture: The Cisco Nexus 2000 Series offers a highly cost-effective access-layer architecture for 100 Megabit Ethernet, Gigabit Ethernet, 10 Gigabit Ethernet, mixed Gigabit Ethernet and 10 Gigabit Ethernet servers, Ethernet or unified fabric, physical or virtual server, and rack or blade server environments.

• Flexible physical topologies: The Cisco Nexus 2000 Series architecture allows decoupling of the Layer 1 and 2 topologies, therefore providing flexibility in designing physical architectures, including ToR, middle-of-row (MoR), and EoR deployments, while allowing quick expansion of network capacity and remote line-card portability across multiple parent switches. It is also space optimized for all these architectures.

Some outputs from my lab

SITE2-AGG1# show run int e102/1/1

!Command: show running-config interface Ethernet102/1/1------------------non-default configuration

!Time: Tue Sep 6 07:54:57 2011

version 5.2(1)

interface Ethernet102/1/1

switchport

switchport access vlan 100

SITE2-AGG1# show run int e102/1/1 all

!Command: show running-config interface Ethernet102/1/1 all-------------------Default and non-default configuration

!Time: Tue Sep 6 07:55:03 2011

version 5.2(1)

interface Ethernet102/1/1

no description

shutdown

lacp port-priority 32768

lacp rate normal

switchport

switchport mode access

no switchport dot1q ethertype

switchport access vlan 100

spanning-tree port-priority 128

spanning-tree cost auto

spanning-tree link-type auto

spanning-tree port type edge

spanning-tree bpduguard enable

no spanning-tree bpdufilter

speed auto

duplex auto

flowcontrol receive off

flowcontrol send on

no link debounce

no beacon

delay 1

snmp trap link-status

logging event port link-status default

logging event port trunk-status default

mdix auto

SITE2-AGG1#

Hope this helps

Jerry Ye · ‎09-05-2011

Yes, if you block BPDU on your switch, you shouldn't have issue to connect it to the N2K (except worry about loop).

I will not comment on the N2K design but I can suggest you to look at the below link about forwarding model of the N2K. N2K has no local switching intellegent where communication of different hosts in the same FEX will need to go through the N5K to forward traffic. I thinking is (not official) since the intellegent is still on the N5K and running STP will require allocation of switch resources, you can connect 12+ FEXs (N50x0 allows 12 FEXs max and number of FEXs allowed in the N55xx is much higher) into the N5K where if we allow all ports to become regular switchport (to listen to STP, etc.), how much memory/resource will required in the N5K? It would be a lot.

http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_521/Configuring_the_Cisco_Nexus_2000_Series_Fabric_Extender_rel_5_2_chapter1.html#con_1197054

To your last point, I believe the default configuration on the N2K's HIF is spanning-tree port type edge (portfast). If you want to connect a dot1q trunk server into the N2K, you can change the HIF configuration to spanning-tree port type edge trunk.

HTH,

jerry

visitor68 · ‎09-06-2011

Thank you, folks, for your prompt answers. Really appreciate it.

I cant access the links. I am logged into the site, but it doesnt work. I keep getting "Forbidden File". I log in again and get it again....cycle... Thanks

Jerry Ye · ‎09-06-2011

Sorry, try again.

http://www.cisco.com/en/US/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_521/Configuring_the_Cisco_Nexus_2000_Series_Fabric_Extender_rel_5_2_chapter1.html#con_1197054

Regards,

jerry

Surya ARBY · ‎10-04-2011

If you want to connect a switch with redundancy to some FEX, use flexlink instead of spanning-tree on the downstream switch.

visitor68 · ‎01-06-2012

In this thread, we cleared up the notion that you cannot connect a dot1q trunk FROM A SWITCH to a FEX because each FEX Host port is hard-coded for BPDUGUARD. So, any BPDU coming from a dot1q switched downlink will force the FEX Host port into the errdisable state.

HOWEVER, I am hearing that one CAN indeed connect a dot1q trunk from a switch to a FEX now - something has changed, or not (?). I dont know.

Can anyone at Cisco please clarify this? I am working with Cisco at a c.ient site and they have shown me that you can indeed do this. But I dont know if they are engaged in a science experiment or if this is now indeed a supported design.

Thanks

Amit Singh · ‎01-06-2012

Do not get confused between the dot1q trunk port and the spanning tree running on the switches. You can run a dot1q trunk from a server NIC or CNA back to the FEX host ports. That does not mean that NIC or CNA will run spanning-tree to the ports connected to the FEX. 802.1q tagging is different than enabling the spanning-tree and sending BPDU's on a specific port. Even the some of the host PC/Laptop Nic's have capability to form a dot1q trunk to th switchport. You need 802.1q tagging on the CNA as you will be forwarding both SAN and LAN traffic on the same port and you need to tag the specific vlans for LAN and SAN traffic.

You can connect any device that is not running spanning-tree or sending the BPDU's on the ports connected to the FEX's.As mentioned by the other folks, either use BPDU filtering or Flex-links to connect any switch to the FEX ports.

Hope this helps.

Cheers,

-amit singh

visitor68 · ‎01-06-2012

Amit, I am NOT getting confused between a dot1q trunk from a NIC and a dot1q trunk from a SWITCH- that is why I specifically asked about a dot1q trunk from a switch, with the words "FROM A SWITCH" in capital letters. See above.

My question is whether we can now connect a dot1q trunk FROM A SWITCH with STP RUNNING to a Cisco FEX Host Port. The Cisco account team is saying YES and TAC is saying YES.

We need a definitive statement from Cisco.

Jerry Ye · ‎01-06-2012

It is not recommended since it is not designed for that.

You can like what Amit said, block BPDU from the switch if it is running STP or use Flexlink.

HTH,

jerry

visitor68 · ‎01-06-2012

jerry, I do not want to bl.ock BPDUs. I definitely want to keep STP enabled on both ends.....so my question is about connecting a full blown switch with STP running to a FEX Host port.....

So what is Flex Link?

Jerry Ye · ‎01-06-2012

If you are talking about regular switch running STP, then the answer is NO.

FlexLink is another L2 loop avoidance technology which doesn't use STP. It works but convergence is not as fast as STP. Here is the link you can read about FlexLink:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750x_3560x/software/release/12.2_53_se/configuration/guide/swflink.html

HTH,

jerry