Cisco 9348 and FEX - strange oddity.

Orlando_Native · ‎10-25-2022

Ok, so I just received some 9348 switches. Since that product is NX-OS based; I wondered (without looking at the support chart FIRST) if they supported FEX's; since the NX-OS versions one can run on them do. And since they have 4 SFP+ and 2 QSFP ports.

I now know from the doc that they aren't supposed to. But my question is; WHY? The reason I'm asking is because of the following:

First; the units I received had NX-OS 7 on them. Cisco recommends 9.3.10. So I upgraded.

I then started the necessary configuration (other than the networking necessary to upgrade the firmware). I used a 9336 switch config as a template.

The 9336 DOES have FEX's attached.

What's weird is ALL the FEX associated commands from the 9336 completed WITHOUT ERROR on the 9348. I attached a FEX to the 9348; and it downloaded the FEX firmware to the FEX (2248TP-E); and the FEX came on line. The ports on the FEX then showed up on show run output; and one can configure them (as normal). The ones with attached hosts show the links up and ok.

But the caveat is that one can't seem to actually ACCESS them. Even though everything shows they SHOULD be accessible.

The real oddity; though; is that all the commands associated with FEX configuration completed completely normally; with no errors. It's always been my experience when configuring Cisco units is that IF a unit didn't support a particular function; the command would end with an error; eg breakout on a non-breakout-capable port. So I'm wondering why the FEX commands aren't flagged as errors; and why the show commands give responses that would indicate everything is fine; if the switch isn't supposed to support FEX's.

Any ideas why the switches support the commands if they're not supposed to? It would certainly be a nice selling and usability point if there was some easy way to actually get these switches supported with FEX's.

show fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------
107 ILOM FEX PR07 Online N2K-C2248TP-E-1GE FOX2045GE7X

show int brief

Eth1/53/1 1 eth fabric up none 10G(D) 107
Eth1/53/2 1 eth fabric up none 10G(D) 107
Eth1/53/3 1 eth access down Link not connected auto(D) --
Eth1/53/4 1 eth access down Link not connected auto(D) --
Eth1/54/1 1 eth fabric up none 10G(D) 107
Eth1/54/2 1 eth fabric up none 10G(D) 107
Eth1/54/3 1 eth access down Link not connected auto(D) --
Eth1/54/4 1 eth access down Link not connected auto(D) --

------------------------------------------------------------------------------------------
Port-channel VLAN Type Mode Status Reason Speed Protocol
Interface
------------------------------------------------------------------------------------------
Po107 1 eth fabric up none a-10G(D) none

--------------------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
--------------------------------------------------------------------------------
Eth107/1/1 316 eth access down Link not connected auto(D) --
Eth107/1/2 316 eth access down Link not connected auto(D) --
Eth107/1/3 316 eth access down Link not connected auto(D) --
Eth107/1/4 316 eth access down Link not connected auto(D) --
Eth107/1/5 316 eth access down Link not connected auto(D) --
Eth107/1/6 316 eth access down Link not connected auto(D) --
Eth107/1/7 316 eth access down Link not connected auto(D) --
Eth107/1/8 316 eth access down Link not connected auto(D) --
Eth107/1/9 316 eth access down Link not connected auto(D) --
Eth107/1/10 316 eth access down Link not connected auto(D) --
Eth107/1/11 316 eth access down Link not connected auto(D) --
Eth107/1/12 316 eth access down Link not connected auto(D) --
Eth107/1/13 316 eth access down Link not connected auto(D) --
Eth107/1/14 316 eth access down Link not connected auto(D) --
Eth107/1/15 316 eth access down Link not connected auto(D) --
Eth107/1/16 316 eth access down Link not connected auto(D) --
Eth107/1/17 316 eth access down Link not connected auto(D) --
Eth107/1/18 316 eth access down Link not connected auto(D) --
Eth107/1/19 316 eth access down Link not connected auto(D) --
Eth107/1/20 316 eth access down Link not connected auto(D) --
Eth107/1/21 316 eth access down Link not connected auto(D) --
Eth107/1/22 316 eth access down Link not connected auto(D) --
Eth107/1/23 316 eth access down Link not connected auto(D) --
Eth107/1/24 316 eth access down Link not connected auto(D) --
Eth107/1/25 316 eth access down Link not connected auto(D) --
Eth107/1/26 316 eth access down Link not connected auto(D) --
Eth107/1/27 316 eth access down Link not connected auto(D) --
Eth107/1/28 316 eth access down Link not connected auto(D) --
Eth107/1/29 316 eth access down Link not connected auto(D) --
Eth107/1/30 316 eth access down Link not connected auto(D) --
Eth107/1/31 316 eth access down Link not connected auto(D) --
Eth107/1/32 316 eth access down Link not connected auto(D) --
Eth107/1/33 316 eth access down Link not connected auto(D) --
Eth107/1/34 316 eth access down Link not connected auto(D) --
Eth107/1/35 316 eth access down Link not connected auto(D) --
Eth107/1/36 316 eth access down Link not connected auto(D) --
Eth107/1/37 316 eth access down Link not connected auto(D) --
Eth107/1/38 316 eth access down Link not connected auto(D) --
Eth107/1/39 316 eth access down Link not connected auto(D) --
Eth107/1/40 316 eth access down Link not connected auto(D) --
Eth107/1/41 316 eth access down Link not connected auto(D) --
Eth107/1/42 316 eth access down Link not connected auto(D) --
Eth107/1/43 316 eth access down Link not connected auto(D) --
Eth107/1/44 316 eth access down Link not connected auto(D) --
Eth107/1/45 316 eth access down Link not connected auto(D) --
Eth107/1/46 316 eth access down Link not connected auto(D) --
Eth107/1/47 316 eth access up none 1000(D) --
Eth107/1/48 316 eth access down Link not connected auto(D) --

Sergiu.Daniluk · ‎10-25-2022

Hi @Orlando_Native

Could you be more specific what is not working?

From the above command outputs it looks like you have one interface which is up on the FEX.

Cheers,

Sergiu

Orlando_Native · ‎10-25-2022

Well, first of all allegedly this *shouldn't* be working *at all* according to the documentation. The FEX support matrix says connection to a Nexus 9348 switch is not supported. Yet one can configure one; and from all show command output; it's working normally.

The port in question is connected to vlan 316; which is a 10.145.204.0/22 network. Vlan 316 IS part of the trunk definition on the uplink from the 9348 to the core (if I create a vlan 316 interface on the 9348; I can ping units on the subnet. And vice versa.)

Address of the 9348 on the 316 vlan:

/sbin/ping 10.145.204.9
10.145.204.9 is alive

Address of unit connected to the core:

ping 10.145.204.243
PING 10.145.204.243 (10.145.204.243): 56 data bytes
64 bytes from 10.145.204.243: icmp_seq=0 ttl=254 time=0.674 ms
64 bytes from 10.145.204.243: icmp_seq=1 ttl=254 time=0.679 ms
64 bytes from 10.145.204.243: icmp_seq=2 ttl=254 time=0.609 ms
64 bytes from 10.145.204.243: icmp_seq=3 ttl=254 time=0.532 ms
64 bytes from 10.145.204.243: icmp_seq=4 ttl=254 time=0.606 ms

--- 10.145.204.243 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.532/0.619/0.679 ms

Network definition on the host ILOM: (316 vlan)

commitpending = (Cannot show property)
dhcp_clientid = none
dhcp_server_ip = none
ipaddress = 10.145.207.67
ipdiscovery = static
ipgateway = 10.145.204.1
ipnetmask = 255.255.252.0
link_state = up
macaddress = 00:10:E0:CC:C0:7A
managementport = MGMT
outofbandmacaddress = 00:10:E0:CC:C0:7A
pendingipaddress = 10.145.207.67
pendingipdiscovery = static
pendingipgateway = 10.145.204.1
pendingipnetmask = 255.255.252.0
pendingmanagementport = MGMT
pendingvlan_id = (none)
sidebandmacaddress = 00:10:E0:CC:C0:7B
state = enabled
vlan_id = (none)

Doing a test from the ILOM to the subnet gateway and then to the host whose ping to the 9348 succeeded previously; and to the vlan 316 interface on the switch itself:

-> cd test
/SP/network/test

-> show

/SP/network/test
Targets:

Properties:
ping = (Cannot show property)
ping6 = (Cannot show property)

Commands:
cd
set
show

-> set ping=10.145.204.1
set: Ping test failed

-> set ping=10.145.204.243
set: Ping test failed

-> set ping=10.145.204.9
set: Ping test failed

So obviously "traffic" isn't making it's way from the FEX port to the 9348. Even though everything else seems to be.

Sergiu.Daniluk · ‎10-25-2022

Ah ok, I missed the hw model. Indeed is marked as not supported, and probably the reason of being unsupported is because forwarding doesn't work properly.

Now why NXOS allows commands even though things are not supported? Very good question.

First, the NXOS image is the same for all 9K platforms, which means the same commands, by default, are available on all models.

Second, because a feature is not yet supported, doesn't mean it will not be supported in the feature. So it is easier to add a note in config guide and remove it later if supportability will change, rather then changing the code twice (hide the commands and then add them back).

Hope this information was useful.

Take care,

Sergiu