Cut-through vs Store and Forward in ACI / NX-OS N9K

[pille1234]

Hi there,

I am coming from an N5K - N7K setup, where I know that N5k regularly do cut through switching and N7K always store and forward.

That is relevant to know in regards to how broken frames (CRC errors) are discarded or spread through the fabric.

Now I know if I do VXLAN/EVPN on the N5K they become store and forward as well.

How is this with N9K in NX-OS and ACI-Mode? Is there a difference between spine and leaf? If they are cut-through, would we see broken frames to spread through the L2-domain as well?

Is there a differene on N9K NX-OS, if we do simple switching vs VXLAN/EVPN?

[Sergiu.Daniluk · Accepted solution]

Hi @pille1234 

ACI is configured to work in store-and-forward, thus any CRC received will be dropped locally on the leaf. However, starting with 4.2.something (i am running 4.2.5k) you can actually see if the received corrupted frame is link local generated, or if they are stomped by other cut-through switch in the network. 

This is the description of these counters from the help window:

CRC Errors (FCS Errors)
The number of valid size frames with Frame Check Sequence (FCS) errors but no framing errors.

FCS error count is the number of frames that were received with a bad checksum (CRC value) in the Ethernet frame. These frames are dropped and not propagated onto other ports.

Stomped CRC Errors (packets)
The number of corrupted frames that were cut-through switched.

Regarding Neuxs 9000 running NXOS, by default they run in cut-through. If you enable VXLAN they change the switching mode into store-and-forward.  will keep running in cut-through.

Stay safe,

Sergiu

[Prasann Magadum]

Nexus 9000 running in NX-OS operate in the cut-through regardless the VXLAN is enabled or not, below are the lab results

93180FX-07# sh switching-mode
Configured switching mode: Cut through                  <<<<<
 
Module Number                   Operational Mode
 
     1                          Cut-Through
93180FX-07# sh feature | inc enabled
icam                   1          enabled
sshServer              1          enabled
 
93180FX-07# conf
Enter configuration commands, one per line. End with CNTL/Z.
93180FX-07(config)# feature nv overlay
93180FX-07(config)# feature  vn-segment-vlan-based
93180FX-07(config)# exit
 
 
93180FX-07# show feature | inc enabled
icam                   1          enabled
nve                    1          enabled            <<< Feature enabled
sshServer              1          enabled   
vnseg_vlan             1          enabled            <<< Feature enabled
 
 
93180FX-07# show switching-mode
Configured switching mode: Cut through
 
Module Number                   Operational Mode
     1                          Cut-Through
93180FX-07#

++ changing switching mode on Nexus 9000 series doesn’t require reload. Reload is required on Nexus 5000 series

[Sergiu.Daniluk]

Hi @Prasann Magadum 

"Nexus 9000 running in NX-OS operate in the cut-through regardless the VXLAN is enabled or not" - to avoid confusion here, if I am not mistaking, what you are trying to say is that if the 9k switch is configured in cut-through (default config) and you enable vxlan, the switching mode doesn't change to store and forward automatically.

However, having the switch in cut-through with vxlan configured, and having a working vxlan setup with leafs configured in cut-through, are two different things.

As a test, I guess you need to sniff the underlay and see if the original frame is full or not. I need to think about it

EDIT: ok I know - what you can test is basically craft an ICMP packet with bad checksum and see if the packets are sent or not over the vxlan overlay. If yes, then cut-though is really functioning while VXLAN is enabled.

Thanks,

Sergiu

[Prasann Magadum]

Hi @Sergiu.Daniluk ;

Yes, the Nexus 9k Doesn't change the switching mode when VXLAN is enabled.

[Sergiu.Daniluk]

But did you tested if it really works with vxlan traffic?

[Prasann Magadum]

The VxLAN  traffic works ( ping with the MTU of 9000 with df-bit set)
Note: this is tested in the Nexus 9300 running in NX-OS mode

[SulthanGhazali]

Hi @Sergiu.Daniluk 

how to make sure ACI is using store-and-forward? do you have a command to verify this or documentation explaining this?

Thank you in advance