cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
236
Views
0
Helpful
0
Replies

Cisco Nexus 9K Priority flow control don't work

KLon
Level 1
Level 1

Hi.

We have 9336C-FX2 and i trying to configure it for SMB-Direct RDMA using PFC. If i globally enable PFC watch-dog forced, Queue goes to state Shutdown even if watchdog is off traffic won't goes. Howewer on old 9396PX with N9K-M12PQ module everys fine.

9336C-FX2

KLon_0-1712922647233.png

9396PX

KLon_1-1712925619635.png

On the screenshots same host at the same time.

 

switch(config)# show queuing pfc-queue

slot 1
=======


+----------------------------------------------------+
Global watch-dog interval [Enabled]
Forced Global watch-dog [Enabled]
+----------------------------------------------------+

+----------------------------------------------------+
Global PFC watchdog configuration details

PFC watch-dog poll interval : 100 ms
PFC watch-dog shutdown multiplier : 1
PFC watch-dog auto-restore multiplier : 10
PFC watch-dog fixed-restore multiplier : 0
PFC watchdog internal-interface multiplier : 2
+----------------------------------------------------+

+-------------------------------------------------------------+
| Port PFC Watchdog (VL bmap) State (Shutdown) |
+-------------------------------------------------------------+
Ethernet1/1 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/2 Enabled ( 0x8 ) - - - - Y - - -
Ethernet1/3 Enabled ( 0x8 ) - - - - Y - - -
Ethernet1/4 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/5 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/6 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/7 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/8 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/9 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/10 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/11 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/12 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/13 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/14 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/15 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/16 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/17 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/18 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/19 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/20 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/21 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/22 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/23 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/24 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/25 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/26 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/27 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/28 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/29 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/30 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/31 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/32 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/33 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/34 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/35 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/36 Disabled ( 0x0 ) - - - - - - - -

 

switch(config)# show interface priority-flow-control

slot 1
=======

============================================================
Port Mode Oper(VL bmap) RxPPP TxPPP
============================================================

Ethernet1/1 Auto Off 0 0
Ethernet1/2 On On (8) 196 0
Ethernet1/3 On On (8) 0 0
Ethernet1/4 Auto Off 0 0
Ethernet1/5 Auto Off 0 0
Ethernet1/6 Auto Off 0 0
Ethernet1/7 Auto Off 0 0
Ethernet1/8 Auto Off 0 0
Ethernet1/9 Auto Off 0 0
Ethernet1/10 Auto Off 0 0
Ethernet1/11 Auto Off 0 0
Ethernet1/12 Auto Off 0 0
Ethernet1/13 Auto Off 0 0
Ethernet1/14 Auto Off 0 0
Ethernet1/15 Auto Off 0 0
Ethernet1/16 Auto Off 0 0
Ethernet1/17 Auto Off 0 0
Ethernet1/18 Auto Off 0 0
Ethernet1/19 Auto Off 0 0
Ethernet1/20 Auto Off 0 0
Ethernet1/21 Auto Off 0 0
Ethernet1/22 Auto Off 0 0
Ethernet1/23 Auto Off 0 0
Ethernet1/24 Auto Off 0 0
Ethernet1/25 Auto Off 0 0
Ethernet1/26 Auto Off 0 0
Ethernet1/27 Auto Off 0 0
Ethernet1/28 Auto Off 0 0
Ethernet1/29 Auto Off 0 0
Ethernet1/30 Auto Off 0 0
Ethernet1/31 Auto Off 0 0
Ethernet1/32 Auto Off 0 0
Ethernet1/33 Auto Off 0 0
Ethernet1/34 Auto Off 0 0
Ethernet1/35 Auto Off 0 0
Ethernet1/36 Auto Off 0 0

If PFC watch-dog forced off:

switch(config)# show queuing pfc-queue detail

slot 1
=======


+----------------------------------------------------+
Global watch-dog interval [Enabled]
Forced Global watch-dog [Disabled]
+----------------------------------------------------+

+----------------------------------------------------+
Global PFC watchdog configuration details

PFC watch-dog poll interval : 100 ms
PFC watch-dog shutdown multiplier : 1
PFC watch-dog auto-restore multiplier : 10
PFC watch-dog fixed-restore multiplier : 0
PFC watchdog internal-interface multiplier : 2
+----------------------------------------------------+

+----------------------------------------------------+
Ethernet1/2 Interface PFC watchdog: [Enabled]
Disable-action : No
PFC watch-dog interface-multiplier : 0
+----------------------------------------------------+
+----------------------------------------------------+
| QOS GROUP 3 [Active] PFC [YES] PFC-COS [3]
+----------------------------------------------------+
| | Stats |
+----------------------------------------------------+
| Shutdown| 1|
| Restored| 1|
| Total pkts drained| 0|
| Total pkts dropped| 0|
| Total pkts drained + dropped| 0|
| Aggregate pkts dropped| 0|
| Total Ingress pkts dropped| 136|
| Aggregate Ingress pkts dropped| 136|
+----------------------------------------------------+

+----------------------------------------------------+
Ethernet1/3 Interface PFC watchdog: [Enabled]
Disable-action : No
PFC watch-dog interface-multiplier : 0
+----------------------------------------------------+
+----------------------------------------------------+
| QOS GROUP 3 [Active] PFC [YES] PFC-COS [3]
+----------------------------------------------------+
| | Stats |
+----------------------------------------------------+
| Shutdown| 1|
| Restored| 1|
| Total pkts drained| 0|
| Total pkts dropped| 0|
| Total pkts drained + dropped| 0|
| Aggregate pkts dropped| 0|
| Total Ingress pkts dropped| 0|
| Aggregate Ingress pkts dropped| 0|
+----------------------------------------------------+

 

 

Here 9336C-FX2 config of QoS and Interfaces:

policy-map type network-qos qos_network
class type network-qos c-8q-nq-default
mtu 9216
class type network-qos c-8q-nq3
mtu 9216
pause pfc-cos 3

class-map type qos match-any ROCE
match cos 3
policy-map type qos QOS_MARKING
class ROCE
set qos-group 3
policy-map type queuing QOS_EGRESS_PORT
class type queuing c-out-8q-q6
bandwidth remaining percent 0
class type queuing c-out-8q-q5
bandwidth remaining percent 0
class type queuing c-out-8q-q4
bandwidth remaining percent 0
class type queuing c-out-8q-q3
bandwidth remaining percent 80
class type queuing c-out-8q-q2
bandwidth remaining percent 0
class type queuing c-out-8q-q1
bandwidth remaining percent 0
class type queuing c-out-8q-q-default
bandwidth remaining percent 20
class type queuing c-out-8q-q7
bandwidth remaining percent 0
policy-map type queuing QOS_INGRESS_PORT
class type queuing c-in-q6
bandwidth remaining percent 0
class type queuing c-in-q5
bandwidth remaining percent 80
class type queuing c-in-q4
bandwidth remaining percent 0
class type queuing c-in-q3
bandwidth remaining percent 0
class type queuing c-in-q2
bandwidth remaining percent 0
class type queuing c-in-q1
bandwidth remaining percent 0
class type queuing c-in-q-default
bandwidth remaining percent 20
class type queuing c-in-q7
bandwidth remaining percent 0
system qos
service-policy type network-qos qos_network

interface Ethernet1/2
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type edge trunk
speed 40000
duplex full
no negotiate auto
service-policy type qos input QOS_MARKING
service-policy type queuing output QOS_EGRESS_PORT
no shutdown

interface Ethernet1/3
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type edge trunk
speed 40000
duplex full
no negotiate auto
service-policy type qos input QOS_MARKING
service-policy type queuing output QOS_EGRESS_PORT
no shutdown

Here 9396PX config of QoS and Interfaces:

class-map type network-qos ROCE_NETWORK_CLASS
match qos-group 3
policy-map type network-qos ROCE_NETWORK_POLICY
class type network-qos ROCE_NETWORK_CLASS
mtu 9216
pause pfc-cos 3
class type network-qos c-nq-default
mtu 9216

class-map type qos match-all ROCE_CLASS
match cos 3
match dscp 26
policy-map type qos ROCE_POLICY
class ROCE_CLASS
set qos-group 3
policy-map type queuing ROCE_QUEUING_OUT
class type queuing c-out-q3
bandwidth remaining percent 70
class type queuing c-out-q2
bandwidth remaining percent 0
class type queuing c-out-q1
bandwidth remaining percent 0
class type queuing c-out-q-default
bandwidth remaining percent 30
system qos
service-policy type network-qos ROCE_NETWORK_POLICY

interface Ethernet2/1
switchport mode trunk
ip dhcp snooping trust
priority-flow-control mode on
spanning-tree port type edge trunk
service-policy type qos input ROCE_POLICY
service-policy type queuing output ROCE_QUEUING_OUT

interface Ethernet2/2
switchport mode trunk
ip dhcp snooping trust
priority-flow-control mode on
spanning-tree port type edge trunk
service-policy type qos input ROCE_POLICY
service-policy type queuing output ROCE_QUEUING_OUT

 

 

 

0 Replies 0