Hi.
We have 9336C-FX2 and i trying to configure it for SMB-Direct RDMA using PFC. If i globally enable PFC watch-dog forced, Queue goes to state Shutdown even if watchdog is off traffic won't goes. Howewer on old 9396PX with N9K-M12PQ module everys fine.
9336C-FX2
9396PX
On the screenshots same host at the same time.
switch(config)# show queuing pfc-queue
slot 1
=======
+----------------------------------------------------+
Global watch-dog interval [Enabled]
Forced Global watch-dog [Enabled]
+----------------------------------------------------+
+----------------------------------------------------+
Global PFC watchdog configuration details
PFC watch-dog poll interval : 100 ms
PFC watch-dog shutdown multiplier : 1
PFC watch-dog auto-restore multiplier : 10
PFC watch-dog fixed-restore multiplier : 0
PFC watchdog internal-interface multiplier : 2
+----------------------------------------------------+
+-------------------------------------------------------------+
| Port PFC Watchdog (VL bmap) State (Shutdown) |
+-------------------------------------------------------------+
Ethernet1/1 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/2 Enabled ( 0x8 ) - - - - Y - - -
Ethernet1/3 Enabled ( 0x8 ) - - - - Y - - -
Ethernet1/4 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/5 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/6 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/7 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/8 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/9 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/10 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/11 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/12 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/13 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/14 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/15 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/16 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/17 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/18 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/19 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/20 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/21 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/22 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/23 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/24 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/25 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/26 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/27 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/28 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/29 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/30 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/31 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/32 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/33 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/34 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/35 Disabled ( 0x0 ) - - - - - - - -
Ethernet1/36 Disabled ( 0x0 ) - - - - - - - -
switch(config)# show interface priority-flow-control
slot 1
=======
============================================================
Port Mode Oper(VL bmap) RxPPP TxPPP
============================================================
Ethernet1/1 Auto Off 0 0
Ethernet1/2 On On (8) 196 0
Ethernet1/3 On On (8) 0 0
Ethernet1/4 Auto Off 0 0
Ethernet1/5 Auto Off 0 0
Ethernet1/6 Auto Off 0 0
Ethernet1/7 Auto Off 0 0
Ethernet1/8 Auto Off 0 0
Ethernet1/9 Auto Off 0 0
Ethernet1/10 Auto Off 0 0
Ethernet1/11 Auto Off 0 0
Ethernet1/12 Auto Off 0 0
Ethernet1/13 Auto Off 0 0
Ethernet1/14 Auto Off 0 0
Ethernet1/15 Auto Off 0 0
Ethernet1/16 Auto Off 0 0
Ethernet1/17 Auto Off 0 0
Ethernet1/18 Auto Off 0 0
Ethernet1/19 Auto Off 0 0
Ethernet1/20 Auto Off 0 0
Ethernet1/21 Auto Off 0 0
Ethernet1/22 Auto Off 0 0
Ethernet1/23 Auto Off 0 0
Ethernet1/24 Auto Off 0 0
Ethernet1/25 Auto Off 0 0
Ethernet1/26 Auto Off 0 0
Ethernet1/27 Auto Off 0 0
Ethernet1/28 Auto Off 0 0
Ethernet1/29 Auto Off 0 0
Ethernet1/30 Auto Off 0 0
Ethernet1/31 Auto Off 0 0
Ethernet1/32 Auto Off 0 0
Ethernet1/33 Auto Off 0 0
Ethernet1/34 Auto Off 0 0
Ethernet1/35 Auto Off 0 0
Ethernet1/36 Auto Off 0 0
If PFC watch-dog forced off:
switch(config)# show queuing pfc-queue detail
slot 1
=======
+----------------------------------------------------+
Global watch-dog interval [Enabled]
Forced Global watch-dog [Disabled]
+----------------------------------------------------+
+----------------------------------------------------+
Global PFC watchdog configuration details
PFC watch-dog poll interval : 100 ms
PFC watch-dog shutdown multiplier : 1
PFC watch-dog auto-restore multiplier : 10
PFC watch-dog fixed-restore multiplier : 0
PFC watchdog internal-interface multiplier : 2
+----------------------------------------------------+
+----------------------------------------------------+
Ethernet1/2 Interface PFC watchdog: [Enabled]
Disable-action : No
PFC watch-dog interface-multiplier : 0
+----------------------------------------------------+
+----------------------------------------------------+
| QOS GROUP 3 [Active] PFC [YES] PFC-COS [3]
+----------------------------------------------------+
| | Stats |
+----------------------------------------------------+
| Shutdown| 1|
| Restored| 1|
| Total pkts drained| 0|
| Total pkts dropped| 0|
| Total pkts drained + dropped| 0|
| Aggregate pkts dropped| 0|
| Total Ingress pkts dropped| 136|
| Aggregate Ingress pkts dropped| 136|
+----------------------------------------------------+
+----------------------------------------------------+
Ethernet1/3 Interface PFC watchdog: [Enabled]
Disable-action : No
PFC watch-dog interface-multiplier : 0
+----------------------------------------------------+
+----------------------------------------------------+
| QOS GROUP 3 [Active] PFC [YES] PFC-COS [3]
+----------------------------------------------------+
| | Stats |
+----------------------------------------------------+
| Shutdown| 1|
| Restored| 1|
| Total pkts drained| 0|
| Total pkts dropped| 0|
| Total pkts drained + dropped| 0|
| Aggregate pkts dropped| 0|
| Total Ingress pkts dropped| 0|
| Aggregate Ingress pkts dropped| 0|
+----------------------------------------------------+
Here 9336C-FX2 config of QoS and Interfaces:
policy-map type network-qos qos_network
class type network-qos c-8q-nq-default
mtu 9216
class type network-qos c-8q-nq3
mtu 9216
pause pfc-cos 3
class-map type qos match-any ROCE
match cos 3
policy-map type qos QOS_MARKING
class ROCE
set qos-group 3
policy-map type queuing QOS_EGRESS_PORT
class type queuing c-out-8q-q6
bandwidth remaining percent 0
class type queuing c-out-8q-q5
bandwidth remaining percent 0
class type queuing c-out-8q-q4
bandwidth remaining percent 0
class type queuing c-out-8q-q3
bandwidth remaining percent 80
class type queuing c-out-8q-q2
bandwidth remaining percent 0
class type queuing c-out-8q-q1
bandwidth remaining percent 0
class type queuing c-out-8q-q-default
bandwidth remaining percent 20
class type queuing c-out-8q-q7
bandwidth remaining percent 0
policy-map type queuing QOS_INGRESS_PORT
class type queuing c-in-q6
bandwidth remaining percent 0
class type queuing c-in-q5
bandwidth remaining percent 80
class type queuing c-in-q4
bandwidth remaining percent 0
class type queuing c-in-q3
bandwidth remaining percent 0
class type queuing c-in-q2
bandwidth remaining percent 0
class type queuing c-in-q1
bandwidth remaining percent 0
class type queuing c-in-q-default
bandwidth remaining percent 20
class type queuing c-in-q7
bandwidth remaining percent 0
system qos
service-policy type network-qos qos_network
interface Ethernet1/2
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type edge trunk
speed 40000
duplex full
no negotiate auto
service-policy type qos input QOS_MARKING
service-policy type queuing output QOS_EGRESS_PORT
no shutdown
interface Ethernet1/3
switchport
switchport mode trunk
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type edge trunk
speed 40000
duplex full
no negotiate auto
service-policy type qos input QOS_MARKING
service-policy type queuing output QOS_EGRESS_PORT
no shutdown
Here 9396PX config of QoS and Interfaces:
class-map type network-qos ROCE_NETWORK_CLASS
match qos-group 3
policy-map type network-qos ROCE_NETWORK_POLICY
class type network-qos ROCE_NETWORK_CLASS
mtu 9216
pause pfc-cos 3
class type network-qos c-nq-default
mtu 9216
class-map type qos match-all ROCE_CLASS
match cos 3
match dscp 26
policy-map type qos ROCE_POLICY
class ROCE_CLASS
set qos-group 3
policy-map type queuing ROCE_QUEUING_OUT
class type queuing c-out-q3
bandwidth remaining percent 70
class type queuing c-out-q2
bandwidth remaining percent 0
class type queuing c-out-q1
bandwidth remaining percent 0
class type queuing c-out-q-default
bandwidth remaining percent 30
system qos
service-policy type network-qos ROCE_NETWORK_POLICY
interface Ethernet2/1
switchport mode trunk
ip dhcp snooping trust
priority-flow-control mode on
spanning-tree port type edge trunk
service-policy type qos input ROCE_POLICY
service-policy type queuing output ROCE_QUEUING_OUT
interface Ethernet2/2
switchport mode trunk
ip dhcp snooping trust
priority-flow-control mode on
spanning-tree port type edge trunk
service-policy type qos input ROCE_POLICY
service-policy type queuing output ROCE_QUEUING_OUT
