QoS on Nexus 3000

[dukenuk96]

Hi all!

I'm trying to resolve the following task - we have a Nexus 3000 switch which acts as transit L2 switch between our routers, several ISPs and private WAN links. I want to preserve bandwith for routing and management protocols to prevent user-generated traffic from affecting routing adjacencies and management applications (RDP, SSH, etc..)

I have read configuration guide, several articles and watched Cisco Live videos about QoS implementation on Nexus switches. Then my brain yielded the following configuration:

!!! ACLs to catch specific traffic patterns
!
ip access-list AL-Routing-Protocols
 10 remark # BFD Echo #
 20 permit udp any any eq 3785
 30 permit udp any eq 3785 any
 40 permit tcp any any eq 3785
 50 permit tcp any eq 3785 any
 60 remark # BFD Control #
 70 permit udp any any eq 3784
 80 permit udp any eq 3784 any
 90 permit tcp any any eq 3784
 100 permit tcp any eq 3784 any
 110 remark # BFD Multihop Control
 120 permit udp any any eq 4784
 130 permit udp any eq 4784 any
 140 permit tcp any any eq 4784
 150 permit tcp any eq 4784 any
 160 permit ospf any any
 170 permit tcp any any eq bgp
 180 permit tcp any eq bgp any
!
ip access-list AL-Management-Protocols
 10 permit tcp any any eq 22
 20 permit tcp any eq 22 any
 30 permit tcp any any eq 3389
 40 permit tcp any eq 3389 any
 50 permit udp any any eq 3389
 60 permit udp any eq 3389 any
 70 permit tcp any any eq 23
 80 permit tcp any eq 23 any
!
ip access-list AL-Our-AS-Addrs-Only
 10 permit ip 169.254.0.0/16 169.254.0.0/16
!
ip access-list AL-any-IP-Traffic
 10 permit ip any any
!
class-map type qos match-any CM-QoS-Routing-Protocols
 match access-group name AL-Routing-Protocols
!
class-map type qos match-any CM-QoS-Management-Protocols
 match access-group name AL-Management-Protocols
!
class-map type qos match-any CM-QoS-Our-AS-Traffic-Only
 match access-group name AL-Our-AS-Addrs-Only
!
!
policy-map type qos PM-QoS-Edge-Class-and-Internal-pre-Mark
 class CM-QoS-Routing-Protocols
 set qos-group 5
 class CM-QoS-Management-Protocols
 set qos-group 4
 class CM-QoS-Our-AS-Traffic-Only
 set qos-group 3
!
!
class-map type queuing CM-Queuing-Routing-Protocols
 match qos-group 5
!
class-map type queuing CM-Queuing-Management-Protocols
 match qos-group 4
!
class-map type queuing CM-Queuing-Our-AS-Traffic-Only
 match qos-group 3
!
!
!
policy-map type queuing PM-Queuing-Internet-Edge
 class type queuing CM-Queuing-Routing-Protocols
 bandwidth percent 5
 priority
 class type queuing CM-Queuing-Management-Protocols
 bandwidth percent 20
 class type queuing CM-Queuing-Our-AS-Traffic-Only
 bandwidth percent 30
 class type queuing class-default
 bandwidth percent 30
!
!
interface e1/51
 service-policy type qos input PM-QoS-Edge-Class-and-Internal-pre-Mark
 service-policy type queuing output PM-Queuing-Internet-Edge

Then I try to verify what's happening on the interface:

SD2-5-8-22-L2SW-155(config)# sh queuing interface e1/51
Ethernet1/51 queuing information:
 qos-group sched-type oper-bandwidth
 0 WRR 30
 qos-group 0
 HW MTU: 1500 (1500 configured)
 drop-type: drop, xon: 0, xoff: 0
 Statistics:
 Ucast pkts sent over the port : 66122156488
 Ucast bytes sent over the port : 42691127066217
 Mcast pkts sent over the port : 46358
 Mcast bytes sent over the port : 3038776
 Ucast pkts dropped : 6363974
 Ucast bytes dropped : 9145820663
 Mcast pkts dropped : 0
 Mcast bytes dropped : 0

 Pkts dropped by RX thresholds : 0
 Bytes dropped by RX thresholds : 0
!
SD2-5-8-22-L2SW-155(config)# sh policy-map interface e1/51


Global statistics status : enabled

 NOTE: Type qos policy-map configured on VLAN will take precedence 
 over system-qos policy-map for traffic on the VLAN 


Ethernet1/51

 Service-policy (qos) input: PM-QoS-Edge-Class-and-Internal-pre-Mark 
 policy statistics status: enabled

 Class-map (qos): CM-QoS-Routing-Protocols (match-any)
 Match: access-group AL-Routing-Protocols
 set qos-group 5

 Class-map (qos): CM-QoS-Management-Protocols (match-any)
 Match: access-group AL-Management-Protocols
 set qos-group 4

 Class-map (qos): CM-QoS-Our-AS-Traffic-Only (match-any)
 Match: access-group AL-Our-AS-Addrs-Only
 set qos-group 3

 Class-map (qos): class-default (match-any)
 Match: any 
 set qos-group 0

 Service-policy (queuing) input: default-in-policy 
 policy statistics status: enabled

 Class-map (queuing): class-default (match-any)
 Match: qos-group 0 
 bandwidth percent 100 

 Service-policy (queuing) output: PM-Queuing-Internet-Edge 
 policy statistics status: enabled

 Class-map (queuing): CM-Queuing-Routing-Protocols (match-any)
 Match: qos-group 5 
 bandwidth percent 5 
 priority
 queue dropped pkts : 0 current depth pkts: 0
 queue dropped bytes : 0 current depth bytes: 0
 queue transmit pkts: 0 queue transmit bytes: 0
 queue max depth pkts: 0 queue max depth bytes: 0

 Class-map (queuing): CM-Queuing-Management-Protocols (match-any)
 Match: qos-group 4 
 bandwidth percent 20 
 queue dropped pkts : 0 current depth pkts: 0
 queue dropped bytes : 0 current depth bytes: 0
 queue transmit pkts: 0 queue transmit bytes: 0
 queue max depth pkts: 0 queue max depth bytes: 0

 Class-map (queuing): CM-Queuing-Our-AS-Traffic-Only (match-any)
 Match: qos-group 3 
 bandwidth percent 30 
 queue dropped pkts : 0 current depth pkts: 0
 queue dropped bytes : 0 current depth bytes: 0
 queue transmit pkts: 0 queue transmit bytes: 0
 queue max depth pkts: 0 queue max depth bytes: 0

 Class-map (queuing): class-default (match-any)
 Match: qos-group 0 
 bandwidth percent 30 
 queue dropped pkts : 6363974 current depth pkts: 0
 queue dropped bytes : 9145820663 current depth bytes: 0
 queue transmit pkts: 66122180569 queue transmit bytes: 42691145285775
 queue max depth pkts: 0 queue max depth bytes: 0

Strange things on the interface - qos-group 0 is the only one active with oper-bandwidth of 30, so part of the configuration is applied. Also I do see that policy-map for marking (type qos PM-QoS-Edge-Class-and-Internal-pre-Mark) and for queuing (type queuing PM-Queuing-Internet-Edge) are applied, but there is no traffic in them. Moreover, there are drops in default queue, but total traffic is about 150mbps on the interface and if I understand queuing correctly, it should drop packets only in case of congestion when there is traffic in other or priority queues competing for remaining bandwidth.

Please tell me what am I doing wrong? I know there are more objects like system qos and network-qos, but from configuration guide I cannot get solid understanding why and how do I need to configure them if I want to configure classification, marking and bandwidth management only on a few interfaces and not on the whole switch.

Thanks.

[Robert Hoeger]

Hi,

first of all it seems that qos-group 5-7 is already defined and in use for "internal usage" (you use qos-group 5):

"By default, queues 6 and 7 are reserved for control plane traffic and queue 5 for SPAN traffic. So you

can configure four classes along with the default class."

=> default class is qos-group 0, so only qos-group 1-4 can be configured!

You cannot see any statistics because you have to "activate" the qos-groups "internally" - see Cisco Live Breakout Session 2930 (London):

"The network-qos policy instantiates the QoS-Groups when applied to the system policy. This enables the QoS-Groups and interface statistics collection per QoS-Group."

Example:

Define the Class-Maps and match the QoS-Group values

N3K-1(config)# class-map type network-qos qos-group-1

N3K-1(config-cmap-nq)# match qos-group 1

N3K-1(config-cmap-nq)# class-map type network-qos qos-group-2

N3K-1(config-cmap-nq)# match qos-group 2

N3K-1(config-cmap-nq)# class-map type network-qos qos-group-3

N3K-1(config-cmap-nq)# match qos-group 3

Define the Policy-Map and match the Class-Maps previously defined

N3K-1(config)# policy-map type network-qos qos-groups

N3K-1(config-pmap-nq)# class type network-qos qos-group-1

N3K-1(config-pmap-nq)# class type network-qos qos-group-2

N3K-1(config-pmap-nq)# class type network-qos qos-group-3

Apply the Policy-Map to the system (this activated the groups)

N3K-1(config)# system qos

N3K-1(config-sys-qos)# service-policy type network-qos qos-groups

Best regards

Robert