Configuring QoS for Cisco 3750,3850 and 9300 models breaks dhcp and port-channel link connections

[Eric R. Jones]

This is our current configuration we are trying to deploy on our Edges.
Edge configuration:
ip arp inspection vlan $vlans
ip dhcp snooping vlan $vlans
ip dhcp snooping
ip igmp snooping

interface port-channel $port-channel
ip dhcp snooping trust
ip arp inspection trust

class-map match-all C2_VOICE
match ip dscp 47
class-map match-all VOICE
match ip dscp ef
class-map match-all VIDEO
match ip dscp af41
class-map match-all PREFERRED_DATA
match ip dscp af33

policy-map QOS_POLICY_SWITCHPORT
class C2_VOICE

priority level 1 percent 10
class VOICE
priority level 2 percent 10
class VIDEO
bandwidth percent 10
class PREFERRED_DATA
bandwidth remaining percent 10
class class-default
bandwidth remaining percent 60

int range g1/1/1-2
service-policy output QOS_POLICY_SWITCHPORT

interface range g1/0/1 - 24
spanning-tree guard root
switchport block unicast
ip verify source
storm-control unicast level bps 62000000
storm-control broadcast level bps 5000000

Distribution:
The distribution switches have "ip arp inspection trust" configured on the port-channel link.
The distribution switches don't have "ip dhcp snooping trust" enabled on them.

######################################################################
When deployed, the switch remains operational; however, our hosts lose connection to the dhcp server and/or connectivity grinds to halt to the point where trying to ssh into the device fails.

They start getting APIPA addresses and so far the only thing that fixes this is to remove the i"p arp inpsection" and "ip dhcp snooping" configurations.
Not sure which specific ones of these settings, "trust" or "vlan" could be causing the issue, but we regain connectivity.
I did some reading and it mentions having "ip dhcp relay information trusted" configured on the SVI on the distribution side.
Having this means "ip dhcp snooping trust" is not required on the port-channels.

I did some more testing and applied the "ip arp inspection trust" and "ip dhcp inspection trust" to the port channels on both the edge and the distribution. This didn't change anything as long as "ip arp inspection vlan $vlans" was configured globally.

I opened a TAC case and the engineer passed this on to me which I read, see the link below. Based on the reading I believe my configuration is workable but that I'm missing something else that may be required on the distribution/core side.

For those who know, this is a STIG requirement and not the only one required but this one focuses on QoS.

For the QoS configuration on the 9300 and 3850 please refer to the guide below, https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/16-5/configuration_guide/qos/b_165_qos_9300_cg/b_165_qos_9300_9500_cg_chapter_01.html

[hemmerling]

Looks like you are trying to STIG some devices.
Be careful, a lot of their examples are not cut and paste or will even function in some cases.

As far as your question goes, you will need to apply:
1) apply ip dhcp snooping globally
2) apply the DHCP trust command on the trunks/port-channels that go towards your DHCP server
3) assign it to the VLANs that use DHCP
4) wait for after all the computers/devices in the VLANs to request their DHCP address from the DHCP server (those detected IPs will be put in the switch local DHCP binding table)
5) then you can turn on arp inspection (and I'm guessing IPSG).
Warning if you don't save that DHCP table somewhere, then every time that switch reboots, you will have to wait for it to be filled again, and your users will not be happy.

Might be easier to just use 802.1x on your user ports, doing so makes it so you don't need to use dhcp snooping, arp inspection, and ipsg. (according to the STIGs, don't ask me why)

To use QOS the way you listed, you have to have the traffic tagged already, otherwise all your traffic will be CS0.
That is unless you have "trust device cisco-phone" on your user ports, and then all you will have is whatever tags the phones set (EF/CS5?) and CS0 on everything else (which is better than nothing as your phones will keep working if you have bandwidth issues).
I mention the last part because when you get to the point where you are creating complicated service-policies on the input from your user ports to tag that traffic, having that trust command will remove all that tagging you think you are doing, your hit counts will be going up, and you will think you have tagged traffic, but you won't, you will have voip tagged phone traffic and CS0 for everything else. Don't take my word for it, packet capture the traffic on a switch above the one you think you are tagging via input policy.

Read:
https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/16-5/configuration_guide/qos/b_165_qos_9300_cg/b_165_qos_9300_9500_cg_chapter_01.html

Look for the line "both CoS and DSCP values are set to "0" and any input policy will not take effect"

How did I find your thread, by looking up the seemingly bogus "C2_VOICE" reference, I'm guessing it's part of the "QoS DODIN Technical Profile" that also doesn't exist anywhere outside of the STIGs themselves. 

[Joseph W. Doherty]

"To use QOS the way you listed, you have to have the traffic tagged already, otherwise all your traffic will be CS0.
That is unless you have "trust device cisco-phone" on your user ports, and then all you will have is whatever tags the phones set (EF/CS5?) and CS0 on everything else (which is better than nothing as your phones will keep working if you have bandwidth issues).
I mention the last part because when you get to the point where you are creating complicated service-policies on the input from your user ports to tag that traffic, having that trust command will remove all that tagging you think you are doing, your hit counts will be going up, and you will think you have tagged traffic, but you won't, you will have voip tagged phone traffic and CS0 for everything else. Don't take my word for it, packet capture the traffic on a switch above the one you think you are tagging via input policy."

I believe you misunderstand or not fully understand the documentation you've referenced and/or QoS in general and/or what you wrote doesn't correctly convey what you really have in mind; causing, I further believe, some inaccuracies in the above statements.

For example, "That is unless you have "trust device cisco-phone" on your user ports, and then all you will have is whatever tags the phones set (EF/CS5?) and CS0 on everything else  . . .", when in your referenced documentation we have: 

Trust Behavior for Wired Ports

For wired ports that are connected to the device (end points such as IP phones, laptops, cameras, telepresence units, or other devices), their DSCP, precedence, or CoS values coming in from these end points are trusted by the device and therefore are retained in the absence of any explicit policy configuration.

I.e. the default is trust, and L3 ToS and/or L2 CoS are retained, using "trust device cisco-phone" is NOT required for connected hosts to have their packets/frames ToS/CoS markings retained.

Or:  "To use QOS the way you listed, you have to have the traffic tagged already, otherwise all your traffic will be CS0."

This statement is difficult to untangle, starting with, L3 ToS can be intentionally set to zero.  I.e. besides being a default, when ToS isn't being intentionally tagged, it can also be a ToS tag.

The listed egress policy does have a class for CS0, also used for any other non-zero ToS values not matched in another class.

Here, I suspect, you intended to describe that for the OP's QoS to work as listed (intended) packets would all need their correct/appropriate ToS markings.  It's not just "tagged" vs. CS0 because CS0 usage might be correct or incorrect.  Likewise non-zero tagged traffic hitting this egress policy might have the correct markings or incorrect markings.

Or, if we dig into "trust device cisco-phone", documentation mentions "When using this command in an AutoQoS configuration", unclear if used (if possible) outside of AutoQoS if command operates differently, and documentation also mentions "If the connected peer device is a corresponding device, input policy will take effect." which appears to contradict what you describe (unless you meant if a non-Cisco VoIP phone was being used).

[Joseph W. Doherty]

I don't believe it's the QoS (egress) configuration statements causing your issues, but the snooping, inspection and verify statements.