Some help with something I am missing with my QOS and VOIP

[MD500Pilot]

A little backstory, we have a smallish network that consists of a Nexus9000 and three Cisco 3560Gs running IOS 15.0(1). The Nexus9000 is a 10GB core switch utilized mostly for internal backup servers, Proxmox virtualization server and TrueNAS core storage equipment, the rest of the network including access points are all connected to the 3560s. The 3560s are connected to each other via dual gigabit portchannel trunks, VTP handles VLANs across the network. The 9000 is connected to one of the 3650s also via a portchannel trunk link.

We have an IP phone system that we have had for years. This is a Switchvox system running version 7.9 (latest). We have about 15 phones on the network and we use SIP trunking exclusively for dial tone. For network sanity, I have configured a number of VLANs with VLAN5 being the VLAN for our VOIP traffic. We do not have a huge amount of traffic on our network at all.

This setup has been great for quite some time, but recently we noticed a bit of an issue with our phone system, mostly with internal calls where we started to get some garbled calls. Having been a CCNP and CCDP in my former data center days, I immediately started to think that I might finally need to get some QOS in place since we were growing and adding more network devices to the mix. I assumed that a simple vlan-based QOS solution would be a quick fix, but frankly, I am now doubting both my configuration and my thought behind the QOS on the whole. 

All of my traffic is segregated administratively into VLANs based upon the traffic. 100% of my VOIP traffic is on VLAN5 with the only exception being the traffic that leaves VLAN5 for the SIP trunk to our dial tone provider. All of my internal media traffic is on VLAN50, my Corosync traffic for my Proxmox clusters are on the 10G 9000 on VLAN100, guest wifi traffic is on VLAN66, and so forth. I just assumed that somewhere traffic was beating up on VLAN5 and so the best way to address that was to apply QOS to VLAN5 and nowhere else.

So I set it all up (see below) but I am not seeing any traffic on VLAN5 matching what I configured so I am at a loss. Is it because no VOIP traffic really leaves VLAN5? This might make sense except that 100% of my inbound and outbound SIP traffic has to traverse the VLAN5 interface to get to the SIP carrier. Also, am I thinking clearly about how the QOS should behave?

Here is the 10,000' view: 

1) Turn on QOS on all of my switches (mls qos)

2) Tag 100% of the traffic in VLAN5 as EF

3) Set every physical port connected to phones, VOIP switch, and trunks carrying that traffic with 'mls qos vlan-based'

4) Magically all of the traffic tagged EF will take priority over all other traffic on my network regardless of what VLAN it is in

A lot of my reading and setting up came from THIS link here at Cosco. 

After setting everything up and letting it run overnight, I do not see any traffic on VLAN5 being matched:

AR01#show policy-map interface vlan5
 Vlan5 

  Service-policy input: VOIP-TRAFFIC

    Class-map: VOIP (match-any)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: access-group name voip-devices
        0 packets, 0 bytes
        5 minute rate 0 bps

    Class-map: class-default (match-any)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any 

So I am at a loss to figure out what I have done wrong, or where my thinking might be wrong in regards to how the QOS should work. Is turning on QOS on my switches and tagging all of my VOIP traffic even though it never leaves VLAN5 enough to prioritize this traffic in my switches enough or do I need to do more to make this work? How do I verify my configuration (shown below) to make sure it is accurate? 

Any nuggets of wisdom would be greatly appreciated.

!--- All of my VOIP is 10.200.5.0/24 on VLAN5

!--- mls qos is active
AR01#sh mls qos
QoS is enabled
QoS ip packet dscp rewrite is enabled

!---QOS Mapping
mls qos map cos-dscp 0 8 16 24 32 46 48 56


!--- My Access list
ip access-list extended voip-devices
 permit ip 10.200.5.0 0.0.0.255 any

!--- My Class/policy maps
class-map match-any VOIP
  match access-group name voip-devices
!
policy-map VOIP-TRAFFIC
 class VOIP
   set dscp ef

!--- My VLAN5 Interface configuration
!
interface Vlan5
 description Connection to VOIP network
 ip address 10.200.5.1 255.255.255.0
 service-policy input VOIP-TRAFFIC

!-- VLAN5 Stats
AR01#sh int vlan5
Vlan5 is up, line protocol is up 
  Hardware is EtherSVI, address is 001c.b086.01c3 (bia 001c.b086.01c3)
  Description: Connection to VOIP network
  Internet address is 10.200.5.1/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 2000 bits/sec, 2 packets/sec
  5 minute output rate 1000 bits/sec, 2 packets/sec
     7384576 packets input, 1411102567 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     2790480 packets output, 220615292 bytes, 0 underruns
     0 output errors, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

!-- My physical interface configs (all VOIP hardware)
interface GigabitEthernet0/5
 description Connection to Switchvox Phone System
 switchport access vlan 5
 switchport mode access
 mls qos vlan-based

!-- policy-map output for VLAN5
AR01#show policy-map interface vlan5
 Vlan5 

  Service-policy input: VOIP-TRAFFIC

    Class-map: VOIP (match-any)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: access-group name voip-devices
        0 packets, 0 bytes
        5 minute rate 0 bps

    Class-map: class-default (match-any)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

[paul driver]

Hello
Try this:
class-map access-ports_cm
match input-interface x/x -x/y

policy-map VLAN-child
class access-ports_cm
police xxxxx yyyyy exceed-action drop

policy-map VLAN-Parent
class VOIP
set dscp EF
service-policy VLAN-child

int vlan x
service-policy input VLAN-Parent

[Joseph W. Doherty]

The 3560 and 3750 series are both a bit infamous for dropping frames/packets, usually during some form of traffic bursting, as neither has a huge amount of interface buffer resources.  If QoS is enabled on these switches, especially with default QoS settings, frame/packet drops often is even worst due to the way the switch "reserves" buffer space for interfaces and/or egress queues.  (The latter, though, can often by mitigated by some QoS buffer tuning.)

That aside, if I've understood what you've written, QoS, for just VoIP traffic alone (at least w/o FQ) is ineffective.  QoS comes into use, mostly, when dealing with different types/kinds of traffic, e.g. VoIP and FTP, sharing a link/port, concurrently.

As you mention trunks, it's generally likely to be sharing different types/kinds/VLANs of traffic on those.  (BTW, this can also happen on edge port "trunks", i.e. access ports hosting a data and VoIP VLAN.)

So, the first thing you want to do is check if there are any registered drops on any ports.  Show interface might reveal such, but if it doesn't, you might need to show interface/ASIC stats (I don't recall the actual command).

Also, BTW, I recall early 15.x releases weren't without issues. One of our leading "experts", knowing 3560/3750 issues is @Leo Laohoo

I'm not saying your 15.x version has anything directly to due with the issue you're now describing, but Leo might suggest a "better/stable" version to avoid other issues.

[Leo Laohoo]

Thanks, @Joseph W. Doherty. 

What IOS version is the switch on?  Don't just say, "15.0(1)", I want the entire lot.

[MD500Pilot]

Thank you all for the help:

Is this what you are looking for Leo?

c3560-ipservicesk9-mz.150-1.SE.bin

Also, I am not opposed to upgrading switches if necessary or moving my VOIP equipment to a better switch. I do have a Cisco 9K for all of my 10G stuff, I have enough ports to move over all of my VOIP stuff even though it is just 1G connections, would that be a better option than leaving it on the 3560s?

Again, than you for your help, it is greatly appreciated.

[Leo Laohoo]

Yikes, 15.0(1)SE0.  How did anyone manage to survive on that train-wreck of a release?

12.2(55)SE and 15.0(2)SE trains are considered as the most stable IOS.  Upgrade to the latest 15.0(2)SE and see if it makes any difference. 

[MD500Pilot]

So would this be the right version:

c3560-ipservicesk9-mz.150-2.SE11.bin

[Leo Laohoo]

Yes, that is the one.

[Joseph W. Doherty]

Upgrading, may not be necessary to mitigate your issue, also, possibly, might not even need to upgrade IOS too (although, if Leo says "yikes", probably a really, really good idea to migrate to a "better" IOS version).

Again, check for interface drops.