RoomKit Quality Issues

[mwood000111]

Brief summary of issue is that sometimes when starting a meeting, whether it is scheduled or impromptu, the initial connection to the other client (usually another RoomKit in another location) experiences latency, lag in audio/video. Initial troubleshooting step is to disconnect the call, or reboot the RoomKit, and then immediately dial back in and the experience is perfect.  Control Hub shows packet loss, from the moment the call begins.  This isnt a burst of traffic where we see a moment or two of this type of behavior and then goes back to normal.  We have auto qos trust dscp enabled on ports for the RoomKits and on trunk links out to public internet.  Anyone see issues like this before?  TAC just says its a network problem, which is a poor excuse.  We feel this is more of an issue with the call parameters and negotiations between the two sides when the call starts, which leads us to believe its more software/code related.  Any thoughts would be appreciated.  Thanks. 

[davisjaron]

I am inclined to agree with TAC on this one that it's a network issue.  But just to dive a bit deeper, what version of code are you using?

I can't really give you a step by step instruction on how to fix this, it's going to take a fair amount of troubleshooting your network, but here is my suggestion of what to look at.  Video, just like voice, is subject to quality issues caused by jitter, latency, and packet loss.  When initializing a call, your network traffic is, presumably, being filtered through one or more firewalls along the route.  These firewalls are inspecting the packets and then either passing them on or dropping them.  If your call is being processed, I presume they are mostly being passed on, but that could easily cause hit latency or jitter, which would easily disrupt the quality of your video call.  Plus you also mentioned high packet loss being shown on the device.

Check your firewalls, make sure your SIP and RTP ports are trusted and can pass through from your Video/Voice VLAN and that on your receiving side that it trusts those packets.

Good luck on troubleshooting the issue.

[mwood000111]

Thanks for the info.  What code version are you referring to?  We are a Cisco shop so traffic is from a C9300 to ASA 5516-X and then edge is a ISR 4331.  VLAN for this traffic is wide open.  No restrictions to subnets or ports provided by Cisco.  Typical global_policy on FW so it is inspecting SIP, not RTP.  We have a service-policy for AF41 tagged traffic to provide it priority on FW interfaces.

policy-map WebEx_VoIP_Policy
description WebEx and VoIP
class WebEx
priority

service-policy WebEx_VoIP_Policy interface xxx
service-policy WebEx_VoIP_Policy interface xxx

Cisco Control Hub is reporting packet loss but we are not seeing the drops from sh policy-map int x/x/x on the switch.  While it may be network related, Im trying to rule out our side and from what I can see, I dont think there is anything restricting the flow of traffic.  Should I remove QoS from FW and or edge?  Traffic is tagged properly at the source and should retain correct tagging throughout our network.  Thanks for the info again. 

[Mike_Brezicky]

You mentioned control hub so I assume these are cloud registered. You can place as much QoS on the LAN, Firewall, edge router, etc. but you cant QoS the internet and even point to point calls between two cloud registered room kits will see internet traffic. So while it may not be an internal LAN issue, it is possible the internet link getting traffic to the webex cloud is dropping packets.

If you want an extreme option to rule out any LAN/WAN issue, you could load up a demo Video Mesh node and initiate the call - which should keep all RTP traffic on the WAN and only small amount of SIP to the cloud.

[mwood000111]

Thanks for the reply Mike.  You are correct, these are registered to the cloud. Yes, possibility of internet traffic getting dropped outside of our network, within ISPs network is totally legit.  Just want to make sure everything within our network is copacetic before we escalate with TAC.  Can you elaborate a bit more on your extreme option?  Thanks.

[Mike_Brezicky]

You can download the Video Mesh demo note from Control Hub. In short, it allows your cloud registered endpoints to keep the call on prem instead of defaulting to the cloud / internet connection. The video mesh node would be the bridge / processor for the call instead of in the webex cloud. Its a pretty neat and flexible offering. But in your case, you can run it, make a few calls utilizing the node and validate your on - prem quality.

You can read more details here:
https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cloudCollaboration/wbxt/videomesh/cmgt_b_webex-video-mesh-deployment-guide/cmgt_b_hybrid-media-deployment-guide_chapter_01.html

[mwood000111]

Thanks for the link Mike!  Ill investigate this solution to see if we can stand this up and demo/test our network.  Appreciate it.