WebEx Edge Audio Dead Air Problem

[RICHARD MESSINGER]

Have a customer that has implemented WebEx Edge Audio and is fairly happy with the results.  But there is a continuing issue where randomly a user will get connected to the WebEx meeting and Call in from their Desk phone.

They get prompted for the meeting number and the participant id and when they hit the # key, they get dead air instead of the connection beep and joining to the meeting.  They will hang upi and on the second try everything works.

Does not seem to be any pattern.  No congestion on the internal network that we can see and because of the randomness, it is impossible to get logs from the Expressways.

Anyone else seeing this?

We are working with TAC but unless we have the logs there isnt much they can do.

[joe.funk]

I have same problem, did you ever get a resolution?

[RICHARD MESSINGER]

WebEx DE was looking at the logs from a failed call and a good call.

On the failed call WebEx is sending out the re-invite but the customer's Expressway-e does not receive it.

DE wants to blame the customer's firewall but we are pointing out the the traffic is going through the same firewall and firewall rule set for both bad and good calls so that does not wash.

I think we are dealing with a flaky Internet traffic that either delays that re-nvite or it gets routed wrong.  I think DE should try and resend the re-invite is they do not see the 1xx response in a second and then drop the call is nothing comes back.

SR 686783360 is the case we opened.

[diggy]

We are having the exact same issue with audio from desktop ip phones connecting to Webex through our Edge Gateways and when the user enters their user ID number...dead air.  Hang up, try again and usually works...if they just hit pound it seems to always work.

The issue is random, and infrequent, but happens enough that it is a pain point.

Cisco is intent on pointing it to our firewall as the issue.  We are using a Palo firewall with no application inspection on the Edge Audio rules, and the Edge Audio rules at the top for good measure.

What firewall are you using?

[RICHARD MESSINGER]

My customer is also using a Palo Alto Firewall.

Customer's security people say that they are not seeing anything being reported as dropped but it is very hard to have a packet capture running all the time on both sides of the firewall to capture the packet stream, since it is not all the time and is not easily reproducable.

Tried to get Palo Alto TAC involved with little help as far as I can see.

Did you ever solve the problem?

[RICHARD MESSINGER]

we have not found the problem yet but there is a strong feeling that it has to do with the Palo Alto Firewall.

But we have not been able to capture packets on both sides of the Palo Alto to show that it is dropping the packet randomly.

What kind of Firewall do you have?

Did you ever find the problem?

[diggy]

Not yet.  We changed our UCM so all Webex calls aren't routed through the Edgegateway for the time being while we were in holiday blackout.  Now that blackout period is over we are going to start troubleshooting again.

Out plan to packet capture going forward will be twofold.

First we will make changes to UCM to have a limited set of VOIP desktop phones to route through the Edgegateway for Webex SIP calls.  This will limit the amount of Edge Audio traffic going through the Edgegateway and will limit the amount of data being captured.

Second we have moved our packet capture focus to the Internet router.  We have Cisco ASR routers in front of the Palo's so we have created a packet capture monitor there with ACLs to match only on our Edgegateway, CIsco Webex network list, and the Edge Audio SIP control ports (5060-5065 basically) as it is the Cisco SIP control packets are the cause.

That packet capture will run 24/7 and when we get a report of an incident we will stop the capture and export it and send it to Cisco as well as look at it ourselves.  We will look to confirm our Internet router gets the SIP control packet.

If we don't see the proper SIP control packet exchanges then that will rule out the Palos.

If we see the SIP control packets on the Internet routers, then we will completely concentrate on the Palos are the source of the issue.  Also we plan on having a packet capture going on the Palos as well as the amount of traffic will be drastically smaller as we will only have maybe 10 desktop phones using the Edgegateway so the Palo captures won't be so intense...still I hate how limited the Palo packet capture filters are...only four and you can't leverage host groups or service groups in those filters...it really hurts you.

We plan on actively troubleshooting again in the next couple weeks so hopefully will have some more info.

We are using PA5050 running 8.1.5

[techcontroller23]

Diggy, 

Can you check your Expressway logs for the following error codes?

Response-code="422"

"Reason="Session Timer too small"

[RICHARD MESSINGER]

Yes, we see these Errors and Cisco TAC confirms that these are red herring errors that are handled during the normal negociations.  We see them on good calls and bad calls. 

In doing testing last week, could not get it to fail to get something in either packet capture or error logs in Expressways or PaloAlto firewall.  One of the members of my team said that in a previous life, Palo Alto was known to randomly drop packets when it was doing Threat analysis and not report the packets being dropped when it is inspecting TCP Ports 5060-5062.  The fix that was reported was to turn off the inspection for these ports.

[techcontroller23]

Rich,
I was referring to the other commenter on this post. I know what is being said for this specific case, because "I am the customer".

[joe.funk]

I am having same issue so interested in what you find.

[RICHARD MESSINGER]

If you have a TAC case open, Can you please share so I can reference it in our case.

Our case is SR 686783360

[joe.funk]

We have two open. One is for webex edge dead air after attendee id is pressed, the other is for cisco vtc video endpoints dropping calls.

SR 687773982
SR 688067412