VOIP Latency Issue On First Call Of Day From One Subnet

[andeporter]

Hello All,

I'm hoping someone can help me out with an incredibly frustrating issue I'm facing with our Cisco VOIP system. We have approximately 25 Cisco 8851 VOIP phones at 3 separate locations. The majority of the phones work correctly, without issue, 100% of the time. One of the small VPN sites phones behave in a very strange way....

Our setup is:

192.168.12.X - Main site where CU servers live (ASA and Cisco switches)
192.168.3.X - Site where issue is occuring (across Site-To-Site ASA VPN) (ASA and 1 Cisco switch)

192.168.2.X - Another (small) remote office (we have not seen the issue yet) (ASA and 1 Cisco switch)

It seems to be that if any of the 3 phones at that location sit idle for too long (I can't pinpoint how long), the first (and only the first) call that is received or placed on those phones has about a 3-5 second latency issue:

• If they place a call from that site, the receiver cannot hear them speaking for 3-5 seconds
• If they receive a call at that site, same deal

The really strange thing is after the first call, all subsequent calls work fine until the idle time is reached again. This only happens at this one particular VPN (ASA to ASA) location. We have not experienced any other issues at this location as far and VPN connectivity issues, latency problems, dropped packets, etc. 

Troubleshooting that I've done thus far:

• Turned off skinny inspection on the ASA where the issue is occurring
• Ran a continuous ping from far end to local end (to ensure VPN tunnel is staying "active")
• Updated ASA firmware
• Reset the phones and rebuilt their configs
• Reviewed firewall config
• Reviewed switch config

I would be glad to provide configs for any hardware involved, or anything else that someone more experienced that myself might need to get to the bottom of this issue. I have searched the forums and have seen similar issues, but nothing that seems to be the same as my issue.

Thanks all for any assistance you can provide!

[andeporter]

I wanted to add additional information that I've found on this issue:

When the phone has sat idle for long enough, upon picking up the handset, there is no dial tone until after placing the first call, either internal or external. I could really use some help on this issue!

[Anthony Holloway]

I mean, I'm not security expert, but damnit if this doesn't sound like a VPN tunnel establishment issue. I think you could prove this very quickly by simply moving a phone from an affected site, to a working site, and observing the behavior of the phone. Or vice versa. I think you'll learn pretty quickly that the problem is the site, and not the phone, and thus, your VPN/Network, and not your phones.

[andeporter]

Anthony,

Thanks for the response, and I agree with you on that, but it still leaves me with the issue. I've done everything I can think of with the site-to-site VPN tunnel as well, including rebuilding it at both ends, but the problem persists (and incredibly randomly at that). I'm halfway tempted to believe that it might be an ISP issue, but I hate to pass blame without proof.

Any ASA/VOIP guys/gals out there that could chime in?

Thx,

Ap.

[Anthony Holloway]

I doubt you'll find ASA people in this forum. Maybe cross post your problem to the security/vpn forum?

https://community.cisco.com/t5/vpn-and-anyconnect/bd-p/6001-discussions-vpn

Also, what about moving the phones between sites to prove the problem does not follow the phones, and stays with the site? Is that something you've done, or can/will do?

[andeporter]

Anthony,

Fair enough. I'll cross post over there and see what I can dig up. I can take the phones offsite to another VPN location and they work as expected, which is part of what leads me to believe that it is switch or ASA related, since these are the only other 2 pieces of hardware in the mix.

Thanks again for the replies.

Thx,

Ap.

[Paul Kazzi]

On ASA you can run "show crypto ipsec sa" before you make your first call of the day and see if the local and remote idents/subnets representing your local and remote voice/phone traffic are being actively tunneled (not sure how your crypto map interesting traffic is defined). Run the same command after call is established with two way audio to see if it suddenly appears then, if it wasn't earlier.

You mentioned that you are running ping across the 2 sites, make sure that ICMP traffic is from phone VLAN to phone VLAN for the test and from phone VLAN to CUCM/s subnet/s). 

Also for troubleshooting purposes you could clear the crypto ipsec sa and isakmp sa and retest to see if it reproduces problem but may momentarily affect other traffic whilst tunnels are being reestablished.

[andeporter]

Paul,

This is very helpful information. I will give this a shot and post the results here. I've always used the GUI on the ASAs, but am learning the CLI as I go here.

Thx,

Ap.

[andeporter · Accepted solution]

To All,

I'm circling back to provide the solution to this issue, just in case anyone else runs into this problem.

After extensive discussions w/ Cisco TAC engineers from the VOIP, CUCM, and Firewall teams, we determined that the issue was ultimately that the Security Associations (SAs) inside of the IPSec VPN tunnel were going down after periods of inactivity on the VPN tunnel (ASA to ASA) between the two sites.

This is apparently by design in order to not consume unnecessary bandwidth on the tunnel and provide additional security across the VPN tunnel.

I was able to determine that the problem was with the firewalls by running the following command from the ASA CLI to "reset" the VPN tunnel between the two ASAs (which would also drop the SAs on the tunnel):

fwhostname# vpn-sessiondb logoff ipaddress x.x.x.x (with x.x.x.x being the other end firewalls VPN IP address)

This command drops and re-establishes the VPN tunnel, leaving the SAs not established (mimicking the scenario that causes the issue)

After running the command, the VOIP (Cisco 8851) would show "Phone Registering" followed by "Phone Services Unavailable" for about 10 seconds. After that, the first call would experience the 4 second delay, with all subsequent calls working with no delay. The reason for this is that the first call would actually have to establish the SAs for the subnets inside the VPN tunnel.

Cisco's solution was to set up a continuous ping between the subnet of the phone and the subnet of CUCM in order to make certain that the SAs inside the tunnel were constantly passing traffic.

I did not like this solution - while it does "work" it seems like a hack job. 

What I ended up doing was setting the phones at the remote sites to require Media Termination Point in the CUCM settings for the phone. This can be done in CUCM by navigating to: Device >> Phone >> then searching for the phone(s) at the remote site. Click on the phone you want to modify, then scroll down to the Protocol Specific Information section, and check the box for Media Termination Point Required, then click Save >> Apply Config

This will force the VOIP phone to communicate with the call manager on a steady interval, as well as keep call manager inline when sending/receiving calls (as opposed to letting the firewalls/switches handle the call audio). To me, this is a much better solution than a continuous ping. 

I hope this helps someone else out there!