IP Phone SSL VPN to ASA using AnyConnect

[Jason Burns]

• Purpose
• Functional Overview
• CUCM Places ASA Certificate Hash and VPN URL in Phone Config
• Phone Connects to ASA
• Software Versions
• Phone Models
• CUCM Configuration
• Phone Configuration
• ASA Configuration
• Additional Requirements:
• Troubleshooting Steps
• Common Issues

Purpose

Starting in CUCM 8.0.1 and IP Phone Firmware 9.X, IP Phones are now able to directly connect to an ASA using the AnyConnect VPN. This  document will help address some common issues encountered during intial configuration. This guide will act as a supplement to the Official IP Phone VPN Documentation.

Functional Overview

Before we get into versions and model numbers let's look at how the feature works.

CUCM Places ASA Certificate Hash and VPN URL in Phone Config

Before the phone is ready for VPN, it must first be provisioned using the internal network. This requires direct access to the CUCM TFTP server.

The first step after the ASA is fully configured is to take the ASA HTTPS Certificate and upload it to the CUCM server. This allows the CUCM server to build an IP phone config file that tells the phone how to get to the ASA. The CUCM requires some additional configuration to associate the uploaded certificate with a VPN Profile that can be assigned to the phone.

Here is an example of the IP Phone VPN section of a phone's config file after performing the required configuration:

jasburns@jasburns-gentoo /home/jasburns $ tftp 14.48.44.80
tftp> get SEP0011215A1AE3.cnf.xml.sgn
Received XXXX bytes in 0.0 seconds
jasburns@jasburns-gentoo /home/jasburns $ cat SEP0011215A1AE3.cnf.xml.sgn
..........
<vpnGroup>
[Some Lines Omitted]
<addresses>
<url1>https://X.X.X.X/PhoneVPN</url1>

</addresses>

<credentials>

<hashAlg>0</hashAlg>

<certHash1>1eD9l3VEI9DGWQGKlNBGE1bRhUg=</certHash1>

</credentials>

</vpnGroup>

Note that the URL is printed exactly as entered on the VPN Gateway Configuration page in CUCM. Make sure the IP Phone can resolve this address.

Even more interesting is the Cert Hash. The IP phone configuration does not contain the entire certificate, merely a SHA1 Base64 encoded hash of the certificate.

You can compare the certificate hash in the IP phone configuration file to the cert hash of the actual file on the ASA or CUCM if you copy it to a computer running OpenSSL (either Windows, Linux, or Mac)

$ cat r2800.cisco.com.pem

-----BEGIN CERTIFICATE-----

<Base64 value of the cert omitted>

-----END CERTIFICATE-----

openssl x509 -in r2800.cisco.com.pem -noout -fingerprint

SHA1 Fingerprint=D5:E0:FD:97:75:44:23:D0:C6:59:01:8A:94:D0:46:13:56:D1:85:48

This is the SHA1 Fingerprint in Hexadecimal form. In the configuration file this value is instead printed as the Base64 value. I used the following website to convert from Hex to Base64:

http://tomeko.net/online_tools/hex_to_base64.php

This method can be used to verify the certificate loaded onto and presented by the ASA matches the certificate hash loaded into the phone.

Phone Downloads Configuration

This part is extremely important. The phone must download the configuration (including the certificate hash in Base64) while it is inside the network and has direct access to the CUCM TFTP server.

The phone has to be provisioned inside the network before it can be moved outside the network and use the VPN feature.

Phone Connects to ASA

After internal provisioning has been completed, the phone can be moved to the external network for VPN access. Here the Corporate Phone has been moved to a Home location.

Depending on the phone's configuration it will either automatically attempt to connect to the VPN gateway, or will connect once manually initiated. If auto network detect is enabled, the phone will try to ping the TFTP server. If there is no response to this ping request the phone will automatically bring up the VPN process on the phone.

The phone connects on TCP port 443 over HTTPS to the ASA. The ASA responds back with the configured certificate, hopefully the same certificate uploaded to CUCM. In additional TCP 443 (Transport Layer Security, or TLS), the phone will also connect on UDP 443 for DTLS (Datagram Transport Layer Security).

Phone Verifies Presented Certificate

The phone console logs show us the hash of the certificate that the ASA presents in Hex form:

3943: INF 18:10:22.354209 VPNC: vpnc_save_to_file: wrote: </tmp/leaf.crt>, 479 bytes

3944: NOT 18:10:22.355351 VPNC: cert_vfy_cb: peer cert saved: /tmp/leaf.crt

3945: NOT 18:10:22.361892 SECD: Leaf cert hash = D5E0FD97754423D0C659018A94D0461356D18548

3946: NOT 18:10:22.362574 SECD: Hash was found in the trust list

3947: NOT 18:10:22.400294 VPNC: VPN cert chain trusted

These messages show us that the phone was able to validate the certificate that the ASA presented. The cert presented matched the hash in the configuration file.

At this point the phone will establish an SSL session with the ASA and continue setting up the VPN tunnel.

All communication will now flow between the phone and the ASA in an encrypted tunnel. Once the traffic reaches the ASA it will be decrypted and forwarded along to any location in the network that the phone would like to connect to.

The beauty of this solution is that the phone obtains an address on the Internal network that is typically not filtered. The phone can connect using SCCP, SIP, HTTP, HTTPS to any server inside the Corporate Network. This allows advanced phone services and features to function that might not work through ASA Phone Proxy.

Software Versions

CUCM >= 8.0.1.100000-4

IP Phone >= 9.0(2)SR1S - SCCP

ASA >= 8.0.4

Anyconnect VPN Pkg >= 2.4.1012

Note: A "Premium" license and an "AnyConnect for Cisco VPN Phone" license is required.  The part number for the "AnyConnect for Cisco VPN Phone" is L-ASA-AC-PH-55XX= where XX = 05,10,20,40,50,80.

Phone Models

7942 / 7962 / 7945 / 7965 / 7975 / 8961 / 9951 / 9971. For a complete list of supported phones in your CUCM version go to:

https://<CUCM Server IP Address>:8443/cucreports/systemReports.do

Unified CM Phone Feature List

Generate a new report

Feature: Virtual Private Network

http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/security/8_5_1/secugd/secuvpn.html

CUCM Configuration

The following document provides a complete set of configuration  tasks required to configure CUCM for this feature:

http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/security/8_5_1/secugd/secuvpn.html

Note: Please make sure the URL for the VPN Gateway contains the full and correct address to reach the IP Phone VPN tunnel-group on the ASA.

Phone Configuration

1. Use a supported phone model per the CUCM Supported Models / Features report.
2. Register the phone to the CUCM server on the Internal network
3. Configure the IP phone with a TFTP server manually.
4. Move the phone to the external network.

ASA Configuration

Configure Anyconnect VPN access on ASA to provide network access.

See  http://www.cisco.com/en/US/products/ps6120/products_configuration_example09186a00808efbd2.shtml  for example configuration.

The lateset CUCM Security Guide also provides sample ASA configuration.

Additional Requirements:

1. The ASA must have the AnyConnect for Cisco VPN Phone Licensed feature enabled.  Licensing info can be found using show version command.
2. Group-policy must not be configured with split tunnel or split exclude.  Only tunnel all is the supported tunneling policy
3. The tunnel-group used can not be the DefaultWEBVPNGroup.   Create another tunnel-group and use "group-url https://x.x.x.x/phonevpn enable to map to the correct tunnel-group.
4. DTLS must be enabled and negotiated for operation.  This  requires both tcp/443 and udp/443 to be open and allowed on all devices between the ASA and the phone.

 

Troubleshooting Steps

1. Plug the phone into the internal network.  This will test whether the phone's configuration works prior to adding VPN.
2. Connect with AnyConnect on a PC from the outside to the ASA.  This will confirm that the ASA is configured correctly for Anyconnect
3. From the connected PC try to ping the TFTP server and CUCM server.  This will test basic ip connectivity to the two servers.
4. From the PC try to download the TFTP config file for the phone in question "tftp -i <TFTP Server> GET  SEP<Mac Address>.cnf.xml"  This will test that the tftp service is reachable and serving files.
5. From the PC try to telnet to TCP Port 2000 on the CUCM server "telnet <CUCM IP> 2000". This should immediately come back with a new line and a blank cursor.  This will test connectivity to the CUCM SCCP port, for SIP registrations use port 5060 instead.

 

Common Issues

1. One-way or no voice.  The phone registers and makes calls but no audio is heard.  Confirm routing between the two phone/rtp stream endpoints.
2. Auto Network Detect does not reliably work in IP Phone Firmware 9.0(2), but does work as expected in 9.2(1).
   1. Auto Network Detect allows the phone to detect whether it is inside or outside the network. If outside it will bring up the VPN, if inside, it will connect directly.
   2. The phone uses a series of pings to the TFTP server to determine whether it is outside the network. If pings to the TFTP server fail, the VPN GUI will be brought up on the phone and the phone will attempt to access the VPN URL.
3. Username and Password authentication from the phone does NOT support the SPACE character in either the username or the password.

[parviz]

Thanks a lot

[poidogpon]

Can anyone confirm that they got Video to work on a 9971 using this method? Is there any official Cisco documentation regarding video over VPN phones? Bandwidth requirements, etc? I've got the 9971 making  calls, but video just comes up black.Any help is appreciated.

[Jean-Christophe Majchrzak]

Hi, I confirm it works. Check if your camera is powered on and also your region settings. Using internet sometimes upload bandwidth is insufficient. You should create a dedicated site for VPN phones in order to manage bandwidth. At least you need 400 kbps for audio+video to have video.

JC

[churchiladmin]

I hate to resurrect a dead thread but the solution to our "'webvpn=' not found in cookie" was to disable Secure Desktop for the tunnel-group we set up for our phones.

tunnel-group VPNPhone webvpn-attributes
 authentication certificate
 group-url https://asa5520-c.cisco.com/VPNPhone enable
 without-csd

Source:
http://www.cisco.com/c/en/us/support/docs/security/anyconnect-secure-mobility-client/116162-trouble-anyconnect-vpn-phone-00.html#anc5

[Chi Fai Leung]

Hi Jason,

CUCM 9.X --- ASA ----ssl vpn ---- 8941

I tested the SSL VPN on the 8941, that is fail.

On my 7942, that is working ... Any idea on the 8941 SSL VPN Fail?

7942 / 7962 / 7945 / 7965 / 7975 / 8961 / 9951 / 9971. For a complete list of supported phones in your CUCM version go to:https://<CUCM Server IP Address>:8443/cucreports/systemReports.do
Unified CM Phone Feature List
Generate a new report
Feature: Virtual Private Network

[dallassplawn1]

Does anyone know if there is any way to adjust the anyconnect timeout limitation on the phone?

We have several users that use these phones as their primary endpoint.  Most of the time the vpn works great, however, if the user loses internet connection for 3 minutes or less, the phone will attempt to reestablish a vpn...but only for about 2 minutes.  After the 2 minute mark, the phone doesn't attempt to reconnect to the vpn automatically, instead...the user gets a retry prompt on the phone.  I can understand why there is limitation on the phone, but I would like to see the phone attempt for a little longer than 2 minutes...or at least try again instead of giving up after 2 or 3 minutes.

[Chi Fai Leung]

Hi,

I deployed the 3X VPN phones and two phones cannot show the calling/called history. That mean when user called/received a call, which will not show up on the call history, but that would not happen in the internal network. Tried to replace the phone and record ... any idea? 

[Joseph Martini]

Make sure you have set an alternate TFTP on the phone (https://tools.cisco.com/bugsearch/bug/CSCum10683/?reffering_site=dumpcr).  You likely need to manually set this because the external DHCP server won't provide a valid option 150.

[don.click1]

Are there any known limitations or anything that would cause configuration modifications from UCM to not be deployed to the remote phone, over the VPN tunnel?  I have a couple of 7965 phones that when on the corporate network directly, can change backgrounds, ring tones, etc - but if they are connected via VPN, they cannot.  If I make a change to the phones settings while connected over VPN, only settings changing the DN or number of lines seem to take. The other settings show in UCM, but not on the vpn connected phone.  I opened a case on this, but we haven't gotten anywhere yet.

tia!

[Dan Isai Cortez Castro]

Can I have, AnyConnect Essentials license and L-ASA-AC-PH-55xx license ? or

it´s necesary to have the license AnyConnect  premium and L-ASA-AC-PH-55xx?

[Jay Young]

Dan,

Anyconnect licensing has changed over the last year.  You'll need to purchase an organization-wide "plus" license.  This will enable you to get an activation-key for your ASA that will enable phone-based sslvpn.

See page 4

http://www.cisco.com/c/dam/en/us/products/security/anyconnect-og.pdf

-Jay

[luke.maxwell]

Hi,

We have around 30 VPN phones connected to our system via an ASA5512 on a single internet link. We are deploying a new pair of 5515x ASA's onto a new internet line and want to be able to migrate the connected VPN phones onto the new ASA's without havig to bring the phones back to the HQ so they can get updated configuration files.

Can this be done?

If so can you let me now what steps we need to take to support this migration?

Thanks

Luke

[Devinder Sharma]

Thanks so much Jay for this great and detailed walk thru. I am trying to achieve the same thing and have been generally successful but then for a combined case of users  laptops (not daisy chained to phones) and phones, I have challenges.

Currently customer has a single ASA running self signed cert and few 7942G phones out in the field running SSL VPN only for these phones and IPSec RA VPN for remote users. Trying in the lab to prepare to replace single ASA with a new redundant pair  of ASAs as also to add support for SSL VPN for users. That implies running a Go Daddy signed cert. I have simulated the environment in lab and then tested first with self signed certs and that worked as expected. Also tested users SSL VPN (username / password) via second tunnel-group (and group-urls are accordingly https://OutsideInterfacePublicIP/Phone and https://OutsideInterfacePublicIP  for self signed case) and phones using cert authentication with ASA self signed ID cert. That also worked with of course cert warnings for the users.

Then I uploaded Go Daddy signed Intermedaite Root and ID certs into ASA trustpoint and also applied these on the my test PBX (CUCM on a VM) via OS adminsitration and then CUCM adminsitration / VPN gateway/ group / profile and amended tunnel-groups to use  https://sslvpn.domain.com/Phone and https://sslvpn.domain.com/) and changed SSL trust-point on outside to use this new cert (we can only use one SSL cert, which is fine as long as the clients / phones have a hash of the same pushed down via CUCM).

With GoDaddy signed cert, I dont get any cert warnings on the laptop as we will expect. And this works every single time. On the phone, it worked fine as well initially couple of times ( with phone still powered thru a Power injector, with only ethernet cable unplugged / plugged back in). The debugs on the ASA and Phone console logs indicated all proceeding with sucess.

Then I did a reboot to phone and VPN authentication failed comes up.ASA will stop at "CERT API thread sleeps!" and next step of going thru tunnel-groups did not happen. Looking thru Phone console logs, shows Phone successfully resolving the VPN Group-url FQDN to the outside IP of ASA and ssl handshake starts. But then it complains after displaying the GoDaddy Root Cert "unable to get local issuer certificate"and then "certificate not trusted". However the laptop keeps working after disconnection / reconnection of SSL VPN withoit any cert warnings. After deleting the Group and tunnel groups and recreating the configuration, it again worked but then same failure issues for last 6 hours. I verified the cert hash pushed down by downloading the config file from CUCM using tftpd64 and compared the hashes from original cert and one convereted from hex to base64 and these are identical. I have deleted the ITL files on phones, did factory reset and retested and I continue to have the VPN authentication failure issues. 

I also tested with just the ID cert in the VPN gatway as phone will only compare the ASA ID cert hash and not validate by the CA cert, but same thing happens.

The ASAs have proper SSL VPN license for Cisco phones.

Client will be okay to use self signed ASA ID for phones, but we can only have a single ID cert on an interface, so this then is problem for the laptop users who dont want to see cert warnings. clearly I have some issues with phone validating the GoDaddy Root / ASA ID certs but it does work sometimes and when it worked, it took 2 to 3 minutes to login.

Any advice on this by your self or other members who may have implemented Third party certs based solution for Phones+Laptops, will be much appreciated as I am running behind the schedule.

Thanks again and I look forward to some help here.

 Devinder Sharma

[Devinder Sharma]

Hello Jason again,

Since laptop was working without any error messages and I have these cryptic messages of cert failure from phone, to me it appeares that I had wrong interpretation of  the error mesages. while I still continue getting the same VPN Authentication Failed, I changed the CUCM VPN gateway and corresponding tunnel-group to use username / password for client authentiation and this time, phone connected with no error (though I noticed that it takes like a minute to VPN, but after that it takes over 3 minutes to get its configuration downloaded, even if I disable the TFTP, Skinny inspection at ASA). I may have some issues with the Cisco_Manufacturing_CA cert or the MIC on phone is an issue. I downloaded  Cisco_Manufacturing_CA cert again from CUCM, removed this trustpoint from ASA and then recreated and authenticated Cisco_Manufacturing_CA cert again and I still get the same issue. Unfortunately I dont have any other VPN capable phone to test in lab. 

Should I assume that MICs are saved under NVRAM and that after full factory default, the MIC will be installed?

Thanks and I appreciate any / all recommendations.

[Devinder Sharma]

Hello All,

I seem to have resolved issue by following:

1. Excessive delay to connect to  CUCM after VPN connection was because of DNS issues. I had my test PBX using a DNS name and not IP address and while I assumed that on VPN with DNS server addressed supplied to VPN client via group-policy, it still was trying to use the publci DNS. I realize that customer has IP addresses for CUCM server, so did same with my lab.

2. The cert authentication issue was resolved after I removed all trustpoints other than the Go Daddy (that contains the GD Root / Intermediate chain and ASA ID Cert) and Cisco Manufacturing CA. Since we are not using LSC and it is not a mixed mode cluster, I removed the CAPF and Callmanager and CiscoRoot CA 2048 certs. Now with just the two trustpoints, everything is working.

3. Cisco recomemndation is to not use MIC, but I will only agree to that if Cisco Manufacturing CA is compromized. If a phone is lost, then customer's plan is to delete the corresponding mac address for the phone. This is what they have been doing with existing VPN phones. and they are not in any sensitive business. Any concerns with this approach please?

Thanks so much for this valuable tutorial.