The error "Error 51: Unable to communicate with the VPN subsystem" is caused by an issue with the interprocess/driver communication.
Purpose:
This document will attempt to track known reasons and workaround/fixes for this error.
Issue 1: PPP network interfaces (unlikely to be seen today)
There is a known issue with Mac OS X 10.4 and the earlier versions of 4.9, such as 4.9.000.0050 when using PPP based interfaces (e.g. Dial-Up). This issue is documented in Cisco bug ID CSCsd51157. A workaround when hitting this issue was to restart the Cisco IPSec VPN Services after the VPN Point-to-Point Protocol (PPP) based connection is up.
The method to restart the VPN Service is:
/System/Library/StartupItems/CiscoVPN/CiscoVPN stop
/System/Library/StartupItems/CiscoVPN/CiscoVPN start
Resolution:
Upgrade to Mac OS X 10.4.9 or later and Cisco IPSec VPN 4.9.01.0800 or later.
Issue 2: 64bit Mac OSX Kernel
If one tries to use the Cisco IPSec VPN Client on Mac OS X running the 64-bit kernel one will receive the error 51 failure.
Cause:
The Cisco IPSec cliet for Mac OS X does not support the 64 bit kernel. The VPN driver only has i386 and PPC extensions, not x86_64 extensions.
Test:
To do a test 32bit boot one can hold the 3 and 2 buttons during Apple System Boot. If this works then we have proved that your issue is the 64bit kernel.
Resolution:
1. If running Mac OS 10.6 or later, Use the built-in Mac OS X IPSec client.
2. If the built-in Mac OS X client is unavailable, re-configure your Mac to boot into the 32bit kernel. This issue had not been much of a concern until recently, March 2011, when Apple began to release their Macbook Pro systems configured to boot into 64bit by default
Further Information
Default Architecture
http://support.apple.com/kb/HT3770
These Macs use the 64-bit kernel by default in Mac OS X v10.6.
* Mac Pro (Mid 2010)
* MacBook Pro (Early 2011)
You can see which kernel you are using in System Profiler:
1. Choose About This Mac from the Apple menu.
2. Click More Info.
3. Select Software in the Contents pane.
4. Look for "64-bit Kernel and Extensions: Yes (or No)" under the System Software Overview heading.
How to Set the Boot Architecture
http://support.apple.com/kb/ht3773
Issue 3: Mac OS X 10.7 (Lion) or later
If one tries to use the Cisco IPSec VPN Client on Mac OS X 10.7 (Lion) one might encounter this error.
Cause:
The Cisco IPSec client for Mac OS X does not support 10.7 (Lion) or later.
Resolution:
Use the built-in Mac OS IPsec client. Configuration information can be found in the VPN Client for Mac OS X FAQ.
Overall: For IKEv1 IPSec support please use the Apple built-in client.
Using the Apple built-in client will help ensure support as the Mac OS Evolves. For further information on Mac OS X VPN Support, please read the VPN Client for Mac OS X FAQ.