The error "Error 51: Unable to communicate with the VPN subsystem" is caused by an issue with the interprocess/driver communication.
This document will attempt to track known reasons and workaround/fixes for this error.
Issue 1: PPP network interfaces (unlikely to be seen today)
There is a known issue with Mac OS X 10.4 and the earlier versions of 4.9, such as 4.9.000.0050 when using PPP based interfaces (e.g. Dial-Up). This issue is documented in Cisco bug ID CSCsd51157. A workaround when hitting this issue was to restart the Cisco IPSec VPN Services after the VPN Point-to-Point Protocol (PPP) based connection is up.
Upgrade to Mac OS X 10.4.9 or later and Cisco IPSec VPN 4.9.01.0800 or later.
Issue 2: 64bit Mac OSX Kernel
If one tries to use the Cisco IPSec VPN Client on Mac OS X running the 64-bit kernel one will receive the error 51 failure.
The Cisco IPSec cliet for Mac OS X does not support the 64 bit kernel. The VPN driver only has i386 and PPC extensions, not x86_64 extensions.
To do a test 32bit boot one can hold the 3 and 2 buttons during Apple System Boot. If this works then we have proved that your issue is the 64bit kernel.
1. If running Mac OS 10.6 or later, Use the built-in Mac OS X IPSec client.
2. If the built-in Mac OS X client is unavailable, re-configure your Mac to boot into the 32bit kernel. This issue had not been much of a concern until recently, March 2011, when Apple began to release their Macbook Pro systems configured to boot into 64bit by default