Yes, PAT is the problem. PAT and IPSec don't work well together, since PAT uses the TCP/UDP port number to differentiate between sessions, and IPSec is not a TCP/UDP protocol (it sits right on top of IP). The connection is established successfully because that is done with ISAKMP, which is a UDP protocol, so that can be PAT'd OK. The data is sent in IPSec packets, which can't be PAT'd.
You'll have to create a static one-to-one translation for the client and then it'll work fine.
Also, in PIX 6.3 code (not released yet), there is supposed to be support for IPSec thru PAT (IPSec passthru), so watch out for it.