Please enable: inspect ipsec-pass-thru on your ASA global policy map.
Here is more information on the command:
http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/i2.html#wp1740887
However, it is only supported if you configure NAT, not on PAT.
It times out basically because UDP traffic in general has a very shortlived timeout. If it's been idle for 2 minutes (this is the default behaviour), then it clears down the session. In general, example of UDP, ie: DNS, SNMP, they are shortlive sessions and it doesn't require the session to be up for a long time.
Since you are using NAT-T to encapsulate the ESP packet to use UDP/4500, the same behaviour holds. Hence, you are seeing it gets timeout.
Depending on what device you are using to terminate the VPN and since you have NAT-T enabled, you can reduce the NAT-T keepalive to it sends keepalives more often within shorter period of time to ensure that there is continous traffic on the VPN, hence it doesn't timeout on the ASA.
Alternatively, if the VPN server supports encapsulation on TCP, that would be better as TCP is typically not short lived, and you can also change the default timeout on TCP session specific to just that VPN traffic.
Hope that helps.