this is what you need to do,
in your out to self inspect the following traffic and if you have self to out do the same
udp 500 - isakmp
udp 4500 - if vpn with client or hardware which is behind nat device
ip 50 - esp
i think you have already done these, just make sure these ports are both src and dst
now in your out to in (probably in your case internet to vlan)
inspect ip traffic from remote network to your network
and preferably do the same in in to out (vlan to internet)
this is it
hope it helps