I would not expect to see performance degradation just by implementing ASR configuration.
Now, are you seeing the issue when going to the servers thru the VIP or actually you see it when going directly to the servers?
Can you get simultaneous sniffer traces to see if the CSS is really adding delays?
Keep in mind that since the CSS will read upper layers (depending on your config it might read layer 4 and 5 data) is expected to see more delay added that a router or switch would.
Anyway this should not represent a problem.
Hope it helps