Unfortunatly there is no "easy" answer to your question. It depends on many things, such as how much traffic is being policy routed. As you stated, policy routed traffic in not CEF switched and so there is a noticible performance hit. From your post it is not clear what you are policy routing... My suggestion is if you do not need to policy-route, don't. It makes configurations more complex and debugging much harder.
What I would suggest is turning off the policy routing and seeing how the latency (and CPU utilization!) changes.