In my opinion the biggest downside comes with troubleshooting. Anytime you add another device in the path it's one more potential area you need to consider. Part of this comes with the configuration of the load balancer itself. There are no shortage of problems that can occur when the load balancer isn't set up right. Typically these are all sorted pretty early on during testing, and from then on it's usually a non issue. It's usually pretty quick to rule out the load balancer once you're familiar with them.
As far as services go, I would not load balance pxgrid v2 (2.4) or SXP. I would deploy both of these services with standard node redundancy in active/active form.
Using ISE for TrustSec or SDA has only a slight difference that you need to be aware of with the persistence configuration. The CTS pac provisioning process does not include the calling station ID radius attribute. If you're using the Cisco Community ISE LB guides then you need to add compound persistence such as calling station ID + Nas ip to address this. I made a post addressing this with the Citrix config example. The F5 guide addresses this with a persistence fallback irule.
When dealing with a large deployment I will always choose a load balancer. I think the config and HA simplicity it brings outweighs the potential added troubleshooting steps.
I wouldn't hesitate to consider load balancing, set up right it works great.
.jpg "VIP Advisor"){kind=icon}
