I think to do this per virtual server you would have to define it via the match statements with an ACL in the class-map. You'll have to test to verify it works that way, but it seems reasonable to me.
access-list ACLMatch line 8 extended deny ip host 10.10.10.10 host 192.168.1.1
access-list ACLMatch line 16 extended permit ip 10.10.10.0 255.255.255.0 host 192.168.1.1
class-map match-any VS_IndividualACL
2 match access-list ACLMatch