You can add a RULE in the authorization policy allowing only users in that particular identity group to access the device providing VPN service , and you need to make sure that there is no hit for users in that particular group.
So you customize the conditions and add the identity group as part of the game.
-----------------------------------------------------------------------------------
please make sure to rate correct answers