Hi,
You can whitelist only IP addresses and sending server hostnames in HAT (sendergroup) and can use incoming mail policy to whitelist domains. The mail is processed in the following manner (just providing overview):
HAT -->.... RAT...-->Incoming Policies (various engines Anti-spam-->Antivirus--> etc.) -->Delivery Queue
For more details, please refer the below link:
https://www.cisco.com/c/en/us/td/docs/security/esa/esa12-0/user_guide/b_ESA_Admin_Guide_12_0/b_ESA_Admin_Guide_chapter_011.htmlTo answer your other query on precedence, HAT will take precedence over incoming mail policy as it works at the initial connection level to the ESA appliance.
I hope the above helps.
Cheers,
Pratham