Hello Thaung,
Encryption in this case i would imagine is SSL/TLS encryption.
Attached is a comprehensive guide for this:
https://www.cisco.com/c/en/us/support/docs/security/email-security-appliance/118844-technote-esa-00.html
Otherwise, in short.
Exchange to ESA
This is ICID level, you can check the TLS settings for this in GUI > Mail Policies > HAT Overview
Exchange should be in the RELAYLIST (or sendergroup associated to a RELAY mail flow policy).
Click on the associated mail flow policy (Should be RELAY/RELAYED) and scroll down to security features.
Verify TLS setting (preferred or required).
Preferred means it can fall back down to unencrypted plain text if TLS fails.
For ESA to Exchange.
This is DCID level, you can verify these settings in Mail Policies > Destination Control.
If there is no entries, it will use the DEFAULT and TLS settings set here.
Else I would recommend you to add a new entry, add your recipient domain here.
Enable the TLS settings to your requirements.
After done, commit changes.
You will be able to see on the Message Tracking/mail_logs the TLS protocol and cipher when it's negotiated successfully.
Regards,
Matthew