Will, you need to put the both ip addresses for both ISP's in the crypto map of the remote site, this can be done by:
crypto map set peer
Also on the site where you have the two ISPs you need to apply the same crypto map on both interfaces facing both ISPs.
P.S: You will need to configure SLA on the ASA facing the two ISPs (which I asume you alredy have for ISP failover).
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: