IPSec is a standard, and there is nothing in the standard that allows for multicast or broadcast traffic to go through it.
Specifically, the IPSec RFC (ftp://ftp.isi.edu/in-notes/rfc2401.txt) states things like:
A security association is uniquely identified by a triple consisting
of a Security Parameter Index (SPI), an IP Destination Address, and a
security protocol (AH or ESP) identifier. In principle, the
Destination Address may be a unicast address, an IP broadcast
address, or a multicast group address. However, IPsec SA management
mechanisms currently are defined only for unicast SAs.
and
The receiver-orientation of the Security Association implies that, in
the case of unicast traffic, the destination system will normally
select the SPI value. By having the destination select the SPI
value, there is no potential for manually configured Security
Associations to conflict with automatically configured (e.g., via a
key management protocol) Security Associations or for Security
Associations from multiple sources to conflict with each other. For
multicast traffic, there are multiple destination systems per
multicast group. So some system or person will need to coordinate
among all multicast groups to select an SPI or SPIs on behalf of each
multicast group and then communicate the group's IPsec information to
all of the legitimate members of that multicast group via mechanisms
not defined here.
Multiple senders to a multicast group SHOULD use a single Security
Association (and hence Security Parameter Index) for all traffic to
that group when a symmetric key encryption or authentication
algorithm is employed. In such circumstances, the receiver knows only
that the message came from a system possessing the key for that
multicast group. In such circumstances, a receiver generally will
not be able to authenticate which system sent the multicast traffic.
Specifications for other, more general multicast cases are deferred
to later IPsec documents.
Sorry to quote RFC's at you, but we simply follow the standard and the standard didn't support this. You can work around it by defining a GRE/IPSec connection, but all that's really doing is encapsulating the broadcast/multicast into a unicast GRe packet first, then encrypting that unicast packet.
