We have a CSVB installed. Now there is the problem that emails with big attachments are not going through the CSVB. I found that you can define the max message size under "mail flow policies". Is there also an option on the CSVB that you can generate an SMTP Error message if somebody is sending an email with a too big attachment?
usually an SMTP response code is not required as the sending mail server should honor the SIZE advertisement in the SMTP banners after sending the EHLO or HELO command. The SIZE advertisement on the CSVB is directly taken from your Mail Flow Policy configuration here, so in fact the sending mail server should not even try to send the message if he knows that the message size is too big to be accepted ;-)
However, if a sending mail server is happily ignoring the SIZE advertisement, the CSVB will generate an SMTP response code with 'Message size too big' during the SMTP data conversation. This can then also be found in the mail_logs of the CSVB.
So to answer your question, apart from the Masimum Message Size configuration in the respective Mail Flow Policies there's no other configuration required to inform the sending mail server that messages over this size will not be accepted.