Compromised Internal SMTP servers

[Jason Meyer]

This week I have had three (of the roughly 4000) internal SMTP servers that are behind our IronPort appliances either get compromised with a virus or have an application generate high volumes (500K) of e-mails.

For the SMTP server that was comprimised with a virus the e-mails it was generating was a phishing attempt asking for recipients e-mail addresses and passwords.  Our outgoing e-mail policies do scan for viruses and SPAM and found the e-mail to be clean.  The e-mail contained a link that was valid.  By the time that I was alerted to the fact about 230K e-mails had gone out to external recipients and the workstation had been powered off.  I was able to get one of the e-mails and did a TRACE on it and at that time (roughly 24 hours after the event) the e-mail was detected as SPAM.  So, no custom filter was built for the specific e-mail to block future ones.

For the applications, a coding error was to blame causing lots of e-mails to be generated and then routed through our IronPort appliances to our internal Exchange server.  Our Exchange server was keeping up with the load but was queueing up more e-mail in the transport queues than normal, this is what alerted me to this problem.  I created custom filters on our IronPort appliances to block these e-mails (to protect our internal Exchange system) until the developers could get their servers stopped.  Some of the e-mail was legitimate from these servers so I could not just block their connection attemps by HAT.

In thinking about this problem I probably should have some mail flow limits on our internal SMTP servers, currently they are set at unlimited e-mails.

Is there a good way to determine what those limits should be?

Is Cisco/IronPort working on any logic that can learn a 'normal' amount of e-mail for an internal server and then when volume goes beyond this 'normal' volume and a throttling mechanism automaticaly begins?

Are there any alert mechanisms that can tell me when an internal SMTP server is going beyond it's normal volume?

Other than using Mail Flow Policy to throttle sending are there any good ways to prevent the above from happening?

I appreciate all shared thoughts.  Oh, and of course, long live the IronPort Nation.

[Christopher Smith]

Hi Jason,

The question of outbound limits can be a hard one to answer. This is because every customers enviroment and needs are different. Typically most customers I have worked with have very loose limits outbound as they don't want to restrict outbound traffic. Having little or no restrictions outbound can also be a problem as you can end up in a situation where you are trying to push out more traffic than a specific host will accept. This is not uncommon with domains like Yahoo. As you have seen if an internal system becomes compromised this can turn into a much larger issue. Ideally you would want try to put together some data on what your average outbound traffic is like during a normal day and look at the trends within that data to understand what your average is like. Once you have that data you can use that to compare against what your settings are in your outbound policy.

One addtional thing to keep in mind is that the antivirus scanning works well outbound however outbound spam scanning (including web reputation rules) will not be nearly as accurate or effective outbound as they are for inbound traffic.

Currently there is not much in the way of built in monitoring or live reporting on outbound delivery. There are several feature requests for some similar functionality so it is very possible that something like this will be added in a future release of AsyncOS. Currently the best way to deal with this is throught the use of the outbound policy, and or filters.

Christopher C Smith
CSE
Cisco IronPort Customer Support. 

[Jason Meyer]

Appreciate the input Chris, I wanted to make sure I wasn't missing anything that could easily be done.  Currenlty I am looking at the mail volumes to try to determine what kind of limits I can set.  The first hurdle that I'm running into is it the limits are set in "recipients" per hour and the reports that I'm find are "messages" per hour.  Am I correct that these are very different numbers, one message could have a thousand recipients?

Jason