ESA URL filtering logging

[antoniyganchev]

I am following the "ESA URL Filtering Enablement and Best Practices" guide:

http://www.cisco.com/c/en/us/support/docs/security/email-security-appliance/118775-technote-esa-00.html

It works as shown BUT when I send a message with TWO OR MORE links, it would only show ONE of them. E.g. I sent a message with google.com and cisco,com, and got this in the message log:

Thu Mar 23 18:09:21 2017 Info: MID 687 interim verdict using engine: CASE spam negative
Thu Mar 23 18:09:21 2017 Info: MID 687 using engine: CASE spam negative
Thu Mar 23 18:09:21 2017 Info: MID 687 interim AV verdict using Sophos CLEAN
Thu Mar 23 18:09:21 2017 Info: MID 687 antivirus negative
Thu Mar 23 18:09:21 2017 Info: MID 687 AMP file reputation verdict : CLEAN
Thu Mar 23 18:09:21 2017 Info: MID 687 URL http://cisco.com has reputation 8.77090249266 matched url-reputation-rule
Thu Mar 23 18:09:21 2017 Info: MID 687 Custom Log Entry: URL found
Thu Mar 23 18:09:21 2017 Info: MID 687 Outbreak Filters: verdict negative
Thu Mar 23 18:09:21 2017 Info: MID 687 queued for delivery

Basically, I have 2 questions. 

1. Does this mean that the ESA only checks one link (a bug)?

2. If it checks all, how can I can force it to report all URLs?

Thanks

[Libin Varghese]

Hi,

All URLs in the emails are scanned by the URL filtering feature.

I would think the displayed URL is the only one that matched the filter created by you.

To log all URLs I would recommend adding a filter with URL reputation range -10 to +10 as well as no URL reputation. The action can be just a log entry like you have currently.

Thank You!

Libin Varghese

[antoniyganchev]

Hi Libin,

This is exactly what I have: one filter for -10 to 10 and one for no reputation. The -10to10 filter is as follows:

CCondition

URL Reputation

url-reputation(-10.00, 10.00 , "")

Action

Add Log Entry

log-entry("URL found")

The links I tested are from high reputation domains - e.g. google.com. google.ca cnn.com cisco.com etc

There seems to be no logic whatsoever - try it yourself.

Try with 2,3,4 domains, change the order of the links, etc ... It always reports one, but there is no logic in which one - e.g. sometimes the it reports the first one, sometimes it reports the last one, and the alphabetical order does not seem to matter.

[antoniyganchev]

I played a bit more. I added filters from 0 to 5, 5 to 10 and NoRep.

If I send these 3 links, it works as expected:

http://google.com

http://qweqweqweqweqw.com

http://mail.ru

------->

Fri Mar 24 13:47:28 2017 Info: MID 709 AMP file reputation verdict : CLEAN
Fri Mar 24 13:47:28 2017 Info: MID 709 URL http://google.com/ has reputation 8.05838754054 matched url-reputation-rule
Fri Mar 24 13:47:28 2017 Info: MID 709 Custom Log Entry: URL found 5to10
Fri Mar 24 13:47:28 2017 Info: MID 709 URL http://mail.ru has reputation 1.96267748896 matched url-reputation-rule
Fri Mar 24 13:47:28 2017 Info: MID 709 Custom Log Entry: URL0to5
Fri Mar 24 13:47:28 2017 Info: MID 709 URL http://qweqweqweqweqw.com has reputation noscore matched url-reputation-rule
Fri Mar 24 13:47:28 2017 Info: MID 709 Custom Log Entry: URL NoRep
Fri Mar 24 13:47:28 2017 Info: MID 709 Outbreak Filters: verdict negative
Fri Mar 24 13:47:28 2017 Info: MID 709 queued for delivery

--------<

Now, I add some more domains that fall into these groups - e.g. cnn into 5 to 10 group, abv.bg in the 0 to 5 group and another nonsense domain. This is where it goes wrong:

http://google.com

http://qweqweqweqweqw.com

http://mail.ru

http://abv.bg

http://cnn.com

http://121212123434343.com

The results is again only 3 domains:

------>

ri Mar 24 14:04:16 2017 Info: MID 713 AMP file reputation verdict : CLEAN
Fri Mar 24 14:04:16 2017 Info: MID 713 URL http://cnn.com has reputation 5.03835548289 matched url-reputation-rule
Fri Mar 24 14:04:16 2017 Info: MID 713 Custom Log Entry: URL found 5to10
Fri Mar 24 14:04:16 2017 Info: MID 713 URL http://abv.bg has reputation 1.55489991465 matched url-reputation-rule
Fri Mar 24 14:04:16 2017 Info: MID 713 Custom Log Entry: URL0to5
Fri Mar 24 14:04:16 2017 Info: MID 713 URL http://qweqweqweqweqw.com/ has reputation noscore matched url-reputation-rule
Fri Mar 24 14:04:16 2017 Info: MID 713 Custom Log Entry: URL NoRep
Fri Mar 24 14:04:16 2017 Info: MID 713 Outbreak Filters: verdict negative
Fri Mar 24 14:04:16 2017 Info: MID 713 queued for delivery

------<

[antoniyganchev]

Any ideas or I should open a tac?

Thanks

[Libin Varghese]

After review I was able to locate the below defects which match the description.

URL Logging, does not write all URLs in mail_logs

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCva41817/?reffering_site=dumpcr

SDS URL Category filter not categorizing URLs

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz05197

These defects are not yet marked as fixed and hence this would be expected behavior.

- Libin V

[joelbland]

Libin,

Do you know when this fix is planned for release?

Thanks

[Libin Varghese]

Hi,

We are expecting a fix to be available for this in Async OS 11, however no confirmed ETA on the release yet as it is being evaluated and tested.

- Libin V

[joelbland]

We are still experiencing this issue in AsyncOS 11.0.0-264. Is there an estimate on when this will be resolved?

Thanks