One of the more common ticket-generating topics that we see from our Cisco IronPort Email Security Appliance(ESA) customers, are delivery related issues. Among the various types of delivery issues that can occur, a common question is why a message was "queued for delivery." The purpose of this blog is to unveil any mystery about messages that have a last status of "queued for delivery," and to provide some basic troubleshooting technics regarding this topic.
What can cause delivery delays? How does IronPort message delivery work?
There are many variables involved, so it is difficult to say what is causing specific messages to be delayed, without investigating each occurrence individually. However, here is a basic overview of how delivery works on the ESA:
Queues:
Once a message is fully scanned through the workqueue, the message recipients are divided up by destination domain in the delivery queue. The ESA scans the destination domain queues on an ongoing basis; the more destinations in memory, the longer each pass of scans will take. If the system is under heavy load, this can delay the delivery queue scans.
Network:
Messages that are larger will take longer to deliver than smaller messages. If the network connection or path to the destination server is slow for any reason, mail delivery will be slow as well. If there are network errors reaching a particular destination server, the messages will be re-queued for another connection. If there are DNS errors looking up a destination domain or all hosts are unreachable, all mail for that domain will be re-queued until the issue is resolved. Network and/or issue at the destination server end are the most common reason for a message to be stuck as "queued for delivery."
Regardless of the reason for the re-queue, there will definitely be a soft bounce code logged somewhere in the mail logs. It is also important to note that if a particular destination domain was not reachable when the ESA attempted to deliver previous messages addressed to the same domain, all proceeding messages addressed to that same domain will not show any immediate delivery attempts. This is primarily why you will find messages with a last status of "queued for delivery."
The next delivery attempts for the unreachable destination domain in question will be made in the background and not tied to any known MID number. Only successful connections will be tied to an MID. This is what can often make this type of issue difficult to troubleshoot. The frequency of delivery attempts is controlled by the 'Initial Period to Wait Before Retrying an Unreachable Host' parameter in the global settings of your ESAs Bounce Profile. While these retries are made in the background, you can actually locate them in the mail logs. Locating these retries requires that you know the IP address(es) of the destination domain in question. Use the IP(s) as your characters string, when using 'grep' against the mail logs. This will show each retry and more importantly, the bounce that was logged. Viewing the bounce that occurs during a failed delivery will often help you understand why the delivery attempts have been failing.
As an example, the following grep was run against a known IP address of a partular destination domain. This particular example shows connection errors occuring between the ESA interface and the destination IP address.
IronPortESA1.example.com> grep '206.132.3.41' mail_logs
mail.@20120328T152739.s:Wed Mar 28 15:27:41 2012 Info: New SMTP DCID 42466271 interface 10.33.130.150 address 206.132.3.41 port 25
mail.@20120328T152739.s:Wed Mar 28 15:27:41 2012 Info: Connection Error: DCID: 42466271 domain: target.bfi0.com IP: 206.132.3.41 port: 25 details: EOF interface: 10.33.130.150 reason: network error
mail.@20120328T152739.s:Wed Mar 28 15:33:53 2012 Info: New SMTP DCID 42466709 interface 10.33.130.150 address 206.132.3.41 port 25
mail.@20120328T152739.s:Wed Mar 28 15:33:53 2012 Info: Connection Error: DCID: 42466709 domain: target.bfi0.com IP: 206.132.3.41 port: 25 details: EOF interface: 10.33.130.150 reason: network error
mail.@20120328T152739.s:Wed Mar 28 15:52:30 2012 Info: New SMTP DCID 42467905 interface 10.33.130.150 address 206.132.3.41 port 25
mail.@20120328T152739.s:Wed Mar 28 15:52:30 2012 Info: Connection Error: DCID: 42467905 domain: target.bfi0.com IP: 206.132.3.41 port: 25 details: EOF interface: 10.33.130.150 reason: network error
mail.@20120328T152739.s:Wed Mar 28 16:25:36 2012 Info: New SMTP DCID 42470002 interface 10.33.130.150 address 206.132.3.41 port 25
Once you know the IP address(es) of the destination server in question, you can also attempt to telnet to them, from the ESA CLI-over port 25. If there is still a delivery issue, this will manifest itself in real-time. You can then use the tools and suggestions in the "How to troubleshoot deliver issues" Knowledge Base article link below, to gain additional information on resolving the issue at hand.
Additional Resources:
