11-18-2015 10:10 AM
In our environment we have 4 X1060 appliances sending and receiving our internet email. We are still on 7.6.1 on these boxes but are currently pursuing an upgrade to 680's.
For now though because of the load on these, we are stuck on 7.6.1 for several more weeks. The issue we have is the boxes run fine with minimal if any workqueue backups daily except on Wednesdays. For a while it was Wednesday mornings beginning perhaps as early as 7:00 am EST but now is closer to noon. Our 4 hosts go from averaging 200 or less messages in the workqueue to suddently 20,000 plus in the workqueue on all 4.. This usually takes a couple hours to clear and users complain of 30 minute plus latency in receiving their email.
We have beat this issue to death in our environment looking at DNS, firewalls, switches, bandwidth etc. I have analyzed the messages coming through and find nothing unusual. Size is average, messages per minute is normal, nothing unusual. I don't see any indication of snowshoe spam or anyone flooding us with connections.
Is there anything that happens weekly on Wednesdays that may explain this? Major CASE rule updates? Virus updates? We use CASE, SOPHOS, SBRS. It just seems the boxes can do lett work for a couple hours on Wednesdays.
I have opened cases on this, but usually get great pushback at our old version. We can't upgrade these hosts as we have been told the newer versions would decrease our capacity. We are in the process of putting in 8 680's to replace these but that is several weeks away.
Thanks in advance for any ideas on this.
11-19-2015 10:05 AM
Tony,
While I hadn't noticed it previously, we ran into this issue yesterday (Wednesday) with the (3) C670's we have that are in a cluster. From approximately 7:00 am PST to 11:45 am PST, we saw a huge spike in our workqueue. We had up to approx. 7,200 messages in each of the workqueues, which caused delays of close to 25 minutes. Our appliances are running the latest and greatest version of AsyncOS: 9.7.0-125.
I checked our updater logs and didn't see any updates having taken place during these times.
11-19-2015 02:48 PM
Hey Tony,
I know this may not be the ideal response, but I would indeed like to recommend to open a case if the issue is present so we can review it for you; as with a case and tunnel access we can check what is in the workqueue and causing the problem.
However if that's not possible, some useful grep commands i would suggest to monitor the amount of emails or types (perhaps there is a large influx of a particular email coming in at these specific times that can be mitigated).
From the CLI:
grep “MID.*Subject” mail_logs
This will return outputs of all new message and their subject in an individual line for easier troubleshooting and less clutter for log reviewing
grep “ready” mail_logs
This command will return outputs of all emails which has been fully accepted for workqueue processing with the sender of the email in question
grep -i “new smtp icid” mail_logs
This command is extremely useful to see if you’re being attacked by a common sending address who is spamming your system that you do not trust, buy locating the common sender IP, you can add them directly to the BLACKLIST to reject it at the connection point
tophosts
This is to show the emails in your delivery queue and notice if there is any hosts which generally has a lot more emails than others
showrecipients
This will show you the emails for that host in more detail such as subject, sender, MID, and how many retries of delivery.
Perhaps there is a large amount of emails you're not actually meant to be receiving and this may be the one impacting your queue.
Once more details are known, you can then start using either message filters to remove invalid emails at the workqueue level based on your gathered information.
To diagnose if your services may be having some form of failure, checking the mail_logs to see which process is taking more time than other (or message tracking) can give an indicator.
Regards,
Matthew
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide