Finally got an answer to this one. We are on version 4.3.1, need to upgrade to 4.3.2 for a fix.
Queueing order not preserved after EAAS restart
Symptom:
After an EAAS process restart or MR PIM failover queue order is not preserved anymore so older emails may appear in the queue after newer emails
Conditions:
After EEAS process restart or MR PIM failover
Queueing order is not preserved anymore, so older emails are later in the queue than newer.
Workaround:
None
This behavior of EAAS is As Designed. The reason behind this is that EAAS is multi-threaded in nature (based on design) to avail better performance, and when you start/re-start the EAAS process and instance, it kicks off multiple internal threads each of which pick-up a batch of activities in the order of queue priority or activity priority or due date, and the submission to ICM need not be in the same order due to the multi-threaded nature. It's the very nature of a multi-threaded application or service and therefore, the order is not necessarily FIFO.
Though this is as per designed behavior in 4.3(1), development is currently in progress for enhancing the current design to ensure queuing order in EAAS after a re-start. Hence re-opening this defect and this will be fixed in 4.3(2).
Like the bug says above the issue is also triggered if there is a MR PIM failover when you restart Cisco Services on your PG. The bug has been fixed in 4.3.2 and there has been no workaround at the moment.
